Overview

Due to increasing amount of data produced from outside of source systems, enterprises are facing difficulties in reading, collecting, and ingesting data into a desired, central database system. An Edge pipeline runs on an edge device with limited resources, receives data from another pipeline or reads the data from the device, and controls the device based on the data.

StreamSets Data Collector (SDC) Edge, an ultra-lightweight agent, is used to create end-to-end data flow pipelines in StreamSets Data Collector and to run the pipelines to read and export data in and out of the systems. In this blog, StreamSets Data Collector Edge is used to read data of air pressure BMP180 sensor from IoT Device (Raspberry Pi3) and StreamSets Data Collector is used to load the data into Amazon Simple Storage Service (S3) via MQTT.

Pre-requisites

• Install StreamSets
• Raspberry Pi3
• BMP180 Sensor
• Amazon S3 Storage

Use Case

• Read air pressure BMP180 sensor data with IoT Device (Raspberry Pi3) and send to MQTT
• Use SDC to load the data into Amazon S3 via MQTT

Synopsis

• Connect BMP180 temperature/pressure sensor with Raspberry Pi3
• Create edge sending pipeline
• Create data collector receiving pipeline

Flow Diagram

Connecting BMP180 Temperature/Pressure Sensor with Raspberry Pi3

I2C bus, a communication protocol, is used by Raspberry Pi3 to communicate with other embedded IoT devices such as temperature sensors, displays, accelerometers, and so on. The I2C bus has two wires called SCL and SDA, where the SCL is a clock line to synchronize all data transfers over the I2C bus and the SDA is a data line. The devices are connected to the I2C bus via the SCL & SDA lines.

To enable I2C drivers on Raspberry Pi3, perform the following:

• Run sudo raspi-config.
• Choose Interfacing Options from the menu as shown in the below diagram:

• Choose I2C as shown in the below diagram:

Note: If I2C is not available in the Interfacing Options, check Advanced Options for I2C availability.

• Click Yes to enable the I2C driver.
• Click Yes again to load the driver by default.
• Add i2c-dev to /etc/modules using the below commands:

pi@raspberrypi:~$ sudo nano /etc/modules
i2c-bcm2708
i2c-dev

• Install i2c-tools using the below command:

pi@raspberrypi:~$ sudo apt-get install python-smbus i2c-tools

• Reboot the Raspberry Pi3 by running back at the command line using the below command:

sudo reboot

• Ensure that the I2C modules are loaded and made active using the below command:

pi@raspberrypi:~$ lsmod | grep i2c

• Connect the Raspberry Pi3 with the BMP180 temperature/pressure sensor as shown in the below diagram:

• Ensure that the hardware and software are working fine with i2cdetect using the below command:

pi@raspberrypi:~$ sudo i2cdetect -y 1

Building Edge Sending Pipeline

To build an Edge Sending Pipeline for reading the sensor data, perform the following:

• Create an SDC Edge Sending pipeline on StreamSets Data Collector.
• Read the data directly from the device (using I2C Address) using “Sensor Reader” component.
• Set the I2C address as “0×77”.
• Use an Expression Evaluator to convert temperature from Celsius to Fahrenheit.
• Publish data to MQTT topic as “bmp_sensor/data”.
• Download and move the SDC Edge pipeline’s executable format (Linux) to device side, where the pipeline runs on the device side (Raspberry Pi3).
• Start SDC Edge from the SDC Edge home directory on the edge device using the following command:

bin/edge –start=

bin/edge --start=sendingpipeline137e204d-1970-48a3-b449-d28e68e5220e

Building Data Collector Receiving Pipeline

To build a data collector receiving pipeline for storing the received data in Amazon S3, perform the following:

• Create a receiving pipeline on the StreamSets Data Collector.
• Use MQTT subscriber component to consume data from MQTT topic (bmp_sensor/data).
• Use Amazon S3 destination component to load the data into Amazon S3.
• Run the receiving pipeline in the StreamSets Data Collector.

The real-time air pressure data collected and stored is shown in the below diagram:

Conclusion

In this blog, we discussed reading the air pressure BMP180 sensor data from Raspberry Pi3 using StreamSets Data Collector Edge and loading the collected data into Amazon S3 via MQTT using StreamSets Data Collector.