Introduction

Hello there! In this blog, we’ll be demonstrating how to make your local MQTT Broker communicate with a pod in your Google Kubernetes Engine (GKE) cluster. We’ll achieve this by sending messages with a simple Python script and safely storing the payloads in a Redis cloud database. This blog will be your constant companion, providing step-by-step instructions, useful commands, and links to make the entire setup process a joy.

MQTT Protocol

MQTT, which stands for Message Queuing Telemetry Transport, is a lightweight messaging protocol designed for efficient communication between devices in the IoT (Internet of Things) and other resource-constrained environments. It follows a publish-subscribe messaging pattern, allowing devices to send and receive messages in a distributed system.

The MQTT protocol is based on the client-server architecture, where clients connect to a central broker. The broker acts as an intermediary, responsible for receiving and distributing messages between clients. Clients can be both publishers and subscribers, meaning they can send messages (publish) and receive messages (subscribe) on specific topics.

MQTT Broker

An Mqtt Broker is a software application that facilitates communication between devices or applications using the MQTT protocol. It acts as a central hub, receiving messages published by devices and distributing them to the intended recipients based on their subscriptions. MQTT brokers enable efficient and reliable messaging in constrained environments, such as IoT systems, by managing message flow, authentication, access control, and quality of service levels. They play a crucial role in building responsive and real-time systems by enabling scalable and asynchronous communication.

Setting Up a HiveMQ MQTT Broker

To set up the HiveMQ MQTT broker, you can follow these general steps:

1. Obtain HiveMQ: Visit the HiveMQ website (https://www.hivemq.com/developers/community/)and download the HiveMQ MQTT broker. Ensure that you choose the appropriate version for your operating system.

2. Installation: Click on the GitHub repository link once the site is loaded on your browser. It will redirect to the Hivemq-community-edition repository.

3. Download the Community Binary Package: Scroll down to in the Readme.doc to the HiveMQ Community Forum section and click on the HiveMQ binary package link and your download will start.

4. Start the script: Navigate to the bin folder in the binary package folder of the hivemq-ce-(version) and run the script based on you’re the operating system on which you are running the HiveMQ. Run.sh for Linux machines and Run.bat for windows machines.

5. Enable Security (Optional): If you require secure communication, HiveMQ supports various security options such as TLS/SSL. Enable and configure security settings in the configuration file, including certificates, encryption, and authentication.

6. Adjust Broker Settings: Review and modify other HiveMQ broker-specific settings in the configuration file, such as message persistence, maximum connections, maximum message size, and QoS levels.

7. Start the HiveMQ Broker: Save the configuration file and start the HiveMQ broker. The specific method to start the broker depends on your operating system and the method of installation you used.

1. Test Connectivity: Use an MQTT client or tool to connect to the HiveMQ broker and publish/subscribe to test the connectivity. You can use GUI-based MQTT clients like MQTT Explorer. You can download and install it from https://mqttx.app/

2. Monitor and Troubleshoot: HiveMQ provides various monitoring and diagnostic features to monitor the broker’s health and troubleshoot issues. Familiarize yourself with the available tools and resources provided by HiveMQ for monitoring and troubleshooting.

NGROK (for making the MQTT Broker publicly accessible)

NGROK is a cross-platform tunneling software that allows you to expose your local server to the internet by creating secure tunnels. It is commonly used during web development and testing to share your locally hosted applications with others or to integrate webhooks with services that require a publicly accessible URL.

To install NGROK, follow these steps:

1. Visit the NGROK website: Go to the official NGROK website at https://ngrok.com/You will find the latest version of NGROK available for download.

2. Sign up for an account (optional): Creating an account is optional but recommended, as it   allows you to access additional features like custom subdomains, more concurrent connections, and usage statistics. You can sign up for a free account on the NGROK website.

3. Download NGROK: After signing up or logging in, download the appropriate version of NGROK for your operating system (Windows, macOS, or Linux) from the downloads page.

4. Extract the downloaded file: Once the download is complete, extract the contents of the NGROK zip file to a directory of your choice.

5. Configure authtoken (optional): If you signed up for an account, you can configure your authtoken by running the following command in your terminal or command prompt:

ngrok authtoken YOUR_AUTH_TOKEN

Replace YOUR_AUTH_TOKEN with the token provided to you after signing up. This step enables additional NGROK features for your account.

6. Run NGROK: Open your terminal or command prompt, navigate to the directory where you extracted the NGROK files, and execute the appropriate command for your operating system:

Windows: ngrok.exe

macOS/Linux

./ngrok

This will start NGROK and display the available options.

7. Create a tunnel: To create a tunnel, use the following command:

ngrok tcp

Replace PORT_NUMBER with the port number on which your local server is running. Here in our case hivemq is running on port 1883. NGROK will generate a public URL that you can use to access your local server from the internet.

Ngrok tcp 1883

8. This will provide us with a ngrok url which makes this mqtt broker accessible over the internet.

9. Now this url can be used as a Broker address for the communication setup.

Redis

Redis is an open-source, in-memory data storage system that can be used as a database, cache, or message broker. It keeps data in memory for fast access and low latency. It supports different data structures and offers commands for data manipulation. Redis can persist data to disk for durability and has replication and clustering capabilities for scalability. It is commonly used as a cache and has features like publish/subscribe messaging. Redis is versatile, fast, and widely used in web applications, caching, analytics, and more.

Redis Cloud database setup

To create a Redis Cloud Database directly from the Redis website, you can follow these steps:

1. Visit the Redis website: Go to the official Redis cloud signin/signup website at https://app.redislabs.com/#/

2.Sign up or log in: If you don’t have an account on Redis Cloud, you will need to sign up by providing the necessary details. If you already have an account, log in using your credentials.

3.Select a plan: Redis Cloud offers different plans based on your requirements, including free and paid options. Choose a plan that suits your needs in terms of memory capacity, data persistence, replication, and other features. Review the pricing details associated with each plan.

4.Configure the database: Provide the required information to configure your Redis Cloud database. This may include choosing the cloud provider (e.g., AWS, GCP, Azure), selecting the region or data center location, specifying the desired memory size, and setting up any additional options or advanced configurations.

5.Set up security: Configure the security settings for your Redis Cloud database. This typically involves setting a strong password or access key and enabling encryption if desired. You may also have options to define network access restrictions or firewall rules.

6.Review and confirm: Double-check all the configuration details and settings you have chosen. Ensure that everything is accurate and meets your requirements. Once you are satisfied, confirm the creation of the Redis Cloud database.

7.Wait for provisioning: The Redis Cloud platform will now provision and set up your database. The time required for this process may vary depending on the chosen plan and cloud provider.

8.Access and connect to the database: Once the provisioning is complete, you will receive connection details such as hostname, port number, and authentication credentials. These details are needed to establish a connection from your applications or client libraries to the Redis Cloud database.

9.Start using Redis: With the connection details, you can now connect to your Redis Cloud database and begin utilizing its features. Refer to the Redis Cloud documentation or guides for instructions on how to interact with the database, manage data, and optimize performance.

Remember to monitor your Redis Cloud database regularly, adjust configurations as needed, and take advantage of any additional features or services provided by Redis Cloud to enhance the performance, scalability, and security of your Redis deployment.

Python Script 

Now in this section, we will create a python script to establish the connection with the MQTT broker which is installed on our local system and now is accessible through the url URL obtained from the previous steps mentioned before.

importpaho.mqtt.clientasmqtt

importredis

# MQTT broker connection details

broker_address = “0.tcp.in.ngrok.io”

broker_port = 16992

# Redis connection details

redis_host = “redis_host_url”

redis_port = 15412

redis_password = “redis_password_value”

# Connect to Redis

redis_client = redis.Redis(host=redis_host, port=redis_port, password=redis_password)

# MQTT on_connect callback

defon_connect(client, userdata, flags, rc):

    ifrc == 0:

        print(“Connected to MQTT broker”)

        client.subscribe(“test”)  # Subscribe to MQTT topic(s) here

        client.publish(“test”, “Device connected”)  # Publish a message to MQTT topic

    else:

        print(“Failed to connect to MQTT broker, return code:”, rc)

# MQTT on_message callback

defon_message(client, userdata, msg):

    topic = msg.topic

    payload = msg.payload.decode(“utf-8”)

    qos = msg.qos

    print(f”Received MQTT message – Topic: {topic}, Payload: {payload}, QoS: {qos}”)

    # Store MQTT message in Redis

    redis_client.set(“logs”, payload)

    print(“Message stored in Redis”)

# Create MQTT client and set callbacks

mqtt_client = mqtt.Client()

mqtt_client.on_connect = on_connect

mqtt_client.on_message = on_message

# Connect to MQTT broker

mqtt_client.connect(broker_address, broker_port)

# Start MQTT client loop

mqtt_client.loop_start()

# Prompt for topic and message

topic = input(“Enter the topic: “)

message = input(“Enter the message: “)

# Publish the message to MQTT topic

mqtt_client.publish(topic, message)

# Keep the script running

whileTrue:

    pass

Creating a Dockerfile

Now to use this script inside a pod we need to create a Dockerfile, which is a collection of steps to build a docker image. Here we are creating a Dockerfile which that will use this python script to establish the MQTT broker connection when the container or pod is created using this image

1. Dockerfile 
2. # Use a base image with Python installed
3. FROMpython:3.9
5. # Set the working directory in the container
6. WORKDIR/app
8. # Install required dependencies
9. RUNpipinstallpaho-mqttredis
11. # Copy the script to the container
12. COPYtest.py/app/test.py
14. # Set the MQTT and Redis connection details as environment variables
15. ENVBROKER_ADDRESS=”NGROK URL”
16. ENVBROKER_PORT=16921
17. ENVREDIS_HOST=”REDIS_URL”
18. ENVREDIS_PORT=15412
19. ENVREDIS_PASSWORD=”REDIS_PASSWORD”
21. # Run the Python script
22. CMDpythontest.py

2. Now to build an image from this Dockerfilewe need to execute the following command:

docker build -t .

3. Now you can see the image will be created and to check whether the docker images is created successfully:

# docker images 

It will display your image with a the latest tag and image id ID.

GKE Google Kubernetes Engine

Google Kubernetes Engine (GKE) is a managed service provided by Google Cloud Platform (GCP) that allows users to deploy, manage, and scale containerized applications using Kubernetes. It simplifies the process of running Kubernetes clusters by handling many of the underlying infrastructure tasks, such as cluster setup, scaling, and monitoring.

With GKE, users can create and manage multiple Kubernetes clusters easily, providing a robust and reliable environment for deploying containerized applications. GKE integrates with other Google Cloud services, enabling seamless integration with storage, networking, and monitoring tools.

Overall, GKE simplifies the management of Kubernetes clusters and provides a scalable and reliable platform for deploying containerized applications, allowing developers to focus on building and running their applications rather than managing the underlying infrastructure.

GCR (Google Container Registry)

Before creating a cluster, we need to push our customized docker image to a container repository in the Google Container Registry so that the image can be pulled by the cluster while creating the pod. To do these following commands needs to be executed:

1. Authenticate Docker with the GCR service:

gcloud auth configure-docker

2. Tag the Docker image with the GCR registry URL:

docker tag mqtt-image asia.gcr.io/[PROJECT-ID]/mqtt-image

3. Push the Docker image to GCR:

docker push asia.gcr.io/[PROJECT-ID]/mqtt-image

GKE autopilot setup

In this section we will be creating an autopilot cluster in the Google Kubernetes Engine. Following these steps will setup the autopilot cluster:

1. Go for the urlhttps://cloud.google.com and search for Kubernetes Engine and hit enter it will redirect you to the Google Kubernetes Engine page.
2. Click on Create, by default GKE provides you an option to create an autopilot cluster so that most of the things inside a cluster are managed by the Kubernetes engine itself. But if you want to manage all by yourself you can go for the standard method. Here we choose the autopilot method.

3. Name your cluster and select your region.

4. After naming your cluster and selecting the region click on Next for networking configuration and choose the network and mode of cluster whether it is going to be public or private cluster, which in our case will be a public cluster and then click on nxt for Advanced settings.

5. Now select the cluster release channel, and we choose Regular channel which is the default option. And Then we review and create the cluster. 

6. Now review the configuration for once and then click on Create cluster. It will take some time to create the cluster so we’ll have to wait till the time it gets provisioned.

Connecting with the cluster 

Now to connect with the cluster we have two options either copy the gcloud command to use the local system after getting it authenticated or else use the Cloud shell.

1. Run the following command on your system and authenticate and enter the cluster 

gcloud container clusters get-credentials mqttcluster –region asia-south1 –project

2. This command will authenticate you and you may interact with you cluster. Now create a manifest file named as pod.yaml with the following configurationwhich is as follows:

1. apiVersion: v1
2. kind: Pod
3. metadata:
4.   name: first-pod
5. spec:
6.   containers:
7.     – name: mqtt
8.       image: asia.gcr.io/idan-dev/mqtt-img
9.       ports:
10.         – containerPort: 1883
11.       resources:
12.         requests:
13.           cpu: “2”
14.           memory: “4Gi”
15.       command: [“python”]
16.       args:
17.         – “-u”
18.         – “-c”
19.         – |
20.           import paho.mqtt.client as mqtt
21.           import redis
22.           broker_address = ‘NGROK_URL’
23.           broker_port = 16921
24.           redis_host = ‘REDIS_HOST_URL’
25.           redis_port = 15412
26.           redis_password = ‘REDIS_PASSWORD’
27.           redis_client = redis.Redis(host=redis_host, port=redis_port, password=redis_password)
29.           def on_connect(client, userdata, flags, rc):
30.               if rc == 0:
31.                   print(‘Connected to MQTT broker’)
32.                   client.subscribe(‘test’)
33.                   client.publish(‘test’, ‘Device connected’)
34.               else:
35.                   print(‘Failed to connect to MQTT broker, return code:’, rc)
37.           def on_message(client, userdata, msg):
38.               topic = msg.topic
39.               payload = msg.payload.decode(‘utf-8’)
40.               qos = msg.qos
41.               print(f’Received MQTT message – Topic: {topic}, Payload: {payload}, QoS: {qos}’)
42.               redis_client.set(‘logs’, payload)
43.               print(‘Message stored in Redis’)
45.           mqtt_client = mqtt.Client()
46.           mqtt_client.on_connect = on_connect
47.           mqtt_client.on_message = on_message
48.           mqtt_client.connect(broker_address, broker_port)
49.           mqtt_client.loop_start()
50.           while True:
51.               pass

3. Now create a pod using the following commands:

kubectl apply -f pod.yaml

Now it will take some time to create a pod and once it is created, we’ll be able to see that the pod is connected with the MQTT Broker which is running on my local system. Your connection is now established between your cluster and your MQTT broker.