Introduction
MQTT is evolving into the de facto protocol in the IoT world due to its topic-based pub/sub messaging pattern, which is quite different from the req/res model in the RESTful world. It requires us to think differently when it comes to designing topic-based APIs, that said, designing in a topic-centralized way.
This document is created with the intention to facilitate MQTT development in terms of message subscribing and publishing, topics design, testing, or even just for fun.
Audience
- Developers who are developing MQTT features.
- Those who are interested in and want to get their hands dirty with MQTT basics.
Goals
- Set up a local MQTT broker and clients for message subscriptions and publications.
- Visualize topics as well as corresponding payloads on the go.
- Inspire topics design or design topics in a standardized way.
Setup MQTT Broker
While there are a bunch of brokers when it comes to Choosing an MQTT broker for your IoT project, the lightweight and open-sourced Mosquitto broker is demonstrated here due to its simplicity and support for MQTT v5 and ACL in its most recent v2 release.
- Install
mosquittoon the host machine(Mac)
Download and installmosquittoas per the guidance from the official homepage.
brew install mosquitto
- Install and run Docker-based Eclipse Mosquitto Broker.
- Pull cedalo/installer docker image
docker pull cedalo/installer:2-macos
- Run from the docker image
docker run -it -v ~/cedalo_platform:/cedalo cedalo/installer:2-macos
- Brings the broker up by running the start script
cd ~/cedalo_platform/
./start.sh
Once started, the following components should be working in the specific ports now
- Eclipse Mosquitto: mqtt://localhost:1883
- Eclipse Streamsheets Web UI: http://localhost:8081
- Mosquitto proxy(Management Center Web UI): http://localhost:8088
Navigate to the Management Web UI to take a look quick at what the dashboard looks like.
Verify message pub & sub
- Makes sure MQTT broker has started
docker ps
The outputs should print the running containers that look like
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bf19c83f9d46 cedalo/installer:2-macos "./docker-entrypoint…" 22 hours ago Up 22 hours loving_kare
e98c90d8deea cedalo/streamsheets:2-milestone "docker-entrypoint.s…" 43 hours ago Up 22 hours 1883/tcp, 6379/tcp, 8080/tcp, 8083/tcp, 27017/tcp, 0.0.0.0:8081->8081/tcp cedalo_platform_streamsheets_1
2722f7f34f1c cedalo/management-center:2 "docker-entrypoint.s…" 43 hours ago Up 22 hours 0.0.0.0:8088->8088/tcp cedalo_platform_management-center_1
3479c5c781c7 eclipse-mosquitto:2-openssl "/docker-entrypoint.…" 43 hours ago Up 22 hours 0.0.0.0:1883->1883/tcp cedalo_platform_mosquitto_1
If the containers were down, try to bring them up
docker run -it -v ~/cedalo_platform:/cedalo cedalo/installer:2-macos
- Creates MQTT client(s) with unique username(s) to connect to the broker.
This is optional if anonymous authentication is allowed, which is not enabled by default as of Mosquitto v2 with the Dynamic security model enabled by default. To enable anonymous authentication, one needs to configallow_anonymousinthe configuration file and restart the broker.
Enables allow_anonymous config
# ~cedalo_platform/mosquitto/config/mosquitto.conf
allow_anonymous true
and restarts the broker to apply the new configuration (if it’s up now)
# ~cedalo_platform
./start.sh
Note: If anonymous authentication is disabled, then both publisher and subscriber should at least hold a client role, from the ACL(Access Control Lists)'s perspective, in order to connect to the broker, otherwise, subscription/publication requests will get denied and end up with an “authentication failed” error.
Here is a screenshot with two clients newly created.
- Subscribes to a test topic
mosquitto_sub -d -k 10 -v -h localhost -u test -P 123456 -t 'test/topic'
Sample outputs:
➜ mosquitto_sub -d -k 10 -v -h localhost -u test -P 123456 -t 'test/topic'
Client (null) sending CONNECT
Client (null) received CONNACK (0)
Client (null) sending SUBSCRIBE (Mid: 1, Topic: test/topic, QoS: 0, Options: 0x00)
Client (null) received SUBACK
Subscribed (mid: 1): 0
Client (null) received PUBLISH (d0, q0, r1, m0, 'test/topic', ... (26 bytes))
test/topic {"greeting": "Cool MQTT2"}
Client (null) sending PINGREQ
Client (null) received PINGRESP
Client (null) sending PINGREQ
Client (null) received PINGRESP
...
Notes:
-
-dindicates to work in debug mode, printing verbose debug info. For example, thePINGREQandPINGRESPpairs show an alive session. -
-kchanges the keep-alive interval value from the default 60 to the specified. -
-vprints the topic name of the published messages. -
-hhostname (or IP address) of the machine the broker is running on. Here, as it’s running in the container on the local machine, solocalhostis used to accept pub/sub requests from terminal clients on the same machine. Other clients running on e.g. a real iPhone/Android device, or a simulator/emulator should use the IP address of the dev machine so as to be able to connect to the broker. -
-pnetwork port to connect to. Defaults to 1883 for plain MQTT and 8883 for MQTT over TLS. -
-u-Puser name and password of the client to authenticate with. Optional if anonymous authentication is allowed supported. -
-tthe target topic to subscribe to.
Other useful options that can be used
-
-Cstops subscribing and automatically disconnects once the specified amount of messages were received. -
-qchanges the QoS level in the pub/sub requests. -
-c-x 60-i ciddisables clean session and shares session states for the same client ID in the new connection. which means the session will never expire after the client disconnects (prior to MQTT v5) or will last at leastxseconds (60 here) before it expires (for MQTT V5+), starting from the time of the most recent connection.
- Publishes test message to a test topic
mosquitto_pub -d -r \
-h 192.168.7.123 \
-u test -P 123456 \
-t 'test/topic' \
-m "{\"greeting\": \"Cool MQTT2\"}"
Sample outputs:
Client (null) sending CONNECT
Client (null) received CONNACK (0)
Client (null) sending PUBLISH (d0, q0, r1, m1, 'test/topic', ... (26 bytes))
Client (null) sending DISCONNECT
Notes:
-
-rindicates the published message will be retained in the broker so that subsequent clients that subscribe to the same topic are able to receive the published message immediately once connected. -
-mspecified the message payload to be published to the target topic'test/topic'. Note that JSON-formatted payload should be escaped. Freeformatter can be used for this purpose.
Here is a screenshot that depicts the activities coming from the broker, terminal publisher, terminal subscriber, Android publisher/subscriber.
- The running broker dynamically prints the sub/sub events from clients.
- The Android client log dumped from
adb logcat -v -t time | grep mqttkotlinsample
- A terminal client subscribing to updates from the test topic.
- A terminal client publishing message to the test topic.
Take also a look at the dynamic topics inside the broker from the Topic Inspector
MQTT Topic Naming Scheme
As we know, topics are organized in a hierarchical way, to keep the hierarchical topic tree flexible, it is important to design the topic tree very carefully and leave room for future use cases.
- MQTT Topic and Payload Design Notes
- Topic-hierarchy-centralized way
Put as much information in the topic fields as possible - Payload-centralized way
Put as much information in the payload as possible
Some principles for topics naming scheme should follow - Human readability
- Traffic minimization
- Granular access control
Some Best Practice suggested by HiveMQ team:
- Never use a leading forward slash.
- Never use spaces in a topic.
- Keep the topic short and concise.
- Use only ASCII characters, avoid non-printable characters.
Last Will And Testament Message (LWT)
LWT is used to notify subscribers (through the broker) that the publisher has lost connectivity somehow, for example, suffered from an unexpected shutdown.
- Starts a publisher with configured LWT message
mosquitto_pub -d -r -q 2 \
--repeat 3 --repeat-delay 30 \
-h 192.168.7.123 -p 1883 \
-u test -P 123456 \
-t 'test/topic' -m "{\"greeting\": \"Last will and testament test\"}" \
--will-topic 'test/lwt' --will-qos 2 --will-payload "publisher has lost connectivity"
--repeat 3 --repeat-delay 30 sending repeatable messages to simulate the publisher is working
--will-topic, --will-qos, --will-payload sends LWT message to the specified topic.
- Starts a subscriber monitoring the LWT message on the will topic
test/lwt
mosquitto_sub -d -k 10 -v \
-h localhost \
-u test -P 123456 \
-t 'test/lwt'
- Kills the publisher and observes if the LWT message was received.
kill -9 `ps aux | grep mosquitto_pub | awk '{print $2}'`
Observed LWT message
...
Client (null) sending PINGREQ
Client (null) received PINGRESP
Client (null) received PUBLISH (d0, q0, r0, m0, 'test/lwt', ... (31 bytes))
test/lwt publisher has lost connectivity
Client (null) sending PINGREQ
Client (null) received PINGRESP
...
QoS Management
- QoS 0, At most once - the message is sent only once and the client and broker take no additional steps to acknowledge delivery (fire and forget).
- QoS 1, the message is re-tried by the sender multiple times until acknowledgment is received (acknowledged delivery).
- QoS 2, the sender and receiver engage in a two-level handshake to ensure only one copy of the message is received (assured delivery).
The two-level and four-step handshake flow ensure the exact delivery in the case of QoS 2.
Note that the QoS value can be applied to either a sub or a pub request, however, the QoS level that’s eventually used to deliver the message from the broker to the subscriber is determined by the lower of the two, for example:
- subscribe message with
qos = xtotopic. - publish a message with
qos = ytotopic. - deliver the message with
qos = min(x, y).
Security Concerns
Authentication
The methodology and procedure to validate that users are who they claim to be before they are allowed to log into a secure system and eventually get access to the system resources. It corresponds to the AUTH flow in the MQTT context.
Authorization
The process of giving the user permission to access a specific resource or function in a secure system, which corresponds to the ACL (Access Control Lists) rules for topics in the MQTT context.
- Mosquitto introduces the Dynamic Security plugin which features role-based topic authorization.
- EMQ X provides client pub/sub ACLs for Permission Management.
User name and password-based authentication
Certificates (TLS) based authentication
The goal here is to use a client certificate rather than plain username password text for authentication.
Prons:
- Encrypt and transport credentials and MQTT payload in a secure way.
- Generate and manage certificates on a per-client basis.
Cons:
MQTT TLS Security in Mosquitto
- Generate CA certificate (for self-signed certificates purpose)
openssl req -new -x509 -days 365 -extensions v3_ca -keyout ca.key -out ca.crt
Note: Just in case of “Error Loading extension section v3_ca” error on MacOS, append the following config to the end of OpenSSL config file.
# sudo vi /etc/ssl/openssl.cnf
[ v3_ca ]
basicConstraints = critical, CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer:always
To inspect details about the generated cert file
openssl x509 -inform pem -in ca.crt -text
- Prepare server certificate (to be signed by trusted CA), which is required by the MQTT broker running in TLS mode.
openssl req -out server.csr -key server.key -new
Create self-signed server certificate
openssl x509 -req \
-in server.csr \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-out server.crt \
-days 365
Note: The CN(Common Name, eg, fully qualified host name) field provided when filling up the certificate details is very important. So be careful to provide it.
Here is an example to explore the details in the generated server.crt using openssl command
- Prepare client certificate (to be signed by trusted CA) which is required to connect to the MQTT broker.
openssl req -out client.csr -key client.key -new
Create a self-signed client certificate
openssl x509 -req \
-in client.csr \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-out client.crt \
-days 365
As of self-signed certificates no longer working in Android Q, the self-signed *.pem certificate should be exported to *.p12 format so as to be installed on Android 10+ devices.
openssl pkcs12 -export -in server.crt -inkey server.key -out server.p12
- Config server certificates
- Move
certstoconfigfolder so that they can be accessed thru the shared folder between the broker (running in the docker) and the host machine.
➜ pwd && ll .
/Users/nling/cedalo_platform/mosquitto/config/cert
total 72
-rw-r--r--@ 1 nling staff 1.4K Apr 20 00:19 ca.crt
-rw-r--r--@ 1 nling staff 1.8K Apr 20 00:19 ca.key
-rw-r--r--@ 1 nling staff 17B Apr 20 01:13 ca.srl
-rw-r--r--@ 1 nling staff 1.1K Apr 20 01:11 client.crt
-rw-r--r--@ 1 nling staff 1.0K Apr 20 01:11 client.csr
-rw-r--r--@ 1 nling staff 1.6K Apr 20 01:10 client.key
-rw-r--r--@ 1 nling staff 1.1K Apr 20 01:13 server.crt
-rw-r--r--@ 1 nling staff 1.0K Apr 20 01:13 server.csr
-rw-r--r--@ 1 nling staff 1.6K Apr 20 01:12 server.key
-rw-r--r--@ 1 nling staff 2.4K Apr 20 15:20 server.p12
- Modify Mostiqutto config to apply the server configs.
(➜ cert vim ../mosquitto.conf)
listener 1883
#allow_anonymous true
persistence true
persistence_location /mosquitto/data/
plugin /usr/lib/mosquitto_dynamic_security.so
plugin_opt_config_file /mosquitto/data/dynamic-security.json
listener 8883
cafile mosquitto/config/cert/ca.crt
certfile mosquitto/config/cert/server.crt
keyfile mosquitto/config/cert/server.key
tls_version tlsv1.2
require_certificate true
use_identity_as_username true
log_type all
Note: It’s possible to listen on both 1883 and 8883 ports.
- Export the 8883 port in the docker config so that the broker is accessed from outside the docker, failing to do so will lead to a “Connection Refused” error.
# ➜ cert vim ../../../docker-compose.yml
services:
mosquitto:
image: eclipse-mosquitto:2-openssl
ports:
- 1883:1883
- 8883:8883
networks:
- cedalo-platform
volumes:
- ./mosquitto/config:/mosquitto/config
- ./mosquitto/data:/mosquitto/data
- Restart the broker to bring it up again, listening on 8883 port.
- Initiate a test subscribe request from the terminal, using the client certificate.
mosquitto_sub -d -k 10 -v \
-i DSN000001 \
-h liot.com -p 8883 \
--cafile ca.crt --cert client.crt --key client.key \
-t test/topic
Congrats if you got the output look like
Client DSN000001 sending CONNECT
Client DSN000001 received CONNACK (0)
Client DSN000001 sending SUBSCRIBE (Mid: 1, Topic: test/topic, QoS: 0, Options: 0x00)
Client DSN000001 received SUBACK
Subscribed (mid: 1): 128
Client DSN000001 sending DISCONNECT
All subscription requests were denied.
Outputs from the docker console
mosquitto_1 | 1618914506: New client connected from 172.20.0.1:63288 as DSN000001 (p2, c1, k10, u'client.liot.com').
mosquitto_1 | 1618914506: Client DSN000001 disconnected.
Note 1: As the hostname of the broker is configured to be liot.com in the CN field, it’s not reachable unless a new entry is configured in the hosts file.
# ➜ cert sudo vim /private/etc/hosts
...
192.168.7.123 liot.com # change to the IP address of the host machine
...
Note 2: Pub/Sub requests from a certificate-authenticated client will be denied due to restrictions introduced in Dynamic Security, so a fake user that stands for clients of this kind should be created and assigned client role, so that they are able to pub and sub. Note that the user name is from the CN (Common Name) field defined in the client certificate.
Comparison Between MQTT and TLS Based Traffics
MQTT by default transports payload in plain text mode, whereas MQTT over TLS will encrypt and decrypt the payload in a secure way.
- Publish test message thru MQTT
mosquitto_pub -r -d \
-u test -P 123456 \
-h 192.168.7.123 -p 1883 \
-t 'test/topic' -m "hello there"
- Publish test message thru MQTT over TLS
mosquitto_pub -d \
--cafile ca.crt --cert client.crt --key client.key \
-h server.liot.com -p 8883 \
-t 'test/topic' -m "hello there, this is secure message"
Here are some references about MQTT over TLS
- Choosing an MQTT broker for your IoT project
- Mosquitto SSL Configuration -MQTT TLS Security
- Creating and Using Client Certificates with MQTT and Mosquitto
- Generating a TLS certificate for mosquitto
- Enable SSL/TLS for EMQ X MQTT broker
- Mosquitto TLS Manual Page.
- SSL and SSL Certificates Explained For Beginners
- How To: DER vs CRT vs CER vs PEM Certificates and How to Convert Them
- HiveMQ. MQTT & MQTT 5 Essentials. 2020, 04/27/2021.
- HiveMQ. X509 Client Certificate Authentication - MQTT Security Fundamentals. 05/25/2015, 04/27/2021.
Others
How to open multiple terminals in docker?
docker exec -it bash
docker ps
docker exec 3479c5c781c7 cat /mosquitto/data/dynamic-security.json
References
- Steve Cope. Using The Mosquitto_pub and Mosquitto_sub MQTT Client Tools- Examples. 02/21/2021, 04/16/2021.
- Homieiot. An MQTT Convention for IoT/M2M.
- OwnTracks. Topics
- MQTT SmartHome. Smart home automation with MQTT as the central message bus.
- Steve Cope. MQTT Last Will and Testament Examples. 02/19/2021, 04/16/2021.
- Mqtt-smarthome.
- MQTT Wiki Page. 04/19/2021.
- OASIS. MQTT Version 5.0 Spec. 03/07/2019, 04/19/2021.