burcham.org.uk

For my tinyML project I’ve been evaluating MQTT and HTTP to get data off my board. I’ve also been looking at using BLE.

I’ve been investigating the ArduinoBLE library. It seems quite nice. It looks like it supports Central and Peripheral. At this stage I think I’m mostly interested in using the microcontroller as a Peripheral. It seems well documented. It can be configured to be event driven (useful if I expect the Central to be sending the Peripheral any data). Supports sending data as notifications (and I believe indications).

On the Central side, it looks like my best option will be the python bleak module. I expect most of the code surrounding BLE handling will be python, so that would work well. Again can be configured to be event driven (I saw lots of mention of async, so I suspect this may even be strongly encouraged.

If I need to any python Peripheral work, there is also the python bless module. I haven’t looked at this much yet.

I’ve done extensive work with bluetooth previously. However, so much of it has been implementing bluetooth stacks. There has been much less using it as part of an actual application to do stuff. It might be nice to use it for this purpose.

Having spent the last few days fiddling with MQTT, I have some thoughts and observations worth documenting.

Mosquitto out of the box isn’t configured for anonymous access. This is better for security, but if you haven’t used the software before can be a bit confusing and lead to a bit of head scratching. To allow anonymous access add the following to /etc/mosquitto/mosquitto.conf:

allow_anonymous true
listener 1883

This will allow anonymous access on clear port 1883. Still not interested in using TLS for the local server, so my microcontroller can talk to it easily.

Using JSON in a MQTT value is really nice. It’s trivial to extract it into a python dictionary. Should be easy to construct simple JSON in C on a microcontroller.

Lots of documentation I’ve seen recommends using https://www.pubnub.com/ as a cloud MQTT server. However, it doesn’t support MQTT over TCP any more. It looks like support was cancelled last year. This doesn’t seem to be made very clear at all. In general pubnub documentation doesn’t feel very user friendly. Its very bitty. It has lots of broken links. There’s a fair bit of stale documentation that is no longer relevant, which isn’t marked as such. Because of this I spent quite a long time analysing my firewall config, as nmap suggested port 1883 and 8883 were being filtered (newsflash, they weren’t). There was also a lot of confusion over which URL to use, which as it turned out, due to the aforementioned cancellation were all nonsense. Servers were present in DNS, but did not respond to ping which also didn’t help. ChatGPT and Gemini were also content to lie through their teeth about the service status, which is hardly surprising since their source material was outdated, obfuscated nonsense.

https://www.hivemq.com/ seems much better. It does actually support MQTT over TCP. It has a free, openly accessible test server. It has a free tier with login for basic messing about. It has good documentation. It has a decent AI chat bot that did a good job of answering my noob questions and got me up and running quickly. Open server allows anonymous access, TLS and clear. Logged in server requires TLS and username/password.

It seems like the best way to handle a microcontroller with weak TLS is to have a local MQTT server which can handle TLS, to forward stuff to the cloud. Doesn’t need to be anything fancy, as its not high traffic. Export data from microcontroller to local MQTT using clear MQTT, MQTT-SN, COAP, BLE etc.

For the purposes of my project, its feeling like a combination of HTTP for bulk payload and MQTT for general sensor data and notifications will be best. The clue is in the name: Message Queuing Telemetry Transport. Use it for sensor readings and management.

The paho MQTT python module seems quite nice.

All in all, quite nifty so far.

I’ve just started looking at a TinyML project I want to do. The first thing I need to do is get some sensor data off my board.

I thought about just bunging it over a uart. In some ways it would be the quickest easiest way to go about things. However, this thing is likely to evolve into something a bit more structured over time, with several different data types, which would necessitate building packets and coming up with some sort of protocol. That’s beginning to sound a bit too much like hard work.

Next thing I considered was sending some JSON. That would make the protocol rather easier, as I could just create some JSON on the microcontroller send it over to the host and use any number of JSON parsers to extract the data.

But, if I’m using JSON I should really send it over HTTP. The board has ethernet and I can get an HTTP server up and running without too trouble.

However, I’ve seen plenty of people talking about MQTT in reference to sensor data. I’ve never used it before. No time like the present!

I’ve got mosquitto installed locally. I’ve managed to setup two subscribers. I’ve published some mock data. I’ve played with wildcards. All in the cli on my PC so far, but I only started a bit before lunch so early days yet. So far it shows promise.

I’m not sure if some sort of hybrid approach where I use MQTT for notifications and HTTP for the actual data (which later down the line may be audiovisual) would be a good idea. I’m not sure what the strengths/weaknesses of MQTT are yet. I know HTTP can do all this without too much bother.

Also, not sure about cloud services for MQTT yet. The microcontroller I’m using doesn’t have hardware crypto, so TLS might be a bit pants. If I do everything locally, I don’t need to bother so much about security. The board I’m using has a ton of other gear that looks useful though. I can’t find another board that has equivalent sensors/connectors for other sensors, has crypto support and is a sane price though.

I think for now, keeping it local will suit my purposes. There’s plenty I don’t know how to do yet, so adding another layer of things I don’t know will just add burden and stress. Maybe I’ll do something more cloudy later.

I think now though its time for a walk in the sun!

Having just spent a while working with SD cards over SPI and an STM32 microcontroller, I made various discoveries, which seem worth documenting, since a lot of it I didn’t find anyone else talking about and had to figure out for myself.

1. An SD card connector is a purely passive device. All SPI hardware resides on the SD card itself. This means if there is no card inserted, there is no SPI device to communicate with. As such MISO should be connected to a pull up or pull down resistor so it isn’t left floating. Almost all SPI breakout boards from that I found already have pull up resistors fitted.
2. In theory Chip Select (CS) can be used for card detection. On the microcontroller the GPIO can be configured as open drain. There is meant to be a pull up resistor on the SD card which will pull the line high when inserted, and should go low when ejected. However, as the previous paragraph stated, there’s often pull up resistors on breakout boards which prevent this from working. Further, a lot of boards use a level shifter, which pulls the line low quite hard and completely isolates the pull up resistor on the SD card. Thus, this mechanism is often unusable. If using such a board, CS must be configured as push/pull.
3. On the connector itself there is often a dedicated Card Detect line. This is a mechanical switch on the connector. Often it isn’t electronically connected to anything, so also potentially is unusable.
4. The SD card spec supports both SDIO and SPI. When the card is first inserted it is in SDIO mode until SPI mode is negotiated. That means when you first insert an SD card in the connector you’re putting a non SPI device on the SPI bus. SDIO uses the same pins as SPI and in many ways is a very similar protocol with similar commands. When in SDIO mode, CS is ignored, unless pulled low for cmd0. That’s rather fun isn’t it?
5. An SD card can be inserted at any time, no matter what else might be going on on the SPI bus. If another device is on the bus, the SD card while in SDIO mode (therefore ignoring CS), can start receiving what it thinks are commands and start acting on them. The SD card will have very, very little idea about byte boundaries if inserted in the middle of a SPI transaction. What it receives and chooses to interpret could be anything.
6. While I haven’t done a thorough survey and so can’t prove anything, it seems like SD cards are mostly used in SDIO mode. SPI mode isn’t used much outside microcontrollers (and even then there are microcontrollers which support SDIO). As such, it seems like SPI mode isn’t particularly widely used, tested or supported. Outside pulling CS low during cmd0, I have a suspicion CS gets ignored. This would mean if there are other devices on the SPI bus, the SD card will suck in all data from the bus. Good luck!
7. While soldered down SPI devices can be planned and designed for, an SD card could be any old thing the user decides to plug in. It could be the most reliable device ever, or a pile of junk barely conforming to the spec (or even not supporting SPI at all, and only vaguely working out of chance due to aforementioned similarities between SPI and SDIO).
8. From a brief glance at the SDIO portion of the spec, it looks like SDIO sends responses on what would be MOSI if it were in SPI mode. It would therefore potentially act as some sort of horrid messed up secondary master on the SPI bus, interfering with legitimate operations. I haven’t investigated this at all, so take with a grain of salt.
9. Just put the SD card on a bus by itself. Avoid pain and suffering.
10. When calling HAL_SPI_Receive, the buffer contents passed in get transmitted to the bus unaltered. Always initialise the buffer first. When talking to SD cards, 0xFF seems like an appropriate value.

I hope this information is some day found by some weary traveler wondering why things aren’t working well.

It’s about time I caught up with the 90’s and started my own blog!