I've been looking for a good introduction to LLMs for embedded firmware for a bit and recently found myself needing a UASP(USB Attached SCSI Protocol) module where I could control the SCSI commands.

Claude already knows about UASP, and ESP chips and their ESP-IDF SDK, so it seemed like a great fit.

Getting started

In VS-Code on Windows, with the ESP-IDF and Claude plugins, Claude got a first pass ready, but it wouldn't enumerate correctly, and wanted me to run it on Linux where we could use dmesg and usbmon. Makes sense, but super tedious.

I booted up my Linux disk in VirtualBox only to find that partially enumerated devices cannot be passed through to VMs. After a couple edit-reboot-run-debug loops, I moved the project to Linux.

Within VS-Code, I couldn't get Claude to run the build/flash/monitor tools from the ESP-IDF VS-Code plugin, and Claude struggled with sudo, which barked about being unable to enter a password without stdin or something. Claude's responses to this were...really strange and somewhat unsettling. A Linux developer in the same situation would want to fix sudo, but Claude would say something like "ooh this indicates something deeply interesting now I'll fix...", bruh. I almost gave up at this point. Just tell me if you need tools dude.

Mostly working development setup

I made a new user account on Linux, with command-line-Claude, connected the Espressif MCPs for their documentation server and idf.py mcp-server tool which allows building and flashing my board direct from Claude. Each time Claude prompted to run a new command with sudo, I'd allow it into /etc/sudoers.d/, like so:

claude  ALL=(ALL) NOPASSWD: /usr/bin/lsusb

Whenever it wanted me to capture logs, I did it like so:

dmesg -c; rm build/logs/*; idf.py monitor ...wait30secondsMaybeUseTimeout...; mv build/logs/*stdout*/ idf_py_stdout; dmesg > dmesg.log

Eventually Claude would patch the project, build and flash it, I would run it and it would re-read the logs and retry. I'd switch x sessions to my regular account, and when Claude was idle, switch back for another go.

Hilarity ensues before success

At first, Claude ran into a missing USB packet. It immediately dug into the designware USB phy's registers and setup a thread to poll them, even though TinyUSB usually handles this for you.

I came back to 'timeout 20 cat /dev/ttyACM0', command timeout with no data, "fascinating!"

Every time it added some logging, it would also tweak the SCSI IUs slightly, either renaming or moving bytes or sizes around. Often we weren't even getting to these packets so I didn't get what was happening. Eventually I saved my browser tabs of SCSI, USB, and UASP standards to a folder and told it to come back once it grokked them and that seemed to help.

Eventually I suggested it to check the return values of usbd_edpt_xfer() - it was failing.

While Claude knows how UASP works, I don't think Claude was aware that UASP can run with or without USB3 streams. On USB2 and below, it works a little differently. To its credit, Claude read the Linux UASP driver source and rearranged the endpoint transfers to match the streamless protocol.

Several times the context memory reached a point where it was too full and speed ground to a token-burning crawl. Seems to be a commonly reported issue with Claude. Eventually it gets bad enough that creating a new session seemed to fix it.

Suddenly, dmesg started spewing an endless list of SCSI device enumerations. Ever wonder what comes after /dev/sdz? /dev/sdaa of course! Claude's SCSI INQUIRY handler replied success for every LUN, so Linux dutifully kept enumerating them at high speed. This was a little tricky to catch the start of the dmesg log since it was going so fast. Eventually raced it by running `idf.py monitor` then plugging in the USB-port cable and hitting enter. This is a weird little effect that might be fun to play with later, but I asked Claude to report just one LUN...

And then it worked. 6MB wear-leveled block device over USB. Made a filesystem, saved some files, ejected, remounted, seems to be working fine! A whopping 2mbps with loads of serial logging, but the protocol is up!

What went well?

Optimizing read-eval-debug loop

As for traditional software development, make it fast or pay the tax. Embedded is especially awful, since flashing a microcontroller is almost never fast. One big upside is that Claude was kicking this off for me, so I never had to wait for it.

Observability

LLMs are limited by the same observability limitations as any programmer. In the embedded space, observability gets weird as you're often testing against physical objects or effects, and you're often limited by the available tools, and sometimes whittling your own. I picked UASP to do since with this board I just needed two USB ports.

Until Claude could read dmesg and idf.py-monitor logs, it was very much stumbling around blindly in Plato's Cave.

Software tooling like tshark and usbmon are best used if the LLM can be setup to run and analyze them itself. And on the electrical side of things, folks are working on plugins for oscilloscopes, power supplies, and other test equipment too. Part of the process for tuning the edit-run-debug loop is going to be connecting the plumbing for these.

MCP Servers

These are terrific, especially for embedded where they're required for integrating hardware I/O, and also for documentation search. The ESP-IDF build and flash mcp service is only 3 months old, and is the next area for extension - with the ESP32-S3 with dual ports(USB-UART and USB-Native), it'll soon be possible for LLMs to pull logs themselves as well. There is also work going on in the electrical space at Tektronix for this.

Familiarity with protocols

Just like regular meatspace firmware development, knowing how to write f(x) and g(x) makes writing f(g(x)) much simpler.

Document Ingress

Explicitly providing the protocol specs fixed a number of oddities where I have to assume it just didn't remember the details of the SCSI packet layouts. I couldn't either.

Closing

Is it worth it? In short, yes. It's like the trains here - sometimes slower than driving, but I can multitask, play with my nieces, photograph, and sleep. Just being async to my life is a boon.

Though sometimes frustrating when it made silly mistakes and plastered over permissions issues rather than asking for help, LLM development doesn't have to be perfect, and it gets a huge bonus just for being asynchronous.

For the banging-the-head-on-the-wall parts of debugging, it was able to handle most of that, just needing me in the backseat to hand it logs as I passed by.

While it was too cold to dive around here, I was editing some photos and really wished I had a few more bits in the images. Since part of the JPEG compression process is red/green/blue to YUV color conversion, I wondered if decompressing a JPEG with floating-point operations would improve it any.

jFloaty is a fork of STB-image that supports encoding and decoding JPEGs direct to floating-point images, as well as an upgraded PNG encoder with dual 8/16-bit support, and some tools like scaling and dithering. ImageMagick is far more powerful, but at the time didn't support 8-bit to floating-point JPEG decode.

After discussing with libjpeg-turbo's maintainer, dcommander, they it wasn't as simple as I thought - the quality improvements I was seeing were mostly because JPEGs store 10-bit DCT bins for 8-bit images, and they patched libjpeg to support rendering 8-bit JPEGs to a 12-bit surface with a configuration change. This is helpful for software like GIMP and Darktable, who will hopefully pull this in too. I've got a patch up for GIMP and am using it at home - much nicer than an external codec and doing JPEG->PFM->GIMP.

Google has also written a floating-point JPEG codec, jpegli though I didn't see it before making my own. They also have a smarter quantizer than the fixed Q-factor-based quantization tables of most encoders.

My son Felix and I went to Kansas City Oasis, as we do Sundays when his schedule aligns.

People there like to see him because he is so young and cute as a button. Before we left he had fallen asleep in Kate's arms and I packed him into the bucket seat, expecting him to be hungry when he awoke. We parked a few blocks away on Bell st, and when we arrived he was awake and alert and had lots of fun looking at everyone.

Eventually he was hungry and he had a good time sitting on me in the back with his bottle. He received lots of comments on his hair, which we first spotted on an ultrasound.

As it came time to leave we were both surprised by the thunder outside. When I carried him to the window for a peek, I saw hail! I had no umbrella and realized I was woefully prepared for this situation.

Our friend Julie offered to watch him until I could return with our car, and I graciously accepted.

About halfway to the car I found a medium sized black dog. He was scared and wet and shivering in the hail and came up to me immediately. Usually I would be more than happy to drive a lost pet home, but I needed to get to my son. But when I opened the door he hopped right in!

I was reminded of simple logic puzzles I had faced as a child: a hunter has a dog, two rabbits, and only one canoe. How can the hunter cross the stream in the least number of trips without mixing incompatible animals on the shore. I could not leave an unknown dog in the car with my son.

I pushed him out. As I drove toward Oasis he followed me for an entire block. A lot of the people at Oasis have, raise, or love dogs, surely if I took the dog to Oasis, we could figure things out from there. I stopped and opened the trunk and he hopped in and I was able to read his dog tags phone number.

On the way I called his owner, who explained that the thunder had scared him into bolting. Just as I arrived, Julie appeared at the door with Felix. I rolled down the window to ask for just a few minutes - I've got to deliver this dog! Just then the dog's owner arrived and we carefully opened the trunk and clipped on the leash and sent him on his way with a thank-you and handshake. I ran in, thanked Julie, washed my hands, grabbed Felix, explained the situation, and thanked Julie one last time.