My first professional firmware job was at Garmin International, making backpacking products, at a time of my life when I backpacked myself. It was a good fit. Our team would go out a few times per year, picking up lunch, then heading to a large park to test, practice, review, and plan. I really enjoyed making a thing I wanted to use, and knowing how I wanted it to work.

Sometime you pick up a tool, and it's intuitive. It's the right tool for the job. It feels like something you'd've made to solve the same problems you have. Android Auto is not one of those products.

Toyota Bugs

Starting a route or canceling a route in Android Auto causes the compass to glitch back and forth.

The entertainment system clock location varies depending on the current page.

They're starting with a lower-contrast display technology and on top of that, in dark mode, the gauge-cluster clock is gray instead of white just to make it that extra bit more difficult to read.

The entertainment system clock does not always match the gauge cluster clock, and is sometimes off by a daylight savings time.

I cannot rev the internal combustion engine in neutral or park. My car had a rattle that only occurs at certain RPMs, but no way for me to diagnose further. This took half a day at the dealership since we couldn't reproduce it easily.

Having two different voice assistance fighting for the audio is confusing. There are voice assistants buttons on both the steering wheel and the entertainment display and they seem also to be listening all the time anyways. Long press opens Android Auto, and short tap cancels Android Auto. Short press opens Toyota Assist, and long press cancels Toyota Assist. This is, of course, incredibly confusing to both assistants.

There are two speed limit indicators, one on the dashboard and another on the entertainment screen in Android Auto. And they don't always match. The dashboard speed limit indicator is camera based, while the Android one is map-based. This means that the dashboard one is usually wrong after a turn but the Android one is usually correct if available. After passing a speed limit sign the camera one is almost always correct and the Android one is only wrong for a short while until the map is updated. I do not want to have to think about which type of speed limit is most of interest to me. The Android Auto speed limit indicator is placed on the farthest location from the driver which makes it distracting to use. You could display the minimum of both systems in a single location. Did I mention that the dashboard speed limit indicator moves around based on other indicator icons?

Connection between phone and carputer is inconsistent. The actual pairing entry in both devices works, however the link between them often fails to come up until I wade deep into the bowels of the settings on the entertainment unit and once again turn on the checkbox to enable Android Auto for this particular phone. 30 years of Bluetooth and connection management is still terrible.

I do like the fuel efficiency, but wowee this thing takes forever to warm up in the winter.

When it's raining hard, the capacitive door locks don't always open.

Android Auto Bugs

Audio stream management is bad. When a new Pandora song starts in the middle of important turn directions, the turn directions stop jarringly mid-word to let the song start.

But when I try to send a text, it reads it back to me but jumbles the turn instructions together with the outgoing text.

Today after configuring a route my car said, "by the way if you want to listen to the news say hey Google...", then abruptly cut itself off with a "Hi how can I help?" Prompt.

Eventually I figured out how to read the news, but didn't have time to finish it. The next time I started the car it started playing the news not from the beginning of the story, but from the beginning of the days news. I still have yet to finish a day's news.

Much of the user interface seems to be engagement driven, which is wildly out of place while driving. It's great to be able to send a text hands-free, but I have to click play to hear the response. This would be fine, if the button to play the text didn't disappear after a couple seconds while I was executing a lane change. Once it times out the message is seemingly gone until another message comes from the same sender. Of course you can always tap the multitask button, followed by the app list button that appears in its place, then scroll down to messages, then open the message, then play it. A simple untimed dismiss button would have worked too.

Many, many things just work better on my phone.

The quietest car I've owned, a microphone barely six inches from my mouth, music paused, can barely understand my voice. And then Gemini has the gall to say, "You probably mean ${THE_THING_I_ACTUALLY_SAID_ALOUD}" The speech to text process isn't just bad, it's buggy too - often cutting me off mid sentence and trying to process that, or waiting indefinitely for who knows what.

Several times now Gemini has responded by dumping a page of software gore: text prompts. If you ask it what happened, it'll lie and claim it didn't glitch out and prompt dump. It's not good at lying though, since there's a log. It does this at least once a week.

Perhaps this is a misplaced expectation, but I want the Android SmartPhoneAssistant to be accessible hands-free in my car just like using a Bluetooth speaker. I don't want Android DumbCarAssistant ever really.

I used to open an email to myself before driving, and dictate into it. The alternative, which is the Google Android auto note subsystem seems to record, but doesn't actually seem to go anywhere. Perhaps my hardcopy is stuck in the mail?

Using Google Maps in my car makes my phone overheat with "3D Buildings have been disabled". 3D buildings are already disabled both in Google Maps on my phone and car.

Gemini is often unusably slow, sometimes taking tens of seconds to be able to send a text message.

Gemini cannot route me to my Google maps favorites. It knows home, work, and loves making suggestions for restaurants, but only if speech to text can name it correctly - our gym, High Blue, will only route to the local high school.

I've been looking for a good introduction to LLMs for embedded firmware for a bit and recently found myself needing a UASP(USB Attached SCSI Protocol) module where I could control the SCSI commands.

Claude already knows about UASP, and ESP chips and their ESP-IDF SDK, so it seemed like a great fit.

Getting started

In VS-Code on Windows, with the ESP-IDF and Claude plugins, Claude got a first pass ready, but it wouldn't enumerate correctly, and wanted me to run it on Linux where we could use dmesg and usbmon. Makes sense, but super tedious.

I booted up my Linux disk in VirtualBox only to find that partially enumerated devices cannot be passed through to VMs. After a couple edit-reboot-run-debug loops, I moved the project to Linux.

Within VS-Code, I couldn't get Claude to run the build/flash/monitor tools from the ESP-IDF VS-Code plugin, and Claude struggled with sudo, which barked about being unable to enter a password without stdin or something. Claude's responses to this were...really strange and somewhat unsettling. A Linux developer in the same situation would want to fix sudo, but Claude would say something like "ooh this indicates something deeply interesting now I'll fix...", bruh. I almost gave up at this point. Just tell me if you need tools dude.

Mostly working development setup

I made a new user account on Linux, with command-line-Claude, connected the Espressif MCPs for their documentation server and idf.py mcp-server tool which allows building and flashing my board direct from Claude. Each time Claude prompted to run a new command with sudo, I'd allow it into /etc/sudoers.d/, like so:

claude  ALL=(ALL) NOPASSWD: /usr/bin/lsusb

Whenever it wanted me to capture logs, I did it like so:

dmesg -c; rm build/logs/*; idf.py monitor ...wait30secondsMaybeUseTimeout...; mv build/logs/*stdout*/ idf_py_stdout; dmesg > dmesg.log

Eventually Claude would patch the project, build and flash it, I would run it and it would re-read the logs and retry. I'd switch x sessions to my regular account, and when Claude was idle, switch back for another go.

Hilarity ensues before success

At first, Claude ran into a missing USB packet. It immediately dug into the designware USB phy's registers and setup a thread to poll them, even though TinyUSB usually handles this for you.

I came back to 'timeout 20 cat /dev/ttyACM0', command timeout with no data, "fascinating!"

Every time it added some logging, it would also tweak the SCSI IUs slightly, either renaming or moving bytes or sizes around. Often we weren't even getting to these packets so I didn't get what was happening. Eventually I saved my browser tabs of SCSI, USB, and UASP standards to a folder and told it to come back once it grokked them and that seemed to help.

Eventually I suggested it to check the return values of usbd_edpt_xfer() - it was failing.

While Claude knows how UASP works, I don't think Claude was aware that UASP can run with or without USB3 streams. On USB2 and below, it works a little differently. To its credit, Claude read the Linux UASP driver source and rearranged the endpoint transfers to match the streamless protocol.

Several times the context memory reached a point where it was too full and speed ground to a token-burning crawl. Seems to be a commonly reported issue with Claude. Eventually it gets bad enough that creating a new session seemed to fix it.

Suddenly, dmesg started spewing an endless list of SCSI device enumerations. Ever wonder what comes after /dev/sdz? /dev/sdaa of course! Claude's SCSI INQUIRY handler replied success for every LUN, so Linux dutifully kept enumerating them at high speed. This was a little tricky to catch the start of the dmesg log since it was going so fast. Eventually raced it by running `idf.py monitor` then plugging in the USB-port cable and hitting enter. This is a weird little effect that might be fun to play with later, but I asked Claude to report just one LUN...

And then it worked. 6MB wear-leveled block device over USB. Made a filesystem, saved some files, ejected, remounted, seems to be working fine! A whopping 2mbps with loads of serial logging, but the protocol is up!

What went well?

Optimizing read-eval-debug loop

As for traditional software development, make it fast or pay the tax. Embedded is especially awful, since flashing a microcontroller is almost never fast. One big upside is that Claude was kicking this off for me, so I never had to wait for it.

Observability

LLMs are limited by the same observability limitations as any programmer. In the embedded space, observability gets weird as you're often testing against physical objects or effects, and you're often limited by the available tools, and sometimes whittling your own. I picked UASP to do since with this board I just needed two USB ports.

Until Claude could read dmesg and idf.py-monitor logs, it was very much stumbling around blindly in Plato's Cave.

Software tooling like tshark and usbmon are best used if the LLM can be setup to run and analyze them itself. And on the electrical side of things, folks are working on plugins for oscilloscopes, power supplies, and other test equipment too. Part of the process for tuning the edit-run-debug loop is going to be connecting the plumbing for these.

MCP Servers

These are terrific, especially for embedded where they're required for integrating hardware I/O, and also for documentation search. The ESP-IDF build and flash mcp service is only 3 months old, and is the next area for extension - with the ESP32-S3 with dual ports(USB-UART and USB-Native), it'll soon be possible for LLMs to pull logs themselves as well. There is also work going on in the electrical space at Tektronix for this.

Familiarity with protocols

Just like regular meatspace firmware development, knowing how to write f(x) and g(x) makes writing f(g(x)) much simpler.

Document Ingress

Explicitly providing the protocol specs fixed a number of oddities where I have to assume it just didn't remember the details of the SCSI packet layouts. I couldn't either.

Closing

Is it worth it? In short, yes. It's like the trains here - sometimes slower than driving, but I can multitask, play with my nieces, photograph, and sleep. Just being async to my life is a boon.

Though sometimes frustrating when it made silly mistakes and plastered over permissions issues rather than asking for help, LLM development doesn't have to be perfect, and it gets a huge bonus just for being asynchronous.

For the banging-the-head-on-the-wall parts of debugging, it was able to handle most of that, just needing me in the backseat to hand it logs as I passed by.

While it was too cold to dive around here, I was editing some photos and really wished I had a few more bits in the images. Since part of the JPEG compression process is red/green/blue to YUV color conversion, I wondered if decompressing a JPEG with floating-point operations would improve it any.

jFloaty is a fork of STB-image that supports encoding and decoding JPEGs direct to floating-point images, as well as an upgraded PNG encoder with dual 8/16-bit support, and some tools like scaling and dithering. ImageMagick is far more powerful, but at the time didn't support 8-bit to floating-point JPEG decode.

After discussing with libjpeg-turbo's maintainer, dcommander, they it wasn't as simple as I thought - the quality improvements I was seeing were mostly because JPEGs store 10-bit DCT bins for 8-bit images, and they patched libjpeg to support rendering 8-bit JPEGs to a 12-bit surface with a configuration change. This is helpful for software like GIMP and Darktable, who will hopefully pull this in too. I've got a patch up for GIMP and am using it at home - much nicer than an external codec and doing JPEG->PFM->GIMP.

Google has also written a floating-point JPEG codec, jpegli though I didn't see it before making my own. They also have a smarter quantizer than the fixed Q-factor-based quantization tables of most encoders.