Zarar's blog

The em-dash that broke the server

Bad news for developers who let AI agents write configuration files. There's a whole class of bug waiting for you which will cost you a day. They're entirely invisible until some script chokes on it harder than Team USA.

I was building out infrastructure-as-code for a small fleet of servers following the usual pattern of baking a machine image and letting each new server configure itself on first boot from a cloud-init script. The agent wrote the cloud-init file, I skimmed it, it looked pretty good and I booted the server expecting the server to be ready for primetime. But the script never ran and it came up as a bare Ubuntu box, every trace of setup scrubbed clean like a red card that happened but also never happened. None of the setup ran, leaving me in that unenviable position of debugging agent work. Shudder.

These are the worst kind of failures because they're quiet and even (correctly) confirmed by cloud-init status which said done even though it had done less work than the Brazilian team. The logs showed the base-image steps running fine. It was as if the config file simply didn't exist, much like the Iraqi defense.

I SSH'd in and pulled the raw config the server had actually received. It was there, byte for byte what we wrote. So the file arrived. cloud-init just refused to run it. I went digging in /var/log/cloud-init.log, and buried in the noise:

Failed loading yaml blob. unacceptable character #x0080: special characters are not allowed
Failed at merging in cloud config part: empty cloud config
Skipping modules 'runcmd' because no applicable config is provided.

There it was staring right at me: #x0080. A non-ASCII byte which had no business being there, kind of like the Uruguayan squad. The YAML parser hit one character it didn't like and threw away the entire script. To cloud-init, our config was now empty so it skipped runcmd (the part that runs the script) and moved on.

The culprit

I went back to the file and looked at the comments, those things we used to write but now let agents do it like everything else. One of them read:

# Rendered by Terraform per droplet — each node gets its own identity

That dash between "droplet" and "each" isn't a hyphen. It's an em-dash: —. Unicode U+2014, three bytes in UTF-8, and one of those bytes is 0x80. The exact character in the error.

We all know what an em-dash is, it's that thing that gives away that an AI wrote something. Apparently they don't just sneak their way into emails, but also in config files. And it turns out a 15-year-old YAML parser has the same reaction to it as humans: ugh.

The lesson

This is a brand new way for a script to fail and it's specific to working with agents. A human writing a config file at a terminal almost never types an em-dash because they have some self-respect. I mean, there isn't even a key on the keyboard for that thing. An LLM produces them by default, even in machine-consumed files which don't care about prose and where a single exotic byte can void the whole document with no error a human would notice in review.

And you won't catch it by reading the file. I read that file. It looked perfect, much like a Spanish striker's finish against a hapless Portugese backline in the 91st minute. Em-dashes and hyphens are visually near-identical at a glance, much like smart quotes and straight quotes. Your eyes will glaze right over it like a 0-0 Switzerland-Qatar thriller enjoyed by none and suffered by all.

So the fix isn't to be more careful because who's careful nowadays? It's to see if we can make it impossible:

I added the grep check to the build. It fails now at commit time instead of silently on a booting server at 2 a.m. This cost me an afternoon and the check costs nothing.