Drop a stock EC2 instance into its UEFI shell, ask the firmware to bring up the network, and you get silence. The firmware has no driver for the network card AWS put in the instance.

I want to be fair to Amazon first, because the reason is good engineering rather than neglect. AWS builds its instance firmware from EDK2, the open-source UEFI, as a patchset they publish, rebuild reproducibly with Nix, and own end to end. They treat it the way firmware should be treated: a small, slow-moving, heavily-audited thing whose one job is to boot the operating system and then get out of the way. The place where features and churn and risk are meant to live is the OS. The firmware is boring on purpose.

A NIC driver is the opposite of boring. ENA is a large, evolving, DMA-driving piece of device code, precisely the surface you do not want to carry in an audited root-of-trust firmware to serve a use case nobody has. And nobody has it, because the firmware does not need the network: every operating system AWS expects you to boot brings its own ENA driver seconds later. OVMF, which AWS's firmware builds on, happens to bundle one network driver, VirtioNetDxe for virtio NICs. That is why the same firmware on Google Cloud, where the NIC is virtio, comes up with a working network. On EC2 the NIC is ENA, Amazon's Elastic Network Adapter, nothing binds it, and the entire IP and TCP stack compiled into the firmware sits there with nothing underneath it.

A completely sound set of trade-offs, right until you turn out to be the rare thing that wants to do networking in UEFI mode: a loader that runs before any operating system and has to fetch the thing it boots. That is stage0, and on EC2 it died at IP4_CONFIG2 NOT_FOUND, because there was no network for it to bring up. To netboot EC2 you have to bring your own NIC driver.

So one had to be written.

The plan

The goal is narrow: produce an EFI_SIMPLE_NETWORK_PROTOCOL, the firmware's notion of "a network card you can send and receive frames on." Produce that, and EDK2's existing IP4, DHCP and TCP stack binds straight on top of it. We are not reimplementing TCP/IP. We are filling the one missing layer at the very bottom and letting the firmware's batteries do the rest. (Writing our own stack with something like smoltcp was on the table for about an hour and then sensibly thrown back.)

The driver is substantially original work, written against Amazon's published ena_com specification on GitHub. Most of the Rust itself was written by Claude, working from that spec. The existing ENA implementations across Linux, DPDK, FreeBSD and iPXE were read for reference, none of them ported from. Working from the spec rather than any one implementation is also what keeps it MIT/Apache, clear of the GPL several of them carry. It is a normal UEFI Driver-Model driver: inert on any machine whose NIC it doesn't recognise (so it does nothing on GCP or in local QEMU), and on EC2 it binds the ENA device and brings it up. It ships embedded inside stage0 as a single measured unit, which matters for reasons we'll get to.

Building it (the easy 80%)

ENA is a reasonable piece of hardware with a documented interface. You set up an admin queue, a pair of DMA rings for submitting configuration commands and reading their completions. Through it you query the device's attributes, read its MAC address, negotiate features, and then create the IO queues, the actual send and receive rings. Then you wire the send and receive paths to the SNP methods the firmware will call.

Most of this is a mechanical translation of the spec, and most of it worked on the first real boot. The admin queue came up. Device attributes returned. The MAC address read back correctly. Every feature negotiation returned success. We got all the way to creating the first completion queue.

The wall

CREATE_CQ, the command that creates a completion queue, returned admin status 6. In ENA's status table that is UNKNOWN_ERROR, the device's catch-all for a rejection it will not explain. Everything around it was healthy. The admin queue worked. Every other command returned 0. The device's status register read ready, with no fatal-error bit, before and after the failure.

Each attempt to change its mind was a full round trip: build the driver, turn the disk into an AMI, launch a real c6i.large, scrape its serial console, power it off. Two dozen reboots that each end in the same five-character error message wear on you. What made it bearable was getting that round trip down to a minute or two, and the tool that did it is worth a short detour.

Skip vmimport: coldsnap and the EBS direct API

The obvious way to boot a locally-built disk image on EC2 is AWS VM Import. You upload the image to an S3 bucket, start an import job, and then you watch it. And watch it. VM Import crawls: it sits at 19% for minutes, then 73% for minutes, to ingest a stage0 disk that is under a megabyte. It also wants an IAM role named vmimport whose whole job is to hand an internal Amazon service broad standing access to your S3 and EBS, which you get to create and manage so the slow thing can run at all.

None of that is necessary. coldsnap, an AWS Labs tool (Rust, as it happens), talks the EBS direct API: it writes your raw disk straight into an EBS snapshot, block by block, and skips every block that is all zeros. No S3 bucket. No import job. No vmimport role, just three narrow EBS permissions on your own identity. For a sub-megabyte, mostly-empty disk it finishes in seconds, because it only ships the blocks that contain anything. Register an AMI from the snapshot and launch.

With coldsnap in the loop, going from a changed line of Rust to the first line of serial output off a fresh instance is a minute or two. The hidden cost of the slow version is not the minutes, it is that a slow loop makes you conservative about what you test, and this was a bug that only ever fell to being willing to try the twenty-fourth thing.

Everything that was wrong

The dead ends are the useful part of a writeup like this, so here they are, roughly in the order I believed them:

  • Queue depth. iPXE creates 32-entry rings. I assumed newer Nitro had raised some minimum, so I tried 256, then the device's own advertised maximum of 1024. Identical failure each time. This was the first instance of a recurring mistake: inventing a magic number and tuning it.

  • MSI-X, the big one. Modern drivers create completion queues in interrupt mode, so I enabled the device's MSI-X capability. Then I programmed all six interrupt table entries with valid (but masked) x86 local-APIC messages. Then I tried it with a real vector, with the "no vector" sentinel, in interrupt mode, in poll mode. Every single combination failed identically, which in hindsight was the tell: the device was not even looking at this. I was carefully adjusting a knob the firmware had already ignored.

  • LLQ. Low Latency Queue, where the driver writes packet descriptors directly into device memory. Nitro v4 enables "wide LLQ" by default, and I convinced myself the firmware gated all queue creation on completing the placement handshake. I negotiated it fully, matched the device's supported feature masks exactly. No change.

  • The async event queue. I initialised its head doorbell the way Linux does, then enabled the real event groups specifically so the device could post me an event explaining the rejection. It posted nothing.

  • Register write ordering. I reordered the admin-queue setup to match ena_com's exact sequence, on the theory that Nitro wired subsystems in a specific window. No change.

  • Readless MMIO. One device register kept reading back as zero, so I suspected direct register reads were unreliable on Nitro. Linux has a whole mechanism for reading registers over DMA instead. I implemented it, purely to compare the two. The values were identical. The register that read as zero was simply write-only.

  • The DMA-address theory. The admin queues, whose addresses I write into device registers, work. The completion queue's address is supplied inside a command instead. I spent two rounds convinced that Nitro's IOMMU treated a command-supplied address differently from a register-supplied one. This is physically impossible, a DMA address is a DMA address once it is on the bus, but it felt plausible enough that I researched UEFI's DMA mapping semantics for an hour before admitting it.

And the meta-mistake, the one actually worth carrying away: for about fifteen of those rounds I kept asserting the CREATE_CQ command was "byte-for-byte identical to what Linux sends." I had reasoned my way to that conclusion, field by field. I had never once dumped the actual bytes. When I finally did, they were perfect, which immediately told me the bug was not in the command at all. Do not reason about a wire format you are capable of reading.

The one u32

So the command was perfect and the device was healthy and it still said no. The gate had to be somewhere in the device's state, set earlier, not in the command itself. Everything Linux establishes before queue creation, I now also did. The one remaining difference was something I had dismissed as cosmetic.

ENA lets the driver register a host info page: a block of metadata about the driver, sent once via SET_FEATURE(HOST_ATTRIBUTES). Operating system type. A spec version. And a driver_version. I had set the OS type and the spec version. I left driver_version at zero, the way any driver written from scratch would.

On c6i, the Nitro firmware refuses to create a completion queue unless that driver_version has a major byte of at least 2.

That is the entire bug. One u32 at offset 172 of a metadata page. And here is why it cost 24 reboots: the command that registers the page returns success. The firmware accepts your page, stores it, and only consults it much later, at queue-creation time, where it fails with a generic error that names nothing. The cause and the symptom are three admin commands and an entire debugging session apart.

I did not find this by deduction. I found it because two other people had been here first: an iPXE commit fixing this exact symptom on c6i.large, and an amzn-drivers bug report from someone writing an illumos ENA driver who hit the same anonymous status 6 on newer Nitro. Set the major version to 2, rebuild, reboot. The completion queue created. The send and receive queues created. DHCP got a lease. The instance fetched its own metadata over a network that, when this started, did not exist. Twenty-four rounds for one byte.

I miss the Sun days

It is worth sitting with how much work that was to make one network card talk, and with how normal that has become. The NIC is excellent, its specification is public on GitHub, and the integration is yours. That is the cloud bargain in miniature: fast, cheap, unbundled parts, and the wiring that used to be a vendor's job is now yours.

I'll cop to some nostalgia. Sun shipped machines whose firmware, OpenBoot, was a whole Forth environment that could netboot the box straight out of the carton, because Sun built the silicon and the firmware and the OS and felt some obligation to make them meet in the middle. That obligation is what got unbundled. You can rebuild it, clearly, one ENA register at a time. But every so often you want to boot a machine from a signed URL before any operating system exists, and you discover the batteries were never included.

There is a real payoff here too, beyond the nostalgia. The reason stage0 exists is to make a boot you can prove: the firmware verifies stage0, and stage0 verifies and measures the payload it loads, so a remote party can attest the whole chain. A NIC driver we wrote, pinned and measured as part of stage0, is one we can include in that proof. A firmware network blob we don't control would not be. So the same work that makes EC2 netboot at all is the work that keeps the chain trustworthy end to end. Owning the bottom of the stack is extra work, and it is also what makes the boot provable.

If you are writing your own ENA driver and you have found this post by searching for CREATE_CQ and status 6: set your host-info driver_version to a major version of 2.