Publishing without downtime

Content on the unikernel is baked into the disk image, so publishing anything means rebooting the guest. One image holds 34 sites, so that reboot is not scoped to the site being published — it is everyone’s outage at once.

The deploy script’s own commentary measures the cost precisely, and it is worse than a brief refusal:

qemu’s hostfwd binds the port the instant qemu starts, so for the ~190 ms between “qemu up” and “guest listening” nginx CONNECTS successfully and the client eats a reset. An L4 router cannot retry past that.

A unikernel has one address space and no exec handoff, so this cannot be fixed inside the guest. The fix is one layer up: run two.

Guests: blue :9443 · green :9445
Sites: 34
Certificates: 38
Image: 135 MB
Bake time: ~1.3 s / 743 MB
Rollback: one flip, no rebuild

The shape

Two guests run permanently on two ports. A deploy bakes into whichever one is not receiving traffic, tests it there, and only then moves the nginx SNI router across with a graceful reload. The old guest keeps running untouched, which makes rollback a second flip rather than another rebuild.

Five gates. The first four abort with traffic still on the old guest; the fifth has already moved traffic, so it reverses it instead.

Every gate is a scar

None of these were designed up front. Each exists because something got past the checks that came before it.

Pre-flight: probe drift

The smoke test iterates a hardcoded list of domain names. The set of domains actually baked lives in a different file. Two hand-maintained lists, and nothing reconciled them — so a domain could be baked and never smoke-tested, and a broken bake for it would sail through and reach traffic.

They agreed only because someone had updated both by hand. Now a mismatch aborts before the bake and names the offenders.

Pre-flight: certificate expiry

An expired certificate used to surface only as a TLS handshake failure during the smoke test — after a full bake and boot, as a confusing symptom rather than a named cause.

The check warns at 21 days and refuses to bake on anything already expired. On its first run it found 16 certificates expiring in 8 days, all failing to renew silently: they used certbot’s standalone authenticator, which binds port 80 to answer the challenge, and nginx had taken that port during the unikernel migration. Every renewal had failed for weeks, logged and unread.

Pre-flight: content sanity

GET / → 200 is a weaker assertion than it looks. Path resolution ends at /index.html, so every path returns 200 — a site stripped to a single file still answers cleanly on every URL.

So the check compares file count and total bytes per site against the last successful deploy, and refuses a drop beyond 20%. Tested by gutting a site from 6 files to 1: the deploy aborted, naming both the count and the size change, and traffic never moved.

The baseline advances only after a verified flip. Otherwise a site could erode in stages, each one within tolerance of the last.

Smoke, then witness

Smoke proves each name returns 200 with a matching certificate on the standby’s own port. The witness probe then fetches one real file per site and checks its exact byte count — something the /index.html fallback cannot satisfy, because that file is a different size. It proves what the guest serves matches what is on disk: a bake that did not take, a truncated file, a docroot resolved wrongly.

13 sites have no witness — they contain only index.html, or nothing small enough with a plain enough name — and rely on the other gates.

Post-flip: the gap nobody had noticed

Everything above validates the standby on its own port. That says nothing about whether nginx is routing to it. A stale upstream include, an SNI map entry still pointing at the old port, a reload that silently kept the previous config — all would survive the flip unnoticed.

So after the flip, the same 34 names are re-probed through :443, the real client path, with full certificate validation. On failure it flips straight back to the guest that is still running.

Tested by removing one domain from the SNI map — baked, certificated, and unroutable:

smoke:    34/34 names serving with a matching cert on :9445
LIVE: green on :9445
  POSTFLIP FAIL timecard.itys.net -> HTTP 000
postflip: 1/34 names FAILING through :443 after flipping to green
auto-rolling back to blue
LIVE: blue on :9443

Smoke said 34/34. The public path said 33/34. Detection to rollback took one second, and every request to a neighbouring site returned 200 throughout.

Does it actually hold?

Measured under continuous load — a request every 50 ms, throughout:

Transition	Requests	Failures
Forward flip	357	0
Rollback	237	0
Sabotaged bake (cert removed)	401	0
Deploy with all gates	439	0
Deploy after auto-rollback test	477	0

Over 1,900 requests across five transitions, including two flips, a refused bake, and an automatic rollback. Nothing dropped.

What it still does not do

One host. blue and green are two guests on the same machine. This survives a bad deploy, not a bad machine.

The witness cannot detect deletion. Its manifest is regenerated from the same source it validates, so removing the witness file just selects the next one. That case belongs to the content-sanity check, which compares against history rather than against itself.

A checker’s first failure is usually itself. The witness probe’s first run reported 14 of 21 names broken. The server was fine; the probe built its URL without the port, so it never reached the guest under test and hit whatever serves those names publicly instead. Worth remembering before trusting a new assertion’s first alarm.