Recent Posts

Subsistence Internet

Jul 31, 2026 • networking, DNS, hardware, monitoring, Proxmox

A storm took my power out for thirty-one hours. The interesting part was over in fourteen minutes.

That is how long the UPS lasted — one UPS, carrying everything. When it quit, all of it went at once: the cluster, the switching, the access points, the gateway.

Fourteen minutes was a great deal less than I had expected, and being that wrong about a number I thought I knew is what sent me looking. I had thirty-one hours to look.

What a Car Battery Taught Me

Somewhere in the dark I wired up a car battery and an inverter.

The first thing that turned up was that both ISPs were still up. Whatever the storm had done, it had not touched their side of the demarc. The Internet had been there the whole time, sitting a few feet from equipment with nothing to run it.

So I fed the modems and my Internet gateway off the improvised supply and brought the path back. The gateway carries a small PoE budget of its own, and that was enough to light one access point — so the WiFi came back too, off a car battery, out of a single box.

And my battery-powered WiFi devices still could not get online.

That sentence is the reason this post exists. I had restored the Internet — carriers, modems, gateway, the entire path — and the devices that wanted it were sitting there with hours of charge in them, and nothing worked.

Without meaning to, I had built the obvious fix and watched it fail. A car battery and an inverter are a crude second power source for the Internet gear, which is exactly the upgrade I would otherwise have bought, installed, and felt clever about. On its own it does nothing.

The Dependency With No Fallback

What I had actually found was that nothing in the home could turn a name into an address.

My DNS servers do two jobs at once. They are authoritative for the internal view of my domain — I run split-horizon DNS, so they answer with internal addresses unknown to public DNS servers. They are also the recursive resolvers for every name outside my domain.

That combination is not a mistake. It is what split-horizon costs. A client is configured with a list of DNS resolvers and uses them for all queries; there is no way to tell a phone to ask one resolver about my domain and a different one about everything else. Choose split-horizon and you have chosen one set of servers that must answer every question every device asks.

Which also means there is no fallback, by construction. I cannot hand clients a public resolver as a backup, because a public resolver does not know my names — it would answer the public questions and break the private ones.

None of that is the error. The error was failing to follow it through, in two directions I had never looked.

I had never noticed that my Internet connectivity depended on my cluster. The ISPs, the gateway, the WiFi — those are the things you would list if someone asked what your Internet depends on, and that night all three of them were working. A device with a perfect path to the Internet and nothing to resolve names with is simply offline, and no amount of health in the rest of the chain rescues it.

And I had never compared the relative power costs. Keeping the Internet and the WiFi alive is a matter of tens of watts — a gateway and an access point, exactly as the car battery demonstrated. Keeping my DNS answers alive meant keeping a three-node Proxmox cluster alive, with its ECC memory and its mirrored storage, which is a different order of magnitude altogether. In terms of power used, I had made the cheapest part of my network depend on the most expensive part, and I had never once put those two numbers beside each other.

The three resolvers were real redundancy, and all of the same kind: three containers on one cluster, which is to say three copies of one dependency in one power domain. Spreading copies across nodes protects you from losing a node. It does nothing about anything all the nodes share.

Two Networks, and Only One of Them Matters at 2 a.m.

The car battery had drawn the line for me without my noticing. What I had chosen to power off it at two in the morning was not a random subset of my equipment — it was a category, and everything I had left dark was the other one. Sketching it out afterward, the gear sorted itself so cleanly that I felt slow for not having seen it before:

The subsistence network — the minimum required to stay connected. The ISP handoffs, the gateway, one small PoE switch, a couple of access points, and a resolver.

The luxury network — everything else. The cluster and its guests, the fast access points, the big switches, file service, cameras, the DVR.

The network names are the useful part. Once you have them, the placement question asks itself: during an outage, is this thing subsistence or luxury? DNS resolution is obviously subsistence. It had been sitting in the luxury tier for years.

Separating the Networks Meant Separating the UPSes

Before the storm there were no tiers, just outlets. One UPS carried the whole load — cluster, switching, access points, gateway, all of it sharing a single battery and therefore a single runtime, set by the heaviest thing plugged into it.

That is the part that could not be fixed by rearranging anything. As long as the subsistence gear draws from the same battery as the cluster, it inherits the cluster’s runtime, and the cluster is where the watts are. Sorting the network into two roles only becomes real when the two roles stop sharing a battery.

So the split had to be physical: a second UPS, carrying nothing but the subsistence network. A permanent, better-behaved version of the car battery on the floor — but wired to a new resolver as well as to the gateway, which is the part the car battery taught me it needed.

Pulling a little load off the luxury UPS buys the luxury network slightly longer runtime, which is a nice fringe benefit. The important benefit is the other direction: the subsistence network, alone on its own battery with a small load, now runs for hours instead of minutes.

Power and Network Paths Have to Agree

This is the point I would keep if I could keep only one.

A UPS gives a device power. It does not give it a path. The subsistence tier is not a set of devices — it is a set of devices plus every hop between them, and all of it has to be on the right side of the line.

Three kinds of wire had to be checked, and they are easy to check separately and get wrong together:

• AC power. The obvious one, and the only one most people trace.
• Ethernet. A device on the subsistence UPS whose uplink switch is on the luxury UPS is a luxury device wearing a disguise. It keeps running and stops being reachable.
• PoE. The one that hides, because the power and the network arrive on the same cable from the same box. An access point is only as protected as the switch port feeding it — which means an AP is on whichever tier its switch is on, no matter which room it lives in.

That last one nearly caught me. Splitting access points across tiers is not a matter of choosing which APs matter; it is a matter of which switch each one plugs into, and that switch’s power decides for you.

The diagram above is really just this rule drawn out: red, blue and green all stop at the dotted line together. The one crossing is a single Ethernet link between the two switches, and it is deliberately the only thing that dies at the boundary.

The Subsistence Tier Is a Watt Budget

That comparison I had never made — the relative power cost of each half of the network — is not a footnote to the design. It is the design.

The subsistence tier is not “the important gear.” It is “the gear that fits in the watt budget.” Every watt on that tier is minutes off the runtime, so the question stops being what do I want during an outage and becomes what can I afford to keep powered. Those are very different questions, and the second one has much shorter answers.

My access points make it concrete. They are the same brand, a generation apart:

The newer ones are genuinely better and earn their place on mains power. They are also nearly twice the draw, they demand more from the switch, and they run hot enough that you notice when you touch one. Fine habits. Expensive ones on battery.

So the older pair are the subsistence APs. They cover the home at twelve watts apiece, and the fast hot ones are allowed to go dark. On the subsistence tier, good enough everywhere beats excellent nearby.

The same contrast, and more starkly, in the two boxes each network hangs off:

The device that roots the entire subsistence network is rated below thirty watts. The single switch at the head of the luxury network is rated for two hundred and ten, before you count anything hanging off it. That gap, repeated in miniature across every device on each side of the line, is why the subsistence UPS now runs for hours where the old shared one ran for fourteen minutes. It is not a better battery. It is a much smaller ask.

Look at the PoE row, though, because that is the one that surprised me. Thirty watts is not much of a budget — and it is comfortably more than a twelve-watt access point needs. That is the whole reason the WiFi came back on the car battery: I powered one box, and one box quietly powered the access point behind it. The subsistence network was always small enough to fit inside its own gateway. I had just never had a reason to notice.

The Survivor

The fix is the box labeled dns4 in the diagram, and it is a Raspberry Pi.

Not another cluster node. A bare Pi running BIND, doing both of the jobs above: a caching recursive resolver and an authoritative secondary of the internal zones. It has to be both, for the same reason the originals are: split-horizon leaves the client no way to split the question, so a survivor that answers only half of them strands it just as completely. Full root recursion, no forwarder, nothing upstream with an opinion about my queries. The zone copies persist to disk with a 28-day expiry, so it keeps answering authoritatively for about four weeks after the primary disappears. That is longer than any outage I intend to have.

Three choices in there were deliberate:

• Same software as the others. BIND, with the same configuration idioms, because a survivor you administer differently is a survivor you will misconfigure.
• Bare metal, not a container. The entire point is to not be on the cluster.
• Not highly available. Making the survivor HA would put it straight back onto the tier I am trying to escape.

It is also stripped down to almost nothing — no desktop, no print service, no mDNS, no RPC — until the only things listening are DNS and SSH. A survivor should be small. Every service it runs is one more way for it to not be there when you need it.

And a spare nobody is told about is not a spare

There was a second, dumber problem, and on its own it would have quietly defeated everything above.

DHCP was handing out exactly one resolver. So even if one of the three had somehow survived, no client would have thought to ask it. My name service was one deep on the wire no matter how many copies were running.

DHCP now hands out all four, with the Pi last. Clients prefer the cluster and fall through to the survivor only when the cluster is gone. The cost is a few resolver timeouts during a total outage, which is exactly the right trade.

What Else Was Riding on the Wrong Tier

Two more things turned out to be the same mistake wearing different hats, and finding them is what convinced me the tier idea was worth the rewiring.

Nothing was ever told to shut down. The UPS knew perfectly well that it was on battery and running out. No server knew, because nothing was wired to tell them. The nodes were not shut down at fourteen minutes; they were cut off. That is a worse outcome than a clean shutdown and it was free to fix. The shutdown notification now runs on dns4, on the subsistence tier — because a process that warns machines about a power failure is worthless if it is on the power that failed.

The monitoring was a guest of the thing it monitors. Checkmk runs in a container on the cluster, so at fourteen minutes I lost the monitoring at the same moment I lost the thing worth monitoring, and had to reconstruct the outage timeline afterward from a log that simply stopped.

That one I did not move, and the reasoning is worth stating because it cuts against the tidy version of this post. Checkmk is heavy. Putting it on the subsistence tier would spend watts — and therefore runtime — on something that is not required to keep the home connected. So it stays a luxury service, and it stays a guest of what it monitors. What moved instead is the small, urgent job: the shutdown notification, which has to survive, and now does.

Three Questions Worth Asking About Your Own Setup

1. Which UPS feeds which machine, and which switch feeds which UPS? I could not answer either when the power came back. Answering them took two evenings with a flashlight and changed the design twice. Until that map exists, no post-mortem you write can say what should have stayed up.
2. Does anything your clients cannot function without live only on the tier that dies first? Name resolution, certainly — especially if a split-horizon setup means there is no public fallback. Then think about what else: a password manager that syncs, a door lock controller, the thing that tells you the power is out.
3. Are clients actually told about the survivor? A survivor nobody queries is decoration.

The batteries will still run out. That was never the part worth fixing. What changed is the order things go dark in, and how long the home can keep answering its own questions on the way down.

Sub-Nanosecond at Home, with GNSS and AI

Jul 14, 2026 • GNSS, timekeeping, talks, AI, hardware

My earlier talk asked whether sub-nanosecond GNSS clock sync was even possible. This one is about building it.

I gave it on 9 July 2026 at the STAC Summit held at the National Physical Laboratory in Teddington — which is a slightly intimidating room in which to discuss timekeeping.

GPS Satellites Are Not Master Clocks

It’s natural to picture GPS satellites as autonomous master clocks, broadcasting the time to anyone who listens. That abstraction is fine until you want position or time better than a handful of meters or nanoseconds — and then it quietly becomes the thing standing in your way.

Here is the question that reframed it for me. There are around 120 atomic clocks circling the Earth, perhaps 60 of them visible at once on a good day. How would you synchronize clocks you can’t touch?

The answer is that you don’t even try. You start them at roughly the same time, let them drift, measure each one’s error against good clocks on the ground, predict what that error will be shortly, and broadcast the prediction. The receiver applies the corrections. The same trick handles the orbits. One Galileo satellite runs about five milliseconds off — an enormous error, and entirely harmless, because it is measured and published.

The master clocks are on Earth. They are not in space. The satellites are how those clocks get broadcast to the rest of the planet, and the earthbound infrastructure — orbits, clock corrections, reference frames, atmospheric models — is what actually limits your accuracy.

Which leads to the point that reframed the whole project for me:

There is a real tradeoff underneath that, and it’s mostly about patience: fast approximate answers versus slow precise ones. A phone wants a blue dot in three seconds. A fixed-position clock can watch the sky for hours, and knowing that the antenna won’t move is itself information you can exploit.

I found it easiest to picture what that patience buys in terms of things you can hold. Process only the pseudorange signals from the satellites and you get a quick answer that sits somewhere inside a very large beach ball — about a meter. Move to carrier phase and it shrinks to a softball. Keep going and you’re down to a ping-pong ball. The practical limit is about the size of a green pea, under a centimeter. That’s the regime I was trying to work in.

The Small Effects That Stop Being Small

Once you’re chasing the last nanoseconds, a list of effects you were previously entitled to ignore starts mattering. The talk highlights three, chosen because they’re the ones people find least intuitive:

• Intersystem bias — the constellations don’t agree with each other, so combining GPS and Galileo means accounting for the offset between them.
• Datum offsets — the land itself moves. Plate motion means a position is only meaningful against a dated reference frame, and “where I am” has a when attached to it.
• Solid-Earth tides — the ground beneath you rises and falls roughly daily, by enough to matter. Not the ocean. The rock.

There’s also a largely visual section on antennas, because they turn out to be a critical part of getting precise answers and they get less attention than they deserve. To give you a sense of the scale, I patiently surveyed the position of an antenna on my roof and got an answer good to about the width of a pencil eraser east-west — but the point being surveyed is five centimeters up inside the antenna, and higher still in a choke ring antenna. Locating an object the size of a dinner plate to the width of a pencil eraser, when the point you care about isn’t visible from outside, is a fair summary of the whole field.

My favorite of the small effects has the best diagnosis attached to it. Watching the sawtooth jitter in my lab one afternoon, the pattern reversed direction partway through, and it took me a while to work out why: I’d switched on the air conditioning. When you compare GPS time to quartz time, you have built a very expensive thermometer.

What I Built

PePPAR-Fix uses raw GNSS carrier phase observations to drive a disciplined oscillator. The talk walks through the block diagram, shows deviation plots against a simple GNSS PPS receiver for comparison, and reports progress toward two-clock agreement.

The parts list totals about $1000, and there’s a photo of one of the prototypes. That number matters to me: this is meant to be reproducible by someone who wants to follow, not a demonstration that it can be done with a budget nobody has outside of national labs.

What Working with AI Actually Looked Like

About four months of fairly intense research in 2026 took me from understanding only the basics of GNSS timing to a much deeper appreciation of what sub-nanosecond performance costs. Along the way I went from zero to roughly 140,000 lines of Python for processing GNSS signals and disciplining precision oscillators.

None of it would have happened without AI, and none of it would have happened without the pioneering work of others either.

The approach that worked for me: give the agents great power, but limit the blast radius if something goes wrong. Broad authority to explore, tight bounds on what a mistake can damage. That’s a design problem more than a trust problem, and getting it right is what made the pace sustainable.

Concretely, that meant a command line interface rather than a chat window, and running with permission prompts switched off — inside a virtual machine that had root access to the lab equipment and not much else. Broad authority, small blast radius. I ran several agents with specialized roles: research and planning, Python coding, driving the instruments and collecting data, and analysis. Some of the slides in the talk were made by agents to convey points I’d asked for.

The hardest lesson was about language, not code. Sloppy language is sloppy thinking. So I had the agents build a project glossary, which every agent must read when it starts and must maintain as new ideas arrive. Precision in the terms turned out to be a prerequisite for precision in the clocks.

If You Want to Build Your Own

One goal was to leave a trail of crumbs.

• PePPAR-Fix — the software, open source.
• The slides include a complete parts list.
• f9tResearch — the measurement tools behind the earlier work, including the dual F9T experiments.

This stands on the shoulders of others, especially Ole Petter Rønningen, who built the first PPP GPSDO, and John Ackermann, N8UR, for an inexpensive time-interval counter and divider.

The Slides

View the presentation on GitHub, download it for offline viewing, or watch the dress rehearsal recording.

As before, the slides are heavy on visuals and light on bullets, so the PDF leaves out what the narration carried. The recording gets you closer.

Proxmox On-Site and Off-Site Backup

Nov 25, 2025 • Proxmox, backup, ZFS, storage

Failure resilience keeps a service running through a failure. Backup gets your data back after one. It’s worth being clear about which of those you’re buying, because a mirrored pool feels like a backup right up until the moment you need one.

A ZFS mirror protects you from a drive dying. It does absolutely nothing about rm -rf in the wrong directory, which it will faithfully replicate to both drives at full speed.

So the useful question isn’t “am I backed up?” but “which failures does each copy of my data actually cover?”

What On-Site Backup Is For

Two things, and they’re worth naming separately because they have different recovery shapes.

Fat-finger recovery. Somebody — probably you — deletes the wrong file, breaks a config, or upgrades something that turns out to be a mistake. This is by far the most common reason to reach for a backup, and it wants frequent snapshots and a fast, low-ceremony restore. You’ll use this one dozens of times for every time you use the others.

Local storage failure beyond what redundancy covers. A mirror survives one drive. A RAIDZ1 pool survives one drive. Lose two in a RAIDZ1, or one in a stripe, and the pool is gone regardless of how good ZFS is. Drives bought at the same time from the same batch have an unhelpful habit of failing at similar times, which makes this less theoretical than it sounds.

On-site backup answers both, and it answers them fast, because the data is sitting on the same network.

What Off-Site Backup Is For

Also two things, and both of them are the ones you never get to practice.

Local storage failure combined with local backup failure. The pool fails, you reach for the on-site backup, and discover that it was on the same shelf, the same UPS, or the same controller.

Site disaster. Fire, flood, theft, a burst pipe above the rack. Everything in the building is gone at once, and the only copies that matter are the ones that weren’t in it.

Off-site is slow to restore from and you will hardly ever touch it. That’s fine. It isn’t for convenience, it’s for the day the building is a write-off.

The Baseline

Proxmox Backup Server does the on-site job well. Backups are incremental and deduplicated, so keeping many restore points costs far less space than keeping many full copies, and it can verify what it’s holding rather than assuming. It can also sync a datastore to a second PBS elsewhere, which is the cleanest route to off-site.

My baseline is a ZFS mirror on non-ECC RAM for both the nodes and the backup server, with every node dumping to PBS regardless of what tier its guests are in. Even a guest with no resilience at all gets backed up, because backup is the one thing that covers the mistake you’re actually going to make.

It’s tempting to argue that a backup server which already replicates off-site doesn’t need storage redundancy of its own — if the pool dies, the off-site copy is still there. That’s true, and I’d still mirror it. A backup server that loses its pool leaves you restoring your fat-finger mistakes across a slow link from the off-site copy, which turns a five-minute annoyance into an afternoon. The mirror isn’t protecting the data, it’s protecting the fast path to it.

Where the Amount of Data Changes Everything

This is the part that makes backup planning messier than it first looks.

Some services carry almost no data. A DNS resolver, an NTP server, a small internal web app: a configuration file, a package list, some logs. These are ideal candidates for everything. ZFS replication between nodes is nearly free, so they can run HA and fail over in seconds. Backups are small, so you can keep many restore points and push them off-site over an ordinary internet connection without thinking about it.

Other services churn large files continuously. A DVR is the clearest example: hours of new video every day, most of it never watched. Every option above gets expensive at once.

• Replication would have to move gigabytes constantly, saturating the drives and the network to protect data that mostly isn’t worth protecting.
• HA is therefore off the table, which is fine — nobody is harmed by a recorder being down for an hour.
• Off-site backup would need an upload budget that no residential link is going to provide, and would burn it on footage with a shelf life measured in days.

The mistake is applying one policy to both. If you treat the DVR like the DNS server, you’ll spend a fortune and still miss recordings. If you treat the DNS server like the DVR, you’ll take a fifteen-minute outage for something that could have failed over in seconds.

That last cell is the one worth dwelling on. “Off-site the whole DVR” is unaffordable, but “off-site nothing from it” is usually wrong too. There’s normally a small, precious subset — the recordings you actually kept, the config that took an afternoon to get right — and separating that from the bulk at the storage layout level is what makes the decision cheap.

Test the Restore

A backup you have never restored from is a hypothesis.

Restoring a guest to a different node, or to a scratch VM, takes a few minutes and tells you whether the thing you’ve been diligently running every night actually produces a working service. It also tells you how long a real restore takes, which is the number you’ll want when something is broken and someone asks when it’ll be back.

I’d rather find out on a Tuesday afternoon than during the disaster.

Back to Resilience

The dividing line here — how much data a service carries — is the same one that decides which resilience tier it belongs in. Small-data services get HA and generous backups because both are cheap. Large-data services get neither, and instead get a clear-eyed decision about which parts of their data are actually worth keeping.

Work that out once per service, write it down, and most of the hard questions answer themselves.

Entry-Level Failure Resilience with Proxmox VE

Nov 24, 2025 • Proxmox, ZFS, storage, networking, hardware

If Murphy’s Law tells us that anything that can go wrong will go wrong, then planning failure resilience means working through anything that can go wrong. For each kind of failure, the question I keep coming back to is a simple one: how long is the service outage it creates, and what does shortening that outage cost?

I adopted virtualization to modernize and simplify the administration of my network services. Focusing on failure resilience is perhaps less common in a homelab, but it’s what I actually cared about. The values that drive a high-end homelab overlap substantially with a small IT shop or an SMB: a handful of services that people notice immediately when they stop, and nobody on call at 3 a.m.

This post is the entry-level version. It is not about building something impressive. It is about the smallest configuration that turns each failure from an evening of work into a few seconds of nothing.

Backups are the other half of this and get their own post. Resilience keeps a service running through a failure. Backup gets your data back after one. They are different jobs, and it is worth being clear which one you are buying.

Hardware Baseline

I’m done with spinning drives, noisy fans, and high power consumption 24x7. So I’ve stayed with compact, low-TDP CPUs and M.2 NVMe storage.

At the low end you’re likely to spend more on storage than on compute no matter what you do, so I think it’s worth paying a bit more for name-brand drives with good warranties and reliability records. I’ve stayed with the Samsung 990 line, but beware of the confusing naming.

There are three: the 990 EVO, the 990 EVO Plus, and the 990 PRO, each costing a bit more than the last. The PRO is the one with a dedicated DRAM cache, and I don’t see that adding much value behind the large DRAM requirements of ZFS. The slightly higher performance of the EVO Plus likely makes no difference either, given the limited CPU power of low-TDP parts and the network interface bottleneck in front of them. So I’d stay with the plain 990 EVO unless you need the 4 TB density, which the EVO tops out below.

Storage Layout

ZFS is clearly the right choice for local storage here. Ceph is a possibility for distributed storage, but it’s overkill for a homelab and it wants more nodes, more network, and more attention than this scale justifies.

Always mirror the boot device. A Proxmox node whose boot drive fails is a node that is simply gone until you rebuild it, and rebuilding it is an evening, not a minute. A two-drive ZFS mirror turns that into a drive swap you can do whenever you feel like it.

It is tempting to stripe instead, because striping two drives gets you the capacity of both. Resist it.

Failure Modes

Here is the whole exercise: list what can break, work out the outage, then decide what shortening it is worth.

Boot Device Failure

The node can’t boot because its boot device has failed.

Without mitigation, the outage runs until you have reinstalled Proxmox and restored the guests, which realistically means hours and a lot of attention.

Configure the boot device as a ZFS mirror across two drives. The node keeps running on the surviving drive and the outage becomes zero. Replacing the failed drive is scheduled work rather than emergency work.

Network Link Failure

The node’s hardware is fine and the switch is fine, but the physical link between them has failed. A cable, a port, a transceiver.

Without mitigation, everything on that node is unreachable until someone physically visits it.

Configure bonded interfaces using LACP across multiple physical links to the same switch. Failover is sub-second and nobody notices.

Network Switch Failure

The node’s hardware is fine, but the switch it connects to has failed. LACP to a single switch does nothing for you here — the whole bond is behind the thing that broke.

Configure multiple interfaces with MLAG using links to two different switches. This is the point where the cost starts to climb, because it needs a second switch and a pair that supports MLAG between them. It is also the first mitigation on this list I would consider genuinely optional for a homelab.

Compute Failure

The physical hardware underlying a node fails in some way that isn’t storage: loss of power, a kernel panic, a CPU that bursts into flames.

This is the one that HA is for, and it is where entry-level gets interesting.

What High Availability Actually Requires

Proxmox VE will fail a guest over to another node automatically, but only inside a cluster with quorum.

The second thing to understand is where the guest’s disk lives. Automatic failover needs the guest’s storage to be available on the node taking over. Without shared storage, that means ZFS replication: Proxmox periodically sends snapshots of the guest’s disk to the other nodes so a copy is already there when it’s needed.

Replication runs on a schedule, and that schedule is your data loss window. Replicate every fifteen minutes and a node failure costs you up to fifteen minutes of that guest’s writes. Replicate every minute and you pay for it in constant disk and network traffic. Nothing about HA makes this go away — it is the price of not having shared storage, and it is the number to decide deliberately rather than accept by default.

So the honest summary of automatic failover is: the service comes back in seconds, on a slightly older copy of its data.

Putting It Together

Not everything deserves the same treatment. I find it useful to sort guests into tiers and be explicit about what each one buys.

A few things are true across all of them. The node boots to run the guest, and the guest boots to serve clients, so both need to survive a drive failure. Some guests also touch bulk storage that is far too large to replicate. And every node dumps to a backup server regardless of tier, because none of the above protects you from deleting the wrong thing.

There is one more thing this table quietly assumes, and it took a blackout to show me what. Every tier above describes what happens when a node fails, not what happens when the room does. Three HA resolvers spread across three nodes are still one power failure away from silence, which turned out to deserve a post of its own.

Where This Runs Out

The tiers above are really a proxy for one question: how much data does the guest carry?

A DNS resolver or an NTP server holds a configuration file and not much else. Replicating it every few minutes is nearly free, so it can sit in Tier 1 and fail over in seconds.

A DVR churning large files is the opposite case. Replicating it continuously would saturate the network and the drives to protect data that is, frankly, replaceable. It belongs in Tier 3 with a sensible backup and no HA at all.

That dividing line — how much data a service carries — turns out to decide almost everything about both its resilience and its backups, which is where the backup post picks up.

Installing Proxmox on Asustor Flashstor 6 Gen 2 FS6806X

Nov 22, 2025 • Proxmox, hardware, networking, storage

I bought the Asustor FS6806X due to its appealing features for a home lab NAS and virtualization host.

1. Six hot-swappable M.2 NVMe drive bays
2. Up to 64 GB of ECC RAM
3. Built-in 10 GbE NIC

I discovered significant challenges while installing Proxmox on the FS6806X:

1. The built-in 10 GbE NIC is not supported by the Proxmox kernel.
2. Without an HDMI port, there’s no way to interact with the Proxmox installer directly.

I worked around these, but it’s not for the faint of heart.

Add a Video Card for Setup

1. Get an M.2 to PCI-e riser cable
2. Get a PC PSU hotwired to power the riser card through a floppy power connector
3. Get a video card for the PCI-e slot
4. Disassemble the FS6806X, install the riser cable and NVMe flash
5. Add a USB keyboard and monitor

Boot Configuration

It seems that the FX6806X boots from an internal USB flash device containing their ADM installer. We must configure the BIOS to boot from an external USB device for Proxmox installation and from NVMe flash after installation. Asus also recommends disabling their USB flash drive when installing another OS to protect it from accidental deletion, and so that it’s around for recovery later.

1. Plug in Proxmox installer on a USB drive
2. Get into the BIOS by holding F2 during power-on
3. Reorder drives on boot menu so Proxmox installer comes first, then NVMe flash drives, then internal USB flash drive
4. Optional, but recommended: disable internal USB flash drive

• BIOS AMD menu -> AMD CBS -> FCH Common Options -> USB Configuration Options -> USB1 2.0 Port Enable-> Disabled

5. Save and exit BIOS

Add an External 10 GbE NIC and an External Slow NIC

The built-in 10 GbE NIC uses the amd_xgbe driver, which is present in the Proxmox kernel. However, the NIC has to be initialized correctly by the BIOS or a device tree overlay or the driver can’t use it. I also see reports of regressions in driver support for this chipset. After a day of trying to get the built-in 10 GbE NIC to work, I gave up and got an external 10 GbE NIC. Using the external 10GbE NIC requires additional packages, so there’s a catch-22.

Adding an external slow NIC gets around the catch-22 by providing network connectivity to install the necessary packages to support the external 10 GbE NIC. It also provides the comfort of knowing that there’s another way into the box if the 10 GbE NIC has issues later. This alternative was important to me since I wanted to use all six M.2 NVMe bays for storage, and didn’t want the PCIe video card hanging dangerously off to the side.

1. Get a compatible external 10 GbE NIC, such as the IOCREST 10 GbE NIC
2. Get a common, slow USB NIC dongle that Proxmox supports out of the box.

A complication here is that Linux hosts don’t do well with two interfaces on the same LAN. I’m fortunate to have a separate VLAN for PTP where I could address the slow external NIC.

Install Proxmox 9.1

1. Set up the root filesystem and network on the slow USB NIC.
2. Reboot and apt install bolt polkit to get support for Thunderbolt hotplugging devices.
3. boltctl list to get the UUID of the external 10 GbE NIC.
4. boltctl enroll <UUID> to authorize the external 10 GbE NIC.
5. lspci -nn | grep -i ether and expect to see something like

e8:00.2 Ethernet controller [0200]: Advanced Micro Devices, Inc. [AMD] XGMAC 10GbE Controller [1022:1458]

Naming Interfaces

Someday the internal 10 GbE NIC may work, and it has at least two network interfaces until then, so confusion would be likely if naming weren’t clear. I wanted to name the on-board 10 GbE NIC ‘board0’, the external 10 GbE NIC ‘prod0’, and the slow external NIC ‘oob1’ (out-of-band management). Knowing the MAC addresses of each NIC allowed persistent names to be set by creating files in /etc/systemd/network/.

root@pve:~# cat /etc/systemd/network/20-oob1.link
[Match]
MACAddress=3c:18:a0:00:00:01

[Link]
Name=oob1
root@pve:~# cat /etc/systemd/network/30-prod0.link
[Match]
MACAddress=88:c9:b3:00:00:02

[Link]
Name=prod0
root@pve:~# cat /etc/systemd/network/40-board0.link
[Match]
MACAddress=78:72:64:00:00:03

[Link]
Name=board0
root@pve:~# cat /etc/systemd/network/99-disable-predictable-names.link
[Link]
NamePolicy=

Aside from the naming consistency, I wanted the external interfaces to come up at boot, but also if they were hot-plugged later. Udev rules, and the right tweaks to /etc/network/interfaces, make that possible.

root@pve:~# ls -l /etc/udev/rules.d/
total 11
lrwxrwxrwx 1 root root   9 Nov 22 10:29 60-bridge-network-interface.rules -> /dev/null
-rw-r--r-- 1 root root  82 Nov 24 11:34 70-prod0.rules
-rw-r--r-- 1 root root 140 Nov 24 13:32 99-ifupdown-hotplug.rules
-rw-r--r-- 1 root root 116 Nov 24 14:21 99-oob-bridge.rules
-rw-r--r-- 1 root root 168 Nov 24 16:08 99-prod0-bridge.rules
root@pve:~# cat /etc/udev/rules.d/60-bridge-network-interface.rules
root@pve:~# cat /etc/udev/rules.d/70-prod0.rules
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="88:c9:b3:00:00:02", NAME="prod0"
root@pve:~# cat /etc/udev/rules.d/99-ifupdown-hotplug.rules
ACTION=="add", SUBSYSTEM=="net", NAME=="oob0", RUN+="/sbin/ifup oob0"
ACTION=="add", SUBSYSTEM=="net", NAME=="oob1", RUN+="/sbin/ifup oob1"
root@pve:~# cat /etc/udev/rules.d/99-oob-bridge.rules
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="3c:18:a0:00:00:01", RUN+="/sbin/ip link set dev oob1 master vmbr2"
root@pve:~# cat /etc/udev/rules.d/99-prod0-bridge.rules
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="88:c9:b3:00:00:02", \
    RUN+="/sbin/ip link set dev prod0 up", \
    RUN+="/sbin/ip link set dev prod0 master vmbr0"
root@pve:~#

And /etc/network/interfaces:

auto lo
iface lo inet loopback

# Onboard amd_xgbe 10GbE interface that doesn't seem to work with Proxmox 8.4 or 9.1
iface board0 inet manual

# Thunderbolt 10GbE
auto prod0
iface prod0 inet manual
    pre-up ip link set $IFACE up

auto vmbr0
iface vmbr0 inet static
	address 192.168.1.30/24
	gateway 192.168.1.1
	bridge-ports prod0
	bridge-stp off
	bridge-fd 0
	bridge-waitport 15

### Second BACK-DOOR / OOB MGMT
auto oob1
iface oob1 inet manual
    pre-up ip link set $IFACE up

auto vmbr2
iface vmbr2 inet static
    address 192.168.99.33/24
    bridge-ports oob1
    bridge-stp off
    bridge-fd 0

source /etc/network/interfaces.d/*

Cleanup

1. Remove M.2 to PCI-e riser and video card and replace with the last NVMe flash drive.
2. Add rpool mirror to sixth NVMe drive

sgdisk --zap-all /dev/nvme5n1
sgdisk -R=/dev/nvme5n1 /dev/nvme0n1
sgdisk -G /dev/nvme5n1
zpool attach rpool /dev/nvme5n1p3

3. Configure sixth NVMe drive for booting:

proxmox-boot-tool format /dev/nvme5n1p2
proxmox-boot-tool init /dev/nvme5n1p2

Longer Term

I expect that support for the amd_xgbe driver will be added to Proxmox in the future. When that happens, I can remove the external 10GbE NIC, and run Proxmox on the built-in 10 GbE NIC.

Proxmox Mail Notification

Nov 18, 2025 • Proxmox, Checkmk, monitoring, Postfix

Not long after you start monitoring Proxmox with Checkmk, you may see unhandled service problems like this one.

Proxmox has plenty to tell you — a failed backup, a degraded array, a certificate about to expire — but out of the box it has no way to send any of it. The mail piles up in the local Postfix queue and sits there.

What follows is how to give Postfix a route out, using an external SMTP relay.

1. Make sure your Proxmox server has good Internet connectivity and DNS resolution configured.

2. Configure relay services for Postfix to use an external SMTP server to send mail.

• I’ll use Google in this example. You’ll need to set up 2FA with Google and create an App Password.
• Create /etc/postfix/sasl_passwd with content like this:

[smtp.gmail.com]:587    user@gmail.hosted.domain:SixteenCharacterAppPasswordNoSpaces

3. Configure Postfix to use the relay.

• Add lines like these to /etc/postfix/main.cf:

relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

• Delete a blank line setting relayhost =, if it exists, to suppress a warning.

4. Create Postfix hash db file for sasl_passwd:

postmap /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db

5. Install encryption libraries and certificates necessary for relay communication:

apt-get install libsasl2-modules ca-certificates

6. Restart Postfix to apply changes:

systemctl restart postfix

7. Run the Postfix queue check to verify mail can be sent:

postqueue -f

It might take a few minutes to drain, especially if there are many messages queued.

8. See if there’s anything left in the queue.

postqueue -p

You should see no entries if everything is working.

Monitoring Proxmox with Checkmk RAW

Nov 17, 2025 • Proxmox, Checkmk, monitoring, storage

Proxmox VE and Proxmox Backup Server are both based on Debian Linux, so we can use the Debian-packaged Checkmk RAW Linux agent to monitor them. This procedure also installs the smart plugin to monitor disk health via S.M.A.R.T., which is especially important for Proxmox nodes with NVMe storage.

1. Create a Checkmk host folder to hold Proxmox VE nodes and Proxmox Backup Servers that will be in your cluster.

2. Inside that folder, create host for each Proxmox VE node and Proxmox Backup Server.

Carefully note hostnames as they will be needed later.

3. Download Checkmk agent Debian package to ProxMox node or PBS and install:

• Browse to Setup->Agents->Linux->Packaged Agents.
• Right-click on check-mk-agent_VERSION_all.deb and copy link address.

cd /tmp
wget http://your.checkmk.server/monitoring/check_mk/agents/check-mk-agent_VERSION_all.deb # Paste URL from above here
dpkg -i check-mk-agent_VERSION_all.deb
rm check-mk-agent_VERSION_all.deb

4. Register agent with Checkmk RAW

cmk-agent-ctl register --server your.checkmk.server:8000 --site monitoring --user agent_registration --hostname HOSTNAME_BEING_REGISTERED --password 'AGENT_REGISTRATION_USER_PASSWORD' --trust-cert

Expect to see Registration complete.

5. Get URL of smart plugin for Checkmk agent:

• Browse to Setup->Agents->Linux->Plug-ins.
• Right-click on smart and copy link address.

6. Download smart plugin to Proxmox node:

wget -O /usr/lib/check_mk_agent/plugins/smart http://your.checkmk.server/monitoring/check_mk/agents/plugins/smart # Paste URL from above here
chmod +x /usr/lib/check_mk_agent/plugins/smart

7. Disable smartd because it blocks access by Checkmk:

systemctl disable --now smartd

8. Verify Checkmk agent is working and smart plugin is loaded:

check_mk_agent | grep '<<<smart>>>'

You should see output like:

<<<smart>>>

9. Configure Checkmk RAW to monitor new services from smart plugin:

• Browse to Setup->Hosts->Your Proxmox Node->Host->Run service discovery.
• Review services, and click Accept All.
• Commit and activate Checkmk changes.

Keeping Time with Proxmox and Ntpsec

Nov 16, 2025 • Proxmox, DNS, NTP, monitoring, ntpsec

Tight network time sync requires consistent latency between packet arrival and user space processing. We can’t expect consistent latency inside a container or VM. And even if we could, we wouldn’t want unprivileged containers or VMs to be steering the Proxmox node clock that is shared by all.

My old homelab configuration had many hosts that were configured to expect DNS resolution and NTP from two well-known IP addresses formerly on bare metal hosts that also provided many other services. The other services have since been migrated to a group of high availability LXC containers delivering just one service each. This eliminated cross-service interactions and allowed services to keep running, even in the face of Proxmox node failures.

Making the jump to Proxmox required different treatment for DNS resolution and NTP services. NTP servers can be referenced by name, but DNS resolvers can’t, due to the obvious chicken/egg problem.

The DNS resolver IP address configured into all my homelab hosts could be moved to HA LXCs easily enough. I created PTR records going to dns{1-3} for these addresses to handle any inverse lookups. It seems unlikely that A records would ever be needed, but I added them for symmetry. No changes to /etc/resolv.conf were required, but it was a good opportunity to audit all my host configs.

The privilege constraints above led me to run time sync daemons directly on my Proxmox nodes for consistent latency between network and user space. I also tried running pass-through or “proxy” time sync daemons inside high-availability LXC containers bound to the well-known IP addresses so that I wouldn’t have to reconfigure NTP clients.

That worked, but I didn’t like the jitter added between the bare metal and containerized time daemons. And it turned out to be easy enough to find all clients using my old NTP servers by address: ntpq -c mrulist. That was a small enough list that I just edited all their NTP client configs to specify servers by the names ntp{1-5}, which I added as additional A records for my bare metal Proxmox hosts.

I’ve chosen ntpsec for my sync daemon on the Proxmox cluster nodes. The default systemd-timesyncd.service isn’t really designed to discipline the system clock to match master clocks. Rather, it steps the system clock when the difference to a master clock is above a threshold, while making occasional frequency adjustments below this threshold.

I considered chrony because it runs continuously, measures multiple masters against each other, chooses the best, and disciplines the system clock from it. However, chrony is optimized for TSC discontinuities from CPU speed steps, sleep/wake, and CPU scheduling latency etc. We should not see these discontinuities in Proxmox nodes. Chrony would be great if we couldn’t run a time sync daemon directly on the Proxmox node.

Ntpsec is my preferred time sync daemon because it supports the newer NTS protocol, which authenticates time servers and so prevents man-in-the-middle attacks. Moreover, it directly supports the ntpviz visualization tools that let you easily see how your NTP servers are performing.

apt install ntpsec
apt install ntpsec-ntpviz # Optional

Monitor your NTP server performance using ntpviz: http://ProxmoxHost/ntpviz/

The Last Nanoseconds to UTC

Nov 12, 2025 • GNSS, timekeeping, talks

I set out to research the limits of clock sync precision — effectively chasing the last nanoseconds to UTC.

Is sub-nanosecond UTC sync possible? Literally, no. Practically, yes.

That sounds like a dodge, and it isn’t. The gap between those two answers is the most useful thing I learned, and it turns on a distinction that’s easy to skate past: agreeing with another clock is not the same as being right about UTC.

I gave this talk at the London and New York STAC Summits, to audiences who care about this for financial markets.

Why You Literally Cannot Get Below About 5 ns

Two things stand in the way, and neither is about your equipment.

The constellations themselves sit about 5 ns from UTC. Even the best GNSS constellations have a standard deviation around 5 ns relative to UTC. That isn’t a rumour — it’s published. Look for sigma_GPS in a recent Circular-T from BIPM and check me on it.

UTC doesn’t exist in real time. A national lab like NIST has atomic clocks that do far better than 5 ns, but UTC is a virtual timescale, computed after the fact from contributions around the world.

The mechanism is worth spelling out, because it’s the part that surprises people. Each country’s time lab watches the satellites, compares what comes down against its own earthbound clocks, and reports the differences to the BIPM in Paris. Then — weeks to months later — the BIPM publishes what UTC was.

There is no wire you can plug into that carries UTC as it happens. As I put it in the talk: everybody else on Earth is predicting UTC. The BIPM is defining it, in arrears.

So if you’re getting time from GNSS rather than from a national lab, you cannot literally be closer than about 5 ns to UTC. That ceiling is structural, and no equipment budget moves it.

Why You Can Practically Do Far Better

Now the interesting part.

In my homelab, two high-quality GNSS receiver modules — u-blox F9T — agreed with each other to 550 picoseconds over half a day, while each wandered 6 to 7 nanoseconds against GPS.

Two conditions made that number possible, and I skipped past both in the talk for lack of time. You need to track a single constellation, because the constellations don’t agree with each other to better than about 5 ns. And you need a dual-frequency receiver, so the ionosphere can be measured out rather than modelled. Take either away and the result doesn’t hold.

Whether that’s good news depends entirely on your question.

If you need to prove to a regulator that a timestamp is within some tolerance of UTC, the 5 ns floor is your problem and no amount of care in the lab removes it.

But if what you actually need is for two machines to agree on the order of events — which is what most people in markets are really asking — then the constellation’s offset from UTC is a common-mode error that cancels. You care about the 550 ps, not the 5 ns.

Most of the confusion I’ve seen in this area comes from conflating those two questions.

The Part That Bites in a Datacenter

All of the above assumes you can put an antenna where it needs to be. In a datacenter you usually can’t, and this is where the theory meets a roof.

The roof blocks the sky, so the antenna goes on top of it — either one you install or one the facility provides. Either way, as a tenant you are likely missing two numbers: where the antenna actually is, and how long the feedline is that runs from it down to your rack.

They cost you time for entirely different reasons, so they are worth taking apart separately.

The solution is for a point inside the antenna

GNSS doesn’t solve for the time at your receiver. It solves for the time at the antenna phase center — a spot several centimeters up inside the antenna housing, invisible from outside, and in a different place for every model. It is the antenna phase center’s position that goes into the equations, not the mounting bracket’s.

Get that position wrong and the error lands directly in your clock. Time error is proportional to position error, though the constant relating the two is not simply the speed of light — it depends on which direction you are wrong in, and on where the satellites happen to be while you are wrong. The bound is the part worth carrying around: a foot of position error costs at most about a nanosecond, and in practice usually less.

It is a static offset. It does not average away, and no amount of patience improves it.

Nothing measures the feedline for you

The second number is the cable, and it is the one people find surprising.

Signal travels more slowly down coax than through free space — typically around two-thirds to four-fifths of c — so a foot of feedline costs you rather more than a foot of free space, call it 1.2 to 1.5 nanoseconds. A run from a rooftop antenna to a rack inside can easily be a hundred feet of it.

Now, the reason this is a surprise rather than a nuisance. Inside the distribution network, NTP, PTP and White Rabbit all measure the delay between nodes and subtract it, which is why time looks the same everywhere downstream. That machinery is genuinely good, and it teaches people that delay is a solved problem.

It stops at the receiver. The antenna feedline — and the pulse-per-second wiring on the other side of the receiver — sit outside the compensated region entirely. Nothing on your network measures them, because nothing on your network can see them.

What Actually Removes Them

Configuration, not measurement. The antenna phase center’s position and the feedline’s delay are numbers you configure into the GNSS receiver, which then corrects for both. Get them right and they disappear.

Which is why not knowing them is the problem, rather than an inconvenience. Leave either at a default, or at a guess, and you are left with a constant, unknown offset in every timestamp the receiver produces. Both are errors a beautiful clock cannot fix — oscillator quality does nothing for either — and that is what makes this a datacenter problem rather than a timekeeping one.

What I Haven’t Tested Yet

This is the state of GNSS clock sync as I see it in late 2025. Two developments look like they could move the floor and I haven’t put either on the bench:

• Galileo HAS, which may bring their standard deviation down.
• Fugro AtomiChron, which promises sub-nanosecond sync to their own traceable reference timescale.

The Slides

View the presentation on GitHub, or download it for offline viewing. The measurement tools are in f9tResearch.

My presentation style is heavy on visuals and light on bullets, with the key points carried by narration — so the PDF necessarily gives you less than the room got. The compensation runs the other way at the end: the live audiences never had enough screen time to sit with the final Takeaways and Best Practices slides, and you do.

Having established what’s possible, the obvious next question was whether I could build one. That became a sequel.