DudeFS is a distributed key-value store for a small trust group where the storage nodes that replicate data and arbitrate writes do not need access to keys or values. The application decides, per row, whether to store data in plaintext or encrypted. Plaintext rows give the application prefix scans and counts; encrypted rows have their names blinded by a keyed PRF (BLAKE2b) and their values sealed with AEAD (XChaCha20-Poly1305), so that nodes see only opaque tokens and ciphertext. Both modes go through the same consensus, the same log, and the same Merkle-rooted state. The encrypted side supports key rotation.

It is built for small, contested state on semi-trusted machines: a few writers, kilobytes of configuration and keys and claims, 3 to 7 nodes on cheap VPS providers. Because the storage nodes do not need TEE hardware, they can be the cheapest instances available -- the confidentiality comes from the cryptography, not from the machine.

This post describes what it is and how it works. DudeFS is part of the lockboot project and on GitHub.

Roles

Five roles cooperate to run a cluster:

  • Anchor: the root of trust. Generates genesis, issues grants (certificates) to every other role, then goes offline. This is the one key the data owner holds.
  • Node: a consensus participant. Runs quorum rounds, settles transactions, replicates state, serves reads. A cluster needs 2f+1 nodes to tolerate f failures.
  • Manager: an operational delegate of the anchor. Drives roster changes, key rotations, and client grants through the consensus layer. A manager's authority is a certificate chain back to the anchor.
  • Client: a light client. Syncs committed state from nodes, submits transactions, and exposes a unix socket for worker processes. The client holds the sealed keys that let it decrypt the store; workers connect through it.
  • Compactor: periodically compacts the log and produces checkpoints for fast bootstrap of new or recovering nodes.

Plaintext and encrypted modes

The store supports two modes per row, chosen by the application at write time:

Plaintext (epoch 0). The name is stored as-is and the value is unencrypted. Nodes can read these rows, which means the application gets prefix scans and counts -- the primitives that PlaintextMap uses for coordination metadata like queue state and worker registration. The management store (store 0) is entirely plaintext-keyed, because nodes need to read it to enforce access rules: grants, roster membership, epoch counters.

Encrypted (epoch > 0). The name is run through a keyed PRF (BLAKE2b with a per-store blinding secret) to produce a 32-byte name token. The value is sealed with XChaCha20-Poly1305 under a per-item key derived from the store's current epoch master and the name token. The AAD binds the ciphertext to its store, name token, and epoch, so a ciphertext moved between slots or epochs does not decrypt. Nodes index by the opaque token and store the ciphertext. Without the blinding secret they cannot map a token back to a name; without the epoch master they cannot decrypt a value.

The blinding secret and the per-epoch master keys are distributed to each authorised client as X25519 sealed boxes -- each copy encrypted to that client's Ed25519 public key. These sealed wraps live in the management store. A client unseals its own copy and derives the keys it needs. A node does not hold any client's private key, so the sealed wraps are opaque to it.

Consensus

Transactions are signed by their author (Ed25519) and submitted to the cluster. Consensus proceeds in time-bucketed rounds:

  1. Collect. Each node advertises which transaction hashes it holds (HELD). Nodes exchange missing bodies so that, by the end, every node in the quorum has the same set.
  2. Finalize. At the bucket boundary, each node computes a deterministic slice -- the largest set of transactions held by at least a quorum -- screens it against the current state (checking guards, authority, epoch validity), signs the resulting slice hash, and broadcasts its signature.
  3. Ratify. A quorum of matching signatures ratifies the block.
  4. Settle. Each node applies the ratified transactions to its local state, computes anchors (block number, height, prev_block hash, state root, state accumulator, log accumulator), signs the anchors, and broadcasts. A quorum agreeing on anchors settles the block.

The two phases -- ratify the content, then settle the state -- mean the block's identity is fixed before the state transition happens, and the state transition is verified by quorum agreement on the resulting roots. If any node computes a different state root, the settlement round diverges and the block is abandoned.

The quorum threshold is ceiling(2n/3). With 5 nodes, the quorum is 4; with 7, it is 5. Any two quorums overlap, which prevents forks. The majority partition makes progress; a minority partition stalls.

Settlement verification

Each settled block carries:

  • A state root: the root of a sparse Merkle tree over every live row.
  • Two accumulators: elliptic-curve point accumulators (EC-Acc over Ed25519), one for the live state and one for the log. These are homomorphic -- adding a row adds a point, removing it subtracts -- so the accumulator tracks the exact multiset of rows without growing. Two nodes with the same accumulator hold exactly the same rows.
  • A block hash that chains to the previous block.
  • A multisig (Ed25519, bitmap-indexed) from the quorum that settled it.

A light client walking the chain can verify that each block's prev_block matches the prior block's hash, and that the multisig has enough signers from the roster it trusts. To verify a specific value, it requests a Merkle proof from a node and checks it against the state root in the block it holds.

Key distribution and rotation

Each data store has its own key hierarchy:

  • A blinding secret per store per client. Derives the name key, which produces name tokens. This is permanent -- name tokens are sparse Merkle tree paths, so rotating the blinding secret would relocate every row in the store.
  • An epoch master per store per epoch per client. Derives the value key for that epoch.

Key rotation bumps the store's epoch counter. The settlement layer enforces that writes carry either the current or immediately previous epoch (grace window during rotation) and that epochs only move forward. The rotation -- epoch bump plus every sealed wrap for every authorised client -- lands in one transaction.

Each store rotates independently. Re-keying store 2 does not re-wrap anyone on store 1.

Where it fits in lockboot

DudeFS is a component of the lockboot project, not a layer in the boot chain. The boot chain -- stage0 fetches and verifies a payload, stage1 does the same for stage2, each extending TPM PCR 14 -- handles admission and measurement independently. DudeFS has no mechanism for serving boot payloads or deciding what a machine should run.

What it provides is coordination after boot. The general pattern:

  1. A machine boots through the lockboot chain. The TPM holds a measurement of exactly which code is running.
  2. The machine produces a TPM attestation (via vaportpm), proving its boot state to a remote verifier.
  3. The data owner (anchor or manager) verifies the attestation and authorises the machine's client keypair -- a management transaction that commits a grant, sealed key wraps, and a blinding secret atomically.
  4. The client daemon on the attested machine unseals its keys and syncs from the cluster. Workers connect to its unix socket.

If the machine reboots into different code, the PCR values change and the attestation no longer matches what the data owner authorised. The grant can be revoked -- revocation re-issues every certificate the revoked key attested, in one transaction, so that nothing the revoked identity signed is left dangling.

Data structures

On top of the encrypted store, three structures provide application-level primitives:

ManagedMap. An encrypted key-value map with a count, a membership accumulator, and a positional index. Used for any state where the values should be opaque to nodes.

PlaintextMap. A prefix-scoped key-value map over epoch-0 (unencrypted) names. Used for coordination metadata that nodes or other participants need to scan without decryption: queue state, worker registration, lease tracking.

Queue and Worker. A lease-based job queue built on PlaintextMap. Jobs are submitted to a pending set, claimed by a worker with a deadline, moved to an active set, and completed or reclaimed on expiry. A dead worker's job returns to the pending set when its deadline passes -- no central coordinator, just a timestamp and a reclaim scan.

Versus what?

etcd, Consul, ZooKeeper. The same niche: small coordination state, replicated, consensus-backed. All three assume trusted infrastructure -- nodes can read every key and value they store. Encryption at rest, where available, is transparent: the node decrypts to serve a query. If you trust your infrastructure, these are mature and well-understood. DudeFS is for the case where you do not -- the nodes replicate and arbitrate without access to the plaintext of encrypted rows.

HashiCorp Vault, cloud secrets managers. Built for key material, but centralised. The Vault server (or the cloud provider's service) holds plaintext and serves it to authorised callers. The trust boundary is the server itself. In DudeFS the storage layer is a quorum of nodes that individually cannot decrypt anything. The trust boundary is the client, not the server.

MongoDB CSFLE. The closest analogy on the crypto side: the server stores ciphertext it cannot decrypt, the client holds the keys. But MongoDB is a database at a different scale and for a different purpose. CSFLE is added to an existing system; in DudeFS the encrypted-or-plaintext choice, key distribution, and epoch rotation are part of the base design.

Encrypted SQLite (SQLCipher). Encrypts the whole database file, but it is single-node and the process holding the key reads everything. No replication, no consensus, no per-row choice between plaintext and encrypted.

TEE-based confidential chains (Secret Network, Oasis Sapphire, Phala). These put confidential execution inside a hardware enclave on the chain's own nodes -- the node is the trust boundary, and you trust the TEE. DudeFS inverts that: the consensus nodes are blind via cryptography, not hardware, and do not need a TEE at all. Attestation is used only on the client side, to verify the machine that will decrypt the data, not the machines that store it. The trade-off is real in both directions: TEE-based chains can compute over encrypted state on-node (confidential smart contracts), which DudeFS cannot -- the encrypted data must be decrypted at the client. In exchange, DudeFS's confidentiality does not depend on enclave security. The SGX side-channel attacks demonstrated against Secret Network (xAPIC, MMIO stale-data) are not a relevant threat class here because the storage nodes never hold key material in any form, hardware-protected or otherwise.

ICP (Internet Computer) vetKeys. A threshold-cryptographic approach: nodes collectively derive user-specific keys, sharded so that no single node ever holds the full key. The confidentiality comes from the threshold protocol, not from hardware enclaves or client-side encryption. DudeFS does not do threshold key generation -- keys are minted by the data owner and distributed as sealed boxes to specific clients. The trust models are different: vetKeys distributes trust across subnet nodes via threshold crypto; DudeFS places no trust in the storage nodes at all and pushes decryption entirely to the client.

Implementation

DudeFS is about 12,000 lines of Python (3.12+, stdlib plus PyNaCl). The rest of lockboot is Rust and no_std. The protocol was designed and iterated in Python; the bottleneck is quorum round-trips over the network, not local computation. There are 537 tests.

The store is SQLite with WAL mode. The network layer is TCP with optional SOCKS5 (for onion transport). Block times are derived from physical measurements -- maximum RTT, clock skew, retry budget -- and default to human-scaled intervals.

Try it

The testbed CLI boots a local cluster:

make install
dude tb create --nodes 3 --mgmt 1
dude tb provision
dude tb start

From there:

dude --home testbed/.rw0 client put mykey "some value"
dude --home testbed/.rw0 client get mykey

That value is sitting encrypted on three SQLite databases whose nodes do not know what it says.