Questions? Email [email protected] · All prices are one-time unless stated otherwise
D
Datapad Projects
Concepts

How syncing works

Content-defined chunking, global de-duplication, manifests, and delta transfers explained.

Files become chunks

When the client prepares a file, it splits the file into pieces using content-defined chunking. Rather than cutting every fixed number of bytes, the chunk boundaries are chosen from the data itself, so inserting or removing bytes near the start does not shift every later boundary. A small edit only re-chunks the area around the change.

Chunks are addressed by their hash

Each chunk is identified by the SHA-256 of its contents. Because the name is the hash, two chunks with the same bytes are the same object. The server stores each unique chunk once and shares it across every file and every user that contains it — global de-duplication. Copying a large file, or many people storing the same document, costs almost nothing extra.

A file version is a manifest

A file is recorded as a manifest: an ordered list of the chunk hashes that make it up, plus its size and a version number. Every save creates a new manifest. Reassembling a file means fetching its chunks in order and concatenating them.

Pushing only what changed

To upload, the client first asks the server which of the file's chunk hashes it does not already have. Only those missing chunks are sent; everything the server already holds is skipped. The client then writes the new manifest, declaring the version it started from — the server accepts it only if that is still the current version, so a concurrent edit is caught rather than overwritten (see Conflicts and versioning). Re-uploading a file after a tiny edit transfers just the few changed chunks.

Pulling and verifying

To download, the client fetches the latest manifest, retrieves any chunks it is missing, and checks each chunk against its expected hash before using it. The file is rebuilt in a temporary file and moved into place atomically, so a sync that is interrupted never leaves a half-written file where your real one was.

Reconciling a folder

A folder sync is push-then-pull. The client keeps a small record of what it last synced for each path — the version, modification time, and size — and uses it to tell what changed locally. New and changed files are pushed; then any server versions newer than what the client has are pulled down.

Deletions

Deletions travel as tombstones. When you remove a file locally, the client records a deletion for that path on the server; when another device sees that tombstone, it removes its local copy. Deleting is just another kind of version, so it propagates the same way edits do.

Staying up to date

A running watcher does not have to poll to notice changes from your other devices. It holds open a lightweight event stream to the server, and the server sends a short notification whenever one of your files reaches a new version. The watcher reacts by reconciling, so an edit on one device shows up on the others within about a second. A slow periodic poll remains as a fallback, and the stream reconnects on its own after a server restart or a network blip.

Further reading

Concurrent edits and the safety rules around them are covered in Conflicts and versioning.