Archive & Sync

Backup

GitHub Edit page

Backup

birdclaw can write the canonical SQLite store as deterministic JSONL shards that Git can diff and merge. The backup repo is the long-lived, version-controlled record; the SQLite file is just a fast local index built from it.

#Layout

.gitattributes
manifest.json
data/accounts.jsonl
data/profiles.jsonl
data/profile_affiliations.jsonl
data/profile_snapshots.jsonl
data/profile_bio_entities.jsonl
data/tweets/YYYY.jsonl
data/tweets/unknown.jsonl
data/tweet_revisions.jsonl
data/tweet_subordinate_tombstones.jsonl
data/collections/likes.jsonl
data/collections/bookmarks.jsonl
data/dms/conversations.jsonl
data/dms/YYYY.jsonl
data/moderation/blocks.jsonl
data/moderation/mutes.jsonl
data/follow_snapshots.jsonl
data/follow_snapshot_members.jsonl
data/follow_edges.jsonl
data/follow_events.jsonl
data/lists/lists.jsonl
data/lists/members.jsonl
data/<logical-shard>.part-NNNN.jsonl

Design rules:

  • tweets are sharded by year for human browsing, partial loads, and yearly analysis
  • DMs are sharded by year with conversation_id in each row, so Git stays fast while preserving conversation membership
  • collection-only tweets with unknown timestamps go to data/tweets/unknown.jsonl instead of pretending they belong to 1970
  • tweet tombstones and revisions preserve explicit deletions, subordinate media/quote deletion state, and observable edit chains without re-exposing deleted or superseded rows through search
  • likes and bookmarks are stored as collection edges and mirrored into the timeline rows so existing queries keep working
  • profiles include bio, follower/following counts, profile URL, location, verification type, structured URL entities, and raw profile JSON so the snapshot is meaningful on its own
  • profile affiliations preserve X badge/highlighted-label organization edges separately from profile rows
  • profile snapshots preserve deduplicated profile-history states for identity evidence over time
  • profile bio entities preserve extracted @handle, domain, and company-phrase identity hints, including inactive historical values
  • follow graph shards preserve followers/following snapshots, snapshot members, current edges, and append-only churn events
  • X Lists preserve owned-List freshness/completeness metadata and current/ended member edges
  • no SQLite WAL/SHM, FTS shadow tables, or transient live cache rows ever land in the backup
  • line endings for hashed JSONL and manifest files stay LF on every platform via the generated .gitattributes
  • large logical shards split deterministically into numbered parts capped at 48 MiB, keeping every Git blob below common hosted-repository limits without Git LFS

The manifest pins per-shard byte counts, row counts, and SHA hashes. Validation walks every shard and verifies they line up.

#backup export

Write text shards to a local directory. Validates the manifest by default.

birdclaw backup export --repo ~/Projects/birdclaw-store --json
birdclaw backup export --repo ~/Projects/birdclaw-store --commit --push

Flags:

  • --repo <path> — target directory (created if missing)
  • --commit — create a Git commit in the backup repo
  • --push — implies --commit and pushes the backup repo
  • --no-validate — skip post-export validation (not recommended)

The data/ directory is fully rewritten on every export. Anything outside data/ (your README, license, hooks) is left alone. Logical shards at or below 48 MiB keep their existing filename. Larger shards become files such as data/profile_snapshots.part-0001.jsonl; import and validation accept both layouts.

#backup sync

The recommended round-trip workflow:

birdclaw backup sync \
  --repo ~/Projects/backup-birdclaw \
  --remote https://github.com/steipete/backup-birdclaw.git \
  --json

What sync does:

  1. clones / configures the backup Git repo if needed
  2. pulls the backup repo before reading
  3. merge-imports remote backup rows into local SQLite
  4. exports the local union back into deterministic text shards
  5. commits and pushes the backup repo

Git operations are rooted at the configured repoPath. If that directory sits inside another worktree, Birdclaw initializes or uses a separate repository there instead of staging backup files into the enclosing project.

This is what makes birdclaw safe across multiple machines: each machine can sync independently, and the merge step preserves rows that only one side has.

#backup import

birdclaw backup import ~/Projects/birdclaw-store --json

Validates the backup first (unless --no-validate), then merge-imports rows into local SQLite. Local-only rows are preserved by default.

--restore restores exactly from backup and deletes local portable rows first. Use it when you want a known-clean state from the backup, e.g. on a new machine. The older --replace spelling remains as a deprecated alias.

The FTS5 shadow tables for tweets and DMs are rebuilt from the JSONL text after import, so search is immediately available.

The portable database fingerprint is computed after the write transaction commits and streams rows in canonical order inside a read snapshot. Large first imports therefore keep bounded memory, do not hold the SQLite write lock while producing the verification hash, and remain atomically usable even while the fingerprint is being calculated.

Revision topology reconciliation visits only multi-revision or explicitly edge-connected components. Canonical singleton revisions are imported directly instead of each triggering an identical recursive component query and rewrite.

#backup validate

birdclaw backup validate ~/Projects/birdclaw-store --json

Checks:

  • manifest.json is well-formed
  • every listed shard exists
  • per-shard byte counts match
  • per-shard row counts match
  • per-shard SHA hashes match
  • every JSONL row parses

Exits non-zero on validation failure. Run it in CI before publishing a backup, or after a noisy disk event.

#Auto-sync config

{
	"backup": {
		"repoPath": "/Users/steipete/Projects/backup-birdclaw",
		"remote": "https://github.com/steipete/backup-birdclaw.git",
		"autoSync": true,
		"staleAfterSeconds": 900
	}
}

When autoSync is enabled:

  • read paths (CLI search, inbox, API status/query, web startup) pull + merge from Git only when the last backup check is older than staleAfterSeconds
  • data-changing commands (compose, sync, blocks, mutes, etc.) run a full backup sync afterward
  • the freshness window is per-process; nothing pings Git on every command

Set BIRDCLAW_BACKUP_AUTO_SYNC=0 to disable auto-sync for one process — useful for local debugging.

#Why not commit birdclaw.sqlite?

  • the FTS5 shadow tables are non-portable and create huge diffs
  • WAL / SHM files churn constantly
  • transient live response caches are noise
  • a SQLite blob is opaque to Git review

Sharded JSONL is the opposite: human-readable, Git-friendly, and reproducible.

#See also

  • Jobs — scheduled bookmark sync writes a JSONL audit log alongside the backup output
  • CLI reference — every flag for every backup subcommand
  • Configuration — auto-sync env var and config keys