ITYS I Told You So

3 min

TX: TAR eXtender for resilient archives

How TX turns plain tarballs into self-validating, chunked, resumable archives that are easy to ship, verify, and repair.

TX (TAR eXtender) is a tiny, dependency-light utility for building, moving, and verifying tar archives at internet scale. It sits between classic tar and modern backup systems, adding manifests, chunking, and integrity checks without pulling in a full database or heavyweight agent.

If you want reproducible tarballs that can be resumed mid-transfer, diffed across sites, and verified long after they leave your machine, TX is the glue. You can grab the latest build at tx.itys.net.

🔗Why another archive helper?

  • Plain tar lacks durable manifests and integrity metadata.
  • Whole-archive retries are painful when a 40 GB upload drops at 98%.
  • Long-haul transfers (home lab -> cloud, colo -> cold storage) need easy resume and post-copy verification.
  • We wanted something you could curl and run without Rust/Cargo already present.

🔗What TX adds

  • Chunked output: splits archives into fixed-size parts (e.g., 256 MiB) so you can resume uploads and retry only the missing pieces.
  • Signed manifest header: writes the JSON manifest (hashes, sizes, tree metadata) into the first bytes of the archive (Track 1) ahead of the tar stream, so you do not lose it as a sidecar.
  • Streaming-friendly: reads from stdin, writes to stdout by default; no temp staging required.
  • Integrity-first: tx verify replays hashes and sizes; tx repair can refetch only the corrupted chunk.
  • Boring dependencies: static build, no runtime services or databases.

🔗Quick start

Create a chunked archive with an embedded manifest:

tx pack \
  --input ./data \
  --output ./exports/site.tx.tar \
  --chunksize 256MiB

Verify the archive later (either side of a transfer):

tx verify \
  --input ./exports/site.tx.tar

Extract after verification:

tx unpack --input ./exports/site.tx.tar --output ./restore

🔗Recipes

  • Remote pushes: pipe into ssh to avoid local disk writes.

    tx pack --input ./data --chunksize 128MiB \
| ssh user@remote "cat > /srv/archive.tx.tar"

  • Object storage: upload chunks in parallel; the embedded manifest rides in Track 1, so you do not have to chase a separate file for audits.

  • Cold storage audits: schedule tx verify against a mounted bucket to catch bit-rot without restoring the data.

🔗How it works

  • Reads the file tree, writes tar entries in a stable, sorted order.
  • Emits fixed-size chunks and SHA-256 per chunk plus whole-archive digests.
  • Records metadata in manifest_version, chunks, tree, and digests keys inside the JSON prelude at the start of the file (Track 1).
  • Keeps the tar stream portable: the payload after Track 1 is a standard tar stream; TX reads the manifest first, then hands off extraction.

🔗Operational notes

  • TX prefers deterministic inputs: run with a consistent --mtime when you want byte-identical outputs across builders.

  • For huge trees, set --chunksize to your network’s sweet spot; 64–256 MiB works well for S3-compatible uploads.

  • Memory footprint stays near the chunk size; TX streams rather than buffering entire archives.

  • Builds are published at tx.itys.net with SHA-256 sums in sha256.txt. Verify the binary before first use:

    curl -O https://tx.itys.net/tx-linux-amd64
curl -O https://tx.itys.net/sha256.txt
grep tx-linux-amd64 sha256.txt | sha256sum --check
chmod +x tx-linux-amd64
sudo mv tx-linux-amd64 /usr/local/bin/tx

🔗Roadmap

  • Optional Blake3 digests for faster verification on low-power hardware.
  • Built-in push helper with retry/backoff for S3 and rsync targets.
  • TUF-style signed release metadata to harden binary distribution.

TX keeps tar simple while adding the reliability knobs long-haul transfers deserve. If you kick the tires, feedback is welcome—open an issue or drop a note via tx.itys.net.***