shardseal

ShardSeal is a C++20 library and command-line tool that adds repairability to an existing directory without storing a second full copy.

It splits the logical file stream into fixed-size chunks, groups them into stripes, and writes Reed–Solomon parity shards plus SHA-256 integrity metadata into one sidecar archive. Later, ShardSeal can identify damaged or missing chunks and reconstruct them as long as each stripe has not lost more shards than its configured parity capacity.

This is useful for cold archives, photo collections, scientific datasets, local model weights, removable drives, long-lived project snapshots, and any data where bit rot or partial loss is more likely than total media loss.

Why it is different from a backup

A backup stores another complete copy. ShardSeal stores only repair information.

With the default 8+3 layout, every eight data chunks receive three parity chunks. The parity payload is roughly 37.5% of the protected data size, and any three missing or corrupted shards in a stripe can be tolerated. You can choose another redundancy ratio.

ShardSeal is not a replacement for an off-site backup: losing both the original directory and the sidecar archive remains unrecoverable. Its role is efficient local self-healing.

Features

• Systematic Reed–Solomon erasure coding over GF(256).
• Detects silent corruption with SHA-256 per data chunk and per parity shard.
• Manifest authentication covers the archive header, file index, data hashes, and parity hashes.
• Restores missing files, truncated files, enlarged files, and corrupted ranges.
• Writes recovery into a separate directory and atomically publishes the completed result.
• Refuses recovery before writing output when any stripe exceeds parity capacity.
• Deterministic file ordering and deterministic archive identity.
• JSON output for automation.
• Dependency-free C++20 implementation.
• Library API, installable CMake package, CLI, and end-to-end tests.

Build

Requirements:

• CMake 3.20 or newer
• A C++20 compiler: GCC 11+, Clang 14+, MSVC 2022+, or Apple Clang with C++20 filesystem support

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

The executable is build/shardseal on Linux/macOS and build/Release/shardseal.exe with a multi-config Windows generator.

Install system-wide or into a prefix:

cmake --install build --prefix /your/prefix

A downstream CMake project can then use:

find_package(ShardSeal 1 CONFIG REQUIRED)
target_link_libraries(your_target PRIVATE ShardSeal::shardseal)

CLI

Create repair data

shardseal protect ./photos ./photos.shardseal

Custom redundancy and chunk size:

shardseal protect ./dataset ./dataset.shardseal --data 10 --parity 4 --chunk 4M

Parameters:

• --data N: data shards per stripe; default 8
• --parity N: parity shards per stripe; default 3
• --chunk SIZE: chunk size in bytes or with K, M, G; default 1M
• --json: machine-readable output

The total of data and parity shards must not exceed 255.

Verify a directory

shardseal verify ./photos.shardseal ./photos

Exit codes:

• 0: all data and parity are healthy
• 2: damage exists but every affected stripe is recoverable
• 3: at least one stripe is unrecoverable
• 1: archive or I/O error
• 64: invalid command-line arguments

Repair into a new directory

shardseal repair ./photos.shardseal ./photos ./photos-recovered

To replace an existing output directory only after a complete verified reconstruction:

shardseal repair ./photos.shardseal ./photos ./photos-recovered --overwrite

ShardSeal never repairs the damaged source directory in place. Healthy chunks are copied from the source, damaged chunks are decoded from the remaining data and parity shards, and every output chunk is checked against its original SHA-256 hash before publication.

Inspect an archive

shardseal inspect ./photos.shardseal
shardseal inspect ./photos.shardseal --json

Library API

#include <shardseal/shardseal.hpp>

int main() {
    shardseal::ProtectOptions options;
    options.data_shards = 8;
    options.parity_shards = 3;
    options.chunk_size = 1024 * 1024;

    const auto archive = shardseal::protect_directory(
        "./important-data",
        "./important-data.shardseal",
        options
    );

    const auto state = shardseal::verify_directory(
        "./important-data.shardseal",
        "./important-data"
    );

    if (!state.healthy() && state.recoverable()) {
        shardseal::repair_directory(
            "./important-data.shardseal",
            "./important-data",
            "./important-data-recovered"
        );
    }
}

The complete public API is in include/shardseal/shardseal.hpp.

Recovery model

For K data shards and M parity shards, every stripe contains K+M encoded shards. The generator matrix is a systematic form of a Vandermonde matrix over GF(256). Any K surviving shards reconstruct the original K data shards. Therefore, up to M erasures per stripe are recoverable.

ShardSeal treats a shard as erased when:

• the corresponding file is absent;
• the file is too short to read the chunk;
• the chunk hash differs;
• a parity shard is unreadable or its hash differs.

A larger-than-expected source file is also reported as damaged; recovery writes the exact protected size.

Archive format v1

All integers are unsigned little-endian.

1. 64-byte header:
   • 8-byte magic SHSEAL1\0
   • format version
   • data shard count
   • parity shard count
   • chunk size
   • file count
   • data chunk count
   • stripe count
   • original byte count
   • metadata byte count
2. Metadata:
   • sorted file paths and sizes
   • SHA-256 of every logical data chunk
   • SHA-256 of every parity shard
3. 32-byte SHA-256 manifest digest over the header and metadata
4. Fixed-size parity shards in stripe order

The archive ID shown by the CLI is the manifest digest. Since parity hashes are inside the authenticated metadata, it commits to the complete parity payload as well.

Operational boundaries

• Regular files are protected. Symbolic links are deliberately ignored to avoid path-escape and link-target ambiguity.
• Empty files are preserved; empty directory entries, ownership, permissions, timestamps, ACLs, and extended attributes are not archived.
• Protection expects source files to remain unchanged during creation; a short read aborts the operation.
• The sidecar archive must be stored independently enough that the same incident does not destroy both it and the source.
• Cryptographic hashes provide integrity detection, not encryption or authenticity against an attacker who can replace the entire archive. Sign the archive externally when adversarial replacement matters.

Project layout

include/shardseal/shardseal.hpp  Public library API
src/shardseal.cpp                Archive, verification, and repair engine
src/gf256.cpp                    GF(256), matrices, Reed–Solomon generator
src/sha256.cpp                   Dependency-free SHA-256
src/main.cpp                     CLI
tests/test_main.cpp             End-to-end corruption and recovery tests

License

MIT. See LICENSE.

Source · MIT