clonetool

Block-level sync for a large file or block device between two machines (or two paths on the same machine), driven from a third, passive "manager" machine. Single static Go binary, no runtime dependencies beyond the system ssh client for remote endpoints.

Two subcommands:

• sync — mirror one file/device onto another (the original tool; see most of this README).
• clone-disk — clone a whole system disk: boot record + partition table + per-filesystem data, rebuilt on the target then filled from the source. Same manager/agent/SSH/self-deploy model as sync; leans on the standard disk tools already present on a rescue system (sfdisk, ntfsclone, partclone.*, e2image, rsync, …). See clone-disk below.

How it works

• You run clonetool sync on the manager. Source, destination, and manager may be three entirely different machines — the manager never reads or writes a single block itself. It only spawns and talks to two control agents (clonetool agent --role control, one per side, local subprocess or over ssh) that do the stat/prepare/transfer work.
• The two agents compare content by SHA-256 hash per fixed-size block. A block is only re-read once: it's hashed from the buffer it was read into, and that same buffer is what gets sent on if it differs — never read twice.
• Source and destination are scanned in parallel. The destination side streams the hash of each block as it computes it, in block order; the source side consumes that stream and immediately reads, compares and (if it differs) sends that block. So the destination scan, the source scan and the transfer of changed blocks all overlap — there is no "scan the whole destination, then start" phase. The status line shows the transfer progress with a trailing scan H/M until the destination fingerprint is complete.
• No state is kept between runs. Every sync re-reads and re-hashes the destination's current content and compares the source against that, so a re-run only moves the blocks that actually differ and nothing needs to be trusted from a previous run. (The destination side always does the destination hashing — the remote sink in push mode, the local destination agent in pull mode.)
• Bulk data goes directly between source and destination, not through the manager. Each run tries:
  1. push — the source agent connects straight to the destination host over SSH and streams changed blocks to it.
  2. pull (fallback) — if push isn't possible (no route/keys in that direction), the destination agent connects to the source host instead and pulls.
  3. If neither direction works, the job fails with both reasons named. Run the manager on the source or destination host, or set up SSH connectivity in at least one direction.
  • When source and destination are both local to the manager, no SSH is used at all — the source agent spawns the write-side helper as a plain local subprocess.
• Self-deploy: if a remote endpoint has no runnable clonetool on PATH (or wherever --remote-bin points), or the one that's there is a different build than this binary (see below), the manager streams this binary to ~/.clonetool/bin/clonetool on that host over the existing SSH connection and uses it — no install, no root. Disable with --deploy=false (a stale or missing binary is then an error, except a merely-stale --remote-bin is used anyway with a warning, since there's no way to correct it). If the copied binary won't execute there (wrong CPU architecture) the error says so; build one for the remote's arch (CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build) and put it on PATH or pass --remote-bin.
• Version/staleness check: every binary has a build timestamp baked in (see Build below) and clonetool version prints it. Before using a remote clonetool, the manager runs its version there and compares that timestamp against its own — a mismatch (e.g. the remote was self-deployed from an older build of this tool) is treated the same as "not runnable" and triggers the self-deploy above, so the remote binary is kept in sync with whatever you're running locally. Only the build timestamp is compared, not GOOS/GOARCH — those are expected to differ across a cross-compiled deploy.
• sudo: if reading or writing an endpoint that is a block device fails with a permission error, --sudo=auto (the default) transparently restarts that side's agent — and the helper it spawns on the peer — under sudo. Locally that may prompt for a password on your terminal; on a remote host it uses sudo -n, so passwordless sudo (NOPASSWD) must be configured there (a password prompt can't work — the transfer protocol owns the SSH stdout stream). --sudo=always elevates from the start; --sudo=never never does.
• Sizing rules:
  • Destination is a block device: it can't be resized, so if the source is larger the job fails; otherwise exactly min(source, dest) bytes are synced and the remainder of the device is left untouched.
  • Destination is a regular file: it's truncated (created if missing) to exactly the source's size, growing or shrinking it. Shrinking an existing non-empty file prompts for confirmation unless --yes is passed.

Build

CGO_ENABLED=0 go build -o clonetool .

Cross-compile for another OS/arch by setting GOOS/GOARCH (no cgo, so these all work from any host):

CGO_ENABLED=0 GOOS=linux   GOARCH=arm64 go build -o clonetool        .
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o clonetool.exe    .
CGO_ENABLED=0 GOOS=darwin  GOARCH=arm64 go build -o clonetool-darwin .

./build.sh writes all three of the above (plus linux/amd64) into dist/. It also stamps every binary it builds with the same build timestamp (via -ldflags -X main.buildTime=...), which is how the manager tells a stale self-deployed remote binary apart from a current one (see "Version/staleness check" above) — build all your binaries for a given release with one ./build.sh run so they compare equal to each other. A plain go build with no -ldflags leaves the timestamp at "dev", which still works for this check (it only ever compares equal to another literal copy of the same dev binary, never to a real timestamp).

Copy the resulting binary to the manager host. Source and destination hosts get it automatically (see self-deploy above), or place it yourself and point --remote-bin at it. The binary is architecture-specific — cross-compile if your hosts differ. Self-deploy streams a POSIX shell script over SSH, so a Windows host can be the manager or a local endpoint but cannot be an automatic deploy target — put clonetool.exe on it yourself and point --remote-bin at it.

Windows

clonetool runs on Windows and can clone physical drives and volumes, not just files:

# Whole disk to an image file
clonetool sync --source \\.\PhysicalDrive2 --dest D:\backup\disk2.img

# Image file back onto a disk (must not be larger than the disk)
clonetool sync --source D:\backup\disk2.img --dest \\.\PhysicalDrive2

# A single volume
clonetool sync --source \\.\E: --dest \\.\F:

• Raw-disk paths are \\.\PhysicalDrive<n> (whole disk) or \\.\<X>: (a volume). \\?\ also works. Forward slashes are accepted.
• Run from an elevated (Administrator) console to open a raw disk. There is no sudo fallback on Windows; --sudo is ignored. A permission error tells you to elevate.
• A raw-disk destination should have no mounted filesystem in use (take the disk offline in Disk Management, or target a volume that nothing else has open) — Windows blocks writes to a disk region owned by a mounted volume. Reading a live disk as the source is fine.
• Raw-disk I/O must be sector-aligned. --block-size must be a multiple of the drive's sector size (512 or 4096); the default 4M is. clonetool handles the final partial block itself.
• Device size is read with IOCTL_DISK_GET_LENGTH_INFO.

clone-disk

clonetool clone-disk --source LOC --dest LOC [options] clones a whole disk — the boot record, the partition table, and the contents of every partition — by rebuilding the layout on the target and then filling it from the source. Orchestrated from the manager exactly like sync: one control agent per side, data straight between source and target (push, or pull fallback), self-deploy, --sudo escalation.

It does not reimplement any filesystem knowledge — it calls the standard tools a rescue system already has and just moves their bytes between the two machines.

What it copies

1. Partition table. The source agent runs sfdisk -d (the canonical restorable dump for both GPT and MBR). The manager turns it into a device-independent restore script — dropping the device: line and last-lba so it re-sizes for the target, keeping the GPT disk GUID and every partition GUID/PARTUUID (so existing fstab / BCD / GRUB references still resolve) unless --new-ids is given. The target agent feeds that to sfdisk and re-reads the table.
2. Boot record / boot code. The 440-byte MBR bootstrap on every disk, plus — on MBR disks — the whole gap before the first partition (where GRUB's core.img lives), are raw-copied through the same block-diff engine sync uses, addressed as byte windows. On GPT disks the GPT structures come from sfdisk/sgdisk; a BIOS-boot partition (EF02) is always cloned raw.
3. Partition data, per partition, by the best available method:
   • fs-image (default when the tool exists): stream a filesystem-aware image — ntfsclone for NTFS, partclone.<fs> for ext*/xfs/btrfs/f2fs/fat/exfat, e2image as an ext fallback. Only used blocks move.
   • raw: the sync block-diff engine over the partition's byte window. Used for swap, unrecognised filesystems, --raw N,…, and whenever no image tool is installed. Re-runs move only changed blocks.
   • file-level (--file-level N,…): mkfs on the target + rsync (planned; not yet implemented — use --raw or an fs-image type).

Target sizing (same rules as sync)

• Target is a device: never grown. If the source layout fits, it is reproduced as-is and any trailing space on the target is left untouched. If the last partition(s) overflow, --allow-shrink will shrink them (filesystem then partition, from the last inward) with ntfsresize / resize2fs — this resizes the source filesystem in place before imaging, so it also needs --yes. Without --allow-shrink an over-large source is a hard error naming the partition.
• Target is a file: created and truncated to just what the layout needs (or --image-size SIZE). Shrinking an existing image prompts unless --yes.

NTFS / Windows partitions

• From a Linux rescue system: fully supported. ntfsclone clones Windows NTFS partitions (incl. Win10/11) at cluster level, copying only used clusters and preserving every NTFS feature. The volume must be clean — Windows Fast Startup and hibernation leave NTFS dirty; boot Windows once and shut down fully, or run ntfsfix -d first. ntfsclone does not fix booting: keep the partition GUIDs (default) so the existing BCD resolves, or run bcdboot from Windows recovery afterwards. This is the same approach Clonezilla uses.
• On Windows itself: raw / VSS only. There is no ntfsclone on Windows. clone-disk reads the partition table via IOCTL_DISK_GET_DRIVE_LAYOUT_EX, reproduces it by raw-copying the leading sectors (and the backup GPT), and block-clones each partition as a byte window of \\.\PhysicalDriveN. With --vss (default) it takes a Volume Shadow Copy of each NTFS volume first for a crash-consistent point-in-time source. This is not free-space-aware (a $Bitmap-driven skip may come later), and shrinking to a smaller target is not supported when the source is native Windows — use a target at least as large, or run the clone from a Linux rescue system to get ntfsclone.

Bootloader

By default clone-disk only reproduces boot structures and preserves disk/partition IDs — enough for a like-for-like replacement disk to boot. --reinstall-bootloader additionally mounts the cloned root (+ ESP), bind-mounts /dev /proc /sys, and runs grub-install + update-grub / grub-mkconfig in a chroot (Linux targets). It is best-effort and never fails the clone. For Windows, run bcdboot C:\Windows /s S: /f ALL from a recovery environment.

Examples

# Whole disk to an image file (image is sized to the layout)
clonetool clone-disk --source /dev/sda --dest /srv/sda.img

# Disk to disk on another host, orchestrated from a third machine
clonetool clone-disk --source box1:/dev/nvme0n1 --dest box2:/dev/nvme0n1

# Onto a smaller SSD, shrinking the last (data) partition to fit
clonetool clone-disk --source /dev/sda --dest /dev/sdb --allow-shrink --yes

# Fresh IDs so the clone can sit next to the original
clonetool clone-disk --source /dev/sda --dest /dev/sdb --new-ids

# Force a raw block clone of a partition an image tool would otherwise handle
clonetool clone-disk --source /dev/sda --dest /dev/sdb --raw 2

Tools the agents call (checked at probe time, reported if missing)

sfdisk, sgdisk, partprobe/blockdev, blkid, lsblk, losetup, ntfsclone, ntfsresize, ntfsfix, partclone.*, e2image, resize2fs, dumpe2fs, e2fsck, mkfs.*, mount/umount, rsync, grub-install, update-grub/grub-mkconfig. Windows: vssadmin / wmic, bcdboot, diskpart. A missing tool just means the affected partitions fall back to a raw block copy (or the run stops with a clear message if that isn't safe, e.g. a partition that must shrink).

clone-disk options

Flag	Default	Meaning
--parts LIST	all	Only clone these partition numbers (data only; the table still lists them all).
--raw LIST	—	Force a raw block clone for these partitions.
--file-level LIST	—	mkfs + rsync these partitions (not yet implemented).
--file-level-auto	off	Use file-level for fs types with no image cloner.
--allow-shrink	off	Permit shrinking trailing partitions to fit a smaller target (resizes the source fs in place; needs --yes).
--no-shrink	off	Never shrink; fail if the target is too small.
--new-ids	off	Randomize the GPT disk GUID / MBR signature on the target.
--reinstall-bootloader	off	Run grub-install / grub-mkconfig on the target after copy (Linux).
--vss	on	Windows source: take a Volume Shadow Copy per NTFS volume.
--image-size SIZE	auto	File target: image size (default = enough for the layout).

--block-size --job --yes --sudo --deploy --ssh --ssh-opt --remote-bin --connect-timeout --manager-host mean the same as for sync.

Usage

clonetool sync --source LOC --dest LOC [options]

LOC is either a local path (/dev/sdb, ./image.bin) or [user@]host:path for a path reached over SSH.

# Same machine
clonetool sync --source /dev/sda --dest /dev/sdb

# Two remote machines, orchestrated from a third
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img

# Re-run any time; only changed blocks move
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img

Options:

Flag	Default	Meaning
--block-size	4M	Block size (accepts K/M/G suffixes).
--job	—	Optional label shown in progress/log output.
--yes	off	Don't prompt before shrinking an existing destination file.
--sudo	auto	Block-device privilege escalation: auto (on a permission error), always, or never. Remote elevation needs passwordless sudo.
--deploy	true	Copy this binary to remote hosts that lack a runnable clonetool. --deploy=false to disable.
--connect-timeout	8	SSH connect timeout (seconds) used for the push/pull direction probe.
--ssh	ssh	ssh binary to use.
--ssh-opt	—	Extra -o OPT passed to ssh (repeatable).
--remote-bin	clonetool	Path to clonetool on remote hosts.
--manager-host	local hostname	Address a peer should use to reach this machine, needed only when source or dest is local to the manager and the other side is remote and ends up needing to dial back in (pull fallback).

clonetool version prints the binary's GOOS/GOARCH and build timestamp, e.g. clonetool linux/amd64 build=2024-06-01T12:00:00Z (the timestamp is used internally for the self-deploy staleness check above).

The status line shows read throughput for both sides and write throughput for the destination separately: rd(src) is the source reading and comparing its blocks, rd(dst) is the destination fingerprinting its current content (only while the trailing scan H/M segment is present), and wr(dst) is the destination actually writing changed blocks.

Caveats

• Block-device size detection is implemented on Linux (BLKGETSIZE64) and Windows (IOCTL_DISK_GET_LENGTH_INFO). On other systems only regular files can be synced.
• If a destination path doesn't exist yet, it's created as a regular file — clonetool won't create device nodes, so double-check device paths for typos before running.
• SSH host keys are accepted on first connect (StrictHostKeyChecking=accept-new) and rejected if they later change, same as normal SSH behavior.
• Because every run re-hashes both sides in full, a re-sync costs a full read of source and destination even when little changed — the win is in the bytes transferred, not the bytes read. The two reads run in parallel; the status line shows a trailing scan H/M until the destination fingerprint catches up.
• agent is an internal subcommand spawned automatically by sync; it's not meant to be run by hand, though it will work standalone for debugging.