# clonetool

Block-level sync for a large file or block device between two machines (or
two paths on the same machine), driven from a third, passive "manager"
machine. Single static Go binary, no runtime dependencies beyond the
system `ssh` client for remote endpoints.

Two subcommands:

- **`sync`** — mirror one file/device onto another (the original tool; see
  most of this README).
- **`clone-disk`** — clone a whole system disk: boot record + partition
  table + per-filesystem data, rebuilt on the target then filled from the
  source. Same manager/agent/SSH/self-deploy model as `sync`; leans on the
  standard disk tools already present on a rescue system (`sfdisk`,
  `ntfsclone`, `partclone.*`, `e2image`, `rsync`, …). See
  [clone-disk](#clone-disk) below.

## How it works

- You run `clonetool sync` on the **manager**. Source, destination, and
  manager may be three entirely different machines — the manager never
  reads or writes a single block itself. It only spawns and talks to two
  **control agents** (`clonetool agent --role control`, one per side,
  local subprocess or over `ssh`) that do the stat/prepare/transfer work.
- The two agents compare content by **SHA-256 hash per fixed-size block**.
  A block is only re-read once: it's hashed from the buffer it was read
  into, and that same buffer is what gets sent on if it differs — never
  read twice.
- **Source and destination are scanned in parallel.** The destination side
  streams the hash of each block *as it computes it*, in block order; the
  source side consumes that stream and immediately reads, compares and (if
  it differs) sends that block. So the destination scan, the source scan
  and the transfer of changed blocks all overlap — there is no "scan the
  whole destination, then start" phase. The status line shows the transfer
  progress with a trailing `scan H/M` until the destination fingerprint is
  complete.
- **No state is kept between runs.** Every sync re-reads and re-hashes the
  destination's *current* content and compares the source against that, so
  a re-run only moves the blocks that actually differ and nothing needs to
  be trusted from a previous run. (The destination side always does the
  destination hashing — the remote `sink` in push mode, the local
  destination agent in pull mode.)
- Bulk data goes **directly between source and destination**, not through
  the manager. Each run tries:
  1. **push** — the source agent connects straight to the destination
     host over SSH and streams changed blocks to it.
  2. **pull** (fallback) — if push isn't possible (no route/keys in that
     direction), the destination agent connects to the source host
     instead and pulls.
  3. If neither direction works, the job fails with both reasons named.
     Run the manager on the source or destination host, or set up SSH
     connectivity in at least one direction.
  - When source and destination are **both local** to the manager, no SSH
    is used at all — the source agent spawns the write-side helper as a
    plain local subprocess.
- **Self-deploy:** if a remote endpoint has no runnable `clonetool` on
  `PATH` (or wherever `--remote-bin` points), the manager streams *this*
  binary to `~/.clonetool/bin/clonetool` on that host over the existing
  SSH connection and uses it — no install, no root. Disable with
  `--deploy=false`. If the copied binary won't execute there (wrong CPU
  architecture) the error says so; build one for the remote's arch
  (`CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build`) and put it on `PATH`
  or pass `--remote-bin`.
- **sudo:** if reading or writing an endpoint that is a **block device**
  fails with a permission error, `--sudo=auto` (the default) transparently
  restarts that side's agent — and the helper it spawns on the peer —
  under `sudo`. Locally that may prompt for a password on your terminal;
  **on a remote host it uses `sudo -n`, so passwordless sudo (NOPASSWD)
  must be configured there** (a password prompt can't work — the transfer
  protocol owns the SSH stdout stream). `--sudo=always` elevates from the
  start; `--sudo=never` never does.
- Sizing rules:
  - Destination is a **block device**: it can't be resized, so if the
    source is larger the job fails; otherwise exactly `min(source, dest)`
    bytes are synced and the remainder of the device is left untouched.
  - Destination is a **regular file**: it's truncated (created if
    missing) to exactly the source's size, growing or shrinking it.
    Shrinking an existing non-empty file prompts for confirmation unless
    `--yes` is passed.

## Build

```
CGO_ENABLED=0 go build -o clonetool .
```

Cross-compile for another OS/arch by setting `GOOS`/`GOARCH` (no cgo, so
these all work from any host):

```
CGO_ENABLED=0 GOOS=linux   GOARCH=arm64 go build -o clonetool        .
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o clonetool.exe    .
CGO_ENABLED=0 GOOS=darwin  GOARCH=arm64 go build -o clonetool-darwin .
```

`./build.sh` writes all three of the above (plus linux/amd64) into `dist/`.

Copy the resulting binary to the manager host. Source and destination
hosts get it automatically (see self-deploy above), or place it yourself
and point `--remote-bin` at it. The binary is architecture-specific —
cross-compile if your hosts differ. Self-deploy streams a POSIX shell
script over SSH, so a **Windows** host can be the manager or a local
endpoint but cannot be an automatic deploy target — put `clonetool.exe`
on it yourself and point `--remote-bin` at it.

## Windows

clonetool runs on Windows and can clone **physical drives and volumes**,
not just files:

```
# Whole disk to an image file
clonetool sync --source \\.\PhysicalDrive2 --dest D:\backup\disk2.img

# Image file back onto a disk (must not be larger than the disk)
clonetool sync --source D:\backup\disk2.img --dest \\.\PhysicalDrive2

# A single volume
clonetool sync --source \\.\E: --dest \\.\F:
```

- Raw-disk paths are `\\.\PhysicalDrive<n>` (whole disk) or `\\.\<X>:` (a
  volume). `\\?\` also works. Forward slashes are accepted.
- **Run from an elevated (Administrator) console** to open a raw disk.
  There is no `sudo` fallback on Windows; `--sudo` is ignored. A
  permission error tells you to elevate.
- A raw-disk **destination** should have no mounted filesystem in use
  (take the disk offline in Disk Management, or target a volume that
  nothing else has open) — Windows blocks writes to a disk region owned by
  a mounted volume. Reading a live disk as the **source** is fine.
- Raw-disk I/O must be sector-aligned. `--block-size` must be a multiple
  of the drive's sector size (512 or 4096); the default 4M is. clonetool
  handles the final partial block itself.
- Device size is read with `IOCTL_DISK_GET_LENGTH_INFO`.

## clone-disk

`clonetool clone-disk --source LOC --dest LOC [options]` clones a whole
disk — the boot record, the partition table, and the contents of every
partition — by **rebuilding** the layout on the target and then filling it
from the source. Orchestrated from the manager exactly like `sync`: one
control agent per side, data straight between source and target (push, or
pull fallback), self-deploy, `--sudo` escalation.

It does not reimplement any filesystem knowledge — it calls the standard
tools a rescue system already has and just moves their bytes between the two
machines.

### What it copies

1. **Partition table.** The source agent runs `sfdisk -d` (the canonical
   restorable dump for both GPT and MBR). The manager turns it into a
   device-independent restore script — dropping the `device:` line and
   `last-lba` so it re-sizes for the target, keeping the GPT disk GUID and
   every partition GUID/PARTUUID (so existing `fstab` / BCD / GRUB
   references still resolve) unless `--new-ids` is given. The target agent
   feeds that to `sfdisk` and re-reads the table.
2. **Boot record / boot code.** The 440-byte MBR bootstrap on every disk,
   plus — on MBR disks — the whole gap before the first partition (where
   GRUB's `core.img` lives), are raw-copied through the same block-diff
   engine `sync` uses, addressed as byte windows. On GPT disks the
   GPT structures come from `sfdisk`/`sgdisk`; a BIOS-boot partition
   (`EF02`) is always cloned raw.
3. **Partition data**, per partition, by the best available method:
   - **fs-image** (default when the tool exists): stream a
     filesystem-aware image — `ntfsclone` for NTFS, `partclone.<fs>` for
     ext*/xfs/btrfs/f2fs/fat/exfat, `e2image` as an ext fallback. Only
     used blocks move.
   - **raw**: the `sync` block-diff engine over the partition's byte
     window. Used for swap, unrecognised filesystems, `--raw N,…`, and
     whenever no image tool is installed. Re-runs move only changed blocks.
   - **file-level** (`--file-level N,…`): `mkfs` on the target + `rsync`
     *(planned; not yet implemented — use `--raw` or an fs-image type)*.

### Target sizing (same rules as `sync`)

- **Target is a device:** never grown. If the source layout fits, it is
  reproduced as-is and any trailing space on the target is left untouched.
  If the last partition(s) overflow, `--allow-shrink` will shrink them
  (filesystem then partition, from the last inward) with `ntfsresize` /
  `resize2fs` — this **resizes the source filesystem in place** before
  imaging, so it also needs `--yes`. Without `--allow-shrink` an
  over-large source is a hard error naming the partition.
- **Target is a file:** created and truncated to just what the layout
  needs (or `--image-size SIZE`). Shrinking an existing image prompts
  unless `--yes`.

### NTFS / Windows partitions

- **From a Linux rescue system: fully supported.** `ntfsclone` clones
  Windows NTFS partitions (incl. Win10/11) at cluster level, copying only
  used clusters and preserving every NTFS feature. The volume must be
  *clean* — Windows Fast Startup and hibernation leave NTFS dirty; boot
  Windows once and shut down fully, or run `ntfsfix -d` first. `ntfsclone`
  does not fix booting: keep the partition GUIDs (default) so the existing
  BCD resolves, or run `bcdboot` from Windows recovery afterwards. This is
  the same approach Clonezilla uses.
- **On Windows itself: raw / VSS only.** There is no `ntfsclone` on
  Windows. `clone-disk` reads the partition table via
  `IOCTL_DISK_GET_DRIVE_LAYOUT_EX`, reproduces it by raw-copying the
  leading sectors (and the backup GPT), and block-clones each partition
  as a byte window of `\\.\PhysicalDriveN`. With `--vss` (default) it
  takes a Volume Shadow Copy of each NTFS volume first for a
  crash-consistent point-in-time source. This is **not** free-space-aware
  (a `$Bitmap`-driven skip may come later), and shrinking to a smaller
  target is not supported when the source is native Windows — use a
  target at least as large, or run the clone from a Linux rescue system
  to get `ntfsclone`.

### Bootloader

By default `clone-disk` only *reproduces* boot structures and preserves
disk/partition IDs — enough for a like-for-like replacement disk to boot.
`--reinstall-bootloader` additionally mounts the cloned root (+ ESP),
bind-mounts `/dev /proc /sys`, and runs `grub-install` + `update-grub` /
`grub-mkconfig` in a chroot (Linux targets). It is best-effort and never
fails the clone. For Windows, run `bcdboot C:\Windows /s S: /f ALL` from a
recovery environment.

### Examples

```
# Whole disk to an image file (image is sized to the layout)
clonetool clone-disk --source /dev/sda --dest /srv/sda.img

# Disk to disk on another host, orchestrated from a third machine
clonetool clone-disk --source box1:/dev/nvme0n1 --dest box2:/dev/nvme0n1

# Onto a smaller SSD, shrinking the last (data) partition to fit
clonetool clone-disk --source /dev/sda --dest /dev/sdb --allow-shrink --yes

# Fresh IDs so the clone can sit next to the original
clonetool clone-disk --source /dev/sda --dest /dev/sdb --new-ids

# Force a raw block clone of a partition an image tool would otherwise handle
clonetool clone-disk --source /dev/sda --dest /dev/sdb --raw 2
```

### Tools the agents call (checked at probe time, reported if missing)

`sfdisk`, `sgdisk`, `partprobe`/`blockdev`, `blkid`, `lsblk`, `losetup`,
`ntfsclone`, `ntfsresize`, `ntfsfix`, `partclone.*`, `e2image`,
`resize2fs`, `dumpe2fs`, `e2fsck`, `mkfs.*`, `mount`/`umount`, `rsync`,
`grub-install`, `update-grub`/`grub-mkconfig`. Windows: `vssadmin` /
`wmic`, `bcdboot`, `diskpart`. A missing tool just means the affected
partitions fall back to a raw block copy (or the run stops with a clear
message if that isn't safe, e.g. a partition that must shrink).

### clone-disk options

| Flag | Default | Meaning |
|---|---|---|
| `--parts LIST` | all | Only clone these partition numbers (data only; the table still lists them all). |
| `--raw LIST` | — | Force a raw block clone for these partitions. |
| `--file-level LIST` | — | `mkfs` + `rsync` these partitions (not yet implemented). |
| `--file-level-auto` | off | Use file-level for fs types with no image cloner. |
| `--allow-shrink` | off | Permit shrinking trailing partitions to fit a smaller target (resizes the **source** fs in place; needs `--yes`). |
| `--no-shrink` | off | Never shrink; fail if the target is too small. |
| `--new-ids` | off | Randomize the GPT disk GUID / MBR signature on the target. |
| `--reinstall-bootloader` | off | Run `grub-install` / `grub-mkconfig` on the target after copy (Linux). |
| `--vss` | on | Windows source: take a Volume Shadow Copy per NTFS volume. |
| `--image-size SIZE` | auto | File target: image size (default = enough for the layout). |

`--block-size --job --yes --sudo --deploy --ssh --ssh-opt --remote-bin
--connect-timeout --manager-host` mean the same as for `sync`.

## Usage

```
clonetool sync --source LOC --dest LOC [options]
```

`LOC` is either a local path (`/dev/sdb`, `./image.bin`) or
`[user@]host:path` for a path reached over SSH.

```
# Same machine
clonetool sync --source /dev/sda --dest /dev/sdb

# Two remote machines, orchestrated from a third
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img

# Re-run any time; only changed blocks move
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img
```

Options:

| Flag | Default | Meaning |
|---|---|---|
| `--block-size` | `4M` | Block size (accepts `K`/`M`/`G` suffixes). |
| `--job` | — | Optional label shown in progress/log output. |
| `--yes` | off | Don't prompt before shrinking an existing destination file. |
| `--sudo` | `auto` | Block-device privilege escalation: `auto` (on a permission error), `always`, or `never`. Remote elevation needs passwordless sudo. |
| `--deploy` | `true` | Copy this binary to remote hosts that lack a runnable `clonetool`. `--deploy=false` to disable. |
| `--connect-timeout` | `8` | SSH connect timeout (seconds) used for the push/pull direction probe. |
| `--ssh` | `ssh` | ssh binary to use. |
| `--ssh-opt` | — | Extra `-o OPT` passed to ssh (repeatable). |
| `--remote-bin` | `clonetool` | Path to clonetool on remote hosts. |
| `--manager-host` | local hostname | Address a peer should use to reach this machine, needed only when source or dest is local to the manager *and* the other side is remote and ends up needing to dial back in (pull fallback). |

`clonetool version` prints the binary's `GOOS/GOARCH` (used internally for
the self-deploy check).

## Caveats

- Block-device size detection is implemented on Linux (`BLKGETSIZE64`) and
  Windows (`IOCTL_DISK_GET_LENGTH_INFO`). On other systems only regular
  files can be synced.
- If a destination path doesn't exist yet, it's created as a regular
  file — clonetool won't create device nodes, so double-check device
  paths for typos before running.
- SSH host keys are accepted on first connect (`StrictHostKeyChecking=accept-new`)
  and rejected if they later change, same as normal SSH behavior.
- Because every run re-hashes both sides in full, a re-sync costs a full
  read of source and destination even when little changed — the win is in
  the bytes transferred, not the bytes read. The two reads run in
  parallel; the status line shows a trailing `scan H/M` until the
  destination fingerprint catches up.
- `agent` is an internal subcommand spawned automatically by `sync`; it's
  not meant to be run by hand, though it will work standalone for
  debugging.