324 lines
16 KiB
Markdown
324 lines
16 KiB
Markdown
# clonetool
|
|
|
|
Block-level sync for a large file or block device between two machines (or
|
|
two paths on the same machine), driven from a third, passive "manager"
|
|
machine. Single static Go binary, no runtime dependencies beyond the
|
|
system `ssh` client for remote endpoints.
|
|
|
|
Two subcommands:
|
|
|
|
- **`sync`** — mirror one file/device onto another (the original tool; see
|
|
most of this README).
|
|
- **`clone-disk`** — clone a whole system disk: boot record + partition
|
|
table + per-filesystem data, rebuilt on the target then filled from the
|
|
source. Same manager/agent/SSH/self-deploy model as `sync`; leans on the
|
|
standard disk tools already present on a rescue system (`sfdisk`,
|
|
`ntfsclone`, `partclone.*`, `e2image`, `rsync`, …). See
|
|
[clone-disk](#clone-disk) below.
|
|
|
|
## How it works
|
|
|
|
- You run `clonetool sync` on the **manager**. Source, destination, and
|
|
manager may be three entirely different machines — the manager never
|
|
reads or writes a single block itself. It only spawns and talks to two
|
|
**control agents** (`clonetool agent --role control`, one per side,
|
|
local subprocess or over `ssh`) that do the stat/prepare/transfer work.
|
|
- The two agents compare content by **SHA-256 hash per fixed-size block**.
|
|
A block is only re-read once: it's hashed from the buffer it was read
|
|
into, and that same buffer is what gets sent on if it differs — never
|
|
read twice.
|
|
- **Source and destination are scanned in parallel.** The destination side
|
|
streams the hash of each block *as it computes it*, in block order; the
|
|
source side consumes that stream and immediately reads, compares and (if
|
|
it differs) sends that block. So the destination scan, the source scan
|
|
and the transfer of changed blocks all overlap — there is no "scan the
|
|
whole destination, then start" phase. The status line shows the transfer
|
|
progress with a trailing `scan H/M` until the destination fingerprint is
|
|
complete.
|
|
- **No state is kept between runs.** Every sync re-reads and re-hashes the
|
|
destination's *current* content and compares the source against that, so
|
|
a re-run only moves the blocks that actually differ and nothing needs to
|
|
be trusted from a previous run. (The destination side always does the
|
|
destination hashing — the remote `sink` in push mode, the local
|
|
destination agent in pull mode.)
|
|
- Bulk data goes **directly between source and destination**, not through
|
|
the manager. Each run tries:
|
|
1. **push** — the source agent connects straight to the destination
|
|
host over SSH and streams changed blocks to it.
|
|
2. **pull** (fallback) — if push isn't possible (no route/keys in that
|
|
direction), the destination agent connects to the source host
|
|
instead and pulls.
|
|
3. If neither direction works, the job fails with both reasons named.
|
|
Run the manager on the source or destination host, or set up SSH
|
|
connectivity in at least one direction.
|
|
- When source and destination are **both local** to the manager, no SSH
|
|
is used at all — the source agent spawns the write-side helper as a
|
|
plain local subprocess.
|
|
- **Self-deploy:** if a remote endpoint has no runnable `clonetool` on
|
|
`PATH` (or wherever `--remote-bin` points), the manager streams *this*
|
|
binary to `~/.clonetool/bin/clonetool` on that host over the existing
|
|
SSH connection and uses it — no install, no root. Disable with
|
|
`--deploy=false`. If the copied binary won't execute there (wrong CPU
|
|
architecture) the error says so; build one for the remote's arch
|
|
(`CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build`) and put it on `PATH`
|
|
or pass `--remote-bin`.
|
|
- **sudo:** if reading or writing an endpoint that is a **block device**
|
|
fails with a permission error, `--sudo=auto` (the default) transparently
|
|
restarts that side's agent — and the helper it spawns on the peer —
|
|
under `sudo`. Locally that may prompt for a password on your terminal;
|
|
**on a remote host it uses `sudo -n`, so passwordless sudo (NOPASSWD)
|
|
must be configured there** (a password prompt can't work — the transfer
|
|
protocol owns the SSH stdout stream). `--sudo=always` elevates from the
|
|
start; `--sudo=never` never does.
|
|
- Sizing rules:
|
|
- Destination is a **block device**: it can't be resized, so if the
|
|
source is larger the job fails; otherwise exactly `min(source, dest)`
|
|
bytes are synced and the remainder of the device is left untouched.
|
|
- Destination is a **regular file**: it's truncated (created if
|
|
missing) to exactly the source's size, growing or shrinking it.
|
|
Shrinking an existing non-empty file prompts for confirmation unless
|
|
`--yes` is passed.
|
|
|
|
## Build
|
|
|
|
```
|
|
CGO_ENABLED=0 go build -o clonetool .
|
|
```
|
|
|
|
Cross-compile for another OS/arch by setting `GOOS`/`GOARCH` (no cgo, so
|
|
these all work from any host):
|
|
|
|
```
|
|
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o clonetool .
|
|
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o clonetool.exe .
|
|
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o clonetool-darwin .
|
|
```
|
|
|
|
`./build.sh` writes all three of the above (plus linux/amd64) into `dist/`.
|
|
|
|
Copy the resulting binary to the manager host. Source and destination
|
|
hosts get it automatically (see self-deploy above), or place it yourself
|
|
and point `--remote-bin` at it. The binary is architecture-specific —
|
|
cross-compile if your hosts differ. Self-deploy streams a POSIX shell
|
|
script over SSH, so a **Windows** host can be the manager or a local
|
|
endpoint but cannot be an automatic deploy target — put `clonetool.exe`
|
|
on it yourself and point `--remote-bin` at it.
|
|
|
|
## Windows
|
|
|
|
clonetool runs on Windows and can clone **physical drives and volumes**,
|
|
not just files:
|
|
|
|
```
|
|
# Whole disk to an image file
|
|
clonetool sync --source \\.\PhysicalDrive2 --dest D:\backup\disk2.img
|
|
|
|
# Image file back onto a disk (must not be larger than the disk)
|
|
clonetool sync --source D:\backup\disk2.img --dest \\.\PhysicalDrive2
|
|
|
|
# A single volume
|
|
clonetool sync --source \\.\E: --dest \\.\F:
|
|
```
|
|
|
|
- Raw-disk paths are `\\.\PhysicalDrive<n>` (whole disk) or `\\.\<X>:` (a
|
|
volume). `\\?\` also works. Forward slashes are accepted.
|
|
- **Run from an elevated (Administrator) console** to open a raw disk.
|
|
There is no `sudo` fallback on Windows; `--sudo` is ignored. A
|
|
permission error tells you to elevate.
|
|
- A raw-disk **destination** should have no mounted filesystem in use
|
|
(take the disk offline in Disk Management, or target a volume that
|
|
nothing else has open) — Windows blocks writes to a disk region owned by
|
|
a mounted volume. Reading a live disk as the **source** is fine.
|
|
- Raw-disk I/O must be sector-aligned. `--block-size` must be a multiple
|
|
of the drive's sector size (512 or 4096); the default 4M is. clonetool
|
|
handles the final partial block itself.
|
|
- Device size is read with `IOCTL_DISK_GET_LENGTH_INFO`.
|
|
|
|
## clone-disk
|
|
|
|
`clonetool clone-disk --source LOC --dest LOC [options]` clones a whole
|
|
disk — the boot record, the partition table, and the contents of every
|
|
partition — by **rebuilding** the layout on the target and then filling it
|
|
from the source. Orchestrated from the manager exactly like `sync`: one
|
|
control agent per side, data straight between source and target (push, or
|
|
pull fallback), self-deploy, `--sudo` escalation.
|
|
|
|
It does not reimplement any filesystem knowledge — it calls the standard
|
|
tools a rescue system already has and just moves their bytes between the two
|
|
machines.
|
|
|
|
### What it copies
|
|
|
|
1. **Partition table.** The source agent runs `sfdisk -d` (the canonical
|
|
restorable dump for both GPT and MBR). The manager turns it into a
|
|
device-independent restore script — dropping the `device:` line and
|
|
`last-lba` so it re-sizes for the target, keeping the GPT disk GUID and
|
|
every partition GUID/PARTUUID (so existing `fstab` / BCD / GRUB
|
|
references still resolve) unless `--new-ids` is given. The target agent
|
|
feeds that to `sfdisk` and re-reads the table.
|
|
2. **Boot record / boot code.** The 440-byte MBR bootstrap on every disk,
|
|
plus — on MBR disks — the whole gap before the first partition (where
|
|
GRUB's `core.img` lives), are raw-copied through the same block-diff
|
|
engine `sync` uses, addressed as byte windows. On GPT disks the
|
|
GPT structures come from `sfdisk`/`sgdisk`; a BIOS-boot partition
|
|
(`EF02`) is always cloned raw.
|
|
3. **Partition data**, per partition, by the best available method:
|
|
- **fs-image** (default when the tool exists): stream a
|
|
filesystem-aware image — `ntfsclone` for NTFS, `partclone.<fs>` for
|
|
ext*/xfs/btrfs/f2fs/fat/exfat, `e2image` as an ext fallback. Only
|
|
used blocks move.
|
|
- **raw**: the `sync` block-diff engine over the partition's byte
|
|
window. Used for swap, unrecognised filesystems, `--raw N,…`, and
|
|
whenever no image tool is installed. Re-runs move only changed blocks.
|
|
- **file-level** (`--file-level N,…`): `mkfs` on the target + `rsync`
|
|
*(planned; not yet implemented — use `--raw` or an fs-image type)*.
|
|
|
|
### Target sizing (same rules as `sync`)
|
|
|
|
- **Target is a device:** never grown. If the source layout fits, it is
|
|
reproduced as-is and any trailing space on the target is left untouched.
|
|
If the last partition(s) overflow, `--allow-shrink` will shrink them
|
|
(filesystem then partition, from the last inward) with `ntfsresize` /
|
|
`resize2fs` — this **resizes the source filesystem in place** before
|
|
imaging, so it also needs `--yes`. Without `--allow-shrink` an
|
|
over-large source is a hard error naming the partition.
|
|
- **Target is a file:** created and truncated to just what the layout
|
|
needs (or `--image-size SIZE`). Shrinking an existing image prompts
|
|
unless `--yes`.
|
|
|
|
### NTFS / Windows partitions
|
|
|
|
- **From a Linux rescue system: fully supported.** `ntfsclone` clones
|
|
Windows NTFS partitions (incl. Win10/11) at cluster level, copying only
|
|
used clusters and preserving every NTFS feature. The volume must be
|
|
*clean* — Windows Fast Startup and hibernation leave NTFS dirty; boot
|
|
Windows once and shut down fully, or run `ntfsfix -d` first. `ntfsclone`
|
|
does not fix booting: keep the partition GUIDs (default) so the existing
|
|
BCD resolves, or run `bcdboot` from Windows recovery afterwards. This is
|
|
the same approach Clonezilla uses.
|
|
- **On Windows itself: raw / VSS only.** There is no `ntfsclone` on
|
|
Windows. `clone-disk` reads the partition table via
|
|
`IOCTL_DISK_GET_DRIVE_LAYOUT_EX`, reproduces it by raw-copying the
|
|
leading sectors (and the backup GPT), and block-clones each partition
|
|
as a byte window of `\\.\PhysicalDriveN`. With `--vss` (default) it
|
|
takes a Volume Shadow Copy of each NTFS volume first for a
|
|
crash-consistent point-in-time source. This is **not** free-space-aware
|
|
(a `$Bitmap`-driven skip may come later), and shrinking to a smaller
|
|
target is not supported when the source is native Windows — use a
|
|
target at least as large, or run the clone from a Linux rescue system
|
|
to get `ntfsclone`.
|
|
|
|
### Bootloader
|
|
|
|
By default `clone-disk` only *reproduces* boot structures and preserves
|
|
disk/partition IDs — enough for a like-for-like replacement disk to boot.
|
|
`--reinstall-bootloader` additionally mounts the cloned root (+ ESP),
|
|
bind-mounts `/dev /proc /sys`, and runs `grub-install` + `update-grub` /
|
|
`grub-mkconfig` in a chroot (Linux targets). It is best-effort and never
|
|
fails the clone. For Windows, run `bcdboot C:\Windows /s S: /f ALL` from a
|
|
recovery environment.
|
|
|
|
### Examples
|
|
|
|
```
|
|
# Whole disk to an image file (image is sized to the layout)
|
|
clonetool clone-disk --source /dev/sda --dest /srv/sda.img
|
|
|
|
# Disk to disk on another host, orchestrated from a third machine
|
|
clonetool clone-disk --source box1:/dev/nvme0n1 --dest box2:/dev/nvme0n1
|
|
|
|
# Onto a smaller SSD, shrinking the last (data) partition to fit
|
|
clonetool clone-disk --source /dev/sda --dest /dev/sdb --allow-shrink --yes
|
|
|
|
# Fresh IDs so the clone can sit next to the original
|
|
clonetool clone-disk --source /dev/sda --dest /dev/sdb --new-ids
|
|
|
|
# Force a raw block clone of a partition an image tool would otherwise handle
|
|
clonetool clone-disk --source /dev/sda --dest /dev/sdb --raw 2
|
|
```
|
|
|
|
### Tools the agents call (checked at probe time, reported if missing)
|
|
|
|
`sfdisk`, `sgdisk`, `partprobe`/`blockdev`, `blkid`, `lsblk`, `losetup`,
|
|
`ntfsclone`, `ntfsresize`, `ntfsfix`, `partclone.*`, `e2image`,
|
|
`resize2fs`, `dumpe2fs`, `e2fsck`, `mkfs.*`, `mount`/`umount`, `rsync`,
|
|
`grub-install`, `update-grub`/`grub-mkconfig`. Windows: `vssadmin` /
|
|
`wmic`, `bcdboot`, `diskpart`. A missing tool just means the affected
|
|
partitions fall back to a raw block copy (or the run stops with a clear
|
|
message if that isn't safe, e.g. a partition that must shrink).
|
|
|
|
### clone-disk options
|
|
|
|
| Flag | Default | Meaning |
|
|
|---|---|---|
|
|
| `--parts LIST` | all | Only clone these partition numbers (data only; the table still lists them all). |
|
|
| `--raw LIST` | — | Force a raw block clone for these partitions. |
|
|
| `--file-level LIST` | — | `mkfs` + `rsync` these partitions (not yet implemented). |
|
|
| `--file-level-auto` | off | Use file-level for fs types with no image cloner. |
|
|
| `--allow-shrink` | off | Permit shrinking trailing partitions to fit a smaller target (resizes the **source** fs in place; needs `--yes`). |
|
|
| `--no-shrink` | off | Never shrink; fail if the target is too small. |
|
|
| `--new-ids` | off | Randomize the GPT disk GUID / MBR signature on the target. |
|
|
| `--reinstall-bootloader` | off | Run `grub-install` / `grub-mkconfig` on the target after copy (Linux). |
|
|
| `--vss` | on | Windows source: take a Volume Shadow Copy per NTFS volume. |
|
|
| `--image-size SIZE` | auto | File target: image size (default = enough for the layout). |
|
|
|
|
`--block-size --job --yes --sudo --deploy --ssh --ssh-opt --remote-bin
|
|
--connect-timeout --manager-host` mean the same as for `sync`.
|
|
|
|
## Usage
|
|
|
|
```
|
|
clonetool sync --source LOC --dest LOC [options]
|
|
```
|
|
|
|
`LOC` is either a local path (`/dev/sdb`, `./image.bin`) or
|
|
`[user@]host:path` for a path reached over SSH.
|
|
|
|
```
|
|
# Same machine
|
|
clonetool sync --source /dev/sda --dest /dev/sdb
|
|
|
|
# Two remote machines, orchestrated from a third
|
|
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img
|
|
|
|
# Re-run any time; only changed blocks move
|
|
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img
|
|
```
|
|
|
|
Options:
|
|
|
|
| Flag | Default | Meaning |
|
|
|---|---|---|
|
|
| `--block-size` | `4M` | Block size (accepts `K`/`M`/`G` suffixes). |
|
|
| `--job` | — | Optional label shown in progress/log output. |
|
|
| `--yes` | off | Don't prompt before shrinking an existing destination file. |
|
|
| `--sudo` | `auto` | Block-device privilege escalation: `auto` (on a permission error), `always`, or `never`. Remote elevation needs passwordless sudo. |
|
|
| `--deploy` | `true` | Copy this binary to remote hosts that lack a runnable `clonetool`. `--deploy=false` to disable. |
|
|
| `--connect-timeout` | `8` | SSH connect timeout (seconds) used for the push/pull direction probe. |
|
|
| `--ssh` | `ssh` | ssh binary to use. |
|
|
| `--ssh-opt` | — | Extra `-o OPT` passed to ssh (repeatable). |
|
|
| `--remote-bin` | `clonetool` | Path to clonetool on remote hosts. |
|
|
| `--manager-host` | local hostname | Address a peer should use to reach this machine, needed only when source or dest is local to the manager *and* the other side is remote and ends up needing to dial back in (pull fallback). |
|
|
|
|
`clonetool version` prints the binary's `GOOS/GOARCH` (used internally for
|
|
the self-deploy check).
|
|
|
|
## Caveats
|
|
|
|
- Block-device size detection is implemented on Linux (`BLKGETSIZE64`) and
|
|
Windows (`IOCTL_DISK_GET_LENGTH_INFO`). On other systems only regular
|
|
files can be synced.
|
|
- If a destination path doesn't exist yet, it's created as a regular
|
|
file — clonetool won't create device nodes, so double-check device
|
|
paths for typos before running.
|
|
- SSH host keys are accepted on first connect (`StrictHostKeyChecking=accept-new`)
|
|
and rejected if they later change, same as normal SSH behavior.
|
|
- Because every run re-hashes both sides in full, a re-sync costs a full
|
|
read of source and destination even when little changed — the win is in
|
|
the bytes transferred, not the bytes read. The two reads run in
|
|
parallel; the status line shows a trailing `scan H/M` until the
|
|
destination fingerprint catches up.
|
|
- `agent` is an internal subcommand spawned automatically by `sync`; it's
|
|
not meant to be run by hand, though it will work standalone for
|
|
debugging.
|