# WARNING THIS TOOL IS COMPLETELY VIBE-CODED!!! THESE LINES ARE THE ONLY ONE I HAVE WRITTEN MYSELF!!!! I SWEAR I DID NOT READ MORE THAN 20 LINES OF CODE! USE AT YOUR OWN RISK (AS ALWAYS) I needed a Tool to sync 2 Files from a to b with differential transfer, all managed from a workstation c. Maybe i use this to learn go... Looks nice, builds fast, runs everywhere with one Binary. # clonetool Block-level sync for a large file or block device between two machines (or two paths on the same machine), driven from a third, passive "manager" machine. Single static Go binary, no runtime dependencies beyond the system `ssh` client for SSH endpoints. Runs on **Linux and Windows**. Endpoints are reached either over SSH (as below) or, for a host with no SSH server (typically Windows), by starting `clonetool listen` there and connecting to it over an encrypted, password- protected TCP channel — see **Listen mode (no SSH)** below. ## How it works - You run `clonetool sync` on the **manager**. Source, destination, and manager may be three entirely different machines — the manager never reads or writes a single block itself. It only spawns and talks to two **control agents** (`clonetool agent --role control`, one per side, local subprocess or over `ssh`) that do the stat/prepare/transfer work. - The two agents compare content by **SHA-256 hash per fixed-size block**. A block is only re-read once: it's hashed from the buffer it was read into, and that same buffer is what gets sent on if it differs — never read twice. - **Source and destination are scanned in parallel.** The destination side streams the hash of each block *as it computes it*, in block order; the source side consumes that stream and immediately reads, compares and (if it differs) sends that block. So the destination scan, the source scan and the transfer of changed blocks all overlap — there is no "scan the whole destination, then start" phase. The status line shows the transfer progress with a trailing `scan H/M` until the destination fingerprint is complete. - **No state is kept between runs.** Every sync re-reads and re-hashes the destination's *current* content and compares the source against that, so a re-run only moves the blocks that actually differ and nothing needs to be trusted from a previous run. (The destination side always does the destination hashing — the remote `sink` in push mode, the local destination agent in pull mode.) - Bulk data goes **directly between source and destination**, not through the manager. Each run tries: 1. **push** — the source agent connects straight to the destination host over SSH and streams changed blocks to it. 2. **pull** (fallback) — if push isn't possible (no route/keys in that direction), the destination agent connects to the source host instead and pulls. 3. If neither direction works, the job fails with both reasons named. Run the manager on the source or destination host, or set up SSH connectivity in at least one direction. - When source and destination are **both local** to the manager, no SSH is used at all — the source agent spawns the write-side helper as a plain local subprocess. - **Self-deploy:** if a remote endpoint has no runnable `clonetool` on `PATH` (or wherever `--remote-bin` points), **or the one that's there is a different build than this binary** (see below), the manager streams *this* binary to `~/.clonetool/bin/clonetool` on that host over the existing SSH connection and uses it — no install, no root. Disable with `--deploy=false` (a stale or missing binary is then an error, except a merely-stale `--remote-bin` is used anyway with a warning, since there's no way to correct it). If the copied binary won't execute there (wrong CPU architecture) the error says so; build one for the remote's arch (`CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build`) and put it on `PATH` or pass `--remote-bin`. - **Version/staleness check:** every binary has a build timestamp baked in (see Build below) and `clonetool version` prints it. Before using a remote `clonetool`, the manager runs its `version` there and compares that timestamp against its own — a mismatch (e.g. the remote was self-deployed from an older build of this tool) is treated the same as "not runnable" and triggers the self-deploy above, so the remote binary is kept in sync with whatever you're running locally. Only the build timestamp is compared, not `GOARCH` — that is expected to differ across a cross-compiled deploy. - **sudo:** if reading or writing an endpoint that is a **block device** fails with a permission error, `--sudo=auto` (the default) transparently restarts that side's agent — and the helper it spawns on the peer — under `sudo`. Locally that may prompt for a password on your terminal; **on a remote host it uses `sudo -n`, so passwordless sudo (NOPASSWD) must be configured there** (a password prompt can't work — the transfer protocol owns the SSH stdout stream). `--sudo=always` elevates from the start; `--sudo=never` never does. - Sizing rules: - Destination is a **block device**: it can't be resized, so if the source is larger the job fails; otherwise exactly the source's size is synced and the remainder of the device is left untouched. - Destination is a **regular file**: it's truncated (created if missing) to exactly the source's size, growing or shrinking it. Shrinking an existing non-empty file prompts for confirmation unless `--yes` is passed. ## Listen mode (no SSH) For a host that has no SSH server — a Windows machine, typically — start clonetool there in **listen mode** and point the manager's location at it with a `tcp://host:port/path` spec instead of `host:path`. There is no self-deploy over this path: you copy the binary to that host yourself and start it by hand. ``` # On the source host (e.g. Windows), started manually: clonetool listen --bind :9000 --password s3cret # On the manager, pull that source into a local/SSH destination: clonetool sync --source tcp://WINHOST:9000/C:\data\disk.img \ --dest backup:/srv/disk.img --password s3cret ``` - A listening host only ever **accepts** connections; it never dials out. So when the **source** is a `tcp://` endpoint, clonetool always uses the **pull** direction — the destination side connects to the listener and pulls the changed blocks. The destination must therefore have a network route to `host:port`. (Push is skipped for a listener source.) - **Password:** required for any `tcp://` endpoint, on both the listener (`clonetool listen`) and the manager (`clonetool sync`). Provide it with `--password`, `--password-file PATH`, or the `CLONETOOL_PASSWORD` environment variable (env/file keep it out of the process list and shell history). The manager passes it to the destination agent over the already- encrypted control channel, never on a command line. - **Security:** the channel is TLS 1.3 (the listener uses a fresh, in-memory self-signed certificate). The password is verified by a challenge-response bound to the TLS session (HMAC over the connection's exported keying material, key stretched with PBKDF2), so a man-in-the-middle can't authenticate even though the certificate isn't checked against a CA. Both ends authenticate each other; a wrong password fails the connection before any file is touched. Uses only the Go standard library — still one static binary. - The listener serves connections until you stop it with Ctrl-C. A single sync opens two connections to it (one for the stat, one for the data pull), so it must stay running for the whole job. ## Build ``` CGO_ENABLED=0 go build -o clonetool . ``` Cross-compile for another architecture by setting `GOARCH` (no cgo, so this works from any host): ``` CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o clonetool . ``` `./build.sh` writes linux/amd64 and linux/arm64 binaries into `dist/`. It also stamps every binary it builds with the same build timestamp (via `-ldflags -X main.buildTime=...`), which is how the manager tells a stale self-deployed remote binary apart from a current one (see "Version/staleness check" above) — build all your binaries for a given release with one `./build.sh` run so they compare equal to each other. A plain `go build` with no `-ldflags` leaves the timestamp at `"dev"`, which still works for this check (it only ever compares equal to another literal copy of the same `dev` binary, never to a real timestamp). Copy the resulting binary to the manager host. Source and destination hosts get it automatically (see self-deploy above), or place it yourself and point `--remote-bin` at it. The binary is architecture-specific — cross-compile if your hosts differ. ## Usage ``` clonetool sync --source LOC --dest LOC [options] ``` `LOC` is one of: - a local path — `/dev/sdb`, `./image.bin`, `C:\data\disk.img`, `\\.\PhysicalDrive0` - `[user@]host:path` — reached over SSH - `tcp://host:port/path` — a host running `clonetool listen` (see **Listen mode** above); no SSH needed there ``` # Same machine clonetool sync --source /dev/sda --dest /dev/sdb # Whole disk to an image file clonetool sync --source /dev/sda --dest /srv/sda.img # Two remote machines, orchestrated from a third clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img # Windows source with no SSH: it runs `clonetool listen --bind :9000 --password p` clonetool sync --source tcp://winbox:9000/C:\data\disk.img \ --dest backup-host:/srv/win.img --password p # Re-run any time; only changed blocks move clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img ``` Options for `sync`: | Flag | Default | Meaning | |---|---|---| | `--block-size` | `4M` | Block size (accepts `K`/`M`/`G` suffixes). | | `--job` | — | Optional label shown in progress/log output. | | `--yes` | off | Don't prompt before shrinking an existing destination file. | | `--sudo` | `auto` | Block-device privilege escalation: `auto` (on a permission error), `always`, or `never`. Remote elevation needs passwordless sudo. No-op on Windows (run as Administrator instead). | | `--deploy` | `true` | Copy this binary to SSH hosts that lack a runnable `clonetool`. `--deploy=false` to disable. (Never applies to `tcp://` endpoints.) | | `--connect-timeout` | `8` | SSH/TLS connect timeout (seconds) used for the push/pull direction probe. | | `--ssh` | `ssh` | ssh binary to use. | | `--ssh-opt` | — | Extra `-o OPT` passed to ssh (repeatable). | | `--remote-bin` | `clonetool` | Path to clonetool on remote SSH hosts. | | `--manager-host` | local hostname | Address a peer should use to reach this machine, needed only when source or dest is local to the manager *and* the other side is remote and ends up needing to dial back in (pull fallback). | | `--password` | — | Shared password for a `tcp://` listen endpoint. | | `--password-file` | — | Read the listen-endpoint password from a file (or set `CLONETOOL_PASSWORD`). | Options for `listen`: | Flag | Default | Meaning | |---|---|---| | `--bind` | `:9000` | Address to listen on, e.g. `:9000` or `0.0.0.0:9000`. | | `--password` | — | Shared password (or `--password-file`, or `CLONETOOL_PASSWORD`). Required. | | `--password-file` | — | Read the shared password from a file. | `clonetool version` prints the binary's `GOOS/GOARCH` and build timestamp, e.g. `clonetool linux/amd64 build=2024-06-01T12:00:00Z` (the timestamp is used internally for the self-deploy staleness check above). The status line shows read throughput for both sides and write throughput for the destination separately: `rd(src)` is the source reading and comparing its blocks, `rd(dst)` is the destination fingerprinting its current content (only while the trailing `scan H/M` segment is present), and `wr(dst)` is the destination actually writing changed blocks. ## Caveats - Block-device sizing uses `BLKGETSIZE64` on Linux and `IOCTL_DISK_GET_LENGTH_INFO` on Windows (`\\.\PhysicalDrive0`, `\\.\C:`). On other platforms (e.g. macOS) only regular files are supported; a device path there errors out. - **Windows raw disks:** syncing a `\\.\PhysicalDrive*` requires running clonetool as **Administrator**, and the disk should be offline/unmounted (a live, mounted volume can refuse writes or give inconsistent reads). Raw-disk I/O is sector-aligned automatically; `--block-size` must be a multiple of the sector size. `--sudo` does nothing on Windows. - If a destination path doesn't exist yet, it's created as a regular file — clonetool won't create device nodes, so double-check device paths for typos before running. - SSH host keys are accepted on first connect (`StrictHostKeyChecking=accept-new`) and rejected if they later change, same as normal SSH behavior. - Because every run re-hashes both sides in full, a re-sync costs a full read of source and destination even when little changed — the win is in the bytes transferred, not the bytes read. The two reads run in parallel; the status line shows a trailing `scan H/M` until the destination fingerprint catches up. - `agent` is an internal subcommand spawned automatically by `sync`; it's not meant to be run by hand, though it will work standalone for debugging.