264 lines
13 KiB
Markdown
264 lines
13 KiB
Markdown
# WARNING
|
|
THIS TOOL IS COMPLETELY VIBE-CODED!!!
|
|
|
|
THESE LINES ARE THE ONLY ONE I HAVE WRITTEN MYSELF!!!!
|
|
|
|
I SWEAR I DID NOT READ MORE THAN 20 LINES OF CODE!
|
|
|
|
USE AT YOUR OWN RISK (AS ALWAYS)
|
|
|
|
|
|
I needed a Tool to sync 2 Files from a to b with differential transfer, all managed from a workstation c.
|
|
|
|
Maybe i use this to learn go...
|
|
|
|
Looks nice, builds fast, runs everywhere with one Binary.
|
|
|
|
|
|
|
|
# clonetool
|
|
|
|
Block-level sync for a large file or block device between two machines (or
|
|
two paths on the same machine), driven from a third, passive "manager"
|
|
machine. Single static Go binary, no runtime dependencies beyond the
|
|
system `ssh` client for SSH endpoints.
|
|
|
|
Runs on **Linux and Windows**. Endpoints are reached either over SSH (as
|
|
below) or, for a host with no SSH server (typically Windows), by starting
|
|
`clonetool listen` there and connecting to it over an encrypted, password-
|
|
protected TCP channel — see **Listen mode (no SSH)** below.
|
|
|
|
## How it works
|
|
|
|
- You run `clonetool sync` on the **manager**. Source, destination, and
|
|
manager may be three entirely different machines — the manager never
|
|
reads or writes a single block itself. It only spawns and talks to two
|
|
**control agents** (`clonetool agent --role control`, one per side,
|
|
local subprocess or over `ssh`) that do the stat/prepare/transfer work.
|
|
- The two agents compare content by **SHA-256 hash per fixed-size block**.
|
|
A block is only re-read once: it's hashed from the buffer it was read
|
|
into, and that same buffer is what gets sent on if it differs — never
|
|
read twice.
|
|
- **Source and destination are scanned in parallel.** The destination side
|
|
streams the hash of each block *as it computes it*, in block order; the
|
|
source side consumes that stream and immediately reads, compares and (if
|
|
it differs) sends that block. So the destination scan, the source scan
|
|
and the transfer of changed blocks all overlap — there is no "scan the
|
|
whole destination, then start" phase. The status line shows the transfer
|
|
progress with a trailing `scan H/M` until the destination fingerprint is
|
|
complete.
|
|
- **No state is kept between runs.** Every sync re-reads and re-hashes the
|
|
destination's *current* content and compares the source against that, so
|
|
a re-run only moves the blocks that actually differ and nothing needs to
|
|
be trusted from a previous run. (The destination side always does the
|
|
destination hashing — the remote `sink` in push mode, the local
|
|
destination agent in pull mode.)
|
|
- Bulk data goes **directly between source and destination**, not through
|
|
the manager. Each run tries:
|
|
1. **push** — the source agent connects straight to the destination
|
|
host over SSH and streams changed blocks to it.
|
|
2. **pull** (fallback) — if push isn't possible (no route/keys in that
|
|
direction), the destination agent connects to the source host
|
|
instead and pulls.
|
|
3. If neither direction works, the job fails with both reasons named.
|
|
Run the manager on the source or destination host, or set up SSH
|
|
connectivity in at least one direction.
|
|
- When source and destination are **both local** to the manager, no SSH
|
|
is used at all — the source agent spawns the write-side helper as a
|
|
plain local subprocess.
|
|
- **Self-deploy:** if a remote endpoint has no runnable `clonetool` on
|
|
`PATH` (or wherever `--remote-bin` points), **or the one that's there is
|
|
a different build than this binary** (see below), the manager streams
|
|
*this* binary to `~/.clonetool/bin/clonetool` on that host over the
|
|
existing SSH connection and uses it — no install, no root. Disable with
|
|
`--deploy=false` (a stale or missing binary is then an error, except a
|
|
merely-stale `--remote-bin` is used anyway with a warning, since there's
|
|
no way to correct it). If the copied binary won't execute there (wrong
|
|
CPU architecture) the error says so; build one for the remote's arch
|
|
(`CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build`) and put it on `PATH`
|
|
or pass `--remote-bin`.
|
|
- **Version/staleness check:** every binary has a build timestamp baked in
|
|
(see Build below) and `clonetool version` prints it. Before using a
|
|
remote `clonetool`, the manager runs its `version` there and compares
|
|
that timestamp against its own — a mismatch (e.g. the remote was
|
|
self-deployed from an older build of this tool) is treated the same as
|
|
"not runnable" and triggers the self-deploy above, so the remote binary
|
|
is kept in sync with whatever you're running locally. Only the build
|
|
timestamp is compared, not `GOARCH` — that is expected to differ across
|
|
a cross-compiled deploy.
|
|
- **sudo:** if reading or writing an endpoint that is a **block device**
|
|
fails with a permission error, `--sudo=auto` (the default) transparently
|
|
restarts that side's agent — and the helper it spawns on the peer —
|
|
under `sudo`. Locally that may prompt for a password on your terminal;
|
|
**on a remote host it uses `sudo -n`, so passwordless sudo (NOPASSWD)
|
|
must be configured there** (a password prompt can't work — the transfer
|
|
protocol owns the SSH stdout stream). `--sudo=always` elevates from the
|
|
start; `--sudo=never` never does.
|
|
- Sizing rules:
|
|
- Destination is a **block device**: it can't be resized, so if the
|
|
source is larger the job fails; otherwise exactly the source's size is
|
|
synced and the remainder of the device is left untouched.
|
|
- Destination is a **regular file**: it's truncated (created if
|
|
missing) to exactly the source's size, growing or shrinking it.
|
|
Shrinking an existing non-empty file prompts for confirmation unless
|
|
`--yes` is passed.
|
|
|
|
## Listen mode (no SSH)
|
|
|
|
For a host that has no SSH server — a Windows machine, typically — start
|
|
clonetool there in **listen mode** and point the manager's location at it
|
|
with a `tcp://host:port/path` spec instead of `host:path`. There is no
|
|
self-deploy over this path: you copy the binary to that host yourself and
|
|
start it by hand.
|
|
|
|
```
|
|
# On the source host (e.g. Windows), started manually:
|
|
clonetool listen --bind :9000 --password s3cret
|
|
|
|
# On the manager, pull that source into a local/SSH destination:
|
|
clonetool sync --source tcp://WINHOST:9000/C:\data\disk.img \
|
|
--dest backup:/srv/disk.img --password s3cret
|
|
```
|
|
|
|
- A listening host only ever **accepts** connections; it never dials out.
|
|
So when the **source** is a `tcp://` endpoint, clonetool always uses the
|
|
**pull** direction — the destination side connects to the listener and
|
|
pulls the changed blocks. The destination must therefore have a network
|
|
route to `host:port`. (Push is skipped for a listener source.)
|
|
- **Password:** required for any `tcp://` endpoint, on both the listener
|
|
(`clonetool listen`) and the manager (`clonetool sync`). Provide it with
|
|
`--password`, `--password-file PATH`, or the `CLONETOOL_PASSWORD`
|
|
environment variable (env/file keep it out of the process list and shell
|
|
history). The manager passes it to the destination agent over the already-
|
|
encrypted control channel, never on a command line.
|
|
- **Security:** the channel is TLS 1.3 (the listener uses a fresh, in-memory
|
|
self-signed certificate). The password is verified by a challenge-response
|
|
bound to the TLS session (HMAC over the connection's exported keying
|
|
material, key stretched with PBKDF2), so a man-in-the-middle can't
|
|
authenticate even though the certificate isn't checked against a CA. Both
|
|
ends authenticate each other; a wrong password fails the connection before
|
|
any file is touched. Uses only the Go standard library — still one static
|
|
binary.
|
|
- The listener serves connections until you stop it with Ctrl-C. A single
|
|
sync opens two connections to it (one for the stat, one for the data
|
|
pull), so it must stay running for the whole job.
|
|
|
|
## Build
|
|
|
|
```
|
|
CGO_ENABLED=0 go build -o clonetool .
|
|
```
|
|
|
|
Cross-compile for another architecture by setting `GOARCH` (no cgo, so
|
|
this works from any host):
|
|
|
|
```
|
|
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o clonetool .
|
|
```
|
|
|
|
`./build.sh` writes linux/amd64 and linux/arm64 binaries into `dist/`. It
|
|
also stamps every binary it builds with the same build timestamp (via
|
|
`-ldflags -X main.buildTime=...`), which is how the manager tells a stale
|
|
self-deployed remote binary apart from a current one (see "Version/staleness
|
|
check" above) — build all your binaries for a given release with one
|
|
`./build.sh` run so they compare equal to each other. A plain `go build`
|
|
with no `-ldflags` leaves the timestamp at `"dev"`, which still works for
|
|
this check (it only ever compares equal to another literal copy of the same
|
|
`dev` binary, never to a real timestamp).
|
|
|
|
Copy the resulting binary to the manager host. Source and destination
|
|
hosts get it automatically (see self-deploy above), or place it yourself
|
|
and point `--remote-bin` at it. The binary is architecture-specific —
|
|
cross-compile if your hosts differ.
|
|
|
|
## Usage
|
|
|
|
```
|
|
clonetool sync --source LOC --dest LOC [options]
|
|
```
|
|
|
|
`LOC` is one of:
|
|
|
|
- a local path — `/dev/sdb`, `./image.bin`, `C:\data\disk.img`, `\\.\PhysicalDrive0`
|
|
- `[user@]host:path` — reached over SSH
|
|
- `tcp://host:port/path` — a host running `clonetool listen` (see **Listen
|
|
mode** above); no SSH needed there
|
|
|
|
```
|
|
# Same machine
|
|
clonetool sync --source /dev/sda --dest /dev/sdb
|
|
|
|
# Whole disk to an image file
|
|
clonetool sync --source /dev/sda --dest /srv/sda.img
|
|
|
|
# Two remote machines, orchestrated from a third
|
|
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img
|
|
|
|
# Windows source with no SSH: it runs `clonetool listen --bind :9000 --password p`
|
|
clonetool sync --source tcp://winbox:9000/C:\data\disk.img \
|
|
--dest backup-host:/srv/win.img --password p
|
|
|
|
# Re-run any time; only changed blocks move
|
|
clonetool sync --source db1:/dev/vdb --dest backup-host:/srv/db1.img
|
|
```
|
|
|
|
Options for `sync`:
|
|
|
|
| Flag | Default | Meaning |
|
|
|---|---|---|
|
|
| `--block-size` | `4M` | Block size (accepts `K`/`M`/`G` suffixes). |
|
|
| `--job` | — | Optional label shown in progress/log output. |
|
|
| `--yes` | off | Don't prompt before shrinking an existing destination file. |
|
|
| `--sudo` | `auto` | Block-device privilege escalation: `auto` (on a permission error), `always`, or `never`. Remote elevation needs passwordless sudo. No-op on Windows (run as Administrator instead). |
|
|
| `--deploy` | `true` | Copy this binary to SSH hosts that lack a runnable `clonetool`. `--deploy=false` to disable. (Never applies to `tcp://` endpoints.) |
|
|
| `--connect-timeout` | `8` | SSH/TLS connect timeout (seconds) used for the push/pull direction probe. |
|
|
| `--ssh` | `ssh` | ssh binary to use. |
|
|
| `--ssh-opt` | — | Extra `-o OPT` passed to ssh (repeatable). |
|
|
| `--remote-bin` | `clonetool` | Path to clonetool on remote SSH hosts. |
|
|
| `--manager-host` | local hostname | Address a peer should use to reach this machine, needed only when source or dest is local to the manager *and* the other side is remote and ends up needing to dial back in (pull fallback). |
|
|
| `--password` | — | Shared password for a `tcp://` listen endpoint. |
|
|
| `--password-file` | — | Read the listen-endpoint password from a file (or set `CLONETOOL_PASSWORD`). |
|
|
|
|
Options for `listen`:
|
|
|
|
| Flag | Default | Meaning |
|
|
|---|---|---|
|
|
| `--bind` | `:9000` | Address to listen on, e.g. `:9000` or `0.0.0.0:9000`. |
|
|
| `--password` | — | Shared password (or `--password-file`, or `CLONETOOL_PASSWORD`). Required. |
|
|
| `--password-file` | — | Read the shared password from a file. |
|
|
|
|
`clonetool version` prints the binary's `GOOS/GOARCH` and build timestamp,
|
|
e.g. `clonetool linux/amd64 build=2024-06-01T12:00:00Z` (the timestamp is
|
|
used internally for the self-deploy staleness check above).
|
|
|
|
The status line shows read throughput for both sides and write throughput
|
|
for the destination separately: `rd(src)` is the source reading and
|
|
comparing its blocks, `rd(dst)` is the destination fingerprinting its
|
|
current content (only while the trailing `scan H/M` segment is present),
|
|
and `wr(dst)` is the destination actually writing changed blocks.
|
|
|
|
## Caveats
|
|
|
|
- Block-device sizing uses `BLKGETSIZE64` on Linux and
|
|
`IOCTL_DISK_GET_LENGTH_INFO` on Windows (`\\.\PhysicalDrive0`, `\\.\C:`).
|
|
On other platforms (e.g. macOS) only regular files are supported; a
|
|
device path there errors out.
|
|
- **Windows raw disks:** syncing a `\\.\PhysicalDrive*` requires running
|
|
clonetool as **Administrator**, and the disk should be offline/unmounted
|
|
(a live, mounted volume can refuse writes or give inconsistent reads).
|
|
Raw-disk I/O is sector-aligned automatically; `--block-size` must be a
|
|
multiple of the sector size. `--sudo` does nothing on Windows.
|
|
- If a destination path doesn't exist yet, it's created as a regular
|
|
file — clonetool won't create device nodes, so double-check device
|
|
paths for typos before running.
|
|
- SSH host keys are accepted on first connect (`StrictHostKeyChecking=accept-new`)
|
|
and rejected if they later change, same as normal SSH behavior.
|
|
- Because every run re-hashes both sides in full, a re-sync costs a full
|
|
read of source and destination even when little changed — the win is in
|
|
the bytes transferred, not the bytes read. The two reads run in
|
|
parallel; the status line shows a trailing `scan H/M` until the
|
|
destination fingerprint catches up.
|
|
- `agent` is an internal subcommand spawned automatically by `sync`; it's
|
|
not meant to be run by hand, though it will work standalone for
|
|
debugging.
|