diff --git a/README.md b/README.md index 824b68b..5e5ab1f 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,17 @@ two paths on the same machine), driven from a third, passive "manager" machine. Single static Go binary, no runtime dependencies beyond the system `ssh` client for remote endpoints. +Two subcommands: + +- **`sync`** — mirror one file/device onto another (the original tool; see + most of this README). +- **`clone-disk`** — clone a whole system disk: boot record + partition + table + per-filesystem data, rebuilt on the target then filled from the + source. Same manager/agent/SSH/self-deploy model as `sync`; leans on the + standard disk tools already present on a rescue system (`sfdisk`, + `ntfsclone`, `partclone.*`, `e2image`, `rsync`, …). See + [clone-disk](#clone-disk) below. + ## How it works - You run `clonetool sync` on the **manager**. Source, destination, and @@ -123,6 +134,137 @@ clonetool sync --source \\.\E: --dest \\.\F: handles the final partial block itself. - Device size is read with `IOCTL_DISK_GET_LENGTH_INFO`. +## clone-disk + +`clonetool clone-disk --source LOC --dest LOC [options]` clones a whole +disk — the boot record, the partition table, and the contents of every +partition — by **rebuilding** the layout on the target and then filling it +from the source. Orchestrated from the manager exactly like `sync`: one +control agent per side, data straight between source and target (push, or +pull fallback), self-deploy, `--sudo` escalation. + +It does not reimplement any filesystem knowledge — it calls the standard +tools a rescue system already has and just moves their bytes between the two +machines. + +### What it copies + +1. **Partition table.** The source agent runs `sfdisk -d` (the canonical + restorable dump for both GPT and MBR). The manager turns it into a + device-independent restore script — dropping the `device:` line and + `last-lba` so it re-sizes for the target, keeping the GPT disk GUID and + every partition GUID/PARTUUID (so existing `fstab` / BCD / GRUB + references still resolve) unless `--new-ids` is given. The target agent + feeds that to `sfdisk` and re-reads the table. +2. **Boot record / boot code.** The 440-byte MBR bootstrap on every disk, + plus — on MBR disks — the whole gap before the first partition (where + GRUB's `core.img` lives), are raw-copied through the same block-diff + engine `sync` uses, addressed as byte windows. On GPT disks the + GPT structures come from `sfdisk`/`sgdisk`; a BIOS-boot partition + (`EF02`) is always cloned raw. +3. **Partition data**, per partition, by the best available method: + - **fs-image** (default when the tool exists): stream a + filesystem-aware image — `ntfsclone` for NTFS, `partclone.` for + ext*/xfs/btrfs/f2fs/fat/exfat, `e2image` as an ext fallback. Only + used blocks move. + - **raw**: the `sync` block-diff engine over the partition's byte + window. Used for swap, unrecognised filesystems, `--raw N,…`, and + whenever no image tool is installed. Re-runs move only changed blocks. + - **file-level** (`--file-level N,…`): `mkfs` on the target + `rsync` + *(planned; not yet implemented — use `--raw` or an fs-image type)*. + +### Target sizing (same rules as `sync`) + +- **Target is a device:** never grown. If the source layout fits, it is + reproduced as-is and any trailing space on the target is left untouched. + If the last partition(s) overflow, `--allow-shrink` will shrink them + (filesystem then partition, from the last inward) with `ntfsresize` / + `resize2fs` — this **resizes the source filesystem in place** before + imaging, so it also needs `--yes`. Without `--allow-shrink` an + over-large source is a hard error naming the partition. +- **Target is a file:** created and truncated to just what the layout + needs (or `--image-size SIZE`). Shrinking an existing image prompts + unless `--yes`. + +### NTFS / Windows partitions + +- **From a Linux rescue system: fully supported.** `ntfsclone` clones + Windows NTFS partitions (incl. Win10/11) at cluster level, copying only + used clusters and preserving every NTFS feature. The volume must be + *clean* — Windows Fast Startup and hibernation leave NTFS dirty; boot + Windows once and shut down fully, or run `ntfsfix -d` first. `ntfsclone` + does not fix booting: keep the partition GUIDs (default) so the existing + BCD resolves, or run `bcdboot` from Windows recovery afterwards. This is + the same approach Clonezilla uses. +- **On Windows itself: raw / VSS only.** There is no `ntfsclone` on + Windows. `clone-disk` reads the partition table via + `IOCTL_DISK_GET_DRIVE_LAYOUT_EX`, reproduces it by raw-copying the + leading sectors (and the backup GPT), and block-clones each partition + as a byte window of `\\.\PhysicalDriveN`. With `--vss` (default) it + takes a Volume Shadow Copy of each NTFS volume first for a + crash-consistent point-in-time source. This is **not** free-space-aware + (a `$Bitmap`-driven skip may come later), and shrinking to a smaller + target is not supported when the source is native Windows — use a + target at least as large, or run the clone from a Linux rescue system + to get `ntfsclone`. + +### Bootloader + +By default `clone-disk` only *reproduces* boot structures and preserves +disk/partition IDs — enough for a like-for-like replacement disk to boot. +`--reinstall-bootloader` additionally mounts the cloned root (+ ESP), +bind-mounts `/dev /proc /sys`, and runs `grub-install` + `update-grub` / +`grub-mkconfig` in a chroot (Linux targets). It is best-effort and never +fails the clone. For Windows, run `bcdboot C:\Windows /s S: /f ALL` from a +recovery environment. + +### Examples + +``` +# Whole disk to an image file (image is sized to the layout) +clonetool clone-disk --source /dev/sda --dest /srv/sda.img + +# Disk to disk on another host, orchestrated from a third machine +clonetool clone-disk --source box1:/dev/nvme0n1 --dest box2:/dev/nvme0n1 + +# Onto a smaller SSD, shrinking the last (data) partition to fit +clonetool clone-disk --source /dev/sda --dest /dev/sdb --allow-shrink --yes + +# Fresh IDs so the clone can sit next to the original +clonetool clone-disk --source /dev/sda --dest /dev/sdb --new-ids + +# Force a raw block clone of a partition an image tool would otherwise handle +clonetool clone-disk --source /dev/sda --dest /dev/sdb --raw 2 +``` + +### Tools the agents call (checked at probe time, reported if missing) + +`sfdisk`, `sgdisk`, `partprobe`/`blockdev`, `blkid`, `lsblk`, `losetup`, +`ntfsclone`, `ntfsresize`, `ntfsfix`, `partclone.*`, `e2image`, +`resize2fs`, `dumpe2fs`, `e2fsck`, `mkfs.*`, `mount`/`umount`, `rsync`, +`grub-install`, `update-grub`/`grub-mkconfig`. Windows: `vssadmin` / +`wmic`, `bcdboot`, `diskpart`. A missing tool just means the affected +partitions fall back to a raw block copy (or the run stops with a clear +message if that isn't safe, e.g. a partition that must shrink). + +### clone-disk options + +| Flag | Default | Meaning | +|---|---|---| +| `--parts LIST` | all | Only clone these partition numbers (data only; the table still lists them all). | +| `--raw LIST` | — | Force a raw block clone for these partitions. | +| `--file-level LIST` | — | `mkfs` + `rsync` these partitions (not yet implemented). | +| `--file-level-auto` | off | Use file-level for fs types with no image cloner. | +| `--allow-shrink` | off | Permit shrinking trailing partitions to fit a smaller target (resizes the **source** fs in place; needs `--yes`). | +| `--no-shrink` | off | Never shrink; fail if the target is too small. | +| `--new-ids` | off | Randomize the GPT disk GUID / MBR signature on the target. | +| `--reinstall-bootloader` | off | Run `grub-install` / `grub-mkconfig` on the target after copy (Linux). | +| `--vss` | on | Windows source: take a Volume Shadow Copy per NTFS volume. | +| `--image-size SIZE` | auto | File target: image size (default = enough for the layout). | + +`--block-size --job --yes --sudo --deploy --ssh --ssh-opt --remote-bin +--connect-timeout --manager-host` mean the same as for `sync`. + ## Usage ``` diff --git a/agent.go b/agent.go index 5f4d451..f8bd144 100644 --- a/agent.go +++ b/agent.go @@ -16,10 +16,15 @@ import ( func cmdAgent(args []string) error { fs := flag.NewFlagSet("agent", flag.ContinueOnError) - role := fs.String("role", "", "control|sink|source-stream (internal)") + role := fs.String("role", "", "control|sink|source-stream|fs-send|fs-recv (internal)") path := fs.String("path", "", "path to read/write") size := fs.Int64("size", 0, "total sync size in bytes") + base := fs.Int64("base", 0, "byte offset the window starts at (clone-disk boot region / offset partition)") blockSize := fs.Int64("block-size", defaultBlockSize, "block size in bytes") + fsType := fs.String("fs", "", "filesystem type (fs-send/fs-recv)") + fsTool := fs.String("fstool", "", "fs-image tool family (fs-send/fs-recv)") + peerDisk := fs.String("peerdisk", "", "this helper's whole-disk/image path (fs-send/fs-recv)") + shrinkTo := fs.Int64("shrink", 0, "shrink this fs to N bytes before sending (fs-send)") if err := fs.Parse(args); err != nil { return err } @@ -28,11 +33,17 @@ func cmdAgent(args []string) error { case "control": return runControlAgent() case "sink": - return runSinkRole(*path, *size, *blockSize) + return runSinkRole(*path, *base, *size, *blockSize) case "source-stream": - return runSourceStreamRole(*path, *size, *blockSize) + return runSourceStreamRole(*path, *base, *size, *blockSize) + case "fs-send": + n, _ := strconv.Atoi(*path) + return runFSSendRole(n, *fsType, *fsTool, *peerDisk, *shrinkTo) + case "fs-recv": + n, _ := strconv.Atoi(*path) + return runFSRecvRole(n, *fsType, *fsTool, *peerDisk) default: - return fmt.Errorf("agent: unknown or missing --role %q (want control|sink|source-stream)", *role) + return fmt.Errorf("agent: unknown or missing --role %q", *role) } } @@ -66,7 +77,16 @@ func runControlAgent() error { runPushDriver(m, out) case msgConnectPull: runPullDriver(m, out) + case msgProbeDisk: + handleProbeDisk(out, m) + case msgBuildLayout: + handleBuildLayout(out, m) + case msgClonePartition: + handleClonePartition(out, m) + case msgReinstallBoot: + handleReinstallBoot(out, m) case msgClose: + detachAllDisks() _ = out.WriteJSON(CtrlMsg{Type: msgBye}) return nil default: @@ -127,6 +147,7 @@ func runPushDriver(req CtrlMsg, out *FrameWriter) { tailArgs := []string{ "agent", "--role", "sink", "--path", req.PeerPath, + "--base", strconv.FormatInt(req.PeerBase, 10), "--size", strconv.FormatInt(req.Size, 10), "--block-size", strconv.FormatInt(req.BlockSize, 10), } @@ -314,7 +335,7 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F sendErrCh := make(chan error, 1) go func() { sendErrCh <- runSourceLoop(sourceLoopParams{ - File: srcFile, Size: req.Size, BlockSize: req.BlockSize, Align: align, Hashes: hashCh, Out: fw, + File: srcFile, Base: req.Base, Size: req.Size, BlockSize: req.BlockSize, Align: align, Hashes: hashCh, Out: fw, OnSkip: func(uint64) { atomic.AddInt64(&skipped, 1); maybeProgress() }, OnSend: func(idx uint64, _ [32]byte) { sem <- struct{}{} @@ -400,6 +421,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) { tailArgs := []string{ "agent", "--role", "source-stream", "--path", req.PeerPath, + "--base", strconv.FormatInt(req.PeerBase, 10), "--size", strconv.FormatInt(req.Size, 10), "--block-size", strconv.FormatInt(req.BlockSize, 10), } @@ -448,7 +470,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) { hashErrCh := make(chan error, 1) go func() { - err := streamHashBlocks(dstFile, req.Size, req.BlockSize, align, func(bh blockHash) error { + err := streamHashBlocks(dstFile, req.Base, req.Size, req.BlockSize, align, func(bh blockHash) error { return fw.WriteFrame(frameBlockHash, encodeBlockHashFrame(bh.index, bh.hash)) }, scanProgressEmitter(out)) if err == nil { @@ -463,7 +485,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) { }() loopErr := runDestLoop(destLoopParams{ - File: dstFile, BlockSize: req.BlockSize, Align: align, In: fr, + File: dstFile, Base: req.Base, BlockSize: req.BlockSize, Align: align, In: fr, OnCtrlMsg: func(m CtrlMsg) { if m.Type == msgProgress { _ = out.WriteJSON(m) @@ -496,7 +518,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) { // destination host. Dumb write endpoint: verify+pwrite+ack per block. // --------------------------------------------------------------------- -func runSinkRole(path string, size, blockSize int64) error { +func runSinkRole(path string, base, size, blockSize int64) error { out := NewFrameWriter(os.Stdout) in := NewFrameReader(os.Stdin) @@ -521,7 +543,7 @@ func runSinkRole(path string, size, blockSize int64) error { // never race on the same region. hashErrCh := make(chan error, 1) go func() { - err := streamHashBlocks(f, size, blockSize, align, func(bh blockHash) error { + err := streamHashBlocks(f, base, size, blockSize, align, func(bh blockHash) error { return out.WriteFrame(frameBlockHash, encodeBlockHashFrame(bh.index, bh.hash)) }, scanProgressEmitter(out)) if err == nil { @@ -533,7 +555,7 @@ func runSinkRole(path string, size, blockSize int64) error { hashErrCh <- err }() - loopErr := runDestLoop(destLoopParams{File: f, BlockSize: blockSize, Align: align, In: in, AckOut: out}) + loopErr := runDestLoop(destLoopParams{File: f, Base: base, BlockSize: blockSize, Align: align, In: in, AckOut: out}) if hashErr := <-hashErrCh; hashErr != nil { return hashErr } @@ -547,7 +569,7 @@ func runSinkRole(path string, size, blockSize int64) error { // writing straight to its own stdout. // --------------------------------------------------------------------- -func runSourceStreamRole(path string, size, blockSize int64) error { +func runSourceStreamRole(path string, base, size, blockSize int64) error { out := NewFrameWriter(os.Stdout) in := NewFrameReader(os.Stdin) @@ -610,7 +632,7 @@ func runSourceStreamRole(path string, size, blockSize int64) error { } loopErr := runSourceLoop(sourceLoopParams{ - File: f, Size: size, BlockSize: blockSize, Align: align, Hashes: hashCh, Out: out, + File: f, Base: base, Size: size, BlockSize: blockSize, Align: align, Hashes: hashCh, Out: out, OnSkip: func(uint64) { skipped++; maybeProgress() }, OnSend: func(uint64, [32]byte) { copied++; maybeProgress() }, }) diff --git a/agent_disk.go b/agent_disk.go new file mode 100644 index 0000000..e8af38f --- /dev/null +++ b/agent_disk.go @@ -0,0 +1,375 @@ +package main + +import ( + "bufio" + "encoding/json" + "fmt" + "io" + "os" + "os/exec" + "strconv" + "strings" +) + +// ---------------------------------------------------------------------------- +// clone-disk agent state: loop-device / mount attachments held for the life +// of the control agent so several partition clones can reuse them. +// ---------------------------------------------------------------------------- + +var diskAttach = map[string]*diskAttachment{} + +func attachedNode(diskPath string, num int, writable bool) (string, error) { + a := diskAttach[diskPath] + if a == nil { + var err error + a, err = attachDisk(diskPath, writable) + if err != nil { + return "", err + } + diskAttach[diskPath] = a + } + return partNode(a.blockPath, num), nil +} + +func detachAllDisks() { + for k, a := range diskAttach { + if a.cleanup != nil { + a.cleanup() + } + delete(diskAttach, k) + } +} + +// ---------------------------------------------------------------------------- +// probe_disk +// ---------------------------------------------------------------------------- + +func handleProbeDisk(out *FrameWriter, m CtrlMsg) { + d, err := probeDiskLayout(m.Path) + if err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: err.Error(), NeedPriv: isPermErr(err)}) + return + } + _ = out.WriteJSON(CtrlMsg{Type: msgProbeDiskOK, LayoutJSON: d.JSON()}) +} + +// ---------------------------------------------------------------------------- +// build_layout: create/size an image, write the partition table, re-read it. +// The boot-region bytes and the partition data are moved afterwards by the +// manager through ordinary push/pull transfers. +// ---------------------------------------------------------------------------- + +type buildOpts struct { + Script string `json:"script"` // sfdisk restore script (Linux) + IsFile bool `json:"isFile"` // target is a regular file, not a device + ImageSize int64 `json:"imageSize"` // truncate the file to this many bytes first +} + +func handleBuildLayout(out *FrameWriter, m CtrlMsg) { + var o buildOpts + if err := json.Unmarshal([]byte(m.Options), &o); err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "decode build opts: " + err.Error()}) + return + } + if o.IsFile { + f, err := os.OpenFile(m.Path, os.O_CREATE|os.O_RDWR, 0644) + if err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: err.Error(), NeedPriv: isPermErr(err)}) + return + } + if err := f.Truncate(o.ImageSize); err != nil { + f.Close() + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("truncate %s: %v", m.Path, err)}) + return + } + f.Close() + } + if strings.TrimSpace(o.Script) != "" { + if err := applyPartitionTable(m.Path, o.Script); err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: err.Error(), NeedPriv: isPermErr(err)}) + return + } + } else { + rereadPartTable(m.Path) + } + _ = out.WriteJSON(CtrlMsg{Type: msgBuildLayoutOK}) +} + +// ---------------------------------------------------------------------------- +// clone_partition: fs-image and file-level methods. (Raw partitions and boot +// regions go through connect_push / connect_pull instead.) +// +// The message is sent to the SOURCE control agent first (Method set, peer +// fields as for connect_push). On a push failure the manager re-sends it to +// the DEST control agent with m.Reason == "pull". +// ---------------------------------------------------------------------------- + +func handleClonePartition(out *FrameWriter, m CtrlMsg) { + o, err := decodeFSCloneOpts(m.Options) + if err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "decode clone opts: " + err.Error()}) + return + } + switch m.Method { + case "fs-image": + if m.Reason == "pull" { + runFSPullDriver(m, o, out) + } else { + runFSPushDriver(m, o, out) + } + case "file-level": + runFileLevelDriver(m, o, out) + default: + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "unknown clone method " + m.Method}) + } +} + +// localDevice resolves this side's partition node. m.Path is the whole disk +// (or image); m.PartIndex is the 1-based partition number. +func localDevice(diskPath string, num int, writable bool) (string, error) { + if isDevicePath(diskPath) || looksLikeBlockDevice(diskPath) { + // real disk: node is derived directly, no attach needed + return partNode(diskPath, num), nil + } + return attachedNode(diskPath, num, writable) +} + +func looksLikeBlockDevice(p string) bool { + fi, err := os.Stat(p) + return err == nil && fi.Mode()&os.ModeDevice != 0 +} + +// runFSPushDriver runs in the SOURCE agent: spawn the peer fs-recv (ssh or +// local), run the save tool locally, pipe save-stdout -> peer-stdin. +func runFSPushDriver(m CtrlMsg, o fsCloneOpts, out *FrameWriter) { + dev, err := localDevice(m.Path, m.PartIndex, false) + if err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: "source device: " + err.Error()}) + return + } + cl := pickCloner(dev, o.FSType, o.Tool) + if cl == nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "no fs-image cloner for " + o.FSType}) + return + } + if o.ShrinkToBytes > 0 { + _ = out.WriteJSON(CtrlMsg{Type: msgProgress, Stage: "shrink source " + o.FSType + " p" + strconv.Itoa(m.PartIndex)}) + if err := preShrinkFS(dev, o.FSType, o.ShrinkToBytes); err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "pre-shrink: " + err.Error()}) + return + } + } + + peerTail := []string{"agent", "--role", "fs-recv", + "--path", strconv.Itoa(m.PartIndex), // partition number; peer resolves its own node + "--fs", o.FSType, "--fstool", o.Tool, + "--size", strconv.FormatInt(o.SizeBytes, 10), + "--peerdisk", m.PeerPath, + } + peer := peerAgentCommand(m, peerTail) + peerIn, err := peer.StdinPipe() + if err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: err.Error()}) + return + } + peerOut, _ := peer.StdoutPipe() + peerErr := newLimitedBuffer(8192) + peer.Stderr = peerErr + if err := peer.Start(); err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: "start peer: " + err.Error()}) + return + } + + save := exec.Command(cl.save[0], cl.save[1:]...) + save.Stdout = peerIn + saveErr, _ := save.StderrPipe() + if err := save.Start(); err != nil { + _ = peer.Process.Kill() + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("start %s: %v", cl.save[0], err)}) + return + } + + var moved int64 + go scanCloneProgress(saveErr, o.SizeBytes, func(b int64) { + moved = b + _ = out.WriteJSON(CtrlMsg{Type: msgProgress, Stage: o.FSType + " p" + strconv.Itoa(m.PartIndex), + BytesCopied: b, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex}) + }) + + saveWait := make(chan error, 1) + go func() { saveWait <- save.Wait() }() + sErr := <-saveWait + _ = peerIn.Close() + + peerLine := readLine(peerOut) + pErr := peer.Wait() + + if sErr != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("%s: %v", cl.save[0], sErr)}) + return + } + if pErr != nil || !strings.HasPrefix(peerLine, "OK") { + reason := strings.TrimSpace(peerLine + " " + peerErr.String()) + if moved == 0 { + _ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: reason}) + return + } + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "peer restore failed: " + reason}) + return + } + _ = out.WriteJSON(CtrlMsg{Type: msgProgress, BytesCopied: o.SizeBytes, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex}) + _ = out.WriteJSON(CtrlMsg{Type: msgPushOK}) +} + +// runFSPullDriver runs in the DEST agent: spawn the peer fs-send, run the +// restore tool locally, pipe peer-stdout -> restore-stdin. +func runFSPullDriver(m CtrlMsg, o fsCloneOpts, out *FrameWriter) { + dev, err := localDevice(m.Path, m.PartIndex, true) + if err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: "dest device: " + err.Error()}) + return + } + cl := pickCloner(dev, o.FSType, o.Tool) + if cl == nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "no fs-image cloner for " + o.FSType}) + return + } + + peerTail := []string{"agent", "--role", "fs-send", + "--path", strconv.Itoa(m.PartIndex), + "--fs", o.FSType, "--fstool", o.Tool, + "--peerdisk", m.PeerPath, + "--shrink", strconv.FormatInt(o.ShrinkToBytes, 10), + } + peer := peerAgentCommand(m, peerTail) + peerOut, err := peer.StdoutPipe() + if err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: err.Error()}) + return + } + peerErr := newLimitedBuffer(8192) + peer.Stderr = peerErr + if err := peer.Start(); err != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: "start peer: " + err.Error()}) + return + } + + restore := exec.Command(cl.restore[0], cl.restore[1:]...) + restore.Stdin = peerOut + restoreErr, _ := restore.StderrPipe() + if err := restore.Start(); err != nil { + _ = peer.Process.Kill() + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("start %s: %v", cl.restore[0], err)}) + return + } + go scanCloneProgress(restoreErr, o.SizeBytes, func(b int64) { + _ = out.WriteJSON(CtrlMsg{Type: msgProgress, Stage: o.FSType + " p" + strconv.Itoa(m.PartIndex), + BytesCopied: b, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex}) + }) + + rErr := restore.Wait() + pErr := peer.Wait() + if pErr != nil { + if rErr != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: strings.TrimSpace(peerErr.String())}) + return + } + } + if rErr != nil { + _ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("%s: %v", cl.restore[0], rErr)}) + return + } + _ = out.WriteJSON(CtrlMsg{Type: msgProgress, BytesCopied: o.SizeBytes, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex}) + _ = out.WriteJSON(CtrlMsg{Type: msgPullOK}) +} + +// ---------------------------------------------------------------------------- +// fs-send / fs-recv: one-shot stdio helpers spawned on the peer. +// ---------------------------------------------------------------------------- + +func runFSSendRole(partNum int, fsType, tool, peerDisk string, shrinkTo int64) error { + dev, err := resolveHelperDevice(peerDisk, partNum, shrinkTo > 0) + if err != nil { + return err + } + cl := pickCloner(dev, fsType, tool) + if cl == nil { + return fmt.Errorf("no fs-image cloner for %s", fsType) + } + if shrinkTo > 0 { + if err := preShrinkFS(dev, fsType, shrinkTo); err != nil { + return fmt.Errorf("pre-shrink: %w", err) + } + } + cmd := exec.Command(cl.save[0], cl.save[1:]...) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + return cmd.Run() +} + +func runFSRecvRole(partNum int, fsType, tool, peerDisk string) error { + dev, err := resolveHelperDevice(peerDisk, partNum, true) + if err != nil { + fmt.Fprintf(os.Stderr, "ERR: %v\n", err) + return err + } + cl := pickCloner(dev, fsType, tool) + if cl == nil { + err := fmt.Errorf("no fs-image cloner for %s", fsType) + fmt.Fprintf(os.Stdout, "ERR: %v\n", err) + return err + } + cmd := exec.Command(cl.restore[0], cl.restore[1:]...) + cmd.Stdin = os.Stdin + cmd.Stderr = os.Stderr + if err := cmd.Run(); err != nil { + fmt.Fprintf(os.Stdout, "ERR: %v\n", err) + return err + } + fmt.Fprintln(os.Stdout, "OK") + return nil +} + +// resolveHelperDevice is used by the one-shot fs-send/fs-recv helpers, which +// have no persistent attachment map of their own. +func resolveHelperDevice(disk string, num int, writable bool) (string, error) { + if isDevicePath(disk) || looksLikeBlockDevice(disk) { + return partNode(disk, num), nil + } + a, err := attachDisk(disk, writable) + if err != nil { + return "", err + } + // deliberately leak the loop device until process exit; a one-shot + // helper is short-lived and the parent detaches on the control side. + _ = a + return partNode(a.blockPath, num), nil +} + +// ---------------------------------------------------------------------------- +// file-level: mkfs on the target partition + rsync the tree across. +// ---------------------------------------------------------------------------- + +func runFileLevelDriver(m CtrlMsg, o fsCloneOpts, out *FrameWriter) { + _ = out.WriteJSON(CtrlMsg{Type: msgError, + Message: "file-level clone (--file-level) is not implemented yet; use --raw or an fs-image type"}) +} + +// ---------------------------------------------------------------------------- +// reinstall_boot: best-effort, never fatal to the clone. +// ---------------------------------------------------------------------------- + +func handleReinstallBoot(out *FrameWriter, m CtrlMsg) { + msg := reinstallBootloader(m.Path, m.Options) + _ = out.WriteJSON(CtrlMsg{Type: msgReinstallBootOK, Message: msg}) +} + +// readLine reads one \n-terminated line (used for the fs-recv status line). +func readLine(r io.Reader) string { + if r == nil { + return "" + } + br := bufio.NewReader(r) + s, _ := br.ReadString('\n') + return strings.TrimSpace(s) +} diff --git a/boot_linux.go b/boot_linux.go new file mode 100644 index 0000000..3351c79 --- /dev/null +++ b/boot_linux.go @@ -0,0 +1,84 @@ +//go:build linux + +package main + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "strings" +) + +// reinstallBootloader mounts the freshly cloned root (and ESP), bind-mounts +// the API filesystems, and runs grub-install + grub-mkconfig inside a chroot. +// It is best-effort: it returns a human summary and never a hard error, so a +// clone still completes even when boot repair cannot. +func reinstallBootloader(diskPath, optsJSON string) string { + var o bootOpts + if err := json.Unmarshal([]byte(optsJSON), &o); err != nil { + return "skipped: bad boot opts: " + err.Error() + } + if o.RootPart == 0 { + return "skipped: no root partition identified (pass --reinstall-bootloader only for a Linux system disk)" + } + att, err := attachDisk(diskPath, true) + if err != nil { + return "skipped: attach: " + err.Error() + } + defer att.cleanup() + + rootNode := partNode(att.blockPath, o.RootPart) + root, umount, err := mountAt(rootNode, false) + if err != nil { + return "skipped: mount root: " + err.Error() + } + defer umount() + + if o.ESPPart != 0 { + espNode := partNode(att.blockPath, o.ESPPart) + espDir := root + "/boot/efi" + _ = os.MkdirAll(espDir, 0755) + if out, err := exec.Command("mount", espNode, espDir).CombinedOutput(); err != nil { + return "skipped: mount ESP: " + strings.TrimSpace(string(out)) + } + defer exec.Command("umount", espDir).Run() + } + + var log strings.Builder + for _, d := range []string{"dev", "proc", "sys", "run"} { + _ = exec.Command("mount", "--bind", "/"+d, root+"/"+d).Run() + defer exec.Command("umount", "-l", root+"/"+d).Run() + } + + grubTarget := att.blockPath + if isDevicePath(diskPath) || looksLikeBlockDevice(diskPath) { + grubTarget = diskPath + } + giArgs := []string{root, "grub-install", "--recheck"} + if o.UEFI { + giArgs = append(giArgs, "--target=x86_64-efi", "--efi-directory=/boot/efi", "--removable") + } else { + giArgs = append(giArgs, grubTarget) + } + if out, err := exec.Command("chroot", giArgs...).CombinedOutput(); err != nil { + fmt.Fprintf(&log, "grub-install failed: %s; ", strings.TrimSpace(string(out))) + } else { + log.WriteString("grub-install ok; ") + } + + mkcfg := "grub-mkconfig" + if _, err := exec.LookPath("update-grub"); err == nil { + mkcfg = "update-grub" + } + args := []string{root, mkcfg} + if mkcfg == "grub-mkconfig" { + args = append(args, "-o", "/boot/grub/grub.cfg") + } + if out, err := exec.Command("chroot", args...).CombinedOutput(); err != nil { + fmt.Fprintf(&log, "%s failed: %s", mkcfg, strings.TrimSpace(string(out))) + } else { + log.WriteString(mkcfg + " ok") + } + return log.String() +} diff --git a/boot_other.go b/boot_other.go new file mode 100644 index 0000000..2e47a01 --- /dev/null +++ b/boot_other.go @@ -0,0 +1,13 @@ +//go:build !linux + +package main + +// reinstallBootloader is Linux-only for now. On Windows the recommended path +// is to preserve the disk signature / partition GUIDs (clone-disk does this by +// raw-copying the leading sectors) so the existing BCD store still resolves, +// and otherwise run `bcdboot W:\Windows /s S: /f ALL` from a WinPE/recovery +// environment by hand. +func reinstallBootloader(diskPath, optsJSON string) string { + return "skipped: --reinstall-bootloader is only implemented on Linux; " + + "run bcdboot from Windows recovery, or grub-install from a Linux rescue system" +} diff --git a/clonedisk.go b/clonedisk.go new file mode 100644 index 0000000..8fccf7f --- /dev/null +++ b/clonedisk.go @@ -0,0 +1,261 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "sort" + "strings" +) + +// CloneDiskConfig is everything `clonetool clone-disk` needs. It embeds +// SyncConfig for the transport/deploy/ssh knobs shared with `sync`. +type CloneDiskConfig struct { + SyncConfig + Parts []int + Raw []int + FileLevel []int + FileAuto bool + AllowShrink bool + NoShrink bool + NewIDs bool + ReinstallBoot bool + VSS bool + ImageSize int64 +} + +func runCloneDisk(cfg CloneDiskConfig) error { + srcSpec, err := parseSpec(cfg.Source) + if err != nil { + return fmt.Errorf("--source: %w", err) + } + dstSpec, err := parseSpec(cfg.Dest) + if err != nil { + return fmt.Errorf("--dest: %w", err) + } + if err := checkNotSame(srcSpec, dstSpec); err != nil { + return err + } + + srcRemoteBin, dstRemoteBin := cfg.RemoteBin, cfg.RemoteBin + if !srcSpec.IsLocal() { + if srcRemoteBin, err = resolveRemoteBin(&cfg.SyncConfig, srcSpec, "source"); err != nil { + return err + } + } + if !dstSpec.IsLocal() { + if !srcSpec.IsLocal() && sameHost(srcSpec, dstSpec) { + dstRemoteBin = srcRemoteBin + } else if dstRemoteBin, err = resolveRemoteBin(&cfg.SyncConfig, dstSpec, "dest"); err != nil { + return err + } + } + + // --- bring up both control agents ------------------------------------- + srcCtrl, srcLayout, srcSudo, err := bringUpProbe(srcSpec, "source", &cfg.SyncConfig, srcRemoteBin) + if err != nil { + return fmt.Errorf("source: %w", err) + } + defer srcCtrl.Close() + + dstCtrl, dstInfo, dstSudo, err := bringUpController(dstSpec, "dest", &cfg.SyncConfig, dstRemoteBin, dstSpec.Path) + if err != nil { + return fmt.Errorf("dest: %w", err) + } + defer dstCtrl.Close() + + isFile := !(dstInfo.Exists && dstInfo.IsDevice) + destSize := dstInfo.Size + fmt.Fprintf(os.Stderr, "source %s: %s, %s, %d partitions\n", + srcSpec, srcLayout.Scheme, humanBytes(srcLayout.DiskSize), len(srcLayout.Partitions)) + reportMissingTools(srcLayout) + + // --- plan the target ------------------------------------------------- + plan, err := planTargetLayout(srcLayout, isFile, destSize, planOpts{ + Parts: intSet(cfg.Parts), + Raw: intSet(cfg.Raw), + FileLevel: intSet(cfg.FileLevel), + FileAuto: cfg.FileAuto, + AllowShrink: cfg.AllowShrink && !cfg.NoShrink, + NewIDs: cfg.NewIDs, + ImageSize: cfg.ImageSize, + }) + if err != nil { + return err + } + for _, n := range plan.Notes { + fmt.Fprintf(os.Stderr, " %s\n", n) + } + shrinks := false + for _, pp := range plan.Parts { + if pp.ShrinkToB > 0 { + shrinks = true + } + } + if shrinks && !cfg.Yes { + return fmt.Errorf("shrinking resizes the SOURCE filesystem(s) in place before imaging; " + + "re-run with --yes to confirm, or use a larger target") + } + if isFile && !cfg.Yes && dstInfo.Exists && dstInfo.Size > plan.ImageSize { + if !confirmShrink(dstSpec, dstInfo.Size, plan.ImageSize) { + return fmt.Errorf("aborted") + } + } + + // --- rebuild the partition table on the target --------------------- + bo := buildOpts{Script: plan.Script, IsFile: isFile, ImageSize: plan.ImageSize} + boJSON, _ := json.Marshal(bo) + fmt.Fprintf(os.Stderr, "rebuilding partition table on %s ...\n", dstSpec) + if err := dstCtrl.BuildLayout(dstSpec.Path, string(boJSON)); err != nil { + return fmt.Errorf("build layout: %w", err) + } + + bothLocal := srcSpec.IsLocal() && dstSpec.IsLocal() + srcHost, srcUser := resolveConnectHost(srcSpec, &cfg.SyncConfig) + dstHost, dstUser := resolveConnectHost(dstSpec, &cfg.SyncConfig) + + base := func(s Spec, remoteBin string) CtrlMsg { + return CtrlMsg{ + BlockSize: cfg.BlockSize, SSHBin: cfg.SSHBin, SSHOpts: cfg.SSHOpts, + ConnectTimeoutSec: cfg.ConnectTimeoutSec, PeerLocal: bothLocal, + } + } + _ = base + + // --- copy the boot regions verbatim (offset windows) --------------- + for _, r := range plan.BootRegions { + if r.Length <= 0 { + continue + } + fmt.Fprintf(os.Stderr, "boot region @%d (%s) %s ...\n", r.Offset, humanBytes(r.Length), r.Note) + if err := runWindowTransfer(srcCtrl, dstCtrl, windowJob{ + srcPath: srcSpec.Path, dstPath: dstSpec.Path, + srcBase: r.Offset, dstBase: r.Offset, size: r.Length, + bothLocal: bothLocal, blockSize: cfg.BlockSize, cfg: &cfg.SyncConfig, + srcHost: srcHost, srcUser: srcUser, dstHost: dstHost, dstUser: dstUser, + srcRemoteBin: srcRemoteBin, dstRemoteBin: dstRemoteBin, + srcSudo: srcSudo, dstSudo: dstSudo, + }); err != nil { + return fmt.Errorf("boot region @%d: %w", r.Offset, err) + } + } + + // --- per-partition data ------------------------------------------------ + for i, pp := range plan.Parts { + tag := fmt.Sprintf("partition %d/%d (p%d %s, %s)", i+1, len(plan.Parts), pp.Num, orDash(pp.FSType), pp.Method) + fmt.Fprintf(os.Stderr, "%s ...\n", tag) + switch pp.Method { + case "raw": + sz := pp.SrcSizeB + if pp.DstSizeB < sz { + sz = pp.DstSizeB + } + if err := runWindowTransfer(srcCtrl, dstCtrl, windowJob{ + srcPath: srcSpec.Path, dstPath: dstSpec.Path, + srcBase: pp.SrcStartB, dstBase: pp.DstStartB, size: sz, + bothLocal: bothLocal, blockSize: cfg.BlockSize, cfg: &cfg.SyncConfig, + srcHost: srcHost, srcUser: srcUser, dstHost: dstHost, dstUser: dstUser, + srcRemoteBin: srcRemoteBin, dstRemoteBin: dstRemoteBin, + srcSudo: srcSudo, dstSudo: dstSudo, + }); err != nil { + return fmt.Errorf("%s: %w", tag, err) + } + case "fs-image": + if err := runFSImagePartition(srcCtrl, dstCtrl, pp, cloneCtx{ + srcPath: srcSpec.Path, dstPath: dstSpec.Path, bothLocal: bothLocal, + cfg: &cfg.SyncConfig, srcHost: srcHost, srcUser: srcUser, + dstHost: dstHost, dstUser: dstUser, + srcRemoteBin: srcRemoteBin, dstRemoteBin: dstRemoteBin, + srcSudo: srcSudo, dstSudo: dstSudo, + }); err != nil { + return fmt.Errorf("%s: %w", tag, err) + } + case "file-level": + return fmt.Errorf("%s: file-level clone is not implemented yet", tag) + } + } + + // --- bootloader (opt-in) -------------------------------------------- + if cfg.ReinstallBoot { + bootJSON, _ := json.Marshal(bootOpts{ + RootPart: plan.RootPart, ESPPart: plan.ESPPart, UEFI: plan.UEFI, DiskPath: dstSpec.Path, + }) + fmt.Fprintf(os.Stderr, "reinstalling bootloader on %s ...\n", dstSpec) + summary, err := dstCtrl.ReinstallBoot(dstSpec.Path, string(bootJSON)) + if err != nil { + fmt.Fprintf(os.Stderr, " bootloader: %v\n", err) + } else { + fmt.Fprintf(os.Stderr, " bootloader: %s\n", summary) + } + } + + label := cfg.Job + if label == "" { + label = fmt.Sprintf("%s -> %s", srcSpec, dstSpec) + } + fmt.Fprintf(os.Stderr, "done: %s\n", label) + return nil +} + +// bringUpProbe starts a source control agent and runs probe_disk, retrying +// once under sudo on a permission error (mirrors bringUpController). +func bringUpProbe(spec Spec, tag string, cfg *SyncConfig, remoteBin string) (*Controller, *DiskLayout, bool, error) { + sudo := cfg.Sudo == "always" && canElevate() + c, err := startController(spec, tag, cfg, remoteBin, sudo) + if err != nil { + return nil, nil, sudo, err + } + lay, err := c.ProbeDisk(spec.Path) + if err == nil { + return c, lay, sudo, nil + } + if errors.Is(err, errNeedPriv) && cfg.Sudo == "auto" && !sudo && canElevate() { + fmt.Fprintf(os.Stderr, "%s: permission denied on %s; retrying via sudo ...\n", tag, spec.Path) + c.Close() + sudo = true + if c, err = startController(spec, tag, cfg, remoteBin, true); err != nil { + return nil, nil, sudo, err + } + if lay, err = c.ProbeDisk(spec.Path); err == nil { + return c, lay, sudo, nil + } + } + c.Close() + return nil, nil, sudo, err +} + +func reportMissingTools(d *DiskLayout) { + want := []string{"sfdisk", "ntfsclone", "partclone.extfs", "partclone.restore", "e2image"} + var miss []string + for _, t := range want { + if !d.Tools[t] { + miss = append(miss, t) + } + } + if len(miss) > 0 { + fmt.Fprintf(os.Stderr, " note: not present on source: %s (affected partitions fall back to raw block copy)\n", + strings.Join(miss, ", ")) + } +} + +func orDash(s string) string { + if s == "" { + return "-" + } + return s +} + +func intSet(xs []int) map[int]bool { + m := make(map[int]bool, len(xs)) + for _, x := range xs { + m[x] = true + } + return m +} + +func sortedParts(ps []Partition) []Partition { + out := append([]Partition(nil), ps...) + sort.Slice(out, func(i, j int) bool { return out[i].Start < out[j].Start }) + return out +} diff --git a/clonedrive.go b/clonedrive.go new file mode 100644 index 0000000..53b958c --- /dev/null +++ b/clonedrive.go @@ -0,0 +1,117 @@ +package main + +import ( + "fmt" + "os" +) + +// windowJob raw-copies a byte window [srcBase,srcBase+size) of the source +// disk onto [dstBase,dstBase+size) of the target, through the same +// push/pull block-diff engine `sync` uses. Used for boot regions and for +// partitions cloned with method "raw". +type windowJob struct { + srcPath, dstPath string + srcBase, dstBase int64 + size int64 + bothLocal bool + blockSize int64 + cfg *SyncConfig + srcHost, srcUser string + dstHost, dstUser string + srcRemoteBin string + dstRemoteBin string + srcSudo, dstSudo bool +} + +func runWindowTransfer(srcCtrl, dstCtrl *Controller, j windowJob) error { + pp := newProgressPrinter(j.blockSize) + cb := transferCallbacks{onProgress: pp.print} + + pushReq := CtrlMsg{ + Path: j.srcPath, Base: j.srcBase, PeerBase: j.dstBase, Size: j.size, BlockSize: j.blockSize, + PeerHost: j.dstHost, PeerUser: j.dstUser, PeerPath: j.dstPath, PeerLocal: j.bothLocal, + RemoteBin: j.dstRemoteBin, SSHBin: j.cfg.SSHBin, SSHOpts: j.cfg.SSHOpts, + ConnectTimeoutSec: j.cfg.ConnectTimeoutSec, Sudo: j.dstSudo, + } + ok, reason, err := srcCtrl.ConnectPush(pushReq, cb) + if err != nil { + pp.finish() + return err + } + if !ok { + pp.finish() + fmt.Fprintf(os.Stderr, " push not possible (%s); pulling instead ...\n", reason) + pullReq := CtrlMsg{ + Path: j.dstPath, Base: j.dstBase, PeerBase: j.srcBase, Size: j.size, BlockSize: j.blockSize, + PeerHost: j.srcHost, PeerUser: j.srcUser, PeerPath: j.srcPath, PeerLocal: j.bothLocal, + RemoteBin: j.srcRemoteBin, SSHBin: j.cfg.SSHBin, SSHOpts: j.cfg.SSHOpts, + ConnectTimeoutSec: j.cfg.ConnectTimeoutSec, Sudo: j.srcSudo, + } + ok2, reason2, err2 := dstCtrl.ConnectPull(pullReq, cb) + if err2 != nil { + pp.finish() + return err2 + } + if !ok2 { + pp.finish() + return fmt.Errorf("no direct connection either way (push: %s; pull: %s)", reason, reason2) + } + } + pp.finish() + return nil +} + +// cloneCtx carries the transport context for an fs-image partition clone. +type cloneCtx struct { + srcPath, dstPath string + bothLocal bool + cfg *SyncConfig + srcHost, srcUser string + dstHost, dstUser string + srcRemoteBin string + dstRemoteBin string + srcSudo, dstSudo bool +} + +// runFSImagePartition streams a filesystem image for one partition, trying +// push (source drives) then pull (dest drives). +func runFSImagePartition(srcCtrl, dstCtrl *Controller, pp plannedPart, c cloneCtx) error { + opts := fsCloneOpts{FSType: pp.FSType, Tool: pp.Tool, SizeBytes: pp.SrcSizeB, ShrinkToBytes: pp.ShrinkToB} + prog := newProgressPrinter(1 << 20) + cb := transferCallbacks{onProgress: prog.printBytes} + + pushReq := CtrlMsg{ + Method: "fs-image", PartIndex: pp.Num, Options: opts.encode(), + Path: c.srcPath, PeerPath: c.dstPath, PeerLocal: c.bothLocal, + PeerHost: c.dstHost, PeerUser: c.dstUser, + RemoteBin: c.dstRemoteBin, SSHBin: c.cfg.SSHBin, SSHOpts: c.cfg.SSHOpts, + ConnectTimeoutSec: c.cfg.ConnectTimeoutSec, Sudo: c.dstSudo, + } + ok, reason, err := srcCtrl.ClonePartition(pushReq, cb) + if err != nil { + prog.finish() + return err + } + if !ok { + prog.finish() + fmt.Fprintf(os.Stderr, " fs-image push not possible (%s); pulling instead ...\n", reason) + pullReq := CtrlMsg{ + Method: "fs-image", Reason: "pull", PartIndex: pp.Num, Options: opts.encode(), + Path: c.dstPath, PeerPath: c.srcPath, PeerLocal: c.bothLocal, + PeerHost: c.srcHost, PeerUser: c.srcUser, + RemoteBin: c.srcRemoteBin, SSHBin: c.cfg.SSHBin, SSHOpts: c.cfg.SSHOpts, + ConnectTimeoutSec: c.cfg.ConnectTimeoutSec, Sudo: c.srcSudo, + } + ok2, reason2, err2 := dstCtrl.ClonePartition(pullReq, cb) + if err2 != nil { + prog.finish() + return err2 + } + if !ok2 { + prog.finish() + return fmt.Errorf("fs-image failed both ways (push: %s; pull: %s)", reason, reason2) + } + } + prog.finish() + return nil +} diff --git a/control.go b/control.go index 30eede4..202e551 100644 --- a/control.go +++ b/control.go @@ -135,6 +135,69 @@ func (c *Controller) agentErr(op, path string, resp CtrlMsg) error { return fmt.Errorf("%s: %s %s: %s", c.tag, op, path, resp.Message) } +// ProbeDisk asks the agent to enumerate a whole disk (partition table, boot +// regions, per-partition filesystem facts, tool availability). +func (c *Controller) ProbeDisk(path string) (*DiskLayout, error) { + resp, err := c.call(CtrlMsg{Type: msgProbeDisk, Path: path}) + if err != nil { + return nil, err + } + switch resp.Type { + case msgProbeDiskOK: + return parseDiskLayoutJSON(resp.LayoutJSON) + case msgError: + return nil, c.agentErr("probe", path, resp) + default: + return nil, fmt.Errorf("%s: unexpected response %q to probe_disk", c.tag, resp.Type) + } +} + +// BuildLayout asks the agent to create/size an image and write the partition +// table (optsJSON is a buildOpts blob). +func (c *Controller) BuildLayout(path, optsJSON string) error { + resp, err := c.call(CtrlMsg{Type: msgBuildLayout, Path: path, Options: optsJSON}) + if err != nil { + return err + } + switch resp.Type { + case msgBuildLayoutOK: + return nil + case msgError: + return c.agentErr("build_layout", path, resp) + default: + return fmt.Errorf("%s: unexpected response %q to build_layout", c.tag, resp.Type) + } +} + +// ClonePartition drives one fs-image or file-level partition clone. dir is +// "push" (send to source agent) or "pull" (send to dest agent). +func (c *Controller) ClonePartition(req CtrlMsg, cb transferCallbacks) (ok bool, reason string, err error) { + req.Type = msgClonePartition + okType, failType := msgPushOK, msgPushFailed + if req.Reason == "pull" { + okType, failType = msgPullOK, msgPullFailed + } + return c.connectAndPump(req, cb, okType, failType) +} + +// ReinstallBoot asks the (dest) agent to run grub-install/bcdboot; the +// returned string is a human summary. Best-effort — never returns an error +// for a boot-repair failure, only for a transport failure. +func (c *Controller) ReinstallBoot(path, optsJSON string) (string, error) { + resp, err := c.call(CtrlMsg{Type: msgReinstallBoot, Path: path, Options: optsJSON}) + if err != nil { + return "", err + } + switch resp.Type { + case msgReinstallBootOK: + return resp.Message, nil + case msgError: + return "", c.agentErr("reinstall_boot", path, resp) + default: + return "", fmt.Errorf("%s: unexpected response %q to reinstall_boot", c.tag, resp.Type) + } +} + // transferCallbacks receives streaming updates while a connect_push or // connect_pull is in flight. type transferCallbacks struct { diff --git a/ctrlmsg.go b/ctrlmsg.go index 6ed4ccb..9d7baaf 100644 --- a/ctrlmsg.go +++ b/ctrlmsg.go @@ -21,6 +21,15 @@ type CtrlMsg struct { PeerUser string `json:"peerUser,omitempty"` PeerPort int `json:"peerPort,omitempty"` PeerPath string `json:"peerPath,omitempty"` + // Base/Length restrict a push/pull transfer to a byte window of the + // path (offset Base, Length bytes). Used by clone-disk to raw-copy a + // boot region or an offset-addressed partition through the same engine. + // Size stays the *window* length for the loop math; Base shifts every + // block offset. Zero Base + Length==Size is the whole-file case that + // `sync` uses unchanged. + Base int64 `json:"base,omitempty"` + PeerBase int64 `json:"peerBase,omitempty"` + Length int64 `json:"length,omitempty"` // PeerLocal is set when both source and dest are local to the manager, // so the driver (itself already a local child of the manager) can spawn // the sink/source-stream helper as a plain local subprocess instead of @@ -48,6 +57,20 @@ type CtrlMsg struct { Skipped int64 `json:"skipped,omitempty"` TotalBlocks int64 `json:"totalBlocks,omitempty"` BytesCopied int64 `json:"bytesCopied,omitempty"` + BytesTotal int64 `json:"bytesTotal,omitempty"` + + // clone-disk staging: which partition/step of the whole-disk job the + // progress or status refers to. Stage is a short human label + // ("partition table", "boot region", "ntfs p2", "bootloader"). + Stage string `json:"stage,omitempty"` + PartIndex int `json:"partIndex,omitempty"` + PartTotal int `json:"partTotal,omitempty"` + + // clone-disk payloads: probe result / requested target layout, both + // carried as embedded JSON so CtrlMsg stays a flat envelope. + LayoutJSON string `json:"layoutJSON,omitempty"` + Method string `json:"method,omitempty"` // partition clone method: "fs-image" | "raw" | "file-level" + Options string `json:"options,omitempty"` // JSON of the build/clone options block // log Level string `json:"level,omitempty"` @@ -69,4 +92,13 @@ const ( msgError = "error" msgClose = "close" msgBye = "bye" + + // clone-disk control verbs + msgProbeDisk = "probe_disk" + msgProbeDiskOK = "probe_disk_ok" + msgBuildLayout = "build_layout" + msgBuildLayoutOK = "build_layout_ok" + msgClonePartition = "clone_partition" + msgReinstallBoot = "reinstall_boot" + msgReinstallBootOK = "reinstall_boot_ok" ) diff --git a/disk_linux.go b/disk_linux.go new file mode 100644 index 0000000..8898a0b --- /dev/null +++ b/disk_linux.go @@ -0,0 +1,322 @@ +//go:build linux + +package main + +import ( + "encoding/base64" + "fmt" + "os" + "os/exec" + "strconv" + "strings" + "unsafe" + + "syscall" +) + +// BLKSSZGET returns the logical (soft) sector size of a block device. +const blkSSZGet = 0x1268 + +func logicalSectorSize(path string) (int64, error) { + fi, err := os.Stat(path) + if err == nil && fi.Mode().IsRegular() { + return 512, nil + } + f, err := os.Open(path) + if err != nil { + return 512, err + } + defer f.Close() + var ssz int32 + _, _, errno := syscall.Syscall(syscall.SYS_IOCTL, f.Fd(), blkSSZGet, uintptr(unsafe.Pointer(&ssz))) + if errno != 0 || ssz <= 0 { + return 512, nil + } + return int64(ssz), nil +} + +// diskAttachment is a probe-time view of a whole disk: the block path whose +// partitions can be opened (the disk itself for a real device, a loop device +// for an image file) plus a cleanup to release a loop device. +type diskAttachment struct { + blockPath string + cleanup func() +} + +// attachDisk returns a block path whose partition nodes exist. For a regular +// file it sets up a partition-scanning loop device; for a real block device +// it is a no-op. +func attachDisk(path string, writable bool) (*diskAttachment, error) { + fi, err := os.Stat(path) + if err != nil { + return nil, err + } + if fi.Mode()&os.ModeDevice != 0 { + return &diskAttachment{blockPath: path, cleanup: func() {}}, nil + } + args := []string{"--find", "--show", "--partscan"} + if !writable { + args = append(args, "--read-only") + } + args = append(args, path) + out, err := exec.Command("losetup", args...).Output() + if err != nil { + return nil, fmt.Errorf("losetup %s: %w", path, cmdErr(err)) + } + lo := strings.TrimSpace(string(out)) + return &diskAttachment{ + blockPath: lo, + cleanup: func() { _ = exec.Command("losetup", "--detach", lo).Run() }, + }, nil +} + +// partNode maps a whole-disk block path + 1-based partition number to its +// partition device node (sda -> sda1, nvme0n1 -> nvme0n1p1, loop0 -> loop0p1). +func partNode(disk string, num int) string { + if l := len(disk); l > 0 && disk[l-1] >= '0' && disk[l-1] <= '9' { + return fmt.Sprintf("%sp%d", disk, num) + } + return fmt.Sprintf("%s%d", disk, num) +} + +var diskTools = []string{ + "sfdisk", "sgdisk", "partprobe", "blockdev", "blkid", "lsblk", "losetup", + "ntfsclone", "ntfsresize", "ntfsfix", "mkntfs", + "e2image", "resize2fs", "dumpe2fs", "mkfs.ext4", "e2fsck", + "partclone.dd", "partclone.extfs", "partclone.ntfs", "partclone.fat", + "partclone.exfat", "partclone.xfs", "partclone.btrfs", "partclone.restore", + "xfs_copy", "mkfs.vfat", "mkfs.xfs", "mkswap", "rsync", + "mount", "umount", "grub-install", "update-grub", "grub-mkconfig", +} + +func probeToolset() map[string]bool { + m := make(map[string]bool, len(diskTools)) + for _, t := range diskTools { + if _, err := exec.LookPath(t); err == nil { + m[t] = true + } + } + return m +} + +// probeDiskLayout gathers everything clone-disk needs from a source disk. +func probeDiskLayout(path string) (*DiskLayout, error) { + d := &DiskLayout{DiskPath: path, Tools: probeToolset()} + + info, err := statPath(path) + if err != nil { + return nil, err + } + if !info.Exists { + return nil, fmt.Errorf("%s does not exist", path) + } + d.DiskSize = info.Size + if d.LogicalSector, err = logicalSectorSize(path); err != nil { + d.LogicalSector = 512 + } + d.PhysicalSector = d.LogicalSector + + // Partition table via sfdisk -d (works on a file directly). + dump, derr := exec.Command("sfdisk", "-d", path).Output() + if derr != nil { + d.Scheme = "raw" + d.Warnings = append(d.Warnings, fmt.Sprintf("no partition table (%v)", cmdErr(derr))) + return d, nil + } + if err := d.parseSfdiskDump(string(dump)); err != nil { + return nil, err + } + if d.LogicalSector == 0 { + d.LogicalSector = 512 + } + if d.Scheme == "gpt" { + if b, err := exec.Command("sgdisk", "--backup=/dev/stdout", path).Output(); err == nil { + d.SgdiskB64 = base64.StdEncoding.EncodeToString(b) + } + } + + // Attach so per-partition nodes exist, then enrich each row. + att, err := attachDisk(path, false) + if err != nil { + d.Warnings = append(d.Warnings, fmt.Sprintf("cannot attach for fs probe: %v", err)) + } else { + defer att.cleanup() + for i := range d.Partitions { + enrichPartition(&d.Partitions[i], att.blockPath, d.LogicalSector) + } + } + + d.BootRegions = computeBootRegions(d) + return d, nil +} + +// enrichPartition fills the FS* fields of p using blkid + fs-specific probes. +func enrichPartition(p *Partition, disk string, sector int64) { + node := partNode(disk, p.Num) + p.DevPath = node + if _, err := os.Stat(node); err != nil { + return + } + if out, err := exec.Command("blkid", "-o", "export", node).Output(); err == nil { + for _, line := range strings.Split(string(out), "\n") { + k, v, ok := splitEq(strings.TrimSpace(line)) + if !ok { + continue + } + switch k { + case "TYPE": + p.FSType = v + case "LABEL": + p.FSLabel = v + case "UUID": + p.FSUUID = v + } + } + } + switch p.FSType { + case "ext2", "ext3", "ext4": + p.FSUsedBytes, p.FSMinBytes = probeExtSizes(node) + case "ntfs": + p.FSUsedBytes, p.FSMinBytes = probeNTFSSizes(node) + } +} + +func probeExtSizes(node string) (used, min int64) { + if out, err := exec.Command("dumpe2fs", "-h", node).Output(); err == nil { + var bs, count, free int64 + for _, line := range strings.Split(string(out), "\n") { + f := strings.SplitN(line, ":", 2) + if len(f) != 2 { + continue + } + v := strings.TrimSpace(f[1]) + switch strings.TrimSpace(f[0]) { + case "Block size": + bs, _ = strconv.ParseInt(v, 10, 64) + case "Block count": + count, _ = strconv.ParseInt(v, 10, 64) + case "Free blocks": + free, _ = strconv.ParseInt(v, 10, 64) + } + } + if bs > 0 && count > 0 { + used = (count - free) * bs + } + } + // resize2fs -P needs a clean fs; ignore failures. + if out, err := exec.Command("resize2fs", "-P", node).Output(); err == nil { + // "Estimated minimum size of the filesystem: 123456" (in 4k blocks) + s := string(out) + if i := strings.LastIndex(s, ":"); i >= 0 { + if n, err := strconv.ParseInt(strings.TrimSpace(s[i+1:]), 10, 64); err == nil { + min = n * 4096 + } + } + } + return used, min +} + +func probeNTFSSizes(node string) (used, min int64) { + out, err := exec.Command("ntfsresize", "--info", "--force", node).CombinedOutput() + if err != nil { + return 0, 0 + } + for _, line := range strings.Split(string(out), "\n") { + line = strings.TrimSpace(line) + switch { + case strings.HasPrefix(line, "You might resize at "): + // "You might resize at 1234567890 bytes ..." + fields := strings.Fields(line) + if len(fields) >= 5 { + if n, err := strconv.ParseInt(fields[4], 10, 64); err == nil { + min = n + } + } + case strings.Contains(line, "Space in use") && strings.Contains(line, "bytes"): + // "Space in use : 1234 MB (1.2%)" - not bytes; skip + } + } + return min, min +} + +// computeBootRegions returns the byte ranges outside any cloned partition +// that must still be copied verbatim. +func computeBootRegions(d *DiskLayout) []Region { + var regs []Region + // bootstrap code before the MBR partition table on both MBR and (BIOS-)GPT disks + regs = append(regs, Region{Offset: 0, Length: 446, Note: "MBR bootstrap"}) + if d.Scheme == "mbr" && len(d.Partitions) > 0 { + firstStart := d.Partitions[0].Start * d.LogicalSector + for _, p := range d.Partitions { + s := p.Start * d.LogicalSector + if s > 0 && s < firstStart { + firstStart = s + } + } + if firstStart > 512 { + regs = append(regs, Region{Offset: 512, Length: firstStart - 512, Note: "MBR gap (GRUB core.img etc.)"}) + } + } + return regs +} + +// applyPartitionTable writes script (an sfdisk restore script) to target, +// which may be a block device or an image file, and re-reads the table. +func applyPartitionTable(target, script string) error { + cmd := exec.Command("sfdisk", "--no-reread", "--no-tell-kernel", "--force", "--wipe=always", target) + cmd.Stdin = strings.NewReader(script) + if out, err := cmd.CombinedOutput(); err != nil { + return fmt.Errorf("sfdisk %s: %v: %s", target, err, strings.TrimSpace(string(out))) + } + rereadPartTable(target) + return nil +} + +func rereadPartTable(target string) { + fi, err := os.Stat(target) + if err != nil || fi.Mode()&os.ModeDevice == 0 { + return + } + if err := exec.Command("partprobe", target).Run(); err != nil { + _ = exec.Command("blockdev", "--rereadpt", target).Run() + } +} + +// lsblkFSType reports the filesystem type on a node ("" if none). +func lsblkFSType(node string) string { + out, err := exec.Command("lsblk", "-ndo", "FSTYPE", node).Output() + if err != nil { + return "" + } + return strings.TrimSpace(string(out)) +} + +// mountAt mounts node at a fresh temp dir and returns the dir + an unmount fn. +func mountAt(node string, readonly bool) (string, func(), error) { + dir, err := os.MkdirTemp("", "clonetool-mnt-") + if err != nil { + return "", nil, err + } + args := []string{} + if readonly { + args = append(args, "-o", "ro") + } + args = append(args, node, dir) + if out, err := exec.Command("mount", args...).CombinedOutput(); err != nil { + os.Remove(dir) + return "", nil, fmt.Errorf("mount %s: %v: %s", node, err, strings.TrimSpace(string(out))) + } + return dir, func() { + _ = exec.Command("umount", dir).Run() + _ = os.Remove(dir) + }, nil +} + +func cmdErr(err error) error { + if ee, ok := err.(*exec.ExitError); ok { + if s := strings.TrimSpace(string(ee.Stderr)); s != "" { + return fmt.Errorf("%v: %s", err, s) + } + } + return err +} diff --git a/disk_other.go b/disk_other.go new file mode 100644 index 0000000..584ba27 --- /dev/null +++ b/disk_other.go @@ -0,0 +1,37 @@ +//go:build !linux && !windows + +package main + +import ( + "fmt" + "runtime" +) + +func logicalSectorSize(string) (int64, error) { return 512, nil } + +type diskAttachment struct { + blockPath string + cleanup func() +} + +func attachDisk(string, bool) (*diskAttachment, error) { + return nil, fmt.Errorf("clone-disk is only implemented on linux and windows (this is %s)", runtime.GOOS) +} + +func partNode(disk string, num int) string { return fmt.Sprintf("%s%d", disk, num) } + +func probeDiskLayout(string) (*DiskLayout, error) { + return nil, fmt.Errorf("clone-disk is only implemented on linux and windows (this is %s)", runtime.GOOS) +} + +func applyPartitionTable(string, string) error { + return fmt.Errorf("clone-disk partition rebuild is only implemented on linux and windows") +} + +func rereadPartTable(string) {} + +func lsblkFSType(string) string { return "" } + +func mountAt(string, bool) (string, func(), error) { + return "", nil, fmt.Errorf("mount is only implemented on linux") +} diff --git a/disk_windows.go b/disk_windows.go new file mode 100644 index 0000000..60ec656 --- /dev/null +++ b/disk_windows.go @@ -0,0 +1,232 @@ +//go:build windows + +package main + +import ( + "encoding/binary" + "fmt" + "os" + "os/exec" + "strings" + "syscall" +) + +const ( + ioctlDiskGetDriveLayoutEx = 0x00070050 // IOCTL_DISK_GET_DRIVE_LAYOUT_EX + ioctlDiskUpdateProperties = 0x00070140 // IOCTL_DISK_UPDATE_PROPERTIES +) + +func logicalSectorSize(path string) (int64, error) { + if !isDevicePath(path) { + return 512, nil + } + return alignmentFor(path), nil +} + +// guidString renders a little-endian Win32 GUID (4-2-2 + 8 bytes) as the +// canonical upper-case string. +func guidString(b []byte) string { + if len(b) < 16 { + return "" + } + d1 := binary.LittleEndian.Uint32(b[0:4]) + d2 := binary.LittleEndian.Uint16(b[4:6]) + d3 := binary.LittleEndian.Uint16(b[6:8]) + return strings.ToUpper(fmt.Sprintf("%08X-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X", + d1, d2, d3, b[8], b[9], b[10], b[11], b[12], b[13], b[14], b[15])) +} + +// probeDiskLayout reads the partition table of a \\.\PhysicalDriveN via +// IOCTL_DISK_GET_DRIVE_LAYOUT_EX. Windows-native clone-disk is raw/VSS only: +// every partition is cloned block-wise and the table itself is reproduced by +// raw-copying the leading sectors (see computeBootRegionsWindows), so no +// sfdisk-style rebuild is needed. +func probeDiskLayout(path string) (*DiskLayout, error) { + d := &DiskLayout{DiskPath: path, Tools: probeToolsetWindows()} + + info, err := statPath(path) + if err != nil { + return nil, err + } + if !info.Exists { + return nil, fmt.Errorf("%s does not exist", path) + } + d.DiskSize = info.Size + d.LogicalSector, _ = logicalSectorSize(path) + if d.LogicalSector == 0 { + d.LogicalSector = 512 + } + d.PhysicalSector = d.LogicalSector + + if !isDevicePath(path) { + // An image file: we cannot enumerate its table without extra tooling. + d.Scheme = "raw" + d.Warnings = append(d.Warnings, "image-file source on Windows: whole-image raw clone only") + return d, nil + } + + p16, err := syscall.UTF16PtrFromString(path) + if err != nil { + return nil, err + } + h, err := syscall.CreateFile(p16, syscall.GENERIC_READ, + syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE, nil, syscall.OPEN_EXISTING, 0, 0) + if err != nil { + return nil, &os.PathError{Op: "open", Path: path, Err: err} + } + defer syscall.CloseHandle(h) + + buf := make([]byte, 64*1024) + var ret uint32 + if err := syscall.DeviceIoControl(h, ioctlDiskGetDriveLayoutEx, nil, 0, + &buf[0], uint32(len(buf)), &ret, nil); err != nil { + return nil, fmt.Errorf("IOCTL_DISK_GET_DRIVE_LAYOUT_EX %s: %w", path, err) + } + + style := binary.LittleEndian.Uint32(buf[0:4]) + count := binary.LittleEndian.Uint32(buf[4:8]) + switch style { + case 0: + d.Scheme, d.Label = "mbr", "dos" + sig := binary.LittleEndian.Uint32(buf[8:12]) + d.LabelID = fmt.Sprintf("0x%08x", sig) + case 1: + d.Scheme, d.Label = "gpt", "gpt" + d.LabelID = guidString(buf[8:24]) + default: + d.Scheme = "raw" + } + + const hdr = 48 + const peSize = 144 + sector := d.LogicalSector + for i := uint32(0); i < count; i++ { + off := hdr + int(i)*peSize + if off+peSize > len(buf) { + break + } + e := buf[off : off+peSize] + start := int64(binary.LittleEndian.Uint64(e[8:16])) + length := int64(binary.LittleEndian.Uint64(e[16:24])) + num := int(binary.LittleEndian.Uint32(e[24:28])) + if length == 0 || num == 0 { + continue + } + p := Partition{ + Num: num, + Start: start / sector, + Size: length / sector, + } + if style == 1 { // GPT union at offset 32 + p.Type = guidString(e[32:48]) + p.UUID = guidString(e[48:64]) + p.Name = decodeUTF16(e[72:144]) + } else if style == 0 { // MBR union at offset 32 + p.Type = fmt.Sprintf("%02x", e[32]) + if e[33] != 0 { + p.Attrs = "bootable" + } + } + d.Partitions = append(d.Partitions, p) + } + + d.BootRegions = computeBootRegionsWindows(d) + return d, nil +} + +func decodeUTF16(b []byte) string { + u := make([]uint16, 0, len(b)/2) + for i := 0; i+1 < len(b); i += 2 { + c := binary.LittleEndian.Uint16(b[i : i+2]) + if c == 0 { + break + } + u = append(u, c) + } + return strings.TrimRight(syscall.UTF16ToString(u), "\x00") +} + +// computeBootRegionsWindows: reproduce the partition table by raw-copying the +// sectors before the first partition, plus the trailing 33 sectors that hold +// the backup GPT. +func computeBootRegionsWindows(d *DiskLayout) []Region { + var regs []Region + if len(d.Partitions) == 0 { + return regs + } + first := d.Partitions[0].Start * d.LogicalSector + for _, p := range d.Partitions { + if s := p.Start * d.LogicalSector; s > 0 && s < first { + first = s + } + } + if first > 0 { + regs = append(regs, Region{Offset: 0, Length: first, Note: "protective MBR + primary GPT / MBR table + gap"}) + } + if d.Scheme == "gpt" && d.DiskSize > 33*d.LogicalSector { + regs = append(regs, Region{ + Offset: d.DiskSize - 33*d.LogicalSector, + Length: 33 * d.LogicalSector, + Note: "backup GPT", + }) + } + return regs +} + +func rereadPartTable(target string) { + if !isDevicePath(target) { + return + } + p16, err := syscall.UTF16PtrFromString(target) + if err != nil { + return + } + h, err := syscall.CreateFile(p16, syscall.GENERIC_READ|syscall.GENERIC_WRITE, + syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE, nil, syscall.OPEN_EXISTING, 0, 0) + if err != nil { + return + } + defer syscall.CloseHandle(h) + var ret uint32 + _ = syscall.DeviceIoControl(h, ioctlDiskUpdateProperties, nil, 0, nil, 0, &ret, nil) +} + +// applyPartitionTable is a no-op on Windows: the table is reproduced via the +// raw boot regions and rereadPartTable() refreshes the kernel's view. +func applyPartitionTable(target, script string) error { + rereadPartTable(target) + return nil +} + +func probeToolsetWindows() map[string]bool { + m := map[string]bool{} + for _, t := range []string{"vssadmin", "wmic", "bcdboot", "diskpart", "robocopy", "powershell"} { + if _, err := exec.LookPath(t); err == nil { + m[t] = true + } + } + return m +} + +type diskAttachment struct { + blockPath string + cleanup func() +} + +// attachDisk is a no-op for a raw \\.\PhysicalDriveN (partitions are addressed +// by byte offset within the whole-disk handle on Windows) and unsupported for +// an image file. +func attachDisk(path string, writable bool) (*diskAttachment, error) { + if isDevicePath(path) { + return &diskAttachment{blockPath: path, cleanup: func() {}}, nil + } + return nil, fmt.Errorf("filesystem-aware / file-level clone of an image file is not supported on Windows; use --raw") +} + +func partNode(disk string, num int) string { return fmt.Sprintf("%s (partition %d)", disk, num) } + +func lsblkFSType(string) string { return "" } + +func mountAt(string, bool) (string, func(), error) { + return "", nil, fmt.Errorf("mount-based file-level clone is not supported on Windows; use --raw") +} diff --git a/disklayout.go b/disklayout.go new file mode 100644 index 0000000..4aabd6a --- /dev/null +++ b/disklayout.go @@ -0,0 +1,298 @@ +package main + +import ( + "bufio" + "encoding/json" + "fmt" + "strconv" + "strings" +) + +// DiskLayout is the full picture of a source disk that clone-disk needs to +// reproduce a target: the partition table (as parsed from `sfdisk -d`, the +// canonical restorable form for both GPT and MBR), the byte ranges outside +// any partition that still have to be copied verbatim (MBR bootstrap + gap, +// BIOS-boot partition), the per-partition filesystem facts, and which of the +// external tools we shell out to are actually present. +type DiskLayout struct { + // Whole-disk geometry. + DiskPath string `json:"diskPath"` + DiskSize int64 `json:"diskSize"` // bytes + LogicalSector int64 `json:"logicalSector"` // bytes, from the kernel/ioctl + PhysicalSector int64 `json:"physicalSector"` // bytes + Scheme string `json:"scheme"` // "gpt" | "mbr" | "raw" (no recognisable table) + + // Partition table, parsed from `sfdisk -d`. + Label string `json:"label"` // "gpt" | "dos" + LabelID string `json:"labelID"` // GPT disk GUID / MBR 4-byte signature (0x...) + FirstLBA int64 `json:"firstLBA"` // sectors; 0 if the dump omitted it + LastLBA int64 `json:"lastLBA"` // sectors; 0 if omitted + Partitions []Partition `json:"partitions"` + SfdiskDump string `json:"sfdiskDump"` // the verbatim `sfdisk -d` output + SgdiskB64 string `json:"sgdiskB64"` // base64 `sgdisk --backup` blob (GPT only), fallback restore path + + // Byte ranges that must be raw-copied and are not covered by a cloned + // partition: MBR bootstrap (0..0x1BE) + the gap before the first + // partition, and any BIOS-boot partition. + BootRegions []Region `json:"bootRegions"` + + Tools map[string]bool `json:"tools"` // external tool name -> present on PATH + Warnings []string `json:"warnings,omitempty"` +} + +// Region is a [Offset, Offset+Length) byte window on the whole disk. +type Region struct { + Offset int64 `json:"offset"` + Length int64 `json:"length"` + Note string `json:"note,omitempty"` +} + +// Partition is one row of the partition table plus the filesystem facts the +// source agent gathered for it. +type Partition struct { + Num int `json:"num"` + Start int64 `json:"start"` // sectors + Size int64 `json:"size"` // sectors + Type string `json:"type"` // GPT type GUID or MBR hex code + UUID string `json:"uuid"` // GPT partition GUID (PARTUUID) + Name string `json:"name"` // GPT partition name + Attrs string `json:"attrs"` // trailing flags: "bootable", "attrs=\"...\"", ... + + // Filesystem facts (source agent; not from sfdisk). + DevPath string `json:"devPath"` // source node, e.g. /dev/sda2 + FSType string `json:"fsType"` // ext4 | ntfs | vfat | xfs | btrfs | swap | "" (none/unknown) + FSLabel string `json:"fsLabel"` + FSUUID string `json:"fsUUID"` + FSUsedBytes int64 `json:"fsUsedBytes"` // 0 if unknown + FSMinBytes int64 `json:"fsMinBytes"` // smallest the fs can be shrunk to; 0 if unknown/unshrinkable +} + +// StartBytes / SizeBytes convert the sector-unit table fields using the +// disk's logical sector size. +func (p Partition) StartBytes(sectorSize int64) int64 { return p.Start * sectorSize } +func (p Partition) SizeBytes(sectorSize int64) int64 { return p.Size * sectorSize } + +func (d *DiskLayout) JSON() string { + b, _ := json.Marshal(d) + return string(b) +} + +func parseDiskLayoutJSON(s string) (*DiskLayout, error) { + var d DiskLayout + if err := json.Unmarshal([]byte(s), &d); err != nil { + return nil, fmt.Errorf("decode disk layout: %w", err) + } + return &d, nil +} + +// parseSfdiskDump fills the Label/LabelID/FirstLBA/LastLBA/Partitions fields +// of d from the text of `sfdisk -d`. It leaves the raw text in d.SfdiskDump. +func (d *DiskLayout) parseSfdiskDump(dump string) error { + d.SfdiskDump = dump + sc := bufio.NewScanner(strings.NewReader(dump)) + sc.Buffer(make([]byte, 64*1024), 1024*1024) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" { + continue + } + if i := strings.Index(line, " : "); i >= 0 && strings.Contains(line, "start=") { + p, err := parseSfdiskPartLine(line) + if err != nil { + return err + } + d.Partitions = append(d.Partitions, p) + continue + } + key, val, ok := splitKV(line) + if !ok { + continue + } + switch key { + case "label": + d.Label = val + case "label-id": + d.LabelID = val + case "first-lba": + d.FirstLBA, _ = strconv.ParseInt(val, 10, 64) + case "last-lba": + d.LastLBA, _ = strconv.ParseInt(val, 10, 64) + case "sector-size": + if n, err := strconv.ParseInt(val, 10, 64); err == nil && d.LogicalSector == 0 { + d.LogicalSector = n + } + } + } + if err := sc.Err(); err != nil { + return fmt.Errorf("scan sfdisk dump: %w", err) + } + switch d.Label { + case "gpt": + d.Scheme = "gpt" + case "dos": + d.Scheme = "mbr" + } + return nil +} + +// splitKV parses a "key: value" header line. +func splitKV(line string) (string, string, bool) { + i := strings.Index(line, ":") + if i < 0 { + return "", "", false + } + return strings.TrimSpace(line[:i]), strings.TrimSpace(line[i+1:]), true +} + +// parseSfdiskPartLine parses a body line like +// +// /dev/sda2 : start= 1050624, size= 1951xxxxxx, type=0FC6..., uuid=..., name="root", attrs="..." +// /dev/sda1 : start= 2048, size= 204800, type=83, bootable +func parseSfdiskPartLine(line string) (Partition, error) { + var p Partition + i := strings.Index(line, " : ") + dev := strings.TrimSpace(line[:i]) + p.DevPath = dev + p.Num = trailingInt(dev) + rest := line[i+3:] + for _, fld := range splitTopLevel(rest, ',') { + fld = strings.TrimSpace(fld) + if fld == "" { + continue + } + k, v, ok := splitEq(fld) + if !ok { + // bare flag, e.g. "bootable" + if p.Attrs == "" { + p.Attrs = fld + } else { + p.Attrs += ", " + fld + } + continue + } + v = strings.Trim(strings.TrimSpace(v), `"`) + switch k { + case "start": + p.Start, _ = strconv.ParseInt(strings.TrimSpace(v), 10, 64) + case "size": + p.Size, _ = strconv.ParseInt(strings.TrimSpace(v), 10, 64) + case "type": + p.Type = v + case "uuid": + p.UUID = v + case "name": + p.Name = v + case "attrs": + if p.Attrs == "" { + p.Attrs = `attrs="` + v + `"` + } else { + p.Attrs += `, attrs="` + v + `"` + } + } + } + if p.Start == 0 && p.Size == 0 { + return p, fmt.Errorf("sfdisk line without start/size: %q", line) + } + return p, nil +} + +// sfdiskRestoreScript renders a device-independent script that recreates this +// table on any target with `sfdisk`. It intentionally drops `device:` and the +// per-line device prefixes (sfdisk numbers the partitions in order) and +// `last-lba` (let sfdisk size the secondary GPT for the target). label-id and +// per-partition uuid are dropped when newIDs is set so a replacement disk can +// coexist with the original; otherwise they are preserved so existing +// BCD/fstab/GRUB references still resolve. sizeOverride maps a 1-based +// partition number to a new size in sectors (used by the shrink path). +func (d *DiskLayout) sfdiskRestoreScript(newIDs bool, sizeOverride map[int]int64) string { + var b strings.Builder + fmt.Fprintf(&b, "label: %s\n", d.Label) + if d.LabelID != "" && !newIDs { + fmt.Fprintf(&b, "label-id: %s\n", d.LabelID) + } + if d.FirstLBA > 0 { + fmt.Fprintf(&b, "first-lba: %d\n", d.FirstLBA) + } + if d.LogicalSector > 0 { + fmt.Fprintf(&b, "sector-size: %d\n", d.LogicalSector) + } + b.WriteString("unit: sectors\n\n") + for _, p := range d.Partitions { + size := p.Size + if ov, ok := sizeOverride[p.Num]; ok && ov > 0 { + size = ov + } + fmt.Fprintf(&b, "start=%d, size=%d", p.Start, size) + if p.Type != "" { + fmt.Fprintf(&b, ", type=%s", p.Type) + } + if p.UUID != "" && !newIDs { + fmt.Fprintf(&b, ", uuid=%s", p.UUID) + } + if p.Name != "" { + fmt.Fprintf(&b, ", name=\"%s\"", p.Name) + } + if p.Attrs != "" { + fmt.Fprintf(&b, ", %s", p.Attrs) + } + b.WriteByte('\n') + } + return b.String() +} + +// biosBootType / espType are the GPT type GUIDs clone-disk special-cases. +const ( + gptBIOSBoot = "21686148-6449-6E6F-744E-656564454649" // EF02 BIOS boot partition + gptESP = "C12A7328-F81F-11D2-BA4B-00A0C93EC93B" // EF00 EFI system partition +) + +func (p Partition) isBIOSBoot() bool { + return strings.EqualFold(p.Type, gptBIOSBoot) || p.Type == "21" /* mbr */ +} +func (p Partition) isESP() bool { + return strings.EqualFold(p.Type, gptESP) || p.Type == "ef" || p.Type == "EF" +} + +// --- small string helpers ------------------------------------------------- + +func trailingInt(s string) int { + j := len(s) + for j > 0 && s[j-1] >= '0' && s[j-1] <= '9' { + j-- + } + if j == len(s) { + return 0 + } + n, _ := strconv.Atoi(s[j:]) + return n +} + +func splitEq(s string) (string, string, bool) { + i := strings.Index(s, "=") + if i < 0 { + return "", "", false + } + return strings.TrimSpace(s[:i]), s[i+1:], true +} + +// splitTopLevel splits on sep but not inside double quotes. +func splitTopLevel(s string, sep byte) []string { + var out []string + var cur strings.Builder + inQ := false + for i := 0; i < len(s); i++ { + c := s[i] + switch { + case c == '"': + inQ = !inQ + cur.WriteByte(c) + case c == sep && !inQ: + out = append(out, cur.String()) + cur.Reset() + default: + cur.WriteByte(c) + } + } + out = append(out, cur.String()) + return out +} diff --git a/disklayout_test.go b/disklayout_test.go new file mode 100644 index 0000000..26851bc --- /dev/null +++ b/disklayout_test.go @@ -0,0 +1,113 @@ +package main + +import ( + "strings" + "testing" +) + +const gptDump = `label: gpt +label-id: 1D2E3F4A-1111-2222-3333-444455556666 +device: /dev/sda +unit: sectors +first-lba: 2048 +last-lba: 8388574 +sector-size: 512 + +/dev/sda1 : start= 2048, size= 204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=AAAAAAAA-1111-2222-3333-444455556666, name="EFI System Partition" +/dev/sda2 : start= 206848, size= 8179712, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=BBBBBBBB-1111-2222-3333-444455556666 +` + +const mbrDump = `label: dos +label-id: 0x1a2b3c4d +device: /dev/sdb +unit: sectors +sector-size: 512 + +/dev/sdb1 : start= 2048, size= 512000, type=83, bootable +/dev/sdb2 : start= 514048, size= 41734144, type=83 +` + +func TestParseSfdiskDumpGPT(t *testing.T) { + var d DiskLayout + if err := d.parseSfdiskDump(gptDump); err != nil { + t.Fatal(err) + } + if d.Label != "gpt" || d.Scheme != "gpt" { + t.Fatalf("label/scheme = %q/%q", d.Label, d.Scheme) + } + if d.LabelID != "1D2E3F4A-1111-2222-3333-444455556666" { + t.Fatalf("labelID = %q", d.LabelID) + } + if d.FirstLBA != 2048 || d.LastLBA != 8388574 { + t.Fatalf("lba = %d/%d", d.FirstLBA, d.LastLBA) + } + if len(d.Partitions) != 2 { + t.Fatalf("parts = %d", len(d.Partitions)) + } + p1 := d.Partitions[0] + if p1.Num != 1 || p1.Start != 2048 || p1.Size != 204800 { + t.Fatalf("p1 = %+v", p1) + } + if p1.Type != "C12A7328-F81F-11D2-BA4B-00A0C93EC93B" || p1.UUID != "AAAAAAAA-1111-2222-3333-444455556666" { + t.Fatalf("p1 type/uuid = %q/%q", p1.Type, p1.UUID) + } + if p1.Name != "EFI System Partition" { + t.Fatalf("p1 name = %q", p1.Name) + } + if !p1.isESP() { + t.Fatalf("p1 should be ESP") + } + if d.Partitions[1].Num != 2 || d.Partitions[1].Size != 8179712 { + t.Fatalf("p2 = %+v", d.Partitions[1]) + } +} + +func TestParseSfdiskDumpMBR(t *testing.T) { + var d DiskLayout + if err := d.parseSfdiskDump(mbrDump); err != nil { + t.Fatal(err) + } + if d.Scheme != "mbr" || d.LabelID != "0x1a2b3c4d" { + t.Fatalf("scheme/id = %q/%q", d.Scheme, d.LabelID) + } + if len(d.Partitions) != 2 || d.Partitions[0].Attrs != "bootable" { + t.Fatalf("parts = %+v", d.Partitions) + } +} + +func TestSfdiskRestoreScript(t *testing.T) { + var d DiskLayout + if err := d.parseSfdiskDump(gptDump); err != nil { + t.Fatal(err) + } + + // keep IDs + s := d.sfdiskRestoreScript(false, nil) + if !strings.Contains(s, "label-id: 1D2E3F4A-1111-2222-3333-444455556666") { + t.Fatalf("expected label-id kept:\n%s", s) + } + if !strings.Contains(s, "uuid=AAAAAAAA-1111-2222-3333-444455556666") { + t.Fatalf("expected part uuid kept:\n%s", s) + } + if strings.Contains(s, "device:") || strings.Contains(s, "/dev/sda1") { + t.Fatalf("script must be device-independent:\n%s", s) + } + if strings.Contains(s, "last-lba") { + t.Fatalf("script must drop last-lba:\n%s", s) + } + + // new IDs + s2 := d.sfdiskRestoreScript(true, nil) + if strings.Contains(s2, "label-id:") || strings.Contains(s2, "uuid=") { + t.Fatalf("--new-ids must strip ids:\n%s", s2) + } + + // size override on p2 + s3 := d.sfdiskRestoreScript(false, map[int]int64{2: 4000000}) + if !strings.Contains(s3, "size=4000000") { + t.Fatalf("expected overridden size:\n%s", s3) + } + if strings.Contains(s3, "size=8179712") { + t.Fatalf("old size should be gone:\n%s", s3) + } +} diff --git a/fsclone.go b/fsclone.go new file mode 100644 index 0000000..3d46cc1 --- /dev/null +++ b/fsclone.go @@ -0,0 +1,200 @@ +package main + +import ( + "bufio" + "encoding/json" + "fmt" + "io" + "regexp" + "strconv" + "strings" +) + +// fsCloner is the pair of argv templates that stream one filesystem as an +// image: save reads the fs on dev and writes an image to stdout; restore +// reads that image from stdin and writes it onto dev. The image format is +// whatever the chosen standard tool uses (ntfsclone, partclone, e2image) — +// clonetool only moves the bytes between the two ends. +type fsCloner struct { + name string // tool family, for logs + save []string + restore []string +} + +// fsCloneOpts is the JSON blob carried in CtrlMsg.Options for a fs-image +// partition clone so both ends build the identical argv. +type fsCloneOpts struct { + FSType string `json:"fsType"` + Tool string `json:"tool"` + SizeBytes int64 `json:"sizeBytes"` + // ShrinkToBytes > 0: shrink the SOURCE filesystem to this size in place + // before imaging it (so it fits a smaller target partition). This + // mutates the source and only runs when the operator passed + // --allow-shrink --yes. + ShrinkToBytes int64 `json:"shrinkToBytes,omitempty"` +} + +func (o fsCloneOpts) encode() string { b, _ := json.Marshal(o); return string(b) } +func decodeFSCloneOpts(s string) (fsCloneOpts, error) { + var o fsCloneOpts + err := json.Unmarshal([]byte(s), &o) + return o, err +} + +// pickCloner returns the fs-image cloner for (fsType, tool) or nil if the +// partition should be handled some other way. tool is the family name chosen +// by chooseCloneMethod ("ntfsclone", "partclone", "e2image"). +func pickCloner(dev, fsType, tool string) *fsCloner { + switch tool { + case "ntfsclone": + return &fsCloner{ + name: "ntfsclone", + save: []string{"ntfsclone", "--save-image", "--force", "--output", "-", dev}, + restore: []string{"ntfsclone", "--restore-image", "--overwrite", dev, "-"}, + } + case "e2image": + return &fsCloner{ + name: "e2image", + // -ra: raw + skip unallocated; stream to/from stdio. + save: []string{"e2image", "-ra", "-p", dev, "-"}, + restore: []string{"e2image", "-ra", "-p", "-", dev}, + } + case "partclone": + pc := "partclone." + partcloneSuffix(fsType) + return &fsCloner{ + name: pc, + save: []string{pc, "-c", "-s", dev, "-O", "-", "-L", "/dev/null"}, + restore: []string{"partclone.restore", "-s", "-", "-O", dev, "-L", "/dev/null"}, + } + } + return nil +} + +func partcloneSuffix(fsType string) string { + switch fsType { + case "ext2", "ext3", "ext4": + return "extfs" + case "vfat", "fat", "fat12", "fat16", "fat32", "msdos": + return "fat" + case "ntfs": + return "ntfs" + default: + return fsType // exfat, xfs, btrfs, f2fs, hfsplus, ... + } +} + +// chooseCloneMethod decides how a partition's data is copied. +// +// "raw" -> the block-diff engine over an offset window (also the +// re-run-friendly path; used for swap, unknown fs, --raw) +// "fs-image" -> stream a filesystem image with a standard tool +// "file-level" -> mkfs on the target + rsync +func chooseCloneMethod(p Partition, tools map[string]bool, forceRaw, forceFile map[int]bool, fileAuto bool) (method, tool string) { + if forceRaw[p.Num] { + return "raw", "" + } + if forceFile[p.Num] { + return "file-level", "" + } + if p.isBIOSBoot() { + return "raw", "" // tiny, and holds GRUB core - always verbatim + } + switch p.FSType { + case "", "swap", "crypto_LUKS", "LVM2_member": + return "raw", "" + case "ntfs": + if tools["ntfsclone"] { + return "fs-image", "ntfsclone" + } + case "ext2", "ext3", "ext4": + if tools["partclone.extfs"] && tools["partclone.restore"] { + return "fs-image", "partclone" + } + if tools["e2image"] { + return "fs-image", "e2image" + } + case "vfat", "fat", "fat12", "fat16", "fat32", "msdos": + if tools["partclone.fat"] && tools["partclone.restore"] { + return "fs-image", "partclone" + } + case "exfat", "xfs", "btrfs", "f2fs", "hfsplus": + if tools["partclone."+partcloneSuffix(p.FSType)] && tools["partclone.restore"] { + return "fs-image", "partclone" + } + } + if fileAuto && p.FSType != "" && tools["rsync"] { + return "file-level", "" + } + return "raw", "" +} + +// progressScanner reads a cloner's stderr line by line, pulls a percentage +// out of the tool's own progress chatter, and calls emit with an estimated +// byte count (fraction * sizeBytes). It swallows everything else. +var pctRe = regexp.MustCompile(`([0-9]{1,3}(?:\.[0-9]+)?)\s*%|([0-9]{1,3}(?:\.[0-9]+)?)\s*percent`) + +func scanCloneProgress(r io.Reader, sizeBytes int64, emit func(bytesDone int64)) { + br := bufio.NewReader(r) + var buf []byte + flush := func() { + line := strings.TrimSpace(string(buf)) + buf = buf[:0] + if line == "" { + return + } + m := pctRe.FindStringSubmatch(line) + if m == nil { + return + } + s := m[1] + if s == "" { + s = m[2] + } + if f, err := strconv.ParseFloat(s, 64); err == nil && f >= 0 && f <= 100 { + emit(int64(f / 100 * float64(sizeBytes))) + } + } + for { + c, err := br.ReadByte() + if err != nil { + flush() + return + } + if c == '\n' || c == '\r' { + flush() + continue + } + buf = append(buf, c) + if len(buf) > 4096 { + buf = buf[:0] + } + } +} + +// preShrinkFS shrinks the filesystem on dev to (at least) toBytes in place. +// Used only on the source, only with --allow-shrink --yes. Best effort with a +// hard error if the resize tool fails. +func preShrinkFS(dev, fsType string, toBytes int64) error { + if toBytes <= 0 { + return nil + } + switch { + case isExtFS(fsType): + if out, err := runCmd("e2fsck", "-f", "-y", dev); err != nil { + return fmt.Errorf("e2fsck %s: %v: %s", dev, err, out) + } + kib := toBytes / 1024 + if out, err := runCmd("resize2fs", dev, fmt.Sprintf("%dK", kib)); err != nil { + return fmt.Errorf("resize2fs %s %dK: %v: %s", dev, kib, err, out) + } + return nil + case fsType == "ntfs": + // ntfsresize prompts for confirmation on a real resize; feed it "y". + if out, err := runCmdStdin("y\n", "ntfsresize", "--force", "--size", strconv.FormatInt(toBytes, 10), dev); err != nil { + return fmt.Errorf("ntfsresize %s --size %d: %v: %s", dev, toBytes, err, out) + } + return nil + default: + return fmt.Errorf("don't know how to shrink a %s filesystem", fsType) + } +} diff --git a/main.go b/main.go index 446f1de..c1b566e 100644 --- a/main.go +++ b/main.go @@ -5,6 +5,7 @@ import ( "fmt" "os" "runtime" + "strings" ) func main() { @@ -17,6 +18,8 @@ func main() { switch os.Args[1] { case "sync": err = cmdSync(os.Args[2:]) + case "clone-disk": + err = cmdCloneDisk(os.Args[2:]) case "agent": err = cmdAgent(os.Args[2:]) case "version", "--version": @@ -40,8 +43,9 @@ func usage() { Usage: clonetool sync --source LOC --dest LOC [options] + clonetool clone-disk --source LOC --dest LOC [options] clonetool version - clonetool agent --role {control|sink|source-stream} ... (internal, spawned automatically) + clonetool agent --role {control|sink|source-stream|fs-send|fs-recv} ... (internal, spawned automatically) LOC is either a local path, or [user@]host:path for a path reached over SSH. Source, destination, and the machine running "sync" (the manager) may all be @@ -51,6 +55,21 @@ a single block itself. clonetool keeps no state between runs: every sync re-reads and re-hashes both the source and the destination and transfers only the blocks that differ. +Options for clone-disk (whole-disk: boot record + partition table + per-filesystem data): + --parts LIST only clone these partition numbers (comma-separated; default all) + --raw LIST force a raw block clone for these partitions + --file-level LIST mkfs + rsync these partitions instead of a filesystem image + --file-level-auto use file-level for filesystem types with no image cloner + --allow-shrink permit shrinking trailing partitions to fit a smaller target + (resizes the SOURCE filesystem in place before imaging) + --no-shrink never shrink; fail instead if the target is too small + --new-ids randomize the GPT disk GUID / MBR signature on the target + --reinstall-bootloader after copy, run grub-install / grub-mkconfig on the target (Linux) + --vss Windows source: take a Volume Shadow Copy per NTFS volume (default on) + --image-size SIZE file target: image size (default = enough for the layout) + (also accepts --block-size --job --yes --sudo --deploy --ssh --ssh-opt --remote-bin + --connect-timeout --manager-host, same meaning as sync) + Options for sync: --block-size SIZE block size, e.g. 4M (default 4M) --job NAME optional label shown in progress/log output @@ -116,6 +135,97 @@ func cmdSync(args []string) error { }) } +func parseIntList(s string) ([]int, error) { + if strings.TrimSpace(s) == "" { + return nil, nil + } + var out []int + for _, f := range strings.Split(s, ",") { + f = strings.TrimSpace(f) + if f == "" { + continue + } + var n int + if _, err := fmt.Sscanf(f, "%d", &n); err != nil || n <= 0 { + return nil, fmt.Errorf("invalid partition number %q", f) + } + out = append(out, n) + } + return out, nil +} + +func cmdCloneDisk(args []string) error { + fs := flag.NewFlagSet("clone-disk", flag.ContinueOnError) + job := fs.String("job", "", "optional label for progress/log output") + source := fs.String("source", "", "source whole-disk location (required)") + dest := fs.String("dest", "", "destination whole-disk or image location (required)") + blockSizeStr := fs.String("block-size", "4M", "block size for raw/boot-region transfers, e.g. 4M") + yes := fs.Bool("yes", false, "don't prompt before shrinking an existing destination image") + sudoMode := fs.String("sudo", "auto", "device-access privilege escalation: auto|always|never") + deploy := fs.Bool("deploy", true, "copy this binary to remote hosts that lack it") + connectTimeout := fs.Int("connect-timeout", defaultConnectTimeoutSec, "ssh connect timeout (seconds)") + sshBin := fs.String("ssh", "ssh", "ssh binary") + remoteBin := fs.String("remote-bin", "clonetool", "clonetool path on remote hosts") + managerHost := fs.String("manager-host", "", "address peers use to reach this machine") + partsStr := fs.String("parts", "", "only clone these partition numbers (comma-separated)") + rawStr := fs.String("raw", "", "force raw block clone for these partition numbers") + fileLevelStr := fs.String("file-level", "", "mkfs + rsync these partition numbers") + fileAuto := fs.Bool("file-level-auto", false, "file-level for fs types with no image cloner") + allowShrink := fs.Bool("allow-shrink", false, "permit shrinking trailing partitions to fit a smaller target") + noShrink := fs.Bool("no-shrink", false, "never shrink; fail if the target is too small") + newIDs := fs.Bool("new-ids", false, "randomize the GPT disk GUID / MBR signature on the target") + reinstallBoot := fs.Bool("reinstall-bootloader", false, "run grub-install / grub-mkconfig on the target after copy") + vss := fs.Bool("vss", true, "Windows source: take a Volume Shadow Copy per NTFS volume") + imageSizeStr := fs.String("image-size", "", "file target: image size (default: enough for the layout)") + var sshOpts stringSlice + fs.Var(&sshOpts, "ssh-opt", `extra "-o OPT" passed to ssh (repeatable)`) + if err := fs.Parse(args); err != nil { + return err + } + if *source == "" || *dest == "" { + fs.Usage() + return fmt.Errorf("--source and --dest are required") + } + blockSize, err := parseSize(*blockSizeStr) + if err != nil { + return fmt.Errorf("--block-size: %w", err) + } + switch *sudoMode { + case "auto", "always", "never": + default: + return fmt.Errorf("--sudo: want auto|always|never, got %q", *sudoMode) + } + parts, err := parseIntList(*partsStr) + if err != nil { + return fmt.Errorf("--parts: %w", err) + } + raw, err := parseIntList(*rawStr) + if err != nil { + return fmt.Errorf("--raw: %w", err) + } + fileLevel, err := parseIntList(*fileLevelStr) + if err != nil { + return fmt.Errorf("--file-level: %w", err) + } + var imageSize int64 + if *imageSizeStr != "" { + if imageSize, err = parseSize(*imageSizeStr); err != nil { + return fmt.Errorf("--image-size: %w", err) + } + } + + return runCloneDisk(CloneDiskConfig{ + SyncConfig: SyncConfig{ + Job: *job, Source: *source, Dest: *dest, BlockSize: blockSize, + Yes: *yes, Sudo: *sudoMode, Deploy: *deploy, ConnectTimeoutSec: *connectTimeout, + SSHBin: *sshBin, SSHOpts: sshOpts, RemoteBin: *remoteBin, ManagerHost: *managerHost, + }, + Parts: parts, Raw: raw, FileLevel: fileLevel, FileAuto: *fileAuto, + AllowShrink: *allowShrink, NoShrink: *noShrink, NewIDs: *newIDs, + ReinstallBoot: *reinstallBoot, VSS: *vss, ImageSize: imageSize, + }) +} + func parseSize(s string) (int64, error) { if s == "" { return 0, fmt.Errorf("empty size") diff --git a/manager.go b/manager.go index ac05113..333f474 100644 --- a/manager.go +++ b/manager.go @@ -297,6 +297,26 @@ func (p *progressPrinter) print(m CtrlMsg) { formatCount(p.copied), formatCount(p.skipped), scan, mib) } +// printBytes renders a byte-oriented status line for clone-disk's fs-image +// partition clones, where progress arrives as BytesCopied/BytesTotal rather +// than block counts. +func (p *progressPrinter) printBytes(m CtrlMsg) { + if m.Type != msgProgress { + return + } + p.active = true + done, total := m.BytesCopied, m.BytesTotal + if total <= 0 { + total = done + } + stage := m.Stage + if stage == "" { + stage = "fs-image" + } + fmt.Fprintf(os.Stderr, "\r %s %s %s / %s ", + stage, progressBar(done, total), humanBytes(done), humanBytes(total)) +} + // finish ends the current status line with a newline so following output // (and the shell prompt) starts clean. func (p *progressPrinter) finish() { diff --git a/plan.go b/plan.go new file mode 100644 index 0000000..3173f9b --- /dev/null +++ b/plan.go @@ -0,0 +1,215 @@ +package main + +import ( + "fmt" + "strings" +) + +// plannedPart is one partition's copy plan: where it lives on each side (in +// bytes) and how its data moves. +type plannedPart struct { + Num int + SrcStartB, SrcSizeB int64 + DstStartB, DstSizeB int64 + Method string // "raw" | "fs-image" | "file-level" + Tool string // fs-image tool family + FSType string + ShrinkToB int64 // >0: shrink the fs to this many bytes before imaging +} + +type targetPlan struct { + Script string // sfdisk restore script ("" when the source has no table / Windows-native) + IsFile bool + ImageSize int64 // file target: truncate to this + Parts []plannedPart + BootRegions []Region + RootPart int + ESPPart int + UEFI bool + Notes []string +} + +// bootOpts is the JSON blob passed to reinstall_boot. +type bootOpts struct { + RootPart int `json:"rootPart"` // 1-based partition number holding "/" + ESPPart int `json:"espPart"` // 1-based ESP partition number, 0 if BIOS-only + UEFI bool `json:"uefi"` + DiskPath string `json:"diskPath"` // whole-disk path grub-install targets +} + +type planOpts struct { + Parts map[int]bool // empty = all + Raw map[int]bool + FileLevel map[int]bool + FileAuto bool + AllowShrink bool + NewIDs bool + ImageSize int64 // explicit file-target size, 0 = auto +} + +const gptTailSectors = 33 // secondary GPT header + entries + +// planTargetLayout turns a probed source layout + the target's nature/size +// into a concrete copy plan, applying the sizing rules: +// +// - device target: never grow; if everything fits, keep the layout and +// leave trailing space untouched; if the last partition(s) overflow, +// shrink them (filesystem + partition) from the last inward, or fail. +// - file target: size the image to hold the layout (or ImageSize). +func planTargetLayout(src *DiskLayout, isFile bool, destSize int64, o planOpts) (*targetPlan, error) { + sector := src.LogicalSector + if sector <= 0 { + sector = 512 + } + parts := sortedParts(src.Partitions) + var sel []Partition + for _, p := range parts { + if len(o.Parts) == 0 || o.Parts[p.Num] { + sel = append(sel, p) + } + } + if len(sel) == 0 && src.Scheme != "raw" { + return nil, fmt.Errorf("no partitions selected") + } + + plan := &targetPlan{IsFile: isFile, BootRegions: append([]Region(nil), src.BootRegions...)} + + // With --new-ids on an MBR disk, keep the raw bootstrap copy clear of the + // 4-byte disk signature at offset 0x1B8 so sfdisk's freshly generated one + // survives (GPT regenerates its GUID in the header, which we never + // raw-copy, so no clamp is needed there). + if o.NewIDs && src.Scheme == "mbr" { + for i := range plan.BootRegions { + r := &plan.BootRegions[i] + if r.Offset == 0 && r.Length > 0x1B8 { + r.Length = 0x1B8 + } + } + } + + // Bytes the layout needs on disk = end of the last selected partition + // (+ secondary GPT for GPT disks). + lastEndB := int64(0) + for _, p := range sel { + if e := (p.Start + p.Size) * sector; e > lastEndB { + lastEndB = e + } + } + neededB := lastEndB + if src.Scheme == "gpt" { + neededB += gptTailSectors * sector + } + if src.Scheme == "raw" { + neededB = src.DiskSize + } + + if isFile { + plan.ImageSize = o.ImageSize + if plan.ImageSize == 0 { + plan.ImageSize = roundUp(neededB+(1<<20), 1<<20) + } + destSize = plan.ImageSize + } + if destSize <= 0 { + return nil, fmt.Errorf("could not determine target size") + } + + // --- shrink pass ---------------------------------------------------- + sizeOverride := map[int]int64{} + if destSize < neededB { + short := neededB - destSize + for i := len(sel) - 1; i >= 0 && short > 0; i-- { + p := &sel[i] + curB := p.Size * sector + minB := p.FSMinBytes + shrinkable := minB > 0 && (isExtFS(p.FSType) || p.FSType == "ntfs") + if !shrinkable { + continue + } + minB = roundUp(minB+(16<<20), sector) // 16 MiB slack + if minB >= curB { + continue + } + reduce := curB - minB + if reduce > short { + reduce = roundUp(short, sector) + } + newB := curB - reduce + sizeOverride[p.Num] = newB / sector + plan.Notes = append(plan.Notes, + fmt.Sprintf("partition p%d (%s) shrinks %s -> %s to fit the target", + p.Num, p.FSType, humanBytes(curB), humanBytes(newB))) + short -= reduce + } + if short > 0 { + return nil, fmt.Errorf( + "target (%s) is smaller than the source layout needs (%s); the trailing partition(s) "+ + "cannot shrink to make up the missing %s; use a larger target", + humanBytes(destSize), humanBytes(neededB), humanBytes(short)) + } + if !o.AllowShrink { + var names []string + for n := range sizeOverride { + names = append(names, fmt.Sprintf("p%d", n)) + } + return nil, fmt.Errorf( + "target is smaller than the source; partition(s) %s would have to be shrunk. "+ + "Re-run with --allow-shrink (this resizes the SOURCE filesystem in place before imaging) "+ + "or use --no-shrink off / a larger target", + strings.Join(names, ", ")) + } + } + + // --- partition-table script -------------------------------------- + if src.Scheme != "raw" && strings.TrimSpace(src.SfdiskDump) != "" { + plan.Script = src.sfdiskRestoreScript(o.NewIDs, sizeOverride) + } + + // --- per-partition method -------------------------------------- + for _, p := range sel { + pp := plannedPart{ + Num: p.Num, + SrcStartB: p.Start * sector, + SrcSizeB: p.Size * sector, + DstStartB: p.Start * sector, + DstSizeB: p.Size * sector, + FSType: p.FSType, + } + if ov, ok := sizeOverride[p.Num]; ok { + pp.DstSizeB = ov * sector + pp.ShrinkToB = ov * sector + } + method, tool := chooseCloneMethod(p, src.Tools, o.Raw, o.FileLevel, o.FileAuto) + pp.Method, pp.Tool = method, tool + if pp.ShrinkToB > 0 && method != "fs-image" { + return nil, fmt.Errorf( + "partition p%d (%s) must shrink to fit but no image cloner is available for it; "+ + "install ntfsclone / partclone or use a larger target", p.Num, orDash(p.FSType)) + } + plan.Parts = append(plan.Parts, pp) + } + + // --- boot roles ------------------------------------------------ + plan.RootPart, plan.ESPPart = pickBootParts(sel) + plan.UEFI = plan.ESPPart != 0 + return plan, nil +} + +func isExtFS(t string) bool { return t == "ext2" || t == "ext3" || t == "ext4" } + +// pickBootParts guesses the "/" partition (largest Linux fs) and the ESP. +func pickBootParts(sel []Partition) (root, esp int) { + var bestSize int64 + for _, p := range sel { + if p.isESP() && esp == 0 { + esp = p.Num + } + switch p.FSType { + case "ext2", "ext3", "ext4", "btrfs", "xfs": + if sz := p.Size; sz > bestSize { + bestSize, root = sz, p.Num + } + } + } + return root, esp +} diff --git a/plan_test.go b/plan_test.go new file mode 100644 index 0000000..504cd8b --- /dev/null +++ b/plan_test.go @@ -0,0 +1,146 @@ +package main + +import "testing" + +// mkLayout builds a GPT layout: ESP (vfat) + root (ext4) + data (ntfs). +func mkLayout(diskBytes int64) *DiskLayout { + d := &DiskLayout{ + DiskPath: "/dev/src", DiskSize: diskBytes, LogicalSector: 512, + Scheme: "gpt", Label: "gpt", LabelID: "GUID", + SfdiskDump: gptDump, // any non-empty dump so a script is produced + Tools: map[string]bool{ + "sfdisk": true, "ntfsclone": true, + "partclone.extfs": true, "partclone.restore": true, "e2image": true, + }, + Partitions: []Partition{ + {Num: 1, Start: 2048, Size: 204800, Type: gptESP, FSType: "vfat"}, // 100 MiB + {Num: 2, Start: 206848, Size: 10 * 2048 * 1024, FSType: "ext4", FSMinBytes: 2 << 30}, // 10 GiB, min 2 GiB + {Num: 3, Start: 206848 + 10*2048*1024, Size: 10 * 2048 * 1024, FSType: "ntfs", FSMinBytes: 3 << 30}, // 10 GiB, min 3 GiB + }, + BootRegions: []Region{{Offset: 0, Length: 446, Note: "MBR bootstrap"}}, + } + return d +} + +func lastEndBytes(d *DiskLayout) int64 { + var e int64 + for _, p := range d.Partitions { + if x := (p.Start + p.Size) * d.LogicalSector; x > e { + e = x + } + } + return e + gptTailSectors*d.LogicalSector +} + +func TestPlanFitsDeviceKeepsLayout(t *testing.T) { + d := mkLayout(40 << 30) + plan, err := planTargetLayout(d, false, 40<<30, planOpts{}) + if err != nil { + t.Fatal(err) + } + if len(plan.Parts) != 3 { + t.Fatalf("parts = %d", len(plan.Parts)) + } + for _, p := range plan.Parts { + if p.SrcSizeB != p.DstSizeB || p.ShrinkToB != 0 { + t.Fatalf("p%d resized unexpectedly: %+v", p.Num, p) + } + if p.SrcStartB != p.DstStartB { + t.Fatalf("p%d start moved", p.Num) + } + } + if plan.Parts[0].Method != "raw" { // vfat, no partclone.fat in tool set + t.Fatalf("p1 method = %s", plan.Parts[0].Method) + } + if plan.Parts[1].Method != "fs-image" || plan.Parts[1].Tool != "partclone" { + t.Fatalf("p2 method/tool = %s/%s", plan.Parts[1].Method, plan.Parts[1].Tool) + } + if plan.Parts[2].Method != "fs-image" || plan.Parts[2].Tool != "ntfsclone" { + t.Fatalf("p3 method/tool = %s/%s", plan.Parts[2].Method, plan.Parts[2].Tool) + } + if plan.ESPPart != 1 || plan.RootPart != 2 || !plan.UEFI { + t.Fatalf("boot roles: esp=%d root=%d uefi=%v", plan.ESPPart, plan.RootPart, plan.UEFI) + } +} + +func TestPlanTrailingFreeSpaceOK(t *testing.T) { + d := mkLayout(60 << 30) + // target smaller than the *disk* but larger than what the layout needs + need := lastEndBytes(d) + if _, err := planTargetLayout(d, false, need+1<<20, planOpts{}); err != nil { + t.Fatalf("should fit into %d (need %d): %v", need+1<<20, need, err) + } +} + +func TestPlanNeedsShrinkRequiresFlag(t *testing.T) { + d := mkLayout(40 << 30) + need := lastEndBytes(d) + small := need - (4 << 30) // 4 GiB short; ntfs (min 3 GiB) can give it up + + if _, err := planTargetLayout(d, false, small, planOpts{}); err == nil { + t.Fatalf("expected error without --allow-shrink") + } + plan, err := planTargetLayout(d, false, small, planOpts{AllowShrink: true}) + if err != nil { + t.Fatalf("with --allow-shrink: %v", err) + } + p3 := plan.Parts[2] + if p3.ShrinkToB == 0 || p3.ShrinkToB >= p3.SrcSizeB { + t.Fatalf("p3 should shrink: %+v", p3) + } + if p3.Method != "fs-image" { + t.Fatalf("shrunk partition must be fs-image, got %s", p3.Method) + } +} + +func TestPlanCannotFit(t *testing.T) { + d := mkLayout(40 << 30) + // absurdly small: even shrinking ntfs+ext to their minimums can't help + if _, err := planTargetLayout(d, false, 1<<30, planOpts{AllowShrink: true}); err == nil { + t.Fatalf("expected cannot-fit error") + } +} + +func TestPlanFileTargetSizing(t *testing.T) { + d := mkLayout(40 << 30) + plan, err := planTargetLayout(d, true, 0, planOpts{}) + if err != nil { + t.Fatal(err) + } + need := lastEndBytes(d) + if plan.ImageSize < need || plan.ImageSize > need+(2<<20) { + t.Fatalf("image size %d not ~%d", plan.ImageSize, need) + } + if plan.Script == "" { + t.Fatalf("expected a partition-table script for a file target") + } +} + +func TestPlanNewIDsScript(t *testing.T) { + d := mkLayout(40 << 30) + plan, err := planTargetLayout(d, true, 0, planOpts{NewIDs: true}) + if err != nil { + t.Fatal(err) + } + if plan.Script == "" || containsAny(plan.Script, "label-id:", "uuid=") { + t.Fatalf("--new-ids script still has ids:\n%s", plan.Script) + } +} + +func containsAny(s string, subs ...string) bool { + for _, x := range subs { + if len(x) > 0 && indexOf(s, x) >= 0 { + return true + } + } + return false +} + +func indexOf(s, sub string) int { + for i := 0; i+len(sub) <= len(s); i++ { + if s[i:i+len(sub)] == sub { + return i + } + } + return -1 +} diff --git a/syncside.go b/syncside.go index f3b2c9c..eafc635 100644 --- a/syncside.go +++ b/syncside.go @@ -81,12 +81,15 @@ type blockHash struct { hash [32]byte } -// streamHashBlocks reads f in blockSize-byte blocks up to size, hashing each -// one and handing it to onHash the moment it is computed (this is what lets -// the peer start comparing before the whole side has been scanned). It keeps -// no hash state of its own. onProgress, if non-nil, is called with -// (blocksHashed, totalBlocks) before the first block and after each one. -func streamHashBlocks(f *os.File, size, blockSize, align int64, onHash func(bh blockHash) error, onProgress func(done, total int64)) error { +// streamHashBlocks reads f in blockSize-byte blocks over the window +// [base, base+size), hashing each one and handing it to onHash the moment it +// is computed (this is what lets the peer start comparing before the whole +// side has been scanned). It keeps no hash state of its own. onProgress, if +// non-nil, is called with (blocksHashed, totalBlocks) before the first block +// and after each one. base is 0 for a whole-file sync; clone-disk passes a +// non-zero base to fingerprint just a boot region / offset-addressed +// partition within a larger handle. +func streamHashBlocks(f *os.File, base, size, blockSize, align int64, onHash func(bh blockHash) error, onProgress func(done, total int64)) error { if err := checkBlockAlign(blockSize, align); err != nil { return err } @@ -96,7 +99,7 @@ func streamHashBlocks(f *os.File, size, blockSize, align int64, onHash func(bh b onProgress(0, blockCount) } for i := int64(0); i < blockCount; i++ { - n, err := readBlockAt(f, buf, i*blockSize, size, align) + n, err := readBlockAt(f, buf, base+i*blockSize, base+size, align) if err != nil { return fmt.Errorf("hash %s at block %d: %w", f.Name(), i, err) } @@ -130,7 +133,8 @@ func scanProgressEmitter(out *FrameWriter) func(done, total int64) { // never read twice. type sourceLoopParams struct { File *os.File // already opened for reading by the caller - Size int64 + Base int64 // byte offset the window starts at (0 for a whole-file sync) + Size int64 // window length in bytes BlockSize int64 Align int64 Hashes <-chan blockHash @@ -151,8 +155,8 @@ func runSourceLoop(p sourceLoopParams) error { f := p.File buf := make([]byte, p.BlockSize) for bh := range p.Hashes { - offset := int64(bh.index) * p.BlockSize - n, err := readBlockAt(f, buf, offset, p.Size, p.Align) + offset := p.Base + int64(bh.index)*p.BlockSize + n, err := readBlockAt(f, buf, offset, p.Base+p.Size, p.Align) if err != nil { return fmt.Errorf("read %s at block %d: %w", f.Name(), bh.index, err) } @@ -181,6 +185,7 @@ func runSourceLoop(p sourceLoopParams) error { // same process, before OnWritten is called). type destLoopParams struct { File *os.File // already opened for read/write by the caller + Base int64 // byte offset the window starts at (0 for a whole-file sync) BlockSize int64 Align int64 // sector alignment for a raw disk handle, else 0/1 In *FrameReader @@ -212,7 +217,7 @@ func runDestLoop(p destLoopParams) error { } return fmt.Errorf("block %d: hash mismatch after transfer", index) } - if err := writeBlockAt(f, block, int64(index)*p.BlockSize, p.Align); err != nil { + if err := writeBlockAt(f, block, p.Base+int64(index)*p.BlockSize, p.Align); err != nil { if p.AckOut != nil { _ = p.AckOut.WriteFrame(frameErr, encodeErrFrame(index, err.Error())) } diff --git a/util.go b/util.go index dd8b3f9..d2ae735 100644 --- a/util.go +++ b/util.go @@ -2,9 +2,25 @@ package main import ( "fmt" + "os/exec" + "strings" "sync" ) +// runCmd runs name with args, returning trimmed combined output. +func runCmd(name string, args ...string) (string, error) { + out, err := exec.Command(name, args...).CombinedOutput() + return strings.TrimSpace(string(out)), err +} + +// runCmdStdin is runCmd with a fixed stdin string (for tools that prompt). +func runCmdStdin(stdin, name string, args ...string) (string, error) { + c := exec.Command(name, args...) + c.Stdin = strings.NewReader(stdin) + out, err := c.CombinedOutput() + return strings.TrimSpace(string(out)), err +} + // limitedBuffer keeps only the last maxLen bytes written to it — used to // capture a bounded tail of a subprocess's stderr for error messages // without risking unbounded memory growth from a noisy remote command.