DigitalInfinity/clonetool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

clone windows

Browse Source
This commit is contained in:
Alexander Gabriel 2026-09-06 20:15:54 +02:00
parent 20c9017d3b
commit 6405d41c20
21 changed files with 2847 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

142
README.md
View File

@ -5,6 +5,17 @@ two paths on the same machine), driven from a third, passive "manager"
machine. Single static Go binary, no runtime dependencies beyond the
system `ssh` client for remote endpoints.
Two subcommands:
- **`sync`** — mirror one file/device onto another (the original tool; see
most of this README).
- **`clone-disk`** — clone a whole system disk: boot record + partition
table + per-filesystem data, rebuilt on the target then filled from the
source. Same manager/agent/SSH/self-deploy model as `sync`; leans on the
standard disk tools already present on a rescue system (`sfdisk`,
`ntfsclone`, `partclone.*`, `e2image`, `rsync`, …). See
[clone-disk](#clone-disk) below.
## How it works
- You run `clonetool sync` on the **manager**. Source, destination, and
@ -123,6 +134,137 @@ clonetool sync --source \\.\E: --dest \\.\F:
handles the final partial block itself.
- Device size is read with `IOCTL_DISK_GET_LENGTH_INFO`.
## clone-disk
`clonetool clone-disk --source LOC --dest LOC [options]` clones a whole
disk — the boot record, the partition table, and the contents of every
partition — by **rebuilding** the layout on the target and then filling it
from the source. Orchestrated from the manager exactly like `sync`: one
control agent per side, data straight between source and target (push, or
pull fallback), self-deploy, `--sudo` escalation.
It does not reimplement any filesystem knowledge — it calls the standard
tools a rescue system already has and just moves their bytes between the two
machines.
### What it copies
1. **Partition table.** The source agent runs `sfdisk -d` (the canonical
restorable dump for both GPT and MBR). The manager turns it into a
device-independent restore script — dropping the `device:` line and
`last-lba` so it re-sizes for the target, keeping the GPT disk GUID and
every partition GUID/PARTUUID (so existing `fstab` / BCD / GRUB
references still resolve) unless `--new-ids` is given. The target agent
feeds that to `sfdisk` and re-reads the table.
2. **Boot record / boot code.** The 440-byte MBR bootstrap on every disk,
plus — on MBR disks — the whole gap before the first partition (where
GRUB's `core.img` lives), are raw-copied through the same block-diff
engine `sync` uses, addressed as byte windows. On GPT disks the
GPT structures come from `sfdisk`/`sgdisk`; a BIOS-boot partition
(`EF02`) is always cloned raw.
3. **Partition data**, per partition, by the best available method:
- **fs-image** (default when the tool exists): stream a
filesystem-aware image — `ntfsclone` for NTFS, `partclone.<fs>` for
ext*/xfs/btrfs/f2fs/fat/exfat, `e2image` as an ext fallback. Only
used blocks move.
- **raw**: the `sync` block-diff engine over the partition's byte
window. Used for swap, unrecognised filesystems, `--raw N,…`, and
whenever no image tool is installed. Re-runs move only changed blocks.
- **file-level** (`--file-level N,…`): `mkfs` on the target + `rsync`
*(planned; not yet implemented — use `--raw` or an fs-image type)*.
### Target sizing (same rules as `sync`)
- **Target is a device:** never grown. If the source layout fits, it is
reproduced as-is and any trailing space on the target is left untouched.
If the last partition(s) overflow, `--allow-shrink` will shrink them
(filesystem then partition, from the last inward) with `ntfsresize` /
`resize2fs` — this **resizes the source filesystem in place** before
imaging, so it also needs `--yes`. Without `--allow-shrink` an
over-large source is a hard error naming the partition.
- **Target is a file:** created and truncated to just what the layout
needs (or `--image-size SIZE`). Shrinking an existing image prompts
unless `--yes`.
### NTFS / Windows partitions
- **From a Linux rescue system: fully supported.** `ntfsclone` clones
Windows NTFS partitions (incl. Win10/11) at cluster level, copying only
used clusters and preserving every NTFS feature. The volume must be
*clean* — Windows Fast Startup and hibernation leave NTFS dirty; boot
Windows once and shut down fully, or run `ntfsfix -d` first. `ntfsclone`
does not fix booting: keep the partition GUIDs (default) so the existing
BCD resolves, or run `bcdboot` from Windows recovery afterwards. This is
the same approach Clonezilla uses.
- **On Windows itself: raw / VSS only.** There is no `ntfsclone` on
Windows. `clone-disk` reads the partition table via
`IOCTL_DISK_GET_DRIVE_LAYOUT_EX`, reproduces it by raw-copying the
leading sectors (and the backup GPT), and block-clones each partition
as a byte window of `\\.\PhysicalDriveN`. With `--vss` (default) it
takes a Volume Shadow Copy of each NTFS volume first for a
crash-consistent point-in-time source. This is **not** free-space-aware
(a `$Bitmap`-driven skip may come later), and shrinking to a smaller
target is not supported when the source is native Windows — use a
target at least as large, or run the clone from a Linux rescue system
to get `ntfsclone`.
### Bootloader
By default `clone-disk` only *reproduces* boot structures and preserves
disk/partition IDs — enough for a like-for-like replacement disk to boot.
`--reinstall-bootloader` additionally mounts the cloned root (+ ESP),
bind-mounts `/dev /proc /sys`, and runs `grub-install` + `update-grub` /
`grub-mkconfig` in a chroot (Linux targets). It is best-effort and never
fails the clone. For Windows, run `bcdboot C:\Windows /s S: /f ALL` from a
recovery environment.
### Examples
```
# Whole disk to an image file (image is sized to the layout)
clonetool clone-disk --source /dev/sda --dest /srv/sda.img
# Disk to disk on another host, orchestrated from a third machine
clonetool clone-disk --source box1:/dev/nvme0n1 --dest box2:/dev/nvme0n1
# Onto a smaller SSD, shrinking the last (data) partition to fit
clonetool clone-disk --source /dev/sda --dest /dev/sdb --allow-shrink --yes
# Fresh IDs so the clone can sit next to the original
clonetool clone-disk --source /dev/sda --dest /dev/sdb --new-ids
# Force a raw block clone of a partition an image tool would otherwise handle
clonetool clone-disk --source /dev/sda --dest /dev/sdb --raw 2
```
### Tools the agents call (checked at probe time, reported if missing)
`sfdisk`, `sgdisk`, `partprobe`/`blockdev`, `blkid`, `lsblk`, `losetup`,
`ntfsclone`, `ntfsresize`, `ntfsfix`, `partclone.*`, `e2image`,
`resize2fs`, `dumpe2fs`, `e2fsck`, `mkfs.*`, `mount`/`umount`, `rsync`,
`grub-install`, `update-grub`/`grub-mkconfig`. Windows: `vssadmin` /
`wmic`, `bcdboot`, `diskpart`. A missing tool just means the affected
partitions fall back to a raw block copy (or the run stops with a clear
message if that isn't safe, e.g. a partition that must shrink).
### clone-disk options
| Flag | Default | Meaning |
|---|---|---|
| `--parts LIST` | all | Only clone these partition numbers (data only; the table still lists them all). |
| `--raw LIST` | — | Force a raw block clone for these partitions. |
| `--file-level LIST` | — | `mkfs` + `rsync` these partitions (not yet implemented). |
| `--file-level-auto` | off | Use file-level for fs types with no image cloner. |
| `--allow-shrink` | off | Permit shrinking trailing partitions to fit a smaller target (resizes the **source** fs in place; needs `--yes`). |
| `--no-shrink` | off | Never shrink; fail if the target is too small. |
| `--new-ids` | off | Randomize the GPT disk GUID / MBR signature on the target. |
| `--reinstall-bootloader` | off | Run `grub-install` / `grub-mkconfig` on the target after copy (Linux). |
| `--vss` | on | Windows source: take a Volume Shadow Copy per NTFS volume. |
| `--image-size SIZE` | auto | File target: image size (default = enough for the layout). |
`--block-size --job --yes --sudo --deploy --ssh --ssh-opt --remote-bin
--connect-timeout --manager-host` mean the same as for `sync`.
## Usage
```

46
agent.go
View File

@ -16,10 +16,15 @@ import (
func cmdAgent(args []string) error {
fs := flag.NewFlagSet("agent", flag.ContinueOnError)
role := fs.String("role", "", "control|sink|source-stream (internal)")
role := fs.String("role", "", "control|sink|source-stream|fs-send|fs-recv (internal)")
path := fs.String("path", "", "path to read/write")
size := fs.Int64("size", 0, "total sync size in bytes")
base := fs.Int64("base", 0, "byte offset the window starts at (clone-disk boot region / offset partition)")
blockSize := fs.Int64("block-size", defaultBlockSize, "block size in bytes")
fsType := fs.String("fs", "", "filesystem type (fs-send/fs-recv)")
fsTool := fs.String("fstool", "", "fs-image tool family (fs-send/fs-recv)")
peerDisk := fs.String("peerdisk", "", "this helper's whole-disk/image path (fs-send/fs-recv)")
shrinkTo := fs.Int64("shrink", 0, "shrink this fs to N bytes before sending (fs-send)")
if err := fs.Parse(args); err != nil {
return err
}
@ -28,11 +33,17 @@ func cmdAgent(args []string) error {
case "control":
return runControlAgent()
case "sink":
return runSinkRole(*path, *size, *blockSize)
return runSinkRole(*path, *base, *size, *blockSize)
case "source-stream":
return runSourceStreamRole(*path, *size, *blockSize)
return runSourceStreamRole(*path, *base, *size, *blockSize)
case "fs-send":
n, _ := strconv.Atoi(*path)
return runFSSendRole(n, *fsType, *fsTool, *peerDisk, *shrinkTo)
case "fs-recv":
n, _ := strconv.Atoi(*path)
return runFSRecvRole(n, *fsType, *fsTool, *peerDisk)
default:
return fmt.Errorf("agent: unknown or missing --role %q (want control|sink|source-stream)", *role)
return fmt.Errorf("agent: unknown or missing --role %q", *role)
}
}
@ -66,7 +77,16 @@ func runControlAgent() error {
runPushDriver(m, out)
case msgConnectPull:
runPullDriver(m, out)
case msgProbeDisk:
handleProbeDisk(out, m)
case msgBuildLayout:
handleBuildLayout(out, m)
case msgClonePartition:
handleClonePartition(out, m)
case msgReinstallBoot:
handleReinstallBoot(out, m)
case msgClose:
detachAllDisks()
_ = out.WriteJSON(CtrlMsg{Type: msgBye})
return nil
default:
@ -127,6 +147,7 @@ func runPushDriver(req CtrlMsg, out *FrameWriter) {
tailArgs := []string{
"agent", "--role", "sink",
"--path", req.PeerPath,
"--base", strconv.FormatInt(req.PeerBase, 10),
"--size", strconv.FormatInt(req.Size, 10),
"--block-size", strconv.FormatInt(req.BlockSize, 10),
}
@ -314,7 +335,7 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
sendErrCh := make(chan error, 1)
go func() {
sendErrCh <- runSourceLoop(sourceLoopParams{
File: srcFile, Size: req.Size, BlockSize: req.BlockSize, Align: align, Hashes: hashCh, Out: fw,
File: srcFile, Base: req.Base, Size: req.Size, BlockSize: req.BlockSize, Align: align, Hashes: hashCh, Out: fw,
OnSkip: func(uint64) { atomic.AddInt64(&skipped, 1); maybeProgress() },
OnSend: func(idx uint64, _ [32]byte) {
sem <- struct{}{}
@ -400,6 +421,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) {
tailArgs := []string{
"agent", "--role", "source-stream",
"--path", req.PeerPath,
"--base", strconv.FormatInt(req.PeerBase, 10),
"--size", strconv.FormatInt(req.Size, 10),
"--block-size", strconv.FormatInt(req.BlockSize, 10),
}
@ -448,7 +470,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) {
hashErrCh := make(chan error, 1)
go func() {
err := streamHashBlocks(dstFile, req.Size, req.BlockSize, align, func(bh blockHash) error {
err := streamHashBlocks(dstFile, req.Base, req.Size, req.BlockSize, align, func(bh blockHash) error {
return fw.WriteFrame(frameBlockHash, encodeBlockHashFrame(bh.index, bh.hash))
}, scanProgressEmitter(out))
if err == nil {
@ -463,7 +485,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) {
}()
loopErr := runDestLoop(destLoopParams{
File: dstFile, BlockSize: req.BlockSize, Align: align, In: fr,
File: dstFile, Base: req.Base, BlockSize: req.BlockSize, Align: align, In: fr,
OnCtrlMsg: func(m CtrlMsg) {
if m.Type == msgProgress {
_ = out.WriteJSON(m)
@ -496,7 +518,7 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) {
// destination host. Dumb write endpoint: verify+pwrite+ack per block.
// ---------------------------------------------------------------------
func runSinkRole(path string, size, blockSize int64) error {
func runSinkRole(path string, base, size, blockSize int64) error {
out := NewFrameWriter(os.Stdout)
in := NewFrameReader(os.Stdin)
@ -521,7 +543,7 @@ func runSinkRole(path string, size, blockSize int64) error {
// never race on the same region.
hashErrCh := make(chan error, 1)
go func() {
err := streamHashBlocks(f, size, blockSize, align, func(bh blockHash) error {
err := streamHashBlocks(f, base, size, blockSize, align, func(bh blockHash) error {
return out.WriteFrame(frameBlockHash, encodeBlockHashFrame(bh.index, bh.hash))
}, scanProgressEmitter(out))
if err == nil {
@ -533,7 +555,7 @@ func runSinkRole(path string, size, blockSize int64) error {
hashErrCh <- err
}()
loopErr := runDestLoop(destLoopParams{File: f, BlockSize: blockSize, Align: align, In: in, AckOut: out})
loopErr := runDestLoop(destLoopParams{File: f, Base: base, BlockSize: blockSize, Align: align, In: in, AckOut: out})
if hashErr := <-hashErrCh; hashErr != nil {
return hashErr
}
@ -547,7 +569,7 @@ func runSinkRole(path string, size, blockSize int64) error {
// writing straight to its own stdout.
// ---------------------------------------------------------------------
func runSourceStreamRole(path string, size, blockSize int64) error {
func runSourceStreamRole(path string, base, size, blockSize int64) error {
out := NewFrameWriter(os.Stdout)
in := NewFrameReader(os.Stdin)
@ -610,7 +632,7 @@ func runSourceStreamRole(path string, size, blockSize int64) error {
}
loopErr := runSourceLoop(sourceLoopParams{
File: f, Size: size, BlockSize: blockSize, Align: align, Hashes: hashCh, Out: out,
File: f, Base: base, Size: size, BlockSize: blockSize, Align: align, Hashes: hashCh, Out: out,
OnSkip: func(uint64) { skipped++; maybeProgress() },
OnSend: func(uint64, [32]byte) { copied++; maybeProgress() },
})

375
agent_disk.go Normal file
View File

@ -0,0 +1,375 @@
package main
import (
"bufio"
"encoding/json"
"fmt"
"io"
"os"
"os/exec"
"strconv"
"strings"
)
// ----------------------------------------------------------------------------
// clone-disk agent state: loop-device / mount attachments held for the life
// of the control agent so several partition clones can reuse them.
// ----------------------------------------------------------------------------
var diskAttach = map[string]*diskAttachment{}
func attachedNode(diskPath string, num int, writable bool) (string, error) {
a := diskAttach[diskPath]
if a == nil {
var err error
a, err = attachDisk(diskPath, writable)
if err != nil {
return "", err
}
diskAttach[diskPath] = a
}
return partNode(a.blockPath, num), nil
}
func detachAllDisks() {
for k, a := range diskAttach {
if a.cleanup != nil {
a.cleanup()
}
delete(diskAttach, k)
}
}
// ----------------------------------------------------------------------------
// probe_disk
// ----------------------------------------------------------------------------
func handleProbeDisk(out *FrameWriter, m CtrlMsg) {
d, err := probeDiskLayout(m.Path)
if err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: err.Error(), NeedPriv: isPermErr(err)})
return
}
_ = out.WriteJSON(CtrlMsg{Type: msgProbeDiskOK, LayoutJSON: d.JSON()})
}
// ----------------------------------------------------------------------------
// build_layout: create/size an image, write the partition table, re-read it.
// The boot-region bytes and the partition data are moved afterwards by the
// manager through ordinary push/pull transfers.
// ----------------------------------------------------------------------------
type buildOpts struct {
Script string `json:"script"` // sfdisk restore script (Linux)
IsFile bool `json:"isFile"` // target is a regular file, not a device
ImageSize int64 `json:"imageSize"` // truncate the file to this many bytes first
}
func handleBuildLayout(out *FrameWriter, m CtrlMsg) {
var o buildOpts
if err := json.Unmarshal([]byte(m.Options), &o); err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "decode build opts: " + err.Error()})
return
}
if o.IsFile {
f, err := os.OpenFile(m.Path, os.O_CREATE|os.O_RDWR, 0644)
if err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: err.Error(), NeedPriv: isPermErr(err)})
return
}
if err := f.Truncate(o.ImageSize); err != nil {
f.Close()
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("truncate %s: %v", m.Path, err)})
return
}
f.Close()
}
if strings.TrimSpace(o.Script) != "" {
if err := applyPartitionTable(m.Path, o.Script); err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: err.Error(), NeedPriv: isPermErr(err)})
return
}
} else {
rereadPartTable(m.Path)
}
_ = out.WriteJSON(CtrlMsg{Type: msgBuildLayoutOK})
}
// ----------------------------------------------------------------------------
// clone_partition: fs-image and file-level methods. (Raw partitions and boot
// regions go through connect_push / connect_pull instead.)
//
// The message is sent to the SOURCE control agent first (Method set, peer
// fields as for connect_push). On a push failure the manager re-sends it to
// the DEST control agent with m.Reason == "pull".
// ----------------------------------------------------------------------------
func handleClonePartition(out *FrameWriter, m CtrlMsg) {
o, err := decodeFSCloneOpts(m.Options)
if err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "decode clone opts: " + err.Error()})
return
}
switch m.Method {
case "fs-image":
if m.Reason == "pull" {
runFSPullDriver(m, o, out)
} else {
runFSPushDriver(m, o, out)
}
case "file-level":
runFileLevelDriver(m, o, out)
default:
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "unknown clone method " + m.Method})
}
}
// localDevice resolves this side's partition node. m.Path is the whole disk
// (or image); m.PartIndex is the 1-based partition number.
func localDevice(diskPath string, num int, writable bool) (string, error) {
if isDevicePath(diskPath) || looksLikeBlockDevice(diskPath) {
// real disk: node is derived directly, no attach needed
return partNode(diskPath, num), nil
}
return attachedNode(diskPath, num, writable)
}
func looksLikeBlockDevice(p string) bool {
fi, err := os.Stat(p)
return err == nil && fi.Mode()&os.ModeDevice != 0
}
// runFSPushDriver runs in the SOURCE agent: spawn the peer fs-recv (ssh or
// local), run the save tool locally, pipe save-stdout -> peer-stdin.
func runFSPushDriver(m CtrlMsg, o fsCloneOpts, out *FrameWriter) {
dev, err := localDevice(m.Path, m.PartIndex, false)
if err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: "source device: " + err.Error()})
return
}
cl := pickCloner(dev, o.FSType, o.Tool)
if cl == nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "no fs-image cloner for " + o.FSType})
return
}
if o.ShrinkToBytes > 0 {
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Stage: "shrink source " + o.FSType + " p" + strconv.Itoa(m.PartIndex)})
if err := preShrinkFS(dev, o.FSType, o.ShrinkToBytes); err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "pre-shrink: " + err.Error()})
return
}
}
peerTail := []string{"agent", "--role", "fs-recv",
"--path", strconv.Itoa(m.PartIndex), // partition number; peer resolves its own node
"--fs", o.FSType, "--fstool", o.Tool,
"--size", strconv.FormatInt(o.SizeBytes, 10),
"--peerdisk", m.PeerPath,
}
peer := peerAgentCommand(m, peerTail)
peerIn, err := peer.StdinPipe()
if err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: err.Error()})
return
}
peerOut, _ := peer.StdoutPipe()
peerErr := newLimitedBuffer(8192)
peer.Stderr = peerErr
if err := peer.Start(); err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: "start peer: " + err.Error()})
return
}
save := exec.Command(cl.save[0], cl.save[1:]...)
save.Stdout = peerIn
saveErr, _ := save.StderrPipe()
if err := save.Start(); err != nil {
_ = peer.Process.Kill()
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("start %s: %v", cl.save[0], err)})
return
}
var moved int64
go scanCloneProgress(saveErr, o.SizeBytes, func(b int64) {
moved = b
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Stage: o.FSType + " p" + strconv.Itoa(m.PartIndex),
BytesCopied: b, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex})
})
saveWait := make(chan error, 1)
go func() { saveWait <- save.Wait() }()
sErr := <-saveWait
_ = peerIn.Close()
peerLine := readLine(peerOut)
pErr := peer.Wait()
if sErr != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("%s: %v", cl.save[0], sErr)})
return
}
if pErr != nil || !strings.HasPrefix(peerLine, "OK") {
reason := strings.TrimSpace(peerLine + " " + peerErr.String())
if moved == 0 {
_ = out.WriteJSON(CtrlMsg{Type: msgPushFailed, Reason: reason})
return
}
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "peer restore failed: " + reason})
return
}
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, BytesCopied: o.SizeBytes, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex})
_ = out.WriteJSON(CtrlMsg{Type: msgPushOK})
}
// runFSPullDriver runs in the DEST agent: spawn the peer fs-send, run the
// restore tool locally, pipe peer-stdout -> restore-stdin.
func runFSPullDriver(m CtrlMsg, o fsCloneOpts, out *FrameWriter) {
dev, err := localDevice(m.Path, m.PartIndex, true)
if err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: "dest device: " + err.Error()})
return
}
cl := pickCloner(dev, o.FSType, o.Tool)
if cl == nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: "no fs-image cloner for " + o.FSType})
return
}
peerTail := []string{"agent", "--role", "fs-send",
"--path", strconv.Itoa(m.PartIndex),
"--fs", o.FSType, "--fstool", o.Tool,
"--peerdisk", m.PeerPath,
"--shrink", strconv.FormatInt(o.ShrinkToBytes, 10),
}
peer := peerAgentCommand(m, peerTail)
peerOut, err := peer.StdoutPipe()
if err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: err.Error()})
return
}
peerErr := newLimitedBuffer(8192)
peer.Stderr = peerErr
if err := peer.Start(); err != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: "start peer: " + err.Error()})
return
}
restore := exec.Command(cl.restore[0], cl.restore[1:]...)
restore.Stdin = peerOut
restoreErr, _ := restore.StderrPipe()
if err := restore.Start(); err != nil {
_ = peer.Process.Kill()
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("start %s: %v", cl.restore[0], err)})
return
}
go scanCloneProgress(restoreErr, o.SizeBytes, func(b int64) {
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Stage: o.FSType + " p" + strconv.Itoa(m.PartIndex),
BytesCopied: b, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex})
})
rErr := restore.Wait()
pErr := peer.Wait()
if pErr != nil {
if rErr != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgPullFailed, Reason: strings.TrimSpace(peerErr.String())})
return
}
}
if rErr != nil {
_ = out.WriteJSON(CtrlMsg{Type: msgError, Message: fmt.Sprintf("%s: %v", cl.restore[0], rErr)})
return
}
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, BytesCopied: o.SizeBytes, BytesTotal: o.SizeBytes, PartIndex: m.PartIndex})
_ = out.WriteJSON(CtrlMsg{Type: msgPullOK})
}
// ----------------------------------------------------------------------------
// fs-send / fs-recv: one-shot stdio helpers spawned on the peer.
// ----------------------------------------------------------------------------
func runFSSendRole(partNum int, fsType, tool, peerDisk string, shrinkTo int64) error {
dev, err := resolveHelperDevice(peerDisk, partNum, shrinkTo > 0)
if err != nil {
return err
}
cl := pickCloner(dev, fsType, tool)
if cl == nil {
return fmt.Errorf("no fs-image cloner for %s", fsType)
}
if shrinkTo > 0 {
if err := preShrinkFS(dev, fsType, shrinkTo); err != nil {
return fmt.Errorf("pre-shrink: %w", err)
}
}
cmd := exec.Command(cl.save[0], cl.save[1:]...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd.Run()
}
func runFSRecvRole(partNum int, fsType, tool, peerDisk string) error {
dev, err := resolveHelperDevice(peerDisk, partNum, true)
if err != nil {
fmt.Fprintf(os.Stderr, "ERR: %v\n", err)
return err
}
cl := pickCloner(dev, fsType, tool)
if cl == nil {
err := fmt.Errorf("no fs-image cloner for %s", fsType)
fmt.Fprintf(os.Stdout, "ERR: %v\n", err)
return err
}
cmd := exec.Command(cl.restore[0], cl.restore[1:]...)
cmd.Stdin = os.Stdin
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
fmt.Fprintf(os.Stdout, "ERR: %v\n", err)
return err
}
fmt.Fprintln(os.Stdout, "OK")
return nil
}
// resolveHelperDevice is used by the one-shot fs-send/fs-recv helpers, which
// have no persistent attachment map of their own.
func resolveHelperDevice(disk string, num int, writable bool) (string, error) {
if isDevicePath(disk) || looksLikeBlockDevice(disk) {
return partNode(disk, num), nil
}
a, err := attachDisk(disk, writable)
if err != nil {
return "", err
}
// deliberately leak the loop device until process exit; a one-shot
// helper is short-lived and the parent detaches on the control side.
_ = a
return partNode(a.blockPath, num), nil
}
// ----------------------------------------------------------------------------
// file-level: mkfs on the target partition + rsync the tree across.
// ----------------------------------------------------------------------------
func runFileLevelDriver(m CtrlMsg, o fsCloneOpts, out *FrameWriter) {
_ = out.WriteJSON(CtrlMsg{Type: msgError,
Message: "file-level clone (--file-level) is not implemented yet; use --raw or an fs-image type"})
}
// ----------------------------------------------------------------------------
// reinstall_boot: best-effort, never fatal to the clone.
// ----------------------------------------------------------------------------
func handleReinstallBoot(out *FrameWriter, m CtrlMsg) {
msg := reinstallBootloader(m.Path, m.Options)
_ = out.WriteJSON(CtrlMsg{Type: msgReinstallBootOK, Message: msg})
}
// readLine reads one \n-terminated line (used for the fs-recv status line).
func readLine(r io.Reader) string {
if r == nil {
return ""
}
br := bufio.NewReader(r)
s, _ := br.ReadString('\n')
return strings.TrimSpace(s)
}

84
boot_linux.go Normal file
View File

@ -0,0 +1,84 @@
//go:build linux
package main
import (
"encoding/json"
"fmt"
"os"
"os/exec"
"strings"
)
// reinstallBootloader mounts the freshly cloned root (and ESP), bind-mounts
// the API filesystems, and runs grub-install + grub-mkconfig inside a chroot.
// It is best-effort: it returns a human summary and never a hard error, so a
// clone still completes even when boot repair cannot.
func reinstallBootloader(diskPath, optsJSON string) string {
var o bootOpts
if err := json.Unmarshal([]byte(optsJSON), &o); err != nil {
return "skipped: bad boot opts: " + err.Error()
}
if o.RootPart == 0 {
return "skipped: no root partition identified (pass --reinstall-bootloader only for a Linux system disk)"
}
att, err := attachDisk(diskPath, true)
if err != nil {
return "skipped: attach: " + err.Error()
}
defer att.cleanup()
rootNode := partNode(att.blockPath, o.RootPart)
root, umount, err := mountAt(rootNode, false)
if err != nil {
return "skipped: mount root: " + err.Error()
}
defer umount()
if o.ESPPart != 0 {
espNode := partNode(att.blockPath, o.ESPPart)
espDir := root + "/boot/efi"
_ = os.MkdirAll(espDir, 0755)
if out, err := exec.Command("mount", espNode, espDir).CombinedOutput(); err != nil {
return "skipped: mount ESP: " + strings.TrimSpace(string(out))
}
defer exec.Command("umount", espDir).Run()
}
var log strings.Builder
for _, d := range []string{"dev", "proc", "sys", "run"} {
_ = exec.Command("mount", "--bind", "/"+d, root+"/"+d).Run()
defer exec.Command("umount", "-l", root+"/"+d).Run()
}
grubTarget := att.blockPath
if isDevicePath(diskPath) || looksLikeBlockDevice(diskPath) {
grubTarget = diskPath
}
giArgs := []string{root, "grub-install", "--recheck"}
if o.UEFI {
giArgs = append(giArgs, "--target=x86_64-efi", "--efi-directory=/boot/efi", "--removable")
} else {
giArgs = append(giArgs, grubTarget)
}
if out, err := exec.Command("chroot", giArgs...).CombinedOutput(); err != nil {
fmt.Fprintf(&log, "grub-install failed: %s; ", strings.TrimSpace(string(out)))
} else {
log.WriteString("grub-install ok; ")
}
mkcfg := "grub-mkconfig"
if _, err := exec.LookPath("update-grub"); err == nil {
mkcfg = "update-grub"
}
args := []string{root, mkcfg}
if mkcfg == "grub-mkconfig" {
args = append(args, "-o", "/boot/grub/grub.cfg")
}
if out, err := exec.Command("chroot", args...).CombinedOutput(); err != nil {
fmt.Fprintf(&log, "%s failed: %s", mkcfg, strings.TrimSpace(string(out)))
} else {
log.WriteString(mkcfg + " ok")
}
return log.String()
}

13
boot_other.go Normal file
View File

@ -0,0 +1,13 @@
//go:build !linux
package main
// reinstallBootloader is Linux-only for now. On Windows the recommended path
// is to preserve the disk signature / partition GUIDs (clone-disk does this by
// raw-copying the leading sectors) so the existing BCD store still resolves,
// and otherwise run `bcdboot W:\Windows /s S: /f ALL` from a WinPE/recovery
// environment by hand.
func reinstallBootloader(diskPath, optsJSON string) string {
return "skipped: --reinstall-bootloader is only implemented on Linux; " +
"run bcdboot from Windows recovery, or grub-install from a Linux rescue system"
}

261
clonedisk.go Normal file
View File

@ -0,0 +1,261 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"os"
"sort"
"strings"
)
// CloneDiskConfig is everything `clonetool clone-disk` needs. It embeds
// SyncConfig for the transport/deploy/ssh knobs shared with `sync`.
type CloneDiskConfig struct {
SyncConfig
Parts []int
Raw []int
FileLevel []int
FileAuto bool
AllowShrink bool
NoShrink bool
NewIDs bool
ReinstallBoot bool
VSS bool
ImageSize int64
}
func runCloneDisk(cfg CloneDiskConfig) error {
srcSpec, err := parseSpec(cfg.Source)
if err != nil {
return fmt.Errorf("--source: %w", err)
}
dstSpec, err := parseSpec(cfg.Dest)
if err != nil {
return fmt.Errorf("--dest: %w", err)
}
if err := checkNotSame(srcSpec, dstSpec); err != nil {
return err
}
srcRemoteBin, dstRemoteBin := cfg.RemoteBin, cfg.RemoteBin
if !srcSpec.IsLocal() {
if srcRemoteBin, err = resolveRemoteBin(&cfg.SyncConfig, srcSpec, "source"); err != nil {
return err
}
}
if !dstSpec.IsLocal() {
if !srcSpec.IsLocal() && sameHost(srcSpec, dstSpec) {
dstRemoteBin = srcRemoteBin
} else if dstRemoteBin, err = resolveRemoteBin(&cfg.SyncConfig, dstSpec, "dest"); err != nil {
return err
}
}
// --- bring up both control agents -------------------------------------
srcCtrl, srcLayout, srcSudo, err := bringUpProbe(srcSpec, "source", &cfg.SyncConfig, srcRemoteBin)
if err != nil {
return fmt.Errorf("source: %w", err)
}
defer srcCtrl.Close()
dstCtrl, dstInfo, dstSudo, err := bringUpController(dstSpec, "dest", &cfg.SyncConfig, dstRemoteBin, dstSpec.Path)
if err != nil {
return fmt.Errorf("dest: %w", err)
}
defer dstCtrl.Close()
isFile := !(dstInfo.Exists && dstInfo.IsDevice)
destSize := dstInfo.Size
fmt.Fprintf(os.Stderr, "source %s: %s, %s, %d partitions\n",
srcSpec, srcLayout.Scheme, humanBytes(srcLayout.DiskSize), len(srcLayout.Partitions))
reportMissingTools(srcLayout)
// --- plan the target -------------------------------------------------
plan, err := planTargetLayout(srcLayout, isFile, destSize, planOpts{
Parts: intSet(cfg.Parts),
Raw: intSet(cfg.Raw),
FileLevel: intSet(cfg.FileLevel),
FileAuto: cfg.FileAuto,
AllowShrink: cfg.AllowShrink && !cfg.NoShrink,
NewIDs: cfg.NewIDs,
ImageSize: cfg.ImageSize,
})
if err != nil {
return err
}
for _, n := range plan.Notes {
fmt.Fprintf(os.Stderr, " %s\n", n)
}
shrinks := false
for _, pp := range plan.Parts {
if pp.ShrinkToB > 0 {
shrinks = true
}
}
if shrinks && !cfg.Yes {
return fmt.Errorf("shrinking resizes the SOURCE filesystem(s) in place before imaging; " +
"re-run with --yes to confirm, or use a larger target")
}
if isFile && !cfg.Yes && dstInfo.Exists && dstInfo.Size > plan.ImageSize {
if !confirmShrink(dstSpec, dstInfo.Size, plan.ImageSize) {
return fmt.Errorf("aborted")
}
}
// --- rebuild the partition table on the target ---------------------
bo := buildOpts{Script: plan.Script, IsFile: isFile, ImageSize: plan.ImageSize}
boJSON, _ := json.Marshal(bo)
fmt.Fprintf(os.Stderr, "rebuilding partition table on %s ...\n", dstSpec)
if err := dstCtrl.BuildLayout(dstSpec.Path, string(boJSON)); err != nil {
return fmt.Errorf("build layout: %w", err)
}
bothLocal := srcSpec.IsLocal() && dstSpec.IsLocal()
srcHost, srcUser := resolveConnectHost(srcSpec, &cfg.SyncConfig)
dstHost, dstUser := resolveConnectHost(dstSpec, &cfg.SyncConfig)
base := func(s Spec, remoteBin string) CtrlMsg {
return CtrlMsg{
BlockSize: cfg.BlockSize, SSHBin: cfg.SSHBin, SSHOpts: cfg.SSHOpts,
ConnectTimeoutSec: cfg.ConnectTimeoutSec, PeerLocal: bothLocal,
}
}
_ = base
// --- copy the boot regions verbatim (offset windows) ---------------
for _, r := range plan.BootRegions {
if r.Length <= 0 {
continue
}
fmt.Fprintf(os.Stderr, "boot region @%d (%s) %s ...\n", r.Offset, humanBytes(r.Length), r.Note)
if err := runWindowTransfer(srcCtrl, dstCtrl, windowJob{
srcPath: srcSpec.Path, dstPath: dstSpec.Path,
srcBase: r.Offset, dstBase: r.Offset, size: r.Length,
bothLocal: bothLocal, blockSize: cfg.BlockSize, cfg: &cfg.SyncConfig,
srcHost: srcHost, srcUser: srcUser, dstHost: dstHost, dstUser: dstUser,
srcRemoteBin: srcRemoteBin, dstRemoteBin: dstRemoteBin,
srcSudo: srcSudo, dstSudo: dstSudo,
}); err != nil {
return fmt.Errorf("boot region @%d: %w", r.Offset, err)
}
}
// --- per-partition data ------------------------------------------------
for i, pp := range plan.Parts {
tag := fmt.Sprintf("partition %d/%d (p%d %s, %s)", i+1, len(plan.Parts), pp.Num, orDash(pp.FSType), pp.Method)
fmt.Fprintf(os.Stderr, "%s ...\n", tag)
switch pp.Method {
case "raw":
sz := pp.SrcSizeB
if pp.DstSizeB < sz {
sz = pp.DstSizeB
}
if err := runWindowTransfer(srcCtrl, dstCtrl, windowJob{
srcPath: srcSpec.Path, dstPath: dstSpec.Path,
srcBase: pp.SrcStartB, dstBase: pp.DstStartB, size: sz,
bothLocal: bothLocal, blockSize: cfg.BlockSize, cfg: &cfg.SyncConfig,
srcHost: srcHost, srcUser: srcUser, dstHost: dstHost, dstUser: dstUser,
srcRemoteBin: srcRemoteBin, dstRemoteBin: dstRemoteBin,
srcSudo: srcSudo, dstSudo: dstSudo,
}); err != nil {
return fmt.Errorf("%s: %w", tag, err)
}
case "fs-image":
if err := runFSImagePartition(srcCtrl, dstCtrl, pp, cloneCtx{
srcPath: srcSpec.Path, dstPath: dstSpec.Path, bothLocal: bothLocal,
cfg: &cfg.SyncConfig, srcHost: srcHost, srcUser: srcUser,
dstHost: dstHost, dstUser: dstUser,
srcRemoteBin: srcRemoteBin, dstRemoteBin: dstRemoteBin,
srcSudo: srcSudo, dstSudo: dstSudo,
}); err != nil {
return fmt.Errorf("%s: %w", tag, err)
}
case "file-level":
return fmt.Errorf("%s: file-level clone is not implemented yet", tag)
}
}
// --- bootloader (opt-in) --------------------------------------------
if cfg.ReinstallBoot {
bootJSON, _ := json.Marshal(bootOpts{
RootPart: plan.RootPart, ESPPart: plan.ESPPart, UEFI: plan.UEFI, DiskPath: dstSpec.Path,
})
fmt.Fprintf(os.Stderr, "reinstalling bootloader on %s ...\n", dstSpec)
summary, err := dstCtrl.ReinstallBoot(dstSpec.Path, string(bootJSON))
if err != nil {
fmt.Fprintf(os.Stderr, " bootloader: %v\n", err)
} else {
fmt.Fprintf(os.Stderr, " bootloader: %s\n", summary)
}
}
label := cfg.Job
if label == "" {
label = fmt.Sprintf("%s -> %s", srcSpec, dstSpec)
}
fmt.Fprintf(os.Stderr, "done: %s\n", label)
return nil
}
// bringUpProbe starts a source control agent and runs probe_disk, retrying
// once under sudo on a permission error (mirrors bringUpController).
func bringUpProbe(spec Spec, tag string, cfg *SyncConfig, remoteBin string) (*Controller, *DiskLayout, bool, error) {
sudo := cfg.Sudo == "always" && canElevate()
c, err := startController(spec, tag, cfg, remoteBin, sudo)
if err != nil {
return nil, nil, sudo, err
}
lay, err := c.ProbeDisk(spec.Path)
if err == nil {
return c, lay, sudo, nil
}
if errors.Is(err, errNeedPriv) && cfg.Sudo == "auto" && !sudo && canElevate() {
fmt.Fprintf(os.Stderr, "%s: permission denied on %s; retrying via sudo ...\n", tag, spec.Path)
c.Close()
sudo = true
if c, err = startController(spec, tag, cfg, remoteBin, true); err != nil {
return nil, nil, sudo, err
}
if lay, err = c.ProbeDisk(spec.Path); err == nil {
return c, lay, sudo, nil
}
}
c.Close()
return nil, nil, sudo, err
}
func reportMissingTools(d *DiskLayout) {
want := []string{"sfdisk", "ntfsclone", "partclone.extfs", "partclone.restore", "e2image"}
var miss []string
for _, t := range want {
if !d.Tools[t] {
miss = append(miss, t)
}
}
if len(miss) > 0 {
fmt.Fprintf(os.Stderr, " note: not present on source: %s (affected partitions fall back to raw block copy)\n",
strings.Join(miss, ", "))
}
}
func orDash(s string) string {
if s == "" {
return "-"
}
return s
}
func intSet(xs []int) map[int]bool {
m := make(map[int]bool, len(xs))
for _, x := range xs {
m[x] = true
}
return m
}
func sortedParts(ps []Partition) []Partition {
out := append([]Partition(nil), ps...)
sort.Slice(out, func(i, j int) bool { return out[i].Start < out[j].Start })
return out
}

117
clonedrive.go Normal file
View File

@ -0,0 +1,117 @@
package main
import (
"fmt"
"os"
)
// windowJob raw-copies a byte window [srcBase,srcBase+size) of the source
// disk onto [dstBase,dstBase+size) of the target, through the same
// push/pull block-diff engine `sync` uses. Used for boot regions and for
// partitions cloned with method "raw".
type windowJob struct {
srcPath, dstPath string
srcBase, dstBase int64
size int64
bothLocal bool
blockSize int64
cfg *SyncConfig
srcHost, srcUser string
dstHost, dstUser string
srcRemoteBin string
dstRemoteBin string
srcSudo, dstSudo bool
}
func runWindowTransfer(srcCtrl, dstCtrl *Controller, j windowJob) error {
pp := newProgressPrinter(j.blockSize)
cb := transferCallbacks{onProgress: pp.print}
pushReq := CtrlMsg{
Path: j.srcPath, Base: j.srcBase, PeerBase: j.dstBase, Size: j.size, BlockSize: j.blockSize,
PeerHost: j.dstHost, PeerUser: j.dstUser, PeerPath: j.dstPath, PeerLocal: j.bothLocal,
RemoteBin: j.dstRemoteBin, SSHBin: j.cfg.SSHBin, SSHOpts: j.cfg.SSHOpts,
ConnectTimeoutSec: j.cfg.ConnectTimeoutSec, Sudo: j.dstSudo,
}
ok, reason, err := srcCtrl.ConnectPush(pushReq, cb)
if err != nil {
pp.finish()
return err
}
if !ok {
pp.finish()
fmt.Fprintf(os.Stderr, " push not possible (%s); pulling instead ...\n", reason)
pullReq := CtrlMsg{
Path: j.dstPath, Base: j.dstBase, PeerBase: j.srcBase, Size: j.size, BlockSize: j.blockSize,
PeerHost: j.srcHost, PeerUser: j.srcUser, PeerPath: j.srcPath, PeerLocal: j.bothLocal,
RemoteBin: j.srcRemoteBin, SSHBin: j.cfg.SSHBin, SSHOpts: j.cfg.SSHOpts,
ConnectTimeoutSec: j.cfg.ConnectTimeoutSec, Sudo: j.srcSudo,
}
ok2, reason2, err2 := dstCtrl.ConnectPull(pullReq, cb)
if err2 != nil {
pp.finish()
return err2
}
if !ok2 {
pp.finish()
return fmt.Errorf("no direct connection either way (push: %s; pull: %s)", reason, reason2)
}
}
pp.finish()
return nil
}
// cloneCtx carries the transport context for an fs-image partition clone.
type cloneCtx struct {
srcPath, dstPath string
bothLocal bool
cfg *SyncConfig
srcHost, srcUser string
dstHost, dstUser string
srcRemoteBin string
dstRemoteBin string
srcSudo, dstSudo bool
}
// runFSImagePartition streams a filesystem image for one partition, trying
// push (source drives) then pull (dest drives).
func runFSImagePartition(srcCtrl, dstCtrl *Controller, pp plannedPart, c cloneCtx) error {
opts := fsCloneOpts{FSType: pp.FSType, Tool: pp.Tool, SizeBytes: pp.SrcSizeB, ShrinkToBytes: pp.ShrinkToB}
prog := newProgressPrinter(1 << 20)
cb := transferCallbacks{onProgress: prog.printBytes}
pushReq := CtrlMsg{
Method: "fs-image", PartIndex: pp.Num, Options: opts.encode(),
Path: c.srcPath, PeerPath: c.dstPath, PeerLocal: c.bothLocal,
PeerHost: c.dstHost, PeerUser: c.dstUser,
RemoteBin: c.dstRemoteBin, SSHBin: c.cfg.SSHBin, SSHOpts: c.cfg.SSHOpts,
ConnectTimeoutSec: c.cfg.ConnectTimeoutSec, Sudo: c.dstSudo,
}
ok, reason, err := srcCtrl.ClonePartition(pushReq, cb)
if err != nil {
prog.finish()
return err
}
if !ok {
prog.finish()
fmt.Fprintf(os.Stderr, " fs-image push not possible (%s); pulling instead ...\n", reason)
pullReq := CtrlMsg{
Method: "fs-image", Reason: "pull", PartIndex: pp.Num, Options: opts.encode(),
Path: c.dstPath, PeerPath: c.srcPath, PeerLocal: c.bothLocal,
PeerHost: c.srcHost, PeerUser: c.srcUser,
RemoteBin: c.srcRemoteBin, SSHBin: c.cfg.SSHBin, SSHOpts: c.cfg.SSHOpts,
ConnectTimeoutSec: c.cfg.ConnectTimeoutSec, Sudo: c.srcSudo,
}
ok2, reason2, err2 := dstCtrl.ClonePartition(pullReq, cb)
if err2 != nil {
prog.finish()
return err2
}
if !ok2 {
prog.finish()
return fmt.Errorf("fs-image failed both ways (push: %s; pull: %s)", reason, reason2)
}
}
prog.finish()
return nil
}

63
control.go
View File

@ -135,6 +135,69 @@ func (c *Controller) agentErr(op, path string, resp CtrlMsg) error {
return fmt.Errorf("%s: %s %s: %s", c.tag, op, path, resp.Message)
}
// ProbeDisk asks the agent to enumerate a whole disk (partition table, boot
// regions, per-partition filesystem facts, tool availability).
func (c *Controller) ProbeDisk(path string) (*DiskLayout, error) {
resp, err := c.call(CtrlMsg{Type: msgProbeDisk, Path: path})
if err != nil {
return nil, err
}
switch resp.Type {
case msgProbeDiskOK:
return parseDiskLayoutJSON(resp.LayoutJSON)
case msgError:
return nil, c.agentErr("probe", path, resp)
default:
return nil, fmt.Errorf("%s: unexpected response %q to probe_disk", c.tag, resp.Type)
}
}
// BuildLayout asks the agent to create/size an image and write the partition
// table (optsJSON is a buildOpts blob).
func (c *Controller) BuildLayout(path, optsJSON string) error {
resp, err := c.call(CtrlMsg{Type: msgBuildLayout, Path: path, Options: optsJSON})
if err != nil {
return err
}
switch resp.Type {
case msgBuildLayoutOK:
return nil
case msgError:
return c.agentErr("build_layout", path, resp)
default:
return fmt.Errorf("%s: unexpected response %q to build_layout", c.tag, resp.Type)
}
}
// ClonePartition drives one fs-image or file-level partition clone. dir is
// "push" (send to source agent) or "pull" (send to dest agent).
func (c *Controller) ClonePartition(req CtrlMsg, cb transferCallbacks) (ok bool, reason string, err error) {
req.Type = msgClonePartition
okType, failType := msgPushOK, msgPushFailed
if req.Reason == "pull" {
okType, failType = msgPullOK, msgPullFailed
}
return c.connectAndPump(req, cb, okType, failType)
}
// ReinstallBoot asks the (dest) agent to run grub-install/bcdboot; the
// returned string is a human summary. Best-effort — never returns an error
// for a boot-repair failure, only for a transport failure.
func (c *Controller) ReinstallBoot(path, optsJSON string) (string, error) {
resp, err := c.call(CtrlMsg{Type: msgReinstallBoot, Path: path, Options: optsJSON})
if err != nil {
return "", err
}
switch resp.Type {
case msgReinstallBootOK:
return resp.Message, nil
case msgError:
return "", c.agentErr("reinstall_boot", path, resp)
default:
return "", fmt.Errorf("%s: unexpected response %q to reinstall_boot", c.tag, resp.Type)
}
}
// transferCallbacks receives streaming updates while a connect_push or
// connect_pull is in flight.
type transferCallbacks struct {

32
ctrlmsg.go
View File

@ -21,6 +21,15 @@ type CtrlMsg struct {
PeerUser string `json:"peerUser,omitempty"`
PeerPort int `json:"peerPort,omitempty"`
PeerPath string `json:"peerPath,omitempty"`
// Base/Length restrict a push/pull transfer to a byte window of the
// path (offset Base, Length bytes). Used by clone-disk to raw-copy a
// boot region or an offset-addressed partition through the same engine.
// Size stays the *window* length for the loop math; Base shifts every
// block offset. Zero Base + Length==Size is the whole-file case that
// `sync` uses unchanged.
Base int64 `json:"base,omitempty"`
PeerBase int64 `json:"peerBase,omitempty"`
Length int64 `json:"length,omitempty"`
// PeerLocal is set when both source and dest are local to the manager,
// so the driver (itself already a local child of the manager) can spawn
// the sink/source-stream helper as a plain local subprocess instead of
@ -48,6 +57,20 @@ type CtrlMsg struct {
Skipped int64 `json:"skipped,omitempty"`
TotalBlocks int64 `json:"totalBlocks,omitempty"`
BytesCopied int64 `json:"bytesCopied,omitempty"`
BytesTotal int64 `json:"bytesTotal,omitempty"`
// clone-disk staging: which partition/step of the whole-disk job the
// progress or status refers to. Stage is a short human label
// ("partition table", "boot region", "ntfs p2", "bootloader").
Stage string `json:"stage,omitempty"`
PartIndex int `json:"partIndex,omitempty"`
PartTotal int `json:"partTotal,omitempty"`
// clone-disk payloads: probe result / requested target layout, both
// carried as embedded JSON so CtrlMsg stays a flat envelope.
LayoutJSON string `json:"layoutJSON,omitempty"`
Method string `json:"method,omitempty"` // partition clone method: "fs-image" | "raw" | "file-level"
Options string `json:"options,omitempty"` // JSON of the build/clone options block
// log
Level string `json:"level,omitempty"`
@ -69,4 +92,13 @@ const (
msgError = "error"
msgClose = "close"
msgBye = "bye"
// clone-disk control verbs
msgProbeDisk = "probe_disk"
msgProbeDiskOK = "probe_disk_ok"
msgBuildLayout = "build_layout"
msgBuildLayoutOK = "build_layout_ok"
msgClonePartition = "clone_partition"
msgReinstallBoot = "reinstall_boot"
msgReinstallBootOK = "reinstall_boot_ok"
)

322
disk_linux.go Normal file
View File

@ -0,0 +1,322 @@
//go:build linux
package main
import (
"encoding/base64"
"fmt"
"os"
"os/exec"
"strconv"
"strings"
"unsafe"
"syscall"
)
// BLKSSZGET returns the logical (soft) sector size of a block device.
const blkSSZGet = 0x1268
func logicalSectorSize(path string) (int64, error) {
fi, err := os.Stat(path)
if err == nil && fi.Mode().IsRegular() {
return 512, nil
}
f, err := os.Open(path)
if err != nil {
return 512, err
}
defer f.Close()
var ssz int32
_, _, errno := syscall.Syscall(syscall.SYS_IOCTL, f.Fd(), blkSSZGet, uintptr(unsafe.Pointer(&ssz)))
if errno != 0 || ssz <= 0 {
return 512, nil
}
return int64(ssz), nil
}
// diskAttachment is a probe-time view of a whole disk: the block path whose
// partitions can be opened (the disk itself for a real device, a loop device
// for an image file) plus a cleanup to release a loop device.
type diskAttachment struct {
blockPath string
cleanup func()
}
// attachDisk returns a block path whose partition nodes exist. For a regular
// file it sets up a partition-scanning loop device; for a real block device
// it is a no-op.
func attachDisk(path string, writable bool) (*diskAttachment, error) {
fi, err := os.Stat(path)
if err != nil {
return nil, err
}
if fi.Mode()&os.ModeDevice != 0 {
return &diskAttachment{blockPath: path, cleanup: func() {}}, nil
}
args := []string{"--find", "--show", "--partscan"}
if !writable {
args = append(args, "--read-only")
}
args = append(args, path)
out, err := exec.Command("losetup", args...).Output()
if err != nil {
return nil, fmt.Errorf("losetup %s: %w", path, cmdErr(err))
}
lo := strings.TrimSpace(string(out))
return &diskAttachment{
blockPath: lo,
cleanup: func() { _ = exec.Command("losetup", "--detach", lo).Run() },
}, nil
}
// partNode maps a whole-disk block path + 1-based partition number to its
// partition device node (sda -> sda1, nvme0n1 -> nvme0n1p1, loop0 -> loop0p1).
func partNode(disk string, num int) string {
if l := len(disk); l > 0 && disk[l-1] >= '0' && disk[l-1] <= '9' {
return fmt.Sprintf("%sp%d", disk, num)
}
return fmt.Sprintf("%s%d", disk, num)
}
var diskTools = []string{
"sfdisk", "sgdisk", "partprobe", "blockdev", "blkid", "lsblk", "losetup",
"ntfsclone", "ntfsresize", "ntfsfix", "mkntfs",
"e2image", "resize2fs", "dumpe2fs", "mkfs.ext4", "e2fsck",
"partclone.dd", "partclone.extfs", "partclone.ntfs", "partclone.fat",
"partclone.exfat", "partclone.xfs", "partclone.btrfs", "partclone.restore",
"xfs_copy", "mkfs.vfat", "mkfs.xfs", "mkswap", "rsync",
"mount", "umount", "grub-install", "update-grub", "grub-mkconfig",
}
func probeToolset() map[string]bool {
m := make(map[string]bool, len(diskTools))
for _, t := range diskTools {
if _, err := exec.LookPath(t); err == nil {
m[t] = true
}
}
return m
}
// probeDiskLayout gathers everything clone-disk needs from a source disk.
func probeDiskLayout(path string) (*DiskLayout, error) {
d := &DiskLayout{DiskPath: path, Tools: probeToolset()}
info, err := statPath(path)
if err != nil {
return nil, err
}
if !info.Exists {
return nil, fmt.Errorf("%s does not exist", path)
}
d.DiskSize = info.Size
if d.LogicalSector, err = logicalSectorSize(path); err != nil {
d.LogicalSector = 512
}
d.PhysicalSector = d.LogicalSector
// Partition table via sfdisk -d (works on a file directly).
dump, derr := exec.Command("sfdisk", "-d", path).Output()
if derr != nil {
d.Scheme = "raw"
d.Warnings = append(d.Warnings, fmt.Sprintf("no partition table (%v)", cmdErr(derr)))
return d, nil
}
if err := d.parseSfdiskDump(string(dump)); err != nil {
return nil, err
}
if d.LogicalSector == 0 {
d.LogicalSector = 512
}
if d.Scheme == "gpt" {
if b, err := exec.Command("sgdisk", "--backup=/dev/stdout", path).Output(); err == nil {
d.SgdiskB64 = base64.StdEncoding.EncodeToString(b)
}
}
// Attach so per-partition nodes exist, then enrich each row.
att, err := attachDisk(path, false)
if err != nil {
d.Warnings = append(d.Warnings, fmt.Sprintf("cannot attach for fs probe: %v", err))
} else {
defer att.cleanup()
for i := range d.Partitions {
enrichPartition(&d.Partitions[i], att.blockPath, d.LogicalSector)
}
}
d.BootRegions = computeBootRegions(d)
return d, nil
}
// enrichPartition fills the FS* fields of p using blkid + fs-specific probes.
func enrichPartition(p *Partition, disk string, sector int64) {
node := partNode(disk, p.Num)
p.DevPath = node
if _, err := os.Stat(node); err != nil {
return
}
if out, err := exec.Command("blkid", "-o", "export", node).Output(); err == nil {
for _, line := range strings.Split(string(out), "\n") {
k, v, ok := splitEq(strings.TrimSpace(line))
if !ok {
continue
}
switch k {
case "TYPE":
p.FSType = v
case "LABEL":
p.FSLabel = v
case "UUID":
p.FSUUID = v
}
}
}
switch p.FSType {
case "ext2", "ext3", "ext4":
p.FSUsedBytes, p.FSMinBytes = probeExtSizes(node)
case "ntfs":
p.FSUsedBytes, p.FSMinBytes = probeNTFSSizes(node)
}
}
func probeExtSizes(node string) (used, min int64) {
if out, err := exec.Command("dumpe2fs", "-h", node).Output(); err == nil {
var bs, count, free int64
for _, line := range strings.Split(string(out), "\n") {
f := strings.SplitN(line, ":", 2)
if len(f) != 2 {
continue
}
v := strings.TrimSpace(f[1])
switch strings.TrimSpace(f[0]) {
case "Block size":
bs, _ = strconv.ParseInt(v, 10, 64)
case "Block count":
count, _ = strconv.ParseInt(v, 10, 64)
case "Free blocks":
free, _ = strconv.ParseInt(v, 10, 64)
}
}
if bs > 0 && count > 0 {
used = (count - free) * bs
}
}
// resize2fs -P needs a clean fs; ignore failures.
if out, err := exec.Command("resize2fs", "-P", node).Output(); err == nil {
// "Estimated minimum size of the filesystem: 123456" (in 4k blocks)
s := string(out)
if i := strings.LastIndex(s, ":"); i >= 0 {
if n, err := strconv.ParseInt(strings.TrimSpace(s[i+1:]), 10, 64); err == nil {
min = n * 4096
}
}
}
return used, min
}
func probeNTFSSizes(node string) (used, min int64) {
out, err := exec.Command("ntfsresize", "--info", "--force", node).CombinedOutput()
if err != nil {
return 0, 0
}
for _, line := range strings.Split(string(out), "\n") {
line = strings.TrimSpace(line)
switch {
case strings.HasPrefix(line, "You might resize at "):
// "You might resize at 1234567890 bytes ..."
fields := strings.Fields(line)
if len(fields) >= 5 {
if n, err := strconv.ParseInt(fields[4], 10, 64); err == nil {
min = n
}
}
case strings.Contains(line, "Space in use") && strings.Contains(line, "bytes"):
// "Space in use : 1234 MB (1.2%)" - not bytes; skip
}
}
return min, min
}
// computeBootRegions returns the byte ranges outside any cloned partition
// that must still be copied verbatim.
func computeBootRegions(d *DiskLayout) []Region {
var regs []Region
// bootstrap code before the MBR partition table on both MBR and (BIOS-)GPT disks
regs = append(regs, Region{Offset: 0, Length: 446, Note: "MBR bootstrap"})
if d.Scheme == "mbr" && len(d.Partitions) > 0 {
firstStart := d.Partitions[0].Start * d.LogicalSector
for _, p := range d.Partitions {
s := p.Start * d.LogicalSector
if s > 0 && s < firstStart {
firstStart = s
}
}
if firstStart > 512 {
regs = append(regs, Region{Offset: 512, Length: firstStart - 512, Note: "MBR gap (GRUB core.img etc.)"})
}
}
return regs
}
// applyPartitionTable writes script (an sfdisk restore script) to target,
// which may be a block device or an image file, and re-reads the table.
func applyPartitionTable(target, script string) error {
cmd := exec.Command("sfdisk", "--no-reread", "--no-tell-kernel", "--force", "--wipe=always", target)
cmd.Stdin = strings.NewReader(script)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("sfdisk %s: %v: %s", target, err, strings.TrimSpace(string(out)))
}
rereadPartTable(target)
return nil
}
func rereadPartTable(target string) {
fi, err := os.Stat(target)
if err != nil || fi.Mode()&os.ModeDevice == 0 {
return
}
if err := exec.Command("partprobe", target).Run(); err != nil {
_ = exec.Command("blockdev", "--rereadpt", target).Run()
}
}
// lsblkFSType reports the filesystem type on a node ("" if none).
func lsblkFSType(node string) string {
out, err := exec.Command("lsblk", "-ndo", "FSTYPE", node).Output()
if err != nil {
return ""
}
return strings.TrimSpace(string(out))
}
// mountAt mounts node at a fresh temp dir and returns the dir + an unmount fn.
func mountAt(node string, readonly bool) (string, func(), error) {
dir, err := os.MkdirTemp("", "clonetool-mnt-")
if err != nil {
return "", nil, err
}
args := []string{}
if readonly {
args = append(args, "-o", "ro")
}
args = append(args, node, dir)
if out, err := exec.Command("mount", args...).CombinedOutput(); err != nil {
os.Remove(dir)
return "", nil, fmt.Errorf("mount %s: %v: %s", node, err, strings.TrimSpace(string(out)))
}
return dir, func() {
_ = exec.Command("umount", dir).Run()
_ = os.Remove(dir)
}, nil
}
func cmdErr(err error) error {
if ee, ok := err.(*exec.ExitError); ok {
if s := strings.TrimSpace(string(ee.Stderr)); s != "" {
return fmt.Errorf("%v: %s", err, s)
}
}
return err
}

37
disk_other.go Normal file
View File

@ -0,0 +1,37 @@
//go:build !linux && !windows
package main
import (
"fmt"
"runtime"
)
func logicalSectorSize(string) (int64, error) { return 512, nil }
type diskAttachment struct {
blockPath string
cleanup func()
}
func attachDisk(string, bool) (*diskAttachment, error) {
return nil, fmt.Errorf("clone-disk is only implemented on linux and windows (this is %s)", runtime.GOOS)
}
func partNode(disk string, num int) string { return fmt.Sprintf("%s%d", disk, num) }
func probeDiskLayout(string) (*DiskLayout, error) {
return nil, fmt.Errorf("clone-disk is only implemented on linux and windows (this is %s)", runtime.GOOS)
}
func applyPartitionTable(string, string) error {
return fmt.Errorf("clone-disk partition rebuild is only implemented on linux and windows")
}
func rereadPartTable(string) {}
func lsblkFSType(string) string { return "" }
func mountAt(string, bool) (string, func(), error) {
return "", nil, fmt.Errorf("mount is only implemented on linux")
}

232
disk_windows.go Normal file
View File

@ -0,0 +1,232 @@
//go:build windows
package main
import (
"encoding/binary"
"fmt"
"os"
"os/exec"
"strings"
"syscall"
)
const (
ioctlDiskGetDriveLayoutEx = 0x00070050 // IOCTL_DISK_GET_DRIVE_LAYOUT_EX
ioctlDiskUpdateProperties = 0x00070140 // IOCTL_DISK_UPDATE_PROPERTIES
)
func logicalSectorSize(path string) (int64, error) {
if !isDevicePath(path) {
return 512, nil
}
return alignmentFor(path), nil
}
// guidString renders a little-endian Win32 GUID (4-2-2 + 8 bytes) as the
// canonical upper-case string.
func guidString(b []byte) string {
if len(b) < 16 {
return ""
}
d1 := binary.LittleEndian.Uint32(b[0:4])
d2 := binary.LittleEndian.Uint16(b[4:6])
d3 := binary.LittleEndian.Uint16(b[6:8])
return strings.ToUpper(fmt.Sprintf("%08X-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X",
d1, d2, d3, b[8], b[9], b[10], b[11], b[12], b[13], b[14], b[15]))
}
// probeDiskLayout reads the partition table of a \\.\PhysicalDriveN via
// IOCTL_DISK_GET_DRIVE_LAYOUT_EX. Windows-native clone-disk is raw/VSS only:
// every partition is cloned block-wise and the table itself is reproduced by
// raw-copying the leading sectors (see computeBootRegionsWindows), so no
// sfdisk-style rebuild is needed.
func probeDiskLayout(path string) (*DiskLayout, error) {
d := &DiskLayout{DiskPath: path, Tools: probeToolsetWindows()}
info, err := statPath(path)
if err != nil {
return nil, err
}
if !info.Exists {
return nil, fmt.Errorf("%s does not exist", path)
}
d.DiskSize = info.Size
d.LogicalSector, _ = logicalSectorSize(path)
if d.LogicalSector == 0 {
d.LogicalSector = 512
}
d.PhysicalSector = d.LogicalSector
if !isDevicePath(path) {
// An image file: we cannot enumerate its table without extra tooling.
d.Scheme = "raw"
d.Warnings = append(d.Warnings, "image-file source on Windows: whole-image raw clone only")
return d, nil
}
p16, err := syscall.UTF16PtrFromString(path)
if err != nil {
return nil, err
}
h, err := syscall.CreateFile(p16, syscall.GENERIC_READ,
syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE, nil, syscall.OPEN_EXISTING, 0, 0)
if err != nil {
return nil, &os.PathError{Op: "open", Path: path, Err: err}
}
defer syscall.CloseHandle(h)
buf := make([]byte, 64*1024)
var ret uint32
if err := syscall.DeviceIoControl(h, ioctlDiskGetDriveLayoutEx, nil, 0,
&buf[0], uint32(len(buf)), &ret, nil); err != nil {
return nil, fmt.Errorf("IOCTL_DISK_GET_DRIVE_LAYOUT_EX %s: %w", path, err)
}
style := binary.LittleEndian.Uint32(buf[0:4])
count := binary.LittleEndian.Uint32(buf[4:8])
switch style {
case 0:
d.Scheme, d.Label = "mbr", "dos"
sig := binary.LittleEndian.Uint32(buf[8:12])
d.LabelID = fmt.Sprintf("0x%08x", sig)
case 1:
d.Scheme, d.Label = "gpt", "gpt"
d.LabelID = guidString(buf[8:24])
default:
d.Scheme = "raw"
}
const hdr = 48
const peSize = 144
sector := d.LogicalSector
for i := uint32(0); i < count; i++ {
off := hdr + int(i)*peSize
if off+peSize > len(buf) {
break
}
e := buf[off : off+peSize]
start := int64(binary.LittleEndian.Uint64(e[8:16]))
length := int64(binary.LittleEndian.Uint64(e[16:24]))
num := int(binary.LittleEndian.Uint32(e[24:28]))
if length == 0 || num == 0 {
continue
}
p := Partition{
Num: num,
Start: start / sector,
Size: length / sector,
}
if style == 1 { // GPT union at offset 32
p.Type = guidString(e[32:48])
p.UUID = guidString(e[48:64])
p.Name = decodeUTF16(e[72:144])
} else if style == 0 { // MBR union at offset 32
p.Type = fmt.Sprintf("%02x", e[32])
if e[33] != 0 {
p.Attrs = "bootable"
}
}
d.Partitions = append(d.Partitions, p)
}
d.BootRegions = computeBootRegionsWindows(d)
return d, nil
}
func decodeUTF16(b []byte) string {
u := make([]uint16, 0, len(b)/2)
for i := 0; i+1 < len(b); i += 2 {
c := binary.LittleEndian.Uint16(b[i : i+2])
if c == 0 {
break
}
u = append(u, c)
}
return strings.TrimRight(syscall.UTF16ToString(u), "\x00")
}
// computeBootRegionsWindows: reproduce the partition table by raw-copying the
// sectors before the first partition, plus the trailing 33 sectors that hold
// the backup GPT.
func computeBootRegionsWindows(d *DiskLayout) []Region {
var regs []Region
if len(d.Partitions) == 0 {
return regs
}
first := d.Partitions[0].Start * d.LogicalSector
for _, p := range d.Partitions {
if s := p.Start * d.LogicalSector; s > 0 && s < first {
first = s
}
}
if first > 0 {
regs = append(regs, Region{Offset: 0, Length: first, Note: "protective MBR + primary GPT / MBR table + gap"})
}
if d.Scheme == "gpt" && d.DiskSize > 33*d.LogicalSector {
regs = append(regs, Region{
Offset: d.DiskSize - 33*d.LogicalSector,
Length: 33 * d.LogicalSector,
Note: "backup GPT",
})
}
return regs
}
func rereadPartTable(target string) {
if !isDevicePath(target) {
return
}
p16, err := syscall.UTF16PtrFromString(target)
if err != nil {
return
}
h, err := syscall.CreateFile(p16, syscall.GENERIC_READ|syscall.GENERIC_WRITE,
syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE, nil, syscall.OPEN_EXISTING, 0, 0)
if err != nil {
return
}
defer syscall.CloseHandle(h)
var ret uint32
_ = syscall.DeviceIoControl(h, ioctlDiskUpdateProperties, nil, 0, nil, 0, &ret, nil)
}
// applyPartitionTable is a no-op on Windows: the table is reproduced via the
// raw boot regions and rereadPartTable() refreshes the kernel's view.
func applyPartitionTable(target, script string) error {
rereadPartTable(target)
return nil
}
func probeToolsetWindows() map[string]bool {
m := map[string]bool{}
for _, t := range []string{"vssadmin", "wmic", "bcdboot", "diskpart", "robocopy", "powershell"} {
if _, err := exec.LookPath(t); err == nil {
m[t] = true
}
}
return m
}
type diskAttachment struct {
blockPath string
cleanup func()
}
// attachDisk is a no-op for a raw \\.\PhysicalDriveN (partitions are addressed
// by byte offset within the whole-disk handle on Windows) and unsupported for
// an image file.
func attachDisk(path string, writable bool) (*diskAttachment, error) {
if isDevicePath(path) {
return &diskAttachment{blockPath: path, cleanup: func() {}}, nil
}
return nil, fmt.Errorf("filesystem-aware / file-level clone of an image file is not supported on Windows; use --raw")
}
func partNode(disk string, num int) string { return fmt.Sprintf("%s (partition %d)", disk, num) }
func lsblkFSType(string) string { return "" }
func mountAt(string, bool) (string, func(), error) {
return "", nil, fmt.Errorf("mount-based file-level clone is not supported on Windows; use --raw")
}

298
disklayout.go Normal file
View File

@ -0,0 +1,298 @@
package main
import (
"bufio"
"encoding/json"
"fmt"
"strconv"
"strings"
)
// DiskLayout is the full picture of a source disk that clone-disk needs to
// reproduce a target: the partition table (as parsed from `sfdisk -d`, the
// canonical restorable form for both GPT and MBR), the byte ranges outside
// any partition that still have to be copied verbatim (MBR bootstrap + gap,
// BIOS-boot partition), the per-partition filesystem facts, and which of the
// external tools we shell out to are actually present.
type DiskLayout struct {
// Whole-disk geometry.
DiskPath string `json:"diskPath"`
DiskSize int64 `json:"diskSize"` // bytes
LogicalSector int64 `json:"logicalSector"` // bytes, from the kernel/ioctl
PhysicalSector int64 `json:"physicalSector"` // bytes
Scheme string `json:"scheme"` // "gpt" | "mbr" | "raw" (no recognisable table)
// Partition table, parsed from `sfdisk -d`.
Label string `json:"label"` // "gpt" | "dos"
LabelID string `json:"labelID"` // GPT disk GUID / MBR 4-byte signature (0x...)
FirstLBA int64 `json:"firstLBA"` // sectors; 0 if the dump omitted it
LastLBA int64 `json:"lastLBA"` // sectors; 0 if omitted
Partitions []Partition `json:"partitions"`
SfdiskDump string `json:"sfdiskDump"` // the verbatim `sfdisk -d` output
SgdiskB64 string `json:"sgdiskB64"` // base64 `sgdisk --backup` blob (GPT only), fallback restore path
// Byte ranges that must be raw-copied and are not covered by a cloned
// partition: MBR bootstrap (0..0x1BE) + the gap before the first
// partition, and any BIOS-boot partition.
BootRegions []Region `json:"bootRegions"`
Tools map[string]bool `json:"tools"` // external tool name -> present on PATH
Warnings []string `json:"warnings,omitempty"`
}
// Region is a [Offset, Offset+Length) byte window on the whole disk.
type Region struct {
Offset int64 `json:"offset"`
Length int64 `json:"length"`
Note string `json:"note,omitempty"`
}
// Partition is one row of the partition table plus the filesystem facts the
// source agent gathered for it.
type Partition struct {
Num int `json:"num"`
Start int64 `json:"start"` // sectors
Size int64 `json:"size"` // sectors
Type string `json:"type"` // GPT type GUID or MBR hex code
UUID string `json:"uuid"` // GPT partition GUID (PARTUUID)
Name string `json:"name"` // GPT partition name
Attrs string `json:"attrs"` // trailing flags: "bootable", "attrs=\"...\"", ...
// Filesystem facts (source agent; not from sfdisk).
DevPath string `json:"devPath"` // source node, e.g. /dev/sda2
FSType string `json:"fsType"` // ext4 | ntfs | vfat | xfs | btrfs | swap | "" (none/unknown)
FSLabel string `json:"fsLabel"`
FSUUID string `json:"fsUUID"`
FSUsedBytes int64 `json:"fsUsedBytes"` // 0 if unknown
FSMinBytes int64 `json:"fsMinBytes"` // smallest the fs can be shrunk to; 0 if unknown/unshrinkable
}
// StartBytes / SizeBytes convert the sector-unit table fields using the
// disk's logical sector size.
func (p Partition) StartBytes(sectorSize int64) int64 { return p.Start * sectorSize }
func (p Partition) SizeBytes(sectorSize int64) int64 { return p.Size * sectorSize }
func (d *DiskLayout) JSON() string {
b, _ := json.Marshal(d)
return string(b)
}
func parseDiskLayoutJSON(s string) (*DiskLayout, error) {
var d DiskLayout
if err := json.Unmarshal([]byte(s), &d); err != nil {
return nil, fmt.Errorf("decode disk layout: %w", err)
}
return &d, nil
}
// parseSfdiskDump fills the Label/LabelID/FirstLBA/LastLBA/Partitions fields
// of d from the text of `sfdisk -d`. It leaves the raw text in d.SfdiskDump.
func (d *DiskLayout) parseSfdiskDump(dump string) error {
d.SfdiskDump = dump
sc := bufio.NewScanner(strings.NewReader(dump))
sc.Buffer(make([]byte, 64*1024), 1024*1024)
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
if line == "" {
continue
}
if i := strings.Index(line, " : "); i >= 0 && strings.Contains(line, "start=") {
p, err := parseSfdiskPartLine(line)
if err != nil {
return err
}
d.Partitions = append(d.Partitions, p)
continue
}
key, val, ok := splitKV(line)
if !ok {
continue
}
switch key {
case "label":
d.Label = val
case "label-id":
d.LabelID = val
case "first-lba":
d.FirstLBA, _ = strconv.ParseInt(val, 10, 64)
case "last-lba":
d.LastLBA, _ = strconv.ParseInt(val, 10, 64)
case "sector-size":
if n, err := strconv.ParseInt(val, 10, 64); err == nil && d.LogicalSector == 0 {
d.LogicalSector = n
}
}
}
if err := sc.Err(); err != nil {
return fmt.Errorf("scan sfdisk dump: %w", err)
}
switch d.Label {
case "gpt":
d.Scheme = "gpt"
case "dos":
d.Scheme = "mbr"
}
return nil
}
// splitKV parses a "key: value" header line.
func splitKV(line string) (string, string, bool) {
i := strings.Index(line, ":")
if i < 0 {
return "", "", false
}
return strings.TrimSpace(line[:i]), strings.TrimSpace(line[i+1:]), true
}
// parseSfdiskPartLine parses a body line like
//
// /dev/sda2 : start= 1050624, size= 1951xxxxxx, type=0FC6..., uuid=..., name="root", attrs="..."
// /dev/sda1 : start= 2048, size= 204800, type=83, bootable
func parseSfdiskPartLine(line string) (Partition, error) {
var p Partition
i := strings.Index(line, " : ")
dev := strings.TrimSpace(line[:i])
p.DevPath = dev
p.Num = trailingInt(dev)
rest := line[i+3:]
for _, fld := range splitTopLevel(rest, ',') {
fld = strings.TrimSpace(fld)
if fld == "" {
continue
}
k, v, ok := splitEq(fld)
if !ok {
// bare flag, e.g. "bootable"
if p.Attrs == "" {
p.Attrs = fld
} else {
p.Attrs += ", " + fld
}
continue
}
v = strings.Trim(strings.TrimSpace(v), `"`)
switch k {
case "start":
p.Start, _ = strconv.ParseInt(strings.TrimSpace(v), 10, 64)
case "size":
p.Size, _ = strconv.ParseInt(strings.TrimSpace(v), 10, 64)
case "type":
p.Type = v
case "uuid":
p.UUID = v
case "name":
p.Name = v
case "attrs":
if p.Attrs == "" {
p.Attrs = `attrs="` + v + `"`
} else {
p.Attrs += `, attrs="` + v + `"`
}
}
}
if p.Start == 0 && p.Size == 0 {
return p, fmt.Errorf("sfdisk line without start/size: %q", line)
}
return p, nil
}
// sfdiskRestoreScript renders a device-independent script that recreates this
// table on any target with `sfdisk`. It intentionally drops `device:` and the
// per-line device prefixes (sfdisk numbers the partitions in order) and
// `last-lba` (let sfdisk size the secondary GPT for the target). label-id and
// per-partition uuid are dropped when newIDs is set so a replacement disk can
// coexist with the original; otherwise they are preserved so existing
// BCD/fstab/GRUB references still resolve. sizeOverride maps a 1-based
// partition number to a new size in sectors (used by the shrink path).
func (d *DiskLayout) sfdiskRestoreScript(newIDs bool, sizeOverride map[int]int64) string {
var b strings.Builder
fmt.Fprintf(&b, "label: %s\n", d.Label)
if d.LabelID != "" && !newIDs {
fmt.Fprintf(&b, "label-id: %s\n", d.LabelID)
}
if d.FirstLBA > 0 {
fmt.Fprintf(&b, "first-lba: %d\n", d.FirstLBA)
}
if d.LogicalSector > 0 {
fmt.Fprintf(&b, "sector-size: %d\n", d.LogicalSector)
}
b.WriteString("unit: sectors\n\n")
for _, p := range d.Partitions {
size := p.Size
if ov, ok := sizeOverride[p.Num]; ok && ov > 0 {
size = ov
}
fmt.Fprintf(&b, "start=%d, size=%d", p.Start, size)
if p.Type != "" {
fmt.Fprintf(&b, ", type=%s", p.Type)
}
if p.UUID != "" && !newIDs {
fmt.Fprintf(&b, ", uuid=%s", p.UUID)
}
if p.Name != "" {
fmt.Fprintf(&b, ", name=\"%s\"", p.Name)
}
if p.Attrs != "" {
fmt.Fprintf(&b, ", %s", p.Attrs)
}
b.WriteByte('\n')
}
return b.String()
}
// biosBootType / espType are the GPT type GUIDs clone-disk special-cases.
const (
gptBIOSBoot = "21686148-6449-6E6F-744E-656564454649" // EF02 BIOS boot partition
gptESP = "C12A7328-F81F-11D2-BA4B-00A0C93EC93B" // EF00 EFI system partition
)
func (p Partition) isBIOSBoot() bool {
return strings.EqualFold(p.Type, gptBIOSBoot) || p.Type == "21" /* mbr */
}
func (p Partition) isESP() bool {
return strings.EqualFold(p.Type, gptESP) || p.Type == "ef" || p.Type == "EF"
}
// --- small string helpers -------------------------------------------------
func trailingInt(s string) int {
j := len(s)
for j > 0 && s[j-1] >= '0' && s[j-1] <= '9' {
j--
}
if j == len(s) {
return 0
}
n, _ := strconv.Atoi(s[j:])
return n
}
func splitEq(s string) (string, string, bool) {
i := strings.Index(s, "=")
if i < 0 {
return "", "", false
}
return strings.TrimSpace(s[:i]), s[i+1:], true
}
// splitTopLevel splits on sep but not inside double quotes.
func splitTopLevel(s string, sep byte) []string {
var out []string
var cur strings.Builder
inQ := false
for i := 0; i < len(s); i++ {
c := s[i]
switch {
case c == '"':
inQ = !inQ
cur.WriteByte(c)
case c == sep && !inQ:
out = append(out, cur.String())
cur.Reset()
default:
cur.WriteByte(c)
}
}
out = append(out, cur.String())
return out
}

113
disklayout_test.go Normal file
View File

@ -0,0 +1,113 @@
package main
import (
"strings"
"testing"
)
const gptDump = `label: gpt
label-id: 1D2E3F4A-1111-2222-3333-444455556666
device: /dev/sda
unit: sectors
first-lba: 2048
last-lba: 8388574
sector-size: 512
/dev/sda1 : start= 2048, size= 204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=AAAAAAAA-1111-2222-3333-444455556666, name="EFI System Partition"
/dev/sda2 : start= 206848, size= 8179712, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=BBBBBBBB-1111-2222-3333-444455556666
`
const mbrDump = `label: dos
label-id: 0x1a2b3c4d
device: /dev/sdb
unit: sectors
sector-size: 512
/dev/sdb1 : start= 2048, size= 512000, type=83, bootable
/dev/sdb2 : start= 514048, size= 41734144, type=83
`
func TestParseSfdiskDumpGPT(t *testing.T) {
var d DiskLayout
if err := d.parseSfdiskDump(gptDump); err != nil {
t.Fatal(err)
}
if d.Label != "gpt" || d.Scheme != "gpt" {
t.Fatalf("label/scheme = %q/%q", d.Label, d.Scheme)
}
if d.LabelID != "1D2E3F4A-1111-2222-3333-444455556666" {
t.Fatalf("labelID = %q", d.LabelID)
}
if d.FirstLBA != 2048 || d.LastLBA != 8388574 {
t.Fatalf("lba = %d/%d", d.FirstLBA, d.LastLBA)
}
if len(d.Partitions) != 2 {
t.Fatalf("parts = %d", len(d.Partitions))
}
p1 := d.Partitions[0]
if p1.Num != 1 || p1.Start != 2048 || p1.Size != 204800 {
t.Fatalf("p1 = %+v", p1)
}
if p1.Type != "C12A7328-F81F-11D2-BA4B-00A0C93EC93B" || p1.UUID != "AAAAAAAA-1111-2222-3333-444455556666" {
t.Fatalf("p1 type/uuid = %q/%q", p1.Type, p1.UUID)
}
if p1.Name != "EFI System Partition" {
t.Fatalf("p1 name = %q", p1.Name)
}
if !p1.isESP() {
t.Fatalf("p1 should be ESP")
}
if d.Partitions[1].Num != 2 || d.Partitions[1].Size != 8179712 {
t.Fatalf("p2 = %+v", d.Partitions[1])
}
}
func TestParseSfdiskDumpMBR(t *testing.T) {
var d DiskLayout
if err := d.parseSfdiskDump(mbrDump); err != nil {
t.Fatal(err)
}
if d.Scheme != "mbr" || d.LabelID != "0x1a2b3c4d" {
t.Fatalf("scheme/id = %q/%q", d.Scheme, d.LabelID)
}
if len(d.Partitions) != 2 || d.Partitions[0].Attrs != "bootable" {
t.Fatalf("parts = %+v", d.Partitions)
}
}
func TestSfdiskRestoreScript(t *testing.T) {
var d DiskLayout
if err := d.parseSfdiskDump(gptDump); err != nil {
t.Fatal(err)
}
// keep IDs
s := d.sfdiskRestoreScript(false, nil)
if !strings.Contains(s, "label-id: 1D2E3F4A-1111-2222-3333-444455556666") {
t.Fatalf("expected label-id kept:\n%s", s)
}
if !strings.Contains(s, "uuid=AAAAAAAA-1111-2222-3333-444455556666") {
t.Fatalf("expected part uuid kept:\n%s", s)
}
if strings.Contains(s, "device:") || strings.Contains(s, "/dev/sda1") {
t.Fatalf("script must be device-independent:\n%s", s)
}
if strings.Contains(s, "last-lba") {
t.Fatalf("script must drop last-lba:\n%s", s)
}
// new IDs
s2 := d.sfdiskRestoreScript(true, nil)
if strings.Contains(s2, "label-id:") || strings.Contains(s2, "uuid=") {
t.Fatalf("--new-ids must strip ids:\n%s", s2)
}
// size override on p2
s3 := d.sfdiskRestoreScript(false, map[int]int64{2: 4000000})
if !strings.Contains(s3, "size=4000000") {
t.Fatalf("expected overridden size:\n%s", s3)
}
if strings.Contains(s3, "size=8179712") {
t.Fatalf("old size should be gone:\n%s", s3)
}
}

200
fsclone.go Normal file
View File

@ -0,0 +1,200 @@
package main
import (
"bufio"
"encoding/json"
"fmt"
"io"
"regexp"
"strconv"
"strings"
)
// fsCloner is the pair of argv templates that stream one filesystem as an
// image: save reads the fs on dev and writes an image to stdout; restore
// reads that image from stdin and writes it onto dev. The image format is
// whatever the chosen standard tool uses (ntfsclone, partclone, e2image) —
// clonetool only moves the bytes between the two ends.
type fsCloner struct {
name string // tool family, for logs
save []string
restore []string
}
// fsCloneOpts is the JSON blob carried in CtrlMsg.Options for a fs-image
// partition clone so both ends build the identical argv.
type fsCloneOpts struct {
FSType string `json:"fsType"`
Tool string `json:"tool"`
SizeBytes int64 `json:"sizeBytes"`
// ShrinkToBytes > 0: shrink the SOURCE filesystem to this size in place
// before imaging it (so it fits a smaller target partition). This
// mutates the source and only runs when the operator passed
// --allow-shrink --yes.
ShrinkToBytes int64 `json:"shrinkToBytes,omitempty"`
}
func (o fsCloneOpts) encode() string { b, _ := json.Marshal(o); return string(b) }
func decodeFSCloneOpts(s string) (fsCloneOpts, error) {
var o fsCloneOpts
err := json.Unmarshal([]byte(s), &o)
return o, err
}
// pickCloner returns the fs-image cloner for (fsType, tool) or nil if the
// partition should be handled some other way. tool is the family name chosen
// by chooseCloneMethod ("ntfsclone", "partclone", "e2image").
func pickCloner(dev, fsType, tool string) *fsCloner {
switch tool {
case "ntfsclone":
return &fsCloner{
name: "ntfsclone",
save: []string{"ntfsclone", "--save-image", "--force", "--output", "-", dev},
restore: []string{"ntfsclone", "--restore-image", "--overwrite", dev, "-"},
}
case "e2image":
return &fsCloner{
name: "e2image",
// -ra: raw + skip unallocated; stream to/from stdio.
save: []string{"e2image", "-ra", "-p", dev, "-"},
restore: []string{"e2image", "-ra", "-p", "-", dev},
}
case "partclone":
pc := "partclone." + partcloneSuffix(fsType)
return &fsCloner{
name: pc,
save: []string{pc, "-c", "-s", dev, "-O", "-", "-L", "/dev/null"},
restore: []string{"partclone.restore", "-s", "-", "-O", dev, "-L", "/dev/null"},
}
}
return nil
}
func partcloneSuffix(fsType string) string {
switch fsType {
case "ext2", "ext3", "ext4":
return "extfs"
case "vfat", "fat", "fat12", "fat16", "fat32", "msdos":
return "fat"
case "ntfs":
return "ntfs"
default:
return fsType // exfat, xfs, btrfs, f2fs, hfsplus, ...
}
}
// chooseCloneMethod decides how a partition's data is copied.
//
// "raw" -> the block-diff engine over an offset window (also the
// re-run-friendly path; used for swap, unknown fs, --raw)
// "fs-image" -> stream a filesystem image with a standard tool
// "file-level" -> mkfs on the target + rsync
func chooseCloneMethod(p Partition, tools map[string]bool, forceRaw, forceFile map[int]bool, fileAuto bool) (method, tool string) {
if forceRaw[p.Num] {
return "raw", ""
}
if forceFile[p.Num] {
return "file-level", ""
}
if p.isBIOSBoot() {
return "raw", "" // tiny, and holds GRUB core - always verbatim
}
switch p.FSType {
case "", "swap", "crypto_LUKS", "LVM2_member":
return "raw", ""
case "ntfs":
if tools["ntfsclone"] {
return "fs-image", "ntfsclone"
}
case "ext2", "ext3", "ext4":
if tools["partclone.extfs"] && tools["partclone.restore"] {
return "fs-image", "partclone"
}
if tools["e2image"] {
return "fs-image", "e2image"
}
case "vfat", "fat", "fat12", "fat16", "fat32", "msdos":
if tools["partclone.fat"] && tools["partclone.restore"] {
return "fs-image", "partclone"
}
case "exfat", "xfs", "btrfs", "f2fs", "hfsplus":
if tools["partclone."+partcloneSuffix(p.FSType)] && tools["partclone.restore"] {
return "fs-image", "partclone"
}
}
if fileAuto && p.FSType != "" && tools["rsync"] {
return "file-level", ""
}
return "raw", ""
}
// progressScanner reads a cloner's stderr line by line, pulls a percentage
// out of the tool's own progress chatter, and calls emit with an estimated
// byte count (fraction * sizeBytes). It swallows everything else.
var pctRe = regexp.MustCompile(`([0-9]{1,3}(?:\.[0-9]+)?)\s*%|([0-9]{1,3}(?:\.[0-9]+)?)\s*percent`)
func scanCloneProgress(r io.Reader, sizeBytes int64, emit func(bytesDone int64)) {
br := bufio.NewReader(r)
var buf []byte
flush := func() {
line := strings.TrimSpace(string(buf))
buf = buf[:0]
if line == "" {
return
}
m := pctRe.FindStringSubmatch(line)
if m == nil {
return
}
s := m[1]
if s == "" {
s = m[2]
}
if f, err := strconv.ParseFloat(s, 64); err == nil && f >= 0 && f <= 100 {
emit(int64(f / 100 * float64(sizeBytes)))
}
}
for {
c, err := br.ReadByte()
if err != nil {
flush()
return
}
if c == '\n' || c == '\r' {
flush()
continue
}
buf = append(buf, c)
if len(buf) > 4096 {
buf = buf[:0]
}
}
}
// preShrinkFS shrinks the filesystem on dev to (at least) toBytes in place.
// Used only on the source, only with --allow-shrink --yes. Best effort with a
// hard error if the resize tool fails.
func preShrinkFS(dev, fsType string, toBytes int64) error {
if toBytes <= 0 {
return nil
}
switch {
case isExtFS(fsType):
if out, err := runCmd("e2fsck", "-f", "-y", dev); err != nil {
return fmt.Errorf("e2fsck %s: %v: %s", dev, err, out)
}
kib := toBytes / 1024
if out, err := runCmd("resize2fs", dev, fmt.Sprintf("%dK", kib)); err != nil {
return fmt.Errorf("resize2fs %s %dK: %v: %s", dev, kib, err, out)
}
return nil
case fsType == "ntfs":
// ntfsresize prompts for confirmation on a real resize; feed it "y".
if out, err := runCmdStdin("y\n", "ntfsresize", "--force", "--size", strconv.FormatInt(toBytes, 10), dev); err != nil {
return fmt.Errorf("ntfsresize %s --size %d: %v: %s", dev, toBytes, err, out)
}
return nil
default:
return fmt.Errorf("don't know how to shrink a %s filesystem", fsType)
}
}

112
main.go
View File

@ -5,6 +5,7 @@ import (
"fmt"
"os"
"runtime"
"strings"
)
func main() {
@ -17,6 +18,8 @@ func main() {
switch os.Args[1] {
case "sync":
err = cmdSync(os.Args[2:])
case "clone-disk":
err = cmdCloneDisk(os.Args[2:])
case "agent":
err = cmdAgent(os.Args[2:])
case "version", "--version":
@ -40,8 +43,9 @@ func usage() {
Usage:
clonetool sync --source LOC --dest LOC [options]
clonetool clone-disk --source LOC --dest LOC [options]
clonetool version
clonetool agent --role {control|sink|source-stream} ... (internal, spawned automatically)
clonetool agent --role {control|sink|source-stream|fs-send|fs-recv} ... (internal, spawned automatically)
LOC is either a local path, or [user@]host:path for a path reached over SSH.
Source, destination, and the machine running "sync" (the manager) may all be
@ -51,6 +55,21 @@ a single block itself.
clonetool keeps no state between runs: every sync re-reads and re-hashes both
the source and the destination and transfers only the blocks that differ.
Options for clone-disk (whole-disk: boot record + partition table + per-filesystem data):
--parts LIST only clone these partition numbers (comma-separated; default all)
--raw LIST force a raw block clone for these partitions
--file-level LIST mkfs + rsync these partitions instead of a filesystem image
--file-level-auto use file-level for filesystem types with no image cloner
--allow-shrink permit shrinking trailing partitions to fit a smaller target
(resizes the SOURCE filesystem in place before imaging)
--no-shrink never shrink; fail instead if the target is too small
--new-ids randomize the GPT disk GUID / MBR signature on the target
--reinstall-bootloader after copy, run grub-install / grub-mkconfig on the target (Linux)
--vss Windows source: take a Volume Shadow Copy per NTFS volume (default on)
--image-size SIZE file target: image size (default = enough for the layout)
(also accepts --block-size --job --yes --sudo --deploy --ssh --ssh-opt --remote-bin
--connect-timeout --manager-host, same meaning as sync)
Options for sync:
--block-size SIZE block size, e.g. 4M (default 4M)
--job NAME optional label shown in progress/log output
@ -116,6 +135,97 @@ func cmdSync(args []string) error {
})
}
func parseIntList(s string) ([]int, error) {
if strings.TrimSpace(s) == "" {
return nil, nil
}
var out []int
for _, f := range strings.Split(s, ",") {
f = strings.TrimSpace(f)
if f == "" {
continue
}
var n int
if _, err := fmt.Sscanf(f, "%d", &n); err != nil || n <= 0 {
return nil, fmt.Errorf("invalid partition number %q", f)
}
out = append(out, n)
}
return out, nil
}
func cmdCloneDisk(args []string) error {
fs := flag.NewFlagSet("clone-disk", flag.ContinueOnError)
job := fs.String("job", "", "optional label for progress/log output")
source := fs.String("source", "", "source whole-disk location (required)")
dest := fs.String("dest", "", "destination whole-disk or image location (required)")
blockSizeStr := fs.String("block-size", "4M", "block size for raw/boot-region transfers, e.g. 4M")
yes := fs.Bool("yes", false, "don't prompt before shrinking an existing destination image")
sudoMode := fs.String("sudo", "auto", "device-access privilege escalation: auto|always|never")
deploy := fs.Bool("deploy", true, "copy this binary to remote hosts that lack it")
connectTimeout := fs.Int("connect-timeout", defaultConnectTimeoutSec, "ssh connect timeout (seconds)")
sshBin := fs.String("ssh", "ssh", "ssh binary")
remoteBin := fs.String("remote-bin", "clonetool", "clonetool path on remote hosts")
managerHost := fs.String("manager-host", "", "address peers use to reach this machine")
partsStr := fs.String("parts", "", "only clone these partition numbers (comma-separated)")
rawStr := fs.String("raw", "", "force raw block clone for these partition numbers")
fileLevelStr := fs.String("file-level", "", "mkfs + rsync these partition numbers")
fileAuto := fs.Bool("file-level-auto", false, "file-level for fs types with no image cloner")
allowShrink := fs.Bool("allow-shrink", false, "permit shrinking trailing partitions to fit a smaller target")
noShrink := fs.Bool("no-shrink", false, "never shrink; fail if the target is too small")
newIDs := fs.Bool("new-ids", false, "randomize the GPT disk GUID / MBR signature on the target")
reinstallBoot := fs.Bool("reinstall-bootloader", false, "run grub-install / grub-mkconfig on the target after copy")
vss := fs.Bool("vss", true, "Windows source: take a Volume Shadow Copy per NTFS volume")
imageSizeStr := fs.String("image-size", "", "file target: image size (default: enough for the layout)")
var sshOpts stringSlice
fs.Var(&sshOpts, "ssh-opt", `extra "-o OPT" passed to ssh (repeatable)`)
if err := fs.Parse(args); err != nil {
return err
}
if *source == "" || *dest == "" {
fs.Usage()
return fmt.Errorf("--source and --dest are required")
}
blockSize, err := parseSize(*blockSizeStr)
if err != nil {
return fmt.Errorf("--block-size: %w", err)
}
switch *sudoMode {
case "auto", "always", "never":
default:
return fmt.Errorf("--sudo: want auto|always|never, got %q", *sudoMode)
}
parts, err := parseIntList(*partsStr)
if err != nil {
return fmt.Errorf("--parts: %w", err)
}
raw, err := parseIntList(*rawStr)
if err != nil {
return fmt.Errorf("--raw: %w", err)
}
fileLevel, err := parseIntList(*fileLevelStr)
if err != nil {
return fmt.Errorf("--file-level: %w", err)
}
var imageSize int64
if *imageSizeStr != "" {
if imageSize, err = parseSize(*imageSizeStr); err != nil {
return fmt.Errorf("--image-size: %w", err)
}
}
return runCloneDisk(CloneDiskConfig{
SyncConfig: SyncConfig{
Job: *job, Source: *source, Dest: *dest, BlockSize: blockSize,
Yes: *yes, Sudo: *sudoMode, Deploy: *deploy, ConnectTimeoutSec: *connectTimeout,
SSHBin: *sshBin, SSHOpts: sshOpts, RemoteBin: *remoteBin, ManagerHost: *managerHost,
},
Parts: parts, Raw: raw, FileLevel: fileLevel, FileAuto: *fileAuto,
AllowShrink: *allowShrink, NoShrink: *noShrink, NewIDs: *newIDs,
ReinstallBoot: *reinstallBoot, VSS: *vss, ImageSize: imageSize,
})
}
func parseSize(s string) (int64, error) {
if s == "" {
return 0, fmt.Errorf("empty size")

20
manager.go
View File

@ -297,6 +297,26 @@ func (p *progressPrinter) print(m CtrlMsg) {
formatCount(p.copied), formatCount(p.skipped), scan, mib)
}
// printBytes renders a byte-oriented status line for clone-disk's fs-image
// partition clones, where progress arrives as BytesCopied/BytesTotal rather
// than block counts.
func (p *progressPrinter) printBytes(m CtrlMsg) {
if m.Type != msgProgress {
return
}
p.active = true
done, total := m.BytesCopied, m.BytesTotal
if total <= 0 {
total = done
}
stage := m.Stage
if stage == "" {
stage = "fs-image"
}
fmt.Fprintf(os.Stderr, "\r %s %s %s / %s ",
stage, progressBar(done, total), humanBytes(done), humanBytes(total))
}
// finish ends the current status line with a newline so following output
// (and the shell prompt) starts clean.
func (p *progressPrinter) finish() {

215
plan.go Normal file
View File

@ -0,0 +1,215 @@
package main
import (
"fmt"
"strings"
)
// plannedPart is one partition's copy plan: where it lives on each side (in
// bytes) and how its data moves.
type plannedPart struct {
Num int
SrcStartB, SrcSizeB int64
DstStartB, DstSizeB int64
Method string // "raw" | "fs-image" | "file-level"
Tool string // fs-image tool family
FSType string
ShrinkToB int64 // >0: shrink the fs to this many bytes before imaging
}
type targetPlan struct {
Script string // sfdisk restore script ("" when the source has no table / Windows-native)
IsFile bool
ImageSize int64 // file target: truncate to this
Parts []plannedPart
BootRegions []Region
RootPart int
ESPPart int
UEFI bool
Notes []string
}
// bootOpts is the JSON blob passed to reinstall_boot.
type bootOpts struct {
RootPart int `json:"rootPart"` // 1-based partition number holding "/"
ESPPart int `json:"espPart"` // 1-based ESP partition number, 0 if BIOS-only
UEFI bool `json:"uefi"`
DiskPath string `json:"diskPath"` // whole-disk path grub-install targets
}
type planOpts struct {
Parts map[int]bool // empty = all
Raw map[int]bool
FileLevel map[int]bool
FileAuto bool
AllowShrink bool
NewIDs bool
ImageSize int64 // explicit file-target size, 0 = auto
}
const gptTailSectors = 33 // secondary GPT header + entries
// planTargetLayout turns a probed source layout + the target's nature/size
// into a concrete copy plan, applying the sizing rules:
//
// - device target: never grow; if everything fits, keep the layout and
// leave trailing space untouched; if the last partition(s) overflow,
// shrink them (filesystem + partition) from the last inward, or fail.
// - file target: size the image to hold the layout (or ImageSize).
func planTargetLayout(src *DiskLayout, isFile bool, destSize int64, o planOpts) (*targetPlan, error) {
sector := src.LogicalSector
if sector <= 0 {
sector = 512
}
parts := sortedParts(src.Partitions)
var sel []Partition
for _, p := range parts {
if len(o.Parts) == 0 || o.Parts[p.Num] {
sel = append(sel, p)
}
}
if len(sel) == 0 && src.Scheme != "raw" {
return nil, fmt.Errorf("no partitions selected")
}
plan := &targetPlan{IsFile: isFile, BootRegions: append([]Region(nil), src.BootRegions...)}
// With --new-ids on an MBR disk, keep the raw bootstrap copy clear of the
// 4-byte disk signature at offset 0x1B8 so sfdisk's freshly generated one
// survives (GPT regenerates its GUID in the header, which we never
// raw-copy, so no clamp is needed there).
if o.NewIDs && src.Scheme == "mbr" {
for i := range plan.BootRegions {
r := &plan.BootRegions[i]
if r.Offset == 0 && r.Length > 0x1B8 {
r.Length = 0x1B8
}
}
}
// Bytes the layout needs on disk = end of the last selected partition
// (+ secondary GPT for GPT disks).
lastEndB := int64(0)
for _, p := range sel {
if e := (p.Start + p.Size) * sector; e > lastEndB {
lastEndB = e
}
}
neededB := lastEndB
if src.Scheme == "gpt" {
neededB += gptTailSectors * sector
}
if src.Scheme == "raw" {
neededB = src.DiskSize
}
if isFile {
plan.ImageSize = o.ImageSize
if plan.ImageSize == 0 {
plan.ImageSize = roundUp(neededB+(1<<20), 1<<20)
}
destSize = plan.ImageSize
}
if destSize <= 0 {
return nil, fmt.Errorf("could not determine target size")
}
// --- shrink pass ----------------------------------------------------
sizeOverride := map[int]int64{}
if destSize < neededB {
short := neededB - destSize
for i := len(sel) - 1; i >= 0 && short > 0; i-- {
p := &sel[i]
curB := p.Size * sector
minB := p.FSMinBytes
shrinkable := minB > 0 && (isExtFS(p.FSType) || p.FSType == "ntfs")
if !shrinkable {
continue
}
minB = roundUp(minB+(16<<20), sector) // 16 MiB slack
if minB >= curB {
continue
}
reduce := curB - minB
if reduce > short {
reduce = roundUp(short, sector)
}
newB := curB - reduce
sizeOverride[p.Num] = newB / sector
plan.Notes = append(plan.Notes,
fmt.Sprintf("partition p%d (%s) shrinks %s -> %s to fit the target",
p.Num, p.FSType, humanBytes(curB), humanBytes(newB)))
short -= reduce
}
if short > 0 {
return nil, fmt.Errorf(
"target (%s) is smaller than the source layout needs (%s); the trailing partition(s) "+
"cannot shrink to make up the missing %s; use a larger target",
humanBytes(destSize), humanBytes(neededB), humanBytes(short))
}
if !o.AllowShrink {
var names []string
for n := range sizeOverride {
names = append(names, fmt.Sprintf("p%d", n))
}
return nil, fmt.Errorf(
"target is smaller than the source; partition(s) %s would have to be shrunk. "+
"Re-run with --allow-shrink (this resizes the SOURCE filesystem in place before imaging) "+
"or use --no-shrink off / a larger target",
strings.Join(names, ", "))
}
}
// --- partition-table script --------------------------------------
if src.Scheme != "raw" && strings.TrimSpace(src.SfdiskDump) != "" {
plan.Script = src.sfdiskRestoreScript(o.NewIDs, sizeOverride)
}
// --- per-partition method --------------------------------------
for _, p := range sel {
pp := plannedPart{
Num: p.Num,
SrcStartB: p.Start * sector,
SrcSizeB: p.Size * sector,
DstStartB: p.Start * sector,
DstSizeB: p.Size * sector,
FSType: p.FSType,
}
if ov, ok := sizeOverride[p.Num]; ok {
pp.DstSizeB = ov * sector
pp.ShrinkToB = ov * sector
}
method, tool := chooseCloneMethod(p, src.Tools, o.Raw, o.FileLevel, o.FileAuto)
pp.Method, pp.Tool = method, tool
if pp.ShrinkToB > 0 && method != "fs-image" {
return nil, fmt.Errorf(
"partition p%d (%s) must shrink to fit but no image cloner is available for it; "+
"install ntfsclone / partclone or use a larger target", p.Num, orDash(p.FSType))
}
plan.Parts = append(plan.Parts, pp)
}
// --- boot roles ------------------------------------------------
plan.RootPart, plan.ESPPart = pickBootParts(sel)
plan.UEFI = plan.ESPPart != 0
return plan, nil
}
func isExtFS(t string) bool { return t == "ext2" || t == "ext3" || t == "ext4" }
// pickBootParts guesses the "/" partition (largest Linux fs) and the ESP.
func pickBootParts(sel []Partition) (root, esp int) {
var bestSize int64
for _, p := range sel {
if p.isESP() && esp == 0 {
esp = p.Num
}
switch p.FSType {
case "ext2", "ext3", "ext4", "btrfs", "xfs":
if sz := p.Size; sz > bestSize {
bestSize, root = sz, p.Num
}
}
}
return root, esp
}

146
plan_test.go Normal file
View File

@ -0,0 +1,146 @@
package main
import "testing"
// mkLayout builds a GPT layout: ESP (vfat) + root (ext4) + data (ntfs).
func mkLayout(diskBytes int64) *DiskLayout {
d := &DiskLayout{
DiskPath: "/dev/src", DiskSize: diskBytes, LogicalSector: 512,
Scheme: "gpt", Label: "gpt", LabelID: "GUID",
SfdiskDump: gptDump, // any non-empty dump so a script is produced
Tools: map[string]bool{
"sfdisk": true, "ntfsclone": true,
"partclone.extfs": true, "partclone.restore": true, "e2image": true,
},
Partitions: []Partition{
{Num: 1, Start: 2048, Size: 204800, Type: gptESP, FSType: "vfat"}, // 100 MiB
{Num: 2, Start: 206848, Size: 10 * 2048 * 1024, FSType: "ext4", FSMinBytes: 2 << 30}, // 10 GiB, min 2 GiB
{Num: 3, Start: 206848 + 10*2048*1024, Size: 10 * 2048 * 1024, FSType: "ntfs", FSMinBytes: 3 << 30}, // 10 GiB, min 3 GiB
},
BootRegions: []Region{{Offset: 0, Length: 446, Note: "MBR bootstrap"}},
}
return d
}
func lastEndBytes(d *DiskLayout) int64 {
var e int64
for _, p := range d.Partitions {
if x := (p.Start + p.Size) * d.LogicalSector; x > e {
e = x
}
}
return e + gptTailSectors*d.LogicalSector
}
func TestPlanFitsDeviceKeepsLayout(t *testing.T) {
d := mkLayout(40 << 30)
plan, err := planTargetLayout(d, false, 40<<30, planOpts{})
if err != nil {
t.Fatal(err)
}
if len(plan.Parts) != 3 {
t.Fatalf("parts = %d", len(plan.Parts))
}
for _, p := range plan.Parts {
if p.SrcSizeB != p.DstSizeB || p.ShrinkToB != 0 {
t.Fatalf("p%d resized unexpectedly: %+v", p.Num, p)
}
if p.SrcStartB != p.DstStartB {
t.Fatalf("p%d start moved", p.Num)
}
}
if plan.Parts[0].Method != "raw" { // vfat, no partclone.fat in tool set
t.Fatalf("p1 method = %s", plan.Parts[0].Method)
}
if plan.Parts[1].Method != "fs-image" || plan.Parts[1].Tool != "partclone" {
t.Fatalf("p2 method/tool = %s/%s", plan.Parts[1].Method, plan.Parts[1].Tool)
}
if plan.Parts[2].Method != "fs-image" || plan.Parts[2].Tool != "ntfsclone" {
t.Fatalf("p3 method/tool = %s/%s", plan.Parts[2].Method, plan.Parts[2].Tool)
}
if plan.ESPPart != 1 || plan.RootPart != 2 || !plan.UEFI {
t.Fatalf("boot roles: esp=%d root=%d uefi=%v", plan.ESPPart, plan.RootPart, plan.UEFI)
}
}
func TestPlanTrailingFreeSpaceOK(t *testing.T) {
d := mkLayout(60 << 30)
// target smaller than the *disk* but larger than what the layout needs
need := lastEndBytes(d)
if _, err := planTargetLayout(d, false, need+1<<20, planOpts{}); err != nil {
t.Fatalf("should fit into %d (need %d): %v", need+1<<20, need, err)
}
}
func TestPlanNeedsShrinkRequiresFlag(t *testing.T) {
d := mkLayout(40 << 30)
need := lastEndBytes(d)
small := need - (4 << 30) // 4 GiB short; ntfs (min 3 GiB) can give it up
if _, err := planTargetLayout(d, false, small, planOpts{}); err == nil {
t.Fatalf("expected error without --allow-shrink")
}
plan, err := planTargetLayout(d, false, small, planOpts{AllowShrink: true})
if err != nil {
t.Fatalf("with --allow-shrink: %v", err)
}
p3 := plan.Parts[2]
if p3.ShrinkToB == 0 || p3.ShrinkToB >= p3.SrcSizeB {
t.Fatalf("p3 should shrink: %+v", p3)
}
if p3.Method != "fs-image" {
t.Fatalf("shrunk partition must be fs-image, got %s", p3.Method)
}
}
func TestPlanCannotFit(t *testing.T) {
d := mkLayout(40 << 30)
// absurdly small: even shrinking ntfs+ext to their minimums can't help
if _, err := planTargetLayout(d, false, 1<<30, planOpts{AllowShrink: true}); err == nil {
t.Fatalf("expected cannot-fit error")
}
}
func TestPlanFileTargetSizing(t *testing.T) {
d := mkLayout(40 << 30)
plan, err := planTargetLayout(d, true, 0, planOpts{})
if err != nil {
t.Fatal(err)
}
need := lastEndBytes(d)
if plan.ImageSize < need || plan.ImageSize > need+(2<<20) {
t.Fatalf("image size %d not ~%d", plan.ImageSize, need)
}
if plan.Script == "" {
t.Fatalf("expected a partition-table script for a file target")
}
}
func TestPlanNewIDsScript(t *testing.T) {
d := mkLayout(40 << 30)
plan, err := planTargetLayout(d, true, 0, planOpts{NewIDs: true})
if err != nil {
t.Fatal(err)
}
if plan.Script == "" || containsAny(plan.Script, "label-id:", "uuid=") {
t.Fatalf("--new-ids script still has ids:\n%s", plan.Script)
}
}
func containsAny(s string, subs ...string) bool {
for _, x := range subs {
if len(x) > 0 && indexOf(s, x) >= 0 {
return true
}
}
return false
}
func indexOf(s, sub string) int {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return i
}
}
return -1
}

27
syncside.go
View File

@ -81,12 +81,15 @@ type blockHash struct {
hash [32]byte
}
// streamHashBlocks reads f in blockSize-byte blocks up to size, hashing each
// one and handing it to onHash the moment it is computed (this is what lets
// the peer start comparing before the whole side has been scanned). It keeps
// no hash state of its own. onProgress, if non-nil, is called with
// (blocksHashed, totalBlocks) before the first block and after each one.
func streamHashBlocks(f *os.File, size, blockSize, align int64, onHash func(bh blockHash) error, onProgress func(done, total int64)) error {
// streamHashBlocks reads f in blockSize-byte blocks over the window
// [base, base+size), hashing each one and handing it to onHash the moment it
// is computed (this is what lets the peer start comparing before the whole
// side has been scanned). It keeps no hash state of its own. onProgress, if
// non-nil, is called with (blocksHashed, totalBlocks) before the first block
// and after each one. base is 0 for a whole-file sync; clone-disk passes a
// non-zero base to fingerprint just a boot region / offset-addressed
// partition within a larger handle.
func streamHashBlocks(f *os.File, base, size, blockSize, align int64, onHash func(bh blockHash) error, onProgress func(done, total int64)) error {
if err := checkBlockAlign(blockSize, align); err != nil {
return err
}
@ -96,7 +99,7 @@ func streamHashBlocks(f *os.File, size, blockSize, align int64, onHash func(bh b
onProgress(0, blockCount)
}
for i := int64(0); i < blockCount; i++ {
n, err := readBlockAt(f, buf, i*blockSize, size, align)
n, err := readBlockAt(f, buf, base+i*blockSize, base+size, align)
if err != nil {
return fmt.Errorf("hash %s at block %d: %w", f.Name(), i, err)
}
@ -130,7 +133,8 @@ func scanProgressEmitter(out *FrameWriter) func(done, total int64) {
// never read twice.
type sourceLoopParams struct {
File *os.File // already opened for reading by the caller
Size int64
Base int64 // byte offset the window starts at (0 for a whole-file sync)
Size int64 // window length in bytes
BlockSize int64
Align int64
Hashes <-chan blockHash
@ -151,8 +155,8 @@ func runSourceLoop(p sourceLoopParams) error {
f := p.File
buf := make([]byte, p.BlockSize)
for bh := range p.Hashes {
offset := int64(bh.index) * p.BlockSize
n, err := readBlockAt(f, buf, offset, p.Size, p.Align)
offset := p.Base + int64(bh.index)*p.BlockSize
n, err := readBlockAt(f, buf, offset, p.Base+p.Size, p.Align)
if err != nil {
return fmt.Errorf("read %s at block %d: %w", f.Name(), bh.index, err)
}
@ -181,6 +185,7 @@ func runSourceLoop(p sourceLoopParams) error {
// same process, before OnWritten is called).
type destLoopParams struct {
File *os.File // already opened for read/write by the caller
Base int64 // byte offset the window starts at (0 for a whole-file sync)
BlockSize int64
Align int64 // sector alignment for a raw disk handle, else 0/1
In *FrameReader
@ -212,7 +217,7 @@ func runDestLoop(p destLoopParams) error {
}
return fmt.Errorf("block %d: hash mismatch after transfer", index)
}
if err := writeBlockAt(f, block, int64(index)*p.BlockSize, p.Align); err != nil {
if err := writeBlockAt(f, block, p.Base+int64(index)*p.BlockSize, p.Align); err != nil {
if p.AckOut != nil {
_ = p.AckOut.WriteFrame(frameErr, encodeErrFrame(index, err.Error()))
}

16
util.go
View File

@ -2,9 +2,25 @@ package main
import (
"fmt"
"os/exec"
"strings"
"sync"
)
// runCmd runs name with args, returning trimmed combined output.
func runCmd(name string, args ...string) (string, error) {
out, err := exec.Command(name, args...).CombinedOutput()
return strings.TrimSpace(string(out)), err
}
// runCmdStdin is runCmd with a fixed stdin string (for tools that prompt).
func runCmdStdin(stdin, name string, args ...string) (string, error) {
c := exec.Command(name, args...)
c.Stdin = strings.NewReader(stdin)
out, err := c.CombinedOutput()
return strings.TrimSpace(string(out)), err
}
// limitedBuffer keeps only the last maxLen bytes written to it — used to
// capture a bounded tail of a subprocess's stderr for error messages
// without risking unbounded memory growth from a noisy remote command.