no timeout?
This commit is contained in:
parent
20c9017d3b
commit
10d84909c0
41
README.md
41
README.md
@ -44,13 +44,25 @@ system `ssh` client for remote endpoints.
|
|||||||
is used at all — the source agent spawns the write-side helper as a
|
is used at all — the source agent spawns the write-side helper as a
|
||||||
plain local subprocess.
|
plain local subprocess.
|
||||||
- **Self-deploy:** if a remote endpoint has no runnable `clonetool` on
|
- **Self-deploy:** if a remote endpoint has no runnable `clonetool` on
|
||||||
`PATH` (or wherever `--remote-bin` points), the manager streams *this*
|
`PATH` (or wherever `--remote-bin` points), **or the one that's there is
|
||||||
binary to `~/.clonetool/bin/clonetool` on that host over the existing
|
a different build than this binary** (see below), the manager streams
|
||||||
SSH connection and uses it — no install, no root. Disable with
|
*this* binary to `~/.clonetool/bin/clonetool` on that host over the
|
||||||
`--deploy=false`. If the copied binary won't execute there (wrong CPU
|
existing SSH connection and uses it — no install, no root. Disable with
|
||||||
architecture) the error says so; build one for the remote's arch
|
`--deploy=false` (a stale or missing binary is then an error, except a
|
||||||
|
merely-stale `--remote-bin` is used anyway with a warning, since there's
|
||||||
|
no way to correct it). If the copied binary won't execute there (wrong
|
||||||
|
CPU architecture) the error says so; build one for the remote's arch
|
||||||
(`CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build`) and put it on `PATH`
|
(`CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build`) and put it on `PATH`
|
||||||
or pass `--remote-bin`.
|
or pass `--remote-bin`.
|
||||||
|
- **Version/staleness check:** every binary has a build timestamp baked in
|
||||||
|
(see Build below) and `clonetool version` prints it. Before using a
|
||||||
|
remote `clonetool`, the manager runs its `version` there and compares
|
||||||
|
that timestamp against its own — a mismatch (e.g. the remote was
|
||||||
|
self-deployed from an older build of this tool) is treated the same as
|
||||||
|
"not runnable" and triggers the self-deploy above, so the remote binary
|
||||||
|
is kept in sync with whatever you're running locally. Only the build
|
||||||
|
timestamp is compared, not `GOOS`/`GOARCH` — those are expected to
|
||||||
|
differ across a cross-compiled deploy.
|
||||||
- **sudo:** if reading or writing an endpoint that is a **block device**
|
- **sudo:** if reading or writing an endpoint that is a **block device**
|
||||||
fails with a permission error, `--sudo=auto` (the default) transparently
|
fails with a permission error, `--sudo=auto` (the default) transparently
|
||||||
restarts that side's agent — and the helper it spawns on the peer —
|
restarts that side's agent — and the helper it spawns on the peer —
|
||||||
@ -84,6 +96,14 @@ CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o clonetool-darwin .
|
|||||||
```
|
```
|
||||||
|
|
||||||
`./build.sh` writes all three of the above (plus linux/amd64) into `dist/`.
|
`./build.sh` writes all three of the above (plus linux/amd64) into `dist/`.
|
||||||
|
It also stamps every binary it builds with the same build timestamp (via
|
||||||
|
`-ldflags -X main.buildTime=...`), which is how the manager tells a stale
|
||||||
|
self-deployed remote binary apart from a current one (see "Version/staleness
|
||||||
|
check" above) — build all your binaries for a given release with one
|
||||||
|
`./build.sh` run so they compare equal to each other. A plain `go build`
|
||||||
|
with no `-ldflags` leaves the timestamp at `"dev"`, which still works for
|
||||||
|
this check (it only ever compares equal to another literal copy of the same
|
||||||
|
`dev` binary, never to a real timestamp).
|
||||||
|
|
||||||
Copy the resulting binary to the manager host. Source and destination
|
Copy the resulting binary to the manager host. Source and destination
|
||||||
hosts get it automatically (see self-deploy above), or place it yourself
|
hosts get it automatically (see self-deploy above), or place it yourself
|
||||||
@ -158,8 +178,15 @@ Options:
|
|||||||
| `--remote-bin` | `clonetool` | Path to clonetool on remote hosts. |
|
| `--remote-bin` | `clonetool` | Path to clonetool on remote hosts. |
|
||||||
| `--manager-host` | local hostname | Address a peer should use to reach this machine, needed only when source or dest is local to the manager *and* the other side is remote and ends up needing to dial back in (pull fallback). |
|
| `--manager-host` | local hostname | Address a peer should use to reach this machine, needed only when source or dest is local to the manager *and* the other side is remote and ends up needing to dial back in (pull fallback). |
|
||||||
|
|
||||||
`clonetool version` prints the binary's `GOOS/GOARCH` (used internally for
|
`clonetool version` prints the binary's `GOOS/GOARCH` and build timestamp,
|
||||||
the self-deploy check).
|
e.g. `clonetool linux/amd64 build=2024-06-01T12:00:00Z` (the timestamp is
|
||||||
|
used internally for the self-deploy staleness check above).
|
||||||
|
|
||||||
|
The status line shows read throughput for both sides and write throughput
|
||||||
|
for the destination separately: `rd(src)` is the source reading and
|
||||||
|
comparing its blocks, `rd(dst)` is the destination fingerprinting its
|
||||||
|
current content (only while the trailing `scan H/M` segment is present),
|
||||||
|
and `wr(dst)` is the destination actually writing changed blocks.
|
||||||
|
|
||||||
## Caveats
|
## Caveats
|
||||||
|
|
||||||
|
|||||||
72
agent.go
72
agent.go
@ -231,7 +231,7 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
var pendingMu sync.Mutex
|
var pendingMu sync.Mutex
|
||||||
pending := make(map[uint64]bool)
|
pending := make(map[uint64]bool)
|
||||||
|
|
||||||
var copied, skipped int64
|
var copied, skipped, srcRead int64
|
||||||
var lastProgress time.Time
|
var lastProgress time.Time
|
||||||
maybeProgress := func() {
|
maybeProgress := func() {
|
||||||
if time.Since(lastProgress) < 500*time.Millisecond {
|
if time.Since(lastProgress) < 500*time.Millisecond {
|
||||||
@ -239,11 +239,24 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
}
|
}
|
||||||
lastProgress = time.Now()
|
lastProgress = time.Now()
|
||||||
_ = out.WriteJSON(CtrlMsg{
|
_ = out.WriteJSON(CtrlMsg{
|
||||||
Type: msgProgress, Copied: atomic.LoadInt64(&copied), Skipped: atomic.LoadInt64(&skipped),
|
Type: msgProgress, Phase: "xfer",
|
||||||
|
Copied: atomic.LoadInt64(&copied), Skipped: atomic.LoadInt64(&skipped), SrcRead: atomic.LoadInt64(&srcRead),
|
||||||
TotalBlocks: blockCount,
|
TotalBlocks: blockCount,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// lastActivityNano records the last moment either side made progress: a
|
||||||
|
// frame arriving from the sink (hashes, ACKs, relayed progress) or the
|
||||||
|
// source loop reading and comparing a block. The idle watchdog below
|
||||||
|
// treats a stall as fatal only when *neither* side has moved for a whole
|
||||||
|
// window — otherwise a check-only run, where every block matches and so no
|
||||||
|
// DATA frames and no ACKs are ever exchanged, trips the watchdog purely
|
||||||
|
// because the wire went quiet while the two scans churned through
|
||||||
|
// identical data.
|
||||||
|
var lastActivityNano int64
|
||||||
|
atomic.StoreInt64(&lastActivityNano, time.Now().UnixNano())
|
||||||
|
markActivity := func() { atomic.StoreInt64(&lastActivityNano, time.Now().UnixNano()) }
|
||||||
|
|
||||||
// hashCh is sized to hold every block hash so the reader below never
|
// hashCh is sized to hold every block hash so the reader below never
|
||||||
// blocks handing hashes off (which, since ACKs share the same stream,
|
// blocks handing hashes off (which, since ACKs share the same stream,
|
||||||
// would otherwise be able to deadlock against the in-flight-send limit).
|
// would otherwise be able to deadlock against the in-flight-send limit).
|
||||||
@ -274,6 +287,7 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
}
|
}
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
markActivity()
|
||||||
switch typ {
|
switch typ {
|
||||||
case frameBlockHash:
|
case frameBlockHash:
|
||||||
idx, h, derr := decodeBlockHashFrame(payload)
|
idx, h, derr := decodeBlockHashFrame(payload)
|
||||||
@ -315,8 +329,10 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
go func() {
|
go func() {
|
||||||
sendErrCh <- runSourceLoop(sourceLoopParams{
|
sendErrCh <- runSourceLoop(sourceLoopParams{
|
||||||
File: srcFile, Size: req.Size, BlockSize: req.BlockSize, Align: align, Hashes: hashCh, Out: fw,
|
File: srcFile, Size: req.Size, BlockSize: req.BlockSize, Align: align, Hashes: hashCh, Out: fw,
|
||||||
OnSkip: func(uint64) { atomic.AddInt64(&skipped, 1); maybeProgress() },
|
OnRead: func(uint64) { markActivity(); atomic.AddInt64(&srcRead, 1); maybeProgress() },
|
||||||
|
OnSkip: func(uint64) { markActivity(); atomic.AddInt64(&skipped, 1); maybeProgress() },
|
||||||
OnSend: func(idx uint64, _ [32]byte) {
|
OnSend: func(idx uint64, _ [32]byte) {
|
||||||
|
markActivity()
|
||||||
sem <- struct{}{}
|
sem <- struct{}{}
|
||||||
pendingMu.Lock()
|
pendingMu.Lock()
|
||||||
pending[idx] = true
|
pending[idx] = true
|
||||||
@ -332,7 +348,11 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
sendCh := sendErrCh
|
sendCh := sendErrCh
|
||||||
ackCh := ackEvents
|
ackCh := ackEvents
|
||||||
var fatalErr error
|
var fatalErr error
|
||||||
idle := time.NewTimer(120 * time.Second)
|
// idleTimeout is how long *both* sides can be silent before the transfer
|
||||||
|
// is declared stalled. The timer may fire sooner; a fire is only fatal
|
||||||
|
// when lastActivityNano confirms nothing has moved for the whole window.
|
||||||
|
const idleTimeout = 120 * time.Second
|
||||||
|
idle := time.NewTimer(idleTimeout)
|
||||||
defer idle.Stop()
|
defer idle.Stop()
|
||||||
for sendCh != nil || ackCh != nil {
|
for sendCh != nil || ackCh != nil {
|
||||||
if !idle.Stop() {
|
if !idle.Stop() {
|
||||||
@ -341,7 +361,7 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
idle.Reset(120 * time.Second)
|
idle.Reset(idleTimeout)
|
||||||
select {
|
select {
|
||||||
case sendErr := <-sendCh:
|
case sendErr := <-sendCh:
|
||||||
sendCh = nil
|
sendCh = nil
|
||||||
@ -367,7 +387,11 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
maybeProgress()
|
maybeProgress()
|
||||||
}
|
}
|
||||||
case <-idle.C:
|
case <-idle.C:
|
||||||
fatalErr = fmt.Errorf("timed out waiting for the sink")
|
idleFor := time.Since(time.Unix(0, atomic.LoadInt64(&lastActivityNano)))
|
||||||
|
if idleFor < idleTimeout {
|
||||||
|
continue // progress somewhere within the window; re-arm and wait
|
||||||
|
}
|
||||||
|
fatalErr = fmt.Errorf("timed out waiting for the sink (no activity for %s)", idleFor.Round(time.Second))
|
||||||
}
|
}
|
||||||
if fatalErr != nil {
|
if fatalErr != nil {
|
||||||
break
|
break
|
||||||
@ -384,7 +408,11 @@ func pumpPush(req CtrlMsg, align int64, srcFile *os.File, fw *FrameWriter, fr *F
|
|||||||
return fmt.Errorf("sink closed the connection with %d block write confirmation(s) still outstanding", n)
|
return fmt.Errorf("sink closed the connection with %d block write confirmation(s) still outstanding", n)
|
||||||
}
|
}
|
||||||
|
|
||||||
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Copied: atomic.LoadInt64(&copied), Skipped: atomic.LoadInt64(&skipped), TotalBlocks: blockCount})
|
_ = out.WriteJSON(CtrlMsg{
|
||||||
|
Type: msgProgress, Phase: "xfer",
|
||||||
|
Copied: atomic.LoadInt64(&copied), Skipped: atomic.LoadInt64(&skipped), SrcRead: atomic.LoadInt64(&srcRead),
|
||||||
|
TotalBlocks: blockCount,
|
||||||
|
})
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -462,6 +490,8 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) {
|
|||||||
hashErrCh <- err
|
hashErrCh <- err
|
||||||
}()
|
}()
|
||||||
|
|
||||||
|
var written int64
|
||||||
|
var lastWritten time.Time
|
||||||
loopErr := runDestLoop(destLoopParams{
|
loopErr := runDestLoop(destLoopParams{
|
||||||
File: dstFile, BlockSize: req.BlockSize, Align: align, In: fr,
|
File: dstFile, BlockSize: req.BlockSize, Align: align, In: fr,
|
||||||
OnCtrlMsg: func(m CtrlMsg) {
|
OnCtrlMsg: func(m CtrlMsg) {
|
||||||
@ -469,7 +499,15 @@ func runPullDriver(req CtrlMsg, out *FrameWriter) {
|
|||||||
_ = out.WriteJSON(m)
|
_ = out.WriteJSON(m)
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
OnWrite: func(uint64) {
|
||||||
|
written++
|
||||||
|
if time.Since(lastWritten) >= 500*time.Millisecond {
|
||||||
|
lastWritten = time.Now()
|
||||||
|
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Phase: "write", Written: written})
|
||||||
|
}
|
||||||
|
},
|
||||||
})
|
})
|
||||||
|
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Phase: "write", Written: written})
|
||||||
hashErr := <-hashErrCh
|
hashErr := <-hashErrCh
|
||||||
|
|
||||||
if hashErr != nil {
|
if hashErr != nil {
|
||||||
@ -533,7 +571,19 @@ func runSinkRole(path string, size, blockSize int64) error {
|
|||||||
hashErrCh <- err
|
hashErrCh <- err
|
||||||
}()
|
}()
|
||||||
|
|
||||||
loopErr := runDestLoop(destLoopParams{File: f, BlockSize: blockSize, Align: align, In: in, AckOut: out})
|
var written int64
|
||||||
|
var lastWritten time.Time
|
||||||
|
loopErr := runDestLoop(destLoopParams{
|
||||||
|
File: f, BlockSize: blockSize, Align: align, In: in, AckOut: out,
|
||||||
|
OnWrite: func(uint64) {
|
||||||
|
written++
|
||||||
|
if time.Since(lastWritten) >= 500*time.Millisecond {
|
||||||
|
lastWritten = time.Now()
|
||||||
|
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Phase: "write", Written: written})
|
||||||
|
}
|
||||||
|
},
|
||||||
|
})
|
||||||
|
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Phase: "write", Written: written})
|
||||||
if hashErr := <-hashErrCh; hashErr != nil {
|
if hashErr := <-hashErrCh; hashErr != nil {
|
||||||
return hashErr
|
return hashErr
|
||||||
}
|
}
|
||||||
@ -599,21 +649,23 @@ func runSourceStreamRole(path string, size, blockSize int64) error {
|
|||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
var copied, skipped int64
|
var copied, skipped, srcRead int64
|
||||||
var lastProgress time.Time
|
var lastProgress time.Time
|
||||||
maybeProgress := func() {
|
maybeProgress := func() {
|
||||||
if time.Since(lastProgress) < 500*time.Millisecond {
|
if time.Since(lastProgress) < 500*time.Millisecond {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
lastProgress = time.Now()
|
lastProgress = time.Now()
|
||||||
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Copied: copied, Skipped: skipped, TotalBlocks: blockCount})
|
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Phase: "xfer", Copied: copied, Skipped: skipped, SrcRead: srcRead, TotalBlocks: blockCount})
|
||||||
}
|
}
|
||||||
|
|
||||||
loopErr := runSourceLoop(sourceLoopParams{
|
loopErr := runSourceLoop(sourceLoopParams{
|
||||||
File: f, Size: size, BlockSize: blockSize, Align: align, Hashes: hashCh, Out: out,
|
File: f, Size: size, BlockSize: blockSize, Align: align, Hashes: hashCh, Out: out,
|
||||||
|
OnRead: func(uint64) { srcRead++; maybeProgress() },
|
||||||
OnSkip: func(uint64) { skipped++; maybeProgress() },
|
OnSkip: func(uint64) { skipped++; maybeProgress() },
|
||||||
OnSend: func(uint64, [32]byte) { copied++; maybeProgress() },
|
OnSend: func(uint64, [32]byte) { copied++; maybeProgress() },
|
||||||
})
|
})
|
||||||
|
_ = out.WriteJSON(CtrlMsg{Type: msgProgress, Phase: "xfer", Copied: copied, Skipped: skipped, SrcRead: srcRead, TotalBlocks: blockCount})
|
||||||
if readErr := <-readErrCh; readErr != nil {
|
if readErr := <-readErrCh; readErr != nil {
|
||||||
return readErr
|
return readErr
|
||||||
}
|
}
|
||||||
|
|||||||
5
build.sh
5
build.sh
@ -4,10 +4,13 @@ set -e
|
|||||||
cd "$(dirname "$0")"
|
cd "$(dirname "$0")"
|
||||||
mkdir -p dist
|
mkdir -p dist
|
||||||
|
|
||||||
|
build_time=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||||
|
|
||||||
build() {
|
build() {
|
||||||
os=$1 arch=$2 out=$3
|
os=$1 arch=$2 out=$3
|
||||||
echo "==> $os/$arch -> dist/$out"
|
echo "==> $os/$arch -> dist/$out"
|
||||||
CGO_ENABLED=0 GOOS="$os" GOARCH="$arch" go build -trimpath -ldflags "-s -w" -o "dist/$out" .
|
CGO_ENABLED=0 GOOS="$os" GOARCH="$arch" go build -trimpath \
|
||||||
|
-ldflags "-s -w -X main.buildTime=$build_time" -o "dist/$out" .
|
||||||
}
|
}
|
||||||
|
|
||||||
build linux amd64 clonetool-linux-amd64
|
build linux amd64 clonetool-linux-amd64
|
||||||
|
|||||||
18
ctrlmsg.go
18
ctrlmsg.go
@ -39,13 +39,23 @@ type CtrlMsg struct {
|
|||||||
Reason string `json:"reason,omitempty"`
|
Reason string `json:"reason,omitempty"`
|
||||||
Message string `json:"message,omitempty"`
|
Message string `json:"message,omitempty"`
|
||||||
|
|
||||||
// progress. Scan and transfer now overlap, so a run emits both "scan"
|
// progress. Scan, source read and destination write all overlap and are
|
||||||
// messages (Hashed/TotalBlocks) and transfer messages (Copied/Skipped)
|
// tracked independently, so a run emits several kinds of "progress"
|
||||||
// interleaved; the printer keeps the latest of each.
|
// message distinguished by Phase, interleaved; the printer keeps the
|
||||||
Phase string `json:"phase,omitempty"` // "scan" while hashing the destination; empty during transfer
|
// latest of each:
|
||||||
|
// "scan" - Hashed/TotalBlocks: destination read throughput while it
|
||||||
|
// fingerprints its current content.
|
||||||
|
// "xfer" - Copied/Skipped/SrcRead/TotalBlocks: source read
|
||||||
|
// throughput as it reads and compares each block (Skipped:
|
||||||
|
// matched the destination; Copied: differed and was sent).
|
||||||
|
// "write" - Written: destination write throughput as changed blocks
|
||||||
|
// are actually applied.
|
||||||
|
Phase string `json:"phase,omitempty"`
|
||||||
Hashed int64 `json:"hashed,omitempty"`
|
Hashed int64 `json:"hashed,omitempty"`
|
||||||
Copied int64 `json:"copied,omitempty"`
|
Copied int64 `json:"copied,omitempty"`
|
||||||
Skipped int64 `json:"skipped,omitempty"`
|
Skipped int64 `json:"skipped,omitempty"`
|
||||||
|
SrcRead int64 `json:"srcRead,omitempty"`
|
||||||
|
Written int64 `json:"written,omitempty"`
|
||||||
TotalBlocks int64 `json:"totalBlocks,omitempty"`
|
TotalBlocks int64 `json:"totalBlocks,omitempty"`
|
||||||
BytesCopied int64 `json:"bytesCopied,omitempty"`
|
BytesCopied int64 `json:"bytesCopied,omitempty"`
|
||||||
|
|
||||||
|
|||||||
31
deploy.go
31
deploy.go
@ -15,14 +15,31 @@ import (
|
|||||||
// no install, no root.
|
// no install, no root.
|
||||||
const deployedRemoteBin = ".clonetool/bin/clonetool"
|
const deployedRemoteBin = ".clonetool/bin/clonetool"
|
||||||
|
|
||||||
// resolveRemoteBin returns the path to a runnable clonetool on spec's host.
|
// resolveRemoteBin returns the path to a clonetool on spec's host that is
|
||||||
// It checks the configured --remote-bin first, then a copy deployed by an
|
// both runnable and the same build as this binary (compared via the
|
||||||
// earlier run, and finally streams this binary over and verifies it runs.
|
// "build=" tag in `version` output, see versionString/remoteBuildTag — an
|
||||||
|
// architecture mismatch alone, expected when cross-compiled, doesn't count
|
||||||
|
// as a mismatch). It checks the configured --remote-bin first, then a copy
|
||||||
|
// deployed by an earlier run, and finally streams this binary over and
|
||||||
|
// verifies it runs. With --deploy=false it can't correct a stale or missing
|
||||||
|
// binary, so it uses --remote-bin anyway if merely out of date (only a
|
||||||
|
// non-runnable one is still an error).
|
||||||
func resolveRemoteBin(cfg *SyncConfig, spec Spec, tag string) (string, error) {
|
func resolveRemoteBin(cfg *SyncConfig, spec Spec, tag string) (string, error) {
|
||||||
if _, code, err := runRemote(cfg, spec, cfg.RemoteBin, "version"); err != nil {
|
localTag := remoteBuildTag(versionString())
|
||||||
|
|
||||||
|
if out, code, err := runRemote(cfg, spec, cfg.RemoteBin, "version"); err != nil {
|
||||||
return "", fmt.Errorf("%s: cannot reach %s over ssh: %w", tag, spec.Host, err)
|
return "", fmt.Errorf("%s: cannot reach %s over ssh: %w", tag, spec.Host, err)
|
||||||
} else if code == 0 {
|
} else if code == 0 {
|
||||||
return cfg.RemoteBin, nil
|
if remoteBuildTag(out) == localTag {
|
||||||
|
return cfg.RemoteBin, nil
|
||||||
|
}
|
||||||
|
if !cfg.Deploy {
|
||||||
|
fmt.Fprintf(os.Stderr, "%s: %s on %s reports %q, a different build than this one (%s); continuing anyway (--deploy=false)\n",
|
||||||
|
tag, cfg.RemoteBin, spec.Host, firstLine(out), versionString())
|
||||||
|
return cfg.RemoteBin, nil
|
||||||
|
}
|
||||||
|
fmt.Fprintf(os.Stderr, "%s: %s on %s reports %q, a different build than this one (%s); looking for an up-to-date copy ...\n",
|
||||||
|
tag, cfg.RemoteBin, spec.Host, firstLine(out), versionString())
|
||||||
}
|
}
|
||||||
|
|
||||||
if !cfg.Deploy {
|
if !cfg.Deploy {
|
||||||
@ -30,11 +47,11 @@ func resolveRemoteBin(cfg *SyncConfig, spec Spec, tag string) (string, error) {
|
|||||||
"install clonetool there or pass --remote-bin", tag, cfg.RemoteBin, spec.Host)
|
"install clonetool there or pass --remote-bin", tag, cfg.RemoteBin, spec.Host)
|
||||||
}
|
}
|
||||||
|
|
||||||
if _, code, err := runRemote(cfg, spec, deployedRemoteBin, "version"); err == nil && code == 0 {
|
if out, code, err := runRemote(cfg, spec, deployedRemoteBin, "version"); err == nil && code == 0 && remoteBuildTag(out) == localTag {
|
||||||
return deployedRemoteBin, nil
|
return deployedRemoteBin, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
fmt.Fprintf(os.Stderr, "%s: no runnable clonetool on %s; copying this binary there ...\n", tag, spec.Host)
|
fmt.Fprintf(os.Stderr, "%s: no up-to-date clonetool on %s; copying this binary there ...\n", tag, spec.Host)
|
||||||
if err := deployBinary(cfg, spec); err != nil {
|
if err := deployBinary(cfg, spec); err != nil {
|
||||||
return "", fmt.Errorf("%s: copy clonetool to %s: %w", tag, spec.Host, err)
|
return "", fmt.Errorf("%s: copy clonetool to %s: %w", tag, spec.Host, err)
|
||||||
}
|
}
|
||||||
|
|||||||
3
main.go
3
main.go
@ -4,7 +4,6 @@ import (
|
|||||||
"flag"
|
"flag"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"runtime"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
@ -20,7 +19,7 @@ func main() {
|
|||||||
case "agent":
|
case "agent":
|
||||||
err = cmdAgent(os.Args[2:])
|
err = cmdAgent(os.Args[2:])
|
||||||
case "version", "--version":
|
case "version", "--version":
|
||||||
fmt.Printf("clonetool %s/%s\n", runtime.GOOS, runtime.GOARCH)
|
fmt.Println(versionString())
|
||||||
return
|
return
|
||||||
case "-h", "--help", "help":
|
case "-h", "--help", "help":
|
||||||
usage()
|
usage()
|
||||||
|
|||||||
88
manager.go
88
manager.go
@ -234,28 +234,62 @@ func confirmShrink(dstSpec Spec, oldSize, newSize int64) bool {
|
|||||||
return line == "y" || line == "yes"
|
return line == "y" || line == "yes"
|
||||||
}
|
}
|
||||||
|
|
||||||
// progressPrinter renders a single rewriting status line on stderr. Scan
|
// rateTracker smooths a monotonically increasing block counter into a
|
||||||
// and transfer now run concurrently, so it keeps the latest of each kind of
|
// blocks/sec rate, sampling no more than a few times a second so a burst of
|
||||||
// update (blocks hashed so far, blocks copied/skipped) and shows them on one
|
// same-tick progress messages doesn't produce a noisy instantaneous rate.
|
||||||
// line, with a rolling copy rate; the "scan H/M" segment disappears once the
|
type rateTracker struct {
|
||||||
|
lastT time.Time
|
||||||
|
lastVal int64
|
||||||
|
rate float64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *rateTracker) update(val int64) {
|
||||||
|
now := time.Now()
|
||||||
|
if r.lastT.IsZero() {
|
||||||
|
r.lastT, r.lastVal = now, val
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dt := now.Sub(r.lastT).Seconds()
|
||||||
|
if dt < 0.4 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
inst := float64(val-r.lastVal) / dt
|
||||||
|
if r.rate == 0 {
|
||||||
|
r.rate = inst
|
||||||
|
} else {
|
||||||
|
r.rate = 0.6*r.rate + 0.4*inst
|
||||||
|
}
|
||||||
|
r.lastT, r.lastVal = now, val
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *rateTracker) mib(blockSize int64) float64 {
|
||||||
|
return r.rate * float64(blockSize) / (1024 * 1024)
|
||||||
|
}
|
||||||
|
|
||||||
|
// progressPrinter renders a single rewriting status line on stderr. Scan,
|
||||||
|
// source read and destination write all run concurrently, each tracked by
|
||||||
|
// its own phase-tagged progress message (see CtrlMsg.Phase), so it keeps the
|
||||||
|
// latest of each and shows them on one line with their own rolling
|
||||||
|
// throughput; the "scan H/M" segment and its read rate disappear once the
|
||||||
// destination fingerprint is complete.
|
// destination fingerprint is complete.
|
||||||
type progressPrinter struct {
|
type progressPrinter struct {
|
||||||
blockSize int64
|
blockSize int64
|
||||||
start time.Time
|
active bool
|
||||||
lastT time.Time
|
|
||||||
lastCopied int64
|
|
||||||
rate float64 // copied blocks/sec, smoothed
|
|
||||||
active bool
|
|
||||||
|
|
||||||
hashed int64
|
hashed int64
|
||||||
copied int64
|
copied int64
|
||||||
skipped int64
|
skipped int64
|
||||||
|
srcRead int64
|
||||||
|
written int64
|
||||||
total int64
|
total int64
|
||||||
|
|
||||||
|
dstReadRate rateTracker // destination scan/hash throughput
|
||||||
|
srcReadRate rateTracker // source read/compare throughput
|
||||||
|
dstWriteRate rateTracker // destination write throughput
|
||||||
}
|
}
|
||||||
|
|
||||||
func newProgressPrinter(blockSize int64) *progressPrinter {
|
func newProgressPrinter(blockSize int64) *progressPrinter {
|
||||||
now := time.Now()
|
return &progressPrinter{blockSize: blockSize}
|
||||||
return &progressPrinter{blockSize: blockSize, start: now, lastT: now}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *progressPrinter) print(m CtrlMsg) {
|
func (p *progressPrinter) print(m CtrlMsg) {
|
||||||
@ -263,22 +297,22 @@ func (p *progressPrinter) print(m CtrlMsg) {
|
|||||||
if m.TotalBlocks > 0 {
|
if m.TotalBlocks > 0 {
|
||||||
p.total = m.TotalBlocks
|
p.total = m.TotalBlocks
|
||||||
}
|
}
|
||||||
if m.Phase == "scan" {
|
switch m.Phase {
|
||||||
|
case "scan":
|
||||||
if m.Hashed > p.hashed {
|
if m.Hashed > p.hashed {
|
||||||
p.hashed = m.Hashed
|
p.hashed = m.Hashed
|
||||||
}
|
}
|
||||||
} else {
|
if p.total > 0 && p.hashed >= p.total {
|
||||||
p.copied, p.skipped = m.Copied, m.Skipped
|
p.dstReadRate.rate = 0 // scan finished; stop showing a frozen rate
|
||||||
now := time.Now()
|
} else {
|
||||||
if dt := now.Sub(p.lastT).Seconds(); dt >= 0.4 {
|
p.dstReadRate.update(p.hashed)
|
||||||
inst := float64(m.Copied-p.lastCopied) / dt
|
|
||||||
if p.rate == 0 {
|
|
||||||
p.rate = inst
|
|
||||||
} else {
|
|
||||||
p.rate = 0.6*p.rate + 0.4*inst
|
|
||||||
}
|
|
||||||
p.lastT, p.lastCopied = now, m.Copied
|
|
||||||
}
|
}
|
||||||
|
case "xfer":
|
||||||
|
p.copied, p.skipped, p.srcRead = m.Copied, m.Skipped, m.SrcRead
|
||||||
|
p.srcReadRate.update(p.srcRead)
|
||||||
|
case "write":
|
||||||
|
p.written = m.Written
|
||||||
|
p.dstWriteRate.update(p.written)
|
||||||
}
|
}
|
||||||
|
|
||||||
done := p.copied + p.skipped
|
done := p.copied + p.skipped
|
||||||
@ -286,15 +320,15 @@ func (p *progressPrinter) print(m CtrlMsg) {
|
|||||||
if total <= 0 {
|
if total <= 0 {
|
||||||
total = done
|
total = done
|
||||||
}
|
}
|
||||||
mib := p.rate * float64(p.blockSize) / (1024 * 1024)
|
|
||||||
|
|
||||||
scan := ""
|
scan := ""
|
||||||
if p.total > 0 && p.hashed < p.total {
|
if p.total > 0 && p.hashed < p.total {
|
||||||
scan = fmt.Sprintf(" scan %s/%s", formatCount(p.hashed), formatCount(p.total))
|
scan = fmt.Sprintf(" scan %s/%s", formatCount(p.hashed), formatCount(p.total))
|
||||||
}
|
}
|
||||||
fmt.Fprintf(os.Stderr, "\r syncing %s %s/%s copied=%s skipped=%s%s %6.1f MiB/s ",
|
fmt.Fprintf(os.Stderr, "\r syncing %s %s/%s copied=%s skipped=%s%s rd(src) %5.1f rd(dst) %5.1f wr(dst) %5.1f MiB/s ",
|
||||||
progressBar(done, total), formatCount(done), formatCount(total),
|
progressBar(done, total), formatCount(done), formatCount(total),
|
||||||
formatCount(p.copied), formatCount(p.skipped), scan, mib)
|
formatCount(p.copied), formatCount(p.skipped), scan,
|
||||||
|
p.srcReadRate.mib(p.blockSize), p.dstReadRate.mib(p.blockSize), p.dstWriteRate.mib(p.blockSize))
|
||||||
}
|
}
|
||||||
|
|
||||||
// finish ends the current status line with a newline so following output
|
// finish ends the current status line with a newline so following output
|
||||||
|
|||||||
17
syncside.go
17
syncside.go
@ -135,7 +135,12 @@ type sourceLoopParams struct {
|
|||||||
Align int64
|
Align int64
|
||||||
Hashes <-chan blockHash
|
Hashes <-chan blockHash
|
||||||
Out *FrameWriter
|
Out *FrameWriter
|
||||||
OnSkip func(index uint64)
|
// OnRead is called immediately after a block has been read and hashed
|
||||||
|
// for comparison — whether it turns out to match or differ — so
|
||||||
|
// callers can track true source read throughput independent of the
|
||||||
|
// send/ack timing tracked by OnSkip/OnSend below.
|
||||||
|
OnRead func(index uint64)
|
||||||
|
OnSkip func(index uint64)
|
||||||
// OnSend is called for a differing block immediately before its DATA
|
// OnSend is called for a differing block immediately before its DATA
|
||||||
// frame goes on the wire — never after. The push driver relies on this
|
// frame goes on the wire — never after. The push driver relies on this
|
||||||
// ordering to record the block as awaiting confirmation before the peer
|
// ordering to record the block as awaiting confirmation before the peer
|
||||||
@ -157,6 +162,9 @@ func runSourceLoop(p sourceLoopParams) error {
|
|||||||
return fmt.Errorf("read %s at block %d: %w", f.Name(), bh.index, err)
|
return fmt.Errorf("read %s at block %d: %w", f.Name(), bh.index, err)
|
||||||
}
|
}
|
||||||
hash := sha256.Sum256(buf[:n])
|
hash := sha256.Sum256(buf[:n])
|
||||||
|
if p.OnRead != nil {
|
||||||
|
p.OnRead(bh.index)
|
||||||
|
}
|
||||||
if hash == bh.hash {
|
if hash == bh.hash {
|
||||||
if p.OnSkip != nil {
|
if p.OnSkip != nil {
|
||||||
p.OnSkip(bh.index)
|
p.OnSkip(bh.index)
|
||||||
@ -189,6 +197,10 @@ type destLoopParams struct {
|
|||||||
// pull mode, where the remote source-stream has no other channel back
|
// pull mode, where the remote source-stream has no other channel back
|
||||||
// to the manager for progress updates). Sink-role callers leave it nil.
|
// to the manager for progress updates). Sink-role callers leave it nil.
|
||||||
OnCtrlMsg func(CtrlMsg)
|
OnCtrlMsg func(CtrlMsg)
|
||||||
|
// OnWrite is called right after a block is durably written — the true
|
||||||
|
// destination write-throughput measurement point, as opposed to
|
||||||
|
// whatever "copied" count a driver derives from the send/ack protocol.
|
||||||
|
OnWrite func(index uint64)
|
||||||
}
|
}
|
||||||
|
|
||||||
func runDestLoop(p destLoopParams) error {
|
func runDestLoop(p destLoopParams) error {
|
||||||
@ -218,6 +230,9 @@ func runDestLoop(p destLoopParams) error {
|
|||||||
}
|
}
|
||||||
return fmt.Errorf("write block %d: %w", index, err)
|
return fmt.Errorf("write block %d: %w", index, err)
|
||||||
}
|
}
|
||||||
|
if p.OnWrite != nil {
|
||||||
|
p.OnWrite(index)
|
||||||
|
}
|
||||||
if p.AckOut != nil {
|
if p.AckOut != nil {
|
||||||
if err := p.AckOut.WriteFrame(frameAck, encodeIndexFrame(index)); err != nil {
|
if err := p.AckOut.WriteFrame(frameAck, encodeIndexFrame(index)); err != nil {
|
||||||
return err
|
return err
|
||||||
|
|||||||
35
version.go
Normal file
35
version.go
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"runtime"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// buildTime is stamped at build time via -ldflags "-X main.buildTime=..."
|
||||||
|
// (build.sh does this). A plain "go build" with no ldflags leaves it at
|
||||||
|
// "dev" — still a meaningful, comparable value: two "dev" builds are only
|
||||||
|
// ever the same binary bytes copied around (self-deploy), never two
|
||||||
|
// independently built ones landing on the same value by chance.
|
||||||
|
var buildTime = "dev"
|
||||||
|
|
||||||
|
// versionString is what `clonetool version` prints, and what a remote
|
||||||
|
// host's own `version` output is compared against (see remoteBuildTag) to
|
||||||
|
// tell whether it's running the same build as this binary.
|
||||||
|
func versionString() string {
|
||||||
|
return fmt.Sprintf("clonetool %s/%s build=%s", runtime.GOOS, runtime.GOARCH, buildTime)
|
||||||
|
}
|
||||||
|
|
||||||
|
// remoteBuildTag extracts the "build=..." tag from a clonetool `version`
|
||||||
|
// line, ignoring its GOOS/GOARCH — those are expected to legitimately
|
||||||
|
// differ across a cross-compiled deploy — and any surrounding ssh/shell
|
||||||
|
// noise on other lines. Returns "" if the line isn't a clonetool version
|
||||||
|
// line at all (e.g. "command not found").
|
||||||
|
func remoteBuildTag(out string) string {
|
||||||
|
line := firstLine(out)
|
||||||
|
i := strings.Index(line, "build=")
|
||||||
|
if i < 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(line[i+len("build="):])
|
||||||
|
}
|
||||||
Loading…
Reference in New Issue
Block a user