Retry a backup on a flaky link
Remote sites on satellite, cellular or busy shared links fail backups for reasons that have nothing to do with the device: a dropped packet during login, a session that stalls for a few seconds. Each failure raises an alert and leaves a gap in the history. This recipe is a small wrapper that runs any other SIE script and, if it fails, waits and tries again a few times before giving up. It doesn’t retry login failures, so a wrong password can’t lock an account.
When to use it
- Backups for some sites fail now and then, and succeed when run again by hand.
- The failures are timeouts or dropped connections, not authentication or command errors.
If a site fails every time, a retry only delays the alert. Fix the link or the script instead.
How it works
The wrapper takes three settings, then the full command line of the script it wraps:
sie-retry.py <attempts> <wait_seconds> <budget_seconds> <command...>- It runs the wrapped script and captures its stdout and stderr.
- A run counts as a success only if it exits
0and its stderr has anSIE-OK:line. The wrapper then prints that run’s stdout and stderr and exits0. - If the stderr shows an authentication failure, it stops straight away, so a wrong password isn’t tried again and again.
- Otherwise it waits (doubling the wait each time) and tries again, as long as the time budget allows another try.
- When it runs out of attempts or time, it prints the last attempt’s stderr and an
ERROR:line, and exits1.
Only the successful attempt’s stdout is printed, so a half-collected config from a failed attempt is never stored.
The template
The wrapper uses the same markers as the script it wraps, so use the wrapped script’s template, with a timeout that covers every attempt:
main: name: "SIE - Netmiko backup with retries" desc: "Runs netmiko-backup.py up to three times on flaky links"
connect: timeout: 600 # must exceed the wrapper's budget (540 below) idletimeout: 600 # the wrapper prints nothing while an attempt runs idleTimeout: 600 protocol: script
# exit_codes is deliberately left out: any non-zero exit fails the run.failure_criteria: error_patterns: - "ERROR:" success_patterns: - "SIE-OK:"idletimeout is as long as timeout here because the wrapper holds the wrapped script’s output until the attempt finishes, so rConfig sees no output in between.
The command
This wraps the Netmiko recipe: up to three attempts, 30 seconds before the second, 60 before the third, and 540 seconds in all.
/usr/bin/python3 {script_path}sie-retry.py 3 30 540 /opt/rconfig-sie/venv/bin/python {script_path}netmiko-backup.py cisco_ios {device_ip} {device_username} {device_password} {device_enable_password}The script
Save it as sie-retry.py in the scripts folder.
#!/usr/bin/env python3"""Run another SIE script, retrying on failure, for the rConfig ScriptIntegration Engine.
Usage: sie-retry.py <attempts> <wait_seconds> <budget_seconds> <command...>
Prints the stdout and stderr of the first successful attempt. A success isexit code 0 with an 'SIE-OK:' line on stderr. Authentication failures are notretried. Exit code 0 on success, 1 when every attempt failed."""
import reimport subprocessimport sysimport time
NO_RETRY = re.compile(r"(?i)(authentication failed|access denied|login incorrect|permission denied \(publickey)")
def fail(message): print(f"ERROR: {message}", file=sys.stderr) sys.exit(1)
def main(): if len(sys.argv) < 5: fail("usage: sie-retry.py <attempts> <wait_seconds> <budget_seconds> <command...>") try: attempts, wait, budget = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]) except ValueError: fail("attempts, wait_seconds and budget_seconds must be whole numbers") command = sys.argv[4:]
started = time.monotonic() last = None for attempt in range(1, attempts + 1): remaining = budget - (time.monotonic() - started) if remaining <= 5: break try: last = subprocess.run(command, capture_output=True, text=True, timeout=remaining) except subprocess.TimeoutExpired: print(f"attempt {attempt} ran out of time", file=sys.stderr) break except OSError as exc: fail(f"cannot run {command[0]}: {exc}")
if last.returncode == 0 and "SIE-OK:" in last.stderr: sys.stdout.write(last.stdout) sys.stderr.write(last.stderr) if attempt > 1: print(f"succeeded on attempt {attempt} of {attempts}", file=sys.stderr) return 0
if NO_RETRY.search(last.stderr): sys.stderr.write(last.stderr) fail("authentication failed, not retrying")
print(f"attempt {attempt} of {attempts} failed (exit {last.returncode})", file=sys.stderr) if attempt < attempts: time.sleep(min(wait, max(0, budget - (time.monotonic() - started) - 5))) wait *= 2
if last is not None: # The wrapped script's ERROR: line, if any, also fails the run. sys.stderr.write(last.stderr) fail(f"no successful attempt out of {attempts}")
if __name__ == "__main__": sys.exit(main())What gets stored
Exactly what the wrapped script stores on its successful attempt. Nothing from failed attempts.
Adapting it
- More patterns that shouldn’t be retried. Add them to
NO_RETRY, for example a command error your device prints that no retry will fix. - Wrapping a script without an
SIE-OK:marker. Add one to the script. Without it, the wrapper can’t tell a real success from an empty run that exited0.