Skip to content
Guides & References

Retry a backup on a flaky link

An SIE recipe with a small wrapper that retries any other SIE script a few times before failing, for sites on satellite, cellular or congested links.

Script Integration EngineUpdated
On this page

Retry a backup on a flaky link

Remote sites on satellite, cellular or busy shared links fail backups for reasons that have nothing to do with the device: a dropped packet during login, a session that stalls for a few seconds. Each failure raises an alert and leaves a gap in the history. This recipe is a small wrapper that runs any other SIE script and, if it fails, waits and tries again a few times before giving up. It doesn’t retry login failures, so a wrong password can’t lock an account.


When to use it

  • Backups for some sites fail now and then, and succeed when run again by hand.
  • The failures are timeouts or dropped connections, not authentication or command errors.

If a site fails every time, a retry only delays the alert. Fix the link or the script instead.


How it works

The wrapper takes three settings, then the full command line of the script it wraps:

sie-retry.py <attempts> <wait_seconds> <budget_seconds> <command...>
  1. It runs the wrapped script and captures its stdout and stderr.
  2. A run counts as a success only if it exits 0 and its stderr has an SIE-OK: line. The wrapper then prints that run’s stdout and stderr and exits 0.
  3. If the stderr shows an authentication failure, it stops straight away, so a wrong password isn’t tried again and again.
  4. Otherwise it waits (doubling the wait each time) and tries again, as long as the time budget allows another try.
  5. When it runs out of attempts or time, it prints the last attempt’s stderr and an ERROR: line, and exits 1.

Only the successful attempt’s stdout is printed, so a half-collected config from a failed attempt is never stored.


The template

The wrapper uses the same markers as the script it wraps, so use the wrapped script’s template, with a timeout that covers every attempt:

sie-retry-template.yml
main:
name: "SIE - Netmiko backup with retries"
desc: "Runs netmiko-backup.py up to three times on flaky links"
connect:
timeout: 600 # must exceed the wrapper's budget (540 below)
idletimeout: 600 # the wrapper prints nothing while an attempt runs
idleTimeout: 600
protocol: script
# exit_codes is deliberately left out: any non-zero exit fails the run.
failure_criteria:
error_patterns:
- "ERROR:"
success_patterns:
- "SIE-OK:"

idletimeout is as long as timeout here because the wrapper holds the wrapped script’s output until the attempt finishes, so rConfig sees no output in between.


The command

This wraps the Netmiko recipe: up to three attempts, 30 seconds before the second, 60 before the third, and 540 seconds in all.

/usr/bin/python3 {script_path}sie-retry.py 3 30 540 /opt/rconfig-sie/venv/bin/python {script_path}netmiko-backup.py cisco_ios {device_ip} {device_username} {device_password} {device_enable_password}

The script

Save it as sie-retry.py in the scripts folder.

sie-retry.py
#!/usr/bin/env python3
"""Run another SIE script, retrying on failure, for the rConfig Script
Integration Engine.
Usage:
sie-retry.py <attempts> <wait_seconds> <budget_seconds> <command...>
Prints the stdout and stderr of the first successful attempt. A success is
exit code 0 with an 'SIE-OK:' line on stderr. Authentication failures are not
retried. Exit code 0 on success, 1 when every attempt failed.
"""
import re
import subprocess
import sys
import time
NO_RETRY = re.compile(r"(?i)(authentication failed|access denied|login incorrect|permission denied \(publickey)")
def fail(message):
print(f"ERROR: {message}", file=sys.stderr)
sys.exit(1)
def main():
if len(sys.argv) < 5:
fail("usage: sie-retry.py <attempts> <wait_seconds> <budget_seconds> <command...>")
try:
attempts, wait, budget = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3])
except ValueError:
fail("attempts, wait_seconds and budget_seconds must be whole numbers")
command = sys.argv[4:]
started = time.monotonic()
last = None
for attempt in range(1, attempts + 1):
remaining = budget - (time.monotonic() - started)
if remaining <= 5:
break
try:
last = subprocess.run(command, capture_output=True, text=True, timeout=remaining)
except subprocess.TimeoutExpired:
print(f"attempt {attempt} ran out of time", file=sys.stderr)
break
except OSError as exc:
fail(f"cannot run {command[0]}: {exc}")
if last.returncode == 0 and "SIE-OK:" in last.stderr:
sys.stdout.write(last.stdout)
sys.stderr.write(last.stderr)
if attempt > 1:
print(f"succeeded on attempt {attempt} of {attempts}", file=sys.stderr)
return 0
if NO_RETRY.search(last.stderr):
sys.stderr.write(last.stderr)
fail("authentication failed, not retrying")
print(f"attempt {attempt} of {attempts} failed (exit {last.returncode})", file=sys.stderr)
if attempt < attempts:
time.sleep(min(wait, max(0, budget - (time.monotonic() - started) - 5)))
wait *= 2
if last is not None:
# The wrapped script's ERROR: line, if any, also fails the run.
sys.stderr.write(last.stderr)
fail(f"no successful attempt out of {attempts}")
if __name__ == "__main__":
sys.exit(main())

What gets stored

Exactly what the wrapped script stores on its successful attempt. Nothing from failed attempts.


Adapting it

  • More patterns that shouldn’t be retried. Add them to NO_RETRY, for example a command error your device prints that no retry will fix.
  • Wrapping a script without an SIE-OK: marker. Add one to the script. Without it, the wrapper can’t tell a real success from an empty run that exited 0.

Type to search.