Back up an API that sends results in pages
Many APIs return a long list a page at a time: the first 100 items, then a link to the next 100. API Collections stores one response per request, so it only ever sees the first page. This recipe follows the links until the last page, joins the results and stores them as one sorted document.
When to use it
- An API list endpoint returns results in pages, and the backup must hold all of them.
- The API is reached at the device’s address with a token, which suits controllers, orchestrators and inventory systems.
APIs page in two common ways, and the script handles both:
| Style | How the next page is given | Seen in |
|---|---|---|
| Link header | A Link: <url>; rel="next" response header |
Meraki, GitHub and many others |
| Next field | A next URL in the JSON body, beside a results list |
NetBox, Django REST APIs |
Check your API’s documentation for which one it uses.
Where the token goes
| Device field | What to enter |
|---|---|
| IP address | The API host’s address or name |
| Password | An API token with read-only access |
| Username | Not used by this script |
If the API uses a certificate from your own certificate authority, save the CA certificate as keys/api-ca.pem in the scripts folder. The script then checks the certificate against it. Don’t turn certificate checks off: the token is sent with every request.
The template
main: name: "SIE - Paginated API backup" desc: "Follows every page of a REST API list and stores the joined result"
connect: timeout: 300 idletimeout: 60 idleTimeout: 60 protocol: script
# exit_codes is deliberately left out: any non-zero exit fails the run.failure_criteria: error_patterns: - "ERROR:" success_patterns: - "SIE-OK: paginated"The command
/usr/bin/python3 {script_path}paginated-api-backup.py {device_ip} {device_password}The script
Save it as paginated-api-backup.py in the scripts folder. Change the settings at the top for your API.
#!/usr/bin/env python3"""Collect every page of one or more REST API lists for the rConfigScript Integration Engine.
Usage: paginated-api-backup.py <api_host> <api_token>
The joined, sorted result goes to stdout as JSON. Progress, errors and thesuccess marker go to stderr. Exit code 0 on success, 1 on any failure.Standard library only."""
import jsonimport osimport reimport sslimport sysimport urllib.errorimport urllib.request
# ---- Settings to adapt for your API ------------------------------------------BASE = "https://{host}/api"ENDPOINTS = { # name in the backup -> first page path "devices": "/dcim/devices/?limit=100", "vlans": "/ipam/vlans/?limit=100",}AUTH_HEADER = ("Authorization", "Token {token}") # or ("Authorization", "Bearer {token}")SORT_KEY = "id" # field that identifies an item; used to sortDROP_FIELDS = {"last_updated", "url", "display"} # fields that change without a config changeMAX_PAGES = 500 # stops a paging loop that never endsCA_FILE = "/var/www/html/rconfig8/current/storage/app/rconfig/scripts/keys/api-ca.pem"# ----------------------------------------------------------------------------
LINK_NEXT = re.compile(r'<([^>]+)>\s*;\s*rel="?next"?')
def note(message): print(message, file=sys.stderr)
def fail(message): print(f"ERROR: {message}", file=sys.stderr) sys.exit(1)
def tls_context(): if os.path.exists(CA_FILE): return ssl.create_default_context(cafile=CA_FILE) return ssl.create_default_context()
def fetch(url, headers, context): """GET one page. Returns the decoded body and the response headers.""" request = urllib.request.Request(url, headers=headers) try: with urllib.request.urlopen(request, timeout=30, context=context) as response: return json.loads(response.read().decode("utf-8")), response.headers except urllib.error.HTTPError as e: fail(f"HTTP {e.code} for {url}") except urllib.error.URLError as e: fail(f"cannot reach {url}: {e.reason}") except json.JSONDecodeError: fail(f"the response for {url} was not valid JSON")
def collect(first_url, headers, context): """Follow every page and return all the items.""" items, url, pages = [], first_url, 0 while url: pages += 1 if pages > MAX_PAGES: fail(f"more than {MAX_PAGES} pages from {first_url}, stopping") body, response_headers = fetch(url, headers, context)
if isinstance(body, dict) and "results" in body: # next field style items.extend(body["results"]) url = body.get("next") elif isinstance(body, list): # Link header style items.extend(body) match = LINK_NEXT.search(response_headers.get("Link", "")) url = match.group(1) if match else None else: fail(f"unexpected response shape from {url}")
note(f"page {pages}: {len(items)} items so far") return items
def tidy(item): if isinstance(item, dict): return {k: tidy(v) for k, v in item.items() if k not in DROP_FIELDS} if isinstance(item, list): return [tidy(v) for v in item] return item
def main(): if len(sys.argv) < 3: fail("usage: paginated-api-backup.py <api_host> <api_token>") host, token = sys.argv[1], sys.argv[2]
base = BASE.format(host=host) headers = { AUTH_HEADER[0]: AUTH_HEADER[1].format(token=token), "Accept": "application/json", "User-Agent": "rconfig-sie-recipe/1.0", } context = tls_context()
backup = {} for name, path in ENDPOINTS.items(): note(f"collecting {name}") items = [tidy(i) for i in collect(base + path, headers, context)] backup[name] = sorted(items, key=lambda i: str(i.get(SORT_KEY, "")))
if not any(backup.values()): fail("every list came back empty, refusing to store an empty backup")
json.dump(backup, sys.stdout, indent=2, sort_keys=True) sys.stdout.write("\n") counts = ", ".join(f"{k}={len(v)}" for k, v in backup.items()) note(f"SIE-OK: paginated {counts}") return 0
if __name__ == "__main__": sys.exit(main())What gets stored
One JSON document with every item from every page, sorted by ID, with the fields that change on their own removed:
{"devices": [{"id": 1,"name": "edge-rtr-01","primary_ip4": "192.0.2.1/24","status": "active"},Adapting it
- Page size. A larger page size (
limit,per_pageorperPage, depending on the API) means fewer requests. Stay within the API’s maximum. - Offset paging. Some APIs only take
offsetandlimit, with no next link. Add a loop that raisesoffsetbylimituntil a page comes back shorter thanlimit. - Rate limits. If the API returns
429, add the wait-and-retry from the REST API recipe. - Flat output. For one line per setting, pass
backupthroughflatten_json_textfrom the cleaning output recipe.