Skip to content
Guides & References

Back up an API that sends results in pages

An SIE recipe that follows every page of a REST API, through Link headers or a next field, and stores one complete, sorted result.

Script Integration EngineUpdated
On this page

Back up an API that sends results in pages

Many APIs return a long list a page at a time: the first 100 items, then a link to the next 100. API Collections stores one response per request, so it only ever sees the first page. This recipe follows the links until the last page, joins the results and stores them as one sorted document.


When to use it

  • An API list endpoint returns results in pages, and the backup must hold all of them.
  • The API is reached at the device’s address with a token, which suits controllers, orchestrators and inventory systems.

APIs page in two common ways, and the script handles both:

Style How the next page is given Seen in
Link header A Link: <url>; rel="next" response header Meraki, GitHub and many others
Next field A next URL in the JSON body, beside a results list NetBox, Django REST APIs

Check your API’s documentation for which one it uses.


Where the token goes

Device field What to enter
IP address The API host’s address or name
Password An API token with read-only access
Username Not used by this script

If the API uses a certificate from your own certificate authority, save the CA certificate as keys/api-ca.pem in the scripts folder. The script then checks the certificate against it. Don’t turn certificate checks off: the token is sent with every request.


The template

sie-paginated-api-template.yml
main:
name: "SIE - Paginated API backup"
desc: "Follows every page of a REST API list and stores the joined result"
connect:
timeout: 300
idletimeout: 60
idleTimeout: 60
protocol: script
# exit_codes is deliberately left out: any non-zero exit fails the run.
failure_criteria:
error_patterns:
- "ERROR:"
success_patterns:
- "SIE-OK: paginated"

The command

/usr/bin/python3 {script_path}paginated-api-backup.py {device_ip} {device_password}

The script

Save it as paginated-api-backup.py in the scripts folder. Change the settings at the top for your API.

paginated-api-backup.py
#!/usr/bin/env python3
"""Collect every page of one or more REST API lists for the rConfig
Script Integration Engine.
Usage:
paginated-api-backup.py <api_host> <api_token>
The joined, sorted result goes to stdout as JSON. Progress, errors and the
success marker go to stderr. Exit code 0 on success, 1 on any failure.
Standard library only.
"""
import json
import os
import re
import ssl
import sys
import urllib.error
import urllib.request
# ---- Settings to adapt for your API ------------------------------------------
BASE = "https://{host}/api"
ENDPOINTS = { # name in the backup -> first page path
"devices": "/dcim/devices/?limit=100",
"vlans": "/ipam/vlans/?limit=100",
}
AUTH_HEADER = ("Authorization", "Token {token}") # or ("Authorization", "Bearer {token}")
SORT_KEY = "id" # field that identifies an item; used to sort
DROP_FIELDS = {"last_updated", "url", "display"} # fields that change without a config change
MAX_PAGES = 500 # stops a paging loop that never ends
CA_FILE = "/var/www/html/rconfig8/current/storage/app/rconfig/scripts/keys/api-ca.pem"
# ----------------------------------------------------------------------------
LINK_NEXT = re.compile(r'<([^>]+)>\s*;\s*rel="?next"?')
def note(message):
print(message, file=sys.stderr)
def fail(message):
print(f"ERROR: {message}", file=sys.stderr)
sys.exit(1)
def tls_context():
if os.path.exists(CA_FILE):
return ssl.create_default_context(cafile=CA_FILE)
return ssl.create_default_context()
def fetch(url, headers, context):
"""GET one page. Returns the decoded body and the response headers."""
request = urllib.request.Request(url, headers=headers)
try:
with urllib.request.urlopen(request, timeout=30, context=context) as response:
return json.loads(response.read().decode("utf-8")), response.headers
except urllib.error.HTTPError as e:
fail(f"HTTP {e.code} for {url}")
except urllib.error.URLError as e:
fail(f"cannot reach {url}: {e.reason}")
except json.JSONDecodeError:
fail(f"the response for {url} was not valid JSON")
def collect(first_url, headers, context):
"""Follow every page and return all the items."""
items, url, pages = [], first_url, 0
while url:
pages += 1
if pages > MAX_PAGES:
fail(f"more than {MAX_PAGES} pages from {first_url}, stopping")
body, response_headers = fetch(url, headers, context)
if isinstance(body, dict) and "results" in body: # next field style
items.extend(body["results"])
url = body.get("next")
elif isinstance(body, list): # Link header style
items.extend(body)
match = LINK_NEXT.search(response_headers.get("Link", ""))
url = match.group(1) if match else None
else:
fail(f"unexpected response shape from {url}")
note(f"page {pages}: {len(items)} items so far")
return items
def tidy(item):
if isinstance(item, dict):
return {k: tidy(v) for k, v in item.items() if k not in DROP_FIELDS}
if isinstance(item, list):
return [tidy(v) for v in item]
return item
def main():
if len(sys.argv) < 3:
fail("usage: paginated-api-backup.py <api_host> <api_token>")
host, token = sys.argv[1], sys.argv[2]
base = BASE.format(host=host)
headers = {
AUTH_HEADER[0]: AUTH_HEADER[1].format(token=token),
"Accept": "application/json",
"User-Agent": "rconfig-sie-recipe/1.0",
}
context = tls_context()
backup = {}
for name, path in ENDPOINTS.items():
note(f"collecting {name}")
items = [tidy(i) for i in collect(base + path, headers, context)]
backup[name] = sorted(items, key=lambda i: str(i.get(SORT_KEY, "")))
if not any(backup.values()):
fail("every list came back empty, refusing to store an empty backup")
json.dump(backup, sys.stdout, indent=2, sort_keys=True)
sys.stdout.write("\n")
counts = ", ".join(f"{k}={len(v)}" for k, v in backup.items())
note(f"SIE-OK: paginated {counts}")
return 0
if __name__ == "__main__":
sys.exit(main())

What gets stored

One JSON document with every item from every page, sorted by ID, with the fields that change on their own removed:

{
"devices": [
{
"id": 1,
"name": "edge-rtr-01",
"primary_ip4": "192.0.2.1/24",
"status": "active"
},

Adapting it

  • Page size. A larger page size (limit, per_page or perPage, depending on the API) means fewer requests. Stay within the API’s maximum.
  • Offset paging. Some APIs only take offset and limit, with no next link. Add a loop that raises offset by limit until a page comes back shorter than limit.
  • Rate limits. If the API returns 429, add the wait-and-retry from the REST API recipe.
  • Flat output. For one line per setting, pass backup through flatten_json_text from the cleaning output recipe.

Type to search.