Clean script output so diffs show real changes
A backup that changes on every run hides the run that matters. Scripts are especially prone to it, because they collect from sources that were never designed to be diffed: APIs that return keys in any order, exports stamped with the time, status lines with counters. This recipe gives you a small Python module of cleaning helpers to import into any SIE script, and explains when to clean in the script and when to use rConfig’s diff exclusions instead.
Clean in the script, or exclude in rConfig?
rConfig’s diff exclusions hide lines when versions are compared. The lines are still stored, so they can still be searched and read. Cleaning in the script removes them before rConfig sees them.
| Use a diff exclusion when | Clean in the script when |
|---|---|
| The line is worth keeping, but not worth a change alert (uptime, a last-change timestamp). | Nobody will ever want the line (an API request ID, an export’s own timestamp). |
| You want the same rule on SSH and script devices alike. | The problem is ordering, which an exclusion can’t fix. |
| You don’t want to touch a script that works. | The structure needs changing, such as flattening nested JSON. |
Ordering is the one problem only the script can fix. An exclusion hides lines, it can’t put them back in the same order every time.
The helpers
Save this as sie_clean.py in the scripts folder. Python looks for imports in the script’s own folder first, so any script there can use import sie_clean.
"""Cleaning helpers for rConfig Script Integration Engine scripts.
Import from a script in the same folder:
import sie_clean text = sie_clean.drop_lines(text, sie_clean.COMMON_VOLATILE)"""
import jsonimport re
# Patterns for lines that change on every run on many platforms. Extend per device.COMMON_VOLATILE = [ r"^\s*! Last configuration change at ", r"^\s*! NVRAM config last updated at ", r"^\s*Building configuration", r"^\s*Current configuration\s*:", r"(?i)\buptime is\b", r"(?i)^\s*#?\s*(generated|exported|created) (on|at)\b",]
def normalise(text): """Use \\n line endings, remove trailing spaces and blank lines at either end.""" lines = text.replace("\r\n", "\n").replace("\r", "\n").split("\n") return "\n".join(line.rstrip() for line in lines).strip("\n") + "\n"
def drop_lines(text, patterns): """Remove every line that matches any of the regular expressions.""" compiled = [re.compile(p) for p in patterns] kept = [line for line in text.splitlines() if not any(c.search(line) for c in compiled)] return "\n".join(kept) + "\n"
def mask(text, pattern, replacement="<removed>"): """Keep a line but replace the part that changes, such as a counter or a token.""" return re.sub(pattern, replacement, text)
def sort_section(text, start, end): """Sort the lines between a start marker line and an end marker line.
For lists whose order means nothing, such as an address list a device prints in a different order each time. Never use it on ordered lists such as ACLs or firewall rules, where the order is the configuration. """ lines = text.splitlines() out, block, inside = [], [], False for line in lines: if not inside and re.search(start, line): inside = True out.append(line) continue if inside and re.search(end, line): out.extend(sorted(block)) block, inside = [], False out.append(line) continue (block if inside else out).append(line) out.extend(sorted(block) if inside else block) return "\n".join(out) + "\n"
def flatten_json(data, prefix=""): """Turn nested JSON into one 'path = value' line per setting, in a stable order.
rConfig trims the indentation from every stored line, so nested JSON loses its shape. Flat lines keep the full path to each value, which diffs, searches and policy checks read far more easily. """ lines = [] if isinstance(data, dict): for key in sorted(data): lines.extend(flatten_json(data[key], f"{prefix}.{key}" if prefix else str(key))) elif isinstance(data, list): for index, item in enumerate(data): lines.extend(flatten_json(item, f"{prefix}[{index}]")) else: lines.append(f"{prefix} = {json.dumps(data)}") return lines
def flatten_json_text(data): return "\n".join(flatten_json(data)) + "\n"Using the helpers in a script
A script that collects text from a device, then cleans it before printing:
import sys
import sie_clean
raw = collect_from_device() # your collection code
text = sie_clean.normalise(raw)text = sie_clean.drop_lines(text, sie_clean.COMMON_VOLATILE + [ r"^\s*ntp clock-period ", # drifts on some Cisco platforms])text = sie_clean.mask(text, r"(session-id )\d+", r"\1<removed>")
sys.stdout.write(text)print("SIE-OK: collection complete", file=sys.stderr)A script that collects JSON from an API and stores it flattened:
import sys
import sie_clean
data = fetch_from_api() # your collection code, returning parsed JSONdata.pop("requestId", None) # response metadata, not configuration
sys.stdout.write(sie_clean.flatten_json_text(data))print("SIE-OK: collection complete", file=sys.stderr)What flattening looks like
The same settings as nested JSON and as flattened lines:
{ "ntp": { "servers": ["192.0.2.123", "198.51.100.123"] }, "snmp": { "enabled": true, "location": "Dublin DC1" }}ntp.servers[0] = "192.0.2.123"ntp.servers[1] = "198.51.100.123"snmp.enabled = truesnmp.location = "Dublin DC1"A change to the SNMP location now shows as one changed line that names the setting, rather than a bare value somewhere in a block of braces. A policy can check snmp.enabled = true with a plain line match.
The trade-off: list items are numbered, so inserting an item at the top of a list renumbers every item after it. For lists whose order means nothing, sort them before flattening.