Skip to content
Guides & References

Back up things that are not network devices

SIE recipes for versioning a database schema, Linux server config files and AWS, Azure or Google Cloud firewall rules, like any device.

Script Integration EngineUpdated
On this page

Back up things that are not network devices

rConfig doesn’t care where the text comes from. If a script prints it and exits cleanly, it’s versioned, diffed and searchable. That makes SIE a simple way to watch the configuration of things around the network: the database an application depends on, the Linux servers that run your DNS or RADIUS, the firewall rules in a cloud account. This page has three short recipes, one for each.

All three use the same template, with one marker each:

sie-not-network-template.yml
main:
name: "SIE - Not a network device"
desc: "Versions the configuration of a database, server or cloud account"
connect:
timeout: 120
idletimeout: 60
idleTimeout: 60
protocol: script
# exit_codes is deliberately left out: any non-zero exit fails the run.
failure_criteria:
error_patterns:
- "ERROR:"
success_patterns:
- "SIE-OK:"

Give each system its own rConfig device record. The IP address is the system’s address, and the name is whatever your team calls it.


A database schema

Stores the structure of a MySQL or MariaDB database (tables, columns, indexes, views and stored procedures) without any of its data. A dropped index or a changed column shows up the same way a changed ACL does on a switch.

The database login goes in an option file in the scripts keys folder, owned by the web user and mode 0600, so the password never appears on a command line:

keys/mysql-schema.cnf
[client]
user=schema_reader
password=example-password

The account only needs to read the schema, never to change it. The exact privileges for dumping views, routines, triggers and events differ between MySQL and MariaDB versions, so check the mysqldump documentation for yours.

Command (the database name is the last word; change it per device):

/bin/bash {script_path}db-schema.sh {device_ip} appdb
db-schema.sh
#!/usr/bin/env bash
#
# Store a MySQL or MariaDB schema, without data, for the rConfig Script
# Integration Engine.
#
# Usage: db-schema.sh <db_host> <database>
set -o pipefail
OPTION_FILE="/var/www/html/rconfig8/current/storage/app/rconfig/scripts/keys/mysql-schema.cnf"
fail() { echo "ERROR: $1" >&2; exit 1; }
HOST="${1:-}"; DB="${2:-}"
[ -n "$HOST" ] && [ -n "$DB" ] || fail "usage: db-schema.sh <db_host> <database>"
[ -r "$OPTION_FILE" ] || fail "cannot read $OPTION_FILE"
echo "dumping the schema of $DB on $HOST" >&2
# --defaults-extra-file must be the first option.
schema=$(mysqldump --defaults-extra-file="$OPTION_FILE" \
--host="$HOST" --no-data --routines --triggers --events \
--skip-dump-date --skip-comments --single-transaction \
"$DB") || fail "mysqldump failed for $DB on $HOST"
# AUTO_INCREMENT counters move as rows are added. They are data, not schema.
schema=$(printf '%s\n' "$schema" | sed -E 's/ AUTO_INCREMENT=[0-9]+//')
[ -n "$schema" ] || fail "the schema for $DB came back empty"
printf '%s\n' "$schema"
echo "SIE-OK: schema of $DB on $HOST" >&2

Linux server config files

Stores a list of configuration files from a Linux server, each under a delimiter line, so a change to named.conf or the RADIUS clients file shows up as a change to that server in rConfig.

It reads the files over SSH with a key. Create a dedicated account on the server that can read only those files, and put its key in the scripts keys folder.

Command:

/bin/bash {script_path}linux-files.sh {device_ip} {device_username}
linux-files.sh
#!/usr/bin/env bash
#
# Store a list of config files from a Linux server, for the rConfig Script
# Integration Engine.
#
# Usage: linux-files.sh <server> <username>
set -o pipefail
KEYS="/var/www/html/rconfig8/current/storage/app/rconfig/scripts/keys"
# Adapt for the server. Only files the account can read.
FILES=(
/etc/named.conf
/etc/chrony.conf
/etc/raddb/clients.conf
/etc/ssh/sshd_config
)
fail() { echo "ERROR: $1" >&2; exit 1; }
HOST="${1:-}"; USER_NAME="${2:-}"
[ -n "$HOST" ] && [ -n "$USER_NAME" ] || fail "usage: linux-files.sh <server> <username>"
for file in "${FILES[@]}"; do
echo "===== $file ====="
ssh -i "$KEYS/files_ed25519" \
-o BatchMode=yes \
-o StrictHostKeyChecking=accept-new \
-o UserKnownHostsFile="$KEYS/known_hosts" \
-o ConnectTimeout=10 \
"$USER_NAME@$HOST" "cat -- '$file'" \
|| fail "cannot read $file on $HOST"
echo
done
echo "SIE-OK: ${#FILES[@]} files from $HOST" >&2

A file the account can’t read fails the whole run. That’s deliberate: a backup that quietly skips the file you care about is worse than one that fails.


Cloud firewall rules (AWS, Azure, Google Cloud)

Stores a cloud account’s firewall rules (AWS security groups, Azure network security groups or Google Cloud VPC firewall rules), flattened to one line per setting and sorted, so an opened port shows up as a single changed line naming the group and the rule.

One script covers all three clouds through their command-line tools: the AWS CLI (aws), the Azure CLI (az) and the Google Cloud CLI (gcloud). Install only the ones you use. Each tool normally keeps its login in the user’s home folder, which the web user usually can’t write to, so the script points each tool at a folder under the scripts keys folder instead.

This script imports the helpers from the cleaning output recipe, so save sie_clean.py in the same folder.

Signing each tool in, once

Give each cloud a read-only identity that can only list firewall rules, then sign the tool in once as the web user (www-data on Ubuntu):

Cloud Identity and permission One-time sign-in
AWS An IAM user or role with ec2:DescribeSecurityGroups Save its key in keys/aws/credentials under a profile name such as [rconfig-readonly]. If rConfig runs on EC2, an instance role needs no file at all.
Azure A service principal with the Reader role on the subscription sudo -u apache env AZURE_CONFIG_DIR=$KEYS/azure az login --service-principal -u <app-id> -p <secret> --tenant <tenant-id>
Google Cloud A service account with the Compute Viewer role, or a custom role holding compute.firewalls.list sudo -u apache env CLOUDSDK_CONFIG=$KEYS/gcloud gcloud auth activate-service-account --key-file=<key.json>

$KEYS is /var/www/html/rconfig8/current/storage/app/rconfig/scripts/keys. Keep those folders mode 0700, owned by the web user.

The commands

The device record’s IP address isn’t used. Make one device per account, subscription or project, with one of these commands:

/usr/bin/python3 {script_path}cloud-firewall.py aws eu-west-1 rconfig-readonly
/usr/bin/python3 {script_path}cloud-firewall.py azure 00000000-0000-0000-0000-000000000000
/usr/bin/python3 {script_path}cloud-firewall.py gcp example-project-id

The script

cloud-firewall.py
#!/usr/bin/env python3
"""Store a cloud account's firewall rules, flattened and sorted, for the rConfig
Script Integration Engine.
Usage:
cloud-firewall.py aws <region> <profile>
cloud-firewall.py azure <subscription_id>
cloud-firewall.py gcp <project_id>
"""
import json
import os
import subprocess
import sys
import sie_clean
KEYS = "/var/www/html/rconfig8/current/storage/app/rconfig/scripts/keys"
TOOLS = {"aws": "/usr/local/bin/aws", "azure": "/usr/bin/az", "gcp": "/usr/bin/gcloud"} # check with: which
# Fields that change without a rule changing.
DROP = {"etag", "provisioningState", "resourceGuid", "creationTimestamp", "fingerprint"}
def fail(message):
print(f"ERROR: {message}", file=sys.stderr)
sys.exit(1)
def run(cloud, args):
env = dict(os.environ)
env.update({
"AWS_SHARED_CREDENTIALS_FILE": f"{KEYS}/aws/credentials",
"AWS_CONFIG_FILE": f"{KEYS}/aws/config",
"AWS_PAGER": "",
"AZURE_CONFIG_DIR": f"{KEYS}/azure",
"CLOUDSDK_CONFIG": f"{KEYS}/gcloud",
})
try:
result = subprocess.run([TOOLS[cloud], *args], capture_output=True, text=True, env=env, timeout=120)
except FileNotFoundError:
fail(f"{TOOLS[cloud]} is not installed")
if result.returncode != 0:
lines = result.stderr.strip().splitlines()
fail(f"{cloud} cli failed: {lines[-1] if lines else 'no error text'}")
return json.loads(result.stdout or "null")
def tidy(value):
if isinstance(value, dict):
return {k: tidy(v) for k, v in value.items() if k not in DROP}
if isinstance(value, list):
return [tidy(v) for v in value]
return value
def main():
if len(sys.argv) < 3 or sys.argv[1] not in TOOLS:
fail("usage: cloud-firewall.py aws <region> <profile> | azure <subscription> | gcp <project>")
cloud, scope = sys.argv[1], sys.argv[2]
if cloud == "aws":
if len(sys.argv) < 4:
fail("usage: cloud-firewall.py aws <region> <profile>")
data = run(cloud, ["ec2", "describe-security-groups", "--region", scope,
"--profile", sys.argv[3], "--output", "json"])
groups = {g["GroupId"]: g for g in (data or {}).get("SecurityGroups", [])}
rule_lists = ("IpPermissions", "IpPermissionsEgress")
elif cloud == "azure":
data = run(cloud, ["network", "nsg", "list", "--subscription", scope, "--output", "json"])
groups = {g["name"]: g for g in data or []}
rule_lists = ("securityRules", "defaultSecurityRules")
else:
data = run(cloud, ["compute", "firewall-rules", "list", "--project", scope, "--format", "json"])
groups = {g["name"]: g for g in data or []}
rule_lists = ()
if not groups:
fail(f"no firewall rules returned for {cloud} {scope}, refusing to store an empty backup")
# Rules in these clouds have no list order (priority is a field), so sorting is safe.
for group in groups.values():
for key in rule_lists:
group[key] = sorted(group.get(key, []), key=lambda r: json.dumps(r, sort_keys=True))
sys.stdout.write(sie_clean.flatten_json_text(tidy(groups)))
print(f"SIE-OK: {len(groups)} {cloud} firewall groups or rules in {scope}", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())

Groups are keyed by ID or name, so adding one doesn’t renumber the others. What stored lines look like:

sg-0a1b2c3d4e5f60001.GroupName = "web-frontend"
sg-0a1b2c3d4e5f60001.IpPermissions[0].FromPort = 443
sg-0a1b2c3d4e5f60001.IpPermissions[0].IpRanges[0].CidrIp = "203.0.113.0/24"
allow-https-web.direction = "INGRESS"
allow-https-web.sourceRanges[0] = "198.51.100.0/24"

Type to search.