High Availability guides

High Availability guides

These guides explain how a highly available rConfig deployment is put together: which node layouts exist, which components have to survive a failure, and what each design choice costs you. The step-by-step material stays in the main docs, starting with HA Options Overview; these pages explain the reasoning behind it.

Start from RTO and RPO, not from a topology

There is no single HA architecture for rConfig. The right design follows from two numbers you agree with the business first:

A team that can tolerate a short outage while someone checks the situation needs a very different build from one with an SLA that demands automatic recovery. Pick the objectives, then pick the topology.

Three ways to lay out the application tier

Active/Standby is the default recommendation because it keeps one source of activity at any moment. Active/Active is the right answer only when its extra moving parts are justified by a zero downtime requirement or very high user concurrency. Horizontal scaling is not failover at all: it limits how much of the estate one outage can affect.

The components that must survive a failure

Making the web servers redundant achieves nothing if the data behind them is a single point of failure. A highly available rConfig has five layers to think about.

rConfig itself needs no code changes for database failover. It connects to whatever endpoint .env names, so the replication tooling or proxy is responsible for moving that endpoint to the new primary.

Traffic steering

The failover mechanism decides how quickly users reach the surviving node and how well split-brain is prevented.

• keepalived is the simplest. Both nodes must share a Layer 2 segment, and the firewall must allow VRRP. Add health scripts so it reacts to application failures, not only to a dead host.
• Corosync/Pacemaker adds service ordering and fencing (STONITH) at the cost of much more configuration. It suits teams that already run it.
• Cloud load balancers remove failed nodes from rotation using HTTP or TCP health checks.

Database

The database choice is a trade between simplicity and data loss on failover. Asynchronous MariaDB replication is simple but can lose recent writes. PostgreSQL with synchronous commit, or a three node Galera cluster, avoids that loss but costs write performance or operational complexity. Whatever you choose, size the replica to carry the full production load, because it becomes the primary.

Shared storage

Configuration files are read and written constantly, so storage latency shows up directly in download speed and UI responsiveness. NFS is easy but is itself a single point of failure unless the NFS service is highly available. GlusterFS removes that with replicas but is harder to run. Cloud file services trade cost for less management. Any distributed filesystem needs working file locking.

Redis

Redis holds the job queue, so it has to be treated as stateful. Two facts from the main docs shape a multi-node design:

• One password, set once. When several nodes share one Redis, securing it is done on one node and the password copied to the rest. Each secure run generates a new password, so running it on every node locks out the nodes already updated. See Redis shared with other nodes.
• Never let Redis evict keys. Eviction policies silently drop queued backups and reports. Set a maxmemory limit with noeviction so a runaway queue fails loudly rather than taking the host down. See Redis Memory Optimization.

On EL10, Valkey runs in place of Redis. The same rules apply.

Queues and the scheduler: where background work runs

Every backup, compliance check, snippet deployment and purge runs as a Horizon job, and the scheduler queues tasks every minute from the web server user’s crontab. Where these run is the hardest part of an HA design.

• In Active/Standby, the active node runs the web interface, Horizon workers and scheduled tasks. Only one node dispatches work, so there is no cross-node queue coordination to design.
• In Active/Active, every node that runs workers or the scheduler can pick up the same work. Without a coordination strategy this means duplicate device connections and repeated compliance checks. The main docs list queue worker coordination as a design item that needs Professional Services input.

Running Horizon on a different host from the web server has a visible side effect: the worker_code row on the System Optimization page reports Not measurable, because it cannot compare the workers with the web node’s code. See Horizon Queue Manager.

HA is not a backup

Replication copies mistakes: corruption or an accidental deletion on the primary propagates to every replica. Keep independent backups of the four parts a restore needs: the database, the storage directory, .env, and the APP_KEY inside it. Without the matching key, restored device credentials are unreadable. If the database runs on its own server, protect it on the same schedule as the application server so the two stay in step. See Backups.

Monitoring and testing

An HA design is only as good as the alerts that trigger it and the last time someone tested it. Monitor replication lag, shared storage mounts, Horizon and Supervisor, and database connectivity on every node. rConfig’s own health checks cover the application layer; external monitoring covers the servers. Test failover on a regular schedule and record the actual recovery time against your RTO. See System Monitoring.

Related reference

• HA Options Overview
• Database Optimization
• Redis Memory Optimization
• Redis Authentication
• Horizon Queue Manager
• Backups
• System Monitoring