Data Purge, Config Retention & History Repair in rConfig 8.1

Data Purge, Config Retention & History Repair (rConfig 8.1)

rConfig 8.1 introduces a modern, highly controlled approach to database cleanup, configuration file retention, and self-healing repair of configuration history. This system replaces multiple older purge commands from rConfig 7.x and earlier.

Important

• The old commands rconfig:purge-configs and rconfig:purge-configs-v2 are deprecated.
• Any scheduled cron tasks referencing these commands must be removed.
• rConfig 8.1 automatically disables legacy purge tasks at runtime and removes any matching task records during upgrade.

All retention, cleanup, and repair functionality now lives in three coordinated subsystems:

1. Data Purge Service (generic tables)
2. Config Retention Service (configs + config_changes + file cleanup)
3. Config History Repair Service (repairs broken version sequences)

You control all of this from:

Settings → Data Purge (/settings/data-purge)

This UI lets you:

• enable/disable purge per table
• change retention days
• configure table-specific options
• inspect Purge Run History
• drill into individual runs to see counts, errors and context
• manually trigger repair tools via CLI using the same configuration

---

Overview of the Three Components

1. DataPurgeService

Handles all non-config tables such as:

• activity_log
• jobs
• failed_jobs
• agent_logs (Vector)
• agent_queues (Vector)
• other high-volume, timestamped tables

This service respects:

• retention days
• batch size
• queue thresholds
• cleanup frequency (daily/weekly/monthly)
• dry-run mode
• queue flags (--force-queue, --no-queue, --queue-threshold)

Workflow (non-config tables)

Notes

• Non-config tables may be deleted directly or via queued jobs.
• rConfig never queues purge jobs when running with --dry-run.
• Purge runs are recorded and visible under Settings → Data Purge → Purge Runs.

---

2. ConfigRetentionService

A specialised retention engine dedicated to:

• configs
• config_changes
• non-downloaded configs (config_downloaded = 0)
• configuration file deletion in storage
• “protected version” logic per device/command

Default Behaviour

• The configs purge setting is disabled by default in 8.1.
• Admins must explicitly enable config retention under Settings → Data Purge.
• When disabled, no config records or files are removed by the retention engine.

Key Safety Features

• Always keeps the final latest_version = 1 config for each device/command.
• Ignores disabled devices – devices with status = STATUS_DISABLED are excluded from retention and repair. Their history is preserved.
• Keeps the last N versions per device+command (min_versions_per_device_command).
• Deletes corresponding config_changes safely and in the correct order.
• Validates timestamp cutoffs and respects separate retention for non-downloaded configs.
• Never queues config purge jobs (always direct execution, regardless of queue flags).

Workflow

The resulting report is:

• Aggregated back into DataPurgeService so the purge run summary remains consistent.
• Recorded into the Purge Runs table with a specific context (e.g. config_retention).

---

3. ConfigHistoryRepairService

This engine repairs broken version chains, which can happen after:

• older purge logic from pre-8.x
• manual DB operations
• long-term retention combined with purges
• version resets (e.g. 254 → 1,2,3…) after legacy purges

What It Fixes

• Incorrect latest_version flags on configs

• Gaps or resets in version sequences

• Missing or invalid previous_config_id links

• config_changes.config_version mismatches vs configs.config_version

• Chains that restart after purges (e.g., version jumps from 254 → 1,2,3…)

• Inconsistent histories across devices and commands, while:

  • Always respecting disabled devices (still repairable, not purged)
  • Remaining idempotent (safe to run multiple times)

Workflow

---

Using the Settings UI (/settings/data-purge)

This is now the primary entry point for all purge and repair configuration.

What You Can Configure

Setting	Description
Retention Days	Number of days to keep records
Enabled	Enable/disable purge for this table
Batch Size	Size of deletion batches (non-config tables only)
Cleanup Frequency	Daily / Weekly / Monthly
min_versions_per_device_command	For configs only — minimum versions to always keep
non_downloaded_retention_days	Optional separate retention for config_downloaded = 0

Note For configs, Enabled is false by default in rConfig 8.1. No config data is purged until you explicitly enable it and choose safe values.

Purge Runs Table (History)

Also located on /settings/data-purge, this shows:

• when purges ran
• dry-run vs real
• record counts and per-table outcomes
• error messages (if any)
• job queue involvement
• context (e.g. data_purge, config_retention, config_repair)

Each run can be used to:

• validate that retention is working as expected
• debug customer issues related to missing data, config history, or DB size
• correlate CLI runs (rconfig:data-purge, rconfig:config-history:repair ) with UI-visible history

---

CLI Usage

1. Full Purge (all tables)

Dry-run:

php artisan rconfig:data-purge --dry-run

Real:

php artisan rconfig:data-purge

---

2. Purge Specific Table

Dry-run:

php artisan rconfig:data-purge --table=activity_log --dry-run

Real:

php artisan rconfig:data-purge --table=activity_log

---

3. Config Retention Only

Dry-run:

php artisan rconfig:data-purge --table=configs --dry-run

Real:

php artisan rconfig:data-purge --table=configs

Notes

• Config retention never queues jobs, even if queue flags are set.
• It is always processed immediately and safely.
• Disabled devices are never included in config purges.

---

4. Repair Config History

Dry-run for a single device:

php artisan rconfig:config-history:repair  --device-id=123 --dry-run

Real for a single device:

php artisan rconfig:config-history:repair  --device-id=123

Repair all devices:

php artisan rconfig:config-history:repair  --all

Tip The repair command is idempotent. Running it multiple times will not damage healthy chains and is safe as a general “health pass” across your estate.

---

Understanding Job Queues

When Jobs Are Queued

Jobs are only queued when purging non-config tables, and when:

• It’s not a dry run
• records_to_delete >= queue_threshold
• purge is not forced direct (no --no-queue)
• you haven’t disabled queue usage via flags
• the queue driver is not sync

Jobs are dispatched to:

PurgeDataJobs

You can monitor them in Horizon:

/horizon

Config retention and history repair:

• never use the queue for deletions
• always run inline within the Artisan command
• still record their outcomes to Purge Runs for auditing

---

Migration Notes (V7 → V8.1)

Remove Old Purge Commands

If your system has cron entries for:

• rconfig:purge-configs
• rconfig:purge-configs-v2

Remove them.

rConfig 8.1:

• disables these commands in code
• removes any matching tasks entries during migration
• surfaces all future retention via rconfig:data-purge and rconfig:config-history:repair

Why This Change Was Needed

Old purge commands:

• could cause version chain resets
• left orphaned config_changes
• didn’t consistently delete files from disk
• didn’t respect protected versions per device/command
• weren’t integrated with the UI or purge run history
• had limited or no reporting and auditing

The new system:

• centralises configuration under Settings → Data Purge
• gives auditable Purge Runs history
• adds strong safety rails around config retention
• adds a dedicated Config History Repair pass

---

Examples

Example 1 — Keep 5 versions per device+command

The options JSON for configs should be:

{
  "enabled": true,
  "min_versions_per_device_command": 5,
  "batch_size": 1000,
  "cleanup_frequency": "daily"
}

---

Example 2 — Purge all logs older than 90 days

UI:

• activity_log → retention_days = 90

CLI:

php artisan rconfig:data-purge --table=activity_log

---

Example 3 — Purge only non-downloaded configs after 7 days

{
  "enabled": true,
  "non_downloaded_retention_days": 7,
  "min_versions_per_device_command": 3
}

---

Troubleshooting

“Nothing is deleting”

Checklist:

• Is purge enabled for that table?
• Are retention days > 0?
• Is today a valid day for the configured frequency (daily/weekly/monthly)?
• Does the table actually have a usable timestamp column?
• Are you running with --dry-run?
• For configs: is the enabled option still false (default)?

---

“Jobs aren’t appearing in Horizon”

• configs never queue
• --dry-run never queues
• threshold not reached for that table
• --force-queue not used
• queue driver may be sync
• Horizon may not be running
• Deletions may already have completed inline if record counts were small

---

“Version numbers reset to 1,2,3 after purge”

Run:

php artisan rconfig:config-history:repair  --all

Then validate with a dry run:

php artisan rconfig:data-purge --table=configs --dry-run

---

Related Documentation

• CLI Commands
• Device Management
• Universal Device Support
• Encryption for Config Files
• Troubleshooting Backups