Data Purge - Automated Data Retention Management in rConfig V8

Data Purge: Automated Data Retention Management in rConfig V8

Data purge functionality in rConfig V8 provides automated retention management for system data, addressing the challenge of database growth over time. As network configuration management systems accumulate logs, activity records, and historical data, database size increases indefinitely without active management, potentially impacting performance and storage costs.

Organizations can leverage rConfig’s data purge capabilities to automatically remove old data based on configurable retention periods, maintain optimal database performance, and comply with data retention policies while preserving recent operational data needed for network management and troubleshooting.

For config-specific retention logic and version repair, see
Data Purge, Config Retention & History Repair.

Understanding Data Purge

What is Data Purge?

Data purge automatically removes records older than specified retention periods from rConfig database tables. The system identifies tables containing time-stamped data, counts records exceeding the retention threshold, and deletes them in controlled batches to maintain database health without impacting system performance.

Key characteristics:

• Automatic execution: Runs on a scheduled basis (nightly at 2:30 AM by default) without manual intervention.
• Table-specific retention: Each database table has independent retention settings.
• Configurable frequency: Purge operations can run daily, weekly, or monthly per table.
• Performance-optimized: Large deletions can be processed via background jobs to prevent system slowdown.
• Selective operation: Individual tables can be enabled or disabled independently.
• Config-aware: The configs table uses a dedicated Config Retention Service with additional safety logic.

Why Data Purge Matters

Without active data retention management, rConfig databases grow continuously as the system accumulates:

• User activity logs tracking authentication events and actions
• Configuration change history across managed devices
• Compliance check results from periodic scans
• Failed job records from scheduled tasks
• Temporary processing data from background operations

This growth impacts storage costs, backup duration, query performance, and system responsiveness. Data purge maintains database efficiency while preserving the operational data needed for current network management activities.

How Data Purge Works

The purge process follows a systematic workflow:

1. Scheduled execution: Purge runs automatically via a scheduled Artisan command (nightly by default).
2. Table evaluation: System reviews settings_purge entries for each configured table.
3. Frequency check: Verifies if purge should run based on cleanup frequency (daily/weekly/monthly).
4. Record identification: Locates records older than the retention period.
5. Deletion strategy: Small datasets delete immediately; large datasets may be queued for background processing.
6. Batch processing: Deletes records in controlled batches to prevent database locks.
7. Result logging: Records deletion activity in Purge Runs and system logs for audit purposes.

Note

The purge process only affects tables explicitly configured in purge settings. Tables without purge settings remain untouched regardless of data age.

---

Configuring Data Purge Settings

Accessing Purge Configuration

To manage data purge settings:

1. Navigate to Settings → Data Purge from the main navigation.
2. The settings table displays all configured tables with their retention parameters.
3. Use search and filters to locate specific tables.
4. Use the action controls to modify retention settings.

In the same screen, you’ll also find Purge Run History, which lists all recent purge executions (including CLI-initiated runs) and their outcomes.

Note

You cannot add arbitrary tables manually; rConfig auto-detects eligible tables based on schema and timestamp columns. You may disable purging for any table as needed, but new tables generally require a system update to be included.

Understanding Purge Parameters

Each table’s purge configuration includes several parameters controlling deletion behavior:

• Table Name
  The database table subject to purging (read-only, cannot be changed).

• Records
  Current count of records in the table (informational, updates on page refresh).

• Data Size
  Total disk space consumed by table data (informational, where available).

• Retention (Days)
  Number of days to retain records before deletion. Records older than this threshold are candidates for purging. Setting this to 0 disables purging for that table.

• Batch Size (non-config tables)
  Number of records deleted in each batch operation. Smaller batches (100–500) reduce database load; larger batches (1000–5000) complete faster but may briefly impact performance. Default: 1000.

• Frequency
  How often purge runs for this table:

  • Daily: purge runs every day.
  • Weekly: purge runs only on a specific weekday (e.g. Sunday).
  • Monthly: purge runs only on the 1st day of each month.

• Enabled
  Master control determining if purge processes this table. Disabled tables skip purging entirely regardless of other settings.

• Config-specific options (configs only)

  • enabled (defaults to false in 8.1)
  • min_versions_per_device_command
  • non_downloaded_retention_days

Tip

Start with conservative retention periods (e.g. 90–180 days) and increase if storage becomes constrained. Shortening retention later is safer than accidentally deleting needed data from overly aggressive initial settings.

Modifying Purge Settings

To update retention parameters for a table:

1. From the Data Purge settings page, locate the table to modify.
2. Click the edit action in the table row.
3. The row enters edit mode with editable fields:
   • Adjust Retention Days to change how long records are kept.
   • Modify Batch Size if experiencing performance issues during purge (non-config tables).
   • Change Frequency to control how often purge runs.
   • Toggle Enabled to activate or deactivate purging.
4. Save your changes.

Changes take effect at the next scheduled purge execution. Modified retention periods do not retroactively delete data; the new threshold applies going forward.

Enabling and Disabling Tables

To quickly enable or disable purging for a table without editing all settings:

1. From the Data Purge settings page, locate the table.
2. Use the row actions to Enable or Disable the table.
3. The table’s enabled status updates immediately.

Disabled tables remain in the configuration but skip all purge operations. Use this to temporarily prevent deletion during investigations, audits, or troubleshooting without losing your retention settings.

Filtering and Search

To quickly locate specific tables:

• Search by table name: Enter text in the search field to filter tables matching the name.
• Show enabled only: Check the “Enabled only” checkbox to hide disabled tables and focus on active purge configurations.
• Sort columns: Click column headers (Table Name, Retention Days, Enabled) to sort the list.
• Clear filters: Reset search and filters to see all tables again.

---

Monitoring Purge Operations

Purge Run History

The primary place to monitor purge behaviour is the Purge Runs table under Settings → Data Purge. Each run entry includes:

• Timestamp of the run
• Initiator (scheduled vs CLI)
• Context (data_purge, config_retention, config_repair)
• Per-table results (records found, deleted, queued)
• Any error messages

From here you can quickly answer:

• “Did last night’s purge run?”
• “How many records were deleted from each table?”
• “Did config retention actually remove anything?”

Viewing System Logs

rConfig also logs purge operations in the system logs:

1. Navigate to Settings → Logs (or System → Application Log).
2. Filter for purge-related entries:
   • Search for "purge" or "PurgeDataJobs".
   • Filter by date range to find specific runs.
3. Review log entries for:
   • Tables processed.
   • Records deleted or queued.
   • Any errors or warnings encountered.

Note

Large deletions processed via background jobs may span multiple log entries as batches complete asynchronously. The initial entry shows records were queued; subsequent entries log batch completion.

Verifying Purge Effectiveness

To confirm purge operations are working correctly:

1. Note the Records count on the Data Purge settings page before a scheduled purge.
2. Check the count after purge execution (refresh the page).
3. Records should decrease if old data existed.
4. Review Purge Runs and system logs to verify deletion occurred.
5. Monitor Data Size over time to confirm database growth is controlled.

If record counts don’t decrease as expected, verify:

• Table is enabled in purge settings.
• Retention period is appropriate (not so long that no records qualify).
• Frequency matches expected schedule (weekly/monthly tables won’t purge daily).
• System logs for errors indicating why purge didn’t execute.

---

Performance Considerations

Queue vs. Direct Deletion

rConfig uses two strategies for deleting records based on volume for non-config tables:

• Direct deletion (below queue threshold):
  Deletes records immediately during the purge execution. Suitable for tables with moderate data volumes.

• Queued deletion (above queue threshold):
  Dispatches background jobs to delete records in batches over time. Prevents long-running deletions from blocking other system operations.

The queue decision also honours CLI flags:

• --force-queue – always queue deletions for non-config tables.
• --no-queue – always delete directly, regardless of record count.
• --queue-threshold – override the default record threshold.

For configs, deletion is never queued. Config retention always runs inline via the Config Retention Service, regardless of queue flags.

Batch Size Tuning

Batch size controls how many records delete in each operation (non-config tables):

Smaller batches (100–500)

• Reduce database load during purge.
• Allow other queries to execute between batches.
• Take longer to complete large purges.
• Recommended for systems with heavy user activity during purge time.

Larger batches (1000–5000)

• Complete purges faster.
• May briefly slow database performance.
• More efficient for systems with light activity during purge time.
• Recommended for overnight processing when users are inactive.

Adjust batch size per table based on that table’s typical deletion volume and system activity patterns.

Timing and Frequency

• Default schedule: Daily at 2:30 AM balances timely data removal with minimal user impact during off-peak hours.
• Frequency options:
  • Daily: Ensures old data never accumulates more than retention period + 1 day.
  • Weekly: Reduces processing overhead for tables with slower growth.
  • Monthly: Appropriate for tables with minimal activity or very long retention.

Consider your organization’s data growth rates and storage constraints when selecting frequencies. Tables with rapid growth (activity logs, job records) benefit from daily purging, while slowly-growing tables can use weekly or monthly schedules.

---

Best Practices

Retention Period Selection

• Start conservatively: Begin with longer retention periods (90–180 days) until you understand your data usage patterns.
• Align with compliance: Match retention periods to regulatory requirements for your industry.
• Consider troubleshooting needs: Retain data long enough to investigate historical issues (often 30–60 days).
• Differentiate by table type:
  • Activity logs: 90–180 days (compliance and security monitoring).
  • Job failures: 30–60 days (operational troubleshooting).
  • Temporary data: 7–14 days (processing artefacts).

Monitoring and Maintenance

• Review quarterly: Check purge effectiveness every 3 months:

  • Database size trends (should stabilise after initial purges).
  • Storage utilisation (should not grow indefinitely).
  • Purge Runs (should show consistent activity and sane counts).

• Investigate anomalies:

  • Verify purge is enabled for high-volume tables.
  • Check retention periods aren’t too long.
  • Review logs for errors preventing deletion.
  • Confirm tables have appropriate timestamp columns.

• Test retention periods:

  • After modifying retention settings, monitor for one complete cycle to ensure:
    • Appropriate amount of data is deleted.
    • Sufficient history remains for operational needs.
    • No unexpected table growth occurs.

Safety and Recovery

• Backup before major changes: Before significantly reducing retention periods, ensure recent backups exist.

• Use dry-run for testing: When manually executing purge, use --dry-run to preview deletions.

• Document retention policies: Internally document:

  • Retention periods for each table type.
  • Business justification and compliance drivers.
  • Review schedule and owners.

• Monitor initially: After enabling purge for the first time, closely monitor for 1–2 weeks to ensure:

  • Deletions occur as expected.
  • System performance remains acceptable.
  • No operational issues arise from data removal.

---

Advanced Usage

Manual Purge Execution

Administrators can manually trigger purge operations outside the scheduled time using the command line:

Basic manual purge (processes all enabled tables):

php artisan rconfig:data-purge

Test without deleting (dry-run mode shows what would be deleted):

php artisan rconfig:data-purge --dry-run

Purge specific table only:

php artisan rconfig:data-purge --table=activity_log

Force background processing (queue all deletions for non-config tables regardless of size):

php artisan rconfig:data-purge --force-queue

Disable background processing (delete everything immediately):

php artisan rconfig:data-purge --no-queue

Adjust queue threshold:

php artisan rconfig:data-purge --queue-threshold=500

Caution

Manual purge execution with —no-queue can cause temporary system slowdown if large amounts of data require deletion. Use this option only during maintenance windows when users are not actively using rConfig.

For config retention and history repair, see: php artisan rconfig:data-purge --table=configs and php artisan rconfig:config-history:repair as documented in Config Purge & History Repair.

When to Use Manual Purge

• Initial cleanup: After enabling purge for the first time, run with --dry-run to verify behaviour.
• Storage emergencies: If database storage becomes critically low, manually trigger purge to immediately free space.
• Maintenance windows: Run manual purge during planned maintenance to control timing and monitor results in real-time.
• Testing changes: After modifying retention settings, manually execute with --dry-run to preview impact.

---

Troubleshooting

Symptom: Purge Not Deleting Records

Diagnosis: Check table configuration and verify records meet deletion criteria.

Possible Causes:

• Table is disabled in purge settings.
• Retention period is longer than the age of existing data.
• Frequency setting prevents purge on current day (e.g., weekly purge but today isn’t the configured day).
• Table lacks an appropriate timestamp column for age determination.
• Retention days set to 0 (purging disabled).
• For configs: the enabled option is still false (default in 8.1).

Resolution Steps:

1. Verify the table shows “Enabled” status on the Data Purge settings page.
2. Check retention period vs. actual age of data in the table.
3. Confirm the frequency matches the current schedule.
4. Review system logs for errors about missing timestamp columns.
5. Ensure retention days is greater than 0 (and enabled = true for configs).
6. Manually execute purge with --dry-run to see what would be deleted.

Symptom: System Slowdown During Purge

Diagnosis: Check batch size and queue configuration.

Possible Causes:

• Batch size too large causing long-running operations.
• Queue disabled (--no-queue) forcing large deletions to process immediately.
• Multiple large tables purging simultaneously.
• Database server resource constraints during purge time.

Resolution Steps:

1. Reduce batch size to 500 or less for affected tables.
2. Ensure queue is not disabled in scheduled commands.
3. Stagger purge frequency to spread load.
4. Consider moving scheduled purge time to a lower-activity period.
5. Monitor database resources (CPU, I/O, memory) during purge.
6. Verify background job workers are running (for queued deletions).

Symptom: Records Count Not Updating After Purge

Diagnosis: Check if purge completed successfully and refresh data.

Possible Causes:

• Page not refreshed after purge completed.
• Background jobs still processing (queued deletion in progress).
• Purge encountered errors and didn’t complete.
• Browser caching old data.

Resolution Steps:

1. Refresh the Data Purge settings page.
2. Check Purge Runs to verify purge completed successfully.
3. If records were queued, check background job queue status.
4. Review logs for errors indicating purge failure.
5. Verify timestamps in logs show recent execution.

Symptom: Database Still Growing Despite Purge

Diagnosis: Verify tables with highest growth have purge enabled and tuned correctly.

Possible Causes:

• High-volume tables not included in purge configuration.
• Retention periods too long for data growth rate.
• New tables created without purge settings.
• Purge disabled globally or for critical tables.
• Data growth exceeds deletion rate.

Resolution Steps:

1. Identify largest tables in the database (Data Size column).
2. Verify largest tables have purge enabled with appropriate retention.
3. Review and potentially reduce retention periods.
4. Check for new tables since initial purge configuration.
5. Monitor data growth rate vs. deletion rate in logs and Purge Runs.
6. Consider more aggressive retention or more frequent purging for high-volume tables.

---

Related Documentation

• Config Purge & History Repair
• System Logs – Viewing purge execution logs and activity
• Performance Tuning – Optimizing rConfig performance
• Database Maintenance – General database health practices
• Backup and Recovery – Protecting data before purge operations

Summary

Data purge in rConfig V8 provides automated data retention management to maintain optimal database performance and comply with organizational data policies. By automatically removing old records based on configurable retention periods, the system prevents indefinite database growth while preserving recent operational data needed for network management activities.

Key takeaways:

• Configure retention per table with periods appropriate to data type and requirements.
• Ensure purging is enabled for high-volume tables (activity logs, job records).
• Monitor effectiveness via Purge Runs, system logs, and database size trends.
• Start with conservative retention (90–180 days) and adjust as needed.
• Use dry-run before deploying aggressive retention policies.
• Coordinate with backups so you can recover if critical data is purged.
• For configs, use the dedicated Config Retention & History Repair tooling for safe, version-aware cleanup.