Data Purge - Automated Data Retention Management in rConfig V8

Data Purge: Automated Data Retention Management in rConfig V8

Data purge functionality in rConfig V8 provides automated retention management for system data, addressing the challenge of database growth over time. As network configuration management systems accumulate logs, activity records, and historical data, database size increases indefinitely without active management, potentially impacting performance and storage costs.

Organizations can leverage rConfig’s data purge capabilities to automatically remove old data based on configurable retention periods, maintain optimal database performance, and comply with data retention policies while preserving recent operational data needed for network management and troubleshooting.

Understanding Data Purge

What is Data Purge?

Data purge automatically removes records older than specified retention periods from rConfig database tables. The system identifies tables containing time-stamped data, counts records exceeding the retention threshold, and deletes them in controlled batches to maintain database health without impacting system performance.

Key characteristics:

• Automatic execution: Runs nightly at 2:30 AM by default without manual intervention
• Table-specific retention: Each database table has independent retention settings
• Configurable frequency: Purge operations can run daily, weekly, or monthly per table
• Performance-optimized: Large deletions process via background jobs to prevent system slowdown
• Selective operation: Individual tables can be enabled or disabled independently

Why Data Purge Matters

Without active data retention management, rConfig databases grow continuously as the system accumulates:

• User activity logs tracking authentication events and actions
• Configuration change history across managed devices
• Compliance check results from periodic scans
• Failed job records from scheduled tasks
• Temporary processing data from background operations

This growth impacts storage costs, backup duration, query performance, and system responsiveness. Data purge maintains database efficiency while preserving the operational data needed for current network management activities.

How Data Purge Works

The purge process follows a systematic workflow:

1. Scheduled execution: Purge runs automatically at 2:30 AM daily (configurable)
2. Table evaluation: System reviews purge settings for each configured table
3. Frequency check: Verifies if purge should run based on cleanup frequency (daily/weekly/monthly)
4. Record identification: Locates records older than the retention period
5. Deletion strategy: Small datasets delete immediately; large datasets queue for background processing
6. Batch processing: Deletes records in controlled batches to prevent database locks
7. Result logging: Records deletion activity in system logs for audit purposes

Note

The purge process only affects tables explicitly configured in purge settings. Tables without purge settings remain untouched regardless of data age.

Configuring Data Purge Settings

Accessing Purge Configuration

To manage data purge settings:

1. Navigate to Settings → Data Purge from the main navigation
2. The settings table displays all configured tables with their retention parameters
3. Use search and filters to locate specific tables
4. Click edit actions to modify retention settings

Note

Note: You may not add new tables manually; rConfig auto-detects eligible tables based on schema and timestamp columns. You may disable purging for any table as needed, but new tables require system updates to be included.

Understanding Purge Parameters

Each table’s purge configuration includes several parameters controlling deletion behavior:

Table Name: The database table subject to purging (read-only, cannot be changed)

Records: Current count of records in the table (informational, updates on page refresh)

Data Size: Total disk space consumed by table data (informational)

Retention (Days): Number of days to retain records before deletion. Records older than this threshold are candidates for purging. Setting to 0 disables purging for that table.

Batch Size: Number of records deleted in each batch operation. Smaller batches (100-500) reduce database load; larger batches (1000-5000) complete faster but may briefly impact performance. Default: 1000.

Frequency: How often purge runs for this table:

• Daily: Purge runs every day at 2:30 AM
• Weekly: Purge runs only on Sundays at 2:30 AM
• Monthly: Purge runs only on the 1st day of each month at 2:30 AM

Enabled: Master control determining if purge processes this table. Disabled tables skip purging entirely regardless of other settings.

Tip

Start with conservative retention periods (e.g., 90-180 days) and increase if storage becomes constrained. Shortening retention later is safer than accidentally deleting needed data from overly aggressive initial settings.

Modifying Purge Settings

To update retention parameters for a table:

1. From the Data Purge settings page, locate the table to modify
2. Click the edit icon in the table row
3. The row enters edit mode with editable fields:
   • Adjust Retention Days to change how long records are kept
   • Modify Batch Size if experiencing performance issues during purge
   • Change Frequency to control how often purge runs
   • Toggle Enabled to activate or deactivate purging
4. Click the checkmark icon to save changes
5. Click the X icon to cancel without saving

Changes take effect at the next scheduled purge execution. Modified retention periods do not retroactively delete data; the new threshold applies going forward.

Enabling and Disabling Tables

To quickly enable or disable purging for a table without editing:

1. From the Data Purge settings page, locate the table
2. Click the three-dot menu icon in the table row
3. Select Enable or Disable from the menu
4. The table’s enabled status updates immediately

Disabled tables remain in the configuration but skip all purge operations. Use this to temporarily prevent deletion during investigations, audits, or troubleshooting without losing your retention settings.

Filtering and Search

To quickly locate specific tables:

Search by table name: Enter text in the search field to filter tables matching the name

Show enabled only: Check the “Enabled only” checkbox to hide disabled tables and focus on active purge configurations

Sort columns: Click column headers (Table Name, Retention Days, Enabled) to sort the list ascending or descending

Clear filters: Click the X icon next to the search field to reset all filters and show all tables

Monitoring Purge Operations

Viewing Purge Activity

rConfig logs all purge operations in the system logs for monitoring and audit purposes:

1. Navigate to Settings → Logs (or System → Application Log)
2. Filter for purge-related entries:
   • Search for “purge” or “PurgeDataJobs”
   • Filter by date range to find specific purge runs
   • Look for entries at 2:30 AM (default execution time)
3. Review log entries for:
   • Tables processed
   • Records deleted or queued
   • Any errors or warnings encountered

Understanding Log Entries

Successful purge: Logs show table name, number of records deleted, processing duration

Queued purge: Logs indicate large deletion was queued for background processing

Skipped tables: Logs explain why tables were skipped (disabled, no records to delete, wrong day for frequency)

Errors: Logs detail any issues encountered (table doesn’t exist, no timestamp column, database errors)

Note

Large deletions processed via background jobs may span multiple log entries as batches complete asynchronously. The initial entry shows records were queued; subsequent entries log batch completion.

Verifying Purge Effectiveness

To confirm purge operations are working correctly:

1. Note the Records count on the Data Purge settings page before scheduled purge
2. Check the count after purge execution (refresh the page)
3. Records should decrease if old data existed
4. Review system logs to verify deletion occurred
5. Monitor Data Size over time to confirm database growth is controlled

If record counts don’t decrease as expected, verify:

• Table is enabled in purge settings
• Retention period is appropriate (not so long that no records qualify)
• Frequency matches expected schedule (weekly/monthly tables won’t purge daily)
• System logs for errors indicating why purge didn’t execute

Performance Considerations

Queue vs. Direct Deletion

rConfig uses two strategies for deleting records based on volume:

Direct deletion (< 1000 records): Deletes records immediately during the scheduled purge execution. Suitable for tables with moderate data volumes where deletion completes quickly.

Queued deletion (≥ 1000 records): Dispatches background jobs to delete records in batches over time. Prevents long-running deletions from blocking other system operations.

The threshold is configurable via command-line options for manual purge execution, but the default of 1000 records balances performance and efficiency for most deployments.

Batch Size Tuning

Batch size controls how many records delete in each operation:

Smaller batches (100-500):

• Reduce database load during purge
• Allow other queries to execute between batches
• Take longer to complete large purges
• Recommended for systems with heavy user activity during purge time

Larger batches (1000-5000):

• Complete purges faster
• May briefly slow database performance
• More efficient for systems with light activity during purge time
• Recommended for overnight processing when users are inactive

Adjust batch size per table based on that table’s typical deletion volume and system activity patterns.

Timing and Frequency

Default schedule: Daily at 2:30 AM balances timely data removal with minimal user impact during off-peak hours.

Frequency options:

• Daily: Ensures old data never accumulates more than retention period + 1 day
• Weekly: Reduces processing overhead for tables with slower growth
• Monthly: Appropriate for tables with minimal activity or very long retention

Consider your organization’s data growth rates and storage constraints when selecting frequencies. Tables with rapid growth (activity logs, job records) benefit from daily purging, while slowly-growing tables can use weekly or monthly schedules.

Best Practices

Retention Period Selection

Start conservatively: Begin with longer retention periods (90-180 days) until you understand your data usage patterns. It’s easier to shorten retention later than recover accidentally deleted data.

Align with compliance: Match retention periods to regulatory requirements for your industry. Some sectors mandate specific retention for audit trails and activity logs.

Consider troubleshooting needs: Retain data long enough to investigate historical issues. If network problems typically require reviewing 30-60 days of history, set retention accordingly.

Differentiate by table type: Apply different retention to different data types:

• Activity logs: 90-180 days (compliance and security monitoring)
• Job failures: 30-60 days (operational troubleshooting)
• Temporary data: 7-14 days (processing artifacts)

Monitoring and Maintenance

Review quarterly: Examine purge effectiveness every 3 months:

• Database size trends (should stabilize after initial purges)
• Storage utilization (should not grow indefinitely)
• Purge log entries (should show consistent activity)

Investigate anomalies: If database continues growing despite purge:

• Verify purge settings are enabled for high-volume tables
• Check retention periods aren’t too long
• Review logs for errors preventing deletion
• Confirm tables have appropriate timestamp columns

Test retention periods: After modifying retention settings, monitor for one complete cycle to ensure:

• Appropriate amount of data is deleted
• Sufficient history remains for operational needs
• No unexpected table growth occurs

Safety and Recovery

Backup before major changes: Before significantly reducing retention periods, ensure recent backups exist. This allows recovery if too much data is deleted.

Use dry-run for testing: When manually executing purge (see Advanced Usage below), use --dry-run flag to preview what would be deleted without actual deletion.

Document retention policies: Maintain internal documentation of:

• Retention periods for each table type
• Business justification for retention durations
• Compliance requirements driving retention decisions
• Review schedule and responsible parties

Monitor initially: After enabling purge for the first time, closely monitor for 1-2 weeks to ensure:

• Deletions occur as expected
• System performance remains acceptable
• No operational issues arise from data removal

Advanced Usage

Manual Purge Execution

Administrators can manually trigger purge operations outside the scheduled time using the command line:

Basic manual purge (processes all enabled tables):

php artisan rconfig:data-purge

Test without deleting (dry-run mode shows what would be deleted):

php artisan rconfig:data-purge --dry-run

Purge specific table only:

php artisan rconfig:data-purge --table=activity_log

Force background processing (queue all deletions regardless of size):

php artisan rconfig:data-purge --force-queue

Disable background processing (delete everything immediately):

php artisan rconfig:data-purge --no-queue

Adjust queue threshold:

php artisan rconfig:data-purge --queue-threshold=500

Caution

Manual purge execution with --no-queue flag can cause temporary system slowdown if large amounts of data require deletion. Use this option only during maintenance windows when users are not actively using rConfig.

When to Use Manual Purge

Initial cleanup: After enabling purge for the first time, manually execute with --dry-run to verify expected behavior before allowing automatic execution.

Storage emergencies: If database storage becomes critically low, manually trigger purge to immediately free space without waiting for scheduled execution.

Maintenance windows: Run manual purge during planned maintenance to control timing and monitor results in real-time.

Testing changes: After modifying retention settings, manually execute with --dry-run to preview impact before automatic purge runs.

Troubleshooting

Symptom: Purge Not Deleting Records

Diagnosis: Check table configuration and verify records meet deletion criteria.

Possible Causes:

• Table is disabled in purge settings
• Retention period is longer than oldest data (no records qualify for deletion)
• Frequency setting prevents purge on current day (e.g., weekly purge running on Tuesday)
• Table lacks appropriate timestamp column for age determination
• Retention days set to 0 (purging disabled)

Resolution Steps:

1. Verify table shows “Enabled” status on Data Purge settings page
2. Check retention period vs. actual age of data in table (records must be older than retention period)
3. Confirm frequency matches current day (weekly = Sunday, monthly = 1st of month)
4. Review system logs for errors about missing timestamp columns
5. Ensure retention days is greater than 0
6. Manually execute purge with --dry-run to see what would be deleted

Symptom: System Slowdown During Purge

Diagnosis: Check batch size and queue configuration.

Possible Causes:

• Batch size too large causing long-running database operations
• Queue disabled forcing large deletions to process immediately
• Multiple large tables purging simultaneously
• Database server resource constraints during purge time

Resolution Steps:

1. Reduce batch size to 500 or less for affected tables
2. Ensure queue is not disabled (check scheduled command doesn’t use --no-queue)
3. Stagger purge frequency (some tables daily, others weekly) to spread load
4. Consider moving scheduled purge time to lower-activity period
5. Monitor database server resources (CPU, I/O, memory) during purge
6. Verify background job queue is processing (check queue workers are running)

Symptom: Records Count Not Updating After Purge

Diagnosis: Check if purge completed successfully and refresh data.

Possible Causes:

• Page not refreshed after purge completed
• Background jobs still processing (queued deletion in progress)
• Purge encountered errors and didn’t complete
• Browser caching old data

Resolution Steps:

1. Refresh the Data Purge settings page (record counts update on refresh)
2. Check system logs to verify purge completed successfully
3. If records were queued, check background job queue status
4. Review logs for errors indicating purge failure
5. Verify timestamp in logs shows recent execution (after expected purge time)
6. Clear browser cache if counts still appear incorrect

Symptom: Database Still Growing Despite Purge

Diagnosis: Verify tables with highest growth have purge enabled.

Possible Causes:

• High-volume tables not included in purge configuration
• Retention periods too long for data growth rate
• New tables created without purge settings
• Purge disabled globally or for critical tables
• Data growth exceeds deletion rate

Resolution Steps:

1. Identify largest tables in database (check Data Size column)
2. Verify largest tables have purge enabled with appropriate retention
3. Review retention periods - reduce if storage is constrained
4. Check for new tables since initial purge configuration
5. Monitor data growth rate vs. deletion rate in logs
6. Consider more aggressive retention or more frequent purging for high-volume tables

Related Documentation

• System Logs - Viewing purge execution logs and activity
• Performance Tuning - Optimizing rConfig performance
• Database Maintenance - General database health practices
• Backup and Recovery - Protecting data before purge operations

Summary

Data purge in rConfig V8 provides automated data retention management to maintain optimal database performance and comply with organizational data policies. By automatically removing old records based on configurable retention periods, the system prevents indefinite database growth while preserving recent operational data needed for network management activities.

Key takeaways for effective data purge:

• Configure retention per table with periods appropriate to data type and operational requirements
• Enable purging for high-volume tables (activity logs, job records) to control database growth
• Monitor purge effectiveness through system logs and database size trends
• Start with conservative retention (90-180 days) and adjust based on storage needs and compliance requirements
• Use appropriate frequencies (daily for high-growth tables, weekly/monthly for slower-growing tables)
• Test with dry-run before deploying aggressive retention policies
• Coordinate with backups to ensure recovery options exist if needed data is purged

Effective data retention management balances database performance with operational requirements, ensuring rConfig remains responsive while retaining sufficient history for troubleshooting, compliance, and network management activities.