Horizon Queue Manager

Horizon Queue Management: Monitoring Task Execution in rConfig V8

Laravel Horizon provides queue management and monitoring for rConfig V8, serving as the backbone for asynchronous job processing. Horizon manages all background operations including device backups, policy compliance checks, snippet deployments, SNMP polling, SNMP trap processing, and system maintenance tasks. Understanding Horizon’s operation is essential for maintaining reliable rConfig automation and diagnosing operational issues.

The majority of rConfig’s backend operations leverage Laravel’s Jobs and Queues system, facilitating the creation of jobs that can be executed in the background. Horizon runs as a persistent background process managed by Supervisor, providing a web-based dashboard for real-time visibility into queue performance, job execution status, and system health.

Understanding Horizon in rConfig

What Horizon Does

Asynchronous job processing: Horizon manages background execution of time-consuming operations, preventing web interface blocking and enabling concurrent processing of multiple tasks.

Queue management: Organizes jobs into prioritized queues, ensuring critical operations (manual downloads, high-priority tasks) process before lower-priority operations.

Worker supervision: Monitors worker processes, automatically restarting failed workers and ensuring consistent job processing capacity.

Performance monitoring: Provides real-time metrics on job throughput, execution times, failure rates, and queue depths, enabling proactive performance management.

Batch processing: Groups related jobs into batches (introduced in V6), allowing individual job failures without blocking entire operations. All devices in a scheduled task are sent as a batch, then delivered to queues as individual jobs for non-blocking execution.

Failed job handling: Captures failed jobs with complete error details, stack traces, and retry capabilities, simplifying troubleshooting and error recovery.

Queue Architecture

rConfig V8 implements multiple specialized queues, processed in First In First Out (FIFO) order:

Queue	Priority	Timeout	Description
HighPriorityQueue	Highest	4 Hrs	Critical jobs requiring immediate processing
ManualDownloadQueue	High	4 Hrs	Manual device configuration downloads triggered by users
PolicyCompliance	High	4 Hrs	Policy compliance validation jobs
TaskDownloadQueue	Medium	4 Hrs	Scheduled task configuration downloads
TaskSnippetQueue	Medium	4 Hrs	Snippet deployment jobs from scheduled tasks
SnmpTrapQueue	Medium	4 Hrs	SNMP trap processing jobs
PurgeDataJobs	Low	4 Hrs	Data cleanup and purge operations
rConfigDefault	Low	4 Hrs	General system jobs (backups, emails, maintenance)
default	Lowest	A few seconds	Final fallback queue (not actively used)

Queue prioritization: Higher priority queues process before lower priority queues. Manual operations (ManualDownloadQueue) take precedence over scheduled operations (TaskDownloadQueue), ensuring interactive user actions complete quickly.

Batch system: As of rConfig V6, all device download jobs are sent as batches. Previously, rConfig processed all devices in a scheduled task as a single job, meaning failures or delays affected the entire operation. With batches:

Each device becomes an individual job within the batch
Individual job failures don’t block other jobs
Better troubleshooting with per-device job status
Non-blocking queue execution
Detailed batch status monitoring in Horizon

How rConfig Uses Horizon

Device backups: Configuration downloads execute as queued jobs in TaskDownloadQueue (scheduled) or ManualDownloadQueue (manual), allowing multiple devices to backup concurrently.

Scheduled tasks: All scheduled operations queue through appropriate queues based on task type for reliable execution.

SNMP polling: Device property collection runs as background jobs, preventing timeouts during bulk polling operations.

SNMP trap processing: Incoming SNMP traps queue to SnmpTrapQueue for asynchronous processing without blocking trap reception.

Policy compliance: Configuration compliance validation processes asynchronously in PolicyCompliance queue, handling large device populations efficiently.

Snippet deployment: Configuration changes deploy through TaskSnippetQueue, providing execution tracking and failure handling.

System maintenance: Backup cleanup, log archival, and purge operations run in PurgeDataJobs and rConfigDefault queues.

Without Horizon running, these operations cannot execute—making Horizon critical to rConfig’s operation.

Accessing Horizon Dashboard

Navigate to the Horizon dashboard via Settings → Queues → Horizon or directly at:

https://your-rconfig-domain.com/horizon

Access is restricted to users with ‘admin’ role.

Horizon Dashboard Sections

Dashboard

High-level overview of queue system operations and health.

Key metrics displayed:

Jobs Per Minute: Current rate of job processing across all queues. Higher numbers indicate active operations (backups running, tasks executing). Lower numbers suggest idle periods or processing bottlenecks.

Processes: Number of active worker processes handling jobs. This should match your configured worker count in Supervisor. If significantly lower, workers may have crashed.

Jobs Past Hour: Total job volume over the last 60 minutes, showing workload trends and peak activity periods.

Failed Jobs: Count of jobs that encountered errors. Occasional failures are normal (device timeouts, connectivity issues), but high failure rates indicate systemic problems.

Job Status Distribution: Visual breakdown of pending, processing, completed, and failed jobs, providing at-a-glance health assessment.

Horizon Status: Shows whether Horizon is actively running and processing jobs or is inactive.

Recent Jobs: List of most recently processed jobs with execution time, status, and queue name.

Horizon Dashboard Overview

Metrics

Detailed metrics and performance data introduced in V7, providing historical analysis of queue system performance.

Available metrics:

Jobs Processed: Total job volume over selected time period, showing workload trends and identifying peak usage patterns.

Throughput: Jobs processed per time unit, indicating system processing capacity and efficiency.

Runtime Analysis: Job execution time statistics, helping identify performance bottlenecks and slow operations.

Jobs Failed: Failure count trends, helping identify when issues started and whether problems are increasing or decreasing.

Jobs Per Minute: Processing rate over time, useful for capacity planning and identifying bottlenecks.

Average Wait Time: How long jobs wait before processing begins. High wait times indicate insufficient worker capacity or queue backlogs.

Average Execution Time: How long jobs take to complete. Increasing execution times may indicate performance degradation.

Time Range Options: View metrics for past hour, 24 hours, 7 days, or custom ranges to identify patterns and trends.

Horizon Metrics View

Batches

Monitor batch job processing status introduced in V7.

Batch information displayed:

Batch ID and creation timestamp
Total jobs in batch
Jobs completed, pending, and failed
Overall batch progress percentage
Individual job status within batch
Batch completion status

Batch benefits:

Non-blocking execution (individual failures don’t block other jobs)
Granular troubleshooting (identify specific failed devices)
Progress monitoring (see completion percentage in real-time)
Better resource utilization (jobs process in parallel)

Use cases: All scheduled device downloads, manual multi-device downloads, bulk compliance checks.

Horizon Batches View

Pending Jobs

Jobs awaiting execution by Laravel application.

Information displayed:

Job type and class name
Queue name and priority
Wait time (how long in queue)
Job payload (parameters)

Normal vs. concerning pending states:

Normal: Short-lived pending state during active processing
Concerning: Jobs remaining pending for extended periods indicate worker capacity issues or processing problems

Completed Jobs

Historical record of successfully executed jobs.

Information displayed:

Job completion timestamp
Execution duration
Queue name
Job payload details

Note: History is cleared on system restart or rConfig update.

Failed Jobs

Dedicated view for jobs that encountered errors during execution.

Information displayed:

Exception type and message
Complete stack trace
Job payload showing exactly what was being processed
Failure timestamp
Queue name and connection
Number of retry attempts

Actions available:

Retry: Re-queue failed job for another attempt
Retry All: Re-queue all failed jobs in batch
Forget: Remove failed job from list (doesn’t fix underlying issue)
Forget All: Clear all failed jobs (use cautiously)

Understanding Job Types in rConfig

Different job types appear in Horizon corresponding to rConfig operations.

Common Job Types

Device Backup Jobs:

Class: App\Jobs\DeviceBackupJob
Purpose: Execute configuration download for specific device
Typical duration: 10-60 seconds depending on device and commands
Common failures: Connection timeouts, authentication errors, prompt mismatches

Scheduled Task Jobs:

Class: App\Jobs\ScheduledTaskJob
Purpose: Execute scheduled operations (backups, compliance, maintenance)
Duration: Varies widely based on task scope
Common failures: Task configuration errors, device unavailability

SNMP Polling Jobs:

Class: App\Jobs\SnmpPollJob
Purpose: Collect device properties via SNMP
Typical duration: 1-10 seconds per device
Common failures: SNMP authentication errors, device timeouts

SNMP Trap Processing Jobs:

Class: App\Jobs\SnmpTrapProcessJob
Purpose: Process incoming SNMP traps
Typical duration: 1-5 seconds per trap
Common failures: Invalid trap format, database issues

Policy Compliance Jobs:

Class: App\Jobs\ComplianceCheckJob
Purpose: Validate configurations against defined policies
Typical duration: 5-30 seconds depending on policy complexity
Common failures: Invalid regex in policies, missing configuration files

Snippet Deployment Jobs:

Class: App\Jobs\SnippetDeploymentJob
Purpose: Deploy configuration snippets to devices
Typical duration: 10-60 seconds depending on snippet size
Common failures: Connection errors, insufficient privileges, command syntax errors

Data Purge Jobs:

Class: App\Jobs\PurgeDataJob
Purpose: Clean old configurations, logs, and temporary files
Typical duration: Varies based on data volume
Common failures: Disk space issues, database lock contention

Interpreting Job Status

Pending: Job is queued but not yet picked up by a worker. Normal if queue has backlog. Concerning if jobs remain pending for extended periods (indicates worker issues).

Processing: Worker is actively executing the job. Should transition to completed or failed within reasonable timeframe based on job type.

Completed: Job finished successfully. Execution time shows how long it took. Review if times seem abnormally long.

Failed: Job encountered error during execution. Review exception details to determine cause.

Retrying: Failed job that’s been re-queued for another attempt. Monitor to see if retry succeeds.

Supervisor Configuration

Supervisor manages the Horizon process, ensuring it runs continuously and automatically restarts if it crashes. Proper Supervisor configuration is critical to Horizon reliability.

Understanding Supervisor

Supervisor is a process control system for Linux/Unix that:

Starts Horizon automatically when system boots
Monitors Horizon process health
Restarts Horizon if it crashes or exits unexpectedly
Provides logging for process startup, shutdown, and errors
Manages multiple worker processes if configured

Why Supervisor matters: Without Supervisor running Horizon, all background jobs stop processing. Scheduled tasks don’t execute, backups don’t run, and the queue fills with pending jobs.

Supervisor Configuration by OS

Supervisor configuration varies slightly between distributions but follows the same principles.

CentOS/RHEL/Rocky/Alma
Ubuntu/Debian

CentOS / RHEL / Rocky / Alma Linux

Supervisor configuration file location:

/etc/supervisord.d/horizon.ini

Configuration file contents:

[program:horizon]
; Horizon rConfig Configuration
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/rconfig8/current/artisan horizon
autostart=true
autorestart=true
user=root
redirect_stderr=true
stdout_logfile=/var/www/html/rconfig8/current/storage/logs/horizon.log
stopwaitsecs=3600

Configuration explanation:

command: Path to Horizon artisan command
autostart: Start Horizon when Supervisor starts
autorestart: Restart Horizon if it exits
user: Run as nginx user (web server user)
numprocs: Number of Horizon processes (typically 1)
stdout_logfile: Where Horizon logs are written
stopwaitsecs: Grace period before killing process (3600 seconds for long-running jobs)

Managing Supervisor:

# Check Supervisor status
systemctl status supervisord

# Start Supervisor
systemctl start supervisord

# Enable Supervisor to start on boot
systemctl enable supervisord

# Restart Supervisor
systemctl restart supervisord

Managing Horizon through Supervisor:

# Check Horizon worker status
supervisorctl status horizon

# Start Horizon worker
supervisorctl start horizon:*

# Stop Horizon worker
supervisorctl stop horizon:*

# Restart Horizon worker
supervisorctl restart horizon:*

# Reload Supervisor configuration
supervisorctl reread
supervisorctl update

Supervisor logs location:

/var/log/supervisor/supervisord.log

Ubuntu / Debian

Supervisor configuration file location:

/etc/supervisor/conf.d/horizon_supervisor.ini

Configuration file contents:

[program:horizon]
; Horizon rConfig Configuration
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/rconfig8/current/artisan horizon
autostart=true
autorestart=true
user=root
redirect_stderr=true
stdout_logfile=/var/www/html/rconfig8/current/storage/logs/horizon.log
stopwaitsecs=3600

Configuration explanation:

command: Path to Horizon artisan command
autostart: Start Horizon when Supervisor starts
autorestart: Restart Horizon if it exits
user: Run as root user
stdout_logfile: Where Horizon logs are written
stopwaitsecs: Grace period before killing process (3600 seconds for long-running jobs)

Managing Supervisor:

# Check Supervisor status
systemctl status supervisor

# Start Supervisor
systemctl start supervisor

# Enable Supervisor to start on boot
systemctl enable supervisor

# Restart Supervisor
systemctl restart supervisor

Managing Horizon through Supervisor:

# Check Horizon worker status
supervisorctl status horizon

# Start Horizon worker
supervisorctl start horizon:*

# Stop Horizon worker
supervisorctl stop horizon:*

# Restart Horizon worker
supervisorctl restart horizon:*

# Reload Supervisor configuration
supervisorctl reread
supervisorctl update

Supervisor logs location:

/var/log/supervisor/supervisord.log

Horizon Configuration via Environment Variables

Horizon’s behavior is controlled by environment variables in the .env file. These settings control worker processes, timeouts, and queue retry behavior.

Important Environment Variables

HORIZON_PROD_TIMEOUT=60
HORIZON_MIN_PROCESS=1
HORIZON_MAX_PROCESS=10
HORIZON_DEFAULT_MAX_PROCESS=5
REDIS_QUEUE_RETRY_AFTER=130

Variable descriptions:

HORIZON_PROD_TIMEOUT:

Number of seconds a job may run before termination as timeout
Default: 60 seconds
Increase for long-running jobs (large backups, extensive compliance checks)
Must be less than REDIS_QUEUE_RETRY_AFTER

HORIZON_MIN_PROCESS:

Minimum number of worker processes Horizon maintains
Default: 1
Applied to all queues
Ensures baseline processing capacity always available

HORIZON_MAX_PROCESS:

Maximum number of worker processes Horizon maintains
Default: 10
Applied to: TaskDownloadQueue, TaskSnippetQueue, ManualDownloadQueue, HighPriorityQueue
Scales up workers during high load periods

HORIZON_DEFAULT_MAX_PROCESS:

Maximum worker processes for default-priority queues
Default: 5
Applied to: PolicyCompliance, rConfigDefault, PurgeDataJobs, SnmpTrapQueue

REDIS_QUEUE_RETRY_AFTER:

Number of seconds to wait before retrying a failed job
Default: 130 seconds
Critical: Must be greater than HORIZON_PROD_TIMEOUT to prevent duplicate job processing
If updating timeout, update this value accordingly

Modifying Environment Variables

Edit the .env file located at /var/www/html/rconfig8/current/.env:

nano /var/www/html/rconfig8/current/.env

Add or modify the variables as needed. After making changes, clear application cache:

cd /var/www/html/rconfig8/current
php artisan rconfig:clear-all

Then restart Horizon if needed:

php artisan rconfig:clear-horizon
# Supervisor automatically restarts it

Important Considerations

Timeout and Retry Settings:

If updating HORIZON_PROD_TIMEOUT, ensure REDIS_QUEUE_RETRY_AFTER is also edited to a value higher than the timeout to prevent job duplication issues
Formula: REDIS_QUEUE_RETRY_AFTER should be at least HORIZON_PROD_TIMEOUT + 30 seconds

Database Connections:

When increasing HORIZON_MAX_PROCESS beyond typical configurations, check the database server’s maximum allowed connections
Default MySQL/MariaDB configurations may need adjustment to accommodate additional connections from increased worker processes
Each worker process can potentially use a database connection
Monitor database connection usage: SHOW STATUS LIKE 'Threads_connected';

Resource Planning:

More workers = higher CPU usage
More workers = higher memory usage
More workers = more database connections
Balance worker count against available system resources
Monitor system performance when adjusting worker counts

Troubleshooting Horizon Issues

Horizon Shows as Inactive

Symptoms: Horizon dashboard shows “Inactive” status, or jobs queue but never process.

Diagnosis:

Check Horizon status via command line:

supervisorctl status horizon

Expected output: Horizon is running.

If output is Horizon is inactive., Horizon is not running.

Check Supervisor status:

supervisorctl status horizon

Expected output: horizon RUNNING pid 12345, uptime 1:23:45

If output shows STOPPED, FATAL, or EXITED, Supervisor isn’t running Horizon.

Check Supervisor itself is running:

# CentOS/RHEL/Rocky/Alma
systemctl status supervisord

# Ubuntu/Debian
systemctl status supervisor

If Supervisor isn’t running, Horizon can’t run.

Resolution Steps:

Step 1: Ensure Supervisor is running:

# CentOS/RHEL/Rocky/Alma
systemctl start supervisord
systemctl enable supervisord

# Ubuntu/Debian
systemctl start supervisor
systemctl enable supervisor

Step 2: Update Supervisor configuration:

# Reload Supervisor configuration files
supervisorctl reread

# Apply configuration changes
supervisorctl update

Step 3: Start Horizon worker:

supervisorctl start horizon:*

Step 4: Verify Horizon is running:

php artisan horizon:status

Should return Horizon is running.

Step 5: Check Horizon dashboard:

Navigate to Settings → Queues → Horizon and verify dashboard shows active status and processing metrics.

Supervisor Not Starting Horizon

Symptoms: Supervisor shows FATAL or EXITED status for horizon.

Diagnosis:

Check Supervisor logs for errors:

# CentOS/RHEL/Rocky/Alma
tail -n 50 /var/log/supervisor/supervisord.log

# Ubuntu/Debian
tail -n 50 /var/log/supervisor/supervisord.log

Check Horizon logs for application errors:

tail -n 50 /var/www/html/rconfig8/current/storage/logs/horizon.log

Common causes and resolutions:

1. Incorrect file paths in configuration:

Verify paths in Supervisor config match your installation:

# Check rConfig installation path
ls -la /var/www/html/rconfig8/current/artisan

# Verify PHP path
which php

Update Supervisor config if paths incorrect, then:

supervisorctl reread
supervisorctl update
supervisorctl start horizon:*

2. Permission issues:

Horizon must run as web server user:

# Check ownership of rConfig directory
ls -la /var/www/html/rconfig8/current/

# Should be owned by nginx (CentOS) or www-data (Ubuntu)
# If not, fix permissions:

# CentOS/RHEL/Rocky/Alma
chown -R nginx:nginx /var/www/html/rconfig8/current/

# Ubuntu/Debian
chown -R www-data:www-data /var/www/html/rconfig8/current/

3. PHP memory limit too low:

You should review our docs under System Settings to set optimal PHP settings.

Jobs Stuck in Pending Status

Symptoms: Jobs queue but never process, remaining in “Pending” status indefinitely.

Diagnosis:

Check if workers are processing jobs:

php artisan horizon:status

Check worker count in Horizon dashboard:

Navigate to Settings → Queues → Horizon
Check “Processes” count
Should match configured numprocs in Supervisor config (typically 1) plus additional workers up to max

Resolution:

If Horizon shows as running but no workers active:

Terminate and restart Horizon:

php artisan rconfig:clear-horizon
# Wait 5 seconds for graceful shutdown
supervisorctl start horizon:*

If workers are active but jobs still pending:

Check for queue priority issues. Jobs in higher priority queues process first. Lower priority jobs may wait if high priority queue is busy.

Increase worker count if consistently backlogged:

Edit .env file:

nano /var/www/html/rconfig8/current/.env

# Increase max processes slightly
HORIZON_MAX_PROCESS=15
HORIZON_DEFAULT_MAX_PROCESS=8

Clear cache and restart:

php artisan rconfig:clear-all
php artisan rconfig:clear-horizon

Jobs Failing Repeatedly

Symptoms: High failure rate in Horizon dashboard, many jobs in Failed Jobs section.

Diagnosis:

Navigate to Settings → Queues → Horizon → Failed Jobs
Click on recent failed job to view details
Review exception message and stack trace
Identify common failure patterns

Common causes and resolutions:

Device connectivity failures:

Exception: Connection timeout, SSH connection refused
Resolution: Verify device reachability, check credentials, verify firewall rules
Action: Fix device/network issues before retrying jobs

Authentication failures:

Exception: Authentication failed, Invalid credentials
Resolution: Update device credentials in rConfig
Action: Correct credentials, then retry failed jobs

Configuration errors:

Exception: Command not found, Invalid regex pattern
Resolution: Fix command configuration or policy definitions
Action: Correct configuration, then retry jobs

System resource exhaustion:

Exception: Out of memory, Maximum execution time exceeded
Resolution: Increase PHP memory limit, optimize job processing
Action: Increase resources or reduce concurrent job count

After resolving root cause:

Retry failed jobs:

Click Retry All in Failed Jobs section
Or retry individual jobs after fixing specific issues

Horizon Process Consuming Excessive Resources

Symptoms: High CPU or memory usage by Horizon process, system slowdown.

Diagnosis:

Check resource usage:

# View Horizon process resource usage
ps aux | grep horizon

# Monitor in real-time
top -p $(pgrep -f horizon)

Common causes and resolutions:

Too many concurrent workers:

Reduce max processes in .env:

nano /var/www/html/rconfig8/current/.env

# Reduce max processes
HORIZON_MAX_PROCESS=5
HORIZON_DEFAULT_MAX_PROCESS=3

# Clear cache and restart
php artisan rconfig:clear-all
php artisan rconfig:clear-horizon

Large job payload or memory leaks:

Restart Horizon periodically to clear memory:

php artisan rconfig:clear-horizon
# Supervisor automatically restarts it

Schedule regular restarts (add to crontab):

# Restart Horizon daily at 3 AM
0 3 * * * cd /var/www/html/rconfig8/current && php artisan rconfig:clear-horizon

Inefficient jobs:

Review job execution times in Horizon metrics. Jobs taking >5 minutes may need optimization:

Reduce scope (fewer devices per job)
Optimize command sets
Reduce SNMP OID count
Split large operations into smaller jobs

Horizon Dashboard Inaccessible

Symptoms: Cannot access Horizon dashboard at /horizon, 404 error or blank page.

Resolution:

Clear application cache:

cd /var/www/html/rconfig8/current
php artisan config:clear
php artisan route:clear
php artisan view:clear
php artisan cache:clear

Verify Horizon is published:

php artisan horizon:install

Check web server configuration:

Ensure web server (Nginx/Apache) is configured to serve rConfig properly and routes are accessible.

Restart services:

systemctl restart nginx php-fpm  # For Nginx
systemctl restart httpd php-fpm  # For Apache

Best Practices

Monitoring

Review Horizon dashboard daily: Check for failed jobs, unusual processing times, or growing queue depths indicating issues.

Monitor system resources: Track CPU, memory, and disk usage to ensure adequate capacity for queue processing.

Set up alerting: Configure monitoring tools (Nagios, Zabbix, etc.) to alert when:

Horizon becomes inactive
Failed job count exceeds threshold
Queue depth grows continuously
Worker processes are not running

Review batch status: Check batch completion rates to identify systematic issues with specific device groups or operations.

Maintenance

Restart Horizon after deployments: Always terminate and restart Horizon after deploying rConfig updates:

php artisan rconfig:clear-horizon
# Supervisor automatically restarts it

Clear failed jobs periodically: After resolving issues and retrying jobs, clear old failed jobs to keep interface clean:

Review failed jobs to understand failure patterns
Retry jobs that might succeed after fixes
Forget jobs that are no longer relevant

Review performance metrics: Weekly review of Horizon metrics identifies trends:

Growing average execution times may indicate performance degradation
Increasing failure rates suggest systemic issues developing
High wait times indicate insufficient worker capacity

Monitor batch failures: Review batch completion statistics to identify patterns:

Specific devices failing consistently
Time-based failure patterns
Queue-specific issues

Capacity Planning

Baseline your workload: Understand normal job volume and execution times to identify when capacity expansion is needed.

Scale workers appropriately: Add workers as device count grows, but balance against system resources:

Monitor database connections usage
Track PHP memory consumption
Watch CPU utilization trends

Stagger scheduled tasks: Distribute task schedules across time to prevent queue spikes overwhelming workers.

Plan for peak loads: Ensure system can handle peak activity periods (scheduled backup windows) without excessive delays:

Test with increased worker counts during off-peak hours
Monitor resource usage during peak periods
Adjust worker counts based on observed patterns

Database connection planning: Calculate maximum potential database connections:

Formula: HORIZON_MAX_PROCESS * number_of_queues * concurrent_jobs_per_worker
Ensure MySQL/MariaDB max_connections setting accommodates this plus regular web traffic
Default MySQL max_connections is often 151—may need increase for larger deployments

Scheduled Tasks - Configure automated operations processed by Horizon
Device Backups - Understanding backup jobs in Horizon
Policy Compliance - Compliance jobs processing
Application Logs - Additional logging beyond Horizon

Quick Reference

Common Commands

# Check Horizon status
php artisan horizon:status

# Terminate Horizon gracefully
php artisan rconfig:clear-horizon

# Pause Horizon (stop processing new jobs)
php artisan horizon:pause

# Continue Horizon (resume processing)
php artisan horizon:continue

# Clear all caches after .env changes
php artisan rconfig:clear-all

# Check Supervisor status
supervisorctl status horizon

# Start Horizon worker
supervisorctl start horizon:*

# Restart Horizon worker
supervisorctl restart horizon:*

# View Horizon logs
tail -f /var/www/html/rconfig8/current/storage/logs/horizon.log

# View Supervisor logs
tail -f /var/log/supervisor/supervisord.log

Environment Variables Quick Reference

# Default values in .env file
HORIZON_PROD_TIMEOUT=60
HORIZON_MIN_PROCESS=1
HORIZON_MAX_PROCESS=10
HORIZON_DEFAULT_MAX_PROCESS=5
REDIS_QUEUE_RETRY_AFTER=130

Critical rule: REDIS_QUEUE_RETRY_AFTER must be greater than HORIZON_PROD_TIMEOUT to prevent duplicate job processing.

Queue Priority Order

HighPriorityQueue - Critical jobs
ManualDownloadQueue - User-triggered downloads
PolicyCompliance - Compliance validation
TaskDownloadQueue - Scheduled downloads
TaskSnippetQueue - Snippet deployments
SnmpTrapQueue - SNMP trap processing
PurgeDataJobs - Data cleanup
rConfigDefault - General system jobs
default - Fallback (rarely used)

Troubleshooting Checklist

Supervisor Configuration Locations

CentOS / RHEL / Rocky / Alma:

Config: /etc/supervisord.d/horizon.ini
Logs: /var/log/supervisor/supervisord.log
Service: supervisord
User: nginx

Ubuntu / Debian:

Config: /etc/supervisor/conf.d/horizon_supervisor.ini
Logs: /var/log/supervisor/supervisord.log
Service: supervisor
User: www-data

Common Horizon Issues and Quick Fixes

Issue	Quick Fix
Horizon inactive	`supervisorctl start horizon:*`
Jobs stuck pending	`php artisan rconfig:clear-horizon`
Failed jobs piling up	Review failed job details, fix root cause, then retry
High memory usage	Reduce HORIZON_MAX_PROCESS in .env
Dashboard inaccessible	`php artisan config:clear && systemctl restart nginx`
After .env changes	`php artisan rconfig:clear-all && php artisan rconfig:clear-horizon`
Worker not starting	Check Supervisor logs, verify paths and permissions

Resource Planning Guidelines

Small deployment (< 100 devices):

HORIZON_MIN_PROCESS=1
HORIZON_MAX_PROCESS=5
HORIZON_DEFAULT_MAX_PROCESS=3
PHP memory_limit=256M
MySQL max_connections=151 (default)

Medium deployment (100-500 devices):

HORIZON_MIN_PROCESS=1
HORIZON_MAX_PROCESS=10
HORIZON_DEFAULT_MAX_PROCESS=5
PHP memory_limit=512M
MySQL max_connections=300

Large deployment (500+ devices):

HORIZON_MIN_PROCESS=2
HORIZON_MAX_PROCESS=15
HORIZON_DEFAULT_MAX_PROCESS=8
PHP memory_limit=1024M
MySQL max_connections=500

Note: Adjust based on actual system resources and workload patterns. Monitor performance and scale incrementally.

Summary

Laravel Horizon is the critical component managing all background job processing in rConfig V8. Understanding how to monitor Horizon’s operation through its dashboard, properly configure Supervisor to ensure Horizon runs reliably, configure environment variables for optimal performance, and troubleshoot common issues ensures smooth rConfig automation.

Key Takeaways

Central to rConfig operation: Horizon manages all background jobs—without it running, backups, compliance checks, and scheduled tasks cannot execute.

Queue-based architecture: Nine specialized queues prioritize different job types, ensuring critical operations (manual downloads, high-priority tasks) process before lower-priority operations.

Batch processing benefits: Introduced in V6, batch processing allows individual job failures without blocking entire operations, providing better troubleshooting and non-blocking execution.

Supervisor dependency: Horizon runs as a Supervisor-managed process. Supervisor must be running and properly configured for Horizon to function.

Environment variable control: Horizon behavior is tuned via .env variables controlling worker counts, timeouts, and retry behavior. Changes require cache clearing and Horizon restart.

Real-time visibility: Horizon dashboard provides immediate insight into job processing, failures, performance metrics, and batch status for proactive issue identification.

Resource considerations: Increasing worker counts and timeouts requires additional system resources including CPU, memory, database connections, and PHP memory limits. Monitor carefully and scale incrementally.

Troubleshooting workflow: When Horizon issues occur, systematically check Horizon status, Supervisor status, configuration files, environment variables, logs, and system resources to identify and resolve root causes.

Regular monitoring: Daily review of Horizon dashboard, batch status, and periodic log analysis prevents small issues from becoming operational problems.

With proper Horizon configuration, monitoring, and maintenance, rConfig provides reliable, scalable automation handling background operations efficiently across networks of any size—from dozens to thousands of devices.

Horizon Queue Manager

Horizon Queue Management: Monitoring Task Execution in rConfig V8

Understanding Horizon in rConfig

What Horizon Does

Queue Architecture

How rConfig Uses Horizon

Accessing Horizon Dashboard

Horizon Dashboard Sections

Dashboard

Metrics

Batches

Pending Jobs

Completed Jobs

Failed Jobs

Understanding Job Types in rConfig

Common Job Types

Interpreting Job Status

Supervisor Configuration

Understanding Supervisor

Supervisor Configuration by OS

CentOS / RHEL / Rocky / Alma Linux

Ubuntu / Debian

Horizon Configuration via Environment Variables

Important Environment Variables

Modifying Environment Variables

Important Considerations

Troubleshooting Horizon Issues

Horizon Shows as Inactive

Supervisor Not Starting Horizon

Jobs Stuck in Pending Status

Jobs Failing Repeatedly

Horizon Process Consuming Excessive Resources

Horizon Dashboard Inaccessible

Best Practices

Monitoring

Maintenance

Capacity Planning

Related Documentation

Quick Reference

Common Commands

Environment Variables Quick Reference

Queue Priority Order

Troubleshooting Checklist

Supervisor Configuration Locations

Common Horizon Issues and Quick Fixes

Resource Planning Guidelines

Summary

Key Takeaways