Horizon Queue Manager

Horizon Queue Management: Monitoring Task Execution in rConfig V8

Prerequisites

rConfig V8 installed with Redis configured as the queue driver.
Supervisor installed on the server and the Horizon worker configured.
Admin role access to view the Queue Manager dashboard.

Laravel Horizon provides queue management and monitoring for rConfig V8, serving as the backbone for asynchronous job processing. Horizon manages all background operations including device backups, policy compliance checks, snippet deployments, SNMP polling, SNMP trap processing, and system maintenance tasks. Understanding Horizon’s operation is essential for maintaining reliable rConfig automation and diagnosing operational issues.

The majority of rConfig’s backend operations leverage Laravel’s Jobs and Queues system, facilitating the creation of jobs that can be executed in the background. Horizon runs as a persistent background process managed by Supervisor, providing a web-based dashboard for real-time visibility into queue performance, job execution status, and system health.

Understanding Horizon in rConfig

What Horizon Does

Asynchronous job processing: Horizon manages background execution of time-consuming operations, preventing web interface blocking and enabling concurrent processing of multiple tasks.

Queue management: Organizes jobs into prioritized queues, ensuring critical operations (manual downloads, high-priority tasks) process before lower-priority operations.

Worker supervision: Monitors worker processes, automatically restarting failed workers and ensuring consistent job processing capacity.

Performance monitoring: Provides real-time metrics on job throughput, execution times, failure rates, and queue depths, enabling proactive performance management.

Batch processing: Groups related jobs into batches (introduced in V6), allowing individual job failures without blocking entire operations. All devices in a scheduled task are sent as a batch, then delivered to queues as individual jobs for non-blocking execution.

Failed job handling: Captures failed jobs with complete error details, stack traces, and retry capabilities, simplifying troubleshooting and error recovery.

Queue Architecture

rConfig V8 implements multiple specialized queues, processed in First In First Out (FIFO) order:

Queue	Priority	Timeout	Description
HighPriorityQueue	Highest	4 Hrs	Critical jobs requiring immediate processing
ManualDownloadQueue	High	4 Hrs	Manual device configuration downloads triggered by users
PolicyCompliance	High	4 Hrs	Policy compliance validation jobs
TaskDownloadQueue	Medium	4 Hrs	Scheduled task configuration downloads
TaskSnippetQueue	Medium	4 Hrs	Snippet deployment jobs from scheduled tasks
SnmpTrapQueue	Medium	4 Hrs	SNMP trap processing jobs
PurgeDataJobs	Low	4 Hrs	Data cleanup and purge operations
rConfigDefault	Low	4 Hrs	General system jobs (backups, emails, maintenance)
ReportBuildQueue (V8.4.0+)	Low	5 Min	Report generation for completed scheduled task batches, isolated on its own queue so it’s never delayed by other queues
default	Lowest	A few seconds	Final fallback queue (not actively used)

Queue prioritization: Higher priority queues process before lower priority queues. Manual operations (ManualDownloadQueue) take precedence over scheduled operations (TaskDownloadQueue), ensuring interactive user actions complete quickly.

Batch system: As of rConfig V6, all device download jobs are sent as batches. Previously, rConfig processed all devices in a scheduled task as a single job, meaning failures or delays affected the entire operation. With batches:

Each device becomes an individual job within the batch
Individual job failures don’t block other jobs
Better troubleshooting with per-device job status
Non-blocking queue execution
Detailed batch status monitoring in Horizon

How rConfig Uses Horizon

Device backups: Configuration downloads execute as queued jobs in TaskDownloadQueue (scheduled) or ManualDownloadQueue (manual), allowing multiple devices to backup concurrently.

Scheduled tasks: All scheduled operations queue through appropriate queues based on task type for reliable execution.

SNMP polling: Device property collection runs as background jobs, preventing timeouts during bulk polling operations.

SNMP trap processing: Incoming SNMP traps queue to SnmpTrapQueue for asynchronous processing without blocking trap reception.

Policy compliance: Configuration compliance validation processes asynchronously in PolicyCompliance queue, handling large device populations efficiently.

Snippet deployment: Configuration changes deploy through TaskSnippetQueue, providing execution tracking and failure handling.

System maintenance: Backup cleanup, log archival, and purge operations run in PurgeDataJobs and rConfigDefault queues.

Without Horizon running, these operations cannot execute—making Horizon critical to rConfig’s operation.

Accessing Horizon Dashboard

Navigate to the Horizon dashboard via External Tools → Queue Manager (opens in a new tab), or directly at:

https://your-rconfig-domain.com/horizon

Access is restricted to users with ‘admin’ role.

Horizon Dashboard Sections

Dashboard

High-level overview of queue system operations and health.

Key metrics displayed:

Jobs Per Minute: Current rate of job processing across all queues. Higher numbers indicate active operations (backups running, tasks executing). Lower numbers suggest idle periods or processing bottlenecks.

Processes: Number of active worker processes handling jobs. This should match your configured worker count in Supervisor. If significantly lower, workers may have crashed.

Jobs Past Hour: Total job volume over the last 60 minutes, showing workload trends and peak activity periods.

Failed Jobs: Count of jobs that encountered errors. Occasional failures are normal (device timeouts, connectivity issues), but high failure rates indicate systemic problems.

Job Status Distribution: Visual breakdown of pending, processing, completed, and failed jobs, providing at-a-glance health assessment.

Horizon Status: Shows whether Horizon is actively running and processing jobs or is inactive.

Recent Jobs: List of most recently processed jobs with execution time, status, and queue name.

The dashboard displays Jobs Per Minute, active Processes, Jobs Past Hour, Failed Jobs count, and current Horizon Status in a single view. An auto-reload button in the top right refreshes metrics continuously.

Metrics

Detailed metrics and performance data introduced in V7, providing historical analysis of queue system performance.

Available metrics:

Jobs Processed: Total job volume over selected time period, showing workload trends and identifying peak usage patterns.

Throughput: Jobs processed per time unit, indicating system processing capacity and efficiency.

Runtime Analysis: Job execution time statistics, helping identify performance bottlenecks and slow operations.

Jobs Failed: Failure count trends, helping identify when issues started and whether problems are increasing or decreasing.

Jobs Per Minute: Processing rate over time, useful for capacity planning and identifying bottlenecks.

Average Wait Time: How long jobs wait before processing begins. High wait times indicate insufficient worker capacity or queue backlogs.

Average Execution Time: How long jobs take to complete. Increasing execution times may indicate performance degradation.

Time Range Options: View metrics for past hour, 24 hours, 7 days, or custom ranges to identify patterns and trends.

The Metrics view shows throughput, runtime analysis, failure trends, and average wait/execution times over selectable time ranges (past hour, 24 hours, 7 days).

Batches

Monitor batch job processing status introduced in V7.

Batch information displayed:

Batch ID and creation timestamp
Total jobs in batch
Jobs completed, pending, and failed
Overall batch progress percentage
Individual job status within batch
Batch completion status

Batch benefits:

Non-blocking execution (individual failures don’t block other jobs)
Granular troubleshooting (identify specific failed devices)
Progress monitoring (see completion percentage in real-time)
Better resource utilization (jobs process in parallel)

Use cases: All scheduled device downloads, manual multi-device downloads, bulk compliance checks.

The Batches view lists each batch with its ID, creation time, total jobs, completed/pending/failed counts, and an overall progress percentage.

Pending Jobs

Jobs awaiting execution by Laravel application.

Information displayed:

Job type and class name
Queue name and priority
Wait time (how long in queue)
Job payload (parameters)

Normal vs. concerning pending states:

Normal: Short-lived pending state during active processing
Concerning: Jobs remaining pending for extended periods indicate worker capacity issues or processing problems

Completed Jobs

Historical record of successfully executed jobs.

Information displayed:

Job completion timestamp
Execution duration
Queue name
Job payload details

Note: History is cleared on system restart or rConfig update.

Failed Jobs

Dedicated view for jobs that encountered errors during execution.

Information displayed:

Exception type and message
Complete stack trace
Job payload showing exactly what was being processed
Failure timestamp
Queue name and connection
Number of retry attempts

Actions available:

Retry: Re-queue failed job for another attempt
Retry All: Re-queue all failed jobs in batch
Forget: Remove failed job from list (doesn’t fix underlying issue)
Forget All: Clear all failed jobs (use cautiously)

Understanding Job Types in rConfig

Different job types appear in Horizon corresponding to rConfig operations.

Common Job Types

Device Backup Jobs:

Class: App\Jobs\BackupRun
Purpose: Execute configuration download for specific device
Typical duration: 10-60 seconds depending on device and commands
Common failures: Connection timeouts, authentication errors, prompt mismatches

Scheduled Task Jobs:

Class: App\Jobs\TaskDownloadRun
Purpose: Execute scheduled operations (backups, compliance, maintenance)
Duration: Varies widely based on task scope
Common failures: Task configuration errors, device unavailability

SNMP Polling Jobs:

Class: App\Jobs\SnmpDevicePollingJob
Purpose: Collect device properties via SNMP
Typical duration: 1-10 seconds per device
Common failures: SNMP authentication errors, device timeouts

SNMP Trap Processing Jobs:

Class: App\Jobs\SnmpProcessTrapJob
Purpose: Process incoming SNMP traps
Typical duration: 1-5 seconds per trap
Common failures: Invalid trap format, database issues

Policy Compliance Jobs:

Class: App\Jobs\PolicyComplianceJob
Purpose: Validate configurations against defined policies
Typical duration: 5-30 seconds depending on policy complexity
Common failures: Invalid regex in policies, missing configuration files

Snippet Deployment Jobs:

Class: App\Jobs\DeviceSnippetJob
Purpose: Deploy configuration snippets to devices
Typical duration: 10-60 seconds depending on snippet size
Common failures: Connection errors, insufficient privileges, command syntax errors

Data Purge Jobs:

Classes: App\Jobs\PurgeTableJob, App\Jobs\PurgeFailedConfigsJob, App\Jobs\PurgePolicyComplianceReportsJob
Purpose: Clean old configurations, logs, and temporary files
Typical duration: Varies based on data volume
Common failures: Disk space issues, database lock contention

Report Generation Jobs (V8.4.0+):

Class: App\Jobs\BuildDownloadReportJob
Purpose: Generates the task report once a scheduled download or snippet batch completes
Typical duration: A few seconds
Common failures: Database write errors, disk write errors
Unlike most job types in rConfig, this job retries automatically up to 3 times with increasing delays between attempts before it’s marked as failed — a single transient error won’t stop a report from being generated

Interpreting Job Status

Pending: Job is queued but not yet picked up by a worker. Normal if queue has backlog. Concerning if jobs remain pending for extended periods (indicates worker issues).

Processing: Worker is actively executing the job. Should transition to completed or failed within reasonable timeframe based on job type.

Completed: Job finished successfully. Execution time shows how long it took. Review if times seem abnormally long.

Failed: Job encountered error during execution. Review exception details to determine cause.

Retrying: Failed job that’s been re-queued for another attempt. Monitor to see if retry succeeds.

Supervisor Configuration

Supervisor manages the Horizon process, ensuring it runs continuously and automatically restarts if it crashes. Proper Supervisor configuration is critical to Horizon reliability.

Understanding Supervisor

Supervisor is a process control system for Linux/Unix that:

Starts Horizon automatically when system boots
Monitors Horizon process health
Restarts Horizon if it crashes or exits unexpectedly
Provides logging for process startup, shutdown, and errors
Manages multiple worker processes if configured

Why Supervisor matters: Without Supervisor running Horizon, all background jobs stop processing. Scheduled tasks don’t execute, backups don’t run, and the queue fills with pending jobs.

Supervisor Configuration by OS

Supervisor configuration varies slightly between distributions but follows the same principles.

CentOS/RHEL/Rocky/Alma
Ubuntu/Debian

CentOS / RHEL / Rocky / Alma Linux

Supervisor configuration file location:

/etc/supervisord.d/horizon.ini

Configuration file contents:

[program:horizon]
; Horizon rConfig Configuration
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/rconfig8/current/artisan horizon
autostart=true
autorestart=true
user=root
redirect_stderr=true
stdout_logfile=/var/www/html/rconfig8/current/storage/logs/horizon.log
stopwaitsecs=3600

Configuration explanation:

command: Path to Horizon artisan command
autostart: Start Horizon when Supervisor starts
autorestart: Restart Horizon if it exits
user: Run as nginx user (web server user)
numprocs: Number of Horizon processes (typically 1)
stdout_logfile: Where Horizon logs are written
stopwaitsecs: Grace period before killing process (3600 seconds for long-running jobs)

Managing Supervisor:

# Check Supervisor status
systemctl status supervisord

# Start Supervisor
systemctl start supervisord

# Enable Supervisor to start on boot
systemctl enable supervisord

# Restart Supervisor
systemctl restart supervisord

Managing Horizon through Supervisor:

# Check Horizon worker status
supervisorctl status horizon

# Start Horizon worker
supervisorctl start horizon:*

# Stop Horizon worker
supervisorctl stop horizon:*

# Restart Horizon worker
supervisorctl restart horizon:*

# Reload Supervisor configuration
supervisorctl reread
supervisorctl update

Supervisor logs location:

/var/log/supervisor/supervisord.log

Ubuntu / Debian

Supervisor configuration file location:

/etc/supervisor/conf.d/horizon_supervisor.ini

Configuration file contents:

[program:horizon]
; Horizon rConfig Configuration
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/rconfig8/current/artisan horizon
autostart=true
autorestart=true
user=root
redirect_stderr=true
stdout_logfile=/var/www/html/rconfig8/current/storage/logs/horizon.log
stopwaitsecs=3600

Configuration explanation:

command: Path to Horizon artisan command
autostart: Start Horizon when Supervisor starts
autorestart: Restart Horizon if it exits
user: Run as root user
stdout_logfile: Where Horizon logs are written
stopwaitsecs: Grace period before killing process (3600 seconds for long-running jobs)

Managing Supervisor:

# Check Supervisor status
systemctl status supervisor

# Start Supervisor
systemctl start supervisor

# Enable Supervisor to start on boot
systemctl enable supervisor

# Restart Supervisor
systemctl restart supervisor

Managing Horizon through Supervisor:

# Check Horizon worker status
supervisorctl status horizon

# Start Horizon worker
supervisorctl start horizon:*

# Stop Horizon worker
supervisorctl stop horizon:*

# Restart Horizon worker
supervisorctl restart horizon:*

# Reload Supervisor configuration
supervisorctl reread
supervisorctl update

Supervisor logs location:

/var/log/supervisor/supervisord.log

Horizon Configuration via Environment Variables

Horizon’s behavior is controlled by environment variables in the .env file. These settings control worker processes, timeouts, and queue retry behavior.

Important Environment Variables

HORIZON_PROD_TIMEOUT=14400
HORIZON_ONE_MIN_PROCESS=1
HORIZON_TWO_MIN_PROCESS=1
HORIZON_MAX_PROCESS=10
HORIZON_DEFAULT_MAX_PROCESS=5
HORIZON_THREE_MIN_PROCESS=1
HORIZON_THREE_MAX_PROCESS=2
HORIZON_THREE_TIMEOUT=300
REDIS_QUEUE_RETRY_AFTER=14800

Variable descriptions:

HORIZON_PROD_TIMEOUT:

Number of seconds a job may run before termination as timeout
Default: 14400 seconds (4 hours), matching the queue timeout
Increase for unusually long-running jobs; decrease only if your workload is known to complete faster
Must be less than REDIS_QUEUE_RETRY_AFTER

HORIZON_ONE_MIN_PROCESS / HORIZON_TWO_MIN_PROCESS (V8.4.0+):

Minimum number of worker processes Horizon maintains, set independently per supervisor:
- HORIZON_ONE_MIN_PROCESS applies to HighPriorityQueue, PolicyCompliance, PurgeDataJobs, rConfigDefault, SnmpTrapQueue
- HORIZON_TWO_MIN_PROCESS applies to TaskDownloadQueue, TaskSnippetQueue, ManualDownloadQueue
Default: 1 for each
Ensures baseline processing capacity is always available on that group of queues, without affecting the other group
If neither variable is set, both fall back to the legacy HORIZON_MIN_PROCESS value for backward compatibility — existing installations that only have HORIZON_MIN_PROCESS set continue to work exactly as before with no changes required

HORIZON_MAX_PROCESS:

Maximum number of worker processes Horizon maintains
Default: 10
Applied to: TaskDownloadQueue, TaskSnippetQueue, ManualDownloadQueue
Scales up workers during high load periods

HORIZON_DEFAULT_MAX_PROCESS:

Maximum worker processes for default-priority queues
Default: 5
Applied to: HighPriorityQueue, PolicyCompliance, rConfigDefault, PurgeDataJobs, SnmpTrapQueue

HORIZON_THREE_MIN_PROCESS / HORIZON_THREE_MAX_PROCESS / HORIZON_THREE_TIMEOUT (V8.4.0+):

Minimum/maximum worker processes and job timeout for the dedicated ReportBuildQueue supervisor
Defaults: 1 minimum, 2 maximum, 300 second timeout
This queue handles report generation only — a low-volume, fast job — so it needs very few workers. It’s kept separate specifically so a busy or backlogged queue elsewhere never delays a report from being generated

REDIS_QUEUE_RETRY_AFTER:

Number of seconds to wait before retrying a failed job
Default: 14800 seconds (just over 4 hours)
Critical: Must be greater than HORIZON_PROD_TIMEOUT (and HORIZON_THREE_TIMEOUT) to prevent duplicate job processing
If updating any timeout, update this value accordingly

Worker Recycling (V8.4.0+)

Horizon worker processes are long-lived — unlike a web request, a worker keeps running and can process thousands of jobs over hours or days without restarting. Left unbounded, this can lead to gradually increasing memory usage over time. Two optional settings let a worker restart itself cleanly between jobs, never interrupting one that’s actively running:

HORIZON_ONE_MAX_TIME=3600
HORIZON_ONE_MAX_JOBS=500
HORIZON_TWO_MAX_TIME=3600
HORIZON_TWO_MAX_JOBS=500
HORIZON_THREE_MAX_TIME=3600
HORIZON_THREE_MAX_JOBS=500

_MAX_TIME: Number of seconds a worker runs before it restarts. Default: 3600 (1 hour).

_MAX_JOBS: Number of jobs a worker processes before it restarts. Default: 500.

A worker restarts when it reaches whichever limit comes first, then Supervisor immediately starts a replacement. Set either value to 0 to disable that particular limit. These are set independently for each supervisor (HorizonOne, HorizonTwo, HorizonThree) — the same three supervisor groups described under HORIZON_ONE_MIN_PROCESS/HORIZON_TWO_MIN_PROCESS above, plus the ReportBuildQueue supervisor.

Checking Horizon from System Optimization (V8.4.0+)

The Horizon Settings card under System Settings >> System Optimization shows two runtime rows above the worker settings:

horizon: whether Horizon is running, with its status, supervisor count, and PID.
worker_code: whether the workers started after the last change to the rConfig code, config cache, or .env. If it shows Needs Optimization, run php artisan horizon:terminate so Supervisor restarts the workers on the current code. It shows Not measurable when Horizon runs on a different host from the web server.

Redis memory use is reported on the same page. See Redis Memory Optimization.

Modifying Environment Variables

Edit the .env file located at /var/www/html/rconfig8/current/.env:

nano /var/www/html/rconfig8/current/.env

Add or modify the variables as needed. After making changes, clear application cache:

cd /var/www/html/rconfig8/current
php artisan rconfig:clear-all

Then restart Horizon if needed:

php artisan rconfig:clear-horizon
# Supervisor automatically restarts it

Important Considerations

Timeout and Retry Settings:

If updating HORIZON_PROD_TIMEOUT, ensure REDIS_QUEUE_RETRY_AFTER is also edited to a value higher than the timeout to prevent job duplication issues
Formula: REDIS_QUEUE_RETRY_AFTER should be at least HORIZON_PROD_TIMEOUT + 30 seconds

Database Connections:

When increasing HORIZON_MAX_PROCESS beyond typical configurations, check the database server’s maximum allowed connections
Default MySQL/MariaDB configurations may need adjustment to accommodate additional connections from increased worker processes
Each worker process can potentially use a database connection
Monitor database connection usage: SHOW STATUS LIKE 'Threads_connected';

Resource Planning:

More workers = higher CPU usage
More workers = higher memory usage
More workers = more database connections
Balance worker count against available system resources
Monitor system performance when adjusting worker counts

Troubleshooting Horizon Issues

Horizon Shows as Inactive

Symptoms: Horizon dashboard shows “Inactive” status, or jobs queue but never process.

Diagnosis:

Check Horizon status via command line:

supervisorctl status horizon

Expected output: Horizon is running.

If output is Horizon is inactive., Horizon is not running.

Check Supervisor status:

supervisorctl status horizon

Expected output: horizon RUNNING pid 12345, uptime 1:23:45

If output shows STOPPED, FATAL, or EXITED, Supervisor isn’t running Horizon.

Check Supervisor itself is running:

# CentOS/RHEL/Rocky/Alma
systemctl status supervisord

# Ubuntu/Debian
systemctl status supervisor

If Supervisor isn’t running, Horizon can’t run.

Resolution Steps:

Step 1: Ensure Supervisor is running:

# CentOS/RHEL/Rocky/Alma
systemctl start supervisord
systemctl enable supervisord

# Ubuntu/Debian
systemctl start supervisor
systemctl enable supervisor

Step 2: Update Supervisor configuration:

# Reload Supervisor configuration files
supervisorctl reread

# Apply configuration changes
supervisorctl update

Step 3: Start Horizon worker:

supervisorctl start horizon:*

Step 4: Verify Horizon is running:

php artisan horizon:status

Should return Horizon is running.

Step 5: Check Horizon dashboard:

Navigate to External Tools → Queue Manager and verify dashboard shows active status and processing metrics.

Supervisor Not Starting Horizon

Symptoms: Supervisor shows FATAL or EXITED status for horizon.

Diagnosis:

Check Supervisor logs for errors:

# CentOS/RHEL/Rocky/Alma
tail -n 50 /var/log/supervisor/supervisord.log

# Ubuntu/Debian
tail -n 50 /var/log/supervisor/supervisord.log

Check Horizon logs for application errors:

tail -n 50 /var/www/html/rconfig8/current/storage/logs/horizon.log

Common causes and resolutions:

1. Incorrect file paths in configuration:

Verify paths in Supervisor config match your installation:

# Check rConfig installation path
ls -la /var/www/html/rconfig8/current/artisan

# Verify PHP path
which php

Update Supervisor config if paths incorrect, then:

supervisorctl reread
supervisorctl update
supervisorctl start horizon:*

2. Permission issues:

Horizon must run as web server user:

# Check ownership of rConfig directory
ls -la /var/www/html/rconfig8/current/

# Should be owned by nginx (CentOS) or www-data (Ubuntu)
# If not, fix permissions:

# CentOS/RHEL/Rocky/Alma
chown -R nginx:nginx /var/www/html/rconfig8/current/

# Ubuntu/Debian
chown -R www-data:www-data /var/www/html/rconfig8/current/

3. PHP memory limit too low:

You should review our docs under System Settings to set optimal PHP settings.

Jobs Stuck in Pending Status

Symptoms: Jobs queue but never process, remaining in “Pending” status indefinitely.

Diagnosis:

Check if workers are processing jobs:

php artisan horizon:status

Check worker count in Horizon dashboard:

Navigate to External Tools → Queue Manager
Check “Processes” count
Should match configured numprocs in Supervisor config (typically 1) plus additional workers up to max

Resolution:

If Horizon shows as running but no workers active:

Terminate and restart Horizon:

php artisan rconfig:clear-horizon
# Wait 5 seconds for graceful shutdown
supervisorctl start horizon:*

If workers are active but jobs still pending:

Check for queue priority issues. Jobs in higher priority queues process first. Lower priority jobs may wait if high priority queue is busy.

Increase worker count if consistently backlogged:

Edit .env file:

nano /var/www/html/rconfig8/current/.env

# Increase max processes slightly
HORIZON_MAX_PROCESS=15
HORIZON_DEFAULT_MAX_PROCESS=8

Clear cache and restart:

php artisan rconfig:clear-all
php artisan rconfig:clear-horizon

Jobs Failing Repeatedly

Symptoms: High failure rate in Horizon dashboard, many jobs in Failed Jobs section.

Diagnosis:

Navigate to External Tools → Queue Manager → Failed Jobs
Click on recent failed job to view details
Review exception message and stack trace
Identify common failure patterns

Common causes and resolutions:

Device connectivity failures:

Exception: Connection timeout, SSH connection refused
Resolution: Verify device reachability, check credentials, verify firewall rules
Action: Fix device/network issues before retrying jobs

Authentication failures:

Exception: Authentication failed, Invalid credentials
Resolution: Update device credentials in rConfig
Action: Correct credentials, then retry failed jobs

Configuration errors:

Exception: Command not found, Invalid regex pattern
Resolution: Fix command configuration or policy definitions
Action: Correct configuration, then retry jobs

System resource exhaustion:

Exception: Out of memory, Maximum execution time exceeded
Resolution: Increase PHP memory limit, optimize job processing
Action: Increase resources or reduce concurrent job count

After resolving root cause:

Retry failed jobs:

Click Retry All in Failed Jobs section
Or retry individual jobs after fixing specific issues

Horizon Process Consuming Excessive Resources

Symptoms: High CPU or memory usage by Horizon process, system slowdown.

Diagnosis:

Check resource usage:

# View Horizon process resource usage
ps aux | grep horizon

# Monitor in real-time
top -p $(pgrep -f horizon)

Common causes and resolutions:

Too many concurrent workers:

Reduce max processes in .env:

nano /var/www/html/rconfig8/current/.env

# Reduce max processes
HORIZON_MAX_PROCESS=5
HORIZON_DEFAULT_MAX_PROCESS=3

# Clear cache and restart
php artisan rconfig:clear-all
php artisan rconfig:clear-horizon

Large job payload or memory leaks:

Restart Horizon periodically to clear memory:

php artisan rconfig:clear-horizon
# Supervisor automatically restarts it

Schedule regular restarts (add to crontab):

# Restart Horizon daily at 3 AM
0 3 * * * cd /var/www/html/rconfig8/current && php artisan rconfig:clear-horizon

Inefficient jobs:

Review job execution times in Horizon metrics. Jobs taking >5 minutes may need optimization:

Reduce scope (fewer devices per job)
Optimize command sets
Reduce SNMP OID count
Split large operations into smaller jobs

Horizon Dashboard Inaccessible

Symptoms: Cannot access Horizon dashboard at /horizon, 404 error or blank page.

Resolution:

Clear application cache:

cd /var/www/html/rconfig8/current
php artisan config:clear
php artisan route:clear
php artisan view:clear
php artisan cache:clear

Verify Horizon is published:

php artisan horizon:install

Check web server configuration:

Ensure web server (Nginx/Apache) is configured to serve rConfig properly and routes are accessible.

Restart services:

systemctl restart nginx php-fpm  # For Nginx
systemctl restart httpd php-fpm  # For Apache

Best Practices

Monitoring

Review Horizon dashboard daily: Check for failed jobs, unusual processing times, or growing queue depths indicating issues.

Monitor system resources: Track CPU, memory, and disk usage to ensure adequate capacity for queue processing.

Set up alerting: Configure monitoring tools (Nagios, Zabbix, etc.) to alert when:

Horizon becomes inactive
Failed job count exceeds threshold
Queue depth grows continuously
Worker processes are not running

Review batch status: Check batch completion rates to identify systematic issues with specific device groups or operations.

Maintenance

Restart Horizon after deployments: Always terminate and restart Horizon after deploying rConfig updates:

php artisan rconfig:clear-horizon
# Supervisor automatically restarts it

Clear failed jobs periodically: After resolving issues and retrying jobs, clear old failed jobs to keep interface clean:

Review failed jobs to understand failure patterns
Retry jobs that might succeed after fixes
Forget jobs that are no longer relevant

Review performance metrics: Weekly review of Horizon metrics identifies trends:

Growing average execution times may indicate performance degradation
Increasing failure rates suggest systemic issues developing
High wait times indicate insufficient worker capacity

Monitor batch failures: Review batch completion statistics to identify patterns:

Specific devices failing consistently
Time-based failure patterns
Queue-specific issues

Capacity Planning

Baseline your workload: Understand normal job volume and execution times to identify when capacity expansion is needed.

Scale workers appropriately: Add workers as device count grows, but balance against system resources:

Monitor database connections usage
Track PHP memory consumption
Watch CPU utilization trends

Stagger scheduled tasks: Distribute task schedules across time to prevent queue spikes overwhelming workers.

Plan for peak loads: Ensure system can handle peak activity periods (scheduled backup windows) without excessive delays:

Test with increased worker counts during off-peak hours
Monitor resource usage during peak periods
Adjust worker counts based on observed patterns

Database connection planning: Calculate maximum potential database connections:

Formula: HORIZON_MAX_PROCESS * number_of_queues * concurrent_jobs_per_worker
Ensure MySQL/MariaDB max_connections setting accommodates this plus regular web traffic
Default MySQL max_connections is often 151—may need increase for larger deployments

What’s next

Scheduled Tasks - Configure automated operations processed by Horizon
Configuration Backups - Understanding backup jobs in Horizon
Policy Compliance Results - Compliance jobs processing
Application Log - Additional logging beyond Horizon
Redis Memory Optimization - Size Redis memory for the job queue

Quick Reference

Common Commands

# Check Horizon status
php artisan horizon:status

# Terminate Horizon gracefully
php artisan rconfig:clear-horizon

# Pause Horizon (stop processing new jobs)
php artisan horizon:pause

# Continue Horizon (resume processing)
php artisan horizon:continue

# Clear all caches after .env changes
php artisan rconfig:clear-all

# Check Supervisor status
supervisorctl status horizon

# Start Horizon worker
supervisorctl start horizon:*

# Restart Horizon worker
supervisorctl restart horizon:*

# View Horizon logs
tail -f /var/www/html/rconfig8/current/storage/logs/horizon.log

# View Supervisor logs
tail -f /var/log/supervisor/supervisord.log

Environment Variables Quick Reference

# Standard values in .env file
HORIZON_PROD_TIMEOUT=14400
HORIZON_ONE_MIN_PROCESS=1
HORIZON_TWO_MIN_PROCESS=1
HORIZON_MAX_PROCESS=10
HORIZON_DEFAULT_MAX_PROCESS=5
HORIZON_ONE_MAX_TIME=3600
HORIZON_ONE_MAX_JOBS=500
HORIZON_TWO_MAX_TIME=3600
HORIZON_TWO_MAX_JOBS=500
HORIZON_THREE_MIN_PROCESS=1
HORIZON_THREE_MAX_PROCESS=2
HORIZON_THREE_TIMEOUT=300
HORIZON_THREE_MAX_TIME=3600
HORIZON_THREE_MAX_JOBS=500
REDIS_QUEUE_RETRY_AFTER=14800

Critical rule: REDIS_QUEUE_RETRY_AFTER must be greater than HORIZON_PROD_TIMEOUT and HORIZON_THREE_TIMEOUT to prevent duplicate job processing.

Legacy variable: HORIZON_MIN_PROCESS still works as a fallback if HORIZON_ONE_MIN_PROCESS/HORIZON_TWO_MIN_PROCESS aren’t set — see Important Environment Variables above.

Queue Priority Order

HighPriorityQueue - Critical jobs
ManualDownloadQueue - User-triggered downloads
PolicyCompliance - Compliance validation
TaskDownloadQueue - Scheduled downloads
TaskSnippetQueue - Snippet deployments
SnmpTrapQueue - SNMP trap processing
PurgeDataJobs - Data cleanup
rConfigDefault - General system jobs
ReportBuildQueue - Report generation (V8.4.0+)
default - Fallback (rarely used)

Troubleshooting Checklist

Supervisor Configuration Locations

CentOS / RHEL / Rocky / Alma:

Config: /etc/supervisord.d/horizon.ini
Logs: /var/log/supervisor/supervisord.log
Service: supervisord
User: nginx

Ubuntu / Debian:

Config: /etc/supervisor/conf.d/horizon_supervisor.ini
Logs: /var/log/supervisor/supervisord.log
Service: supervisor
User: www-data

Common Horizon Issues and Quick Fixes

Issue	Quick Fix
Horizon inactive	`supervisorctl start horizon:*`
Jobs stuck pending	`php artisan rconfig:clear-horizon`
Failed jobs piling up	Review failed job details, fix root cause, then retry
High memory usage	Reduce HORIZON_MAX_PROCESS in .env
Dashboard inaccessible	`php artisan config:clear && systemctl restart nginx`
After .env changes	`php artisan rconfig:clear-all && php artisan rconfig:clear-horizon`
Worker not starting	Check Supervisor logs, verify paths and permissions

Resource Planning Guidelines

Small deployment (< 100 devices):

HORIZON_ONE_MIN_PROCESS=1
HORIZON_TWO_MIN_PROCESS=1
HORIZON_MAX_PROCESS=5
HORIZON_DEFAULT_MAX_PROCESS=3
HORIZON_THREE_MAX_PROCESS=1
PHP memory_limit=256M
MySQL max_connections=151 (default)

Medium deployment (100-500 devices):

HORIZON_ONE_MIN_PROCESS=1
HORIZON_TWO_MIN_PROCESS=1
HORIZON_MAX_PROCESS=10
HORIZON_DEFAULT_MAX_PROCESS=5
HORIZON_THREE_MAX_PROCESS=2
PHP memory_limit=512M
MySQL max_connections=300

Large deployment (500+ devices):

HORIZON_ONE_MIN_PROCESS=2
HORIZON_TWO_MIN_PROCESS=2
HORIZON_MAX_PROCESS=15
HORIZON_DEFAULT_MAX_PROCESS=8
HORIZON_THREE_MAX_PROCESS=2
PHP memory_limit=1024M
MySQL max_connections=500

Note: ReportBuildQueue (HORIZON_THREE_*) handles a low, steady volume of report jobs regardless of device count, so it rarely needs scaling beyond the defaults even in large deployments. Adjust the other values based on actual system resources and workload patterns. Monitor performance and scale incrementally.