rConfig - Queue Manager

2 mins V7 Pro V6 Core

The majority of rConfig’s backend operations leverage Laravel Jobs and Queues system, facilitating the creation of jobs that can be executed in the background. Although a deep understanding of the queueing system is not necessary for most rConfig administrators, a grasp of the fundamentals can be beneficial for basic troubleshooting. Version 7 of rConfig introduces several improvements to the queueing scheduler, including new queue types and a batch queueing system.

Note

The queuing system is very powerful and can allow many hundreds of devices to be downloaded at once using parallel processing. However, it is recommended that a support ticket with rConfig is created to assist in the implementation of the adjustments to the queueing system, as some changes can be detremental if enough resources are not available.

How it works

As of rConfig V7 we have introduced new queues, and a batch queueing system. There are new queues TaskDownloadQueue, TaskSnippetQueue, ManualDownloadQueue, PolicyCompliance, rConfigDefault. All queues are processed in a First In First Out FIFO algorithm. The ‘default’ queue which belongs to ‘supervisor-1’ process is not used.

Queue	Description
TaskDownloadQueue	All scheduled tasks are sent to this queue and are processed in parallel but also in order
TaskSnippetQueue	Manual send of snippets are sent to this queue
ManualDownloadQueue	Manual download of devices are sent to this queue
PolicyCompliance	Policy compliance jobs are sent to this queue
rConfigDefault	This is a final default queue for jobs such as system backup, send email etc..

Batches are a queueing concept available since rConfig V6. Previously, rConfig would process all devices in a scheduled task as a single job. This meant any failures or delays would affect the entire job. No, with batches, all devices to be downloaded in a single scheduled (or manually run) task, will be sent in a ‘batch’, then delivered to the queue ‘downloadqueue’ as individual jobs. This will allow for non-blocking of queues and greater troubleshooting. i.e. of a job or jobs in the batch fails, ir will not block the entire job, all other jobs will complete unimpeded and you can review the batch for any failures. See output below.

rConfig deploys with a Laravel Queue monitoring tool called Laravel Horizon. Access to the Queue Manager is available through the settings menu, and it is restricted to users who possess the ‘admin’ role.

Note

Horizon provides a beautiful dashboard and code-driven configuration for your Laravel powered Redis queues. Horizon allows you to easily monitor key metrics of your queue system such as job throughput, runtime, and job failures.

There are 4 Important sections to the Horizon Application

• The Dashboard
  • High level view of running, completed and failed jobs
  • Horizon and Queuing status
• Metrics (new in V7)
  • Detailed metrics of the queue system
  • Jobs, Throughput, Runtime, etc.
• Batches
  • As of V7 all device download jobs are sent as a batch of jobs. You can monitor status of those batches here.
• Pending Jobs
  • Jobs awaiting executing by the Laravel applications
• Completed Jobs
  • History of all completed jobs.
  • History will be flush on system restart, or rConfig update
• Failed Jobs
  • View log and details of any failed jobs.
  • Any failed jobs should be notified to rConfig Support for investigation
  • Normally, this should be empty

As of v6.11.0, under certain conditions some of the default Horizon queue settings need to be changed. This should be done with caution and only after consultation with rConfig support. Each extra process or increased timeout will consume more resources on the server, this needs to be monitored carefully. The default settings are as follows;

HORIZON_PROD_TIMEOUT=60
HORIZON_MIN_PROCESS=1
HORIZON_MAX_PROCESS=10
HORIZON_DEFAULT_MAX_PROCESS=5
REDIS_QUEUE_RETRY_AFTER=130

These settings can be changed (or if not already present, added) in the .env file. Simply the description for each setting is as follows;

• HORIZON_PROD_TIMEOUT
  • The number of seconds a job may run before it is terminated as a timeout
• HORIZON_MIN_PROCESS
  • The minimum number of processes Horizon should maintain - default 1 - applied to all queues
• HORIZON_MAX_PROCESS
  • The maximum number of processes Horizon should maintain - default 5 - applied to ‘TaskDownloadQueue’, ‘TaskSnippetQueue’, ‘ManualDownloadQueue’ queues
• HORIZON_DEFAULT_MAX_PROCESS
  • The default number of processes Horizon should maintain - default 5 - applied to ‘PolicyCompliance’, ‘rConfigDefault’ queues
• REDIS_QUEUE_RETRY_AFTER
  • The number of seconds to wait before retrying a failed job - default 180

If you update the .env file with new settings, you must run the clear-all command. This can be done from the command line with the following command;

php artisan rconfig:clear-all

Note

Increasing the timeout and max processes will necessitate additional resources on the server, which typically requires an increase in the default PHP memory limit as well. Please refer to the detailed steps provided here: https://www.rconfig.com/support-center/kb-articles/view/increase-php-memory-limit

Environment Variables

Environment variables play a crucial role in configuring our Horizon queueing system. Here are key variables and guidelines on when and how they should be adjusted:

• HORIZON_MIN_PROCESS: Sets the minimum number of processes for each supervisor. Adjust according to base load expectations.
• HORIZON_MAX_PROCESS: Sets the maximum number of worker processes. Note that increasing this value significantly might require an increase in the MySQL/MariaDB max connection count, as each process could potentially use a database connection.
• HORIZON_PROD_TIMEOUT: Configures the timeout for jobs in seconds in the HorizonOne supervisor. It’s important to adjust REDIS_QUEUE_RETRY_AFTER if this value is updated to ensure it is greater than HORIZON_PROD_TIMEOUT to prevent duplicate job processing.

Important Notes

1. Timeout and Retry Settings:
   • If updating the HORIZON_PROD_TIMEOUT value, ensure that the REDIS_QUEUE_RETRY_AFTER environment variable is also edited to a value higher than the timeout to prevent job duplication issues.
2. Database Connections:
   • When increasing HORIZON_MAX_PROCESS beyond typical configurations, check the database server’s maximum allowed connections. Default configurations for MySQL/MariaDB may need adjustment to accommodate additional connections from increased worker processes.