How to fine tune the Bizagi Scheduler in Cluster Environments

<< Click to Display Table of Contents >>

Navigation:  Automation Server > Automation Server configuration and administration guide > System maintenance and monitoring > Environment settings and administration > Management Console > Scheduled jobs administration >

How to fine tune the Bizagi Scheduler in Cluster Environments

Overview

Configuring Bizagi Scheduler for better performance depends on each environment's configuration. It is dependent on three factors: the client's infrastructure, parallelism, and the client's modeling. The best practice when tuning your Bizagi Scheduler is to include performance check tasks that work together with load tests. This allows Scheduler configuration fine-tuning to obtain the best combination in which the performance is optimal.

 

The following document presents a brief explanation of the keys used to do a Scheduler fine tuning. Furthermore, it gives you details about the master and slave schedulers and details on how you can tune up the performance for your Production environment.  

 

Definitions

Take into account the following definitions:

A Master scheduler is the scheduler which coordinates the tasks executed per each slave scheduler and runs additional tasks such as the maintenance tasks. It is mandatory to have one and only one Master scheduler configured.

A Slave scheduler is the scheduler which processes an assigned task.

A core is the individual processor within a CPU.

Processor cores are physical cores within a CPU. For example, if a server has a QUAD-CORE processor, it has four cores.

Threads are a virtual version of a CPU core executing one CPU instruction.

Logical processor, or logical cores, are the number of physical cores times the number threads that can run on each core. This is usually known as HyperThreading. For example, if you have a QUAD-CORE processor with two threads per core, you have 8 logical processors or logical cores.

Multithreading is the capacity of executing multiple CPU instructions at the same time. This can be achieved in two ways: Using multiple cores, or with hyperthreading.

 

Here is graphic as an example of the definitions:

 

Scheduler_05

 

Measuring your infrastructure

The first step is to measure the Scheduler's processing capabilities using the following metrics and measurements:

 

CPU consumption of the server where Schedulers are running

Count of the pending jobs/tasks

 

note_pin

Although there are multiple metrics and variables that you can considered to measure the performance of a server or system, this sections focuses on the performance of the Bizagi Scheduler.

 

To count the number of pending jobs/tasks, run the following queries in the project database:

 

Task  type

Table

Query to fetch pending tasks

Asynchronous activities

ASYNCHWORKITEM

select count(*) from ASYNCHWORKITEM where awState != 0 AND awProcessing != 1

Timers and other jobs

JOB

select count(*) from job where jobEnabled = 1 and jobNextRunTime < GETDATE()

 

Using these two metrics you can decide whether you need to fine-tune the existing Scheduler, or scale-out the infrastructure. The following algorithm guides you through the scale pattern:

 

SchedulerScalingPatterns

 

If your server CPU usage is under 60% you can increase the number of threads in the Scheduler safely, as long as you have threading capabilities. Remember that you need to consider the following parameters: MaxThreadsForCustomJobs, MaxThreadsForJobs, and MaxThreadsForAsynchWorkitems. If your CPU usage is between 60% and 75% you can still increase the Scheduler threading parameters, as long as your server is not affected by other metrics like RAM or disk usage. This depends entirely on your server configuration. If your server exceeds 75% of CPU usage, you can either increase your hardware in the same server, by adding more physical processors or add a new server with a new scheduler. If possible, always scale-out servers, because with one new server you are also including RAM and disk capabilities.

 

We recommend leaving one Scheduler per server and increase the threading capabilities per Scheduler.

 

Configuration files

To fine tune the Bizagi scheduler, locate these files:

 

Locate the web.config file located in C:\Bizagi\Projects\[ProjectName]\WebApplication and open it using the text editor of your choice.

 

FineTune_01

 

Locate the BizAgi.Scheduler.Services.exe.config files located in C:\Bizagi\Projects\[ProjectName]\Scheduler folder of every scheduler.

 

FineTune_02

 

Add the configuration keys required to fine tune your Bizagi Scheduler in the <appsettings> key. Example:

<add key="AsyncTaskWorkingSetSize" value="10"/>

 

Configuration keys

According to the infrastructure of your project, take into account the following minimum configuration:

 

Configuration Key

Description

Default values

Recommendations

Threaded Scheduler configuration

ID_STACK_SIZE

Add this key in the web.config and BizAgi.Scheduler.Services.exe.config files of each scheduler. The key reduces the locks on business IDs assignation during  high rates of case creation.

The default value is 50.

This value should never be decreased and each scheduler must have the same value.

AsynchTaskWorkingSetSize

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the number of asynchronous work items to be retrieved from the database to be processed per thread.

The default value is 10.

The Scheduler reviews within intervals if there are additional tasks to execute.

 

If this value is too low for the expected demand for Scheduler tasks, the Scheduler will take more iterations to retrieve all pending tasks.

JobWorkingSetSize

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the number of tasks of the stack that will be provided to be executed.

The suggested value is 10.

The Scheduler reviews within intervals if there are additional tasks to execute.

 

If this value is too low for the expected demand for Scheduler tasks, the Scheduler will take more iterations to retrieve all pending tasks.

SchedulerAsyncProcessorIntervalInMillis

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key sets the time in milliseconds that define the threads at internal scheduler processor to wait for the next execution of the asynchronous tasks when the BizagiSchedulerInterval was executed.

The suggested value is 500.

If the size of the tasks' batch increases for asynchronous activities, the interval value must be increased. This allows to process more tasks in a greater interval per cycle.

 

If the interval is too long, you are losing processing time, and the number of pending tasks can increase.

SchedulerMaxJobWaitInterval

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the number of wait execution time when the jobs max number is reached.

The suggested value is 1.

If the size of the tasks' batch increases, this interval value must be increased. This let to process more tasks in a greater interval per cycle. It also depends on JobWorkingSize.

 

If the interval is too long, you are losing processing time, and the number of pending tasks can increase.

LaunchThreadForLog

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines you need an extra thread to improve the logging.

The suggested value is 1.

You can change the value to 0 when there are deadlocks.

Enable a threaded option for each function

MaxThreadsForCustomJobs

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the maximum number of threads to run the custom jobs.

The default value is 2.

Take into account that this number CANNOT exceed the number of threading processing capacity of the server. For example, if you have two quad-core processors with two threads per core, the maximum of MaxThreadsForCustomJobs should be 16.

 

This value must consider your hardware resources and the frequency of Scheduler's task execution.

MaxThreadsForJobs

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the maximum number of threads to run the jobs.

The default value is 2.

Take into account that this number CANNOT exceed the number of threading processing capacity of the server. For example, if you have two quad-core processors with two threads per core, the maximum of MaxThreadsForJobs should be 16.

 

This value must consider your hardware resources and the frequency of Scheduler's task execution.

MaxThreadsForAsyncEcm

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the maximum

number of threads to run the ECM when the asynchronous feature is active.

The default value is 2.

Take into account that this number CANNOT exceed the number of threading processing capacity of the server. For example, if you have two quad-core processors with two threads per core, the maximum of MaxThreadsForAsyncJobs should be 16.

 

This value must consider your hardware resources and the frequency of Scheduler's task execution.

MaxThreadsForAsynchWorkitems

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the maximum number of threads to run the asynchronous tasks.

The default value is 10.

Take into account that this number CANNOT exceed the number of threading processing capacity of the server. For example, if you have two quad-core processors with two threads per core, the maximum of MaxThreadsForAsyncWorkItemsJobs should be 16.

 

This value must consider your hardware resources and the frequency of Scheduler's task execution.

Scheduler execution interval

BizagiSchedulerInterval

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key defines the time (in seconds) used by Bizagi to process the work items retrieved  from the database.

The default value is 30.

If the size of the tasks' batch increases, the interval value must be increased. This lets to process more tasks in a greater interval per cycle.

 

If the interval is too long, you are losing processing time, and the number of pending tasks can increase.

MASTER / SLAVE

DisableAsynchCaseClosing

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key is used to name the primary scheduler and ensure that this scheduler will be the only one which runs the maintenance tasks.

The default value is 0.

Set 0 for the Master scheduler and 1 for the Slave scheduler. It is mandatory to have one and only one Master scheduler configured.

DisableInterfaceErrorLogger

Add this key in the BizAgi.Scheduler.Services.exe.config files of each scheduler. This key is used to enable the interface error logger only on the primary scheduler and ensures that this scheduler will be the only one which log all the interface errors.

The default value is 0.

Set 0 for the Master scheduler and 1 for the Slave scheduler. It is mandatory to have one and only one Master scheduler configured.

 

note_pin

Considerations:

The parameters MaxThreadsForCustomJobs, MaxThreadsForJobs, and MaxThreadsForAsynchWorkitems, indicate the maximum number of threads generated by the Scheduler to be executed by any of the logic processors. Nevertheless, software can generate more threads, depending on how fast the logical processors execute them and leave space to execute more. Hence, there is a difference between generated threads and logical processors (threads seen by the processor).

So you can increase the number of threads generated up by the Scheduler to be greater than the number of logical processors multiplying the number of generated threads by the total number of logical processors times 5. Do not increase the value greater than this calculation.

When using Virtual machines, the sum of the parameters MaxThreadsForCustomJobs, MaxThreadsForJobs, and MaxThreadsForAsynchWorkitems should be lower or equals than the number of vCPUs multiplied by the number of logical processors multiplied by the number of generated threads.

 

Tuning up the Scheduler

The tuning of the values for the present keys must be performed according to the results of the performance testing phase of your project. Follow these recommendations to tune up the performance:

 

It is recommended to have an independent machine for the Schedulers.

The ideal performance happens when the number of asynchronous workitems (MaxThreadsForAsynchWorkitems key) is processed in the time frame defined by Bizagi Scheduler Interval (BizagiSchedulerInterval key). Increase the AsynchTaskWorkingSetSize and BizagiSchedulerInterval key values until your asynchronous tasks are processed in an optimal time.

If you reach the maximum value for the MaxThreadsForAsynchWorkitems key, it is recommended to add another scheduler using the default values and then, increase the values until you have the ideal performance.

 

note_pin

It is recommended to perform continuous monitoring of your infrastructure. If you find any increase in terms of Memory or CPU usage, you have to take the decision if you want to tune the schedulers or perform changes over the infrastructure to be able to support the increased demand of resources.

 

Scheduler scaling example

For the sake of example, we have defined a scenario with the following characteristics:

 

75% of the tasks are service tasks configured as asynchronous activities.

The case creation rate is one case per hour.

Each case has on average 25 asynchronous activities executed in parallel.

 

First Scenario: Optimum performance

Due to the high demand of asynchronous activities, the ASYNCWORKITEM table is queried constantly to review pending tasks from the Scheduler. With the following results:

 

Scheduler_01

 

This scenario shows that there is no increment in pending tasks, nor a CPU usage greater than 60%, therefore, you do not need to fine-tune or scale-out.

 

Second Scenario: Fine-tuning the Scheduler

In a second scenario, the system is monitored with the following results:

 

Scheduler_02

 

There is evidence that the number of pending tasks to be executed by the Scheduler is becoming greater, and is increasing over time, meaning that the Scheduler es queuing tasks. Nevertheless, the CPU usage remains on average under 60%, therefore, you can fine-tune the Scheduler to avoid having a cumulative queue of pending tasks.

 

To tune this you can increase the number of threads for asynchronous activities. You can adjust the MaxThreadsForAsynchWorkItems.

 

After the parameter is tuned, the number of pending tasks starts to become lower to reach zero pending tasks.

 

note_pin

You need to assess if the load of the Scheduler is due to Asynchronous tasks, custom jobs, or another type of job. You need to compare the number of pending tasks from the ASYNCHWORKITEM table and the JOB table. Depending on this, you can either increase the threads for asynchronous items (MaxThreadsForAsynchWorkItems) or for another type of job (MaxThreadsForJobs or MaxThreadsForCustomJobs).

 

Third Scenario: Scaling-out the server

In the third scenario the system is monitored during the same times with the following results:

 

Scheduler_03

 

The pending tasks are still being queued because there is an increasing trend on the pending tasks (blue bars). But, in this scenario the CPU usage average is around 80%, above the suggested limit of  60%. These metrics show that the system is overloaded and the CPU resources are not enough to handle all the Scheduler requests.

 

In this scenario, you need to evaluate first if it is possible to scale-out the number of logical processors of your CPU. If yes, you can increase the number of cores or logic processors, and then look to fine-tune the Scheduler by increasing the threads for asynchronous tasks, like the second scenario.

 

If scaling-out the logic processors in the same server is not possible, you can look to scale-out your infrastructure with and additional Scheduler instance. Our recommendation is to keep a maximum of two Schedulers per server. However, this depends on your hardware performance, and if you have more software under the same server.

 

Scheduler_04