Multiple Scheduler services configuration
Each Bizagi project needs an installed service called the Scheduler (installed within Automation Server).
The scheduler is an agent run by Bizagi which is in charge of performing system maintenance, offline and asynchronous tasks, and other jobs which run in the background and can be scheduled (set to fire off periodically).
When running your processes in a .NET platform, the Scheduler runs as a Windows service having its name refer to the specific project's name.
The Scheduler is supported in cluster configuration, in order to support the load balancing of such background tasks.
The following image illustrates a high availability Bizagi system architecture, involving a redundant Scheduler service:
However, for projects strongly relying on background processing, you may install and use more than one Scheduler service, whether by installing them on separate servers (cluster's nodes), by installing them on the same server, or in a combination of the above:
When setting up multiple Scheduler instances, note that though you may have more than 1 service per node, it is always recommended to have these as independently as possible (each using its own dedicated server and resources).
This is so in order to avoid single points of failure (so that in the event of a failure in one node, it does not affect more than 1 Scheduler instance).
For example, you may have separate servers running Scheduler services (if helpful for your project), regardless of having there a Work Portal of Bizagi.
Why configure multiple Scheduler services?
By default each Bizagi project operates with one instance of the Scheduler service.
However, you may configure the use of multiple Scheduler services so that these all work on a batch of background tasks in parallel.
This allows you to achieve either, or both of these 2 aspects:
1. Configuration of a high availability system architecture.
In this setup you would have a Bizagi cluster for load-balancing capabilities, each node having Automation Server operational with its 2 main components: Bizagi Work portal (web application) and the Scheduler.
This way, your system architecture avoids single points of failure (should one node fail, the other continues operating avoiding service disruption).
For more information about setting up a Bizagi cluster refer to Clusters and server management.
2. Scalability of the Scheduler.
In some scenarios and according to the characteristics of the project's implementation, using more than one Scheduler service which processes the background tasks is really helpful to ensure that such background tasks are not taking too much time in queue.
According to your specific analysis on the amount of tasks at a given time that the Scheduler needs to process (i.e sizing), and their expected SLA, you could install and configure more than one Scheduler within the same server, and without the need to do the same for the Work portal (scaling out only the Scheduler).
Before you start
The steps described below, part from your project being already setup (Automation Server properly configured, whether in a cluster or in a single-server system architecture).
Note that though the multiple Scheduler services will distribute the load of background tasks between them, acknowledge the following treatment:
1. Only one Scheduler will perform system maintenance.
System maintenance carried out by the Scheduler involves moving out information from transactional tables in the database and it is carried out at non-working hours.
When having multiple Schedulers, it is important that you predefine which Scheduler will be in charge of performing system maintenance so that you can configure this one accordingly (a "master" Scheduler).
2. Each Scheduler will take a batch of 10 tasks.
Whenever each Scheduler service is ready to process tasks (according to the Scheduler interval definition, as set as described at Bizagi parameters configuration), it will previously lock a batch of 10 tasks even though it will work on one at a time, and it is uncertain how much time will it take to process the first one (or this complete batch).
Batch of 10 tasks, and background tasks processed by the Scheduler in general, follow a FIFO priority algorithm (a First in, First out queue).
The above implies that if the amount of incoming tasks to process at a given moment (in a very short period of time), does not exceed 10, then it is likely that using additional Scheduler service will not balanced the load between them.
What you need to do
To configure multiple Scheduler services, these steps are carried out:
1. Install your additional Scheduler services.
This is done either through the Management Console clustering options or through manual installation.
Special naming considerations to be considered when installing multiple Scheduler services in a same server.
2. Configure the additional Scheduler services.
Recall that only one Scheduler must perform system maintenance.
Additional configuration oriented to the number of threads ran by each Scheduler is optional to optimize parallelism.
Follow the steps below.
1. Install your additional Scheduler services
Additional Scheduler services are created through the Management Console clustering options when creating a first Scheduler service in a given server.
To view more information about setting up a Bizagi cluster that already includes an additional Scheduler, refer to Clusters and server management.
If you are installing multiple Scheduler services in a same server, carry out the following:
1.1 Create a copy (clone) of the Scheduler folder.
Copy, paste and rename the folder located by default at C:\Bizagi\Projects\[your_project]\Scheduler\.
Do this for each additional Scheduler service you require in that same server.
In this example, we will have 3 services on the server:
1.2 Edit the .bat files inside of the new Scheduler folders.
As a best practice, make sure you edit all .bat files to render them ready for whenever needed.
Edit install.bat, installx64.bat, uninstall.bat and uninstallx64.bat by changing the name of the Scheduler service so that it is unique within that same server:
Note that in this example we are using a same name, with a numbering suffix.
For service number three, we have:
Save changes when done.
1.3 Run the installx64.bat (or install.bat if applies) file to install each new Scheduler service.
Click on the installation .bat file according to your server's architecture to install the service.
Do this for each additional Scheduler service to install.
Make sure that you do this with administrator rights and that you get a successful execution:
In the end, you should see your additional Scheduler services installed and listed at your Windows services:
Do not start the multiple services at this point (you will need to first edit their configuration).
2. Configure the additional Scheduler services
Define which one of all of your Scheduler services will perform maintenance tasks.
Consider all of your Scheduler services within the solutions (only 1 among all nodes you may have in a cluster, including multiple services you may have in each node).
We may call that one the "master" Scheduler, and this one will not require further configuration.
For all of the other Scheduler services except the master, edit their .config file located by default as C:\BIZAGI\PROJECTS\[your_project]\SchedulerN3\BizAgi.Scheduler.Services.exe.config, so that you include the following key inside of the <appSettings> element:
<add key="DisableAsynchCaseClosing" value="1"/>
When configuring more than 1 Scheduler service in a same node, it is fundamental that you also change the location of the temporary folder used by each Scheduler instance (in order to avoid issues that may arise when multiple services try to access the same files simultaneously).
To specify a different temporary folder for your Scheduler instances, edit the Project key by ensuring you use non-repeated, different values in each configuration file.
For example, having for 2 instances in a same node:
<add key="Project" value="Scheduler1" /> and <add key="Project" value="Scheduler2" />.
At this point you may save changes, and then start up each Scheduler service (an error-free start-up validates that all of them will work together):
Considerations when using Asynchronous Activities
As additional configuration, you may tune the Schedulers processing parameters by including the following key in their .config file (inside of the <appSettings> element) in order to configure the number of threads each Scheduler service will use:
<add key="MaxThreadsForAsynchWorkitems" value="[number_of_threads]"/>
This key is especially useful when your project expects a large number of asynchronous tasks executions, incoming in large blocks.
When choosing to modify the number of threads used by a Scheduler service (by default it uses one), make sure you consider an appropriate balance between the number of Scheduler services you will be using and the number of threads each one runs in parallel.
Note that modifying this value should be properly analyzed and tested, according to the your project's estimated number of background tasks, the characteristics and estimated processing involved in each of these tasks, and most importantly, according to the characteristics of the underlying hardware (i.e number of cores and processors in your servers).
It is recommended to watch over this value and tune it as needed while carrying out proper acceptance tests for this scenario to use the most accurate distribution.
When using multiple Scheduler services to process multiple Asynchronous Activities, it is recommended to include the following key:
<add key="DisableInterfaceErrorLogger" value="1"/>
This key optimizes performance since it skips logging at the .csv file for threshold and timeouts, and it avoids potential issues when having multiple services writing into the same file.
Acknowledge that when having multiple Scheduler services, all of these services should be stopped before performing a process deployment.
Similarly, with this setup, consider your multiple Scheduler services when upgrading your Bizagi version so that you stop them all and upgrade them all (i.e upgrade the newer assembly files used by them).