ProActive Workflows & Scheduling — User Guide

ProActive Workflow & Scheduler (PWS) User Guide (Workflows, Tasks, Jobs Submission, Resource Management)

Machine Learning Open Studio (ML-OS) User Guide (ready to use palettes with ML Tasks & Workflows)

ProActive Cloud Automation (PCA) User Guide (automate deployment and management of Services)

Administration Guide (Installation, networks, nodes, clusters, users, permissions)

API documentation:   Scheduler REST   Scheduler CLI   Scheduler Java   Workflow Creation Java   Python Client

The following terms are used throughout the documentation:

In order to use Scheduler for executing various computations, one needs to write the execution definition also known as the Workflow definition. A workflow definition is an XML file that adheres to XML schema for ProActive Workflows.

It specifies a number of XML tags for specifying execution steps, their sequence and dependencies. Each execution step corresponds to a task which is the smallest unit of execution that can be performed on a computation resources. There are several types of tasks which caters different use cases.

ProActive Scheduler currently supports three main types of tasks:

For instance, a script task can be used to execute an inline script definition or a script file as an execution step whereas a native task can be used to execute a native executable file.

One can use ProActive Workflow Studio to create and submit workflows graphically . You can simply drag-and-drop various task constructs and draw their dependencies to form complex jobs. It also provides various flow control widgets such as conditional branch, loop, replicate etc to construct complex workflows.

In this tasks graph, we see that task 4 is preceded by task 1, that means the Scheduler waits the end of task 1 execution before launching task 4. In a more concrete way, task 1 could be the calculation of a part of the problem to solve, and task 4 takes the result provided by task 1 and compute another step of the calculation. We introduce here the concept of result passing between tasks. This relation is called a Dependency, and we say that task 4 depends on task 1.

We see that task 1, 2 and 3 are not linked, so these three tasks can be executed in parallel, because they are independent from each other.

The task graph is defined by the user at the time of workflow creation, but can also be modified during the job execution by control flow actions such as Replicate.

A finished job contains the results and logs of each task. Upon failure, a task can be restarted automatically or cancelled.

The quickstart tutorial on try.activeeon.com shows you how to build a simple workflow using script tasks.

We show below an example of a workflow created with the Studio:

At the left part, are illustrated the General Parameters of the workflow with the following information:

In order for Docker tasks to work, the Node must have Docker and Docker Compose installed, please refer to the official Docker documentation to see how to install Docker and Docker Compose.

A Docker task expects the content of a Docker Compose file inside the Script section. You can find out how to write Docker Compose files with the official Docker Compose documentation.

To get started: a simple Docker Compose example is explained below.

The content of the Script section (equal to the content of a Docker Compose file) is:

The above example describes a container which is called 'helloworld'. That container is created from a busybox image, which will run the command 'echo "Hello ProActive"'

The Docker task allows to set parameters to the docker-compose tool with regard to the docker-compose CLI reference.

It supports general parameters as well as commands options (we currently only support options for the up command). You can specify these options by supplying a space character separated list in the generic informations.

The two latter generic informations will be used to generate the following command:

If splitting by space is prohibitive you can specify the split regex in the generic informations with the key docker-compose-options-split-regex. If you supply e.g. "!SPLIT!" as value, then your docker-compose-up-options will need to look like this: "--option1!SPLIT!--option2".

Using native tasks you can easily reuse existing applications and embed them in a workflow. The Scheduler lets you define a native task that takes the name of the executable and list of parameters. Once the executable is wrapped as a task you can easily leverage some of the workflow constructs to run your executable in parallel.

You can find an example of such integration in this XML workflow or you can also build one yourself using the Workflow Studio.

Native application by nature can be tied to a given operating system so if you have different nodes at your disposal, you might need to select a suitable node to run your native task. This can be achieved using selection script.

The PHP task executes a PHP script on a ProActive Node’s local PHP installation. The PHP script will be downloaded from the dataspace. So, the PHP script must be present in the dataspace beforehand. The task can also be modified to download the PHP script from other sources. Log forwarding and log streaming during PHP execution is conveniently available through the Scheduler portal.

Proactive Scheduler supports tasks written in languages other than Java. The currently supported dynamic languages are Groovy, Jython, Python, Ruby and Javascript.

Native scripts can also be executed using a script task. Currently Bash and CMD scripts are supported, simply set the language attribute to bash or cmd and type your script in the workflow.

You can easily embed small scripts in your workflow. The nice thing is that the workflow will be self-contained and will not require to compile your code before executing. However, we recommend that you keep these scripts small to keep the workflow easy to understand.

Please note that the script execution relies on Java implementation of the cited languages thus it may come with some limitations.

You can find an example of a script task in this XML workflow or you can also build one yourself using the Workflow Studio. The quickstart tutorial relies on script task and is a nice introduction to workflows.

Scripts can also be used to decorate tasks with specific actions, we support pre, post, clean and selection scripts.

MPI is often used in the area of parallel computing. The Scheduler integrates with MPI with the concept of multi node tasks. This particular task will acquire several nodes and will expose these nodes to the MPI environment.

Applications built with MPI are often executed using the mpirun command. It accepts several parameters to choose how many CPUs and how many machines will be used. One particular parameter is the machine file that is built by the Scheduler when using multi node tasks. Here is how an MPI application invocation would look like when executed with the Scheduler:

The variable $variables_PA_NODESFILE contains the path to a machine file created by the Scheduler that will be similar to:

Meaning that myApplication will use 2 CPUs on compute-host and 1 CPU on another-host. You can also use $variables_PA_NODESNUMBER to retrieve the number of acquired nodes.

You can find an example of a native task in this XML workflow or you can also build one yourself using the Workflow Studio.

To achieve the best performance for MPI applications, nodes can be selected taking into account their network topology. One might want to select nodes that are as close to each other as possible to obtain better network performance or nodes on different hosts to split the I/O load. Refer to Topology Types for more details.

To run a workflow, the user submits it to ProActive Scheduler. It will be possible to choose the values of all Job Variables when submitting the workflow (see section Job Variables). A verification is performed to ensure the well-formedness of the workflow. A verification will also be performed to ensure that all variables are valid according to their model definition (see section Variable Model). Next, a job is created and inserted into the pending queue and waits until free resources become available. Once the required resources are provided by the ProActive Resource Manager, the job is started. Finally, once the job is finished, it goes to the queue of finished jobs where its result can be retrieved.

You can submit a workflow to the Scheduler using the Workflow Studio, the Scheduler Web Interface or command line tools. For advanced users we also expose REST and JAVA APIs.

It is possible to retrieve multiple logs from a job, these logs can either be:

Unless your account belongs to an administrator group, you can only see the logs of a job that you own.

Once a job or a task is terminated, it is possible to get its result. Unless you belong to the administrator group, you can only get the result of the job that you own. Results can be retrieved using the Scheduler Web Interface or the command line tools.

Dependencies can be set between tasks in a TaskFlow job. It provides a way to execute your tasks in a specified order, but also to forward result of an ancestor task to its children as parameter. Dependency between tasks is then both a temporal dependency and a data dependency.

Dependencies between tasks can be added either in ProActive Workflow Studio or simply in workflow XML as shown below:

The replication allows the execution of multiple tasks in parallel when only one task is defined and the number of tasks to run could change.

The branch construct provides the ability to choose between two alternative task flows, with the possibility to merge back to a common flow.

The loop provides the ability to repeat a set of tasks.

Workflows often relies on task blocks. Task blocks are defined by pairs of start and end tags.

Task blocks are very similar to the parenthesis of most programming languages: anonymous and nested start/end tags. The only difference is that a parenthesis is a syntactical information, whereas task blocks are semantic.

The role of task blocks is to restrain the expressiveness of the system so that a workflow can be statically checked and validated. A treatment that can be looped or iterated will be isolated in a well-defined task block.

Variables can come at hand while developing ProActive applications. Propagate data between tasks, customize jobs, and store useful debug information are a few examples of how variables can ease development. ProActive has basically two groups of variables :

Another way to propagate data from a task to another relies on result variable. Anywhere in a task (usually at the end) you can set a value to a reserved variable named result. This value will be available in tasks depending on it. If a task has several dependencies, results will be an array.

Assuming that we have two tasks task1 and task2 written in Groovy:

and task3 that depends on tasks task1 and task2, then, you can access result values defined by the parents as follows:

For nearly all script languages, the results variable contains a list of TaskResult java object. In order to access the result value, the value() method of this object must be called. Example for Python/Jython:

To perform a control flow action such as if, replicate or loop, a Control Flow Script is executed on the ProActive node. This script takes the result of the task as input; meaning a Java object if it was a Java or Script task, or nothing if it was a native task.

The script is executed on the ProActive node, just after the task’s executable. If the executable is a Java Executable and returns a result, the variable result will be set in the script’s environment so that dynamic decisions can be taken with the task’s result as input. Native Java objects can be used in a Groovy script.

Similarly to how parameters are passed through the result variable to the script, the script needs to define variables specific to each action to determine what action the script will lead to.

The assigned value needs be a strictly positive integer.

The assigned value needs to be the string value if or else.

The assigned value needs to be a boolean.

Failure to set the required variables or to provide a valid control flow script will not be treated gracefully and will result in the failure of the task.

When Control Flow actions such as replicate or loop are performed, some tasks are replicated. To be able to identify replicated tasks uniquely, each replicated task has an iteration index, replication index, and a tag. In addition to help to identify uniquely, these tags are useful to filter tasks by iterations for example.

An Embarrassingly Parallel problem is a problem that is easy to split into smaller independent tasks. With ProActive Scheduler you can tackle this type of problem with the Replicate construct.

The Advanced workflows is an example of an embarrassingly parallel problem where the computation is easily distributed across ProActive Nodes.

A selection script provides an ability for the Scheduler to execute tasks on particular ProActive nodes. E.g. you can specify that a task must be executed on a Unix/Linux system.

A selection script is always executed before the task itself on any candidate node: the execution of a selection script must set the boolean variable selected, that indicates if the candidate node is suitable for the execution of the associated task.

A Java helper org.ow2.proactive.scripting.helper.selection.SelectionUtils is provided for allowing user to simply make some kind of selections. Some script samples are available in PROACTIVE_HOME/samples/scripts/selection.

The following example selects only nodes running on Windows:

You can use variables inside the selection script, these variables must be visible within the task context, i.e., declared on job scope, on task scope, or result variable from a previous task. In the last case, it is mandatory that the task which declares the inherited variable precedes the task using the variable. Below is an example job in XML format which uses a task variable to select Unix/Linux as operating system. In this example, variable operating_system (task scope) resolves to variable operating_system_workflow (job scope).

Another functionality is the possibility to define pre and post scripts. For a given task (Java, Script or Native task), it is possible to launch a script before and after its execution. This possibility can be useful to copy files to a node, or clean a directory before or after task execution, for example.

This is a way to separate from business code the preparation of execution environment and its cleaning. A good example is a script that removes files from a specific directory once the task is finished.

It is possible to start a task under the job owner if the system is configured for that purpose. There are 2 possible ways to run a task under user account (in any case, the administrator should have set computing hosts to authorize one of the 2 methods) :

User must first create a credential containing this key :

This command will create a new credentials with username as login, userpwd as password, using Scheduler public key at config/authentication/keys/pub.key for credentials encryption and using the private SSH key at path/to/private/sshkey provided by administrator. The new credential will be stored in myCredentials.cred

Once created, user must connect the Scheduler using this credential. Then, in order to execute your task under your account set runAsMe=true in the task.

To create a multi-nodes task often used for MPI applications you need to add a parallel environment to the task. Parallel environment describes how many nodes are needed for a task as well as where these nodes should be located. For example if you’d like to run 4 processes in parallel it a scope of one task you should specify it in your task

It’s possible to have errors when your Job is executed. ProActive Scheduler provides several mechanisms to deal with exceptional situations in your Jobs. If a Task contains errors during Job execution, the Proactive Scheduler automatically restarts this Task. The number of automatic restarts is defined by the Workflow owner by assigning an integer value to the counter Number of Automatic Restarts. The user also defines Where he wants that the scheduler restarts In-Error tasks, i.e., on the same Proactive node or on another Proactive node.

If automatic restarts does not fix errors, the Scheduler applies one of the following Task error management policies, i.e., the one defined by the user during Workflow creation:

The ProActive Scheduler allows to schedule Workflows with the Job Planner. Find more information in the Job Planner Service section.

ProActive Scheduler supports the execution of time-based tasks or cron tasks. The users can assign a cron expression to the loop variable of a Loop in a ProActive Job. All tasks which belong that Loop will be executed iteratively and indefinitely. The start time of the next iteration is the next point of time specified by the cron expression.

Using the Scheduler Web Interface, it is possible to access the display of the ProActive Node running a given task. First remote visualization must be activated via the Scheduler configuration files (set novnc.enabled to true).

The task that you want to visualize must output a special string using the following format: PA_REMOTE_CONNECTION;JOB_ID;TASK_ID;vnc;HOSTNAME:PORT where JOB_ID must be the current job id, TASK_ID the current task id and HOSTNAME:PORT the hostname and port where the VNC server runs.

It is the task’s responsibility to print the string. Here is a sample script that starts a Xvnc server and runs a graphical application. It can be used in a native task for instance.

In the Scheduler Web Interface, select the running job and use the Preview tab to enable remote visualization.

When a task has IN_ERROR status, a right click on it allows the user to select Mark as finished and resume. This will set the task status as FINISHED and then resume job execution: this task won’t be executed again, subsequent tasks will execute as if the task executed correctly. This allows the user to manually perform actions to the IN_ERROR task, without cancelling the whole job execution.

If the task has not IN_ERROR status, the Mark as finished and resume option will be in grey, meaning that the action is disabled.

Add a "NOTIFICATION_EVENTS" Generic information, to list events on which a user wants to be notified. List is comma-separated and should be taken from(events are not case sensitive):

Scheduler can send email to the users when one of the above events is received.

Notifications will only be sent for jobs whose generic information contains user’s email address under the "EMAIL" key. Example:

To enable this functionality, the configuration need to be done following the steps introduced in Get Notification on Job Events Configuration.

SubmitJobNoWait and SubmitJobAndWait templates under Controls menu allow you to submit a workflow from another workflow using schedulerapi.

In order to submit a workflow, you need to put the URL of that workflow in the Task Variables under the key workflowURL. You can use any URL that points to a valid workflow xml file. For example, you can add the workflow that you want to execute into Catalog, then retrieve its URL by indicating its Bucket name and Workflow name. Moreover, SubmitJobAndWait will wait for the completion of the submitted workflow before proceeding further with the execution of the dependant tasks.

In order to better control your workflows, two templates Web Validation and Email Validation are provided under the Manuals menu. These templates aim at validating part of your workflows manually. The tasks after the validation template will only be executed once you validate it. You can put your description of the validation under the content variable in Web Validation template, and under message variable in Email Validation template. Three options are provided, the Yes option indicates that you want to validate the workflow and continue the execution of the rest tasks; the Maybe option means that you do not know what to do for the moment; the No option denotes that you do not want to validate the workflow, and the remaining of the workflow will be paused.

In order to validate your workflow using Web Validation, you need to go to the ProActive Notification & Validation portal, the workflow will be validated by clicking on one of the three buttons. By using Email Validation, you will receive an email with three links, the workflow can be validated by clicking on one of the links.

If a shared folder is not an option in your environment, the ProActive Scheduler provides a convenient way to access your data. It has two types of storage on the host where the server is running:

By default, these spaces are linked to folders in the data directory of the ProActive Scheduler host:

But it can be changed in PROACTIVE_HOME/config/scheduler/settings.ini.

All files that are stored in these spaces can be read from all computing nodes. However, data transfer is declarative and not explicit. You need to define which files have to be transferred from a data space to computing nodes by using input files and reciprocally from computing nodes to a data space by defining output files:

Upon the execution of the previous example, Proactive Scheduler automatically transfers all files to a ProActive Node where the task will be executed. The files are put in a special place called Local Space on the ProActive Node. It corresponds to the working directory of a task when the task is in fork mode (default). From a task point of view, files can be accessed normally like shown below:

Then, based on the previous task definition, all the files whose the name starts with titi (e.g titi, titi2, etc.) produced by task1 will be transferred back automatically to Global Space by the Scheduler once the task is finished.

Files can be also transferred from/to User Space by specifying transferFromUserSpace/transferToUserSpace.

Before running your jobs you should first upload your files into either Global or User Space. To do it you can export folders linked to these spaces by one of the standard protocols like FTP, NFS, or use available web interfaces like Pydio.

It is possible to use wildcards in transfer directives, such as:

More information of pattern syntax can be found in the FileSystem class JDK documentation.

In addition to transferring to the Task Local Space, it is also possible to transfer files to the Node host Cache Space.

From the Node host point of view, the Cache Space is unique and shared for all tasks and all ProActive Nodes deployed on the host.

Transfers to the cache are not concurrent, only a single task can transfer to the cache at the same time. I.e if multiple tasks are run in parallel on the same hosts and they all transfer to the cache, these transfers will be executed sequentially.

If a file declared for the transfer is newer than the existing version in the cache, the file will be updated.

In order to transfer files to the cache, simply use the cache transfer directives inside your job:

The files will not be transferred to the Local Space of the task, so it will not be possible to access these files from the current directory.

In order to access these files, use the cachespace variable which contains the location of the cache, for example in groovy:

There are no transfer directives to transfer output files from the cache. Output files transfers are only possible from the Local Space.

Files in the cache space are automatically deleted after a given period. The default invalidation period is two weeks, which means that files older than two weeks will be automatically deleted.

In order to change this period, the ProActive Node must be started with the property node.dataspace.cache.invalidation.period (in miliseconds).

The property node.dataspace.cachedir can also be used to control the cache location on the node.

A Bucket is a collection of ProActive Objects and in particular ProActive Workflows that can be shared between users. The default settings provide the different Buckets, one of them is:

Listing the Catalog for existing Buckets is done using the following HTTP request:

It is also possible to list all Workflows inside a particular Bucket using its bucketName:

Then, we can fetch a particular Workflow based on its name:

Every bucket is identified by a unique bucket name. The bucket naming should follow the following rules:

The Catalog provides versioning of Workflows. You can list all the revisions of a particular Workflow using the HTTP request:

The revision of Workflow is identified by commit_time. Commit time specifies the time when the Workflow’s version was added in the Catalog. You can fetch a Workflow detailed metadata information using its name and commit_time:

A Workflow Revision is created from a ProActive XML Workflow definition. Beside from the id, name and project_name of the Workflow, a few more information are indexed from the XML content such as Generic Information and Workflow variables.

More detailed information on how to create or modify Workflows and Buckets are available in the Admin guide.

The Catalog is secured with a sessionId. Every request to the Catalog must have a valid sessionId inside the request header. The security mechanism is based on groups, the group GROUP:public-objects can be accessed by any user with a valid sessionId. Other groups can be accessed (read and write) if the sessionId corresponds to a user which is part of that group. As an example, a user which is part of the interns group (GROUP:interns) can see and change buckets and their contents which belong to GROUP:interns. The user can only access to the buckets list belonging to him or to his group and plus to public buckets (the GROUP:public-objects associated buckets).

The sessionId is validated against the Scheduler REST api, so if the Catalog does not have access (network outage, firewall) to the Scheduler, no access to the Catalog is possible.

Inside Studio portal there is present a Catalog menu. With help of this menu the user can directly interact from studio view with Catalog storage.

The Workflow Catalog Portal provides interfaces for easy administering the Catalog. The most important of provided features will be described next.

Tasks executed by ProActive Scheduler might need to use a password or another kind of credentials to acces third-party services or applications. For example a user might wish to pass his MySQL password to Native Task which runs a mysql command to dump a database.

In such cases it is often desireable to store the credentials separately from the Workflow definition. ProActive Scheduler provides such facility.

Third-party credentials in ProActive Scheduler are user-defined key-value pairs of strings stored on the server side in encrypted form. The credentals are accessible in the tasks executed by the user via API (Script and Java Tasks) or variable substitution (arguments of Native Tasks and parameters of Java Tasks).

Methods to add or remove the credentials and to list the stored credential keys are exposed in both Java and REST APIs.

End users can manage their credentials using the Scheduler Web Interface (Portal → Manage third-party credentials) or the Command Line Interface.

From inside a Script Task, you can use the Scheduler API binding like the following:

The complete API description can be found in the SchedulerNodeClient JavaDoc.

The traditional way to transfer files from the user or global space to a task is by using file transfer directives as described in chapter Data Spaces.

Where file transfer directives are sufficient to cover usual cases, it is sometimes necessary to manipulate directly the dataspace API to have a finer control level.

Below is an example use of this api inside a Script Task:

The complete description of user and global space APIs can be found in the DataSpaceNodeClient JavaDoc.

This Synchronization API allows for both communication and synchronization between Tasks and between Jobs dynamically at execution. It is based on a Key-Value Store offering atomic operations. It enables to easily put into action all classical synchronization patterns: Critical Sections and Mutual Exclusion, Rendez-vous, Barriers, Producer/Consumer…​

Task synchronization is traditionally handled by the ProActive Scheduler through Task Dependencies, where dependent ProActive Tasks are started only after their parents Tasks completed.

In some cases, static task dependencies are not enough to synchronize tasks execution.

This is the case when a task depends on the state of another task executed in a different branch of a workflow, a sibling task or even another job/workflow.

For example, let’s suppose workflow A contains a single task A_1 starting an apache server. Task A_1 starts the server and remains alive while the server is up. When workflow A is killed, task A_1 stops the Apache server.

A task B_1 of another workflow B wants to interact with this server.

Problem: task B_1 does not know if the apache server is up and how to contact it.

Using Task Synchonization API, task A_1 can advertise the server URL to the Synchronization Service.

Task B_1 can use Task Synchonization API to wait until the apache server is advertised, it can also perform the necessary operations and advertise task A_1 that the operation is complete and the apache server can be terminated.

As we can see, Task Synchronization API can be used for two-ways synchronization.

Synchronizations can be used in any part of a Task (Pre/Post/Clean Scripts, Task Implementation, Selection Script). Using Synchronization API within Selection Scripts is a way to avoid a Task to be started (consuming a Node) and to be actually doing nothing, just waiting for a specific condition to occur.

Scheduler has Earliest deadline first policy (EDF policy) that prioritizes jobs with deadlines. Overall, the general principal is the following: a job that needs to be started as soon as possible to meet its deadline taking into account its execution time has higher priority.

The EDF policy considers two Generic Information items: JOB_DDL and JOB_EXEC_TIME.

JOB_DDL represents job deadline. Job deadline can be set in one of the following format:

If job deadline is absolute then its value equals to JOB_DDL.

If job deadline is relative then its value equals to submission time + JOB_DDL.

JOB_EXEC_TIME represents job expected execution time in the following format: HH:mm:ss , e.g. 12:30:00, 5:30:00, or mm:ss, e.g. 4:30, or ss, e.g. 45 (45 seconds).

EDF policy orders jobs by applying the rules below. The rule considers jobs at submission time when they are still pending, and orders them by priority.

Thus EDF policy applies these rules (in the given order) until one rule actually applies. Here is the ordered list of rules for the EDF policy:

The File connectors allow to import and export data from HTTP, FTP and SFTP servers. We begin by presenting FTP and SFTP connectors then, URL connector.

The SQL connectors allow to import and export data from and to Relational DataBase Management Systems (RDBMS). Currently, we have connectors for MySQL, Oracle, PostgreSQL, Greenplum and SQL Server.

SQL connectors workflows are composed of two tasks: an import task and an export task.

Variables:

All SQL connectors share the same list of variables. Consequently, we describe them in the following table using a unique notation. <RDBMS_NAME> can take one of the following values: {POSTGRES, GPDB, ORACLE, SQL_SERVER, MYSQL}

How to use this task:

When you drag & drop an SQL connector, two tasks will be appended to your workflow: an import task and an export task. You can keep one of them depending on your needs and remove the other or you can use them both.

This task uses the driver given in RMDB_DRIVER to connect to the database. To use another driver, make sure you have it properly installed before (e.g. using pip install <RMDBS_DRIVER>).

The task requires the following third-party credential: {key: <mysql|postgres|mssql|oracle|gpdb>://<<RDBMS_NAME>_USER>@<<RDBMS_NAME>_HOST>:<<RDBMS_NAME>_PORT>, value: <RDBMS_NAME>_PASSWORD}. ; this is a one-time action and will ensure that your credentials are securely encrypted. Please refer to the User documentation to learn how to add third-party credentials.

In the import mode, the output containing the imported data takes one or many of the following forms:

The NoSQL connectors allow to import data from NoSQL Databases. Currently, we have connectors for MongoDB and Cassandra.

Cloud data connectors allow to interact with cloud storage services. Currently we provide support for Amazon S3, Azure Storage and Azure Data Lake.

Proactive’s ERP connectors enable communication and data exchange with popular ERP software providers.

A Docker fork execution environment executes a JVM inside a Docker container. Hence the task which is executes in the Docker fork execution environment can be customized by the JVM settings and by the tool-set and isolation provided by Docker containers.

The Job Planner schedules Workflows. A Workflow is scheduled by the Job Planner if it is associated to a Calendar Definition.

Job Planner uses a Calendar Definition to know how the job will be planned over the time. As shown on the example below, this definition is composed of 4 fields:

Based on the above configuration, the following JSON object will be stored in the Catalog.

This addon generates a gantt chart to visualize executions of jobs and tasks, per node and over the time.

An example of a generated gantt chart

This chart can be generated on-demand, running Gantt_chart workflow from the Analytics bucket in the Catalog.

Workflows are written in XML and we provide a XSD grammar to help you write and validate them. The Workflows XSD is available here and you can also find the documentation here.

This section gathers the available scripts engines and the operating system where they run.

This section describes the available predefined tasks and the operating system where they run.

This section describes useful variables, their scope, and their usage.

The following script names refer to:

The order in this list corresponds to the task life-cycle order. all scripts refers to all above scripts

ProActive Scheduler supports the following nodes topologies for multi-nodes tasks.

For this task the scheduler will try to find a host with 4 nodes. If there is no such a host another one will be used with a bigger capacity (if exists). In this case extra nodes on this host will be also occupied by the task but will not be used for the execution.

For this task the scheduler will try to find 4 nodes on a set of hosts. If there is no such a set which gives you exactly 4 nodes another one will be used with a bigger capacity (if exists). In this case extra nodes on this host will be also occupied by the task but will not be used for the execution.

For this task the scheduler will try to find 4 hosts with one node on each. If number of "single node" hosts are not enough hosts with bigger capacity will be selected. In this case extra nodes on this host will be also occupied by the task but will not be used for the execution.

The ProActive client allows to interact with the Scheduler and Resource Manager. The client has an interactive mode started if you do not provide any command.

The client usage is also available using the -h parameter as shown below: