ProActive Admin Guide

All documentation links

ProActive Workflows & Scheduling (PWS)

PWS User Guide
(Workflows, Workload automation, Jobs, Tasks, Catalog, Resource Management, Big Data/ETL, …)
PWS Modules
- Job Planner
  (Time-based Scheduling)
- Event Orchestration
  (Event-based Scheduling)
- Service Automation
  (PaaS On-Demand, Service deployment and management)
PWS Admin Guide
(Installation, Infrastructure & Nodes setup, Agents,…)

ProActive AI Orchestration (PAIO)

PAIO User Guide
(a complete Data Science and Machine Learning platform, with Studio & MLOps)

API documentation

Scheduler REST Scheduler GraphQL Scheduler CLI Scheduler Java Workflow Creation Java Python Client Jupyter Kernel .NET Client

ProActive Scheduler is a comprehensive Open Source job scheduler and Orchestrator, also featuring Workflows and Resource Management. The user specifies the computation in terms of a series of computation steps along with their execution and data dependencies. The Scheduler executes this computation on a cluster of computation resources, each step on the best-fit resource and in parallel wherever its possible.

On the top left there is the Studio interface which allows you to build Workflows. It can be interactively configured to address specific domains, for instance Finance, Big Data, IoT, Artificial Intelligence (AI). See for instance the Documentation of ProActive AI Orchestration here, and try it online here. In the middle there is the Scheduler which enables an enterprise to orchestrate and automate Multi-users, Multi-application Jobs. Finally, at the bottom right is the Resource manager interface which manages and automates resource provisioning on any Public Cloud, on any virtualization software, on any container system, and on any Physical Machine of any OS. All the components you see come with fully Open and modern REST APIs.

The screenshot above shows the Workflow Execution Portal, which is the main portal of ProActive Workflows & Scheduling and the entry point for end-users to submit workflows manually, monitor their executions and access job outputs, results, services endpoints, etc.

The screenshot above shows the Catalog Portal, where one can store Workflows, Calendars, Scripts, etc. Powerful versioning together with full access control (RBAC) is supported, and users can share easily Workflows and templates between teams, and various environments (Dev, Test, Staging and Prod).

The screenshot above shows the Job Planner Portal, allowing to automate and schedule recurring Jobs. From left to right, you can define and use Calendar Definitions , associate Workflows to calendars, visualize the execution planning for the future, as well as actual executions of the past.

The Gantt screenshot above shows the Gantt View of Job Planner, featuring past job history, current Job being executed, and future Jobs that will be submitted, all in a comprehensive interactive view. You easily see the potential differences between Planned Submission Time and Actual Start Time of the Jobs, get estimations of the Finished Time, visualize the Job that stayed PENDING for some time (in Yellow) and the Jobs that had issues and got KILLED, CANCELLED, or FAILED (in red).

The screenshots above show the Health Dashboard Portal, which is a part of the Analytics service, that displays global statistics and current state of the ProActive server. Using this portal, users can monitor the status of critical resources, detect issues, and take proactive measures to ensure the efficient operation of workflows and services.

The screenshot above shows the Job Analytics Portal, which is a ProActive component, part of the Analytics service, that provides an overview of executed workflows along with their input variables and results.

The screenshot above shows the Event Orchestration Portal, where a user can manage and benefit from Event Orchestration - smart monitoring system. This ProActive component detects complex events and then triggers user-specified actions according to predefined set of rules.

The screenshot above shows the Service Automation Portal which is actually a PaaS automation tool. It allows you to easily manage any On-Demand Services with full Life-Cycle Management (create, deploy, suspend, resume and terminate).

The screenshots above taken from the Resource Manager Portal shows that you can configure ProActive Scheduler to dynamically scale up and down the infrastructure being used (e.g. the number of VMs you buy on Clouds) according to the actual workload to execute.

The screenshot above shows the Notifications Portal, which is a ProActive component that allows a user or a group of users to subscribe to different types of notifications (web, email, sms or a third-party notification) when certain event(s) occur (e.g., job In-Error state, job in Finished state, scheduler in Paused state, etc.).

The administration guide covers cluster setup and cluster administration. Cluster setup includes two main steps:

The installation and configuration of the ProActive Scheduler.
The set up of ProActive Nodes.

1.1. Glossary

The following terms are used throughout the documentation:

ProActive Workflows & Scheduling: The full distribution of ProActive for Workflows & Scheduling, it contains the ProActive Scheduler server, the REST & Web interfaces, the command line tools. It is the commercial product name.

ProActive Scheduler

Can refer to any of the following:

A complete set of ProActive components.
An archive that contains a released version of ProActive components, for example activeeon_enterprise-pca_server-OS-ARCH-VERSION.zip.
A set of server-side ProActive components installed and running on a Server Host.

Resource Manager: ProActive component that manages ProActive Nodes running on Compute Hosts.

Scheduler: ProActive component that accepts Jobs from users, orders the constituent Tasks according to priority and resource availability, and eventually executes them on the resources (ProActive Nodes) provided by the Resource Manager.

Please note the difference between Scheduler and ProActive Scheduler.

REST API: ProActive component that provides RESTful API for the Resource Manager, the Scheduler and the Catalog.

Resource Manager Portal: ProActive component that provides a web interface to the Resource Manager. Also called Resource Manager portal.

Scheduler Portal: ProActive component that provides a web interface to the Scheduler. Also called Scheduler portal.

Workflow Studio: ProActive component that provides a web interface for designing Workflows.

Automation Dashboard: Centralized interface for the following web portals: Workflow Execution, Catalog, Analytics, Job Planner, Service Automation, Event Orchestration and Notifications.

Analytics: Web interface and ProActive component, responsible to gather monitoring information for ProActive Scheduler, Jobs and ProActive Nodes

Job Analytics: ProActive component, part of the Analytics service, that provides an overview of executed workflows along with their input variables and results.

Job Gantt: ProActive component, part of the Analytics service, that builds a Gantt diagram of past Jobs executions.

Node Gantt: ProActive component, part of the Analytics service, that provides an overview of ProActive Nodes usage over time.

Health Dashboard: Web interface, part of the Analytics service, that displays global statistics and current state of the ProActive server.

Job Analytics portal: Web interface of the Analytics component.

Notification Service: ProActive component that allows a user or a group of users to subscribe to different types of notifications (web, email, sms or a third-party notification) when certain event(s) occur (e.g., job In-Error state, job in Finished state, scheduler in Paused state, etc.).

Notification portal: Web interface to visualize notifications generated by the Notification Service and manage subscriptions.

Notification Subscription: In Notification Service, a subscription is a per-user configuration to receive notifications for a particular type of events.

Third Party Notification: A Notification Method which executes a script when the notification is triggered.

Notification Method: Element of a Notification Subscription which defines how a user is notified (portal, email, third-party, etc.).

Notification Channel: In the Notification Service, a channel is a notification container that can be notified and displays notifications to groups of users

Service Automation: ProActive component that allows a user to easily manage any On-Demand Services (PaaS, IaaS and MaaS) with full Life-Cycle Management (create, deploy, pause, resume and terminate).

Service Automation portal: Web interface of the Service Automation.

Workflow Execution portal: Web interface, is the main portal of ProActive Workflows & Scheduling and the entry point for end-users to submit workflows manually, monitor their executions and access job outputs, results, services endpoints, etc.

Catalog: ProActive component that provides storage and versioning of Workflows and other ProActive Objects. It is also possible to query the Catalog for specific Workflows through a REST API and also with Graphql API.

Catalog portal: Web interface of the Catalog component.

Event Orchestration: ProActive component that monitors a system, according to predefined set of rules, detects complex events and then triggers user-specified actions.

Event Orchestration portal: Web interface of the Event Orchestration component.

Scheduling API: ProActive component that offers a GraphQL endpoint for getting information about a ProActive Scheduler instance.

Connector IaaS: ProActive component that enables to do CRUD operations on different infrastructures on public or private Cloud (AWS EC2, Openstack, VMWare, Kubernetes, etc).

Job Planner: ProActive component providing advanced scheduling options for Workflows.

Job Planner portal: Web interface to manage the Job Planner service.

Calendar Definition portal: Web interface, component of the Job Planner that allows to create Calendar Definitions.

Calendar Association portal: Web interface, component of the Job Planner that defines Workflows associated to Calendar Definitions.

Execution Planning portal: Web interface, component of the Job Planner that allows to see future executions per calendar.

Job Planner Gantt portal: Web interface, component of the Job Planner that allows to see past and future executions on a Gantt diagram.

Bucket: ProActive notion used with the Catalog to refer to a specific collection of ProActive Objects and in particular ProActive Workflows.

Server Host: The machine on which ProActive Scheduler is installed.
SCHEDULER_ADDRESS: The IP address of the Server Host.

ProActive Node

One ProActive Node can execute one Task at a time. This concept is often tied to the number of cores available on a Compute Host. We assume a task consumes one core (more is possible, see multi-nodes tasks), so on a 4 cores machines you might want to run 4 ProActive Nodes. One (by default) or more ProActive Nodes can be executed in a Java process on the Compute Hosts and will communicate with the ProActive Scheduler to execute tasks. We distinguish two types of ProActive Nodes:

Server ProActive Nodes: Nodes that are running in the same host as ProActive server;
Remote ProActive Nodes: Nodes that are running on machines other than ProActive Server.

Compute Host: Any machine which is meant to provide computational resources to be managed by the ProActive Scheduler. One or more ProActive Nodes need to be running on the machine for it to be managed by the ProActive Scheduler.

Examples of Compute Hosts:

All machines of a cluster managed by ProActive Scheduler, except Server Host.
All desktop machines in an organisation that have ProActive Agents installed.

Node Source: A set of ProActive Nodes deployed using the same deployment mechanism and sharing the same access policy.

Node Source Infrastructure: The configuration attached to a Node Source which defines the deployment mechanism used to deploy ProActive Nodes.

Node Source Policy: The configuration attached to a Node Source which defines the ProActive Nodes acquisition and access policies.

Scheduling Policy: The policy used by the ProActive Scheduler to determine how Jobs and Tasks are scheduled.
PROACTIVE_HOME: The path to the extracted archive of ProActive Scheduler release, either on the Server Host or on a Compute Host.

Workflow: User-defined representation of a distributed computation. Consists of the definitions of one or more Tasks and their dependencies.

Workflow Revision: ProActive concept that reflects the changes made on a Workflow during it development. Generally speaking, the term Workflow is used to refer to the latest version of a Workflow Revision.

Generic Information: Are additional information which are attached to Workflows or Tasks. See generic information.

Calendar Definition: Is a json object attached by adding it to the Generic Information of a Workflow.

Job: An instance of a Workflow submitted to the ProActive Scheduler. Sometimes also used as a synonym for Workflow.

Job Id: An integer identifier which uniquely represents a Job inside the ProActive Scheduler.

Job Icon: An icon representing the Job and displayed in portals. The Job Icon is defined by the Generic Information workflow.icon.

Task: A unit of computation handled by ProActive Scheduler. Both Workflows and Jobs are made of Tasks. A Task must define a ProActive Task Executable and can also define additional task scripts

Task Id: An integer identifier which uniquely represents a Task inside a Job ProActive Scheduler. Task ids are only unique inside a given Job.

Task Executable: The main executable definition of a ProActive Task. A Task Executable can either be a Script Task, a Java Task or a Native Task.

Script Task: A Task Executable defined as a script execution.

Java Task: A Task Executable defined as a Java class execution.

Native Task: A Task Executable defined as a native command execution.

Additional Task Scripts: A collection of scripts part of a ProActive Task definition which can be used in complement to the main Task Executable. Additional Task scripts can either be Selection Script, Fork Environment Script, Pre Script, Post Script, Control Flow Script or Cleaning Script

Selection Script: A script part of a ProActive Task definition and used to select a specific ProActive Node to execute a ProActive Task.

Fork Environment Script: A script part of a ProActive Task definition and run on the ProActive Node selected to execute the Task. Fork Environment script is used to configure the forked Java Virtual Machine process which executes the task.

Pre Script: A script part of a ProActive Task definition and run inside the forked Java Virtual Machine, before the Task Executable.

Post Script: A script part of a ProActive Task definition and run inside the forked Java Virtual Machine, after the Task Executable.

Control Flow Script: A script part of a ProActive Task definition and run inside the forked Java Virtual Machine, after the Task Executable, to determine control flow actions.

Control Flow Action: A dynamic workflow action performed after the execution of a ProActive Task. Possible control flow actions are Branch, Loop or Replicate.

Branch: A dynamic workflow action performed after the execution of a ProActive Task similar to an IF/THEN/ELSE structure.

Loop: A dynamic workflow action performed after the execution of a ProActive Task similar to a FOR structure.

Replicate: A dynamic workflow action performed after the execution of a ProActive Task similar to a PARALLEL FOR structure.

Cleaning Script: A script part of a ProActive Task definition and run after the Task Executable and before releasing the ProActive Node to the Resource Manager.

Script Bindings: Named objects which can be used inside a Script Task or inside Additional Task Scripts and which are automatically defined by the ProActive Scheduler. The type of each script binding depends on the script language used.

Task Icon: An icon representing the Task and displayed in the Studio portal. The Task Icon is defined by the Task Generic Information task.icon.

ProActive Agent: A daemon installed on a Compute Host that starts and stops ProActive Nodes according to a schedule, restarts ProActive Nodes in case of failure and enforces resource limits for the Tasks.

2. Get Started

ProActive Scheduler can be installed and started either manually or automatically.

For manual installation, you can install and start ProActive:

as a Java process, see section Start ProActive Manually as a Java process.
as a system service, see section Install and start ProActive as a system service.

For automatic installation, you can install and start ProActive:

using Docker Compose (applicable to any machine with Docker and Docker Compose installed), see section Automated ProActive Installation with Docker Compose.
using Kubernetes (applicable to any On-premises or AKS Kubernetes clusters) , see section Automated ProActive Installation with Kubernetes.
using Shell Scripts (applicable to both physical and cloud virtual machines), see section Automated ProActive Installation with Shell Scripts.

We recommend using one of the automated installation methods, as they are faster and simpler.

2.1. Minimum requirements

The minimal hardware requirements for the ProActive server is 4 Cores and 8 GB RAM.

ProActive server is compatible with any 64 bits Linux, Windows or Mac operating system. The server process uses at minimum 4 GB of RAM.

ProActive Nodes are compatible with any Linux, Windows or Mac systems (both 64 bits and 32 bits versions). ProActive Nodes can also run on some Unix systems (such as Solaris, HP-UX), by installing and using a Java Runtime Environment compatible with these systems.

Sizing the ProActive server correctly depends on the number of jobs and tasks executed and submitted per day, the number of nodes connected to the server and the number of concurrent users.

A medium-sized ProActive Scheduler installation uses 12 cores, 24 GB RAM. Large setups use 24 cores and 100 GB RAM.

Additionally, using local ProActive Nodes (deployed directly inside the ProActive server) or having other services running on the same machine as the server can impact the sizing.

2.2. Start ProActive Manually as a Java process

First, Download ProActive Scheduler and unzip the archive. The extracted folder will be referenced as PROACTIVE_HOME in the rest of the documentation. The archive contains all required dependencies with no extra prerequisites.

The first way to start a ProActive Scheduler is by running the proactive-server script. To do so, you can use the following command with no extra configuration.

$PROACTIVE_HOME/bin/proactive-server

It is possible to start the server with a specific number of nodes by adding the -ln parameter. For example, PROACTIVE_HOME/bin/proactive-server -ln 1 will start the proactive server with a single node. In case this parameter is not specified, the number of nodes corresponds to the number of cores of the machine where ProActive server is installed.

The router created on localhost:33647
Starting the scheduler...
Starting the resource manager...
The resource manager with 4 local nodes created on pnp://localhost:41303/
The scheduler created on pnp://localhost:41303/
The web application /automation-dashboard created on http://localhost:8080/automation-dashboard
The web application /studio created on http://localhost:8080/studio
The web application /job-analytics created on http://localhost:8080/job-analytics
The web application /notification-service created on http://localhost:8080/notification-service
The web application /cloud-automation-service created on http://localhost:8080/cloud-automation-service
The web application /catalog created on http://localhost:8080/catalog
The web application /proactive-cloud-watch created on http://localhost:8080/proactive-cloud-watch
The web application /scheduler created on http://localhost:8080/scheduler
The web application /scheduling-api created on http://localhost:8080/scheduling-api
The web application /connector-iaas created on http://localhost:8080/connector-iaas
The web application /rm created on http://localhost:8080/rm
The web application /rest created on http://localhost:8080/rest
The web application /job-planner created on http://localhost:8080/job-planner
*** Get started at http://localhost:8080 ***

The following ProActive Scheduler components and services are started:

Resource Manager
4 local ProActive Nodes
Scheduler
Automation Dashboard
Workflow Studio
Job Analytics
Notification Service
Service Automation
Catalog
Event Orchestration
Scheduler Web Interface
Scheduling Api
Connector IaaS
Resource Manager Web Interface
REST API
Job Planner

The URLs of the Scheduler, Resource Manager, REST API and Web Interfaces are displayed in the output.

The ProActive server can also be started in a specific state by using the command line options --stopped, --paused or --frozen.

When one of these options is used, the scheduler will start in the provided state, which can be changed afterwards using the ProActive Scheduler portal.

See Scheduler Life-Cycle Management for an explanation of the different scheduler states.

2.3. Install and start ProActive as a system service

The second way to start a ProActive Scheduler is to install it as a system service. Depending on your operating system, the installation of ProActive can be done as follows.

2.3.1. How to install ProActive on Windows

Under Windows, it is possible to use the nssm service manager tool to manage a running script as a service. You can configure nssm to absolve all responsibility for restarting it and let Windows take care of recovery actions. In our case, you need to provide to nssm the Path to this script $PROACTIVE_HOME/bin/proactive-server.bat to start ProActive as a service.

2.3.2. How to install ProActive on Linux

Under Linux, the user has to run the install.sh script located under the tools folder as root and follow the installation steps provided in an interactive mode. Generally, it is recommended to keep the default configuration (internal account, protocol used, port , etc) unless you would like to setup a particular one.

Below is an example of an installation procedure with default answers:

$ sudo $PROACTIVE_HOME/tools/install.sh
Directory where to install the scheduler: /opt/proactive
Name of the user starting the ProActive service: proactive
Group of the user starting the ProActive service: proactive
ProActive use internal system accounts which should be modified in a production environment.
Do you want to regenerate the internal accounts ? [Y/n] n
ProActive can integrate with Linux PAM (Pluggable Authentication Modules) to authenticate users of the linux system.
Warning: this will add the proactive account to the linux "shadow" group
Do you want to set PAM integration for ProActive scheduler? [y/N] n
Protocol to use for accessing Web apps deployed by the proactive server: [http/https] http
Port to use for accessing Web apps deployed by the proactive server: 8080
Number of ProActive nodes to start on the server machine: 4
The ProActive server can automatically remove from its database old jobs. This feature also remove the associated job logs from the file system.
Do you want to enable automatic job/node history removal? [Y/n] y
Remove history older than (in days) [30]: 30
Here are the network interfaces available on your machine and the interface which will be automatically selected by ProActive:
Available interfaces:
Eligible addresses:
/xxx.xxx.xxx.xxx
/127.0.0.1

Elected address : /xxx.xxx.xxx.xxx
Elected address :
Do you want to change the network interface used by the ProActive server? [y/N] n
Start the proactive-scheduler service at machine startup? [Y/n] y
Restrict the access to the ProActive installation folder to user proactive? [y/N] n
Restrict the access to the configuration files containing sensitive information to user proactive? [Y/n] y
Do you want to start the scheduler service now? [Y/n] y
If a problem occurs, check output in /var/log/proactive/scheduler

The installation script will create a symbolic link called default under the root path chosen for the installation that will point to the current version folder.

Accordingly, the ProActive server installation can be accessed using /opt/proactive/default path.

Once the server is up, ProActive Web Portals are available at http://localhost:8080/. The user can access to any of the four portals using the default credentials: admin/admin

Your ProActive Scheduler is ready to execute Jobs!

2.3.3. How to upgrade ProActive on Linux

First, Download ProActive Scheduler and unzip the archive inside a temporary folder that will be used only during the installation (for example /home/user/temp). Make sure that you don’t extract the new version archive in the same folder as the current ProActive installation (e.g. not in /opt/proactive) as it would otherwise create conflicts.

To upgrade ProActive, the user has to run the install.sh script located under the tools folder of the extracted new version as root or sudo and follow the installations steps provided in an interactive mode.

When launching the installation script, the answer to the question Directory where to install the scheduler should point to the same base directory where the old version is installed (e.g. /opt/proactive). This will enable the detection of the existing installation and copying addons, data files and the configuration changes from the previous installation to the new one. Also, the existing version installed in /opt/proactive should not be removed prior to the upgrade. Otherwise, the porting of data and settings from the previous version to the new one would be impossible.

The new version installation will also create a symbolic link called default under ProActive installation root directory that points to the new version, and a symbolic link called previous that points to the older version.

It is thus possible to revert the upgrade if anything goes wrong simply by changing the default symbolic link target. In that case, The previous symbolic link can either be removed or pointed to an older installation.

In some cases, this operation can induce some conflicts that can be solved easily using Git command-line tools.

Below is an example of the upgrade installation procedure with default answers:

$ sudo $PROACTIVE_HOME/tools/install.sh
This will install the ProActive scheduler 10.1.0-SNAPSHOT as a service.
Once the installer is started, it must process until the end otherwise the installation may be corrupted.
Do you want to continue? [Y/n] y
Directory where to install the scheduler: /opt/proactive
Previous installation was owned by user proactive with group proactive
Detected an existing ProActive Scheduler installation at /opt/proactive/8.4
Copying addons and data files from the previous installation...
Do you want to port all configuration changes to the new version (do not do this from a version prior to 8.4.0 before executing the patch)? [Y/n] y
Porting configuration into new installation...
[master 61aaa19] Commit all changes before switching from 8.4 to 10.1
4 files changed, 8 insertions(+), 8 deletions(-)
warning: no common commits
Depuis /opt/proactive/8.4/
* [nouvelle branche] master -> old-version/master

Cherry-picking all changes to new installation
Here is the list of JAR files in the new installation 'addons' folder.
If there are duplicates, you need to manually remove outdated versions.
Restrict the access to the ProActive installation folder to user proactive? [y/N] n
Restrict the access to the configuration files containing sensitive information to user proactive? [Y/n] y
Resource Manager credentials are used by remote ProActive Agents to register to the scheduler.
Do you want to start the scheduler service now? [Y/n] y
If a problem occurs, check output in /var/log/proactive/scheduler

Once the upgrade is performed and the new version started for the first time, the ProActive server will upgrade Catalog objects to the new version. This procedure ensures that any object modified manually by a user will not be upgraded.

2.4. Automated ProActive Installation with Docker Compose

ProActive could be easily deployed and started within a fully containerized environment. For this purpose, we provide a ready-to-use Docker Compose configuration that enables you running ProActive Workflows and Scheduling (PWS) as a containerized service. It starts the following services:

ProActive server container
(Optional) ProActive PostgreSQL database container. The database container is not started when using our embedded HSQL Database.

The Docker Compose configuration pulls the ProActive server and database images from the Private Activeeon Docker Registry.

These images can be used out of the box, on any operating system. You can customize the ProActive server and database settings, located in the .env file according to your requirements.

To be able to access the Private Activeeon Docker Registry, a ProActive enterprise licence and access credentials are required. Please contact us at contact@activeeon.com to request access.

Here are the installation steps you need to follow in order to have ProActive running in a fully containerized environment:

2.4.1. Download the Docker Compose Installation Package

We offer a public Docker Compose installation package, enabling you to easily configure and initiate the Docker installation within your environment.

The Docker Compose installation package includes:

The Docker Compose input file (.env): Used to set the environment variables that are then read by the Docker Compose configuration files.
The Docker Compose configuration files: Define and configure the ProActive server and database containers using YAML definitions:
- docker-compose-hsqldb.yml: Used to have ProActive with an embedded HSQL database.
- docker-compose-pg.yml: Used to have ProActive with a Postgres database.

You can proceed to donwload it using the following command:

2.4.2. Configure the Docker Compose Input File

Before starting the ProActive installation, you need first to configure the installation input parameters using the Docker Compose environment file .env.

Click here to display the .env input file

.env:

##########################################################################################################################################
########################## 1. Installation Properties ####################################################################################
##########################################################################################################################################

# Settings of the Docker registry images
# ProActive Docker image namespace
DOCKER_IMAGE_PROACTIVE_NAMESPACE=<proactive_namespace>
# ProActive version tag to be deployed
DOCKER_IMAGE_PROACTIVE_TAG=<proactive_tag>
# Postgres Docker image namespace
DOCKER_IMAGE_POSTGRES_NAMESPACE=<postgres_namespace>
# Postgres version tag to be deployed
DOCKER_IMAGE_POSTGRES_TAG=<postgres_namespace>

##########################################################################################################################################
########################## 2. PWS Server Configuration Properties ########################################################################
##########################################################################################################################################

# The internal hostname used (e.g. localhost, myserver) or IP address (e.g. 127.0.0.1, 192.168.12.1)
PWS_HOST_ADDRESS=<host_address>
# The internal http protocol to use (http or https)
PWS_PROTOCOL=http
# The internal port used (e.g., 8080 for http, 8443 for https)
PWS_PORT=8080
# The internal port to use for PARM communication, 33647 by default
PWS_PAMR_PORT=33647
# Remove jobs history older than (in days), 30 days by default
PWS_JOB_CLEANUP_DAYS=30
# User starting the ProActive Server
PWS_USER_ID=1001
PWS_USER_GROUP_ID=1001
PWS_USER_NAME=activeeon
PWS_USER_GROUP_NAME=activeeon
# Password of the ProActive 'admin' user
PWS_LOGIN_ADMIN_PASSWORD=activeeon
# Number of ProActive agent nodes to deploy on the ProActive Server machine
PWS_WORKER_NODES=4

##########################################################################################################################################
########################## 3. Public Access Configuration Properties #####################################################################
##########################################################################################################################################

# 'ENABLE_PUBLIC_ACCESS' should be set to true when the ProActive Server is deployed behind a reverse proxy or inside a cloud instance
ENABLE_PUBLIC_ACCESS=false
# Public Hostname used (e.g. public.myserver.mydomain) or IP address (e.g. 192.168.12.1)
PWS_PUBLIC_HOSTNAME=<public_host_address>
# Public protocol used (http or https)
PWS_PUBLIC_SCHEME=https
# Public port used (e.g. 8080, 8443)
PWS_PUBLIC_PORT=8443

##########################################################################################################################################
########################## 4. TLS Certificate Configuration Properties ###################################################################
##########################################################################################################################################

# 'ENABLE_TLS_CERTIFICATE_CONFIGURATION' should be set to true in case you have a valid certificate for the host address
ENABLE_TLS_CERTIFICATE_CONFIGURATION=false
# Path of the certificate full chain or PFX keystore (accepted formats are crt, pem or pfx)
TLS_CERTIFICATE_PATH=/opt/proactive/server/backup/fullchain.pem
# If the certificate is provided in pem/crt format, path to the server private key ('TLS_CERTIFICATE_KEY_PATH' is ignored if the certificate is provided as a pfx keystore)
TLS_CERTIFICATE_KEY_PATH=/opt/proactive/server/backup/privkey.pem
# Password of the server private key (if applicable) or pfx keystore
TLS_CERTIFICATE_KEYSTORE_PASSWORD=activeeon

##########################################################################################################################################
########################## 5. LDAP Configuration Properties ##############################################################################
##########################################################################################################################################

# 'ENABLE_LDAP_AUTHENTICATION' should be set to true if you would like to enable the LDAP authentication to the ProActive Server
ENABLE_LDAP_AUTHENTICATION=false
# Url of the ldap server. Multiple urls can be provided separated by commas
LDAP_SERVER_URLS=<ldap_server_urls>
# Scope in the LDAP tree where users can be found
LDAP_USER_SUBTREE=<ldap_user_subtree>
# Scope in the LDAP tree where groups can be found
LDAP_GROUP_SUBTREE=<ldap_group_subtree>
# Login that will be used when binding to the ldap server (to search for a user and its group). The login should be provided using its ldap distinguished name
LDAP_BIND_LOGIN=<ldap_bind_login>
# Password associated with the bind login
LDAP_BIND_PASSWORD=<ldap_bind_password>
# Ldap filter used to search for a user. %s corresponds to the user login name that will be given as parameter to the filter
LDAP_USER_FILTER=<ldap_user_filter>
# Ldap filter used to search for all groups associated with a user. %s corresponds to the user distinguished name found by the user filter
LDAP_GROUP_FILTER=<ldap_group_filter>
# A login name that will be used to test the user and group filter and validate the configuration. The test login name must be provided as a user id
LDAP_TEST_LOGIN=<ldap_test_login>
# Password associated with the test user
LDAP_TEST_PASSWORD=<ldap_test_password>
# List of existing ldap groups that will be associated with a ProActive administrator role. Multiple groups can be provided separated by commas
LDAP_ADMIN_ROLES=<ldap_admin_roles>
# List of existing ldap groups that will be associated with a ProActive standard user role. Multiple groups can be provided separated by commas
LDAP_USER_ROLES=<ldap_user_roles>

##########################################################################################################################################
########################## 6. PWS Users Configuration Properties #########################################################################
##########################################################################################################################################

# 'ENABLE_USERS_CONFIGURATION' should be set to true if you would like to create internal ProActive users via login/password
ENABLE_USERS_CONFIGURATION=true
# A json formatted array where each object defines a set of LOGIN, PASSWORD, and GROUPS of a ProActive internal user
PWS_USERS=[{ "LOGIN": "test_admin", "PASSWORD": "activeeon", "GROUPS": "scheduleradmins,rmcoreadmins" }, { "LOGIN": "test_user", "PASSWORD": "activeeon", "GROUPS": "user" }]

##########################################################################################################################################
########################## 7. External Database Configuration Properties #################################################################
##########################################################################################################################################

# 'ENABLE_EXTERNAL_DATABASE_CONFIGURATION' should be set to true if you are planning to use ProActive with an external database (i.e., postgres)
# It is recommended to not change these parameters unless you would like to change the passwords
ENABLE_EXTERNAL_DATABASE_CONFIGURATION=false
POSTGRES_PASSWORD=postgres
DB_DNS=proactive-database
DB_PORT=5432
DB_DIALECT=org.hibernate.dialect.PostgreSQL94Dialect
DB_RM_PASS=rm
DB_SCHEDULER_PASS=scheduler
DB_CATALOG_PASS=catalog
DB_PCA_PASS=pca
DB_NOTIFICATION_PASS=notification

Required configuration

The .env file mentioned above is ready for use in a minimal installation. However, it does require the definition of five mandatory parameter values:

DOCKER_IMAGE_PROACTIVE_NAMESPACE: Defines the ProActive Docker image namespace (or repository) in the Activeeon private registry.
DOCKER_IMAGE_PROACTIVE_TAG: Defines the ProActive release version.
DOCKER_IMAGE_POSTGRES_NAMESPACE: Defines the ProActive Postgres image namespace.
DOCKER_IMAGE_POSTGRES_TAG: Defines the Postgres version to be used.
PWS_HOST_ADDRESS: Defines the internal hostname or IP address of the machine where ProActive will be installed.

Advanced configuration

For more advanced configuration of the ProActive server, please refer to the parameters listed below:

PWS Server Configuration Properties

This section outlines the global configuration options available to configure the ProActive server.

PWS_PROTOCOL: The internal http protocol to use (http or https). By default, http.
PWS_PORT: The internal port used (e.g., 8080 for http, 8443 for https). By default, 8080.
PWS_PAMR_PORT: The internal port to use for PAMR communication. By default, 33647.
PWS_JOB_CLEANUP_DAYS: Remove job history older than the specified number of days. By default, 30.
PWS_USER_ID: ID of the user starting the ProActive server. By default, 1001.
PWS_USER_GROUP_ID: Group ID of the user starting the ProActive server. By default, 1001.
PWS_USER_NAME: Name of the user starting the ProActive server. By default, activeeon.
PWS_USER_GROUP_NAME: Group of the user starting the ProActive server. By default, activeeon.
PWS_LOGIN_ADMIN_PASSWORD: Password of the ProActive 'admin' user. By default, activeeon.
PWS_WORKER_NODES: Number of ProActive agent nodes to deploy on the ProActive server machine. By default, 4.

Public Access Configuration Properties

This section describes the public REST Urls to be set when you are running the ProActive server behind a reverse proxy or in a cloud instance (e.g., Azure VM)

ENABLE_PUBLIC_ACCESS: Set to true when the ProActive server is deployed behind a reverse proxy or inside a cloud instance. By default, false.
PWS_PUBLIC_HOSTNAME: Public hostname used (e.g. public.myserver.mydomain) or IP address (e.g. 192.168.12.1).
PWS_PUBLIC_SCHEME: Public protocol used (http or https).
PWS_PUBLIC_PORT: Public port used (e.g. 8080, 8443).

TLS Certificate Configuration Properties

In case you would like to set up a TLS certificate for the ProActive server, you can customize it in this section:

ENABLE_TLS_CERTIFICATE_CONFIGURATION: Set to true in case you have a valid certificate for the host address. By default, false.
TLS_CERTIFICATE_PATH: Path of the certificate full chain or PFX keystore (accepted formats are crt, pem or pfx).
TLS_CERTIFICATE_KEY_PATH: If the certificate is provided in pem/crt format, path to the server private key (TLS_CERTIFICATE_KEY_PATH is ignored if the certificate is provided as a pfx keystore).
TLS_CERTIFICATE_KEYSTORE_PASSWORD: Password of the server private key (if applicable) or pfx keystore. By default, activeeon.

The pem/crt/pfx certificate as well as the key have to be placed in the backup folder and their absolute paths need to be provided. See section Required configuration to learn more about the backup folder to be created beforehand.

LDAP Configuration Properties

In case you would like to set up the LDAP authentication for the ProActive server, you can enable the LDAP configuration in this section.

The given parameter values are only examples and actual parameters must match your LDAP organization. This section will require two accounts:

An LDAP_BIND_LOGIN (given as a LDAP distinguished name) that will be used by the ProActive server to execute LDAP queries (search queries only).
An LDAP_TEST_LOGIN (given as a LDAP user login name) that will be used to validate the configuration and guarantee that the connection to the LDAP server is working properly. This test account is only used during the installation.

You can then set up the right LDAP configuration using the following properties:

ENABLE_LDAP_AUTHENTICATION: Set to true if you would like to enable the LDAP authentication to the ProActive server. By default, false.
LDAP_SERVER_URLS: Url of the LDAP server. Multiple urls can be provided separated by commas. Example: ldaps://ldap-server.company.com
LDAP_USER_SUBTREE: Scope in the LDAP tree where users can be found. Example: ou=users,dc=activeeon,dc=com
LDAP_GROUP_SUBTREE: Scope in the LDAP tree where groups can be found. Example: ou=groups,dc=activeeon,dc=com
LDAP_BIND_LOGIN: Login that will be used when binding to the LDAP server (to search for a user and its group). The login should be provided using its LDAP distinguished name. Example: cn=admin,dc=activeeon,dc=com
LDAP_BIND_PASSWORD: Password associated with the bind login.
LDAP_USER_FILTER: LDAP filter used to search for a user. %s corresponds to the user login name that will be given as parameter to the filter. Example: (&(objectclass=inetOrgPerson)(uid=%s))
LDAP_GROUP_FILTER: LDAP filter used to search for all groups associated with a user. %s corresponds to the user distinguished name found by the user filter. Example: (&(objectclass=groupOfUniqueNames)(uniqueMember=%s))
LDAP_TEST_LOGIN: A login name that will be used to test the user and group filter and validate the configuration. The test login name must be provided as a user id. Example: activeeon
LDAP_TEST_PASSWORD: Password associated with the test user.
LDAP_ADMIN_ROLES: List of existing LDAP groups that will be associated with a ProActive administrator role. Multiple groups can be provided separated by commas. Example: activeeon-admins
LDAP_USER_ROLES: List of existing LDAP groups that will be associated with a ProActive standard user role. Multiple groups can be provided separated by commas. Example: activeeon-operators

PWS Users Configuration Properties

In case you would like to create internal user accounts for the ProActive server, you can enable the users configuration in this section:

ENABLE_USERS_CONFIGURATION: Set to true if you would like to create internal ProActive users via login/password. By default, false.
PWS_USERS: A json formatted array where each object defines a set of LOGIN, PASSWORD, and GROUPS of a ProActive internal user. The list of ProActive predefined GROUPS that can be associated with a user are listed and explained following this link: User Permissions

External Database Configuration Properties

These parameters are only used when ProActive is started with a Postgres database. It is recommended to not change these parameters unless you would like to change the passwords.

ENABLE_EXTERNAL_DATABASE_CONFIGURATION: Set to true if you would like to start ProActive with a Postgres database. By default, false.
POSTGRES_PASSWORD: Password of the Postgres database user. By default, postgres.
DB_DNS: DNS of the machine hosting the database. By default, it is hosted in the same network as the ProActive server container, hence, the database container name is referenced.
DB_PORT: Database port. By default, 5432.
DB_DIALECT: Database dialect to be used for hibernate communication. By default, PostgreSQL94Dialect.
DB_RM_PASS: Password of the rm user in the RM database. By default, rm.
DB_SCHEDULER_PASS: Password of the scheduler user in the SCHEDULER database. By default, scheduler.
DB_CATALOG_PASS: Password of the catalog user in the CATALOG database. By default, catalog.
DB_PCA_PASS: Password of the pca user in the PCA database. By default, pca.
DB_NOTIFICATION_PASS: Password of the notification user in the NOTIFICATION database. By default, notification.

2.4.3. Docker Compose Configuration with an Embedded HSQL Database

Once you set the required configuration options in the .env file, you can use now the Docker Compose configuration provided in the installation package (docker-compose-hsqldb.yml) to run the ProActive server container with an embedded HSQL database.

It will start one single container related to the ProActive server.

Click here to display the docker-compose-hsqldb.yml file

docker-compose-hsqldb.yml:

version: '2.1'

services:
  proactive-server:
    restart: on-failure
    image: dockerhub.activeeon.com/${DOCKER_IMAGE_PROACTIVE_NAMESPACE}/proactive-scheduler:${DOCKER_IMAGE_PROACTIVE_TAG}
    container_name: proactive-scheduler
    networks:
      - pa-network
    ports:
      - ${PWS_PORT}:${PWS_PORT}
      - ${PWS_PAMR_PORT}:${PWS_PAMR_PORT}
    env_file:
      - .env
    environment:
      - PWS_PROTOCOL
      - PWS_PORT
    volumes:
      - proactive-default:/opt/proactive/server/default
      - proactive-previous:/opt/proactive/server/previous
      - proactive-backup:/opt/proactive/server/backup
    healthcheck:
      test: [ "CMD", "curl", "-k", "-f", "${PWS_PROTOCOL}://proactive-server:${PWS_PORT}" ]
      interval: 30s
      timeout: 10s
      retries: 30

volumes:
  proactive-default:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/proactive/default'
  proactive-previous:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/proactive/previous'
  proactive-backup:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/proactive/backup'
networks:
  pa-network:

Required configuration

You can use the YAML Docker Compose definition as it is. However, to persist containers’ data, it is necessary to establish local mounted volumes on the ProActive server and database VMs. Hence, you need to create 3 empty folders on the host machine named default, previous, and backup.

$ mkdir -p /opt/proactive/default /opt/proactive/previous /opt/proactive/backup

Then, you have to configure the absolute paths to these 3 folders in the docker-compose-hsqldb.yml file using the device: property (e.g., in the example above 3 folders are created and configured: /opt/proactive/default, /opt/proactive/previous, and /opt/proactive/backup).

...
volumes:
  proactive-default:
    ...
      device: '/opt/proactive/default'
  proactive-previous:
    ...
      device: '/opt/proactive/previous'
  proactive-backup:
    ...
      device: '/opt/proactive/backup'

These folders will host the current installation files (default), the upgraded one (previous), and all previous upgraded installation files (backup).

2.4.4. Docker Compose Configuration with a PostgreSQL Database

To use the Docker Compose configuration with a PostgreSQL database, ensure that the correct configuration options are set in the .env file beforehand. Additionally, enable the advanced property ENABLE_EXTERNAL_DATABASE_CONFIGURATION.

Once done, you can then proceed with the Docker Compose configuration provided in the installation package docker-compose-pg.yml to launch the ProActive server container alongside an external PostgreSQL database container.

It will start two containers, one for the server, and one for the database.

Click here to display the docker-compose-pg.yml file

docker-compose-pg.yml:

version: '2.1'

services:
  proactive-server:
    restart: on-failure
    image: dockerhub.activeeon.com/${DOCKER_IMAGE_PROACTIVE_NAMESPACE}/proactive-scheduler:${DOCKER_IMAGE_PROACTIVE_TAG}
    container_name: proactive-scheduler
    networks:
      - pa-network
    ports:
      - ${PWS_PORT}:${PWS_PORT}
      - ${PWS_PAMR_PORT}:${PWS_PAMR_PORT}
    env_file:
      - .env
    environment:
      - PWS_PROTOCOL
      - PWS_PORT
    volumes:
      - proactive-default:/opt/proactive/server/default
      - proactive-previous:/opt/proactive/server/previous
      - proactive-backup:/opt/proactive/server/backup
    depends_on:
      proactive-database:
        condition: service_healthy
    healthcheck:
      test: [ "CMD", "curl", "-k", "-f", "${PWS_PROTOCOL}://proactive-server:${PWS_PORT}" ]
      interval: 30s
      timeout: 10s
      retries: 30

  proactive-database:
    restart: on-failure
    image: dockerhub.activeeon.com/${DOCKER_IMAGE_POSTGRES_NAMESPACE}/postgres-pa:${DOCKER_IMAGE_POSTGRES_TAG}
    container_name: proactive-database
    networks:
      - pa-network
    env_file:
      - .env
    volumes:
      - proactive-data:/var/lib/postgresql/data
    ports:
      - 5432:5432
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  proactive-default:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/proactive/default'
  proactive-previous:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/proactive/previous'
  proactive-backup:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/proactive/backup'
  proactive-data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/proactive/data'
networks:
  pa-network:

Required configuration

Same as for a Docker Compose installation using an embedded database (See section persistent volumes with compose), you need to create and link an empty folder to host the Postgres data in addition to the basic three default, previous, and backup folders that host the ProActive installation(s).

$ mkdir -p /opt/proactive/default /opt/proactive/previous /opt/proactive/backup /opt/proactive/data

You then need to specify the folder paths in the Docker Compose configuration file docker-compose-pg.yml:

...
volumes:
  proactive-default:
    ...
      device: '/opt/proactive/default'
  proactive-previous:
    ...
      device: '/opt/proactive/previous'
  proactive-backup:
    ...
      device: '/opt/proactive/backup'
  proactive-data:
    ...
      device: '/opt/proactive/data'

2.4.5. Start ProActive Using Docker Compose

$ docker login dockerhub.activeeon.com

To start the ProActive server with a default HSQL database, run:

$ docker-compose -f docker-compose-hsqldb.yml up -d

To start the ProActive server with a PostgreSQL database, run:

$ docker-compose -f docker-compose-pg.yml up -d

To monitor the ProActive server startup, run:

$ docker logs -f proactive-scheduler

Wait until the command output shows that ProActive server is started:

....
Loading workflow and utility packages...
Packages successfully loaded
ProActive started in 246.476727488 seconds

You can then access the ProActive Web Portals at <PWS_PROTOCOL>://<PWS_HOST_ADDRESS>:<PWS_PORT>/. The URL property values are specified in the .env file.

The user can access any of the four portals using the default credentials: admin/<PWS_LOGIN_ADMIN_PASSWORD>. PWS_LOGIN_ADMIN_PASSWORD is, by default, activeeon but you can customize it as well in the .env file.

2.4.6. Database Purge Operation in Docker Compose

In Postgres database, vacuum operations do not by default remove orphaned Large Objects from the database. Identification and removal of these objects is performed by the vacuumlo command. As the ProActive database contains many large objects, it is essential to apply both vacuumlo and vacuum operations in order to control the disk space usage over time. The purge operation is automatically integrated and performed within the Postgres container using a cron.daily process. Though, prior to executing the purge operation, a preliminary configuration step is necessary (this preliminary step is done once for a given ProActive installation). The following commands allow to prepare the database for the automatic purge operations performed within the Postgres database container.

Stop the ProActive server container:

$ docker-compose stop proactive-server

Run the embedded script for table configuration within the database container:

$ docker exec -it proactive-database /bin/bash /tables-configuration.sh

Start the ProActive server again:

$ docker-compose start proactive-server

The installation of ProActive using Docker Compose is now over ! Your ProActive Scheduler is now running as a set of container services !

2.4.7. Upgrade/Downgrade/Backup ProActive Using Docker Compose

ProActive can be easily upgraded using Docker Compose. We support automatic upgrade, downgrade, and backup of ProActive server installations in Docker.

Docker Compose configuration relies on Docker container volumes to host ProActive installation directories:

The proactive-default volume hosts the current and default installation.
The proactive-previous volume hosts the previous installation after upgrade.
The proactive-backup volume hosts all previous installations after multiple upgrades.

In the following, we are going to explain each procedure.

ProActive upgrade

In order to upgrade the ProActive server to a newer version, you have to edit the .env config file to provide a newer version. The current ProActive version is specified using the DOCKER_IMAGE_PROACTIVE_TAG option in the input installation file.

For example, you can replace DOCKER_IMAGE_PROACTIVE_TAG=13.0.0 by DOCKER_IMAGE_PROACTIVE_TAG=13.0.8 to get a maintenance release of ProActive. It is also applicable for minor and major release versions.

Now that a newer version is provided, you can restart the container:

$ docker-compose -f docker-compose-hsqldb.yml down
$ docker-compose -f docker-compose-hsqldb.yml up -d

The new version will be detected and the upgrade process will start at container startup. In fact, the current version will be automatically placed on the proactive-previous volume and the newer version will be placed on the proactive-default volume and started from there.

ProActive backup

The backup of previous ProActive versions is also automatic and it is triggered on every upgrade. When a new upgrade is started, the proactive-previous volume is checked and if it is not empty, its content is automatically moved to the backup volume proactive-backup. Afterwards, the upgrade process starts as previously described.

ProActive downgrade

Using Docker Compose we also support one-level automatic downgrade of ProActive installations located in proactive-default and proactive-previous volumes. To do so, you should provide the previous version of ProActive, as it is specified in the proactive-previous volume, to the DOCKER_IMAGE_PROACTIVE_TAG option in the .env file. This will allow to automatically swap the content of proactive-default and proactive-previous volumes. ProActive will be then started from the proactive-default volume which will contain the last previous version.

2.5. Automated ProActive Installation with Kubernetes

ProActive could be easily deployed and started on AKS (Azure) or bare-metal (On-premises) Kubernetes clusters. For this purpose, we provide a ready-to-use Kubernetes configuration that enables you running ProActive Workflows and Scheduling (PWS) as a set of pods. It allows you to start the following pods:

ProActive server pod
(Optional) ProActive static nodes pods
(Optional) ProActive dynamic nodes pods
(Optional) ProActive PostgreSQL database pod. The database pod is not started when using an embedded HSQL database.

The Kubernetes installation pulls the ProActive server, node, and database images from the Private Activeeon Docker Registry.

To be able to pull images from the Private Activeeon Docker Registry, a ProActive enterprise licence and access credentials are required. Please contact us at contact@activeeon.com to request access and support.

Here are the installation steps you need to follow in order to have ProActive running in a Kubernetes cluster:

2.5.1. Download the Helm Installation Package

Helm is a Kubernetes deployment tool for automating the creation, packaging, configuration, and deployment of applications and services to Kubernetes clusters.

ProActive uses Helm Charts (i.e., Helm package definition) to create and configure the kubernetes deployments. Thus, Helm must be pre-installed in order to deploy the ProActive pods.

Please refer to Helm’s documentation to get started.

Once Helm is set up properly, proceed to download the Helm installation package of ProActive:

The next step is to configure the Helm Chart by setting up the right configuration in the input values.yml file of the Chart.

2.5.2. Configure the Helm Input File

The values.yaml, which is included in the installation package, is necessary to provide the right configuration values before installing the Chart.

Click here to display the Helm input file

values.yaml:

# Default values for Activeeon ProActive Chart
# This is a YAML-formatted file
# It declares variables to be passed into the ProActive Helm templates

# We provide below a summary of the different configuration blocks
# You can search for one of these terms to quickly jump into a specific block

# Configuration Blocks Menu (in order):
# *********** Common Configuration Properties ***************************
# - target:
# - namespace:
# - imageCredentials:
# *********** ProActive Scheduler Server Configuration Properties *******
# - user:
# - protocol:
# - proactiveAdminPassword:
# - housekeepingDays:
# - serverWorkerNodes:
# - ports:
# - database:
# - usersConfiguration:
# - ldapConfiguration:
# *********** Node Sources Configuration Properties *********************
# - nodeSources:
# *********** Kubernetes Resources Configuration Properties *************
# - ingress:
# - persistentVolumes:
# - services:
# - replicas:
# - images:
# - resources:
# - volumeMounts:
# - readinessProbe:
# - livenessProbe:

#***********************************************************************************************************************
#**************************************** Common Configuration Properties **********************************************
#***********************************************************************************************************************

#**************************************** Target Kubernetes cluster ****************************************************
# Supported target Kubernetes clusters: "on-prem" or "aks" clusters
target:
  # The 'cluster' property value could be "aks" or "on-prem"
  cluster: <cluster>

#**************************************** Namespace Configuration ******************************************************
# Namespace to be used to create all ProActive installation resources
# The namespace will be associated to every resource created by the helm chart
namespace: ae-dev

#******************************** Credentials of the Activeeon Private Docker Registry *********************************
# If you do not have access, please contact contact@activeeon.com to request access
imageCredentials:
  registry: dockerhub.activeeon.com
  username: <username>
  password: <password>
  email: support@activeeon.com

#***********************************************************************************************************************
#**************************************** ProActive Scheduler Server Configuration *************************************
#***********************************************************************************************************************

#**************************************** User Starting the ProActive Server *******************************************
# On the Scheduler Pod startup, a new user is created to start the proactive server
# The root user is only used to do the installation
# The server is started by the following user
user:
  userName: activeeon
  groupName: activeeon
  uid: 1000
  gid: 1000

#**************************************** ProActive Web Protocol Configuration *****************************************
# Protocol http or https to use to access the ProActive Server Web Portals (default: http)
# The https requires for now some advanced manual configurations
# The http protocol is recommended to be used
protocol: http

#**************************************** ProActive 'admin' Password Configuration *************************************
# The admin user is an administrator of the platform
# This account is used internally for the installation to perform some actions like getting the session id or creating worker nodes, etc.
# It is used as well to access the ProActive web portals, at the login page
# It important to provide a custom password and then access the web portals using the given password
proactiveAdminPassword: activeeon

#**************************************** Jobs Housekeeping Configuration **********************************************
# The Scheduler provides a housekeeping mechanism that periodically removes finished jobs from the Scheduler and Workflow Execution portals
# It also removes associated logs and all job data from the database to save space
# The following property defines the number of days before jobs cleanup (default: 30)
housekeepingDays: 30

#**************************************** Server Worker Nodes Configuration ********************************************
# Defines the number of worker agent nodes to be created on the Scheduler server Pod
# It is recommended to not create too many nodes on the same Pod as the server to avoid performance overhead
serverWorkerNodes: 1

#**************************************** Ports Configuration **********************************************************
# This section describes the ports to be used by this installation
ports:
  # Port to use to access the ProActive web portals (default: 8080)
  # The web portals will be accessible through the external IP or DNS given by the load balancer service + httpPort
  httpPort: &httpPort 8080
  # Port to use for PAMR communication (default: 33647)
  # It is recommended to use this default port
  pamrPort: &pamrPort 33647
  # Port to use to access the ProActive web portals on an on-prem installation (without a LoadBalancer)
  # The web portals will be accessible through the cluster IP or DNS + nodePort
  nodePort: &nodePort 32000

#**************************************** Database Configuration *******************************************************
# Database configuration properties
database:
  # Enable or disable using an external database
  # If set to 'true', postgres database is used, 'false' sets the default embedded HSQL database
  enabled: true
  # Postgres database configuration
  # The next properties are considered only if the external database is used
  dialect: org.hibernate.dialect.PostgreSQL94Dialect
  # Credentials, used only for postgres configuration
  # They define the passwords of the catalog, notification, pca, rm, and scheduler databases
  passwords:
    catalogPassword: catalog
    notificationPassword: notification
    pcaPassword: pca
    rmPassword: rm
    schedulerPassword: scheduler
    postgresPassword: postgres

#**************************************** File Authentication Configuration ********************************************
# This is the default authentication method provided by ProActive
# The login, password, and groups could be provided in a JSON format
# For more information about the groups and other authentication methods, visit: https://doc.activeeon.com/latest/admin/ProActiveAdminGuide.html#_file
usersConfiguration:
  addUsersConfiguration: true
  users: [{ "LOGIN": "test_admin", "PASSWORD": "activeeon", "GROUPS": "scheduleradmins,rmcoreadmins" },
          { "LOGIN": "test_user", "PASSWORD": "activeeon", "GROUPS": "user" }]

#**************************************** LDAP Authentication Configuration ********************************************
# In addition to file based authentication, we provide a second authentication method, the LDAP authentication
ldapConfiguration:
  # set ‘enableLdapConfiguration’ to true if you want to activate the ldap authentication in addition to the file based authentication
  enableLdapConfiguration: false
  # url of the ldap server. Multiple urls can be provided separated by commas
  serverUrls: "<serverUrls>"
  # Scope in the LDAP tree where users can be found
  userSubtree: "<userSubtree>"
  # Scope in the LDAP tree where groups can be found
  groupSubtree: "<groupSubtree>"
  # login that will be used when binding to the ldap server (to search for a user and its group). The login should be provided using its ldap distinguished name
  bindLogin: "<bindLogin>"
  # encrypted password associated with the bind login
  bindPassword: "<bindPassword>"
  # ldap filter used to search for a user. %s corresponds to the user login name that will be given as parameter to the filter
  userFilter: "<userFilter>"
  # ldap filter used to search for all groups associated with a user. %s corresponds to the user distinguished name found by the user filter
  groupFilter: "<groupFilter>"
  # a login name that will be used to test the user and group filter and validate the configuration. The test login name must be provided as a user id
  testLogin: "<testLogin>"
  # encrypted password associated with the test user
  testPassword: "<testPassword>"
  # list of existing ldap groups that will be associated with a ProActive administrator role. Multiple groups can be provided separated by commas
  adminRoles: "<adminRoles>"
  # list of existing ldap groups that will be associated with a ProActive standard user role. Multiple groups can be provided separated by commas
  userRoles: "<userRoles>"

#***********************************************************************************************************************
#**************************************** Node Sources Configuration ***************************************************
#***********************************************************************************************************************
# The following section describe the worker pods to be created following the Scheduler server startup
nodeSources:
  # The 'dynamicK8s' configuration defines the worker nodes to be created dynamically depending on the workload
  dynamicK8s:
    # True, to set up a Dynamic Kubernetes Node Source at Server startup (false if not)
    enabled: true
    # Name of the Dynamic Kubernetes Node Source
    name: Dynamic-Kube-Linux-Nodes
    # Minimum number of nodes to be deployed by the node source
    minNodes: 4
    # Maximum number of nodes to be deployed
    maxNodes: 15
  # the 'staticLocal' configuration defines a fixed number of worker nodes to be created following the server startup
  staticLocal:
    # True, to set up a Static Kubernetes Node Source at Server startup (false if not)
    enabled: true
    # Name of the Static Kubernetes Node Source
    name: Static-Kube-Linux-Nodes
    # Number of nodes to be deployed
    numberNodes: 4

#***********************************************************************************************************************
#**************************************** Kubernetes Resources Configuration *******************************************
#***********************************************************************************************************************

#**************************************** Ingress Configuration ********************************************************
ingress:
  # Specifies whether the Ingress resource should be enabled (true or false)
  enabled: false
  # Specifies the Ingress class name to be used. Use 'kubectl get ingressclass' to check the available ingress classes in your cluster
  ingressClassName: public
  # Specifies the hostname to be used by the Ingress service
  hostname: <hostname>
  # To configure the TLS certificate for the https mode. We provide three ways:
  # - Provided certificate: a fullchain + private key have to be provided
  # - Self Signed certificate using Cert Manager
  # - Let's Encrypt certificate using Cert Manager
  # You have to enable only one of these certification method exclusively
  providedCertificate:
    # Specifies whether a provided certificate should be used (true or false)
    enabled: true
    tls:
      # The actual content of the TLS certificate file (encoded in Base64). Use for example 'cat fullchain.pem | base64' to get the encrypted certificate file content
      crt: <fullchain.pem>
      # The actual content of the TLS private key file (encoded in Base64). Use for example 'cat privkey.pem | base64' to get the encrypted private key file content
      key: <privkey.pem>
  clusterIssuer:
    selfSigned:
      # Specifies whether a self-signed certificate should be used (true or false)
      enabled: false
    letsEncrypt:
      # Specifies whether Let's Encrypt should be used for obtaining certificates (true or false)
      enabled: false
      # The Let's Encrypt ACME server URL
      server: https://acme-v02.api.letsencrypt.org/directory
      # The email address to be associated with the Let's Encrypt account
      email: test@test.com

#**************************************** Persistent Volumes Configuration *********************************************

persistentVolumes:
  # The following properties define the required settings to create local Node persistent volumes
  # The Local Node Volume will host the Scheduler Server installation files and the external database data
  localVolumeConfig:
    # Describes the storage class configuration
    storageClass:
      # Specifies the provisioner for the storage class as "kubernetes.io/no-provisioner."
      provisioner: kubernetes.io/no-provisioner
      # Sets the volume binding mode to "Immediate."
      volumeBindingMode: Immediate
    # Contains settings related to scheduling and placement of the persistent volumes.
    scheduler:
      persistentVolume:
        # Specifies the storage capacity for the persistent volume
        storage: 20Gi
        # Defines the access modes for the persistent volume as ReadWriteMany
        accessModes: ReadWriteMany
        # Sets the reclaim policy for the persistent volume to "Delete."
        persistentVolumeReclaimPolicy: Delete
        # Specifies the local path for the persistent volume
        localPath: <localPath>
        #Defines node affinity settings
        nodeAffinity:
          # Specifies the node name
          nodeName: <nodeName>
      persistentVolumeClaim:
        # Specifies the access modes for the persistent volume claim as ReadWriteMany
        accessModes: ReadWriteMany
        # Specifies the storage capacity for the persistent volume claim
        storage: 20Gi
    # Contains settings related to the database and placement of the persistent volumes
    # It is used only when the postgres database is enabled
    database:
      persistentVolume:
        storage: 20Gi
        accessModes: ReadWriteMany
        persistentVolumeReclaimPolicy: Delete
        localPath: <localPath>
        nodeAffinity:
          nodeName: <nodeName>
      persistentVolumeClaim:
        accessModes: ReadWriteMany
        storage: 20Gi

  # The following properties define the required settings to create the Azure Disk for data persistence
  # The Azure Disk will host the Scheduler Server installation files and the external database data
  aksDiskVolumeConfig:
    # The 'StorageClass' defines different classes of storage and their associated provisioning policies in a cluster
    # It allows to abstract the underlying storage infrastructure and provide a way to dynamically provision and manage persistent volumes (PVs)
    storageClass:
      # The 'provisioner' specifies the CSI (Container Storage Interface) driver that is responsible for provisioning the storage
      # In this case, 'disk.csi.azure.com' is the provisioner for Azure Disk storage
      provisioner: disk.csi.azure.com
      # The 'reclaimPolicy' determines what happens to the underlying Azure Disk resource when the associated PersistentVolume (PV) is deleted
      # The 'Delete' policy means that the Azure Disk will be deleted when the PV is released
      reclaimPolicy: Delete
      # The 'allowVolumeExpansion' parameter indicates whether the storage volume associated with this StorageClass can be resized after creation
      # Setting it to 'true' allows for volume expansion
      allowVolumeExpansion: true
      # The 'storageaccounttype' specifies the type of Azure storage account to be used for provisioning the Azure Disk
      # In this case, 'StandardSSD_LRS' stands for Standard Solid State Drive (SSD) with Locally Redundant Storage (LRS)
      storageaccounttype: StandardSSD_LRS
      # The 'kind' specifies the type of the storage class
      # In this case, it's set to 'Managed', which implies that the Azure Disk will be managed by the underlying cloud provider (Azure) rather than being manually managed
      kind: Managed
    # The scheduler persistent volume claim for Azure Disk provisioning
    # For memory, PersistentVolumeClaim (PVC) is used to request and allocate storage resources for applications running within the cluster
    scheduler:
      persistentVolumeClaim:
        # The 'storage' property specifies the desired storage capacity for the PVC
        # In this case, it's set to 40Gi, which means you're requesting a PVC with a storage capacity of 40 gigabytes
        storage: 40Gi
        # The 'accessModes' property defines the access modes that the PVC requires
        # In this case, ReadWriteOnce indicates that the volume can be mounted for read and write access by a single pod at a time
        accessModes: ReadWriteOnce
    # The external database (postgres) persistent volume claim for Azure Disk provisioning
    # It is used only when the postgres database is enabled
    database:
      persistentVolumeClaim:
        storage: 20Gi
        accessModes: ReadWriteOnce

#**************************************** Services Configuration *******************************************************
# Define the kubernetes service properties required for the installation
services:
  scheduler:
    # Scheduler Service to access the ProActive Web portals
    webService:
      # The 'name' property specifies the name of the Service port
      # In this case, the Service port is named http
      name: http
      # The 'protocol' property defines the network protocol to use for the Service
      # In this case, it's set to TCP
      protocol: TCP
      # The 'port' property specifies the port number on which the Service will be exposed externally
      # The value *httpPort suggests that it's using a variable named httpPort (defined previously)
      port: *httpPort
      # The 'nodePort' property specifies the Node port number on which the Service will be exposed. It is used only on an on-prem installation
      # The value *nodePort suggests that it's using a variable named nodePort (defined previously)
      nodePort: *nodePort
      # The 'targetPort' property specifies the port to which the traffic will be forwarded from the Service to the pods
      # Again, *httpPort indicates the use of a variable named httpPort (defined previously)
      targetPort: *httpPort
    # Scheduler Service to ensure the communication between Nodes and Scheduler Pods
    pamrService:
      # The 'name' property specifies the name of the Service port
      # In this case, the Service port is named pamr
      name: pamr
      # The 'protocol' property defines the network protocol to use for the Service
      # In this case, it's set to TCP
      protocol: TCP
      # The 'port' property specifies the port number on which the Service will be exposed internally within the cluster
      # The value *pamrPort suggests that it's using a variable named pamrPort (defined previously)
      port: *pamrPort
      # The 'type' property determines the type of the Service
      # In this case, it's set to ClusterIP, which means the Service will be accessible only within the cluster and will have an internal IP
      type: ClusterIP
  # Database service required to enable the communication between the Scheduler and the database
  # The following configuration is not needed and is not taken into account in case the external database is not used
  dbService:
    postgres:
      # The 'protocol' property defines the network protocol to use for the Service
      # In this case, it's set to TCP
      protocol: TCP
      # The 'port' property specifies the port number on which the Service will be exposed internally within the cluster
      # In this case, it's set to 5432, which is used for PostgreSQL database connections
      port: 5432
      # The 'type' property determines the type of the Service
      # In this case, it's set to ClusterIP, which means the Service will be accessible only within the cluster and will have an internal IP
      type: ClusterIP

#**************************************** Replicas Configuration *******************************************************
# Replicas counts of scheduler, node, and database deployments
replicas:
  # The replica count for the scheduler has to be set to 1
  scheduler:
    replicaCount: 1
  # The replica count for the node can be set 0 to N
  # N depends on the number of worker nodes required
  # The number of worker nodes depends on the workload
  node:
    replicaCount: 1
  # The replica count for the database has to be set to 1
  db:
    replicaCount: 1

#**************************************** Container Images Configuration ***********************************************
# Docker images to be used to start the Server, Nodes, initContainers, dind, and database containers
images:
  # The scheduler image represents the container image of the ProActive scheduler server Pod
  scheduler:
    # 'dockerhub.activeeon.com' is the private Activeeon docker registry
    # For example, dev is the namespace used to host the scheduler snapshot images
    # proactive-scheduler is the image name
    repository: dockerhub.activeeon.com/<namespace>/proactive-scheduler
    # The 'tag' defines the scheduler version to be installed
    # For example, 13.1.0-SNAPSHOT-2023-08-28 represents the scheduler snapshot version 13.1.0 released on 2023-08-28
    # The tag could represent a snapshot release, a minor or major release, a release candidate, or a maintenance release
    tag: <proactive_tag>
    # The 'pullPolicy' determines the conditions under which the container image is fetched from the container registry
    # It could be: IfNotPresent, Always, or Never
    # We recommend to set it to 'always' to always pull the latest image updates
    pullPolicy: Always
  # The node image is used by the static worker nodes to instantiate static ProActive agents
  # Custom user application images cannot be provided since the agent is prebuilt and installed in the ProActive Node image
  node:
    repository: dockerhub.activeeon.com/<namespace>/proactive-k8s-node
    tag: <node_tag>
    pullPolicy: Always
  # The following image is used by the dynamic node source to instantiate dynamic work nodes (e.g., your application image name)
  # This custom image requires 'wget' and 'curl' packages to be installed
  # The agent will be installed on the fly on the desired container, at the Pod creation
  # By default, we provide the following basic Linux image that includes standard Linux packages and libraries
  dynamicNodeSource:
    image: dockerhub.activeeon.com/public/proactive-k8s-dynamic-node:latest
  # The initContainers are used to do simple Kubernetes service checks
  # Bash scripts are executed to check the response of the service endpoints
  # For example, the Node pod check if the server is up before creating the agent containers
  initContainers:
    repository: busybox
    tag: latest
    pullPolicy: IfNotPresent
  # Dind image (aka Docker In Docker) allows to run a Docker container within an already running Docker container
  # To make it possible to run containers in the node pod container, this image is required to create the docker daemon and give access to it by the parent node container
  dind:
    repository: docker
    tag: 20.10.7-dind
    pullPolicy: Always
  # The database image is a custom postgres database image hosted in the Activeeon registry
  # It initializes the database schema, add users, etc.
  # You can have more information about the database configuration here: https://doc.activeeon.com/latest/admin/ProActiveAdminGuide.html#_example_configuration_with_postgresql
  db:
    postgres:
      repository: dockerhub.activeeon.com/<namespace>/postgres-pa
      tag: <postgres_tag>
      pullPolicy: Always

#**************************************** Pod Resources Configuration **************************************************
# Define the resources to be allocated for the Scheduler, Nodes, and database Pods
resources:
  scheduler:
    # As minimum requirements, the scheduler pod requires/requests a minimum of 4 CPU cores and 6G of Ram
    requests:
      cpu: 4000m #equivalent to 4 CPU cores
      memory: 6G
  # As minimum requirements, the node pod requires/requests a minimum of 2 CPU cores and 2G of Ram
  node:
    requests:
      cpu: 2000m
      memory: 2G
  # As minimum requirements, the database pod requires/requests a minimum of 2 CPU cores and 4G of Ram
  db:
    postgres:
      requests:
        cpu: 2000m
        memory: 4G

#**************************************** Volume Mounts Configuration **************************************************
# 'volumeMounts' where you define the various volume mounts that should be attached to the container
volumeMounts:
  # The scheduler pod has 3 volume mounts
  scheduler:
    # The 'default' volume mount hosts the scheduler installation files and logs
    # It also hosts the internal database data
    # It is recommended to not change these values
    default:
      # This specifies the directory path within the container where the volume should be mounted
      mountPath: /opt/proactive/server/default
    # The 'previous' volume mount hosts the previous installation files in case an upgrade was made
    previous:
      mountPath: /opt/proactive/server/previous
    # The 'backup' volume mount hosts all previous installations in case multiple upgrades were made
    backup:
      mountPath: /opt/proactive/server/backup
  # The 'db' volume mount hosts the postgres database data
  db:
    postgres:
      mountPath: /var/lib/postgresql/data

#**************************************** Readiness and Liveness Probes Configuration **********************************
# Define the 'readinessProbe' rules for the Scheduler Pod
# This configuration creates a readiness probe that checks the health of the container by sending an HTTP GET request
readinessProbe:
  # This specifies the type of probe to use, which is an HTTP GET request
  # It checks the container's health by sending an HTTP request to a specific endpoint
  httpGet:
    # This indicates that the port for the HTTP GET request is taken from a variable named httpPort (previously defined)
    port: *httpPort
  # This property sets the number of seconds to wait before starting the first probe after the container starts
  # In this case, it's set to 120 seconds
  initialDelaySeconds: 120
  # This property defines the number of seconds between successive probes
  # The container will be probed every 5 seconds
  periodSeconds: 5
  # This property specifies how long the probe should wait for a response from the container before considering the probe as failed
  # It's set to 3 seconds in this case
  timeoutSeconds: 3

# Define the 'livenessProbe' rules for the Scheduler Pod
livenessProbe:
  # This specifies the type of probe to use, which is an HTTP GET request
  # It checks the container's health by sending an HTTP request to a specific endpoint
  httpGet:
    # This indicates that the port for the HTTP GET request is taken from a variable named httpPort (previously defined)
    port: *httpPort
  # This property specifies how long the probe should wait for a response from the container before considering the probe as failed
  # It's set to 3 seconds in this case
  timeoutSeconds: 3
  # This property sets the number of consecutive probe failures required to consider the container as unhealthy
  # In this case, it's set to 24 failures
  failureThreshold: 24
  # This property sets the number of consecutive successful probes required to consider the container as healthy again after it has been marked as unhealthy
  # It's set to 1 success in this case
  successThreshold: 1
  # This property sets the number of seconds to wait before starting the first probe after the container starts
  # In this case, it's set to 500 seconds
  initialDelaySeconds: 500 # we give some time in order to let the scheduler start properly
  # This specifies a 5-second interval between successive probe executions
  periodSeconds: 5

Required configuration

You can use the default parameter values provided in the values.yaml file. However, the following specific properties must be configured as outlined below:

target.cluster: The target Kubernetes cluster for the installation. Possible values: aks or on-prem.
imageCredentials: Contains the required credentials to be able to pull the ProActive images from the private Activeeon registry.
images:
- scheduler.repository: A namespace of the ProActive scheduler image located in the Activeeon private registry.
- scheduler.tag: The ProActive scheduler image version or tag.
- node.repository: A namespace of the ProActive node image located in the Activeeon private registry.
- node.tag: The ProActive node image version or tag.
- db.postgres.repository: A namespace of the ProActive Postgres image located in the Activeeon private registry.
- db.postgres.tag: The ProActive Postgres image version or tag.
To ensure having dynamic ProActive Kubernetes nodes after server startup, you need to copy and paste the target (aks or on-prem) cluster configuration into the files/kube.config file of the Helm installation package.

For an on-premises installation, the Kubernetes node name to host the ProActive scheduler and database pods:

persistentVolumes.localVolumeConfig:
- scheduler.persistentVolume.nodeAffinity.nodeName: Node name
- database.persistentVolume.nodeAffinity.nodeName: Node name

For an on-premises installation, a directory must be created beforehand on the node defined by the nodeAffinity (see nodeAffinity above) to host the ProActive scheduler and database pods' data:

persistentVolumes.localVolumeConfig:
- scheduler.persistentVolume.localPath: Absolute path to the empty directory created on the node
- database.persistentVolume.localPath: Absolute path to the empty directory created on the node

Advanced configuration

In addition to the required parameters, you can set more advanced parameters to your Helm chart. Many of the advanced parameters are already defined in Docker Compose advanced parameters section.

We will enumerate below few (but not all) important parameters that could be useful to customize your Kubernetes installation:

namespace: Namespace to host all ProActive installation resources. By default, ae-dev.
ports:
- httpPort: Http port. By default, 8080.
- nodePort: For on-premises installation, you can customize the Node Port. By default, 32000.
database: You can enable or not using a separate Postgres pod for the database. By default, true.
usersConfiguration: For more details, see section PWS Users Configuration Properties.
ldapConfiguration: For more details, see section LDAP Configuration Properties.
nodeSources:
- dynamicK8s: To enable or not having dynamic ProActive Kubernetes nodes. A minimum (minNodes) and maximum (maxNodes) number of nodes have to be defined. By default, it is enabled.
- staticLocal: To enable or not having static ProActive Kubernetes nodes. A static number of nodes (numberNodes) has to be defined. By default, it is enabled.
persistentVolumes:
- localVolumeConfig: Set the local node volumes settings according to your needs if you are targeting on-premises clusters. By default, local node spaces are provisioned.
- aksDiskVolumeConfig: Set the Azure disk settings according to your needs if you are targeting AKS clusters. By default, Azure disks are provisioned for the scheduler and the database.
resources: Set the resources requests for the scheduler, node, and database pods. By default, minimal settings are defined.
To use https with TLS encryption, enable the use of ingress service. Note that you must first install or enable the Ingress and Cert Manager controllers in your cluster (refer to the commands below). Then, within the same configuration block, enable and configure one of the three TLS certificates to be used: either a provided, a self-signed, or a Let’s Encrypt certificate:
ingress:
- enabled: Set to true to enable the ingress service. By default, false.
- ingressClassName: ingress class. Use kubectl get ingressclass to get the ingress class.
- hostname: hostname of the machine running the ProActive scheduler pod.
- providedCertificate.enabled: Set to true if you provide your own certificate.
  - providedCertificate.tls.crt: The actual content of the TLS certificate file (encoded in Base64). Use for example cat fullchain.pem | base64 to get the encrypted certificate file content.
  - providedCertificate.tls.key: The actual content of the TLS private key file (encoded in Base64). Use for example cat privkey.pem | base64 to get the encrypted private key file content.
- clusterIssuer.selfSigned.enabled: Set to true if you want to use a self-signed certificate.
- clusterIssuer.letsEncrypt.enabled: Set to true if you want to use a valid Let’s Encrypt certificate.
  - clusterIssuer.letsEncrypt.server: The Let’s Encrypt ACME server URL.
  - clusterIssuer.letsEncrypt.email: The email address to be associated with the Let’s Encrypt account.

Set ingress controller:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.0/deploy/static/provider/cloud/deploy.yaml

Set Cert manager controller:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.3/cert-manager.yaml

2.5.3. Start ProActive Using Helm

After setting the right configuration options in the values.yaml file, you can access the Helm installation package and then proceed to install the chart:

$ helm install pa-helm-chart .

After installing the Chart, all ProActive pods are deployed. Using the default configuration, you can monitor pods creation as follows:

$ kubectl get pod -n ae-dev
NAME                                                              READY   STATUS             RESTARTS   AGE
proactive-db-6f886f98f-cdcvs                                      1/1     Running            0          5h34m
proactive-deployment-657b5dc546-sv4v5                             1/1     Running            0          5h19m
dynamic-kube-linux-nodes-node-d4ka127srv-deployment-64bd6dklnj9   2/2     Running            0          5h15m
dynamic-kube-linux-nodes-node-kmrice1asv-deployment-59c87584vtv   2/2     Running            0          5h15m
dynamic-kube-linux-nodes-node-wrbhjijggy-deployment-7f49c4kw9rt   2/2     Running            0          5h15m
dynamic-kube-linux-nodes-node-0pgisrxfqd-deployment-fc6dc5l8jwn   2/2     Running            0          5h15m
node-deployment-74f979d95c-txd8w                                  2/2     Running            0          5h34m

To monitor ProActive server logs, use:

$ kubectl logs -f -n ae-dev proactive-deployment-657b5dc546-sv4v5

You can access the ProActive Web Portals, for example:

On on-premises installation:

http://on-prem-cluster-ip-or-dns:32000

(32000 is the default NodePort)

On AKS installation:

http://aks-cluster-load-balancer-ip-or-dns:8080

The user can access any of the four portals using the default credentials: admin/<proactiveAdminPassword>. proactiveAdminPassword is defined in the values.yaml input file.

2.5.4. Database Purge Operation in Kubernetes

The Postgres database purge process is required on any fresh ProActive installation using the Postgres database, and it explained in section: Docker Compose Purge Operation.

The following commands allow to prepare the database for the automatic purge operations performed within the Postgres database pod:

Delete the ProActive scheduler deployment in the namespace that was given to the values.yaml file. By default, ae-dev is used:

$ kubectl delete deploy -n ae-dev proactive-deployment

Run the embedded database pod script for table configuration. You need to replace <proactive-db-id> with your current ProActive database pod id:

$ kubectl exec -it -n ae-dev <proactive-db-id> -- /tables-configuration.sh

Start the Helm chart again:

$ helm upgrade pa-helm-chart . -f values.yaml

The installation of ProActive using Helm is now over ! Your ProActive Scheduler is now running as a set of pods !

2.5.5. Upgrade ProActive Using Helm

The process of upgrading ProActive in a containerized environment is described in section Upgrade/Downgrade/Backup ProActive Using Docker Compose.

In this section, we will see how to upgrade ProActive in Kubernetes using Helm.

First, the new release version has to be provided in the values.yaml file:

images:
- scheduler.tag: The new ProActive scheduler image version or tag.
- node.tag: The new ProActive node image version or tag.

Once set, we upgrade the installed Helm release using the following command:

$ helm upgrade pa-helm-chart . -f values.yaml

The Scheduler and node pods will be recreated to download the new images and apply the upgrade progress during their startup.

2.6. Automated ProActive Installation with Shell Scripts

ProActive could be easily deployed and started automatically using Shell scripts. For this purpose, we provide a ready-to-use Shell installer that enables you running ProActive Workflows and Scheduling (PWS) as a Linux service.

Here are the installation steps you need to follow in order to have ProActive running in your environment:

2.6.1. Download the Shell Installation Package

We provide public access to the ProActive Shell installation package that you can easily use to configure and start ProActive as a service in your environment.

The installation package contains 4 files:

The inputConfig.json file that includes the required parameters needed for the configuration tuning of the ProActive server. It is formatted as a json file.
The proactiveInstaller.sh file, which is the main installation script. It parses the configuration provided in the inputConfig.json file, proceeds to the configuration, and finally starts up the ProActive server.
The password-template.properties file contains the clear-text passwords required for installation, which will be encrypted during the process. This file is used only for a specific installation method (more details can be found here: Starting ProActive using Shell scripts).
The apiHealthcheck.sh, called by the proactiveInstaller.sh (at the end), to run REST API health check tests to verify the correct behavior of the ProActive server after its startup.
The installPostgres folder, which contains the required scripts to install and initialize the Postgres database depending on the underlying Linux distribution (Debian and Redhat9).
The lib folder, containing the libraries required for the installation (e.g., postgresql jar driver).

You can proceed to download the Shell installation package using the following command:

In order to install the ProActive server on your environment:

You need first to configure the input parameters defined in the inputConfig.json (to better know how to configure the parameters, see next section: Configure the installation input file)
Run the proactiveInstaller.sh installation script (see section Starting ProActive using Shell scripts)

2.6.2. Configure the Installation Input File

The inputConfig.json file, included in the installation package, is necessary to provide the specific configuration values for installing ProActive.

This section explains the configuration parameters required, as specified in the inputConfig.json file.

Click here to display the configuration input file

inputConfig.json:

{
  "INSTALLATION_CONFIGURATION": {
    "ARCHIVE_LOCATION": "<http_url_or_absolute_path_to_proactive_archive>",
    "NEW_INSTALL_OR_UPGRADE": "true",
    "LIBRARIES": "rsync,wget,curl,zip,unzip,sshpass,openssl,passwd,vim,git,cron",
    "INSTALLATION_DIRECTORY": "/opt/proactive"
  },
  "HEALTH_CHECK_CONFIGURATION": {
    "ENABLE_POST_INSTALLATION_HEALTH_CHECK": "true",
    "RUN_HEALTH_CHECK_ONLY": "false"
  },
  "PWS_SERVER_CONFIGURATION": {
    "PWS_HOST_ADDRESS": "<internal_host_address>",
    "PWS_PROTOCOL": "http",
    "PWS_PORT": "8080",
    "PWS_PAMR_PORT": "33647",
    "PWS_JOB_CLEANUP_DAYS": "60",
    "PWS_USER_NAME": "activeeon",
    "PWS_USER_GROUP_NAME": "activeeon",
    "PWS_USER_PASSWORD": "",
    "PWS_LOGIN_ADMIN_PASSWORD": "",
    "PWS_WORKER_NODES": "4"
  },
  "PUBLIC_ACCESS_CONFIGURATION": {
    "ENABLE_PUBLIC_ACCESS": "false",
    "PWS_PUBLIC_HOSTNAME": "<public_host_address>",
    "PWS_PUBLIC_SCHEME": "http",
    "PWS_PUBLIC_PORT": "8080"
  },
  "TLS_CERTIFICATE_CONFIGURATION": {
    "ENABLE_TLS_CERTIFICATE_CONFIGURATION": "false",
    "TLS_CERTIFICATE_PATH": "<absolute_path_to_fullchain.pem>",
    "TLS_CERTIFICATE_KEY_PATH": "<absolute_path_to_privkey.pem>",
    "TLS_CERTIFICATE_KEYSTORE_PASSWORD": ""
  },
  "LDAP_CONFIGURATION": {
    "ENABLE_LDAP_AUTHENTICATION": "false",
    "LDAP_SERVER_URLS": "<ldap_server_urls>",
    "LDAP_USER_SUBTREE": "<ldap_user_subtree>",
    "LDAP_GROUP_SUBTREE": "<ldap_group_subtree>",
    "LDAP_BIND_LOGIN": "<ldap_bind_login>",
    "LDAP_BIND_PASSWORD": "",
    "LDAP_USER_FILTER": "<ldap_user_filter>",
    "LDAP_GROUP_FILTER": "<ldap_group_filter>",
    "LDAP_TEST_LOGIN": "<ldap_test_login>",
    "LDAP_TEST_PASSWORD": "",
    "LDAP_ADMIN_ROLES": "<ldap_admin_roles>",
    "LDAP_USER_ROLES": "<ldap_user_roles>"
  },
  "PWS_USERS_CONFIGURATION": {
    "ENABLE_USERS_CONFIGURATION": "false",
    "PWS_USERS": [
      {
        "LOGIN": "test_admin",
        "PASSWORD": "",
        "GROUPS": "scheduleradmins,rmcoreadmins"
      },
      {
        "LOGIN": "test_user",
        "PASSWORD": "",
        "GROUPS": "user"
      }
    ]
  },
  "EXTERNAL_DATABASE_CONFIGURATION": {
    "ENABLE_EXTERNAL_DATABASE_CONFIGURATION": "false",
    "DB_INSTALL_POSTGRES": "false",
    "DB_TYPE": "postgresql",
    "DB_DRIVER_LOCATION": "lib/postgresql-42.7.2.jar",
    "DB_APPLY_SCHEDULER_CONFIG_TO_ALL": "false",
    "DB_SCHEDULER_CONFIG": {
      "DB_SCHEDULER_HOSTNAME": "localhost",
      "DB_SCHEDULER_PORT": "5432",
      "DB_SCHEDULER_USERNAME": "scheduler",
      "DB_SCHEDULER_PASSWORD": "",
      "DB_SCHEDULER_DIALECT": "default",
      "DB_SCHEDULER_SCHEMA_NAME": "default",
      "DB_SCHEDULER_URL": "default"
    },
    "DB_RM_CONFIG": {
      "DB_RM_HOSTNAME": "localhost",
      "DB_RM_PORT": "5432",
      "DB_RM_USERNAME": "rm",
      "DB_RM_PASSWORD": "",
      "DB_RM_DIALECT": "default",
      "DB_RM_SCHEMA_NAME": "default",
      "DB_RM_URL": "default"
    },
    "DB_CATALOG_CONFIG": {
      "DB_CATALOG_HOSTNAME": "localhost",
      "DB_CATALOG_PORT": "5432",
      "DB_CATALOG_USERNAME": "catalog",
      "DB_CATALOG_PASSWORD": "",
      "DB_CATALOG_DIALECT": "default",
      "DB_CATALOG_SCHEMA_NAME": "default",
      "DB_CATALOG_URL": "default"
    },
    "DB_PSA_CONFIG": {
      "DB_PSA_HOSTNAME": "localhost",
      "DB_PSA_PORT": "5432",
      "DB_PSA_USERNAME": "pca",
      "DB_PSA_PASSWORD": "",
      "DB_PSA_DIALECT": "default",
      "DB_PSA_SCHEMA_NAME": "default",
      "DB_PSA_URL": "default"
    },
    "DB_NOTIFICATION_CONFIG": {
      "DB_NOTIFICATION_HOSTNAME": "localhost",
      "DB_NOTIFICATION_PORT": "5432",
      "DB_NOTIFICATION_USERNAME": "notification",
      "DB_NOTIFICATION_PASSWORD": "",
      "DB_NOTIFICATION_DIALECT": "default",
      "DB_NOTIFICATION_SCHEMA_NAME": "default",
      "DB_NOTIFICATION_URL": "default"
    }
  }
}

Required configuration

You can the use default parameter values as they are given in the inputConfig.json file. However, the following specific properties must be configured as described below:

INSTALLATION_CONFIGURATION
- ARCHIVE_LOCATION: The archive location could be an absolute file path of the ProActive archive or an http(s) link.
PWS_SERVER_CONFIGURATION
- PWS_HOST_ADDRESS: Defines the internal hostname or IP address of the machine where ProActive will be installed.

Advanced configuration

In this section, we describe the advanced configuration of the ProActive server.

Important: Please note that all password fields defined in the configuration sections listed below should be left empty. Their values will be provided at runtime. They will be then encrypted and inserted back into the inputConfig.json file. We provide a more detailed explanation of how to configure the password values in the next section Starting ProActive using Shell scripts.

INSTALLATION_CONFIGURATION
- NEW_INSTALL_OR_UPGRADE: Set to true when it is a first ProActive server installation or an installation upgrade.
- LIBRARIES: The libraries to be installed on the ProActive server machine. The proactiveInstaller.sh script will install them. Default libraries mentioned are necessary for the installation.
- INSTALLATION_DIRECTORY: Path of the folder where ProActive will be installed. You can change the path according to your preference. This path will be created by the Shell installer. No need to create it in advance.
HEALTH_CHECK_CONFIGURATION
- ENABLE_POST_INSTALLATION_HEALTH_CHECK: If enabled, the health check tests will be executed at the end of the proactiveInstaller.sh installation script. It is recommended to enable the health check tests.
- RUN_HEALTH_CHECK_ONLY: If enabled, only health check tests will be executed. It supposes that the installation is already done and that the ProActive server is already running. It is recommended to not activate it unless you would like to run the health checks separately.
PWS_SERVER_CONFIGURATION

See details of PWS_SERVER_CONFIGURATION in this section: PWS Server Configuration Properties.

PUBLIC_ACCESS_CONFIGURATION

See details of PUBLIC_ACCESS_CONFIGURATION in this section: Public Access Configuration Properties.

TLS_CERTIFICATE_CONFIGURATION

See details of TLS_CERTIFICATE_CONFIGURATION in this section: TLS Certificate Configuration Properties.

The pem/crt/pfx certificate as well as the key have to be provided with their absolute paths. No need to place them in the backup folder.

LDAP_CONFIGURATION

See details of LDAP_CONFIGURATION in this section: LDAP Configuration Properties.

PWS_USERS_CONFIGURATION

See details of PWS_USERS_CONFIGURATION in this section: PWS Users Configuration Properties.

EXTERNAL_DATABASE_CONFIGURATION

This section allows you to change the ProActive server database configuration in order to connect to an external database. If set to false, it will use the embedded HSQL database. ProActive requires the following components to be mapped to different database schemas: Resource Manager, Scheduler, Catalog, Service Automation and Notification Service.

EXTERNAL_DATABASE_CONFIGURATION
- ENABLE_EXTERNAL_DATABASE_CONFIGURATION: Set to true if you want to connect to an external Postres database.
- DB_INSTALL_POSTGRES: Set to true if you want to install the Postgres database.
  - The supported operating systems for Postgres database installation are Debian and Redhat9. Postgres 12 will be installed on Debian distributions, and Postgres 13 will be installed on Redhat9 distributions.
  - The setup and initialization of the Postgres database are automatically performed based on the current operating system.
- DB_TYPE: The external database to be used by ProActive, namely postgresql.
- DB_DRIVER_LOCATION: The Postgres driver location. It is recommended to keep the default value.
- DB_APPLY_SCHEDULER_CONFIG_TO_ALL: If set to true, the 5 ProActive components will be configured using the Scheduler DB configuration described below.
- DB_SCHEDULER_CONFIG: Configuration of the 'scheduler' component database
  - DB_SCHEDULER_HOSTNAME: Database hostname used (e.g. localhost, myserver) or IP address (e.g. 127.0.0.1, 192.168.12.1).
  - DB_SCHEDULER_PORT: Database port used (e.g. 9001, 5432).
  - DB_SCHEDULER_USERNAME: Database user name.
  - DB_SCHEDULER_PASSWORD: Database user encrypted password. This value should be left empty.
  - DB_SCHEDULER_DIALECT: Override default database vendor dialect. Defaults are: {hsqldb=org.hibernate.dialect.HSQLDialect, postgresql=org.hibernate.dialect.PostgreSQL94Dialect}.
  - DB_SCHEDULER_SCHEMA_NAME: Override the default database schema name for the selected component. Defaults names are { Resource Manager : "rm", Scheduler : "scheduler", Catalog : "catalog", Service Automation : "pca", Notification Service : "notification" }.
  - DB_SCHEDULER_URL: Sets the JDBC url that will be used to access the target component database. If set to default, the url will be inferred from --hostname, --port and optionally --schema-name options.

The description of the configuration options for the remaining ProActive components (DB_RM_CONFIG, DB_CATALOG_CONFIG, DB_PSA_CONFIG, DB_NOTIFICATION_CONFIG) is the same as described for the scheduler database above.

2.6.3. Starting ProActive Using Shell Scripts

After configuring the necessary settings as described in the previous section, you are ready to begin the installation of the ProActive server using the proactiveInstaller.sh script.

The proactiveInstaller.sh script simplifies the installation and configuration process by automating key tasks such as password encryption and offering flexibility through offline and upgrade modes.

Before exploring the installation options, it is important to note that proactiveInstaller.sh must be executed with root privileges, and you need to grant it execution permission:

$ sudo chmod +x proactiveInstaller.sh

To get an overview of all available options and some examples of installation commands, you can access the help menu by running:

$ sudo ./proactiveInstaller.sh -h

This script supports three different modes for password encryption: via command-line arguments, interactive, and automatic (using a password file). It ensures that you can tailor the installation method based on your environment and security requirements.

The help menu provides a detailed description of the available options:

CLI Arguments Mode (-p passphrase -admin_pwd admin_password -user_pwd user_password): Allows you to pass the passphrase and passwords directly as command-line arguments for minimal installation setups.
Interactive Mode (-i): Prompts you to enter passwords and passphrases interactively, followed by automatic installation.
Automatic Mode (-a -f file_path): Uses a predefined file with clear-text passwords for installation, ideal for automated deployments.

Additional options include:

Offline Mode (-o): Ensures the installation process can be completed without internet access.
Upgrade Mode (-u): Allows upgrading an existing ProActive installation.

The options -o and -u can be used with any installation mode (-i, -a, or -p).

Below is the help output that showcases the available options and example commands:

$ sudo ./proactiveInstaller.sh -h
 This script automates the installation and configuration process for ProActive, including password encryption and optional offline or upgrade modes.
 The script supports 3 modes for passwords encryption: interactive, automatic, and CLI arguments.

 Usage: ./proactiveInstaller.sh [-i | -a -f file_path | -h | -p passphrase [-admin_pwd admin_password -user_pwd user_password] | -u] [-o]
 (Only one of the following installation methods can be used at a time: -i, -a, -p, or -h):
   -i              Proceed with the interactive password encryption and then automatic installation mode
   -a              Proceed with the automatic installation mode (requires -f)
      -f file_path Specify the clear passwords file path to be used in automatic installation mode
   -h              Display this help message
   -p passphrase   Specify the passphrase for password encryption. Without -u, the following passwords are required:
      -admin_pwd   ProActive login admin password (mandatory unless -u is used)
      -user_pwd    ProActive user password (mandatory unless -u is used)
   -o              (optional) Enable offline mode. Can be used with -i, -a, or -p
   -u              (optional) Enable upgrade mode. Can be used with -i, -a, or -p and makes -admin_pwd and -user_pwd optional

 Examples:
   sudo ./proactiveInstaller.sh -i
   sudo ./proactiveInstaller.sh -i -o
   sudo ./proactiveInstaller.sh -i -u
   sudo ./proactiveInstaller.sh -i -o -u
   sudo ./proactiveInstaller.sh -a -f ./password-template.properties
   sudo ./proactiveInstaller.sh -a -f ./password-template.properties -o
   sudo ./proactiveInstaller.sh -a -f ./password-template.properties -u
   sudo ./proactiveInstaller.sh -a -f ./password-template.properties -o -u
   sudo ./proactiveInstaller.sh -p 1234 -admin_pwd adminPass123 -user_pwd userPass456
   sudo ./proactiveInstaller.sh -p 1234 -admin_pwd adminPass123 -user_pwd userPass456 -o
   sudo ./proactiveInstaller.sh -p 1234 -u
   sudo ./proactiveInstaller.sh -p 1234 -u -o

In the following sections, we will present the three methods you can use to install the ProActive server:

CLI Arguments Mode: Passwords provided directly through command-line arguments for a minimal and non-interactive installation.
Interactive Mode: Step-by-step guidance for users to input passwords.
Automatic Mode: Installation using a configuration file with predefined passwords.

ProActive server installation Using CLI Password Arguments

This method provides a simple and minimal installation of the ProActive server. It does not support advanced ProActive configurations such as LDAP, additional Users, TLS, or database setup.

It installs the required configuration of the ProActive server (with embedded database) by passing passwords as command-line arguments. This allows you to supply the required credentials directly via the command line, avoiding the need for interactive password input during installation.

You need to set all advanced configurations such as LDAP, additional Users, TLS, or database setup to false since they are not supported during this installation mode.

During this mode, you must provide a valid passphrase for password encryption using the -p option, followed by the two required passwords for a basic installation: the admin login password (-admin_pwd) and the user password (-user_pwd).

The ProActive installer will use the provided passphrase to encrypt the passwords and insert them, in encrypted format, into the inputConfig.json file.

Example:

$ sudo ./proactiveInstaller.sh -p MySecurePassphrase -admin_pwd AdminPass123 -user_pwd UserPass456

 CLI installation mode is set.
 Running in CLI installation mode.
...

If you want to start the ProActive server installation and configuration offline (without requiring internet access), you can include the -o option in the installation command to ensure it runs in offline mode:

$ sudo ./proactiveInstaller.sh -p MySecurePassphrase -admin_pwd AdminPass123 -user_pwd UserPass456 -o

It is important to ensure that the command-line interface where you provide the passwords is secure, as passwords may be visible in the process list.

Ensure that the passphrase used is strong and secure since it will be used to encrypt sensitive passwords.

ProActive Server Installation with Interactive Password Input Mode

This method offers an interactive way to install the ProActive server, guiding the user step-by-step through the process of entering the necessary passwords.

To use this mode, specify the installation option -i for interactive mode, after which the installer will prompt you to enter the passphrase and the required passwords. It supports providing passwords for advanced installation such as LDAP, Users, TLS, or database setup.

This approach is ideal for users who prefer a hands-on installation experience and want to avoid passing sensitive information directly through the command line, thereby enhancing security.

Once provided, the installer will encrypt the passwords using the passphrase and store them in the inputConfig.json file in encrypted format.

Example:

$ sudo ./proactiveInstaller.sh -i

 Interactive installation mode is set.
 Running in interactive installation mode.
You are going to set the passphrase for password encryption [y/n] y
Enter Passphrase:
Retype Passphrase:
 Passphrase is set.
Do you want to set ProActive admin password? [y/n] y
Enter ProActive admin password:
Retype ProActive admin password:
 ProActive admin password is set.
Do you want to set ProActive user password? [y/n] y
Enter ProActive user password:
Retype ProActive user password:
 ProActive user password is set.
Do you want to set the TLS certificate keystore password? [y/n] n
Do you want to set LDAP configuration passwords? [y/n] n
Do you want to set ProActive users passwords? [y/n] n
Do you want to set ProActive external database passwords? [y/n] n
...

$ sudo ./proactiveInstaller.sh -i -o

ProActive Server Installation with Passwords from a File

This method allows for an automated installation of the ProActive server by using a file to provide the necessary clear-text passwords.

It is ideal for users who want to simplify the installation process by pre-defining passwords in a configuration file. It is suitable for users who prefer to avoid manual input during installation, making it useful for automation.

During this mode, you specify the installation mode using -a option (for automatic) and the path to a clear-text password file using the -f option (e.g., -f ./password-template.properties) that contains the clear-text passwords for various configurations, such as admin, user, LDAP, TLS, or database passwords.

A pre-filled password-template.properties file is provided in the installation package, and you can use it to provide the clear-text passwords. If some advanced configurations are disabled in the inputConfig.json file (e.g., LDAP or database configuration are disabled), the relative passwords will be simply ignored.

The installer will then use the provided passphrase in the password file to encrypt the passwords and insert them into the inputConfig.json file.

Example:

$ sudo ./proactiveInstaller.sh -a -f ./password-template.properties

 Automatic installation mode is set.
 Using file: ./password-template.properties for automatic installation.
 Running in automatic installation mode.
...

$ sudo ./proactiveInstaller.sh -a -f ./password-template.properties -o

After the installation is complete, it is recommended to delete all clear-text passwords to improve security. In case you want to proceed with an installation upgrade (-u), the PASSPHRASE value will be required.

Server monitoring and troubleshooting

While running the installation script, the script will continuously display logs about the in-progress configuration and installation. When the server is starting, you can access the ProActive service logs located at /var/log/proactive/scheduler to monitor the ProActive server startup status:

$ tail -f /var/log/proactive/scheduler

For advanced troubleshooting, you can access the detailed server logs located at the ProActive installation directory INSTALLATION_DIRECTORY/default/logs/Scheduler.log in order to investigate or report any issues. The INSTALLATION_DIRECTORY is set in the inputConfig.json file, default is /opt/proactive.

$ tail -f INSTALLATION_DIRECTORY/default/logs/Scheduler.log

Wait until the logs show:

....
Loading workflow and utility packages...
Packages successfully loaded

ProActive started in 246.476727488 seconds

ProActive web portals access

When the ProActive server is up, you can access the ProActive Web portals at:

If you are using a public address: PWS_PUBLIC_SCHEME://PWS_PUBLIC_HOSTNAME:PWS_PUBLIC_PORT

If you are using a private address only: PWS_PROTOCOL://PWS_HOST_ADDRESS:PWS_PORT

The login is admin and the password is given in the inputConfig.json file referenced as PROACTIVE_ADMIN_PASSWORD. You can also use the internal ProActive users configured in the input file if you have activated ENABLE_USERS_CONFIGURATION property (same, for the LDAP configuration).

Manage the ProActive Linux service lifecycle

When the server is up and running, it will start as a Linux service. Hence, you can manage its lifecycle using these basic commands:

# status of the ProActive Linux service
$ systemctl status proactive-scheduler

# stop the ProActive Linux service
$ systemctl stop proactive-scheduler

# start the ProActive Linux service
$ systemctl start proactive-scheduler

2.6.4. Postgres Database Purge Operation

The Postgres database purge process is required on any fresh ProActive installation using the Postgres database, and it explained in section: Docker Compose Purge Operation. If you are not using a Postgres database in your ProActive installation, please ignore this section.

The following commands allow to prepare the database for the purge operations to be performed within the Postgres database.

Provide execution permissions to the database purge scripts (provided in the installation package):

$ sudo chmod +x ./installPostgres/purge/tables-configuration.sh ./installPostgres/purge/purge-operation.sh

Stop the ProActive server:

$ sudo systemctl stop proactive-scheduler

While the ProActive server is stopped and the Postgres server is running, run the following script in order to alter the type of certain columns in several database tables (this is needed due to a PostgresSQL driver issue):

$ sudo ./installPostgres/purge/tables-configuration.sh

Start the ProActive server again:

$ sudo systemctl start proactive-scheduler

Now, while the ProActive server is running, you can use the given script purge-operation.sh to execute a full vacuum regularly using a cron expression for example:

$ sudo crontab -e

Add the cron expression to run the purge operations:

* 2 * * * ABSOLUTE_PATH_TO/installPostgres/purge/purge-operation.sh

This will run the purge operations every day at 2am as an example. You can configure it as you prefer.

The installation of ProActive is now over !

2.6.5. Upgrade ProActive Using Shell Scripts

The process of upgrading the ProActive server is described in this section: Upgrade/Downgrade/Backup ProActive Using Docker Compose.

In this section, we will see how to upgrade the ProActive server using the provided Shell scripts.

First, a link to the new ProActive release has to be provided in the inputConfig.json file:

"ARCHIVE_LOCATION": Replace the old archive link with a new one.

Once set, we upgrade the installed version using one of the following commands depending on the installation method:

For ProActive installation using CLI password arguments:

$ sudo ./proactiveInstaller.sh -p MySecurePassphrase -u

For ProActive installation with interactive password input mode:

$ sudo ./proactiveInstaller.sh -i -u

For ProActive installation with passwords from a file:

$ sudo ./proactiveInstaller.sh -a -f ./password-template.properties -u

Important: Make sure to use the same passphrase as given during the installation phase.

Note: You can apply the upgrade in offline mode using the -o option together with -u option.

The new version will be automatically detected, and then the upgrade process will be triggered.

2.7. Scheduler Life-Cycle Management

Under the Admin menu from the scheduler portal, the following administrator actions are proposed:

Start the scheduler: Jobs can be submitted. Getting the jobs results is possible. The scheduler can be stopped, frozen, paused or killed.
Stop the scheduler: Jobs cannot be submitted anymore. Already running jobs run to completion, but not pending jobs. Getting the jobs results is possible. The scheduler can be started or killed.
Freeze the scheduler: Every running tasks run to completion, but the running jobs wait for the scheduler to resume. The scheduler can be stopped, resumed or killed.
Resume the scheduler: If the scheduler is Paused, Stopped or Frozen.
Pause the scheduler: Every running jobs run to completion. The scheduler can be stopped, resumed or killed.
Kill the scheduler: Similar to Ctrl-C. No interaction can be done anymore.
Shutdown the scheduler: Every running tasks run to completion before terminating the scheduler.

3. ProActive Scheduler configuration

All configuration files of ProActive Scheduler can be found under PROACTIVE_HOME.

3.1. Java Virtual Machine configuration

Various command-line tools shipped with ProActive (bin/proactive-server, bin/proactive-node) start a Java Virtual Machine. The parameters of the JVM can be modified by editing DEFAULT_JVM_OPTS= in the corresponding script both in Linux and Windows.

For example, to set the maximum heap capacity on the JVM to 6GB in Linux:

Change the line DEFAULT_JVM_OPTS='"-server" "-Dfile.encoding=UTF-8" "-Xms4g"'

into

DEFAULT_JVM_OPTS='"-server" "-Dfile.encoding=UTF-8" "-Xms4g" "-Xmx6g"'

3.2. General configuration

Table 1. ProActive Scheduler configuration files
Component	Description	File	Reference
Scheduler	Scheduling Properties	config/scheduler/settings.ini	Scheduler Properties
Resource Manager	Node management configuration	config/rm/settings.ini	Resources Manager Properties
Web Applications	REST API and Web Applications configuration	config/web/settings.ini	REST API & Web Properties
Networking	Network, firewall, protocols configuration	config/network/node.ini, config/network/server.ini	Network Properties
Security	User logins and passwords	config/authentication/login.cfg	File authentication
	User group assignments	config/authentication/group.cfg	File authentication
	User permissions	config/security.java.policy-server	User Permissions
	LDAP configuration	config/authentication/ldap.cfg	LDAP authentication
Keycloak configuration	config/authentication/keycloak.cfg	Keycloak-based access control	Database
Scheduler configuration	config/scheduler/database.properties	Database configuration
Resource Manager configuration	config/rm/database.properties	Database configuration
Scheduling-api microservice	dist/war/scheduling-api/WEB-INF/classes/application.properties	Database configuration
Job-planner microservice	dist/war/job-planner/WEB-INF/classes/application.properties	Database configuration
Catalog microservice	dist/war/catalog/WEB-INF/classes/application.properties	Database configuration

3.3. Database configuration

Scheduler, Resource Mananger, and some microservices (catalog, cloud-automation-service, notification-service) require to have direct access to the database. Thus, they must each have a correct configuration for each supported database (HSQLDB, Postgres, MySQL, Oracle).

To configure Scheduler or Resource Mananger, one have to modify PROACTIVE_HOME/config/scheduler/database.properties and PROACTIVE_HOME/config/rm/database.properties respectively. For microservices, you have also to modify files in: PROACTIVE_HOME/config/MICROSERVICE-NAME/database.properties.

The PROACTIVE_HOME/tools/configure-db command can be used to apply a database configuration. See Configure Database Command

For HSQLDB, the following five properties must be configured:

hibernate.connection.driver_class=org.hsqldb.jdbc.JDBCDriver

#DB_NAME could be scheduler, rm, or one of the microservices names
#9001 is the defaut DB port of the embedded PA database
hibernate.connection.url=jdbc:hsqldb:hsql://localhost:9001/DB_NAME

hibernate.dialect=org.hibernate.dialect.HSQLDialect

# Username and password
hibernate.connection.username=username
hibernate.connection.password=password

For Postgres 12.2, the following template must be configured:

hibernate.connection.driver_class=org.postgresql.Driver

#DB_NAME could be scheduler, rm, or one of the microservices names
#PORT should correspond to your DB port
hibernate.connection.url=jdbc:postgresql://localhost:PORT/DB_NAME

#PostgreSQLDialect is recommended for Postgres 12.2 version
hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect

# Username and password
hibernate.connection.username=username
hibernate.connection.password=password

For MySQL 8.0, the following template must be configured:

hibernate.connection.driver_class=com.mysql.jdbc.Driver

#DB_NAME could be scheduler, rm, or one of the microservices names
#PORT should correspond to your DB port
hibernate.connection.url=jdbc:mysql://localhost:PORT/DB_NAME

#MySQL5InnoDBDialect is recommended for MySQL 8.0 version
hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect

# Username and password
hibernate.connection.username=username
hibernate.connection.password=password

For Oracle 12.2, the following template must be configured:

hibernate.connection.driver_class=oracle.jdbc.driver.OracleDriver

#DB_NAME could be scheduler, rm, or one of the microservices names
#PORT should correspond to your DB port
hibernate.connection.url=jdbc:oracle:thin:@localhost:PORT:DB_NAME

#Oracle12cDialect is recommended for Oracle 12.2 version
hibernate.dialect=org.hibernate.dialect.Oracle12cDialect

# Username and password
hibernate.connection.username=username
hibernate.connection.password=password

The job-planner, catalog, scheduling-api, cloud-automation-service, notification-service and proactive-cloud-watch microservices must contain a database configuration. For each microservice, a configuration file can be found in: PROACTIVE_HOME/dist/war/MICROSERVICE-NAME/WEB-INF/classes/application.properties. In their application.properties file, you need to set the following five properties for the target database as configured just above:

spring.datasource.url
spring.datasource.username
spring.datasource.password
spring.datasource.driverClassName
spring.jpa.database-platform

Moreover, it is mandatory to put the proper database driver in PROACTIVE_HOME/addons folder.

For example:

For HSQLDB 2.4, the drive hsqldb-2.4.1.jar is already embedded in PROACTIVE_HOME/addons
For MySQL 8.0, you have to get mysql-connector-java-8.0.19.zip driver an place it in PROACTIVE_HOME/addons
For Postgres 12.2, you have to get postgresql-42.2.11.jar driver.
For Oracle 12.2, you have to get ojdbc8-19.3.0.0.jar driver.

If you plan to use another DB version, it is important to get the corresponding database driver version and set up the right dialect version

Moreover, it is also important to notice that the Scheduler and 3 microservices (scheduling-api, job-planner, and proactive-cloud-watch) have to point to the same scheduler database since they use that one.

3.3.1. Example configuration with PostgreSQL

Configuring ProActive with an external database such as PostgreSQL is recommended in a production environment. In the following, we show an example of PostgreSQL installation and configuration to be used along with ProActive.

PostgreSQL version

PostgreSQL versions 12.0 to 12.11 are fully compatible with ProActive. More recent versions of PostgreSQL have not been fully tested yet for compatibility. PostgreSQL server can be installed on the same machine as the ProActive server or on a separate machine.

Initialize the database

To initialize the database, some SQL queries should be executed in the PostgreSQL server machine. In order to do so, the following SQL queries can be defined in a SQL initialization script (default users passwords should be modified):

init.sql

-- Create users
CREATE USER rm WITH PASSWORD 'rm';
CREATE USER scheduler WITH PASSWORD 'scheduler';
CREATE USER catalog WITH PASSWORD 'catalog';
CREATE USER pca WITH PASSWORD 'pca';
CREATE USER notification WITH PASSWORD 'notification';

-- Creating empty databases
CREATE DATABASE rm OWNER rm;
CREATE DATABASE scheduler OWNER scheduler;
CREATE DATABASE catalog OWNER catalog;
CREATE DATABASE pca OWNER pca;
CREATE DATABASE notification OWNER notification;

-- Grant access
GRANT CONNECT ON DATABASE scheduler TO scheduler;
GRANT USAGE ON SCHEMA public TO scheduler;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO scheduler;
GRANT CONNECT ON DATABASE rm TO rm;
GRANT USAGE ON SCHEMA public TO rm;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO rm;
GRANT CONNECT ON DATABASE catalog TO catalog;
GRANT USAGE ON SCHEMA public TO catalog;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO catalog;
GRANT CONNECT ON DATABASE pca TO pca;
GRANT USAGE ON SCHEMA public TO pca;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO pca;
GRANT CONNECT ON DATABASE notification TO notification;
GRANT USAGE ON SCHEMA public TO notification;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO notification;

ALTER SYSTEM SET max_connections ='500';

When the PostgreSQL server is installed on a separate machine, the listen-addresses setting should be set to allow remote connections:

sudo vim /opt/postgres/data/postgresql.conf
listen-addresses = ‘IP_ADDRESS’

Then, execute the initialization script inside the PotsgreSQL database:

psql -f init.sql

Download the Postgres driver

The Postgres driver has to be downloaded and added inside the addons folder of ProActive (SCHEDULER_PATH/addons/).

The link to download a Postgres driver version compatible with Postgres 12.X is: https://jdbc.postgresql.org/download/postgresql-42.2.11.jar

Configure ProActive

To configure ProActive to connect to the Postgres DB, we need to edit the database configuration files as follows.

The database passwords should be modified according to your chosen passwords. Note that passwords can be encrypted in configuration files as explained in section Encrypting configuration passwords

#To easily get the configuration file names, you can execute the following command:
cd SCHEDULER_PATH/dist/war
find . -name ‘database.properties’

#Scheduler config (SCHEDULER_PATH/config/scheduler/database.properties)
hibernate.connection.driver_class=org.postgresql.Driver
hibernate.connection.url=jdbc:postgresql://PG_SERVER_DNS:5432/scheduler
hibernate.dialect=org.hibernate.dialect.PostgreSQL94Dialect
hibernate.connection.username=scheduler
hibernate.connection.password=scheduler

#RM config (SCHEDULER_PATH/config/rm/database.properties)
hibernate.connection.driver_class=org.postgresql.Driver
hibernate.connection.url=jdbc:postgresql://PG_SERVER_DNS:5432/rm
hibernate.dialect=org.hibernate.dialect.PostgreSQL94Dialect
hibernate.connection.username=rm
hibernate.connection.password=rm

#Catalog config (SCHEDULER_PATH/dist/war/catalog/WEB-INF/classes/application.properties)
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://PG_SERVER_DNS:5432/catalog
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL94Dialect
spring.datasource.username=catalog
spring.datasource.password=catalog

#Scheduling config (SCHEDULER_PATH/dist/war/scheduling-api/WEB-INF/classes/application.properties)
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://PG_SERVER_DNS:5432/scheduler
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL94Dialect
spring.datasource.username=scheduler
spring.datasource.password=scheduler

#Job planner config (SCHEDULER_PATH/dist/war/job-planner/WEB-INF/classes/application.properties)
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://PG_SERVER_DNS:5432/scheduler
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL94Dialect
spring.datasource.username=scheduler
spring.datasource.password=scheduler

#PCW config (SCHEDULER_PATH/dist/war/proactive-cloud-watch/WEB-INF/classes/application.properties)
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://PG_SERVER_DNS:5432/scheduler
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL94Dialect
spring.datasource.username=scheduler
spring.datasource.password=scheduler

#PCA config (SCHEDULER_PATH/dist/war/cloud-automation-service/WEB-INF/classes/application.properties)
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://PG_SERVER_DNS:5432/pca
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL94Dialect
spring.datasource.username=pca
spring.datasource.password=pca

#Notification config (SCHEDULER_PATH/dist/war/notification-service/WEB-INF/classes/application.properties)
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://PG_SERVER_DNS:5432/notification
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL94Dialect
spring.datasource.username=notification
spring.datasource.password=notification

Database purge operation

Additionally, a vacuum operation does not by default remove orphaned Large Objects from the database. Identification and removal of these objects is performed by the vacuumlo command.

As the ProActive database contains many large objects, it is essential to apply both vacuumlo and vacuum operations in order to control the disk space usage over time.

In addition, prior to execute the purge operation, a preliminary configuration step is necessary (this preliminary step is done once for a given ProActive installation).

In the following paragraphs, we describe the step by step procedure to perform the database purge operations:

Start the ProActive server to fill up the database

Initially, the ProActive server has to be started at least once to fill up the ProActive database.

Stop the ProActive server

Then, once the ProActive server is up and the web portals become accessible, you have to stop the ProActive server.

PotgreSQL database must remain running in order to perform the next operations.

Prepare the database purge operation

While the ProActive server is stopped and the Postgres server is running, the following SQL queries have to be executed in order to alter the type of certain columns in several database tables (this is needed due to a PotsgreSQL driver issue):

tables-configuration.sql

# rm database
\c rm
alter table nodesourcedata alter column infrastructuretype TYPE oid USING infrastructuretype::oid;
\q


# scheduler database
\c scheduler
alter table environment_modifier_data alter column value TYPE oid USING value::oid;
alter table job_data alter column input_space TYPE oid USING input_space::oid;
alter table job_data alter column out_space TYPE oid USING out_space::oid;
alter table job_data alter column global_space TYPE oid USING global_space::oid;
alter table job_data alter column user_space TYPE oid USING user_space::oid;
alter table job_data alter column description TYPE oid USING description::oid;
alter table job_data_variable alter column variable_value TYPE oid USING variable_value::oid;
alter table job_data_variable alter column variable_model TYPE oid USING variable_model::oid;
alter table job_data_variable alter column variable_description TYPE oid USING  variable_description::oid;
alter table script_data alter column script TYPE oid USING script::oid;
alter table script_data alter column url TYPE oid USING url::oid;
alter table selection_script_data alter column script TYPE oid USING script::oid;
alter table selection_script_data alter column url TYPE oid USING url::oid;
alter table task_data alter column description TYPE oid USING description::oid;
alter table task_data alter column java_home TYPE oid USING java_home::oid;
alter table task_data alter column working_dir TYPE oid USING working_dir::oid;
alter table task_data_variable alter column variable_value TYPE oid USING variable_value::oid;
alter table task_data_variable alter column variable_model TYPE oid USING variable_model::oid;
alter table task_data_variable alter column variable_description TYPE oid USING variable_description::oid;
alter table plannedworkflow alter column failure_message TYPE oid USING failure_message::oid;
alter table rule_error alter column error_message TYPE oid USING error_message::oid;
alter table submittedrule alter column failure_message TYPE oid USING failure_message::oid;
\q

# pca database
\c pca
alter table variable alter column name TYPE oid USING name::oid;
alter table variable alter column value TYPE oid USING value::oid;
\q

Restart the ProActive server

After editing the table columns as described above, you have to start again the ProActive server to take into account the new modifications made to the Postgres database.

Database purge operation

Now, while the ProActive server is running, it is possible to execute a full vacuum by using the following commands:

# rm database
vacuumlo -v rm
vacuumdb -f -v -d rm -t pg_largeobject

# scheduler database
vacuumlo -v scheduler
vacuumdb -f -v -d scheduler -t pg_largeobject

# pca database
vacuumlo -v pca
vacuumdb -f -v -d pca -t pg_largeobject

# check autovacuum
SHOW autovacuum;

The command vacuumdb is executed implicitly if AutoVacuum is enabled. The command vacuumlo consumes a lot of RAM and disk space depending on the number of large objects to be removed.

3.4. Data Spaces configuration

ProActive Data Spaces is an integrated file-transfer mechanism used by ProActive workflows.

Data Spaces are used to transfer files between the server and ProActive Nodes during workflows tasks execution.

ProActive Data Spaces implementation is based on Apache VFS.

The ProActive server defines and starts automatically the following data spaces:

A Global Space where anyone can read/write files.
An User Space which is a personal user data storage.

Global or User Spaces can be configured individually to use one of the following Data Space protocols.

3.4.1. ProActive protocol

This is the default protocol used by the Global and User Spaces. The ProActive Data Space protocol is able to transfer files either by:

a direct copy operation on the file system
a network copy operation through the currently configured ProActive Network Protocol

Accordingly, a ProActive Node connected to the ProActive server and sharing the same file system (e.g. NFS) will be able to transfer files directly through the file system. A ProActive Node connected to the server from a different environment (virtual machine, cloud instance, docker container, etc) will also be able to transfer files to/from the server thanks to the ProActive Data Space protocol.

The ProActive Data Space protocol will use the network protocol configured for the ProActive server (see Network Properties).

In order to configure the data spaces paths on the ProActive server, you can set the following properties in the PROACTIVE_HOME/config/scheduler/settings.ini file:

pa.scheduler.dataspace.defaultglobal.localpath=/path/to/global/space
pa.scheduler.dataspace.defaultuser.localpath=/path/to/user/space/root

The user space will use the path defined in pa.scheduler.dataspace.defaultuser.localpath as its root and create sub-folders according to each user login name.

When these properties are not configured, global and user spaces default locations are:

Global Space : PROACTIVE_HOME/data/defaultglobal
User Space : PROACTIVE_HOME/data/defaultuser

3.4.2. SFTP protocol

The SFTP protocol can be used instead of the ProActive protocol to deploy the User or Global spaces.

ProActive User Space mapped using SFTP has the following benefits:

It allows to map each User Space to the HOME directory of the user on the SFTP server.
It allows file impersonation : a file copied to the User Space will belong to the user system account and not to the ProActive server account.

It has the following constraints and limitations:

Every user’s credentials of the ProActive server must match system users credentials inside the SFTP server. This is typically achieved by enabling LDAP or PAM authentication on the ProActive server (see User Authentication).
Every file transfer will use the SFTP protocol when it’s configured. Note that SFTP protocol is less efficient than the ProActive protocol when transferring files onto and from the local file system.

The SFTP protocol can also be used for the Global Space. In that case, all file transfers will be accomplished by a single system account. Motivations to enable SFTP on the Global Space can be to access a folder:

on a remote server, reachable via SSH.
on the local file system, in case this folder belongs to a different user from the ProActive server user.

SFTP User Space configuration

In order to configure the SFTP protocol for the User Space, the following properties in the PROACTIVE_HOME/config/scheduler/settings.ini file must be defined:

pa.scheduler.dataspace.defaultuser.url=sftp://your-server
pa.scheduler.dataspace.defaultuser.impersonation=true

The user spaces will always be mapped to each user HOME directory on the SFTP server. In order to have more flexible mappings, consider using the VSFTP protocol.

It is also possible to configure the User Space without impersonation. In that case, all connections to the SFTP server will be done under a single user. In order to do this, PROACTIVE_HOME/config/scheduler/settings.ini will contain:

pa.scheduler.dataspace.defaultuser.url=sftp://username@your-server/path
pa.scheduler.dataspace.defaultuser.impersonation=false

See SFTP Global Space configuration for a detailed explanation of this configuration.

SFTP Global Space configuration

In order to configure the SFTP protocol for the Global Space, the following property must be set:

pa.scheduler.dataspace.defaultglobal.url=sftp://username:password@your-server/path

path refers to a folder path relative to the HOME directory of username.

It is also possible to use SSH public key authentication to avoid setting the password in the URL. In that case the configuration would look like:

pa.scheduler.dataspace.defaultglobal.url=sftp://username@your-server/path

An additional configuration will be needed to allow SFTP public key authentication:

In PROACTIVE_HOME/config/network/server.ini and PROACTIVE_HOME/config/network/node.ini, add the following property:
```
proactive.dataspaces.sftp.useSystemPrivateKey=/path/to/private-key
```
In each Node Source defined in the Resource Manager, you must ensure that the java property -Dproactive.dataspaces.sftp.useSystemPrivateKey=/path/to/private-key is set for all Nodes deployed by the Node Source (this configuration is available in the Node Source parameters).
Finally, you must ensure that:
- The private key used does not contain a passphrase.
- The associated public key is added to the authorized_keys of username on the SFTP server.
- The private key is physically present on each Node connected to the Resource Manager. The private key location on the Node’s file system must match the java property proactive.dataspaces.sftp.useSystemPrivateKey configuration for this Node (The private key location can differ from one Node to another).

3.4.3. VSFTP protocol

The VSFTP protocol is a proprietary extension inside ProActive of the SFTP protocol. It complements the SFTP protocol by adding the ability to configure SFTP root directories with a set of predefined environment variables.

By default, only the HOME environment variable is included. Other environment variables can be added using the java property proactive.vfsprovider.vsftp.var_names.

Each environment variable must represent a directory path inside the SFTP server. Each variable must contain a value for all users of the SFTP server.

For example, suppose that proactive.vfsprovider.vsftp.var_names=HOME,WORK,SCRATCH and that these environment variables are resolved for user1 as follows:

HOME=/home/user1
WORK=/work
SCRATCH=/scratch/user1

The VSFTP protocol will then be able to resolve inside the SFTP server the following urls:

vsftp://your-server/$HOME/file1 => /home/user1/file1
vsftp://your-server/$WORK/file2 => /work/file2
vsftp://your-server/$SCRATCH/file3 => /scratch/user1/file3

Additionally, when listing the contents of the server root directory, it will return a list corresponding to the configured variables:

LIST vsftp://your-server/ => [$HOME, $WORK, $SCRATCH]

proactive.vfsprovider.vsftp.var_names thus defines the different folder roots of the SFTP protocol for all users.

In order to configure the VSFTP protocol for the User Space, the following properties in the PROACTIVE_HOME/config/scheduler/settings.ini file must be defined:

pa.scheduler.dataspace.defaultuser.url=vsftp://your-server
pa.scheduler.dataspace.defaultuser.impersonation=true

Additionally, the following Java properties must be added to the ProActive server and all ProActive Nodes connected to the server:

proactive.vfsprovider.vsftp.var_names : a comma-separated list of environment variables which will be used as vsftp root folders for all users. Default value HOME.
proactive.vfsprovider.vsftp.var_command (optional) : the configured command will be executed as the current user inside the SFTP server to retrieve the environment variable(s) value (s). Depending on how the environment variable is defined on the SFTP server (.bashrc, .profile or .ssh/environment files), it is necessary to adapt the command to make sure the appropriate environment script is sourced. The property uses the %VAR% pattern which will be replaced dynamically by the variable name. Default value is echo $%VAR%. Example definition proactive.vfsprovider.vsftp.var_command='source $HOME/.bashrc & echo $%VAR%'.

In order to add the above properties to the ProActive server and local Nodes, you can edit the following files and include these properties:

PROACTIVE_HOME/config/network/server.ini
PROACTIVE_HOME/config/network/node.ini

In order to add these properties to remote ProActive Nodes, the procedure will depend on the deployment strategy:

Nodes started manually: you can add these properties in the command line. For example, ./proactive-node -Dproactive.vfsprovider.vsftp.var_names=HOME,WORK (…).
Nodes started using ProActive Linux Agents: the agent configuration file /opt/proactive-agent/config.xml needs to be edited to include the properties in the <jvmParameters> section. For example:

        <jvmParameters>
            <param>-Dproactive.useIPaddress=true</param>
            <param>-Dproactive.vfsprovider.vsftp.var_names=HOME,WORK</param>
        </jvmParameters>

Nodes started using the ProActive Windows Agents: using the Windows Agent GUI, the properties must be added in the JVM option section. See Configure Agents on Windows.
Nodes started using ProActive Resource Manager Infrastructures: the properties needs to be added in the corresponding infrastructure parameter which allows to define java properties. See Node Source Infrastructures for further details.

3.5. Configure automated backup

ProActive Workflows & Scheduling features an automated backup mechanism which can be used to save the server state at regular intervals. Usage of this mechanism will depend on the database in use by the server:

In case the default database is used (HSQLDB), the backup mechanism can store the database contents, data and log files of the server.
In case another database is used (MySQL, Oracle, etc), the integrated backup mechanism can only store data and log files. Backup of the database contents will need to be activated separately, using the database vendor backup mechanism.

To use the automated backup, edit the file PROACTIVE_HOME/config/shared/settings.ini. Enable backup mechanism by setting pa.server.backup to true. Backup will be performed according to the provided cron expression in pa.server.backup.period. Set file/folders you want to backup into pa.server.backup.targets. Backup archives will appear in the folder specified by pa.server.backup.destination. pa.server.backup.windows controls how many backup archives needs to be kept in the destination folder.

The backup mechanism automatically freezes the scheduler while performing the backup to guarantee data integrity. It expects currently running tasks to terminate in a reasonable amount of time (configurable using the pa.server.backup.possible.delay property). In case a currently running task does not terminate after the expected delay, backup will not be performed.

3.6. Encrypting configuration passwords

Some configuration settings such as database passwords should be encrypted to prevent their exposition. In order to encrypt any configuration setting, the command-line PROACTIVE_HOME/tools/encrypt or PROACTIVE_HOME\tools\encrypt.bat can be used.

Here is an example of usage:

encrypt -d mypass
Encrypted value (to use in configuration files):
ENC(/b0izhUHO846/GmAn8Hbhg==)

The interactive mode can also be used:

encrypt -i
Data to encrypt:*****
Encrypted value (to use in configuration files):
ENC(xtlMWGd0ZK/EzTMyrgPiPw==)

The encrypted value can then be used as displayed by the encrypt command, for example, in PROACTIVE_HOME/config/scheduler/database.properties: hibernate.connection.password=ENC(xtlMWGd0ZK/EzTMyrgPiPw==)

3.7. Configure Global Variables and Generic Information

ProActive Server allows to configure workflow variables or generic information that will apply to all submitted workflows or to some categories of workflows (e.g. workflows with a given name).

This configuration is available in the file PROACTIVE_HOME/config/scheduler/global_variables.xml.

The default configuration is the following:

<?xml version="1.0" encoding="UTF-8"?>
<globalvariables
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="urn:proactive:globalvariables:3.13" xsi:schemaLocation="urn:proactive:globalvariables:3.13 http://www.activeeon.com/public_content/schemas/proactive/globalvariables/3.13/globalvariables.xsd">
        <filter>
            <select>
                <!-- Xpath expressions are used to filter workflows affected by global variables.
                Example: to select workflows belonging to the basic-examples bucket
                <xpath><![CDATA[/job/genericInformation/info[@name='bucketName' and @value='basic-examples']]]></xpath>
                -->
                <xpath><![CDATA[.]]></xpath>
                <!-- more than one xpath expression can be added. In that case, global variables are applied when all xpath expressions match -->
            </select>
            <!-- add global variables or generic information below -->
            <variables>
            </variables>
            <genericInformation>
            </genericInformation>
        </filter>
        <!-- Multiple filters can be added, to apply different set of global variables to different workflows -->
</globalvariables>

This configuration file is composed of <filter> sections. Each filter section describes the global variables and generic information that will be added to a certain category of workflows. Multiple filter sections allow to add different lists of variables to different types of workflows.

A <filter> section contains the following subsections:

<select>: it contains one or more xpath expressions to filter workflow according to xml rules. In order to select a given workflow, the xpath expression evaluated on this workflow must return a non-empty node set (see xpath documentation). The default xpath expression <xpath><![CDATA[.]]></xpath> selects any workflow. If multiple xpath expressions are included, a given workflow must satisfy all expressions in order to be selected. Examples of <select> section:
Select all workflows with name="CronJob"
```
<select>
    <xpath><![CDATA[/job[@name='CronJob']]]></xpath>
</select>
```
Select all workflows in the "basic-examples" catalog bucket
```
<select>
    <xpath><![CDATA[/job/genericInformation/info[@name='bucketName' and @value='basic-examples']]]></xpath>
</select>
```
<variables>: it contains the list of variables that will be added to the selected workflows. The format of this section corresponds to the same format as the <variables> section in the ProActive workflow XML definition. See Job Variables for a description of this format.
<genericInformation>: it contains the list of generic information that will be added to the selected workflows. The format of this section corresponds to the same format as the <genericInformation> section in the ProActive workflow XML definition. See Generic Information for a description of this format.

The following important rules apply to global variables and generic information:

Global variables reload: the definition of global variables and generic information is reloaded every 10 minutes by the ProActive server. It is thus possible to modify the PROACTIVE_HOME/config/scheduler/global_variables.xml configuration and have the modifications taken into account without restarting the server.
Masking rule: if a global variable named VAR1 is defined and a workflow which contains the same variable is submitted, then the definition of VAR1 inside the workflow will take precedence over the global variable.

Variable references: a global variable can reference another global variable using the ${VAR_NAME} syntax. A global generic information can also reference a global variable (but a global generic information cannot reference another global generic information). Examples:

            <variables>
                <variable name="var1" value="value" model=""/>
                <variable name="var2" value="some_${var2}" model=""/> <!-- var2 references var1 -->
            </variables>
            <genericInformation>
                <info name="ginfo1" value="${var1}"/> <!-- ginfo1 references var1 -->
                <info name="ginfo2" value="${ginfo1}"/> <!-- INVALID, a generic information cannot reference another generic info-->
            </genericInformation>

3.8. Manage job labels

A label is a property that can be attached to a job, and changed dynamically by end-users. The Labels give the possibility to tag Jobs, group them and manipulate them by filtering based on these custom labels. These Labels act as a custom status for the Jobs. For example, a company can label jobs which are having errors to be reviewed by operators (label "to-be-reviewed"), and later mark them as "reviewed" when done. Job labels can be used for this specific purpose, but for any other purpose.

In order to use Labels, the Administrators have to define a list of labels that can be applied to jobs. This has to be done in the Admin menu of the Scheduler Portal. In practice, possible labels are stored in the scheduler database. An Admin can add, change, remove authorized labels.

Once a label is applied to a job, it will be stored alongside the job information in the scheduler database. If, in the future, the administrator changes the label name or deletes it, this operation will not be applied retroactively (the original label attached to the job will not be modified).

In User Guide, Adding Labels to Jobs

The constraints of a label name can be configured in the PROACTIVE_HOME/config/scheduler/settings.ini file.

The maximum length is specified by the property pa.scheduler.label.max.length. If this property is not specified, the default value is 20.

The regular expression constraint is specified by the property pa.scheduler.label.regex. If this property is not specified, the default value is ^[a-zA-Z0-9_/-]*$.

3.9. Custom logo configuration

To be able to edit the default logo displayed in ProActive portals, you have to replace the following file PROACTIVE_HOME/dist/war/getstarted/assets/image/custom-logo.png with your company custom logo.

You can use the command SCP to copy the file from your local machine to the ProActive server machine. The dimensions of the custom logo are automatically adapted.

Please note that there’s no need to restart the server, just clear the cache and refresh the Web page.

3.10. Configuration Tools

3.10.1. Configure Http Server Command

The command line PROACTIVE_HOME/tools/configure-http allows configuring the main parameters of the http server.

The protocol (scheme).
The port.
The hostname or IP address used by internal http communications (when a specific hostname must be configured rather than localhost).
a public url (when the ProActive server is behind a reverse proxy or installed in a cloud instance).

The command options with several usage examples are described below:

$> ./configure-http -h
usage: configure-http [-d] [-h] [-H <HOSTNAME>] [-P <PORT>] [--public-hostname <PUBLIC-HOSTNAME>] [--public-port <PUBLIC-PORT>]
[--public-scheme <PUBLIC-SCHEME>] [-S <SCHEME>]

Change ProActive Jetty (Web) Server parameters and apply it to all configuration files.
This command can be used, for example, to change the ProActive server HTTP port or switch the server to the HTTPS protocol.

 -d,--debug                               Debug mode (prints modified files and properties)
 -h,--help                                Display this help
 -H,--hostname <HOSTNAME>                 Hostname used (e.g. localhost, myserver) or IP address (e.g. 127.0.0.1, 192.168.12.1). When
                                          this option is not provided, the hostname appearing in existing configuration files urls will
                                          be unchanged.
 -P,--port <PORT>                         Port used (e.g. 8080, 8443). When this option is not provided, the port configured for the
                                          current scheme (http or https) will be used.
    --public-hostname <PUBLIC-HOSTNAME>   Public Hostname used (e.g. myserver.mydomain) or IP address (e.g. 192.168.12.1). This setting
                                          should be used when the ProActive server is deployed behind a reverse proxy or inside a cloud
                                          instance. This option must be defined when any of public-scheme or public-port is set.
    --public-port <PUBLIC-PORT>           Public port used (e.g. 8080, 8443). This setting should be used when the ProActive server is
                                          deployed behind a reverse proxy or inside a cloud instance. This option must be defined when
                                          any of public-scheme or public-hostname is set.
    --public-scheme <PUBLIC-SCHEME>       Public protocol used (http or https). This setting should be used when the ProActive server
                                          is deployed behind a reverse proxy or inside a cloud instance. This option must be defined
                                          when any of public-hostname or public-port is set.
 -S,--scheme <SCHEME>                     Http protocol to use (http or https). When this option is not provided, the currently
                                          configured protocol will remain unchanged.

Examples:
# Configure scheme to https with port 8444 and default hostname. Debug output to see all modifications.
configure-http -S https -P 8444 -d
# Configure scheme to http with default port and specific hostname
configure-http -S http -H myserver
# Configure public address to use the server behind a reverse-proxy
configure-http --public-scheme=https --public-hostname=myproxy.mycompany.com --public-port=8443

See REST API & Web Properties for the full description of the http server configuration.

3.10.2. Configure Database Command

The command line PROACTIVE_HOME/tools/configure-db allows configuring the ProActive server to use an external database.

ProActive uses by default an embedded database called HSQLDB. In order to use a different database such as PostgreSQL, MySQL or Oracle, the configure-db command can be used to modify the ProActive server configuration files and allow interaction with this external database.

configure-db configures the connection from ProActive components to the database and not the database itself.

Each ProActive component (Resource Manager, Scheduler, Catalog, Service Automation and Notification Service) that uses an external database needs to be configured separately. Accordingly, the configure-db command should be run multiple times.

The command options with several usage examples are described below:

$> ./configure-db -h
usage: configure-db [-c <COMPONENT>] [-d] [-h] [-H <HOSTNAME>] [-p <PASSWORD>] [-P <PORT>] [-s <SCHEMA-NAME>] [-U <URL>] [-u
       <USERNAME>] [-v <DATABASE-VENDOR>] [-z <TIMEZONE>]

Change ProActive server database configuration.
ProActive can either use the integrated HSQLDB database or an external database (PostgreSQL MySQL or Oracle).
ProActive requires the following components to be mapped to different database schemas:
Resource Manager, Scheduler, Catalog, Service Automation and Notification Service.

 -c,--component <COMPONENT>               Target component to configure. Component name can be "rm", "scheduler", "catalog",
                                          "service-automation", "notification" or "all" (to configure all components with a common
                                          configuration).
 -d,--debug                               Debug mode (prints modified files and properties)
 -h,--help                                Display this help
 -H,--hostname <HOSTNAME>                 Database hostname used (e.g. localhost, myserver) or IP address (e.g. 127.0.0.1,
                                          192.168.12.1). This option cannot be used in conjunction with --url.
 -p,--password <PASSWORD>                 Database user password. Password can be provided already encrypted using the 'encypt' command
                                          or in clear text. The password will always be encrypted when stored in configuration files.
 -P,--port <PORT>                         Database port used (e.g. 9001, 5432). If this option is not provided, the database vendor
                                          default port will be used. This option cannot be used in conjunction with --url.
 -s,--schema-name <SCHEMA-NAME>           Override the default database schema name for the selected component. Defaults names are {
                                          Resource Manager : "rm", Scheduler : "scheduler", Catalog : "catalog", Service Automation :
                                          "pca", Notification Service : "notification" }. This option cannot be used when
                                          --component=all or when --database-vendor=hsqldb
 -U,--url <URL>                           Sets the JDBC url that will be used to access the target component database. If not used, the
                                          url will be inferred from --hostname, --port and optionally --schema-name options. This
                                          option cannot be used when --component=all. Note: this parameter is the only way to use the
                                          TNS URL Format with Oracle database.
 -u,--username <USERNAME>                 Database user name
 -v,--database-vendor <DATABASE-VENDOR>   Target database vendor to configure ("hsqldb", "postgresql", "mysql" or "oracle")
 -z,--timezone <TIMEZONE>                 Specific to MySQL, sets the timezone used by the MySQL server. The default is "UTC". This
                                          option cannot be used in conjunction with --url.

Examples:
# Configure HSQLDB for all components with single user and password, default hostname and port. Debug output to see all modifications.
configure-db -c all -v hsqldb -u user -p pwd -d
# Configure PostgreSQL for the Resource Manager component with a specific hostname and port
configure-db -c rm -v postgresql -H myserver -P 5434 -u user -p pwd
# Configure PostgreSQL for the Resource Manager component with a specific hostname, port and schema name
configure-db -c rm -v postgresql -H myserver -P 5434 -s RESOURCE_MANAGER -u user -p pwd
# Configure Oracle for the Resource Manager component with a TNS URL format
configure-db -c rm -v oracle -U
"jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myserver)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=rm))" -u user -p pwd

See Database configuration for a complete explanation of different database vendor configurations.

Configure LDAP / Active Directory Command

The command line PROACTIVE_HOME/tools/configure-ldap allows configuring the ProActive server to use LDAP or Active Directory authentication.

configure-ldap uses the following process in order to configure authentication:

It modifies the various configuration files using the parameters received (LDAP server url, bind login, user/group subtree, etc).
It checks the connection to the LDAP server and installs inside the ProActive Java Runtime Environment SSL certificates in case of LDAPS usage.
It tests authentication with a provided test user's credentials to validate that authentication works properly.
If it succeeds, the configuration is persisted. Otherwise, the configuration is reverted.

The command options with several usage examples are described below:

$> ./configure-ldap -h
usage: configure-ldap [-a] [-d] [--group.attr <GROUP.ATTR>] [--group.filter <GROUP.FILTER>] [--group.subtree <GROUP.SUBTREE>] [-h]
       [-l <LOGIN>] [-o <OUTPUT.FILE>] [-p <PASSWORD>] [-s] [--start.tls.disable.check] [--tenant.attribute <TENANT.ATTRIBUTE>]
       [--test.pwd <TEST.PWD>] [--test.user <TEST.USER>] [--truststore.path <TRUSTSTORE.PATH>] [--truststore.pwd <TRUSTSTORE.PWD>] [-u
       <URL>] [--user.filter <USER.FILTER>] [--user.subtree <USER.SUBTREE>]

Configure ProActive server for LDAP authentication.
ProActive can either use an Active Directory or a classical LDAP server for authentication.

 -a,--auth.disable                          Use this option to disable authentication to the LDAP server (only when the server supports
                                            anonymous connections).
 -d,--debug                                 Debug mode (prints modified files and properties)
    --group.attr <GROUP.ATTR>               The attribute in the group entry that matches the jaas' group name. Default is "cn".
    --group.filter <GROUP.FILTER>           LDAP filter executed when searching for all groups associated with a given user. By
                                            default, the search takes as parameter the user LDAP entry. Option group.filter.use.uid can
                                            be enabled to give as parameter the user identifier instead. The default group filter,
                                            depending on the LDAP server type and group.filter.use.uid option is  :
                                            {LDAP=(&(objectclass=groupOfUniqueNames)(uniqueMember=%s)),
                                            LDAPwithUID=(&(objectclass=posixGroup)(memberUID=%s)),
                                            AD=(&(objectclass=group)(member:1.2.840.113556.1.4.1941:=%s))}
    --group.subtree <GROUP.SUBTREE>         Scope in the LDAP tree where groups can be found. If not set, the value of option
                                            user.subtree will be used.
 -h,--help                                  Display this help
 -l,--login <LOGIN>                         login name used to connect (bind) to the ldap server when executing queries. The login is
                                            usually a LDAP distinguished name (e.g. uid=janedoe,ou=People,dc=activeeon,dc=com).
 -o,--output.file <OUTPUT.FILE>             Ldap configuration file that will be modified. It will be created by copying the default
                                            configuration file if it does not exist.
 -p,--password <PASSWORD>                   Password associated with the login name. Password can be provided already encrypted using
                                            the 'encypt' command or in clear text.The password will always be encrypted when stored in
                                            the ldap configuration file.
 -s,--start.tls                             Enable StartTLS mode.
    --start.tls.disable.check               If StartTLS mode is enable, disable certificate verification check.
    --tenant.attribute <TENANT.ATTRIBUTE>   An LDAP attribute such as "department" or "project" can be used to group categories of
                                            users in an abstract organization called "Tenant".
    --test.pwd <TEST.PWD>                   (Required) Password of the user that will be searched using configured filters. This
                                            parameter is required to validate the configuration and guarantee that the connection and
                                            filters are working properly.
    --test.user <TEST.USER>                 (Required) Identifier of the user that will be searched using configured filters. This
                                            parameter is required to validate the configuration and guarantee that the connection and
                                            filters are working properly.
    --truststore.path <TRUSTSTORE.PATH>     Path to the truststore that will be used to store LDAPS certificates. If the file does not
                                            exist it will be created. The default value is config/authentication/truststore
    --truststore.pwd <TRUSTSTORE.PWD>       Password used to encrypt the truststore that will be created if the LDAP server is using
                                            LDAPS protocol. If not provided the default password will be "activeeon".
 -u,--url <URL>                             (required) Url(s) of the LDAP server(s). Multiple servers can be configured using a
                                            comma-separated list.
    --user.filter <USER.FILTER>             LDAP filter executed when searching for a LDAP user entry which corresponds to a given user
                                            identifier. The default user filter, depending on the LDAP server type is :
                                            {LDAP=(&(objectclass=inetOrgPerson)(uid=%s)), AD=(&(objectclass=user)(sAMAccountName=%s))}
    --user.subtree <USER.SUBTREE>           (required) Scope in the LDAP tree where users can be found.

Examples:
# Configure Ldap authentication for a ldaps server and default filters. Debug output to see all modifications.
configure-ldap --url ldaps://ldap-server1.activeeon.com --user.subtree "ou=users,dc=activeeon,dc=org" --group.subtree
"ou=groups,dc=activeeon,dc=org" --login "cn=admin,dc=activeeon,dc=org" --password "my_bind_pwd" --test.user user1 --test.pwd
"my_test_pwd" --debug

See LDAP authentication for a full explanation of LDAP authentication parameters.

3.10.3. Copy Role Command

The command line PROACTIVE_HOME/tools/copy-role can be used to duplicate a role defined in the PROACTIVE_HOME/config/security.java.policy-server file.

As described in User Permissions, the file security.java.policy-server is used to store the definitions of users groups and their rights within the ProActive server.

The copy-role command is used when a new group of users needs to be created with the same rights as an existing group. It can also be used after configuring LDAP authentication, to map existing ldap groups to one of the default roles defined in security.java.policy-server.

For example, in order to import the ldap group it-admin as an administrator of the ProActive server:

copy-role --source server-admins --destination it-admin

See Default Groups/Roles for a description of all roles defined by default in the ProActive server.

Any modification to the security.java.policy-server file is taken into account only after the ProActive server is restarted.

The command options with several usage examples are described below:

$> ./copy-role -h
usage: copy-role [-d] [-D <DESTINATION>] [-h] [-S <SOURCE>] [-y]

Create a new JaaS role by copying an existing one.
JaaS roles are used to define authorization inside the ProActive server.

 -d,--debug                       Debug mode (prints modified files and properties)
 -D,--destination <DESTINATION>   (required) Name of the destination role (group).
 -h,--help                        Display this help
 -S,--source <SOURCE>             (required) Name of the role (group) that will be copied.
 -y,--overwrite                   Overwrite if the destination role already exists.

Examples:
# Copy existing 'server-admins' role into new role 'my-admin-group'. Debug output to see all modifications.
copy-role --source server-admins --destination my-admin-group --debug

4. User Authentication

In order to log into ProActive server, every user must have a registered account.

A user login information is associated with the following attributes:

a login name: when the user authenticate himself into the ProActive server, he/she will use his registered login.
a domain: when the server runs on a Windows operating system or when the authentication method is configured as LDAP authentication with Multiple Domains, a domain name may be associated with the account.
a password.
a list of groups. A user may belong to one or more groups. Each group has its own set of authorizations as described in chapter User Permissions.
a tenant: a tenant name may be associated with the user account. Tenants represent the isolation of a set of users within the ProActive server. Tenants must first be configured either using Tenant File Configuration, Tenant LDAP attribute configuration or LDAP authentication with Multiple Domains. When LDAP authentication with Multiple Domains is set, the tenant name is equal to the domain name.

User accounts are defined with one of the following authentication methods:

File-based
LDAP
LDAP with Multiple Domains
PAM
Keycloak

4.1. Configure Domains / Tenants

4.1.1. Domains

Within ProActive, a domain is part of a user login information and can either represent:

A Windows domain.
The name of a LDAP server in case of LDAP authentication with Multiple Domains.
Both when Windows domain names and LDAP/Active Directory server names match.

The domain is used either:

to select which LDAP/Active Directory server will be queried in case of LDAP authentication with Multiple Domains.
to select which Windows domain will be used when the user executes RunAsMe workflow tasks inside a ProActive Node running on a Windows Operating System.

Before being able to use domains in login information, the list of allowed domains must be configured on the ProActive server.

When configuring allowed domains, it is important to include an empty domain name. This is because ProActive uses internal accounts that do not belong to any domain. Failing to do so would prevent the ProActive server from starting. An empty domain is included by using the syntax ",domain1,domain2".

There are two properties to configure:

in PROACTIVE_HOME/config/rm/settings.ini

pa.rm.allowed.domains=,domain1,domain2

in PROACTIVE_HOME/config/scheduler/settings.ini

pa.scheduler.allowed.domains=,domain1,domain2

⇒ Allows using either domain1, domain2 or an empty domain name.

When the ProActive server runs on a Windows operating system, allowed domains properties default to ",SERVER_USER_DOMAIN", where SERVER_USER_DOMAIN corresponds to the Windows domain name of the user that starts the ProActive server.

After allowed domains have been configured, the login page of the various portals will display a drop-down list which allows to select the login domain.

Example:

4.1.2. Tenants

Tenants are organization units that regroup class of users in an isolated environment within the ProActive server.

For example, user1 belonging to tenant1 will not be able to see jobs submitted by user2 belonging to tenant2. It is also possible to restrict access to ProActive Nodes based on tenants (a Node Source can restrict its access to a specific tenant name).

Tenants can thus improve the management and governance of an organization’s hybrid compute infrastructure.

User tenants are different from user groups:

A user can belong to several groups but can only belong to a single tenant.
User groups does not allow the same level of isolation as tenants allow (for example, it is not possible to prevent users from seeing jobs that belong to a different group).
User groups allow defining roles (administrators, standard users, etc.) inside the ProActive server. A user tenant does not correspond to a role.

Tenant user membership can be configured using one of the following methods:

a file-based tenant configuration. This file is a group → tenant association which allows to define a collection of groups as tenant.
a LDAP attribute configuration. This method can be used when a single LDAP/Active Directory server is configured as authentication method. In that case, a scalar LDAP attribute defined for all users can be used as tenant name (For example Organization Name, Division, etc.).
a Multi-Domain LDAP configuration. When multi-domain LDAP configuration is used as authentication method, the tenant name will be automatically set to the domain name used during authentication. For example, if user1 logs in from domain1, user1 will also belong to the domain1 tenant. This is the only case where domain name and tenant name will be equal.

A tenant can either be a collection of groups (if defined using a file), an organization unit (if defined using an LDAP attribute) or a LDAP domain (if the authentication method is configured to LDAP with multiple domains).

After tenants definition is configured, it is necessary to configure in the ProActive Server tenant isolation.

Tenants Job Isolation

Tenants Job isolation is the ability to hide submitted jobs across tenants.

For example, when user1 belonging to tenant1 submits a job to the ProActive Scheduler, then user2 belonging to tenant2 will not see this job activity on the Scheduler or Workflow Execution portals.

In order to configure Tenants Job isolation, the setting pa.scheduler.core.tenant.filter must be set to true in PROACTIVE_HOME/config/scheduler/settings.ini.

When this configuration is enabled, standard users will only see jobs according to the user’s tenant. Admin users will still be able to see jobs coming from all tenants.

It is not mandatory to configure tenant isolation. Tenants can simply be used as additional information associated with each user submitted jobs.

Figure 1. Scheduler Portal for "Users" tenant

Figure 2. Scheduler Portal for "Experts" tenant

Figure 3. Scheduler Portal for Admin users

Tenants Node Source Isolation

Apart from job isolation, it is also possible to restrict Node Sources across tenants.

Restricting a Node Source to a specific tenant can be done by using the nodeUsers parameter which is common to all Node Source Policies.

See Policy Common Parameters for more information.

4.2. Select authentication method

By default, the ProActive server is configured to use file-based authentication and has some default accounts ('demo', 'admin', 'user') that can be used to first log into the server.

The default accounts passwords (demo:demo, admin:admin and user:pwd) should be modified using the proactive-users command.

If you would like to change the authentication mechanism to use your LDAP server or use Linux PAM, you need to modify two configuration files:

Resource Manager configuration (PROACTIVE_HOME/config/rm/settings.ini)

#Property that defines the method that has to be used for logging users to the Resource Manager
#It can be one of the following values:
#  - "RMFileLoginMethod" to use file login and group management
#  - "RMLDAPLoginMethod" to use LDAP login management
#  - "RMMultiLDAPLoginMethod" to use LDAP with multiple domains
#  - "RMPAMLoginMethod" to use PAM login management
#  - "RMKeycloakLoginMethod" to use Keycloak login management
pa.rm.authentication.loginMethod=RMFileLoginMethod

Scheduler configuration (PROACTIVE_HOME/config/scheduler/settings.ini)

#Property that define the method that have to be used for logging users to the Scheduler
#It can be one of the following values:
#  - "SchedulerFileLoginMethod" to use file login and group management
#  - "SchedulerLDAPLoginMethod" to use LDAP login management
#  - "SchedulerMultiLDAPLoginMethod" to use LDAP with multiple domains
#  - "SchedulerPAMLoginMethod" to use PAM login management
#  - "SchedulerKeycloakLoginMethod" to use Keycloak login management
pa.scheduler.core.authentication.loginMethod=SchedulerFileLoginMethod

4.3. File authentication

By default, the ProActive server stores users accounts, passwords, group memberships (user or admins), and tenant memberships (group agglomerate or organization), in two files.

The default file path to store users, group and tenant memberships can be modified by changing the following settings:

PROACTIVE_HOME/config/rm/settings.ini

pa.rm.defaultloginfilename=config/authentication/login.cfg
pa.rm.defaultgroupfilename=config/authentication/group.cfg
pa.rm.defaulttenantfilename=config/authentication/tenant.cfg

PROACTIVE_HOME/config/scheduler/settings.ini

pa.scheduler.core.defaultloginfilename=config/authentication/login.cfg
pa.scheduler.core.defaultgroupfilename=config/authentication/group.cfg
pa.scheduler.core.defaulttenantfilename=config/authentication/tenant.cfg

4.3.1. Users

Users and passwords accounts are stored in PROACTIVE_HOME/config/authentication/login.cfg. Each line has to follow the format user:encrypted password.

4.3.2. Groups

Users group membership is stored in PROACTIVE_HOME/config/authentication/group.cfg. For each user registered in login.cfg, a group membership has to be defined in this file. Each line has to look like user:group. Group has to be user to have user rights, scheduleradmins to have administrator rights on the scheduler, rmcoreadmins to have administrator rights on the resource manager, etc. Below is an example group.cfg file:

group.cfg

admin:scheduleradmins
admin:rmcoreadmins
demo:scheduleradmins
demo:rmcoreadmins
citizen-ds:citizen-ds
expert-ds:expert-ds
guest:guests
nsadmin:nsadmins
provider:providers
rm:admin
rm:nsadmins
scheduler:user
scheduler:scheduleradmins
user:user
watcher:watchers

ProActive contains a set of predefined groups such as user and providers. Group rights are defined in PROACTIVE_HOME/config/security.java.policy-server as described in chapter User Permissions.

4.3.3. Tenants

Tenants are organization units that regroup class of users in an isolated environment within the ProActive server.

When defined via a file, tenant organization is stored in PROACTIVE_HOME/config/authentication/tenant.cfg. Each line of this file contains a couple (group_name, tenant_name), using the syntax group_name:tenant_name. To define tenants, an administrator of the platform need to associate one or more groups to each tenant.

Each user can belong to only one tenant. Thus, if a user is a member of multiple groups belonging to different tenants, only the first tenant found will be associated.

By default, the tenant configuration file is empty, below is an example tenant configuration:

tenant.cfg

scheduleradmins:admins
rmcoreadmins:admins
nsadmin:admins
citizen-ds:citizens
expert-ds:experts
user:users

4.3.4. Command Line

In order to create new users, delete an existing user, or modify the groups or password for an existing user, a command-line tool is available.

This command is available inside the tools folder: PROACTIVE_HOME/tools/proactive-users. You can check the command syntax using the -h option.

$ proactive-users -h
usage: proactive-users [-C | -D | -U]  [-g <GROUPS>] [-gf <GROUPFILE>] [-h] [-kf <KEYFILE>] [-l <LOGIN>] [-lf <LOGINFILE>] [-p
       <PASSWORD>] [-sgf <SOURCEGROUPFILE>] [-slf <SOURCELOGINFILE>]

Here are some example usages:

Creating users

$ proactive-users -C -l user1 -p pwd1 -g user
Created user user1 in H:\Install\scheduling\tools\..\config/authentication/login.cfg
Added group user to user user1

The user with login "user1", password "pwd1" and group "user" was created

Updating users

$ proactive-users -U -l user1 -p pwd2 -g nsadmins,admin
Changed password for user user1 in H:\Install\scheduling\tools\..\config/authentication/login.cfg
Added group nsadmins to user user1
Added group admin to user user1

User "user1" now has password pwd2 and groups nsadmins & admin (group "user" was removed).

Deleting users

$ proactive-users -D -l user1
Deleted user user1 in H:\Install\scheduling\tools\..\config/authentication/login.cfg

User "user1" does not exist any more.

Creating multiple users

It is also possible to create multiple users at once using a source login file and a source group file. In that case, the source login file contains, for each user, a line with the format "login:unencrypted_password". The source group file has the same structure as the PROACTIVE_HOME/config/authentication/group.cfg file. This can be used, for example, to convert login files used by ProActive Scheduler versions prior to 7.19.0.
```
$ proactive-users -C -slf source_login.cfg -sgf source_group.cfg
Adding group admin to user admin1
Created user admin1
Adding group user to user user2
Created user user2
Adding group user to user user1
Created user user1
Stored login file in H:\Install\scheduling\tools\..\config/authentication/login.cfg
Stored group file in H:\Install\scheduling\tools\..\config/authentication/group.cfg
```

Updating multiple users

Similarly, it is possible to update existing users with source login or group files. It is possible to update only group membership for existing users, or only passwords, or both.

The example below shows how to update only groups for existing users:

proactive-users -U -sgf source_group_2.cfg
Adding group admin to user user1
Updated user user1
Adding group admin to user user2
Updated user user2
Stored login file in H:\Install\scheduling\tools\..\config/authentication/login.cfg
Stored group file in H:\Install\scheduling\tools\..\config/authentication/group.cfg

4.4. LDAP authentication

The ProActive server is able to connect to an existing LDAP server, to check users' logins, passwords and verify users group memberships.

In order to use the LDAP authentication method, change the default authentication method as described in Select authentication method.

Additionally, multiple LDAP parameters must be configured in a dedicated LDAP configuration file, such as path in LDAP tree users, LDAP groups that define user and admin group membership, URL of the LDAP server, LDAP binding method used by connection and configuration of SSL/TLS if you want a secured connection between the ProActive Resource Manager and LDAP.

The PROACTIVE_HOME/tools/configure-ldap command can be used to configure LDAP and test the configuration. See Configure LDAP / Active Directory Command.

We assume that the existing LDAP server is configured in a way that:

all existing users and groups are located under a single domain
user and group name is defined in cn (Common Name) attribute

# EXAMPLE of user entry
#
# dn: cn=jdoe,dc=example,dc=com
# cn: jdoe
# firstName: John
# lastName: Doe
# objectClass: inetOrgPerson

# EXAMPLE of group entry
#
# dn: cn=mygroup,dc=example,dc=com
# cn: mygroup
# firstName: John
# lastName: Doe
# uniqueMember: cn=djoe,dc=example,dc=com
# objectClass: groupOfUniqueNames

The LDAP configuration is defined in PROACTIVE_HOME/config/authentication/ldap.cfg. You need to:

4.4.1. Set the LDAP server URL

First, you have to define the LDAP URL of your organization. This address corresponds to the property: pa.ldap.url. You have to put a standard LDAP-like URL, for example ldap://myLdap. You can also set a URL with secure access: ldaps://myLdap:636.

Finally, it is possible to define additional URLs that will be used as backup if the primary URL is not responding.

pa.ldap.url=ldap://myLdapServer1 ldap://myLdapServer2

4.4.2. Define user and group filters

The authentication module must be able to find a user ldap entry, given its login name (or any other appropriate ldap attribute). For each user, the module must also be able to find all group ldap entries associated with the user.

The LDAP module uses ldap filters to search for users and groups.

User filters

The user filter is defined using the property pa.ldap.user.filter. The search is performed according to a configurable ldap subtree, controlled by the property pa.ldap.userssubtree.

pa.ldap.userssubtree must be defined according to the base distinguished name of the ldap server. It can be further refined to restrict the search to a specific subtree where the users are located.

Example:

pa.ldap.userssubtree=dc=mycompany,dc=com

pa.ldap.user.filter must be defined according to the ldap server version.

For an OpenLDAP server (V2 or V3), use:

pa.ldap.user.filter=(&(objectclass=inetOrgPerson)(uid=%s))

For an Active Directory server, use:

pa.ldap.user.filter=(&(objectclass=user)(sAMAccountName=%s))

The ldapsearch command can be used to test the execution of user filters.

For example:

ldapsearch -x -b "dc=mycompany,dc=com" -H ldap://myLdapServer1 -D "cn=admin,dc=mycompany,dc=com" -W '(&(objectclass=user)(sAMAccountName=user1))'

Can be used to search that the configured user filter is able to locate the ldap entry for user1 on an Active Directory server

Group filters

The group filter is defined using the property pa.ldap.group.filter. The search is performed according to a configurable ldap subtree, controlled by the property pa.ldap.groupssubtree.

pa.ldap.groupssubtree can be left undefined. In that case, the value configured for pa.ldap.userssubtree will be used.

As group membership search can be time-consuming for large organizations, it is recommended to set pa.ldap.groupssubtree to a subtree that contains groups that will be mapped as ProActive server roles.

Example:

pa.ldap.groupssubtree=ou=groups,ou=mydepartment,dc=mycompany,dc=com

pa.ldap.group.filter must be defined according to the ldap server version.

For an OpenLDAP server V2, use:

pa.ldap.group.filter=(&(objectclass=groupOfUniqueNames)(uniqueMember=%s))

For an OpenLDAP server V3, use:

pa.ldap.group.filter=(&(objectclass=posixGroup)(memberUID=%s))

The following setting must also be enabled for an open ldap server V3:

pa.ldap.group.search.use.uid=true

This tells the authentication module to use the login name instead of distinguished name as parameter of the group filter.

For an Active Directory server, use:

pa.ldap.group.filter=(&(objectclass=group)(member:1.2.840.113556.1.4.1941:=%s))

Finally, the following property controls which LDAP attribute contains the group name in the group ldap entry (it is normally not necessary to change the default value):

pa.ldap.group.name.attr=cn

The ldapsearch command can be used to test the execution of group filters.

For example:

ldapsearch -x -b "dc=mycompany,dc=com" -H ldap://myLdapServer1 -D "cn=admin,dc=mycompany,dc=com" -W '(&(objectclass=group)(member:1.2.840.113556.1.4.1941:=cn=user1,ou=users,dc=mycompany,dc=com))'

Can be used to search that the configured group filter is able to find all groups associated with user1 (contrary to the user filter, the user distinguished name (dn) is given as parameter)

4.4.3. Configure LDAP authentication parameters

By default, the ProActive Scheduler binds to LDAP in anonymous mode. You can change this authentication method by modifying the property pa.ldap.authentication.method. This property can have several values:

none (default value) - the ProActive Resource Manager performs connection to LDAP in anonymous mode.
simple - the ProActive Resource Manager performs connection to LDAP with a specified user/password (see below for user password setting).

You can also specify a SASL mechanism for LDAPv3. There are many SASL available mechanisms: cram-md5, digest-md5, kerberos4. Just set this property to sasl to let the ProActive server choose a SASL authentication mechanism.

If you specify an authentication method different from 'none' (anonymous connection to LDAP), you must specify a login/password for authentication.

There are two properties to set in LDAP configuration file:

pa.ldap.bind.login - sets user name for authentication.
pa.ldap.bind.pwd - sets password for authentication.

The login used should be the distinguished name (dn) of a user with read access on the LDAP server. The password can be encrypted using the encryption command line tool.

4.4.4. Set SSL/TLS parameters

A secured SSL/TLS layer can be useful if your network is not trusted, and critical information is transmitted between the Resource Manager server and LDAP, such as user passwords.

Encryption of the communication to the ldap server can be performed in two ways:

by using the LDAPS protocol (SSL encryption).
by using StartTLS encryption.

SSL encryption

First, set the LDAP URL property pa.ldap.url to a URL of type ldaps://myLdap. Then, set pa.ldap.authentication.method to none so as to delegate authentication to SSL.

For using SSL properly, it is required to specify a certificate and authorities (public keys) for the SSL handshake. Java stores certificates inside a keyStore and public keys inside a trustStore. In most of the cases, adding authority public keys in a trust store is sufficient.

Generate trustStore manually

The following steps show how to create a keystore and trustStore manually using the keytool command (included in the ProActive server installation):

PROCTIVE_HOME/jre/bin/keytool -import -alias myAlias -file myCertificate -keystore myKeyStore

where:

PROCTIVE_HOME is the path to the ProActive server installation
myAlias is the alias name of your certificate.
myCertificate is your private certificate file.
myKeyStore is the new keyStore file produced in output.

This command asks you to enter a password for your keyStore.

Put the LDAP certificate’s public key in a trustStore, with the following keytool command:

PROCTIVE_HOME/jre/bin/keytool -import -alias myAlias -file myPublicKey -keystore myTrustStore

where:

PROCTIVE_HOME is the path to the ProActive server installation
myAlias is the alias name of your certificate’s public key.
myPublicKey is your certificate’s public key file.
myTrustore is the new trustStore file produced in output.

This command asks you to enter a password for your trustStore.

Finally, in config/authentication/ldap.cfg, set keyStore and trustStore created before to their respective passwords:

Set pa.ldap.keystore.path to the path of your keyStore.
Set pa.ldap.keystore.passwd to the password defined previously for keyStore.
Set pa.ldap.truststore.path to the path of your trustStore.
Set pa.ldap.truststore.passwd to the password defined previously for trustStore.

Download the ldap server certificate

Alternatively to the above procedure, the following command can be used to automatically download the ldap server certificates and install it inside the ProActive server Java Runtime Environment default trustStore:

PROCTIVE_HOME/jre/bin/java -jar PROCTIVE_HOME/tools/installcert-usn-20140115.jar ldap_server_hostname_or_ip:ldap_port

where:

PROCTIVE_HOME is the path to the ProActive server installation.
ldap_server_hostname_or_ip is the hostname or ip address of the ldap server.
ldap_port is the port used to connect to the ldap server.

The command is interactive and will ask the java trustStore password. The default password is changeit.

StartTLS encryption

StartTLS is an alternative encryption method to SSL. StartTLS works by opening a connection using an unencrypted protocol, then upgrading this connection to an encrypted channel. The LDAP server must support StartTLS in order to use it.

In order to configure StartTLS inside the ProActive LDAP module, the keystore/trustStore operations described in Generate trustStore manually or Download the ldap server certificate must be executed.

Additionally, the following properties must be set in the LDAP configuration file:

pa.ldap.starttls=true

pa.ldap.connection.pooling=false

4.4.5. Configure tenant attribute

A LDAP attribute can be used to associate a tenant name to an existing user.

Tenants are organization units that regroup class of users in an isolated environment within the ProActive server.

If your LDAP structure contains an user attribute that seems suitable for tenants isolation, set the pa.ldap.tenant.attr property to the name of this attribute. Example atributes: division, department, company, etc.

See Tenants for more information.

4.4.6. Use fall back to file authentication

You can use simultaneously file-based authentication and LDAP-based authentication. In this scenario, ProActive Scheduler checks at first user/password and group memberships in the login and group files, as performed in the File login method. If the user or group attribution is not found in the login/group files, login or groups will be searched via LDAP.

There are two configuration options to control this mechanism (present in the ldap.cfg configuration file):

pa.ldap.authentication.fallback: if set to true, the user login/password will be searched first in the login file, and then on the LDAP server. If set to false, only the LDAP server will be queried.
pa.ldap.group.membership.fallback: if set to true, group memberships will be read from the LDAP server if the user exists, and then from the group file. In that sense, a user will be affected to groups defined in both file and LDAP server. If set to false, only the LDAP server will be queried.
pa.ldap.tenant.membership.fallback: if set to true, tenant memberships will be read from the tenant file if:
- The user does not exist in the LDAP server.
- The pa.ldap.tenant.attr configuration is not defined.
- The pa.ldap.tenant.attr configuration is defined, but the configured attribute is empty for the user.

4.4.7. Specific example of LDAP configuration

Below is an example of LDAP configuration which summarizes the LDAP properties frequently configured in the file config/authentication/ldap.cfg.

# URL of a ldap used for authentication
pa.ldap.url=ldaps://ldap-server.activeeon.com

# path in the LDAP tree users
pa.ldap.userssubtree=ou=users,dc=activeeon,dc=org

# path in the LDAP tree groups
# if empty, then the value for pa.ldap.userssubtree is used
pa.ldap.groupssubtree=ou=groups,dc=activeeon,dc=org

# the filter that allows to find the user dn given its scheduler login
# %s is replaced by the user-submitted login
pa.ldap.user.filter=(&(objectclass=inetOrgPerson)(uid=%s))

# retrieves the groups the user dn belongs to
# the '%s' parameter is the user dn previously retrieved
pa.ldap.group.filter=(&(objectclass=groupOfUniqueNames)(uniqueMember=%s))

# the attribute in the group entry that matches the jaas' group name
pa.ldap.group.name.attr=cn

# authentication method used to connect to LDAP : none (for anonymous connection), simple or a SASL method
pa.ldap.authentication.method=simple

# login name used to perform ldap's binding
# ex uid=narf,ou=People,dc=activeeon,dc=com
pa.ldap.bind.login=cn=admin,dc=activeeon,dc=org

# password used to perform ldap binding
pa.ldap.bind.pwd=change-pwd

# check user password and group membership in login and group files
pa.ldap.authentication.fallback=true

# check user group membership in group file
pa.ldap.group.membership.fallback=true

# check user tenant membership in tenant file
pa.ldap.tenant.membership.fallback=true

4.5. LDAP authentication with Multiple Domains

The ProActive server can be configured to authenticate users according to multiple LDAP domains.

It means that when a user logs in, he/she will need to select which LDAP domain will be used for the authentication.

In that configuration, the user domain name will be automatically recorded as tenant name.

Following is a step-by-step procedure to configure it.

Copy the file PROACTIVE_HOME\config\authentication\ldap.cfg into multiple files, one for each domain.

For example:
```
config\authentication\ldap-domain1.cfg
config\authentication\ldap-domain2.cfg
```
Using the procedure described in chapter LDAP authentication, configure each ldap file according to the domain used (each domain will have its own ldap server, bind login name, user subtree, etc.).
Follow the steps described in chapter Select authentication method to configure RMMultiLDAPLoginMethod and SchedulerMultiLDAPLoginMethod.
In config\rm\settings.ini, uncomment the property pa.rm.multi.ldap.config and define it according to the ldap configuration files you created and their corresponding domain names:
```
pa.rm.multi.ldap.config=domain1:config/authentication/ldap-domain1.cfg,domain2:config/authentication/ldap-domain2.cfg
```
Uncomment also the property pa.rm.allowed.domains and configure with the list of domains that will be available.
```
pa.rm.allowed.domains=,domain1,domains2
```
This expression must start with a comma, it allows login with an empty domain, necessary for internal ProActive users.
In config\scheduler\settings.ini, uncomment the property pa.scheduler.multi.ldap.config and define it according to the ldap configuration files you created and their corresponding domain names:
```
pa.scheduler.multi.ldap.config=domain1:config/authentication/ldap-domain1.cfg,domain2:config/authentication/ldap-domain2.cfg
```
Uncomment also the property pa.scheduler.allowed.domains and configure with the list of domains that will be available.
```
pa.scheduler.allowed.domains=,domain1,domains2
```
This expression must start with a comma, it allows login with an empty domain, necessary for internal ProActive users.

Once all the steps above are completed, the ProActive server can be (re)started in order to make the multi-domain ldap configuration effective.

In multi-domain mode, when a user logs in to any portal of the ProActive server, the user must specify the domain used. A drop-down list of configured domains will appear when login into ProActive portals.

For other methods of login (such as command line or REST api), the domain can be specified using the notation domain\user.

It is essential to allow the ProActive server select the adequate ldap domain configuration where the user is registered.

4.6. PAM authentication

The ProActive Scheduler & Resource Manager are able to interact with Linux PAM (Pluggable Authentication Modules) to check users login/password. It is not currently possible to retrieve linux system group memberships for users. Thus, groups must be managed using the PROACTIVE_HOME/config/authentication/group.cfg file.

In order to enable PAM authentication, follow the steps below:

Configure the PAM login methods in the Scheduler and RM settings:
PROACTIVE_HOME/config/rm/settings.ini
```
pa.rm.authentication.loginMethod=RMPAMLoginMethod
```
PROACTIVE_HOME/config/scheduler/settings.ini
```
pa.scheduler.core.authentication.loginMethod=SchedulerPAMLoginMethod
```
Copy the file PROACTIVE_HOME/config/authentication/proactive-jpam to /etc/pam.d (you must be root to perform this operation).

You can modify this default PAM configuration file if you need specific PAM policies.
Add the user which will start the scheduler process PROACTIVE_HOME/bin/proactive-server to the shadow group
```
sudo usermod -aG shadow your_user
```
After this command, the user, called "your_user" here in this example, will be added to group shadow. You may need to logoff/logon before this modification can be effective.
Associate, for each Linux system users which will connect to the Scheduler, a ProActive group.

Following the procedure described in chapter File authentication, groups must be associated to existing PAM users in the ProActive group file PROACTIVE_HOME/config/authentication/group.cfg.

In case of PAM users, the login file should not be modified as password authentication will be performed at the Linux system level. Accordingly, you should not use the proactive-users command to associate groups to PAM users.

Similarly to the LDAP authentication, there is a fallback mechanism for login authentication, with the difference that it is always activated. A user defined in the ProActive login file always has precedence over the same user defined on the Linux system.

4.7. Keycloak-based access control

Keycloak is an access control solution that allows single sign-on with identity and access management aimed at modern applications and services. ProActive is able to connect to an existing Keycloak server, and perform user authentication and authorization against it.

ProActive uses the OpenID Connect (OIDC) protocol to communicate with Keycloak and acquire an access token for a given user. It further parses the obtained access token and extracts from it the user information, including the user role. Based on the user role, ProActive determines the set of privileges and authorizations to be granted to the user.

In order to perform ProActive access control against a Keycloak server, two steps must be accomplished: (i) deploying and configuring a Keycloak server and (ii) configuring ProActive to connect to the Keycloak server and perform access control against it.

4.7.1. Configuring Keycloak server

The configuration steps mentioned below provide an example of the basic steps needed to configure ProActive access control against Keycloak. For advanced configurations, please refer to the Keycloak documentation.

Pre-requisites

A Keycloak server must be up and in operational mode so ProActive can interact with it.
The Keycloak server’s administrator should select a security realm (or create a new one if needed).

Configuring a new Keycloak client for ProActive

The Keycloak server’s administrator has to configure a new client for ProActive in the chosen realm. A client is an external software application for which the access control is managed via Keycloak. In the section General Settings of the configuration, the parameters following must be properly set (as shown in the screenshot below):

The property Client type specifies the security protocol of the client application. It must be set to OpenID Connect.
The property Client ID specifies the ID of the client application (e.g., ProActive).
The property Name specifies the displayed name of the client (e.g., ProActive Workflows and Scheduling).
The property Description specifies the description of the client (e.g., ProActive workflows engine).

Figure 4. General settings of Keycloak client

Configuring the authentication mechanism of the client

In the section Capability config of the client configuration, the following parameters must be properly set (as shown in the screenshot below):

The property Client authentication must be enabled (i.e., set to On).
The checkbox Standard flow must be ticked.
The checkbox Direct access grants must be ticked.

Keycloak Client Authentication Configuration

Figure 5. Authentication settings of the Keycloak client

Configuring the client roles

In the Roles tab of the client configuration, click on the button Create role and add the following parameters for each user role needed in ProActive.

The property Role name specifies name of the role. It will be used later on when configuring ProActive privileges.
The property Composite should be set to false.
The property Description specifies description of the role.

Figure 6. Roles to be associated to the Keycloak client

Binding client roles to user groups

Once the client configuration is fulfilled, go to the section Groups which lists the groups of users available in Keycloak.
For each group of users that require access to ProActive, select the tab Role mapping and click on the button Assign role.
In the list of roles that will be displayed, select the set of roles to be granted for the considered group of users.

In the search menu, select Filter by client and enter the client name (e.g., ProActive) in order to easily find the roles defined above.

Figure 7. Binding client roles to user groups in Keycloak

4.7.2. Configuring ProActive server

In order to enable ProActive access control against a Keycloak server, the following configuration steps must be accomplished at ProActive side:

Configuring the authentication method for ProActive scheduler and Resource Manager

The Keycloak authentication method must be selected for the scheduler and resource manager, as described in Select authentication method.

Configuring ProActive properties for Keycloak

The configuration file config/authentication/keycloak.cfg must be edited in order the set the proper values of the following properties:

The property keycloak.url should be set to the URL of the Keycloak server (already configured for ProActive).
The property keycloak.https.allow_any_certificate must be set to true when the keycloak server operates in TLS mode and uses a non-trusted (e.g., self-signed) TLS/SSL certificate. Otherwise, this property is set to false.
The property keycloak.realm must take the name of the Keycloak realm in which the ProActive client application is configured.
The property keycloak.proactive.client_id must take as value the name of the ProActive client application in Keycloak.
The property keycloak.proactive.client_secret must take as value the secret credential of the ProActive client application in Keycloak.
The property keycloak.proactive.scope must take as value the name of a scope configured in Keycloak. A scope defines the set of user information/parameters to be included in a Keycloak access token. If the value is empty, the default scope in Keycloak is used.
The property keycloak.public.key must have as value the content of public encryption key of Keycloak (e.g., public RSA key). This key is used to decrypt the access token returned by Keycloak.
The property keycloak.principal_attribute is the parameter in Keycloak access token based on which the user name is determined (e.g., name or given_name).
The property keycloak.use_resource_role_mappings is set to true when the user roles are determined based on the information block resource_access in Keycloak access token. When it is set to false, the user roles are determined based on the information block realm_access in Keycloak access token.
Using the fallback to file authentication: ProActive can be configured to use simultaneously file-based authentication and Keycloak-based access control. In this scenario, ProActive Scheduler checks at first user credentials (login/password) and group memberships in the login and group files, as performed in the File login method. If the user or group attribution is not found in the login/group files, the user credentials and roles will be checked via Keycloak. There are three configuration options to control the fallback mechanism:
- pa.keycloak.authentication.fallback: If set to true (which is the default value), the user credentials will be checked first in the login file, and then in the Keycloak server. If set to false, only the Keycloak server will be queried.
- pa.keycloak.group.membership.fallback: If set to true (which is the default value), group memberships will be read from the Keycloak server if the user exists, and then from the group file. In that sense, a user will be affected to groups defined in both file and Keycloak server. If set to false, only the Keycloak server will be queried.
- pa.keycloak.tenant.membership.fallback: Must be always set to false for the current version of ProActive.

Configuring the granted privileges for user roles

ProActive has a set of predefined roles (also called groups), each one of them grants certain access rights and privileges to the users having the considered role. If the roles configured in Keycloak (for the ProActive client) do not already exist, the ProActive server’s administrator has to create these roles with the appropriate privileges, as explained in the section creating new groups.

The name of the roles to be created in ProActive must be the same as those configured in Keycloak client for ProActive. For example, if the role proactive-workflow-engineer is configured in Keycloak, the role/group to be created in proActive must be as follows:

grant principal org.ow2.proactive.authentication.principals.GroupNamePrincipal "proactive-workflow-engineer" {
  // list of granted privileges
}

Specific example of Keycloak configuration

Below is an example of Keycloak configuration which provides sample values for the Keycloak properties to be configured in the file config/authentication/keycloak.cfg.

# URL of a Keycloak server configured for ProActive authentication
keycloak.url=https://keycloak.activeeon.com:9443

# When Keycloak URL uses the HTTPS protocol, this parameter indicates which kind of TLS certificate to consider for the Keycloak server.
# True means that any certificate can be used including the self-signed ones.
# False means that only valid certificates generated by trusted Certification Authorities are allowed.
keycloak.https.allow_any_certificate=true

# Keycloak realm in which ProActive client is configured
keycloak.realm=activeeon

# Keycloak client credentials (note that ProActive is the client application of Keycloak)
keycloak.proactive.client_id=ProActive
keycloak.proactive.client_secret=w9jvqPPTMvxxxyyyrnessKxt7nCs75NpDZ

# Scope of the information to be returned by Keycloak for the ProActive client.
# An empty value considers the scopes defined for ProActive client in Keycloak.
keycloak.proactive.scope=

# Keycloak RSA public key used to decrypt the access tokens
keycloak.public.key=MIIBIjANBgkutvbS5w0BAQEFAAOCAQ8AMGGBCgKCAQEAsmVExiiZgtnrN7DrPRu6Cx/H1Fx4wOCy1b9sT4w7YjSLfX0d5GN51bDiz5olcvgBNvNIXGXm3Vax8OoOJdIYaq2EiVa0p0ATpzYFIZkKFOkiuriDJd3TpxuJyWIoEg1zUZHNPw0zvFftZ24HByZr51qm/QMBQNvj2tMcPvBgm5tUbIe1bzR3VURXA41+p0Cs4I1kjhWV5B95+SLbxcj/+TMkUt5BVDxPy0nJixM+4zZ/R48OCTpPUnLrGJw+/z5vgbqwwH+ekpI+L08v0wUYmMkqfnykZxOyhtoCUD66pojDoPO60JPDj5qtbiydHwV7DsULvbc406wEe0pN8hKO5QIDAQAB

# The attribute in Keycloak access token to be used to extract the user/principal name
keycloak.principal_attribute=name

# Use the resource_access block in Keycloak access token to extract the user/principal roles
keycloak.use_resource_role_mappings=true

# Authentication fallback Properties
# check user password and group membership in login and group files
# (as performed in FileLogin method), if user is not found in Keycloak.
#
# It use pa.[scheduler.core|RM].defaultloginfilename to authenticate user and
# pa.[scheduler.core|RM].defaultgroupfilename files to check user membership.
# In any case, no fallback is performed if user is found in Keycloak but has entered an incorrect password).
# true or false.
pa.keycloak.authentication.fallback=true

# check user group membership in group file (as performed in FileLogin method)
# if user is not found in its corresponding requested Keycloak group.
# true or false.
pa.keycloak.group.membership.fallback=true

# check user tenant membership in tenant file (as performed in FileLogin method)
# if tenant attribute is not defined or is empty for the corresponding requested Keycloak user.
# true or false.
pa.keycloak.tenant.membership.fallback=false

5. User Permissions

All users authenticated in the ProActive server have their own role according to granted permissions. ProActive uses the standard Java Authentication and Authorization Service (JAAS) to address these needs.

The file PROACTIVE_HOME/config/security.java.policy-server allows configuring fine-grained access for all users, e.g. who has the right to:

Deploy ProActive Nodes
Execute jobs
Pause the Scheduler
etc

5.1. Default Groups/Roles

The following groups are defined by default in the security.java.policy-server file:

user: can submit a job to the scheduler and control this job execution (pause, kill, etc). Cannot interact with other users jobs.
guests: can only log into the scheduler/resource manager, but cannot interact with it.
scheduleradmins: has the same permissions as user but can also control other users' job execution. Can manage the scheduler life-cycle such as pausing or stopping the scheduler.
nsadmins: this group gives the permission to create, edit or remove Node Sources in the resource manager.
providers: this group gives only the permission to add nodes to the resource manager or to an existing Node Source.
rmcoreadmins : this group gives full interaction capability with the resource manager (create, edit, remove Node Sources, add nodes, etc).
server-admins : this group combine the rights of rmcoreadmins and scheduleradmins groups.
admin : this group gives full permission on every aspect, this group is deprecated and is kept for backward compatibility purpose only (use the above server-admins for a global administrator instead).
watchers : this group allows receiving notification events from the scheduler and resource manager, but cannot perform any action. It is mostly used internally.
citizen-ds : this group gives enough permissions to software power users who can do moderate data analysis tasks (studio, catalog-portal, workflow-execution, job-analytics).
expert-ds : this group gives permissions to expert data scientists to collect and clean a wide range of data, and to analyze it to solve business problems (studio, scheduler, catalog-portal, workflow-execution, service-automation, job-analytics, job-gantt, job-planner-calendar-def, job-planner-calendar-def-workflows, job-planner-execution-planning, job-planner-gantt-chart, notification-portal).

Creating New Groups

New groups can be defined by copying an existing group definition and edit its permissions.

The PROACTIVE_HOME/tools/copy-role command can be used to duplicate an existing group inside the security.java.policy-server file. See Copy Role Command.

Any modification to the security.java.policy-server file is taken into account periodically by the ProActive server (by default every 30 seconds). This period can be configured in PROACTIVE_HOME/config/rm/settings.ini:pa.rm.auth.policy.refreshperiod.seconds and PROACTIVE_HOME/config/scheduler/settings.ini:pa.scheduler.auth.policy.refreshperiod.seconds.

A group is defined using the following syntax:

grant principal org.ow2.proactive.authentication.principals.GroupNamePrincipal "mygroup" {
  // roles definition
}

Each group must at least contain the following permissions (without them, the ProActive software will not work properly):

// AuthPermission is requires for those who would like to access any mbean
permission javax.security.auth.AuthPermission "getSubject";
permission java.lang.RuntimePermission "setContextClassLoader";
permission javax.management.MBeanPermission "-#-[-]", "queryNames";
permission javax.management.MBeanPermission "javax.management.MBeanServerDelegate#-[JMImplementation:type=MBeanServerDelegate]", "addNotificationListener";
permission javax.management.MBeanPermission "org.ow2.proactive.scheduler.core.jmx.mbean.MyAccountMBeanImpl#*[*:*]", "*";
permission javax.management.MBeanPermission "org.ow2.proactive.scheduler.core.jmx.mbean.RuntimeDataMBeanImpl#*[*:*]", "*";
permission javax.management.MBeanPermission "org.ow2.proactive.resourcemanager.core.jmx.mbean.MyAccountMBeanImpl#*[*:*]", "*";
permission javax.management.MBeanPermission "org.ow2.proactive.resourcemanager.core.jmx.mbean.RuntimeDataMBeanImpl#*[*:*]", "*";

// Granting file reading permission i.e. to read RRD database via JMX
permission java.io.FilePermission "<<ALL FILES>>", "read";

// API - access to database
permission java.sql.SQLPermission "setLog";
permission java.sql.SQLPermission "callAbort";
permission java.sql.SQLPermission "setSyncFactory";
permission java.sql.SQLPermission "setNetworkTimeout";
permission java.util.PropertyPermission "*", "read, write";
permission java.net.SocketPermission "*", "accept, connect, listen, resolve";

In addition to this common set of permissions, the group should contain permissions related to Resource Manager access, Scheduler access, Dataspace access and Other Components access.

5.2. Resource Manager Permissions

5.2.1. Core Resource Manager Permissions

Resource Manager permissions are defined using the following syntax:

permission org.ow2.proactive.permissions.ServiceRolePermission "org.ow2.proactive.resourcemanager.core.RMCore.<role>"

<role> can be one of:

basic: login, logout.
read: read access to the Resource Manager information such as node sources, nodes, etc.
write: ability to use nodes in the Resource Manager (necessary to submit workflows).
provider: ability to add/remove nodes to/from the Resource Manager.
nsadmin: ability to create, edit, deploy, undeploy or remove node sources.
admin: access to some specific administrative features such as server thread dump, loggers level change.
*: shortcut for all above permissions.

When a group needs to have access to several of the above permissions, it must be specified in separate lines, for example:

permission org.ow2.proactive.permissions.ServiceRolePermission "org.ow2.proactive.resourcemanager.core.RMCore.basic"
permission org.ow2.proactive.permissions.ServiceRolePermission "org.ow2.proactive.resourcemanager.core.RMCore.read"
permission org.ow2.proactive.permissions.ServiceRolePermission "org.ow2.proactive.resourcemanager.core.RMCore.write"

5.2.2. Resource Manager Permissions Advanced Syntax

ServiceRolePermission is a coarse-grained permission definition, in order to fine tune permissions, it is possible to use either of the following syntax:

permission org.ow2.proactive.permissions.MethodCallPermission "org.ow2.proactive.resourcemanager.core.RMCore.<method_name>";
permission org.ow2.proactive.permissions.DeniedMethodCallPermission "org.ow2.proactive.resourcemanager.core.RMCore.<method_name>";

MethodCallPermission is an additive permission definition, which allows a member of the group to call the Java method <method_name> of class org.ow2.proactive.resourcemanager.core.RMCore.

DeniedMethodCallPermission is a subtractive permission definition, which prevents a member of the group from calling the Java method <method_name> of class org.ow2.proactive.resourcemanager.core.RMCore.

The description of org.ow2.proactive.resourcemanager.core.RMCore methods is beyond the scope of this documentation.

5.2.3. Additional Resource Manager Permissions

Additionally, administrative roles should include the following permissions:

permission javax.management.MBeanPermission "org.ow2.proactive.resourcemanager.core.jmx.mbean.AllAccountsMBeanImpl#*[*:*]", "*";
permission javax.management.MBeanPermission "org.ow2.proactive.resourcemanager.core.jmx.mbean.ManagementMBeanImpl#*[*:*]", "*";

5.3. Scheduler Permissions

5.3.1. Core Scheduler Permissions

Scheduler permissions are defined using the following syntax:

permission org.ow2.proactive.permissions.ServiceRolePermission "org.ow2.proactive.scheduler.core.SchedulerFrontend.<role>";

<role> can be one of:

basic: login, logout.
read: read access to scheduler information (list of jobs, etc).
write: ability to submit, pause, resume, kill or remove a job.
admin: access to some specific administrative features such as pause, freeze and stop the scheduler, accounting information, etc.
*: shortcut for all above permissions.

5.3.2. Scheduler Permissions Advanced Syntax

ServiceRolePermission is a coarse-grained permission definition, in order to fine tune permissions, it is possible to use either of the following syntax:

permission org.ow2.proactive.permissions.MethodCallPermission "org.ow2.proactive.scheduler.core.SchedulerFrontend.<method_name>";
permission org.ow2.proactive.permissions.DeniedMethodCallPermission "org.ow2.proactive.scheduler.core.SchedulerFrontend.<method_name>";

MethodCallPermission is an additive permission definition, which allows a member of the group to call the Java method <method_name> of class org.ow2.proactive.scheduler.core.SchedulerFrontend.

5.3.3. Additional Scheduler Permissions

In addition to these standard set of permissions, a few special permissions are defined:

permission org.ow2.proactive.scheduler.permissions.HandleOnlyMyJobsPermission "true";

Mandatory for any user that submits workflows, defines whether each member of the group can interact (kill, pause, resume, etc) with other users jobs (false) or only with the jobs that belong to the user (true).

permission org.ow2.proactive.scheduler.permissions.OtherUsersJobReadPermission;

Optional permission that allows a group of users to have read-only access to other users jobs. With this permission enabled, a user will be able to read the details of other users' jobs (such as logs, results, etc.) but will not be able to perform an operation that will modify the job execution (pause, kill, etc.).

permission org.ow2.proactive.scheduler.permissions.ChangePriorityPermission "1,2,3";

Mandatory for any user that submits workflows, defines the set of Job Priorities that the group is allowed to use in submitted workflows.

Possible values are:

0: Idle
1: Lowest
2: Low
3: Normal
4: High
5: Highest

When a member tries to submit a workflow with a priority not included in the defined set, an error will occur and the submission will be cancelled.

permission org.ow2.proactive.scheduler.permissions.HandleJobsWithGenericInformationPermission "<gi_definitions>";

Optional permission that allows a group of users to interact with any job that contain the mentioned Generic Information.

<gi_definitions> can either be empty or contain a comma separated list of generic information pairs, such as gi1=value1,gi2=value2.

When more than one generic information is specified, then a member can interact with jobs which contain all generic information specified with their values. A job which contain a single generic information in the list will not be elected.

permission org.ow2.proactive.scheduler.permissions.ConnectToResourceManagerPermission;

Administrative permission that allows linking the scheduler to an existing resource manager.

permission org.ow2.proactive.scheduler.permissions.ChangePolicyPermission;

Administrative permission that allows changing the scheduler’s Scheduling Policy.

permission org.ow2.proactive.scheduler.permissions.TenantAllAccessPermission;

Administrative permission that allows access to all tenants when tenant isolation is enabled. See Tenants.

Finally, the following permissions are needed for administrative access to the scheduler:

permission javax.management.MBeanPermission "org.ow2.proactive.scheduler.core.jmx.mbean.AllAccountsMBeanImpl#*[*:*]", "*";
permission javax.management.MBeanPermission "org.ow2.proactive.scheduler.core.jmx.mbean.ManagementMBeanImpl#*[*:*]", "*";

5.4. Dataspace Permissions

Access to the Global Space through Workflow Studio portal or Workflow Execution portal is controlled with the following permission:

permission org.ow2.proactive.permissions.ServiceRolePermission "org.ow2.proactive_grid_cloud_portal.dataspace.RestDataspaceImpl.global.<role>";

<role> can be one of:

read: read access to the dataspace (list contents, download file).
write: write access to the dataspace (create folder, delete folder, upload file).
*: shortcut for all above permissions.

Similarly, access to the User Space through Workflow Studio portal or Workflow Execution portal is controlled with the following permission:

permission org.ow2.proactive.permissions.ServiceRolePermission "org.ow2.proactive_grid_cloud_portal.dataspace.RestDataspaceImpl.user.*";

5.5. Other Components Permissions

In addition to Resource Manager and Scheduler, a few permissions can be defined for other components.

5.5.1. Portals access

permission org.ow2.proactive_grid_cloud_portal.common.PortalAccessPermission "<portal_list>";

Mandatory permission that defines which portals are accessible for members of the group.

<portal_list> is either * to allow accessing all portals or a comma-separated list of portal names:

rm: allow access to the Resource Manager portal.
scheduler: allow access to the Scheduler portal.
studio: allow access to the Workflow Studio portal.
automation-dashboard: allow access to the Automation Dashboard portal, which contain the following portals (note that in order to access any of the portal below, automation-dashboard must be accessible):
- workflow-execution: allow access to the Workflow Execution portal.
- catalog: allow access to the Catalog portal.
- health-dashboard: allow access to the Health Dashboard, part of the Analytics portal.
- job-analytics: allow access to Job Analytics, part of the Analytics portal.
- job-gantt: allow access to the Job Gantt, part of the Analytics portal.
- node-gantt: allow access to the Node Gantt, part of the Analytics portal.
- job-planner-calendar-def: allow access to Calendar Definition, part of the Job Planner portal.
- job-planner-calendar-def-workflows: allow access to Calendar Associations, part of the Job Planner portal.
- job-planner-execution-planning: allow access to the Execution Planning, part of the Job Planner portal.
- job-planner-gantt-chart: allow access to the Gantt Chart, part of the Job Planner portal.
- event-orchestration: allow access to the Event Orchestration portal.
- service-automation: allow access to the Service Automation portal.
- notification-portal: allow access to the Notification portal.

5.5.2. Service Automation permissions

permission org.ow2.proactive.permissions.PcaAdminPermission;

This permission gives administrative access to Service Automation. A member of a group with such permission can interact with any service (not only owned services).

5.5.3. Job Planner permissions

permission org.ow2.proactive.scheduler.permissions.JobPlannerAllAccessPermission;

This permission gives administrative access to Job Planner. A member of a group with such permission can interact with any calendar association (not only owned associations).

5.5.4. Notification Service permissions

permission org.ow2.proactive.permissions.NotificationAdminPermission;

This permission gives administrative access to Notification Service, allowing to delete any notification.

6. Deploy ProActive Nodes

In the ProActive Scheduler, Computational Resources are managed by the ProActive Resource Manager component.

The ProActive Resource Manager supports ProActive Nodes aggregation from heterogeneous environments. As a node is just a process running somewhere, the process of communication to such nodes is unified. The only part which has to be defined is the procedure of nodes deployment which could be quite different depending on infrastructures and their limitations. After installation of the server and node parts, it is possible to configure an automatic nodes deployment. Basically, you can tell the Resource Manager how to launch nodes and when.

In the ProActive Resource Manager, all the computational resources (Compute Hosts) are managed as a set of Node Sources. Each node source contains a set of ProActive Nodes which shares the same deployment mechanism and access policy.

Examples of a Node Source:

a cluster with SSH access where nodes are available from 9 a.m. to 9 p.m.
nodes from Amazon EC2 available permanently for users from group 'cloud'.

As shown in the previous screenshot, the ProActive Resource Manager provides a Resource Manager Web Interface to show and manage all the node sources with its nodes.

Adding a cluster of Compute Hosts to the ProActive Resource Manager requires to create a node source. Its typical process involves copying the necessary ProActive binary files to all those Compute Hosts, running ProActive Nodes on the hosts, and connecting the nodes to the ProActive Resource Manager. There are two principal ways of doing that:

Launch a process of ProActive Node on the Compute Host and connect it to the ProActive Resource Manager
Initiate the deployment from the ProActive Resource Manager

If you are not familiar with ProActive Workflows & Scheduling product you may want to try the first method as it’s directly executed on the Compute Hosts. Combined with ProActive Agents which control and limit the resources utilization on the Compute Hosts, it gives you the same result as the second method.

The second method commands the ProActive Resource Manager to remotely launch the process of ProActive Nodes on Compute Hosts. The method implies that you have a remote access to Compute Hosts (e.g., SSH access).

The ProActive Nodes can be deployed on various kinds of computational resources, such as:

the local Server Host (i.e., the machine where the ProActive server is installed)
remote compute servers
Cloud Platforms (e.g., Amazon Web Services, OpenStack, GoogleComputeEngine)
an existing cluster managed by another scheduler (e.g., SLURM, LSF, PBS)

A node source is defined by two parts of configurations: Node Source Infrastructure and Node Source Policy. Each Node Source Infrastructure defines the deployment mechanism for a specific kind of computational resources. To support the deployment of these various computational resources, the ProActive Scheduler provides a set of supported Node Source Infrastructures. The Node Source Policy controls the resource usage. See the section Control the resource usage for more details. This section concentrates on the deployment mechanism of ProActive Nodes which is defined by the Node Source Infrastructure.

In the ProActive Resource Manager, there is always a default node source consisting of a Default Infrastructure and Static Policy. It is not able to deploy nodes anywhere but makes it possible to add existing nodes to the RM.

When you connect the Compute Hosts to the ProActive Resource Manager through the first method, a node source of Default Infrastructure is automatically created. Through the second method, you just need to choose the proper node source infrastructure and fill out its required parameters.

6.1. Deploy ProActive Nodes manually

Let’s take a closer look at the first method described above. To deploy a ProActive Node from the Compute Host, you have three ways to do it:

using proactive-node command
using node.jar
using proactive-node-autoupdate command

6.1.1. Using proactive-node command

To use proactive-node command, you need to unpack the ProActive node release archive inside the Compute Host and run the following command

$ PROACTIVE_HOME/bin/proactive-node -r pnp://SCHEDULER_ADDRESS:64738

where -r option is used to specify the URL of the Resource Manager. You can find this URL in the output of the ProActive Scheduler (the Resource Manager URL). For example, in case of the following sample output of the ProActive server command:

The router created on localhost:33647
Starting the scheduler...
Starting the resource manager...
The resource manager with 4 local nodes created on pnp://localhost:41303/
The scheduler created on pnp://localhost:41303/
The web application /automation-dashboard created on http://localhost:8080/automation-dashboard
The web application /studio created on http://localhost:8080/studio
The web application /job-analytics created on http://localhost:8080/job-analytics
The web application /notification-service created on http://localhost:8080/notification-service
The web application /cloud-automation-service created on http://localhost:8080/cloud-automation-service
The web application /catalog created on http://localhost:8080/catalog
The web application /proactive-cloud-watch created on http://localhost:8080/proactive-cloud-watch
The web application /scheduler created on http://localhost:8080/scheduler
The web application /scheduling-api created on http://localhost:8080/scheduling-api
The web application /connector-iaas created on http://localhost:8080/connector-iaas
The web application /rm created on http://localhost:8080/rm
The web application /rest created on http://localhost:8080/rest
The web application /job-planner created on http://localhost:8080/job-planner
*** Get started at http://localhost:8080 **

You should use pnp://localhost:41303/ as the Resource Manager URL.

You can also use discovery to let the ProActive Node find the URL to connect to on its own. Simply run proactive-node without any parameter to use discovery. It uses broadcast to retrieve the URL so this feature might not work depending on your network configuration.

If you want to run multiple tasks at the same time on the same machine, you can either start a few proactive-node processes or start multiple nodes from the same process using the -w command line option.

6.1.2. Using node.jar

It is also possible to launch a ProActive Node without copying the ProActive Server release to the Compute Host:

Open a browser on the Compute Host.
Navigate to the Resource Manager Web Interface. You can find the URL in the output of the ProActive Scheduler (the Resource Manager web application URL, e.g., http://localhost:8080/rm).
Use default demo/demo account to access the Resource Manager Web Interface.
Click on 'Portal→Launch' to download node.jar.
Click on 'Portal→Create Credentials' and download your credential file.
Create a Node Source using the infrastructure DefaultInfrastructureManager.
Run it:

$ java -Dproactive.communication.protocol=pnp -jar node.jar -f CREDENTIAL_FILE -s NAME

Where NAME is the name of the node source. It should connect to the ProActive Resource Manager automatically using the discovery mechanism otherwise you might have to set the URL to connect to with the -r parameter.

If you would like to execute several Tasks at the same time on one host, you can either launch several ProActive Node process or use the -w parameter to run multiple nodes in the same process. A node executes one Task at a time.

6.1.3. Using proactive-node-autoupdate command

This method is a combination of the two above, it starts a node from the command line, and makes sure the node library (node.jar) is synchronized with the latest server version.

The proactive-node-autoupdate command acts as a bootstrap and spawns a new java virtual machine with the up-to-date library classpath.

Launching the node is similar to the proactive-node command, additional options must be specified to allow the download of the node.jar from the server:

$ PROACTIVE_HOME/bin/proactive-node-autoupdate -r pnp://SCHEDULER_ADDRESS:64738 -nju http://SCHEDULER_ADDRESS:8080/rest/node.jar -njs /tmp/node.jar -nja

where -nju option is used to specify the http URL of the node.jar. You can find this URL in the output of the ProActive Scheduler, the node.jar URL is built by appending /rest/node.jar to the base HTTP URL of the server.

-nju option is used to specify where the node.jar will be stored locally on the machine.

Finally, -nja option, when enabled, specifies that the proactive-node-autoupdate will be always up, it means that it will automatically restart when the node terminates (for example in case of server upgrades).

6.2. Deploy ProActive Nodes via Agents

In your production environment you might want to control and limit the resources utilization on some or all Compute Hosts, especially if those are desktop machines where people perform their daily activities. Using ProActive Agent you can:

Control the number of ProActive Node processes on each Compute Host
Launch ProActive Nodes automatically when a Compute Host starts
Restart ProActive Nodes if they fail for some reason and reconnect them to ProActive Resource Manager

ProActive Agents exists for both Linux and Windows operating systems.

6.2.1. ProActive Linux Agent

The ProActive Agent for Linux can be downloaded from ActiveEon’s website.

To install the ProActive Agent on Debian based distributions (the archive name will vary depending on the version and architecture):

sudo apt install ./proactive-agent.deb

Some extra dependencies might be required, to install them: sudo apt-get -f install

To install the ProActive Agent on Redhat based distributions (the archive name will vary depending on the version and architecture):

sudo rpm -i proactive-agent.rpm

Some extra dependencies might be required, to install them you need to install each dependency individually using sudo yum install

By default the Linux Agent will launch locally as many nodes as the number of CPU cores available on the host minus one. The URL to use for connecting nodes to the Resource Manager can be configured in /opt/proactive-agent/config.xml, along with several other parameters. If no URL is set, then broadcast is used for auto discovery. However, this feature might not work depending on your network configuration. Consequently, it is recommended to set the Resource Manager URL manually.

Logs related to the Agent daemon are located in /opt/proactive-agent/proactive-agent.log.

Binaries, configuration files and logs related to ProActive nodes started by the agent are available at /opt/proactive-node/, respectively in subfolders dist/lib, config and logs.

Configuring the Linux Agent

To configure the agent behaviour:

Stop the Linux Agent
```
sudo /etc/init.d/proactive-agent stop
```
Update the config.xml file in /opt/proactive-agent folder

In case there is no such directory, create the directory. Then, create the config.xml file or create a symbolic link
```
sudo mkdir -p /opt/proactive-agent
sudo ln -f -s <path-to-config-file> /opt/proactive-agent/config.xml
```
Change the ownership of the config.xml file
```
sudo chown -Rv proactive:proactive config.xml
```
If the group named "proactive" does not exist, create the group and add "proactive" user to the group:
```
sudo groupadd proactive
sudo usermod -ag proactive proactive
```
Start the Linux Agent
```
sudo /etc/init.d/proactive-agent start
```

Uninstalling the Linux Agent

On Debian based distributions:

sudo apt purge proactive-agent

On Redhat based distributions:

sudo yum remove proactive-agent

6.2.2. ProActive Windows Agent

The ProActive Windows Agent is a Windows Service: a long-running executable that performs specific functions and which is designed to not require user intervention. The agent is able to create a ProActive Node on the local machine and to connect it to the ProActive Resource Manager.

After being installed, it:

Loads the user’s configuration
Creates schedules according to the working plan specified in the configuration
Spawns a ProActive Node that will run a specified java class depending on the selected connection type. 3 types of connections are available:
- Local Registration - The specified java class will create a ProActive Node and register it locally.
- Resource Manager Registration - The specified java class will create a ProActive Node and register it in the specified Resource Manager, thus being able to execute java or native tasks received from the Scheduler. It is is important to note that a ProActive Node running tasks can potentially spawn child processes.
- Custom - The user can specify his own java class.
Watches the spawned ProActive Nodes in order to comply with the following limitations:
- RAM limitation - The user can specify a maximum amount of memory allowed for a ProActive Node and its children. If the limit is reached, then all processes are automatically killed.
- CPU limitation - The user can specify a maximum CPU usage allowed for a ProActive Node and its children. If the limit is exceeded by the sum of CPU usages of all processes, they are automatically throttled to reach the given limit.
Restarts the spawned ProActive Node in case of failures with a timeout policy.

6.2.3. Install Agents on Windows

The ProActive Windows Agent installation pack is available on the ProActive website. Run the setup.exe file and follow instructions. When the following dialog appears:

Figure 8. Windows agent installation

Specify the directory that will contain the configuration file named PAAgent-config.xml, note that if this file already exists in the specified directory it will be re-used.
Specify the directory that will contain the log files of the ProActive Agent and the spawned runtimes.
Specify an existing, local account under which the ProActive Nodes will be spawned. It is highly recommended to specify an account that is not part of the Administrators group to isolate the ProActive Node and reduce security risks.
The password is encrypted using Microsoft AES Cryptographic Provider and only Administrators have access permissions to the keyfile (restrict.dat) this is done using the SubInACLtool.
If the specified account does not exist the installation program will prompt the user to create a non-admin account with the required privileges.

Note that the ProActive Agent service is installed under LocalSystem account, this should not be changed, however it can be using the services.msc utility. ('Control Panel→Administrative Tools→Services')
If you want that any non-admin user (except guest accounts) be able to start/stop the ProActive Agent service check the "Allow everyone to start/stop" box. If this option is checked the installer will use the SubInACL tool. If the tool is not installed in the Program Files\Windows Resource Kits\Tools directory the installer will try to download its installer from the official Microsoft page.
The installer will check whether the selected user account has the required privileges. If not follow the steps to add these privileges:
1. In the 'Administrative Tools' of the 'Control Panel', open the 'Local Security Policy'.
2. In 'Security Settings', select 'Local Policies' then select 'User Rights Assignments'.
3. Finally, in the list of policies, open the properties of 'Replace a process-level token' policy and add the needed user. Do the same for 'Adjust memory quotas for a process'. For more information about these privileges refer to the official Microsoft page.

At the end of the installation, the ProActive Agent Control utility should be started. This next section explains how to configure it.

To uninstall the ProActive Windows Agent, simply run 'Start→Programs→ProActiveAgent→Uninstall ProActive Agent'.

6.2.4. Configure Agents on Windows

To configure the Agent, launch 'Start→Programs→ProActiveAgent→AgentControl' program or click on the notify icon if the "Automatic launch" is activated. Double click on the tray icon to open the ProActive Agent Control window. The following window will appear:

Figure 9. Windows Agent GUI

From the ProActive Agent Control window, the user can load a configuration file, edit it, start/stop the service and view logs. A GUI for editing is provided (explained below). Even if it is not recommended, you can edit the configuration file by yourself with your favorite text editor.

It is also possible to change the ProActive Nodes Account using the 'Change Account' button.

When you click on 'GUI Edit', the following window appears:

Figure 10. General configuration

In the general tab, the user can specify:

The ProActive Scheduler location.
The JRE location (the Java Runtime Environment is provided with the agent archive under this path %programfiles(x86)%\ProActive Agent\schedworker\jre).
The numbers of Runtimes and Nodes (the number of spawned processes and the number of ProActive Nodes per process).
The numbers of workers (the number of ProActive Nodes allocated in the Compute Host)
The JVM options. Note that if the parameter contains ${rank}, it will be dynamically replaced by the ProActive Node rank starting from 0.
The On Runtime Exit script. A script executed after a ProActive Node exits. This can be useful to perform additional cleaning operation. Note that the script receives as parameter the PID of the ProActive Node.
The user can set a memory limit that will prevent the spawned processes to exceed a specified amount of RAM. If a spawned process or its child process requires more memory, it will be killed as well as its child processes. Note that this limit is disabled by default (0 means no limit) and a ProActive Node will require at least 128 MBytes.
It is possible to list all available network interfaces by clicking on the "Refresh" button and add the selected network interface name as a value of the proactive.net.interface property by clicking on "Use" button. See the ProActive documentation for further information.
The user can specify the protocol (PNP or PAMR) to be used by the ProActive Node for incoming communications. If the selected protocol is PAMR, then the user has to add -Dproactive.pamr.router.address JVM property.
To ensure that a unique port is used by a ProActive Node, the initial port value will be incremented for each node process and given as value of the -Dproactive.SELECTED_PROTOCOL.port JVM property. If the port chosen for a node is already used, it is incremented until an available port number is found.

Clicking on the 'Connection' tab, the window will look like this:

Figure 11. Connection configuration

In the 'Connection' tab, the user can select between three types of connections:

Local Registration - creates a local ProActive node and registers (advertises) it in a local RMI registry. The node name is optional.
Resource Manager Registration - creates a local ProActive node and registers it in the specified Resource Manager. The mandatory Resource Manager’s URL must comply with the following format: protocol://host:port/. In case of the PAMR protocol, the URL is of the form pamr://resource_manager_id, which is by default pamr://0 (default value of the resource manager id). The PAMR router host and port are provided using JVM properties. The node name and the node source name are optional. Since the Resource Manager requires authentication, the user specifies the file that contains the credential. If no file is specified the default one located in %USERPROFILE%\.proactive\security folder is used.
Custom - the user specifies his own java starter class and the arguments to be given to the main method. The java starter class must be in the classpath when the ProActive Node is started.

Finally, clicking on the "Planning" tab, the window will look like this:

Figure 12. Planning configuration

In the Planning Tab, depending on the selected connection type, the agent will initiate it according to a weekly planning where each plan specifies the connection start time as well as the working duration. The agent will end the connection as well as the ProActive Nodes and its child processes when the plan duration has expired.

Moreover, it is possible to specify the ProActive Node Priority and its CPU usage limit. The behavior of the CPU usage limit works as follows: if the ProActive Node spawns other processes, they will also be part of the limit so that if the sum of CPU% of all processes exceeds the user limit they will be throttled to reach the given limit. Note that if the Priority is set to RealTime the CPU % throttling will be disabled.

The "Always available" makes the agent to run permanently with a Normal Priority and Max CPU usage at 100%.

6.2.5. Launching Windows Agent

Once you have configured the agent, you can start it clicking on the "Start" button of the ProActive Agent Control window. However, before that, you have to ensure that ProActive Scheduler has been started on the address you specified in the agent configuration. You do not need to start a node since it is exactly the job of the agent.

Once started, you may face some problems. You can realise that an error occurred by first glancing at the color of the agent tray icon. If everything goes right, it should keep the blue color. If its color changes to yellow, it means that the agent has been stopped. To see exactly what happened, you can look at the runtime log file located into the agent installation directory and named Executor<runtime number>Process-log.txt.

The main troubles you may have to face are the following ones:

You get an access denied error: this is probably due to your default java.security.policy file which cannot be found. If you want to specify another policy file, you have to add a JVM parameter in the agent configuration. A policy file is supplied in the scheduling directory. To use it, add the following line in the JVM parameter box of the agent configuration (Figure 5.3, “Configuration Editor window - General Tab ”):

-Djava.security.policy=PROACTIVE_HOME/config/security.java.policy-client

You get an authentication error: this is probably due to your default credentials file which cannot be found. In the "Connection" tab of the Configuration Editor (Figure 5.4, “Configuration Editor window - Connection Tab (Resource Manager Registration)”), you can choose the credentials file you want. You can select, for instance, the credentials file located at PROACTIVE_HOME/config/authentication/scheduler.cred or your own credentials file.
The node seems to be well started, but you cannot see it in the Resource Manager interface: in this case, make sure that the port number is the good one. Do not forget that the runtime port number is incremented from the initial ProActive Resource Manager port number. You can see exactly on which port your runtime has been started looking at the log file described above.
The agent does not seem to be starting at all and the log file Executor<runtime number>Process-log.txt (see above) is empty. This issue has been observed on some Windows 10 environments. When facing this problem, open Windows Event Viewer and look at errors associated with application ProActiveAgent. Follow the procedure Windows 10 Patch if you see the following error:

JobManagement.JobException: The management process is running inside a Job, it will not be able to assign child processes to a new Job.
   à JobManagement.JobObject.ProbeForRunningInJob()
   à JobManagement.JobObject.CreateJob()
   à JobManagement.JobObject..ctor(String name)
   à ProActiveAgent.ProActiveRuntimeExecutor..ctor(CommonStartInfo commonStartInfo, Int32 rank)
   à ProActiveAgent.ExecutorsManager..ctor(AgentType configuration)
   à ProActiveAgent.WindowsService.OnStart(String[] args)
   à System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

Windows 10 Patch

In order to solve the issue mentioned above, please do the following:

Download the patch zip from the following link and expand this archive inside an empty folder.
Close the agent GUI if it’s open. In order to do so, in the Windows notification area (aka system tray), find the ProActive Agent icon , right-click on it and select Close. A warning message will appear (simply confirm).
Replace existing files in the ProActive Agent installation folder C:\Program Files (x86)\ProActiveAgent with patch files copied from the expanded zip.
Start the ProActive Agent again. If the agent does not start and display the following error, start the service called ProActiveAgent manually from Windows services user interface (services.exe).

Figure 13. Start failure after Windows 10 Patch

6.2.6. Automate Windows Agent Installation

The automated installation can be run through a command line (cmd.exe) with administrator priviledge. This can be useful to trigger installation. In order to launch the silent installation, the /S option must be passed in the command line to the ProActive Windows Agent installer. Several options are required, such as the user and password.

Here is an example of automated installation command:

ProactiveAgent-8.3.0-standalone-x64-setup.exe /S /USER=proactive /PASSWORD=proactive

Optionally you can add a domain name.

ProactiveAgent-8.3.0-standalone-x64-setup.exe /S /USER=proactive /PASSWORD=proactive /DOMAIN=mydomain

You can also activate compatibility mode from the command line if you have any problems:

set __COMPAT_LAYER=WINXPSP3
ProactiveAgent-8.3.0-standalone-x64-setup.exe /S /USER=proactive /PASSWORD=proactive

Here is the full list of command line options which can be passed to the installer:

/S

Run silently without graphical interface, uninstall any previous installation.

/UN

Run the uninstall only.

/ALLOW

Allow all users to control the service.

/USER=user

Associate user account to manage proactive agent service, create user if it does not exist.

/PASSWORD=pass

Define password for the proactive agent user.

/DOMAIN=dom

Specify dom as the windows domain, by default domain will be set to the machine host name.

/LOG_DIR=path

Define path where the agent log files, default is $PROGRAMFILES\ProActiveAgent\logs.

/CONFIG_DIR=path

Define path where the configuration files are stored, default is $PROGRAMFILES\ProActiveAgent\config.

/USE_ACC

Use current user’s account home as /CONFIG_DIR and /LOG_DIR.

6.2.7. Configuring Linux Or Windows Agents For Auto-Update

The Linux or Windows Agents can be configured to automatically synchronize the node libary (node.jar) with the ProActive server.

The behavior is similar to the proactive-node-autoupdate command, the main JVM will act as a bootstrap and spawn a child JVM with the up-to-date library in its classpath.

In order to enable auto-update, you need to edit the agent configuration file. This file location is:

Linux Agent: /opt/proactive-agent/config.xml
Windows Agent: C:\Program Files (x86)\ProActiveAgent\config\PAAgent-config.xml

The following changes must be performed:

Change the <javaStarterClass> to org.ow2.proactive.resourcemanager.updater.RMNodeUpdater
Add the following <jvmParameters>
- node.jar.url: URL of the node jar (similar to the nju option of the proactive-node-autoupdate command)
- node.jar.saveas: path used to store the node.jar file locally (similar to the njs option of the proactive-node-autoupdate command)
Other JVM properties specified will be forwarded to the spawned JVM, but non-standard options such as -Xmx will not be forwarded. In order to do so, you must declare them using the following syntax:

   <param>-DXtraOption1=Xmx2048m</param>
   <param>-DXtraOption2=Xms256m</param>
   ...

6.3. Deploy ProActive Nodes from the ProActive Resource Manager Interface

The second way of deploying ProActive Nodes is to create Node Sources from the ProActive Resource Manager Interface.

ProActive Resource Manager provides three ways to create a Node Source:

Use the Resource Manager Web Interface ('Add Node Source' menu).
Use the REST API
Use the Command Line

Whichever the way you use to create a node source, ultimately you just need to choose a Node Source Infrastructure and a Node Source Policy that defines rules and limitations of nodes' utilization. Then according to the chosen node source infrastructure and policy, you need to fill out the required parameters.

The choice of Infrastructure Manager depends on the type of your Compute Hosts. The following part of this section presents how to deploy ProActive Nodes on various types of Compute Hosts:

the local Server Host
remote Compute Hosts accessible via SSH
cloud platforms
clusters managed by other schedulers

6.3.1. Deploy local nodes on the ProActive server

In order to deploy the ProActive nodes locally (i.e., on the server host where the ProActive server is running), you can just create a node source with the Local Infrastructure.

The previous screenshot shows how to create a local node source in the Resource Manager Web Interface.

To create a local node source from the command line:

$ PROACTIVE_HOME/bin/proactive-client --createns localNodeSourceName -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.LocalInfrastructure credentialsPath numberOfNodes timeout javaProperties

For example, the complete command to create a local node source with 7 nodes:

$ $PROACTIVE_HOME/bin/proactive-client -createns local_node_source -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.LocalInfrastructure $PROACTIVE_HOME/config/authentication/rm.cred 7 30000 "" -policy org.ow2.proactive.resourcemanager.nodesource.policy.StaticPolicy 'ALL' 'ALL'

See Local Infrastructure reference for details of each parameter.

6.3.2. Deploy ProActive nodes via SSH

In order to create an SSH Node Source you should first configure an SSH access from the server to the Compute Hosts that does not require password. Then create a text file (refered to as HOSTS_FILE below) containig the hostnames of all your Compute Hosts. Each line shoud have the format:

HOSTNAME NODES_COUNT

where NODES_COUNT is the number of ProActive Nodes to start (corresponds to the number of Tasks that can be executed in parallel) on the corresponding host. Lines beginning with # are comments. Here is an example:

# you can use network names
host1.mydomain.com 2
host2.mydomain.com 4
# or ip addresses
192.168.0.10 8
192.168.0.11 16

Then using this file create a Node Source either from the Resource Manager Web Interface or from the command line:

$ PROACTIVE_HOME/bin/proactive-client -createns SSH_node_source --infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.SSHInfrastructure HOSTS_FILE 60000 3 5000 "" /home/user/jdk/bin/java PROACTIVE_HOME Linux "" config/authentication/rm.cred

Don’t forget to replace HOSTS_FILE and PROACTIVE_HOME with the corresponding values.

See SSH Infrastructure reference for details on each parameter.

6.3.3. Deploy ProActive Nodes on Cloud Platforms

The ProActive Resource Manager can automate the deployment of ProActive Nodes on most popular Cloud Platforms. Currently the following Cloud Platforms are supported:

Amazon Web Service EC2
Amazon Web Service Auto Scaling
OpenStack
VMWare
Google Compute Engine
Microsoft Azure
Microsoft Azure Scale Set
Kubernetes

Unlike directly deploying on the compute servers, on the Cloud Platforms one or more virtual machines usually need to be deployed prior to launching a ProActive Node. Each virtual machine corresponds an instance in a cloud Node Source.

So the definition of most cloud Node Source Infrastructures have the following three parameters:

numberOfInstances: specifies the total number of virtual machines to be created on this node source.
numberOfNodesPerInstance: specifies the total number of ProActive Nodes to be deployed in each created virtual machine.
image: specifies which image is used to deploy the virtual machine (e.g., Linux or Windows).

The general process of deploying ProActive Nodes on a Cloud Platform is:

first, create the required number of virtual machines with the specified image
then, on each virtual machine, download the ProActive node.jar and install the required middleware (e.g., java)
finally, on each virtual machine, launch the specified number of ProActive Nodes (i.e., numberOfNodesPerInstance) using node.jar

See Node Source Infrastructure reference for details of each Cloud Node Source Infrastructure parameters.

6.3.4. Deploy ProActive Nodes dynamically via other schedulers (PBS, SLURM, …)

This functionality is also called Meta-Scheduling.

If an existing cluster is available in your organization and this cluster is managed by a native scheduler such as SLURM, LSF or PBS, you may want to execute transparently on this cluster ProActive Workflows and Tasks.

As ProActive Tasks can be executed only on ProActive Nodes, it is necessary to deploy ProActive Nodes on the cluster.

Dynamic deployment of ProActive Nodes on a cluster to execute ProActive Tasks is handled by defining a Native Scheduler Node Source containing three components:

The Native Scheduler Infrastructure: a Node Source Infrastructure which allows to interact with a native scheduler to deploy ProActive Nodes.
The Native Scheduler Policy: a Node Source Policy which interacts with the Native Scheduler Infrastructure to request ProActive Nodes deployment dynamically based on the ProActive Scheduler pending queue.
The Native Scheduler Scheduling Policy: a Scheduling Policy which allows the execution of ProActive Tasks on ProActive Nodes provisioned by the Native Scheduler Policy.

The provisioning of ProActive Nodes is controlled by a specific Generic Information "NS" which can be defined at Task or Job level.

When this generic information is configured for a Task, a ProActive Node will be dynamically deployed on the cluster to execute the Task. This ProActive Node will be associated with the Task and will only accept this Task execution. When the Task terminates, the ProActive Node will also be terminated.

When this generic information is configured for a Job, all Tasks contained in the Job will execute on the cluster. Nodes will be created dynamically to execute Tasks of this Job, with a similar Task/Node association.

The behavior of the Meta-Scheduling feature is summarized on the following diagram (example for PBS integration):

Figure 14. PBS Integration

The following paragraphs explain how to configure the Native Scheduler Infrastructure, Native Scheduler Policy, Native Scheduler Scheduling Policy and execute ProActive tasks using a Native Scheduler Node Source.

Glossary

Cluster: a group of tightly coupled nodes managed by a native scheduler.

Cluster Node: a computer, part of a cluster used to execute cluster jobs.

Native Scheduler: A software which dispatch cluster jobs on cluster nodes, also known as Cluster Manager.

Cluster Job: A running command executed on a cluster by a native scheduler. Also known as Batch Job.

Cluster Job Id: Identifier representing a cluster job inside a native scheduler.

Head Node: A specific cluster node where the native scheduler server runs.

Cluster User: A linux operating system account registered on the cluster.

ProActive Scheduler User: An account registered in the ProActive Scheduler, the account is only registered inside the ProActive Scheduler and does not necessarily match an operating system account.

ProActive Scheduler Process User: The operating system account which started the ProActive Scheduler server process.

Native Scheduler Node Source Configuration

Using the Resource Manager Web Interface, you can create a Node Source used to acquire ProActive Nodes from a Native Scheduler.

From The drop down menu, select NativeSchedulerInfrastructure and NativeSchedulerPolicy.

See the reference Native Scheduler Infrastructure and Native Scheduler Policy for detailed explanation of the node source parameters.

When the node source is created, it will be activated as other node sources (LocalInfrastructure, SSHInfrastructure, etc), but no ProActive Node will appear.

This is expected as the Native Scheduler node source is dynamic, it will only create ProActive Nodes when specific conditions are met.

Configure NativeSchedulerSchedulingPolicy

After creating the Native Scheduler Node Source, it is necessary as well to change the ProActive Scheduling Policy to use the NativeSchedulerSchedulingPolicy.

This policy ensures that tasks are executed on appropriate ProActive Nodes when using a Native Scheduler Node Source.

In order to do that, edit the file PROACTIVE_HOME/config/scheduler/settings.ini and change the following line:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.ExtendedSchedulerPolicy

to:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.NativeSchedulerSchedulingPolicy

Match User Accounts With the Cluster

Submission of jobs to the native scheduler is done by default using a SSH connection.

When a ProActive Task belonging to a given user Alice needs to execute on a NativeScheduler node source, a SSH connection will be performed with the login and password of the Alice user registered in the ProActive Scheduler.

Accordingly, this login/password combination must correspond to a real user on the cluster head node.

Please refer to the User Authentication section in order to manage ProActive Users.

Execute Tasks on a Native Scheduler Node Source

The Generic Information "NS" allows a ProActive Task to be executed on a Native Scheduler node source.

It must contain the name of the target node source. For example, to submit a Task on the "PBS" Node Source:

<task name="PBS_Task">
    <description>
        <![CDATA[ Execute this Task in the PBS node source. ]]>
    </description>
    <genericInformation>
        <info name="NS" value="PBS"/>
    </genericInformation>
    <scriptExecutable>
        <script>
          <code language="groovy">
            <![CDATA[
println "Hello World"
]]>
          </code>
        </script>
    </scriptExecutable>
</task>

The NS value can also be defined at the job level, in that case, every task of this job will be executed on the Native Scheduler node source:

<?xml version="1.0" encoding="UTF-8"?>
<job
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="urn:proactive:jobdescriptor:3.10"
     xsi:schemaLocation="urn:proactive:jobdescriptor:3.10 http://www.activeeon.com/public_content/schemas/proactive/jobdescriptor/3.10/schedulerjob.xsd"
    name="PBS_Job"
    priority="normal"
    onTaskError="continueJobExecution"
     maxNumberOfExecution="2"
>
  <genericInformation>
    <info name="NS" value="PBS"/>
  </genericInformation>
  <taskFlow>
    <task name="Groovy_Task">
      <description>
        <![CDATA[ The simplest task, ran by a groovy engine. ]]>
      </description>
      <scriptExecutable>
        <script>
          <code language="groovy">
            <![CDATA[
println "Hello World"
]]>
          </code>
        </script>
      </scriptExecutable>
    </task>
  </taskFlow>
</job>

It is also possible to define in a ProActive Task or Job the NS_BATCH generic information. This parameter allows to provide custom arguments to nsSubmitCommand.

For example, to submit a task on the PBS Node source, using a specific PBS queue and reserve for this task 2 cluster nodes with 2 cpu each:

<task name="PBS_Task">
  <description>
    <![CDATA[ Runs on the PBS node source on queue1, using 2 nodes * 2 cpus ]]>
  </description>
  <genericInformation>
    <info name="NS" value="PBS"/>
    <info name="NS_BATCH" value="-q queue1 -lnodes=2:ppn=2"/>
  </genericInformation>
  <scriptExecutable>
    <script>
      <code language="groovy">
         <![CDATA[
println "Hello World"
]]>
      </code>
    </script>
  </scriptExecutable>
</task>

Finally, the NS_PRE generic information can be used to enter a single line command which will be executed in the cluster just before the ProActive node starts. It can be used for example to load linux modules required by the task execution.

Native Scheduler Node Life Cycle

As soon as tasks containing the NS generic information are pending, the target Native Scheduler node source will try to deploy ProActive Nodes to execute them.

The node will first appear in Deploying state. If some error occurs prior to the nsSubmitCommand execution (SSH connection, command syntax), the node state will change to Lost, with some explanation about the failure displayed.

If the node remains in Deploying state, it is possible to monitor the job execution on the native scheduler itself, by logging into the head node, and use the native scheduler command tools.

Example using the PBS qstat command:

root@osboxes:/tmp/activeeon_enterprise-node-linux-x64-8.1.0-SNAPSHOT/logs# qstat

Job ID                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
241.osboxes                STDIN            osboxes                0 R batch

root@osboxes:/tmp/activeeon_enterprise-node-linux-x64-8.1.0-SNAPSHOT/logs# qstat -f 241

Job Id: 241.osboxes
    Job_Name = STDIN
    Job_Owner = osboxes@osboxes
    job_state = R
    queue = batch
    server = osboxes
    Checkpoint = u
    ctime = Fri May  4 10:13:06 2018
    Error_Path = osboxes:/tmp/activeeon_enterprise-node-linux-x64-8.1.0-SNAPSH
	OT/logs/Node-osboxes_852t0.out
    exec_host = osboxes/0
    Hold_Types = n
    Join_Path = oe
    Keep_Files = n
    Mail_Points = a
    mtime = Fri May  4 10:13:06 2018
    Output_Path = osboxes:/tmp/activeeon_enterprise-node-linux-x64-8.1.0-SNAPS
	HOT/logs/Node-osboxes_852t0.out
    Priority = 0
    qtime = Fri May  4 10:13:06 2018
    Rerunable = True
    Resource_List.walltime = 01:00:00
    Resource_List.nodes = 1
    Resource_List.nodect = 1
    Resource_List.neednodes = 1
    session_id = 6486
    substate = 42
    Variable_List = PBS_O_QUEUE=batch,PBS_O_HOME=/home/osboxes,
	PBS_O_LOGNAME=osboxes,
	PBS_O_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/b
	in:/usr/games:/usr/local/games,PBS_O_MAIL=/var/mail/osboxes,
	PBS_O_SHELL=/bin/bash,PBS_O_LANG=fr_FR.UTF-8,
	PBS_O_WORKDIR=/home/osboxes,PBS_O_HOST=osboxes,PBS_O_SERVER=osboxes
    euser = osboxes
    egroup = osboxes
    hashname = 241.osboxes
    queue_rank = 237
    queue_type = E
    comment = Job started on Fri May 04 at 10:13
    etime = Fri May  4 10:13:06 2018
    submit_args = -o /tmp/activeeon_enterprise-node-linux-x64-8.1.0-SNAPSHOT/l
	ogs/Node-osboxes_852t0.out -e /tmp/activeeon_enterprise-node-linux-x64
	-8.1.0-SNAPSHOT/logs/Node-osboxes_852t0.out -j oe
    start_time = Fri May  4 10:13:06 2018
    Walltime.Remaining = 3587
    start_count = 1
    fault_tolerant = False
    job_radix = 0
    submit_host = osboxes
    init_work_dir = /home/osboxes
    request_version = 1

When the deployment is successful, the ProActive Node state will change to Free and shortly after to Busy as soon as the associated task will be deployed on the node. After the task completes, the node will be removed and the cluster job will be cancelled using the nsKillCommand.

Troubleshooting

If the cluster job is in running state, but the ProActive Node associated remains in Deploying state, it probably means there is a connection issue between the ProActive Node and the Resource Manager. If the default ProActive Network Protocol is used (PNP), it is necessary to have a two way connection between the cluster and ProActive server. You can refer to the network protocols documentation for more info.

To troubleshoot node deployment, you can:

inspect the output of the cluster job if provided in the nsSubmitCommand
add the following log4j loggers to the ProActive Scheduler Server config/log/server.properties

log4j.logger.org.ow2.proactive.scheduler.policy=DEBUG
log4j.logger.org.ow2.proactive.scheduler.util=DEBUG
log4j.logger.org.ow2.proactive.nativescheduler=DEBUG

inspect the ProActive Scheduler server logs.

6.4. Node Source Infrastructures

A Node Source Infrastructure, also called Infrastructure Manager, is responsible for deploying ProActive Nodes inside a defined Node Source. In most of the cases, it knows how to launch a set of ProActive Nodes on a remote machine. For instance the SSH infrastructure manager connects via SSH to a remote machine and launches nodes.

When creating a node source, a Node Source Infrastructure and a Node Source Policy must be chosen.

New Node Source Infrastructures can be dynamically plugged into the Resource Manager. In order to do that, it is required to add classes extending the InfrastructureManager class in the server class path and update the infrastructure implementation list in the configuration file (PROACTIVE_HOME/config/rm/nodesource/infrastructures).

Node Sources can be created using the Resource Manager portal or the command line (see Command Line Examples for more information).

When using the command line, the order of parameters is important, and each infrastructure section below describes the correct parameter order. Each section shows as well the fully qualified name of the infrastructure, to pass on the command line. On the Resource Manager portal, the order of parameters displayed on the interface is slightly different as parameters are grouped into categories.

6.4.1. Default Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.DefaultInfrastructureManager

Default Infrastructure is designed to be used with ProActive agent. It cannot perform an automatic deployment but any users (including an agent) can add already existing nodes into it. In order to create a node source with this infrastructure, use the Resource Manager portal or run the following command:

$ PROACTIVE_HOME/bin/proactive-client --createns defaultns -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.DefaultInfrastructureManager

This infrastructure has no parameter.

6.4.2. Local Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.LocalInfrastructure

Local Infrastructure can be used to start nodes locally, i.e, on the host running the Resource Manager. In order to create a node source with this infrastructure, use the Resource Manager portal or run the following command:

$PROACTIVE_HOME/bin/proactive-client --createns localns -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.LocalInfrastructure credentialsPath numberOfNodes timeout javaProperties

credentialsPath - (Optional) The absolute path of the credentials file used to set the provider of the nodes. By default it will use the connected user credentials.
maxNodes - The number of nodes to deploy.
nodeTimeout - The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
paProperties - Java options appended to the command used to start the node on the remote host.

6.4.3. SSH Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.SSHInfrastructure

This infrastructure allows to deploy nodes over SSH. It is deprecated in favor of SSH Infrastructure V2 This infrastructure needs 10 arguments, described hereafter:

hostsList - Path to a file containing the hosts on which resources should be acquired. This file should contain one host per line, described as a host name or a public IP address, optionally followed by a positive integer describing the number of runtimes to start on the related host (default to 1 if not specified). Example:
```
rm.example.com
test.example.net 5
192.168.9.10 2
```
nodeTimeOut - The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
maxDeploymentFailure - Maximum number of failed attempts to deploy on host before discarding it.
waitBetweenDeploymentFailures - The time in milliseconds that the Resource Manager waits before retrying to acquire a node for which the deployment failed before.
sshOptions - Options you can pass to the SSHClient executable ( -l mysuser to specify the user for instance ).
javaPath - Path to the java executable on the remote hosts.
schedulingPath - Path to the Scheduler Node installation directory on the remote hosts.
targetOs - One of 'LINUX', 'CYGWIN' or 'WINDOWS' depending on the machines' ( in Hosts List file ) operating system.
javaOptions - Java options appended to the command used to start the node on the remote host.
rmCredentialsPath - (Optional) The absolute path of the 'rm.cred' file to make the deployed nodes able to register to the Resource Manager ( config/authentication/rm.cred ). By default it will use the currently connected user credentials.

6.4.4. SSH Infrastructure V2

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.SSHInfrastructureV2

This infrastructure allows to deploy nodes over SSH. This infrastructure needs 17 arguments, described hereafter:

hostsList - Path to a file containing the hosts on which resources should be acquired. This file should contain one host per line, described as a host name or a public IP address, optionally followed by a positive integer describing the number of runtimes to start on the related host (default to 1 if not specified). Example:
```
rm.example.com
test.example.net 5
192.168.9.10 2
```
nodeTimeOut - The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
maxDeploymentFailure - Maximum number of failed attempts to deploy on host before discarding it.
waitBetweenDeploymentFailures - The time in milliseconds that the Resource Manager waits before retrying to acquire a node for which the deployment failed before.
sshPort - Port used for the SSH connection.
sshUsername - Username of the SSH connection.
sshPassword - Password of the SSH connection.
sshPrivateKey - Alternately, use a private key to authenticate the SSH connection.
sshOptions - Options you can pass to the SSHClient executable.
javaPath - Path to the java executable on the remote hosts.
schedulingPath - Path to the Scheduler Node installation directory on the remote hosts.
targetOs - One of 'LINUX', 'CYGWIN' or 'WINDOWS' depending on the machines' ( in Hosts List file ) operating system.
javaOptions - Java options appended to the command used to start the node on the remote host.
deploymentMode - Specifies how the ProActive node command is started. The deploymentMode can take the following values:
- autoGenerated: when this mode is selected the command to start the ProActive node will be generated automatically. As a result, the ssh call to the nodes will start the ProActive nodes on the infrastructure without modifications to the startup command. This mode is selected by default.
- useStartupScript: starts the ProActive node using the script in the variable %startupScriptStandard%, allowing the user to modify the startup command of the hosts. This mode uses the ProActive node agent identified in the %schedulingPath% variable and the Java Runtime Environment identified in the %javaPath% variable.
- useNodeJarStartupScript: enables connecting to the SSHInfrastructureV2 by launching node.jar. This mode uses %nodeJarUrl% and %startupScriptWitNodeJarDownload% variables to generate the startup command.
if deploymentMode field was set to an empty string, the autoGenerated mode will be automatically selected.
nodeJarUrl - The full URL path to download the ProActive node.jar on each host added to the hostsList. The URL have to be accessible from the hosts. For example, try.activeeon.com/rest/node.jar. Used only when useNodeJarStartupScript is selected.
startupScriptStandard - Nodes startup script to launch the ProActive nodes using a ProActive node agent. The script by default locates Java Runtime Environment and node agent directory using %javaPath% and %schedulingPath% variables respectively. The user can modify or extend this script to execute commands on the host before or after the Proactive node startup. Used only when useStartupScript is selected.
startupScriptWitNodeJarDownload - Nodes startup script to launch the ProActive nodes using node.jar. To run ProActive nodes, this script is also expected to download and install the Java Runtime Environment, then download and execute ProActive node.jar, if they are not already provided by the host. It uses %nodeJarUrl% variable to get the full URL path for downloading the node.jar file. The user can modify or extend this script to execute commands on the host before or after the ProActive node startup. Used only when useNodeJarStartupScript is selected.

If the deployment mode autoGenerated is selected, the startup scripts will be disregarded.

6.4.5. CLI Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.CLIInfrastructure

This generic infrastructure allows to deploy nodes using deployment script written in arbitrary language. The infrastructure just launches this script and waits until the ProActive node is registered in the Resource Manager. Command line infrastructure could be used when you prefer to describe the deployment process using shell scripts instead of Java. Script examples can be found in PROACTIVE_HOME/samples/scripts/deployment. The deployment script has 4 parameters: HOST_NAME, NODE_NAME, NODE_SOURCE_NAME, RM_URL. The removal script has 2 parameters: HOST_NAME and NODE_NAME.

This infrastructure needs 7 arguments, described hereafter:

hostsList - Path to a file containing the hosts on which resources should be acquired. This file should contain one host per line, described as a host name or a public IP address, optionally followed by a positive integer describing the number of runtimes to start on the related host (default to 1 if not specified). Example:
```
rm.example.com
test.example.net 5
192.168.9.10 2
```
nodeTimeOut - The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
maxDeploymentFailure - Maximum number of failed attempts to deploy on host before discarding it.
waitBetweenDeploymentFailures - The time in milliseconds that the Resource Manager waits before retrying to acquire a node for which the deployment failed before.
interpreter - Path to the script interpreter (bash by default).
deploymentScript - A script that launches a ProActive Node and registers it to the RM.
removalScript - A script that removes the node from the Resource Manager.

6.4.6. EC2 Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.AWSEC2Infrastructure

The Elastic Compute Cloud, aka EC2, is an Amazon Web Service, that allows its users to use machines (instances) on demand on the cloud. An EC2 instance is a Xen virtual machine, running on different kinds of hardware, at different prices, but always paid by the hour, allowing lots of flexibility. Being virtual machines, instances can be launched using custom operating system images, called AMI (Amazon Machine Image). For the Resource Manager to use EC2 instances as computing nodes, a specific EC2 Infrastructure as well as AMI creation utilities are provided.

Pre-Requisites

The configuration of the AWS EC2 infrastructure is subjected to several requirements.

The administrator needs both an AWS access key and an AWS secret access key to enable ProActive to authenticate against AWS. Please refer to the AWS documentation to learn how to get them.
The permissions for using a supported AWS region. Currently we only support the following AWS regions: ap-south-1, eu-west-3, eu-west-2, eu-west-1, ap-northeast-2, ap-northeast-1, sa-east-1, ca-central-1, ap-southeast-1, ap-southeast-2, eu-central-1, us-east-1, us-east-2, us-west-1, us-west-2.
Enough resources quota (including the number of instances, subnets, key pairs, etc) for creating your node source underlying instances. The number of maximum allocated instances is to be configured by the administrator, but has also to comply with the limitation of the regions. Detailed information is available on Amazon EC2 Limits Documentation.
The administrator must be in possession of a valid AMI ID to provision instances operating ProActive node. Currently we only support the Linux operating system for AWS.
The Resource Manager should be accessible from the AWS cloud. Please consider replacing PNP by PAMR as communication protocol if the Resource Manager is located behind a NAT gateway. You can get additional information about the PAMR protocol in the section Installation on a Cluster with Firewall.

Infrastructure Configuration

To use a cluster of AWS instances as a computing resource for the ProActive scheduler, the administrator has to create a node source in the Resource Manager with the AWSEC2Infrastructure profile. The configuration form exposes the following fields:

awsKey: Your AWS access key ID (e.g., AKIAIOSFODNN7EXAMPLE) which is retrieved in the section Pre-Requisites.
awsSecretKey: Your AWS secret access key (e.g., wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY) retrieved in the section Pre-Requisites.
numberOfInstances: Total number of AWS instances (i.e., Virtual Machines) to create for this infrastructure.
numberOfNodesPerInstance: Total number of ProActive Nodes to deploy in each created AWS instance.

If all the nodes of an AWS instance are removed, the instance will be terminated. For more information on the terminated state in AWS please see AWS Terminating Instances.
image: Defines which VM image (AMI) will be used to create the AWS instances. The value to provide is the AWS region together with the unique AMI ID, for example: eu-west-1/ami-bff32ccc. This is an optional property. When no image is specified by the user, the default value eu-west-3/ami-03bca18cb3dc173c9 is used. Currently only Linux images are supported for the EC2 infrastructure. Please use AWS Autoscaling Infrastructure for Windows images. Make sure the image is from a supported region (listed in the section Pre-Requisites) and using Linux operating system. More information about finding a Linux AMI ID is available on AWS EC2 guide for finding an AMI.
vmUsername: Specifies the default username of the VM image. This username is used to connect to AWS instances. Note that you should specify a username already provided in your VM image. If you are using a public AMI provided by Amazon Web Service, you can find the default username of your image here. This is an optional property. When it is not specified, the default value ubuntu is used.
vmKeyPairName: Defines the name of the AWS key pair for accessing AWS instances. This is an optional property. When it’s not specified, a default key pair will be created or reused in the given region of the deployment. If specified, the key pair must exit in AWS in the region of deployment, and the vmPrivateKey must be specified as well.
vmPrivateKey: Defines the AWS private key file (a .pem file) corresponding to 'vmKeyPairName' for accessing AWS instances. This is an optional property. When it’s not specified, a default key pair will be created or reused in the given region of the deployment. If specified, the name of the key pair (vmKeyPairName) to which this private key belongs to must be specified as well.
vmType: The AWS VM type required for each VM (e.g. t3.medium). An optional parameter, if not provided, selecting a specific vmType will be ignored. It only supports VM types supported by Apache Jclouds. The list of supported instances can be found at Apache Jclouds Supported Instance Types. vmType accepts the AWS instance type corresponding to each of these constants (for example m4.large, corresponding to the jclouds constant InstanceType.M4_LARGE will be accepted). See AWS EC2 Instance Types for the full list of AWS instance types.

Once the vmType parameter is set, it will override the parameters set for ram and cores.

ram: The minimum required amount of RAM (expressed in Mega Bytes) for each AWS instance that needs to be created. This is an optional property. If not provided, the default value 2048 will be used.
cores: The minimum required amount of virtual cores for each AWS instance that needs to be created. This is an optional property. If not provided, the default value 2 will be used.

If the combination between RAM and CORES does not match any existing AWS instance type, then the closest to the specified parameters will be selected.
securityGroupIds: This option allows you to specify the id(s) of the security group(s) configured as a virtual firewall(s) to control inbound and outbound traffic for the EC2 instances hosting the ProActive nodes. This is an optional property. If not provided, a security group will be automatically created and used. More information regarding Amazon EC2 Security Group available on AWS EC2 Security Groups.
subnetId: The subnetId option allows you to launch the ProActive nodes on EC2 instances, which will run into an existing subnet added to a specific Virtual Private Cloud. This is an optional property. More information regarding Amazon EC2 Virtual Private Cloud (Amazon VPC) available on AWS EC2 Virtual Private Cloud and Amazon EC2 Subnet available on AWS EC2 Virtual Private Cloud and Subnet
rmHostname: The hostname or the public IP address of the Resource Manager. This address needs to be accessible from the AWS cloud. For example, try.activeeon.com.
connectorIaasURL: Connector-iaas is a ProActive service used to interact with IaaS like AWS. By default it runs on the following URL rmHostname/connector-iaas. For example, http://try.activeeon.com:8080/connector-iaas.
nodeJarURL: The full URL path of the node.jar to download the ProActive node.jar on each new created AWS instance. The URL needs to be accessible from the AWS cloud. For example, try.activeeon.com/rest/node.jar.
additionalProperties: Additional Java command properties to be added when starting each ProActive node JVM in AWS instances (e.g. -Dpropertyna me=propertyvalue). This is an optional property.
nodeTimeout: The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
startupScript: VM startup script to launch the ProActive nodes. To run the ProActive nodes, this script is also expected to download and install the Java Runtime Environment, then download and and execute ProActive node.jar, if they are not already provided by the VM image. The script can use some arguments. For example, it can use %nodeJarUrl% to represent the full URL path for downloading the node.jar file. These arguments will be interpreted by EC2 Infrastructure later. This is an optional property. If left blank, the script is automatically generated for the Linux OS. Here is the default value of startupScript:
```
mkdir -p /tmp/node && cd /tmp/node
if ! type -p jre/bin/java; then wget -nv -N https://s3.amazonaws.com/ci-materials/Latest_jre/jre-8u382b05-linux-x64.tar.gz; tar -xf jre-8u382b05-linux-x64.tar.gz; mv jre1.8.0_382b05/ jre; fi
wget -nv %nodeJarUrl%
nohup jre/bin/java -jar node.jar -Dproactive.communication.protocol=%protocol% -Dpython.path=%jythonPath% -Dproactive.pamr.router.address=%rmHostname% -D%instanceIdNodeProperty%=%instanceId% -r %rmUrl% -s %nodeSourceName% %nodeNamingOption% -v %credentials% -w %numberOfNodesPerInstance% %additionalProperties% &
```
spotPrice: This parameter is only used when you want to deploy ProActive nodes on AWS spot instances. It specifies the maximum price that you are willing to pay per hour per instance (your bid price). When the spot price is too low to be satisfied within the node-running timeout (20 minutes by default, configurable through connector-iaas.jclouds.compute.timeout.node-running in the configuration file dist/war/connector-iaas/WEB-INF/classes/application.properties), the node source deployment will be failed and the related spot requests will be cancelled. This is an optional property. More information regarding Amazon spot instances available on AWS Spot Instances. For the current spot instance prices, see Amazon EC2 Spot Instances Pricing.

Using this configuration, you can start a ProActive Resource Manager and Scheduler using the /bin/proactive-server script. An AWS EC2 NodeSource can now be added using the Create Node Source panel in the Resource Manager Portal or the command line interface:

$ PROACTIVE_HOME/bin/proactive-client --createns ec2 -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.AWSEC2Infrastructure awsKey awsSecretKey numberOfInstances numberOfNodesPerInstance image vmUsername vmKeyPairName vmPrivateKey vmType ram cores securityGroupIds subnetId rmHostname connectorIaasURL nodeJarURL additionalProperties nodeTimeout startupScript spotPrice

As AWS is a paying service, when the ProActive server is stopped normally (without removing the created infrastructure), all the created AWS instances will be terminated. When the ProActive server is restarted, these instances will be re-configured as per the previous settings.

If ProActive server is forcibly killed, the created AWS instances will not be terminated. And, when ProActive server is restarted, the infrastructure will be re-configured as per the previous settings. If the instances were deleted at the AWS side, they will be re-created and re-configured.

6.4.7. AWS Autoscaling Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.AwsAutoScalingInfrastructure

Similarly to the EC2 Infrastructure, the AWS Autoscaling Infrastructure operates AWS EC2 service to provide computing nodes to the Resource Manager. However, it implements a different instance management strategy that reduces the delay of node acquisition and node removal process and facilitates inter-node collaboration in the same cluster, thanks to the following changes:

The instances operating the nodes are allocated from a common instance template.
The nodes share the same networking infrastructure through a common Virtual Private Cloud (VPC). The infrastructure supports networking autoconfiguration if no parameter is supplied.

The node source using empty policy will not benefit from this latter management strategy. The deployment with empty policy doesn’t use the shared instance template and networking configuration.

Pre-Requisites

The configuration of the AWS Autoscaling infrastructure is subjected to several requirements.

The administrator needs both an AWS access key and an AWS secret access key to enable ProActive to authenticate against AWS. Please refer to the AWS documentation to learn how to get them.
The AWS region that will support the node cluster must not have reached its resources quota. It has to be able to allocate one instance template in every case situation. If the network autoconfiguration has to be be triggered, the region has to be able to provide one VPC, one subnet, one internet gateway and one security group. The number of maximum allocated instances is to be configured by the administrator, but has also to comply with the limitation of the regions. Detailed information is available in AWS documentation for VPC and Instances.
The administrator must be in possession of a valid AWS keypair and the ID of the Amazon Machine Image (AMI) to provision instances operating ProActive node. If the AMI to use does not propose the Linux operating system, the administrator must supply a provision script to (i) download a Java Runtime Environment (JRE), (ii) download ProActive Node agent (node.jar), (iii) and start up the ProActive agent.
Optionally, if networking autoconfiguration is not triggered, the administrator has to configure (i) a public VPC, (ii) a Subnet complying with with VPC CIDR configuration, (iii) an Internet gateway for that VPC and a (iv) Security Group authorizing HTTPS connection to Internet and PNP or PAMR connection to the Resource Manager. The administrator must be in possession of the IDs of the VPC, of the Subnet and of the Security Group.
The Resource Manager has to be accessible from the AWS cloud. Please consider replacing PNP by PAMR as communication protocol if the Resource Manager is located behind a NAT gateway. You can get additional information about the PAMR protocol in the section Installation on a Cluster with Firewall.

Infrastructure Configuration

To use a cluster of AWS instances as a computing resource for the ProActive scheduler, the administrator has to create a NodeSource in the Resource Manager with the AwsAutoScalingInfrastructure profile. The configuration form exposes the following fields:

vmGroupTagPrefix: Each instance prepared by the connector is flagged with the tag named vmGroupName. This tag is valued with this configuration option and the Node Source name. Concurrent ProActive schedulers can therefore operate concurrent clusters in the same AWS region, with the same node source name, provided that they diverge on the affected value to this parameter. This option is mandatory.
awsKey: This field must be filled with the content of the AWS key from the administrator. This option is mandatory.
awsSecretKey: The administrator must complete this field with the content of their secret access key. This option is mandatory.
maxVms: This parameter defines the number of maximum tolerated instances on the infrastructure: the instance allocations will be systematically blocked if the Resource Manager tries to overpass this threshold. This option is mandatory and cannot exceed 100.
defaultInstanceType: This parameter defines the instance type to use for AWS instance operating ProActive. This parameter should be choosen according to the expected processing to be performed on the node source. This option is mandatory and has to be filled after one AWS InstanceType name (e.g. t3.large).
amiId: The administrator defines in this field the ID of the AMI to use to bootstrap instance operating ProActive nodes. This option is mandatory, has to refer to an existing AMI in the region, and has to comply with AMI ID format.
publicSshKey: The administrator has to provide the name of the AWS keypair to be use to operate the instance supporting nodes. This option is mandatory, and must refer to an existing AWS keypair in the region.
defaultVpcId: This parameter can be filled with the ID of the VPC to use to operate instance operating nodes. If specified, this parameter has to refer to an existing VPC in the region and comply with the VPC ID format. If left blank, the connector will, first, try to get the default VPC ID in the specified region if set, otherwise it will trigger networking autoconfiguration.

defaultSubNetId: The administrator can define which subnet has to be attached to the the instance supporting nodes. If specified, this parameter has to refer to an existing subnet in the region affected to the specified VPC, and has to comply with the subnet ID format. This parameter has to be filled only if the defaultVpcId is also completed. Otherwise, this parameter has to be left blank to trigger networking autoconfiguration.

Please do not trigger networking autoconfiguration if you operate ProActive on AWS with PNP protocol. Otherwise, a new and distinct VPC will be used to operate the nodes created by the NodeSource, preventing their communication with the Resource Manager.

defaultSecurityGroup: This parameter receives the ID of the security group to spawn instances into. If this parameter does not meet the requirement regarding the provided VPC and subnet, a new security group will be generated by default and will be re-used if the same deployment scenario is repeated. This parameter is mandatory, and has to comply with the format of the ID of the AWS security groups.
region: The administrator specifies here the AWS region to allocate the cluster into. This parameter is mandatory and has to be configured after the name of an AWS region. Please see the related documentation to see the available region names.
rmUrl: This field receives the URL to access the Resource Manager from the nodes. The URL must comply with the specification of the communication protocols used by ProActive, and can therefore be prefixed with pnp://, pnps:// or pamr://. This parameter is mandatory.
rmHostname: This field is to be filled with the domain name or the IP of the host operating the Resource Manager. This option is mandatory.
nodeJarURL: URL used to download the node jar on the VM.
additionalProperties: Additional parameters to be used when invoking PA Agent (not used if an externalStartupScript is provided).
externalStartupScript: The administrator has to provide a script to configure AWS instances to work with ProActive. Usually, if not provided by the AMI, this script is expected to download the Java Runtime Environment and ProActive node.jar agent file. This field is expected to contain the content of the script, and start with a Shebang, as an AWS imposes it. If left blank, the script is automatically generated for the Linux OS.
maxNodePerVM: The administrator specifies the amount of nodes to be deployed on each AWS instance. This parameter is mandatory, and has to be an integer equal or greater than one.

Please ensure this parameter is aligned with the capacity of the specified instance type mentioned in the defaultInstanceType field.
deploymentTimeOut: This field contains the delay in seconds for a node to be deployed and to contact back the Resource Manager before being declared as lost. This parameter is mandatory.
cleanDelay: The administrator can define the periodicity in seconds for unused instance removal. This parameter is mandatory.
spotPrice: This parameter is only used when you want to deploy ProActive nodes on AWS spot instances. It specifies the maximum price that you are willing to pay per hour per instance (your bid price). If your bid price is lower than the current spot instance price for your specified instance type, the node source deployment will be failed immediately. This is an optional property. More information regarding Amazon spot instances available on AWS Spot Instances. For the current spot instance prices, see Amazon EC2 Spot Instances Pricing.

Using this configuration, you can start a Resource Manager and a Scheduler using the /bin/proactive-server script as explained in section Get Started. An AWS Autoscaling NodeSource can now be added using the Create Node Source panel in the Resource Manager or the command line interface:

$ PROACTIVE_HOME/bin/proactive-client --createns awsAutoScaling -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.AwsAutoscalingInfrastructure vmGroupTagPrefix awsKey awsSecretKey maxVms defaultInstanceType amiId publicSshKey defaultVpcId defaultSubNetId defaultSecurityGroup region rmUrl rmHostname nodeJarURL additionalProperties externalStartupScript maxNodePerVM deploymentTimeOut cleanDelay spotPrice

When ProActive server is stopped (without removing the created infrastructure), AWS instances will not be terminated. And when ProActive server is restarted, the infrastructure will be re-configured as per the previous settings. If the instances were deleted at the cloud side, they will be re-created and re-configured.

6.4.8. OpenStack Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.OpenstackInfrastructure

To use OpenStack instances as computing nodes, a specific OpenStack Infrastructure can be created using the Resource Manager. This section describes briefly how to make it.

First, you need to have an admin account on your OpenStack server. For more information see OpenStack users and tenants.
The creation of OpenStack Infrastructure asks for an authentication to the OpenStack server and a deployment of instances that will host ProActive nodes.

Use the proper admin username and password to fill in the properties username and password and perform the basic OpenStack authentication. Those two parameters should never change, except if you need for some reason to handle multiple OpenStack accounts.

For more information regarding OpenStack authentication mode see OpenStack authentication mode.

Other properties needed for the authentication are:

domain: The name of the domain to use that refers to the collection of projects and users defining administrative boundaries for managing Identity entities. For more information see OpenStack domain.
endpoint: The hostname or the IP address of the OpenStack server. This address needs to be accessible from the Resource Manager.
scopePrefix: The scope prefix to use. It can be project, projectId, domain or domainId.
scopeValue: The value of the scope prefix.
region: The Region for networks and compute resources to use.
identityVersion: The REST API version of OpenStack installation. For more information see OpenStack API Version.

Properties needed for the deployment of instance are:

image: Defines which image will be used to create the OpenStack instance. The value to provide is the unique image Id.
flavor: Defines the size of the instance. The value to provide is the flavor name or the flavor Id. For more information see OpenStack flavors.
networkId: Defines the identifier of the network to be attached to the instance. This value can be blank to leave the selection to OpenStack.
publicKeyName: Defines the name of the public key to use for a remote connection when the instance is created.

In order to use publicKeyName, the key pair needs to be created and imported first on the OpenStack server. For more information see OpenStack key pair management.
numberOfInstances: Total number of OpenStack instances to create for this infrastructure.
numberOfNodesPerInstance: Total number of ProActive Nodes to deploy in each OpenStack created instance.

If all the nodes of an OpenStack instance are removed, the instance will be terminated.

Other properties for the node deployment in the Create OpenStack Node Source are:

connectorIaasURL: Connector-iaas is a ProActive service used to interact with IaaS like OpenStack. By default it runs on the following URL rmHostname/connector-iaas.
rmHostname: The hostname or the public IP address of the Resource Manager. This address needs to be accessible from the OpenStack server.
downloadCommand: The command to download the ProActive node.jar. This command is executed in all the newly created OpenStack instances. The full URL path of the node.jar to download needs to be accessible from the OpenStack cloud.
additionalProperties: Additional Java command properties to be added when starting each ProActive node JVM in OpenStack instances (e.g. \"-Dpropertyname=propertyvalue\").
nodesInitDelay: Estimated startup time (expressed in millisecond) of the nodes, including the startup time of VMs. After this timeout expired, the nodes will be considered as lost.
startupScript: VM startup script to launch the ProActive nodes. To run the ProActive nodes, this script is also expected to download and install the Java Runtime Environment, then download and and execute ProActive node.jar, if they are not already provided by the VM image. The script can use some arguments. For example, it can use %nodeJarUrl% to represent the full URL path for downloading the node.jar file. These arguments will be interpreted by Openstack Infrastructure later. This is an optional property. If left blank, the script is automatically generated for the Linux OS. Here is the default value of startupScript:
```
mkdir -p /tmp/node && cd /tmp/node
if ! type -p jre/bin/java; then wget -nv -N https://s3.amazonaws.com/ci-materials/Latest_jre/jre-8u382b05-linux-x64.tar.gz; tar -xf jre-8u382b05-linux-x64.tar.gz; mv jre1.8.0_382b05/ jre; fi
wget -nv %nodeJarUrl%
nohup jre/bin/java -jar node.jar -Dproactive.communication.protocol=%protocol% -Dpython.path=%jythonPath% -Dproactive.pamr.router.address=%rmHostname% -D%instanceIdNodeProperty%=%instanceId% -r %rmUrl% -s %nodeSourceName% %nodeNamingOption% -v %credentials% -w %numberOfNodesPerInstance% %additionalProperties% &
```

Using this configuration, you can start a Resource Manager and a Scheduler using the /bin/proactive-server script. An OpenStack NodeSource can now be added using the Create Node Source panel in the Resource Manager or the command line interface:

$ PROACTIVE_HOME/bin/proactive-client --createns openstack -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.OpenStackInfrastructure username password endpoint rmHostname connectorIaasURL image flavor publicKeyName numberOfInstances numberOfNodesPerInstance downloadCommand additionalProperties nodesInitDelay startupScript

When ProActive server is stopped (without removing the created infrastructure), OpenStack instances will not be terminated. And when ProActive server is restarted, the infrastrucutre will be re-configured as per the previous settings. If the instances were deleted at the OpenStack Cloud side, they will be re-created and re-configured.

6.4.9. VMware Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.VMWareInfrastructure

To use VMware instances as computing nodes, a specific VMware Infrastructure can be created using the Resource Manager. This section describes briefly how to make it.

First, you need to have an admin account on your VMware server.For more information see VMware users.
Use the login and password information to fill in the properties vmware_username, vmware_password in the Create Node Source panel located in the Resource Manager. Those two parameters should never change, except if you need for some reason to handle multiple VMware accounts. Other properties in the Create Node Source are:
- endpoint: The hostname or the IP address of the VMware server. This address needs to be accessible from the Resource Manager.
- rmHostname: The hostname or the public IP address of the Resource Manager. This address needs to be accessible from the VMware server.
- connectorIaasURL: Connector-iaas is a service embedded in the Scheduler used to interact with IaaS like VMware. By default it runs on the following URL rmHostname/connector-iaas.
- image: Defines which image will be used to create the VMware instance. The value to provide is the VMware folder together with the unique image Id, for example: ActiveEon/ubuntu.
- minRam: The minimum required amount of RAM expressed in Mega Bytes for each VMware instance that needs to be created.
- minCores: The minimum required amount of virtual cores for each VMware instance that needs to be created.
  
  If the combination between RAM and CORES does not match any existing VMware instance type, then the closest to the specified parameters will be selected.
- vmUsername: Defines the username to log in the instance when it is created.
- vmPassword: Defines the password to log in the instance when it is created.
  
  The username and password are related to the image.
- numberOfInstances: Total number of VMware instances to create for this infrastructure.
- numberOfNodesPerInstance: Total number of ProActive Nodes to deploy in each VMware created instance.
If all the nodes of an VMware instance are removed, the instance will be terminated.

+
- downloadCommand: The command to download the ProActive node.jar. This command is executed in all the newly created VMware instances. The full URL path of the node.jar to download, needs to be accessible from the VMware cloud.
- additionalProperties: Additional Java command properties to be added when starting each ProActive node JVM in VMware instances (e.g. \"-Dpropertyname=propertyvalue\").

Using this configuration, you can start a Resource Manager and a Scheduler using the /bin/proactive-server script. An VMware NodeSource can now be added using the Create Node Source panel in the Resource Manager or the command line interface:

$ PROACTIVE_HOME/bin/proactive-client --createns vmware -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.VmwareInfrastructure username password endpoint rmHostname connectorIaasURL image ram cores vmusername vmpassword numberOfInstances numberOfNodesPerInstance downloadCommand additionalProperties

When ProActive server is stopped (without removing the created infrastructure), VMware instances will not be terminated. And when ProActive server is restarted, the infrastrucutre will be re-configured as per the previous settings. If the instances were deleted at the VMware server side, they will be re-created and re-configured.

6.4.10. GCE Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.GCEInfrastructure

Google Compute Engine, aka GCE, delivers virtual machines running on Google’s infrastructure. To use GCE virtual machines as computing nodes, a specific GCE Infrastructure needs to be created using the Resource Manager. This section describes briefly how to make it.

Pre-Requisites

First, to use the GCE Infrastructure in the Resource Manager, proper Google Cloud credentials are needed for an authentication to the Google Cloud Platform APIs. To obtain them, you can take the following steps:

Go to the Developer Console.
Log in with an account which has the permissions to create service accounts and service account keys (i.e., granted the Service Account Admin and Service Account Key Admin role).
Following the document Creating a service account to create a service account granted the Compute Admin role
Following the document Creating service account keys to create a service key of the type JSON, a JSON file for the created service account key should be downloaded to your machine.

For more information regarding Google Cloud service accounts see Google Cloud Service Accounts.

Infrastructure Configuration

Now, you are ready to create a new node source of the type GCE Infrastructure. The properties needed for the node deployment in the Create GCE Node Source are:

gceCredential: The credentials to perform the basic Google Cloud Platform authentication. Upload the JSON file of a Google Cloud service account key (downloaded in the section Pre-Requisites) to fill in this property.
totalNumberOfInstances: Total number of GoogleComputeEngine instances to create for this infrastructure.
numberOfNodesPerInstance: Total number of ProActive Nodes to deploy in each created GoogleComputeEngine instance.

If all the nodes of a GoogleComputeEngine instance are removed, the instance will be terminated.
vmUsername: Defines the user name that will be used to connect to GoogleComputeEngine instances. If not provided, then GoogleComputeEngine instances will be accessed as the default user. If specified, the corresponding vmPublicKey and vmPrivateKey must be specified as well.
vmPublicKey: Defines the public key to grant a user specified access for the created GoogleComputeEngine instances. If specified, the corresponding vmUsername and vmPrivateKey must be specified as well.
vmPrivateKey: Defines the private key that will be used to connect to GoogleComputeEngine instances. If specified, the corresponding vmUsername and vmPublicKey must be specified as well.
image: Defines which image will be used to create the GoogleComputeEngine instance. The value to provide is the unique name of the image. If not provided, the default value "debian-9-stretch-v20190326" will be used. For more information see Google Compute Engine Images List.
region: The geographic zone for Google Cloud Platform resources to use. If not provided, the default value "us-central1-a" will be used. For more information see Google Compute Engine Regions and Zones.
machineType: The machine type required for each VM (e.g. n2-standard-2). An optional parameter, if not provided, selecting a specific machineType will be ignored. For more information see Google Compute Engine Machine Families Resource

Once the machineType parameter is set, it will override the parameters set for ram and cores

ram: The minimum required amount of RAM (expressed in Mega Bytes) for each GoogleComputeEngine instance to be created. If not provided, the default value 1740 will be used.
cores: The minimum required amount of virtual cores for each GoogleComputeEngine instance to be created. If not provided, the default value 1 will be used.
rmHostname: The hostname or the public IP address of the Resource Manager. This address needs to be accessible from the GoogleComputeEngine server.
connectorIaasURL: Connector-iaas is a ProActive service used to interact with IaaS like GoogleComputeEngine. By default it runs on the following URL rmHostname/connector-iaas.
nodeJarURL: The full URL path of the node.jar to download the ProActive node.jar on each new created GoogleComputeEngine instance. The URL needs to be accessible from the GoogleComputeEngine server.
additionalProperties: Additional Java command properties to be added when starting each ProActive node JVM in GoogleComputeEngine instances (e.g. "-Dpropertyname=propertyvalue").
nodeTimeout: The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
startupScript: VM startup script to launch the ProActive nodes. To run the ProActive nodes, this script is also expected to download and install the Java Runtime Environment, then download and and execute ProActive node.jar, if they are not already provided by the VM image. The script can use some arguments. For example, it can use %nodeJarUrl% to represent the full URL path for downloading the node.jar file. These arguments will be interpreted by GCE Infrastructure later. This is an optional property. If left blank, the script is automatically generated for the Linux OS. Here is the default value of startupScript:
```
mkdir -p /tmp/node && cd /tmp/node
if ! type -p jre/bin/java; then wget -nv -N https://s3.amazonaws.com/ci-materials/Latest_jre/jre-8u382b05-linux-x64.tar.gz; tar -xf jre-8u382b05-linux-x64.tar.gz; mv jre1.8.0_382b05/ jre; fi
wget -nv %nodeJarUrl%
nohup jre/bin/java -jar node.jar -Dproactive.communication.protocol=%protocol% -Dpython.path=%jythonPath% -Dproactive.pamr.router.address=%rmHostname% -D%instanceIdNodeProperty%=%instanceId% -r %rmUrl% -s %nodeSourceName% %nodeNamingOption% -v %credentials% -w %numberOfNodesPerInstance% %additionalProperties% &
```

Using this configuration, you can start a Resource Manager and a Scheduler using the /bin/proactive-server script. A GoogleComputeEngine NodeSource can now be added using the Create Node Source panel in the Resource Manager or the command line interface:

$ PROACTIVE_HOME/bin/proactive-client --createns googlecomputeengine -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.GCEInfrastructure gceCredential totalNumberOfInstances numberOfNodesPerInstance vmUsername vmPublicKey vmPrivateKey image region machineType ram cores rmHostname connectorIaasURL nodeJarURL additionalProperties nodeTimeout startupScript

When ProActive server is stopped, GoogleComputeEngine instances will be automatically deleted. And when ProActive server is restarted, the infrastructure will be recovered as per the previous settings. The required GoogleComputeEngine instances will be re-created and re-configured.

6.4.11. Azure Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.AzureInfrastructure

The Resource Manager allows to deploy ProActive nodes on the Microsoft Azure cloud. This infrastructure will create and manage virtual machines (VMs) with your custom Azure image and host ProActive nodes on them. In this section we will guide you through the requirements and configuration needed to setup an Azure infrastructure.

Pre-Requisites

To create an Azure infrastructure, ProActive relies on an existing Azure user account with a valid subscription. In particular, you will require:

Azure account: You should have a valid Azure account, linked to an Azure subscription with enough budget and core quota to deploy the VMs (and their underlying resources) that you request.
Azure privileges: If your account does not have full Admin privileges on your subscription, please make sure you have rights to:
- Access a resource group of your choice.
- Create VMs in that resource group.
- Delete VMs in that resource group.
Service Principal credentials: ProActive uses Azure Resource Manager API to create Azure instances in the infrastructure. This API requires a set of Active Directory service principal credentials for non-interactive login (client, secret, tenant). If you do not have them, you can create them via the Azure Portal or Azure CLI. Please ask your administrator in case you do not have the privileges to create them. The credentials should be strings of hexadecimal values like 1a2b3c4d-5e6f-1a2b-3c4d-5e6f-1a2b3c4d5e6f.
Custom Azure image: ProActive will create the infrastructure’s instances from an Azure image. You can either use a known image or a custom Linux image available on the resource group of your choice. If you do not have one, you can create it by following this tutorial.

Infrastructure Configuration

Once all the pre-requisites are met, you are ready to create an Azure infrastructure. On the ProActive’s Resource Manager portal, select AzureInfrastructure from the Infrastructure drop-down list.

The following fields are available to configure your Azure infrastructure.

clientId: Corresponds to the Service Principal application ID.
secret: Corresponds to the secret value (password) of the Service Principal.
domain: Corresponds to the Azure domain, tenant ID, or directory ID of your Active Directory.
subscriptionId: The ID of your Azure subscription.

The next four parameters are optional, and apply to advanced Azure endpoint configurations only.

authenticationEndpoint: Azure Authentication endpoint.
managementEndpoint: Azure Management endpoint.
resourceManagerEndpoint: Azure Resource Manager endpoint.
graphEndpoint: Azure AD Graph endpoint.

* * * * *

rmHttpUrl: Here you should provide your server’s URL or public IP (e.g. try.activeeon.com). A default value will be generated from the system properties but you might need to modify it according to your network configuration so that it is accessible from your Azure nodes.
connectorIaasUrl: The URL of your Connector IaaS service. A default value will be generated as well (e.g. http://try.activeeon.com:8080/connector-iaas).

image: Can be, either:

the name or id of your custom image. The image should be located within the provided subscription and accessible to the service principal. It should also be located in the Azure region that you define in your configuration (see region parameter below).

the name of a known image as defined in Azure’s SDK. Accepted values are simple strings, as listed in the following table:

Linux

Windows

UBUNTU_SERVER_16_04_LTS
UBUNTU_SERVER_18_04_LTS (default value)
UBUNTU_SERVER_18_04_LTS_GEN2
UBUNTU_SERVER_20_04_LTS
UBUNTU_SERVER_20_04_LTS_GEN2
UBUNTU_SERVER_22_04_LTS
UBUNTU_SERVER_22_04_LTS_GEN2
UBUNTU_SERVER_24_04_LTS
UBUNTU_SERVER_24_04_LTS_GEN2
DEBIAN_9
DEBIAN_10
CENTOS_8_1
CENTOS_8_3
OPENSUSE_LEAP_15
SLES_15
REDHAT_RHEL_8_2
ORACLE_LINUX_8_1

WINDOWS_SERVER_2012_R2_DATACENTER
WINDOWS_SERVER_2016_DATACENTER
WINDOWS_SERVER_2016_DATACENTER_GEN2
WINDOWS_SERVER_2019_DATACENTER
WINDOWS_SERVER_2019_DATACENTER_GEN2
WINDOWS_DESKTOP_10_PRO
WINDOWS_DESKTOP_10_21H2_PRO_GEN2

Due to a recent change in Debian images that disables VM Extensions, Debian images (DEBIAN_9, DEBIAN_10) are not supported anymore.

imageOSType: Image OS type (A choice between linux and windows). The default value is linux.
vmSizeType: The size of the VMs to be deployed. The default value is Standard_D1_v2. Azure provides an extensive list of Linux VM sizes. If you want to know the available sizes for a specific region or subscription you can use Azure CLI command az vm list-sizes -l {your location}.
vmUsername: Provide a user name for the VM.
vmPassword: Provide a password for your VM. Here is a guideline for Linux passwords on Azure.
vmPublicKey: An optional public RSA key to connect to your VMs via SSH protocol.
resourceGroup: The name of the resource group of your choice. If left blank, the resource group of the image will be used by default.
region: The Azure region where the VMs will be deployed. For a list of the regions supported by your subscription, you can use the CLI command az account list-locations -o table, the right name to use is the one in the column name.
numberOfInstances: Total instances (VMs) to be created. The default value is 1.
numberOfNodesPerInstances: The number of ProActive nodes to be launched in each VM. The default value is 1.
downloadCommand: Command used to download ProActive’s node.jar worker. If left blank a default command will be generated.
privateNetworkCIDR: An optional network Classless Inter-Domain Routing to allocate the new VMs within the private network. The default value is 10.0.0.0/24.
staticPublicIP: A boolean flag to determine whether the public IPs of the infrastructure’s VMs should be static. The default value is true.
additionalProperties: Additional JVM properties to configure your ProActive node. The default values allow to handle both PNP and PAMR protocols. You can add your own properties but be aware that removing the current values might cause the deployment to fail.
linuxStartupScript: Linux VM startup script to launch the ProActive nodes. To run the ProActive nodes, this script is also expected to download and install the Java Runtime Environment, then download and and execute ProActive node.jar, if they are not already provided by the VM image. The script can use some arguments. For example, it can use %nodeJarUrl% to represent the full URL path for downloading the node.jar file. These arguments will be interpreted by Azure Infrastructure later. This is an optional property. If left blank, the script is automatically generated. Here is the default value of linuxStartupScript:
```
mkdir -p /tmp/node && cd /tmp/node
if ! type -p jre/bin/java; then wget -nv -N https://s3.amazonaws.com/ci-materials/Latest_jre/jre-8u382b05-linux-x64.tar.gz; tar -xf jre-8u382b05-linux-x64.tar.gz; mv jre1.8.0_382b05/ jre; fi
wget -nv %nodeJarUrl%
nohup jre/bin/java -jar node.jar -Dproactive.communication.protocol=%protocol% -Dpython.path=%jythonPath% -Dproactive.pamr.router.address=%rmHostname% -D%instanceIdNodeProperty%=%instanceId% -r %rmUrl% -s %nodeSourceName% %nodeNamingOption% -v %credentials% -w %numberOfNodesPerInstance% %additionalProperties% &
```

windowsStartupScript: Powershell script to be run during Windows VM startup for launching the ProActive nodes. This is an optional property. If left blank, the script is automatically generated. Here is the default value of windowsStartupScript:

$download=New-Object System.Net.WebClient;
$download.DownloadFile('http://javadl.oracle.com/webapps/download/AutoDL?BundleId=238729_478a62b7d4e34b78b671c754eaaf38ab', 'c:\jreInstall.exe');
$procInstall=Start-Process -FilePath 'c:\jreInstall.exe' -ArgumentList '/s REBOOT=ReallySuppress INSTALLDIR=c:\jre' -Wait -PassThru;
$procInstall.waitForExit();
$download.DownloadFile('%nodeJarUrl%', 'c:\node.jar');
Start-Process -NoNewWindow 'c:\jre\bin\java' -ArgumentList '-jar', 'c:\node.jar', '-Dproactive.communication.protocol=%protocol%', '-Dproactive.pamr.router.address=%rmHostname%', '-D%instanceIdNodeProperty%=%instanceId%', '-r', '%rmUrl%', '-s', '%nodeSourceName%', '-v', '%credentials%', '-w', '%numberOfNodesPerInstance%', '%additionalProperties%'

The following fields are the optional parameters of the Azure Billing Configuration section. The aim of this section is to configure the automatic cloud cost estimator. It is done by considering all the Azure resources related to your reservation (virtual machines, disks,..). This mechanism relies on the Azure Resource Usage and RateCard APIs (https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/usage-rate-card-overview).

resourceUsageRefreshFreqInMin: Periodical resource usage retrieving delay in min. The default value is 30.
rateCardRefreshFreqInMin: Periodical rate card retrieving delay in min. The default value is 30.
offerId: The Offer ID parameter consists of the "MS-AZR-" prefix, plus the Offer ID number. The default value is "MS-AZR-0003p" (Pay-As-You-Go offer).
currency: The currency in which the resource rates need to be provided. The default value is "USD".
locale: The culture in which the resource metadata needs to be localized. The default value is "en-US".
regionInfo: The 2 letter ISO code where the offer was purchased. The default value is "US".
maxBudget: Your max budget for the Azure resources related to the node source. Also used to compute your global cost in % of your budget. The default value is 50.

The Azure infrastructure offers a simple interface to easily create ProActive nodes on Azure. If your project requires a dynamic and customizable set of Azure VMs, you can opt for an Azure Scale Set Infrastructure.

6.4.12. Azure Scale Set Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.AzureVMScaleSetInfrastructure

We provide a highly customizable infrastructure to deploy ProActive nodes using an Azure Scale Set. This infrastructure will allow you to automatically adjust the number of deployed virtual machines (VM) in response to the current workload, and according to your configured parameters, e.g. min/max number of nodes, number of nodes per VM, or minimal up-time. Auto-scaling will help you increase your processing capacity during peak computation periods, while keeping your expenses low during less intensive or idle periods.

Whether you require to run your workflows on Linux or Windows, use standard or custom VM images, do some pre-configuration steps, or mount a shared file system, the Azure Scale Set infrastructure can handle it. This guide describes the requirements and the various configuration parameters available to deploy an Azure Scale Set infrastructure.

Pre-Requisites

As in our standard Azure Infrastructure, ProActive relies on an existing Azure user account with a valid subscription in order to deploy a Scale Set. These are the specific requirements:

Azure account: You should have a valid Azure account, linked to an Azure subscription with enough budget and service quotas to deploy the Scale Set that you request.
Azure privileges: If your account does not have full Admin privileges on your subscription, please make sure you have rights to:
- Create a resource group in a region of your choice.
- Create a Scale Set in that resource group, and create its underlying resources: Public IP Address, Load Balancer, Network Security Group and Virtual Network.
- Create a Storage account in that resource group, and use its various services (in particular Blobs, Tables and Queues).
- Delete a resource group.
Service Principal credentials: ProActive uses Azure Resource Manager API to manage the Scale Set and Storage resources. This API requires a set of Active Directory service principal credentials for non-interactive login (client, secret, tenant). If you do not have them, you can create them via the Azure Portal or Azure CLI. Please ask your administrator in case you do not have the privileges to create them. The credentials should be strings of hexadecimal values like 1a2b3c4d-5e6f-1a2b-3c4d-5e6f-1a2b3c4d5e6f.

Scale Set Tagging

The infrastructure can dynamically apply tags to the created Scale Set when the infrastructure is attached to a Dynamic Policy.

This feature can be used to track Scale Set usage information in Azure Cost Monitoring. For example, when several computing batches are run subsequently on a Scale Set, it will be possible to know an approximate cost of each individual batch.

When the Dynamic Policy triggers a scale up of the Scale Set and when pending tasks that triggered this scale up have the RESOURCE_TAGS Generic Information defined (a key/value structure), configured tags will be applied to the Scale Set.

Please note that Azure Scale Set do not support individual instance tagging and tags will only be applied globally on the Scale Set. Accordingly, in a cost monitoring scenario, it is only possible to track the cost of subsequent computing batches (batches that run one after another) and not parallel batches on the same Scale Set.

When a tag with a given name has been applied, any new application of the tag will overwrite the previous value.

When the Scale Set scales down to zero instances, all currently applied tags will be removed. This is the only case where tags are removed from the Scale Set.

Finally, please note that tags are applied only when the Scale Set scales up (regardless of its current size). If a pending task with tags defined simply reuse existing instances without triggering a scale up, its configured tags will not be applied to the Scale Set.

Example Scenario:

The scale set is idle with zero instance
5 tasks with tags <batch=batch1,step=step1> are pending and trigger a scale up to 0→5 instances. The tags <batch=batch1,step=step1> will be applied to the Scale Set.
2 tasks terminate and release 2 instances. The Scale Set scales down 5→3, and still contains tags <batch=batch1,step=step1>.
3 tasks with tags <batch=batch1,step=step2> are pending and trigger a scale up 3→6. The tags <batch=batch1,step=step2> will be applied to the Scale Set. The new value for the tag 'step' will overwrite the previous value.
The 6 running tasks terminate and release 6 instances. The Scale Set scales down 6→0. All tags currently applied on the scale set are removed.
5 tasks with tags <batch=batch2,step=step1> are pending and trigger a scale up to 0→5 instances. The tags <batch=batch2,step=step1> will be applied to the Scale Set.
…

Infrastructure Configuration

The Azure Scale Set Infrastructure offers a wide range of possibilities to configure your Scale Set so that:

You have in your VMs everything you need to execute your jobs.
You can control your expenses by defining size and limits of your VMs.

The available configuration parameters are:

azureCredentialFile: Use the provided file selection tool to supply a file with your Azure Credentials. These credentials correspond to those generated with the Service Principal as mentioned in the Pre-Requisites sub-section. The file should be a four-line text document in a .ini like format, and must include the following elements:
```
client=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
key=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
tenant=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
subscription=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
```
Where:
- client: Corresponds to the Service Principal application ID.
- key: Corresponds to the secret value (password) of the Service Principal.
- tenant: Corresponds to the Azure domain, tenant ID, or directory ID of your Active Directory.
- subscription: Is the ID of your Azure subscription.
You can opt to place this file in $PROACTIVE_HOME/config/authentication/azure.creds so that it will available and automatically loaded in case you want to have multiple Azure node sources.
maxVms: Define the maximum number of VMs to be deployed by the scale set at any time. The number cannot exceed 1000. The default value is 100, so you might want to adjust this value according to your budget and expected processing peaks.
maxNodesPerVm: Define the number of ProActive worker nodes to be deployed per VM. As a guideline, you might want to allocate one node per VM core, but you can set the number to better match your processing requirements. Default value is 2.
machineType: Define the type of Azure VM that you want to use for your Scale Set. You can use any size from Azure’s Standard Tier, provided you have enough quota and budget to launch them. The default value is Standard_D1_v2, you should provide a valid type using a similar syntax (case insensitive). Azure provides an extensive list of VM sizes for Linux and Windows VMs. If you want to know the available sizes for a specific region or subscription you can use Azure CLI command az vm list-sizes -l {your location}.
osType: A choice between linux and windows. Default value is linux.

Image: There are four possible ways to specify the source image for your Scale Set VMs: known images, Azure Marketplace, Shared Image Gallery and Custom images. We describe each of them below.

Syntax summary
- Known image: MY_KNOWN_IMAGE
- Marketplace: marketplace#publisher:offer:sku[:version]
- Shared Image Gallery: gallery#resourceGroup:gallery:image[:version]
  - Shared Image Gallery from non-default subscription: gallery#sub#subsriptionId:resourceGroup:gallery:image[:version]
- Custom image: custom#resourceGroup:customImage
Known image. Refers to commonly used images as defined in Azure’s SDK. Accepted values are simple strings, as listed in the following table:

Linux

Windows

Due to a recent change in Debian images that disables VM Extensions, Debian images (DEBIAN_9, DEBIAN_10) are not supported anymore.

Azure Marketplace. If you want to choose a particular image from Azure’s Marketplace, use the prefix marketplace# plus the image’s URN in the form publisher:offer:sku[:version]. Similarly to Azure CLI or PowerShell, ProActive allows you to specify a version of the image (with :version), or it will use the latest one if the version is omitted. Examples of valid inputs are:
marketplace#OpenLogic:CentOS:7.5
marketplace#OpenLogic:CentOS:7.5:latest
marketplace#credativ:Debian:8:8.0.201901221
Shared Image Gallery: The prefix gallery# allows you to use images from an existing Azure Shared Image Gallery. Append the name of your Resource Group, your Gallery and your Image as resourceGroup:gallery:image. Optionally, you can specify a version too (:version); if no version is given, the latest image version will be used by ProActive. For example:
gallery#proactive-rg:activeeon-gallery:my-image
gallery#myRG:myGallery:myImage:1.0
- Shared Image Gallery from non-default subscription You can use images from an existing Azure Shared Image Gallery by manually specifying the subscription.
  The prefix sub# should be added and the subscriptionId should be provided in the next format: gallery#sub#subsriptionId:resourceGroup:gallery:image[:version]
  The non-default subscription must be accessible with your Azure Credentials provided in the azureCredentialFile configuration.
Custom Image: ProActive allows you to use a custom image to create Scale Sets. Use the prefix custom# followed by the resource group and the custom image name. For example:
custom#proactive-rg:custom-ubuntu

sshPublicKey: Optional parameter. In case of Linux, use it to provide a public RSA key to connect to your VMs via SSH. In case of Windows, use it to define a password to connect to your VMs via Remote Desktop. In both cases, the default user is activeeon. If the field is left blank a password will be automatically generated and written in the Scheduler log.
targetNetwork: Optional parameter. Use it to define a specific Azure Virtual Network subnet for your Scale Set. If defined, the format should be {resourceGroupName}:{network}:{subnet}, e.g. myGroup:myVNet:default. If left unset, a new Virtual Network and subnet will be created inside the Resource Group.
restrictStorageAccountToNetwork: Optional parameter. If true and a targetNetwork is specified, the storage account created by the infrastructure will be restricted to the provided network. In that case, the ProActive server must be deployed inside the provided network, as otherwise access to the storage account from the server would not be possible. Additionally, Microsoft.Storage must be enabled in the Service Endpoints section of the subnet configuration. More information about Service Endpoints is available in the Azure Documentation.
azureRegion: Defines the Azure region where the Resource Group and the Scale Set will be created. For the list of the regions supported by your subscription, you can use the CLI command az account list-locations -o table, the right name to use is the one in the column name. The region name should be in lowercase without spaces, e.g. for the region West US 2 use westus2. Default value is westeurope.
storageType: Specifies the type of storage used by the instances' operating system disk. Options include:
- Standard_LRS: HDD storage.
- StandardSSD_LRS: SSD storage.
- Premium_LRS: Premium SDD storage. Premium storage is only available for specific instance types. For example, it is available on STANDARD_D4S_V3 and not STANDARD_D4_V3 instances. Refer to Azure Resource Manager documentation for further information.
instancePriority: Defines the type of VM instances to be deployed for the Scale Set. It can be set to:
- Regular: Regular instances are standard virtual machines.
- Spot: Spot instances have the lowest priority among the three options and offer significant cost savings. They are allocated from surplus capacity and can be preempted with little notice if the capacity is needed for regular or Low-priority instances.
- Low: Low-priority instances have a higher priority than Spot instances. These instances are cheaper than Regular but more expensive than Spot instances. Note that this is to be deprecated in September 30th 2025. Learn more with the Azure Spot documentation.
evictionPolicy: This parameter allows you to specify what action should be taken when an instance is reclaimed by Azure due to capacity constraints when the instancePriority is set to Spot or Low. It can be set to:
- Delete: When an instance is evicted, it is immediately deleted, and any workloads running on the instance are terminated.
- Deallocate: When an instance is evicted, it is deallocated but not immediately deleted. In both cases, ProActive will be able to automatically reschedule any workloads (Tasks) on other available Nodes provided by another Node Source, for instance another Azure Scale Set with regular instances. Learn more with the Azure Spot documentation.
maxPrice: Defines the maximum price in USD that you are willing to pay for Low or Spot virtual machines in the scale set. By providing the value of -1, the VM won’t be evicted for pricing reasons. The max price will be the current price, up to the price for standard VMs. You’ll never be charged above the standard price.
rmUrl: Optional parameter. Provide your server’s URL or public IP (e.g. try.activeeon.com). If left unset, the default Resource Manager URL will be fetched from the system properties.
rmCredentials: Optional parameter. Provide credentials to connect to the Resource Manager. If left blank, the credentials of the logged user will be used.
deploymentTimeout: Delay before the Resource Manager can declare a deploying node as lost. To be set in minutes. You can set it to a reasonable time after which you expect the node to be running or marked as lost otherwise (e.g. 10 minutes). Default value is 120.
cleaningDelay: Periodical cleaning delay, expressed in seconds. Every X seconds the Resource Manager will remove from the Scale Set any VM that has been identified as idle. Default value is 60. Note that setting a very short interval implies more frequent communication between the Resource Manager and Azure, which might impact on performance or responsiveness.
externalStorageAccount: Optional parameter. Use it if you want to define a specific location of your ProActive node.jar worker; leave blank otherwise. If provided, the format should be: {storageAccount}:{sas_key_token}[:has_node.jar].
linuxInternalCustomStartupScriptUrl: If you are using Linux, provide a URL of a valid Linux bash script to configure your nodes. Otherwise, the parameter can be left empty. We strongly recommend to use one of our default scripts listed below. You are free to copy the script and add custom steps, but be aware that any modification to the default script might cause the deployment to fail. Our startup scripts cover the minimum requirements to start ProActive; additional script engines like R can be installed through a userCustomStartupScriptUrl, where you can also add your custom configuration steps.

Sample scripts (might require some customization for specific versions or OS):
- Debian 8 - apt based script, uses systemd for launching the node service. Default value.
  
  https://gist.githubusercontent.com/activeeon-bot/1d4194fefff12cc9c5046c2455bd457b/raw
- CentOS 7 - yum based script, uses systemd for launching the node service.
  
  https://gist.githubusercontent.com/activeeon-bot/c6f303c2c832fa40691874ef76484e92/raw
- CentOS 6 - yum based script, uses init.d for launching the node service.
  
  https://gist.githubusercontent.com/activeeon-bot/7d1c10ede4599a78ce95e0c9201f9746/raw
windowsInternalCustomStartupScriptUrl: If you are using Windows, provide a URL of a valid Windows PowerShell script to configure your nodes. Otherwise, the parameter can be left empty. We strongly recommend to our default script (see link below). You are free to copy the script and add custom steps, but be aware that any modification to the default script might cause the deployment to fail. Our startup scripts cover the minimum requirements to start ProActive; additional script engines like R can be installed through a userCustomStartupScriptUrl, where you can also add your custom configuration steps.
- Windows script - PowerShell script, compatible with Windows Server 2012 or later.
  
  https://gist.githubusercontent.com/activeeon-bot/100ea57c89d6a0975f051a062187bf51/raw
userCustomStartupScriptUrl: Optional parameter. Use it to provide a URL of a valid custom script (.sh for Linux, .ps1 for Windows) that will be called at the VMs' startup. The script is called after the default ProActive configuration and before the node application is actually executed. You can use this script, for instance, to create directories, assign permissions, or download utilities. As a reference, we provide some sample user scripts:
- Debian: apt based script, includes installation of Python3, R, or SciLab.
  
  https://gist.githubusercontent.com/activeeon-bot/fe7c6a89e248f7363f02fa75a641dd39/raw
- CentOS: yum based script, includes installation of Python3 and R.
  
  https://gist.githubusercontent.com/activeeon-bot/94cc622e7640832682f172e23256cd3f/raw

Note that this user custom script will be run as root/admin user.

nodePreCommand: Optional parameter. This multi-line field can be used to provide specific commands to execute right before launching the ProActive’s node.jar application. The main difference with the previous user custom script is that nodePreCommand will be run as the same user as node.jar. This field is thus useful for operations that require matching users, e.g. mounting a shared volume in Windows or creating non-root paths/files in Linux. Please note the OS-specific conditions for these commands:
- Linux: The commands will be appended as ExecStartPre lines to a .service file. Therefore, commands should include their full path, e.g. mkdir should be written as /bin/mkdir or chown as /bin/chown.
- Windows: The commands will be included in a .bat batch file and must therefore be DOS-compatible. As a guideline, here is an unofficial list of DOS commands and a scripting tutorial.
jvmParameters: Add JVM parameters to configure your ProActive node, use the format -Dproperty=value. The default values allow for a correct deployment of the node. You can add your own parameters but be aware that removing the current values might cause the deployment to fail. A non-exhaustive list of node-related JVM parameters can be found in your configuration file $PROACTIVE_HOME/config/network/node.ini
armTemplateUrl: Optional parameter, for advanced configuration only. This parameter is intended for users familiar with Azure Resource Management templates willing to provide a custom resource configuration file. You should input a URL with a valid .json ARM template. If set, this file will override ProActive’s default ARM file.
resourceGroupExistingName: Optional parameter. Defines the existing Resource Group name where only the Azure Scale Set and Storage Account should be created. This parameter should correspond to the existing Azure Resource Group in the specified azureRegion.
The Resource Group will be created with the Node Source unique name if the parameter resourceGroupExistingName is unset.
In case the parameter is set, several conditions have to be met:
- the deployment Scale Set template (to be specified in armTemplateUrl) should contain only a creation of next Azure resources inside Resource Group: the Scale Set and the Storage Account (no Azure Load Balancer, IP address, etc.)
- the target Azure Virtual Network should be provided by the user in the targetNetwork parameter.

enableBilling: Enable billing information (true/false). If true, the following parameters will be considered. The default value is false.
resourceUsageRefreshFreqInMin: Periodical resource usage retrieving delay in min. The default value is 30.
rateCardRefreshFreqInMin: Periodical rate card retrieving delay in min. The default value is 30.
offerId: The Offer ID parameter consists of the "MS-AZR-" prefix, plus the Offer ID number. The default value is MS-AZR-0003p (Pay-As-You-Go offer).
currency: The currency in which the resource rates need to be provided. The default value is USD.
locale: The culture in which the resource metadata needs to be localized. The default value is en-US.
regionInfo: The 2 letter ISO code where the offer was purchased. The default value is US.
maxBudget: Your max budget for the Azure resources related to the node source. Also used to compute your global cost in % of your budget. The default value is 50.

As you can see, these parameters provide a lot of flexibility to configure your infrastructure. When creating your Azure Scale Set node source, the infrastructure should be coupled with a Dynamic Policy. This Policy will additionally define scalability parameters such as limits on the number of deployed nodes or the minimum idle time before a node can be deleted (to optimize node utilization).

6.4.13. Kubernetes Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.KubernetesInfrastructure

The Kubernetes allows to deploy ProActive Nodes on a Kubernetes cluster. This infrastructure aims at creating kubernetes pods containing ProActive Nodes. Each ProActive Node pod will connect and register inside the ProActive Resource Manager.

Interactions between the ProActive Resource Manager and the Kubernetes cluster is based on the Kubernetes Java SDK. Accordingly, it is not mandatory to install the Kubernetes command line on the ProActive server machine.

Pre-Requisites

A Kubernetes cluster must be accessible from the ProActive server.
A Kubernetes configuration file with server url, credentials and other appropriate settings.
A Docker image which contains Java Runtime Environment version 8 (earlier and later versions are currently not supported). This image will be used to start ProActive Nodes.
ProActive server started with the ProActive Message Routing protocol.

Infrastructure Configuration

kubeConfigFile: Use the provided file selection tool to supply a file with your Kubernetes configuration and credentials.

Example configuration file:

apiVersion: v1
clusters:
- cluster:
    server: https://192.168.99.102:8443
    certificate-authority-data: LS0tLS1CRUdJTiBDRV...
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRV...
    client-key-data: LS0tLS1CRUdJTiBSU0E...

It is advised to embed credentials and certificates in the configuation file, instead of pointing to files on the local file system.

kubernetesNamespace: (Optional) Kubernetes namespace if different from default namespace.
cleaningDelaySeconds: (Optional) Cleaning delay, in seconds until cleaning service starts. Default value is 120 seconds.
cleaningFrequencySeconds: (Optional) Cleaning frequency, Kubernetes node cleaning is done every x seconds. Default value is 30 seconds.
scale: Number of pods (ProActive Nodes) to create. This parameter is used when the Node Source Policy is not a Dynamic Policy.
image: (Optional) Docker image used to deploy ProActive Nodes. When it is not set, its default value adoptopenjdk/openjdk8:latest is used.
nodeDeploymentTemplateFile: (Optional) Kubernetes deployment template file for deploying ProActive node pods. This template file informs Kubernetes about the environment needed to run ProActive nodes. Each node uses this template file to create its specific Kubernetes deployment environment. The default Kubernetes template file creates an environment (i.e., a pod) where both a ProActive node and a Docker engine are started. This way, the tasks executed by the ProActive node can also run Docker commands. Such an environment is supported through a Docker-in-Docker approach (i.e., running a Docker daemon inside the Kubernetes host container system). Specifically, this environment consists of one pod with two containers (proactive-node and proactive-docker-daemon). The template file can use the following macros, which will be interpreted and filled with the corresponding parameter values by Kubernetes Infrastructure later.
- ${STARTUP_SCRIPT}: the value of the infrastructure parameter startupScript
- ${DOCKER_IMAGE}: the value of the infrastructure parameter image
- ${NODE_NAME_LOWER_CASE}: the lower case format of the expected ProActive node name which is a random string generated by the Kubernetes infrastructure
- ${NODE_SOURCE_NAME_LOWER_CASE}: the lower case format of the node source name
- ${DEPLOYMENT_NAME}: the generated Kubernetes deployment name which is unique for each ProActive Node
- ${KUBERNETES_NAMESPACE}: the value of the infrastructure parameter kubernetesNamespace
  
  Default nodeDeploymentTemplateFile:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    pa-node-name: ${NODE_NAME_LOWER_CASE}
    pa-node-source-name: ${NODE_SOURCE_NAME_LOWER_CASE}
  name: ${DEPLOYMENT_NAME}  # must be unique
  namespace: ${KUBERNETES_NAMESPACE}
spec:
  progressDeadlineSeconds: 2147483647
  replicas: 1
  revisionHistoryLimit: 2147483647
  selector:
    matchLabels:
      pa-node-name: ${NODE_NAME_LOWER_CASE}  # Label key and value are important so that the cleaning procedure finds this deployment
      pa-node-source-name: ${NODE_SOURCE_NAME_LOWER_CASE} # Label key and value are important so that the cleaning procedure finds this deployment
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        pa-node-name: ${NODE_NAME_LOWER_CASE}
        pa-node-source-name: ${NODE_SOURCE_NAME_LOWER_CASE}
    spec:
      containers:
        - name: proactive-node
          args:
            - -c
            - ${STARTUP_SCRIPT}
          command:
            - /bin/sh
          image: ${DOCKER_IMAGE}
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 64738 # PNP
              protocol: TCP
            - containerPort: 64739 # PNPS
              protocol: TCP
            - containerPort: 33647 # PAMR
              protocol: TCP
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          env:
            - name: DOCKER_HOST
              value: tcp://localhost:2375
          volumeMounts:
            - name: pa-node-data
              mountPath: /tmp
        - name: proactive-docker-daemon
          image: docker:1.12.6-dind
          resources:
              requests:
                  cpu: 20m
                  memory: 512Mi
          securityContext:
              privileged: true
          volumeMounts:
            - name: pa-node-data
              mountPath: /tmp
      volumes:
        - name: pa-node-data
          emptyDir: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

rmHostname: The hostname or the public IP address of the host operating the Resource Manager. This address needs to be accessible from the Kubernetes server. For example, try.activeeon.com.
nodeJarURL: The full URL path of the node.jar to download the ProActive node.jar on each new created pod. The URL needs to be accessible from the Kubernetes pods. For example, try.activeeon.com/rest/node.jar.
jvmProperties: Java options appended to the command used to start the node on the remote host. As the ProActive Message Routing is required by the Kubernetes Infrastructure, PAMR properties must be defined.

Example:

-Dproactive.communication.protocol=pamr -Dproactive.useIPaddress=true -Dproactive.pamr.router.address=try.activeeon.com

startupScript: VM startup script to launch the ProActive nodes. To run the ProActive nodes, this script is also expected to download and install the Java Runtime Environment if it is not already installed in the specified image, then download and and execute ProActive node.jar. The script can use some arguments. For example, it can use %nodeJarUrl% to represent the full URL path for downloading the node.jar file. These arguments will be interpreted by Kubernetes Infrastructure later. This is an optional property. If left blank, a default script is automatically generated for the Linux OS.

Default startupScript:

mkdir -p /tmp/node && cd /tmp/node
if ! command -v wget; then apt-get update; apt-get -y install wget; fi
wget -nv --no-check-certificate %nodeJarUrl%
if ! command -v java; then wget -nv -N https://s3.amazonaws.com/ci-materials/Latest_jre/jre-8u382b05-linux-x64.tar.gz; tar -xf jre-8u382b05-linux-x64.tar.gz; export PATH=/tmp/node/jre1.8.0_382b05/bin/:$PATH; fi
java -jar node.jar -Dproactive.communication.protocol=%protocol% -Dpython.path=%jythonPath% -Dproactive.pamr.router.address=%rmHostname% -D%instanceIdNodeProperty%=%instanceId% -r %rmUrl% -s %nodeSourceName% %nodeNamingOption% -v %credentials% -w %numberOfNodesPerInstance% %additionalProperties%

nodeTimeout: (Optional) The estimated startup time of the nodes (expressed in millisecond). After this timeout, the node will be considered as lost. Default value is 10 minutes.

6.4.14. Load Sharing Facility (LSF) Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.LSFInfrastructure

This infrastructure knows how to acquire nodes from LSF by submitting a corresponding job. It will be submitted through SSH from the RM to the LSF server. This is the static version of the LSF infrastructure, for a more dynamic mechanism, as described in Deploy ProActive Nodes dynamically via other schedulers (PBS, SLURM, …), use the Native Scheduler Infrastructure instead.

$ PROACTIVE_HOME/bin/proactive-client --createns lsf -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.LSFInfrastructure javaPath SSHOptions schedulingPath javaOptions maxNodes nodeTimeout LSFServer RMCredentialsPath bsubOptions

where:

javaPath - path to the java executable on the remote hosts (ie the LSF slaves).
sshOptions - Options you can pass to the SSHClient executable ( -l inria to specify the user for instance )
schedulingPath - path to the Scheduling/RM installation directory on the remote hosts.
javaOptions - Java options appended to the command used to start the node on the remote host.
maxNodes - maximum number of nodes this infrastructure can simultaneously hold from the LSF server. That is useful considering that LSF does not provide a mechanism to evaluate the number of currently available or idle cores on the cluster. This can result to asking more resources than physically available, and waiting for the resources to come up for a very long time as the request would be queued until satisfiable.
nodeTimeOut - The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
serverName - URL of the LSF server, which is responsible for acquiring LSF nodes. This server will be contacted by the Resource Manager through an SSH connection.
rmCredentialsPath - Encrypted credentials file, as created by the create-cred[.bat] utility. These credentials will be used by the nodes to authenticate on the Resource Manager.
submitJobOpt - Options for the bsub command client when acquiring nodes on the LSF master. Default value should be enough in most cases, if not, refer to the documentation of the LSF cluster.

6.4.15. Portable Batch System (PBS) Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.PBSInfrastructure

This infrastructure knows how to acquire nodes from PBS (i.e. Torque) by submitting a corresponding job. It will be submitted through SSH from the RM to the PBS server. This is the static version of the PBS infrastructure, for a more dynamic mechanism, as described in Deploy ProActive Nodes dynamically via other schedulers (PBS, SLURM, …), use the Native Scheduler Infrastructure instead.

$ PROACTIVE_HOME/bin/proactive-client --createns pbs -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.PBSInfrastructure javaPath SSHOptions schedulingPath javaOptions maxNodes nodeTimeout PBSServer RMCredentialsPath qsubOptions

where:

javaPath - path to the java executable on the remote hosts (ie the PBS slaves).
sshOptions - Options you can pass to the SSHClient executable ( -l inria to specify the user for instance )
schedulingPath - path to the Scheduling/RM installation directory on the remote hosts.
javaOptions - Java options appended to the command used to start the node on the remote host.
maxNodes - maximum number of nodes this infrastructure can simultaneously hold from the PBS server. That is useful considering that PBS does not provide a mechanism to evaluate the number of currently available or idle cores on the cluster. This can result to asking more resources than physically available, and waiting for the resources to come up for a very long time as the request would be queued until satisfiable.
nodeTimeOut - The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
serverName - URL of the PBS server, which is responsible for acquiring PBS nodes. This server will be contacted by the Resource Manager through an SSH connection.
rmCredentialsPath - Encrypted credentials file, as created by the create-cred[.bat] utility. These credentials will be used by the nodes to authenticate on the Resource Manager.
submitJobOpt - Options for the qsub command client when acquiring nodes on the PBS master. Default value should be enough in most cases, if not, refer to the documentation of the PBS cluster.

6.4.16. Generic Batch Job Infrastructure

Full name: org.ow2.proactive.resourcemanager.nodesource.infrastructure.GenericBatchJobInfrastructure

Generic Batch Job infrastructure provides users with the capability to add the support of new batch job scheduler by providing a class extending org.ow2.proactive.resourcemanager.nodesource.infrastructure.BatchJobInfrastructure.

This is the static version of the Generic Batch Job infrastructure, for a more dynamic mechanism, as described in Deploy ProActive Nodes dynamically via other schedulers (PBS, SLURM, …), use the Native Scheduler Infrastructure instead (generic also).

Once you have written the implementation of the desired class, you can create a node source which makes usage of this infrastructure by running the following command:

$ PROACTIVE_HOME/bin/proactive-client --createns pbs -infrastructure org.ow2.proactive.resourcemanager.nodesource.infrastructure.GenericBatchJobInfrastructure javaPath SSHOptions schedulingPath javaOptions maxNodes nodeTimeout BatchJobServer RMCredentialsPath subOptions implementationClassName implementationFile

where:

RMURL - URL of the Resource Manager from the batch job scheduler nodes point of view - this is the URL the nodes will try to lookup when attempting to register to the RM after their creation.
javaPath - path to the java executable on the remote hosts (ie the slaves of the batch job scheduler).
sshOptions - Options you can pass to the SSHClient executable ( -l inria to specify the user for instance )
schedulingPath - path to the Scheduling/RM installation directory on the remote hosts.
javaOptions - Java options appended to the command used to start the node on the remote host.
maxNodes - maximum number of nodes this infrastructure can simultaneously hold from the batch job scheduler server.
nodeTimeOut - The estimated startup time of the nodes (expressed in millisecond). After this timeout expired, the node will be considered as lost.
serverName - URL of the batch job scheduler server, which is responsible for acquiring nodes. This server will be contacted by the Resource Manager through an SSH connection.
rmCredentialsPath - Encrypted credentials file, as created by the create-cred[.bat] utility. These credentials will be used by the nodes to authenticate on the Resource Manager.
submitJobOpt - Options for the submit command client when acquiring nodes on the batch job scheduler master.
implementationClassName - Fully qualified name of the implementation of org.ow2.proactive.resourcemanager.nodesource.infrastructure.BatchJobInfrastructure provided by the end user.
implementationFile - The absolute path of the implementation of org.ow2.proactive.resourcemanager.nodesource.infrastructure.BatchJobInfrastructure.

6.4.17. Native Scheduler Infrastructure

The Native Scheduler Infrastructure allows to interact with a native scheduler to deploy ProActive Nodes. This mechanism is described in Deploy ProActive Nodes dynamically via other schedulers (PBS, SLURM, …). This infrastructure must be associated with a Native Scheduler Policy and cannot be associated with any other policy. The infrastructure parameters are described hereafter:

RMCredentialsPath: path to a file which contains the credentials of an administrator user which will own the node source. The ProActive Scheduler Server release contains two admin users credentials files: config/authentication/rm.cred and config/authentication/admin_user.cred
NSFrontalHostAddress: the host name or IP address of the cluster head node.
NSSchedulerHome: the location of the shared ProActive installation on cluster nodes (cluster nodes must be able to access ProActive libraries in order to start ProActive Node). Example /opt/proactive/activeeon_enterprise-node-linux-x64-8.1.0.
javaHome: similarly, cluster nodes must be able to access the java command in order to start ProActive Nodes. ProActive installation includes a Java Runtime Environment under the jre subfolder. Example: /opt/proactive/activeeon_enterprise-node-linux-x64-8.1.0/jre.
jvmParameters: additional options which can be passed to the java command.
sshOptions: additional options which can be passed to the SSH command used to connect to connect to the host name or IP address specified in the NSFrontalHostAddress parameter.
NSNodeTimeoutInSeconds: timeout to wait for the deployment of ProActive Nodes on the cluster. As the time needed to deploy ProActive Nodes depends on the cluster load, this timeout should be a large value. If the timeout is reached, the ProActive Nodes will be in "Lost" state.
ìmpersonationMethod: when a job is submitted to the native scheduler, the submission is performed under the current ProActive Scheduler user. An impersonation is thus performed between the scheduler server process and the target cluster user. This impersonation can be performed using 3 different strategies:
- ssh: in that case the head node is contacted using a SSH command with the current ProActive Scheduler user and password. User/password combination between the ProActive Scheduler and the head node operating system must match.
- none: in that case the head node is contacted using a SSH command with the ProActive Scheduler process user (passwordless SSH). Submission to the native scheduler will be performed with the same account.
- sudo: similar to none regarding the connection to the head node, but a sudo command will be initiated to impersonate as the current ProActive Scheduler user, before doing a job submission.
alternateRMUrl: the URL used by the ProActive Nodes to contact ProActive Resource Manager. This URL is displayed on ProActive server startup. Example: pnp://myserver:64738.
sshPort: port used for SSH connections.
nsPreCommand: a Linux command which can be run before launching ProActive Nodes on the cluster. Can be used as a workaround when some system environment variables are not properly set when starting ProActive Nodes.

nsSubmitCommand: this is the main command used to start ProActive Nodes on the cluster. Depending on the actual native scheduler implementation, nsSubmitCommand will vary, here are examples definitions:

PBS

qsub -N %NS_JOBNAME% -o %LOG_FILE% -j oe %NS_BATCH%

SLURM

sbatch -J %NS_JOBNAME% -o %LOG_FILE% %NS_BATCH%

LSF

bsub -J %NS_JOBNAME% -o %LOG_FILE% -e %LOG_FILE% %NS_BATCH%

The command can use patterns which will be replaced dynamically by the ProActive Resource Manager.

%NS_JOBNAME%

contains a configurable job name dynamically created by the resource manager.

%LOG_FILE%

contains a log file path dynamically created by the resource manager and located in side the NSSchedulerHome installation. This log file is useful to debug errors during cluster job submission.

%PA_USERNAME%

contains the current ProActive Scheduler user.

%NS_BATCH%

contains the arguments defined inside workflow tasks using the NS_BATCH generic information.

nsKillCommand: this is the command used to kill ProActive Nodes started previously by the nsSubmitCommand. Similarly to nsSubmitCommand, nsKillCommand will vary for each native scheduler syntax:

PBS

qdel %NS_JOBID%

SLURM

scancel -n %NS_JOBNAME%

LSF

bkill -J %NS_JOBNAME%

It can use the following patterns:

%NS_JOBNAME%

contains a configurable job name dynamically created by the resource manager.

%NS_JOBID%

contains the job id returned by the native scheduler when submitting the job. Currently, job id can only be used with PBS, when the setting submitReturnsJobId is set to true.

submitReturnsJobId: is the cluster job id returned plainly when calling the nsSubmitCommand. This is the behavior of PBS, and this is why this setting should be set to true when using PBS.
nsJobName: a way to configure the %NS_JOBNAME% pattern. The following patterns can be used:

%PA_TASKID%

contains the ProActive Task and Job ID associated with the node request.

%PA_USERNAME%

contains the current ProActive Scheduler user.
maxDeploymentFailure: number of attempts when starting a ProActive Node on the cluster using the nsSubmitCommand, after all attempts failed, the ProActive Node will be declared as Lost.
singleConnectionUserName: user name for single SSH connection configuration. Single SSH connection credentials can be used when setting impersonation method to sudo or none. It must be used also when enabling the native scheduler metrics acquisition (see below).
singleConnectionPassword: password for single SSH connection configuration. Single SSH connection credentials can be used when setting impersonation method to sudo or none. It must be used also when enabling the native scheduler metrics acquisition (see below).
singleConnectionPrivateKey: private key for single SSH connection configuration. Single SSH connection credentials can be used when setting impersonation method to sudo or none. It must be used also when enabling the native scheduler metrics acquisition (see below).
impersonationCommand: command to use when impersonation method is sudo. The following patterns can be used:

%COMMAND%

contains the command to impersonate.

%PA_USERNAME%

contains the current ProActive Scheduler user.
nativeSchedulerMetricsCommandFile: Native scheduler metrics are displayed on the Resource Manager portal when selecting the node source. In order to enable metrics display, the single SSH connection settings and the two metrics parameters must be defined. nativeSchedulerMetricsCommandFile contains a list of commands which will gather metrics. The file structure is as follows:
- each metric must be a single line command
- each command must be preceded by a comment line # containing the category and metric title separated by |. + Example:
  # PBS Metrics | Jobs qstat
nativeSchedulerMetricsPeriod: Refresh period in seconds of native scheduler metrics. In order to enable metrics display, a value greater than zero must be defined. Default to -1 (disabled)

7. Control the resource usage

7.1. Node Source Policies

You can limit the utilization of resources connected to the ProActive Scheduler in different ways. When you create node sources you can use a Node Source Policy. A node source policy is a set of rules and conditions which describes, inside a Node Source, when and how many nodes have to be acquired or released.

When creating a node source, a Node Source Infrastructure and a Node Source Policy must be chosen. Node Sources can be created using the Resource Manager portal or the command line (see Command Line Examples for more information).

The Node Source Policy interacts internally with the associated Node Source Infrastructure to request new nodes deployments or nodes removal.

Node Sources were designed in a way that:

All logic related to node acquisition is encapsulated in the Infrastructure Manager.
Conditions and rules of node acquisition is described in the Node Source Policy.

In the ProActive Resource Manager, there is always a default node source configured with a DefaultInfrastructureManager and a Static policy. It is not able to deploy nodes anywhere but makes it possible to add existing nodes to the Scheduler (see Deploy ProActive Nodes manually)

Out of the box the Scheduler supports time slot policies, cron policies, load based policies and many others. Please see detailed information about policies in the following sections.

7.1.1. Common Parameters

Each node source policy regardless its specifics has a common part where you describe users, groups or tenants permissions. When you create a policy you must specify them:

nodeUsers - utilization permission that defines who can get nodes for computations from this node source. It has to take one of the following values:
- ME - only the node source creator
- users=user1,user2;groups=group1,group2;tenants=tenant1,tenant2;tokens=t1,t2 - only specific users, groups, tenants or tokens. For example:
  - users=user1 - node access is limited to user1
  - tenants=tenant1 - node access is limited to users belonging to tenant1
  - users=user1;groups=group1 - node access is limited to user1 and all users from group group1
  - users=user1;tokens=t1 - node access is limited to user1 or anyone who specified token t1. If node access is protected by a token, node will not be found by the ProActive Resource Manager when trying to execute workflows, unless the corresponding token is specified inside the workflow(s). To exclude specific users or groups use for example:
  - users=!user1,!user2;groups=!group1,!group2. Tokens can not be excluded, and mixing inclusion with exclusion is not permitted (i.e. users=user1,!user2;groups=!group1,group2).
- ALL - everybody can use nodes from this node source

To specify a token inside a chosen workflow, add the key/value pair NODE_ACCESS_TOKEN:<token> to its generic information.

nodeProviders - provider permission defines who can add nodes to this node source. It should take one of the following values:
- ME - pnly the node source creator
- users=user1,user2;groups=group1,group2;tenants=tenant1,tenant2 - only specific users, groups or tenants (for our example user1, user2, group1, group2, tenant1 or tenant2). It is also possible to specify only groups, only users or only tenants.
- ALL - everybody can add nodes to this node source

The user who created the node source is the administrator of this node source. He can add and removed nodes to it, remove the node source itself, but cannot use nodes if usage policy is set to PROVIDER or PROVIDER_GROUPS.

Node source policy user restrictions are not enforced for members of the deprecated admin group, they will have no effect on such users (see User Permissions).

New Node Source policies can be dynamically plugged into the Resource Manager. In order to do that, it is required to add classes extending the NodeSourcePolicy class in the server class path and update the policy implementation list in the configuration file (PROACTIVE_HOME/config/rm/nodesource/policies).

7.1.2. Static Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.StaticPolicy

Static node source policy starts node acquisition when nodes are added to the node source and never removes them. Nevertheless, nodes can be removed by user request. To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters

7.1.3. Restart Down Node Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.RestartDownNodePolicy

Restart down node policy automatically redeploy nodes when they are disconnected from the Resource Manager (for example after a network failure or a machine reboot). To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
checkNodeStateEach - interval in milliseconds used to check nodes availability. For example, if this interval is set to 30 minutes the policy will redeploy nodes after a disconnection of 30 minutes.

7.1.4. Time Slot Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.TimeSlotPolicy

Time slot policy is aimed to acquire nodes for a specific time window with an ability to do it periodically. To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
acquireTime - Absolute acquire date (e.g. "6/3/10 1:18:45 PM CEST").
releaseTime - Absolute releasing date (e.g. "6/3/10 2:18:45 PM CEST").
period - Period time in millisecond (default is 86400000).
preemptive - Preemptive parameter indicates the way of releasing nodes. If it is true, nodes will be released without waiting the end of jobs running on (default is false).

7.1.5. Cron Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.CronPolicy

Cron policy is aimed to acquire and remove nodes at a specific time defined in the cron syntax. To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
nodeAcquision - The time policy will trigger the deployment of all nodes (e.g. "0 12 \* \* \*" every day at 12.00).
nodeRemoval - The time policy will trigger the removal of all nodes (e.g. "0 13 \* \* \*" every day at 13.00).
preemptive - Preemptive parameter indicates the way of releasing nodes. If it is true, nodes will be released without waiting the end of jobs running on (default is false).
forceDeployment - If for the example above (the deployment starts every day at 12.00 and the removal starts at 13.00) you are creating the node source at 12.30 the next deployment will take place the next day. If you’d like to force the immediate deployment set this parameter to true.

7.1.6. Release Resources When Scheduler Is Idle

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.ReleaseResourcesWhenSchedulerIdle

"Release resources when scheduler is idle" policy removes all nodes from the infrastructure when the scheduler is idle and acquires them when a new job is submitted. This policy may be useful if there is no need to keep nodes alive permanently. Nodes will be released after a specified "idle time". This policy will use a listener of the scheduler, that is why its URL, its user name and its password have to be specified. To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
schedulerURL - URL of the Scheduler, e.g. pnp://mymachine:64738, pamr://0, etc
schedulerCredentialsPath - Path to the credentials used for scheduler authentication.
schedulerAwarePolicyNodeSourceRecoveryDelay - Delay in ms for the Resource Manager to recover a broken node source using the ReleaseResourcesWhenSchedulerIdle policy.
schedulerAwarePolicyNodeSourceRecoveryTrialsNumber - Number of attempts for the Resource Manager to recover to recover a broken node source using the ReleaseResourcesWhenSchedulerIdle policy.
idleTime - Idle time in millisecond to wait before removing all nodes (default is 60000).

7.1.7. Scheduler Loading Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.SchedulerLoadingPolicy

Scheduler loading policy acquires/releases nodes according to the scheduler loading factor. This policy allows to configure the number of resources which will be always enough for the scheduler. Nodes are acquired and released according to scheduler loading factor which is a number of tasks per node.

It is important to correctly configure maximum and minimum nodes that this policy will try to hold. Maximum number should not be greater than potential nodes number which is possible to deploy to underlying infrastructure. If there are more currently acquired nodes than necessary, policy will release them one by one after having waited for a "release period" delay. This smooth release procedure is implemented because deployment time is greater than the release one. Thus, this waiting time deters policy from spending all its time trying to deploy nodes.

To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
schedulerURL - URL of the Scheduler, e.g. pnp://mymachine:64738, pamr://0, etc
schedulerCredentialsPath - Path to the credentials used for scheduler authentication.
schedulerAwarePolicyNodeSourceRecoveryDelay - Delay in ms for the Resource Manager to recover a broken node source using the SchedulerLoadingPolicy.
schedulerAwarePolicyNodeSourceRecoveryTrialsNumber - Number of attempts for the Resource Manager to recover to recover a broken node source using the SchedulerLoadingPolicy.
refreshTime - Time between each calculation of the number of needed nodes.
minNodes - Minimum number of nodes to deploy
maxNodes - Maximum number of nodes to deploy
loadFactor - Number of tasks per node. Actually, if this number is N, it does not means that there will be exactly N tasks executed on each node. This factor is just used to compute the total number of nodes. For instance, let us assume that this factor is 3 and that we schedule 100 tasks. In that case, we will have 34 (= upper bound of 100/3) started nodes. Once one task finished and the refresh time passed, one node will be removed since 99 divided by 3 is 33. When there will remain 96 tasks (assuming that no other tasks are scheduled meanwhile), an other node will be removed at the next calculation time, and so on and so forth…
refreshCycleWindow - Number of refresh cycles used to memorize scheduler load. Actual load will be computed as an average.
releaseDelay - Delay in milliseconds used before removing a free node. This parameter can be used for paid cloud instances such as Amazon, Azure, etc. when their policy define a minimum granularity on instance costs. Example for Amazon : 3600000 ms (1 hour). Default is 0 (not a paid instance, no delay).
threshold - Nodes can be released if the current time is in interval [releaseDelay - threshold, releaseDelay]. Default is 0 (not a paid instance, no delay). Related to releaseDelay. Example for amazon : 600000 ms (10 minutes).

7.1.8. Cron Load Based Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.CronLoadBasedPolicy

The Cron load based policy triggers new nodes acquisition when scheduler is overloaded (exactly like with "Scheduler loading" policy) only within a time slot defined using crontab syntax. All other time the nodes are removed from the Resource Manager. To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
schedulerURL - URL of the Scheduler, e.g. pnp://mymachine:64738, pamr://0, etc
schedulerCredentialsPath - Path to the credentials used for scheduler authentication.
schedulerAwarePolicyNodeSourceRecoveryDelay - Delay in ms for the Resource Manager to recover a broken node source using the CronLoadBasedPolicy.
schedulerAwarePolicyNodeSourceRecoveryTrialsNumber - Number of attempts for the Resource Manager to recover to recover a broken node source using the CronLoadBasedPolicy.
refreshTime - Time between each calculation of the number of needed nodes.
minNodes - Minimum number of nodes to deploy
maxNodes - Maximum number of nodes to deploy
loadFactor - Number of tasks per node. Actually, if this number is N, it does not means that there will be exactly N tasks executed on each node. This factor is just used to compute the total number of nodes. For instance, let us assume that this factor is 3 and that we schedule 100 tasks. In that case, we will have 34 (= upper bound of 100/3) started nodes. Once one task finished and the refresh time passed, one node will be removed since 99 divided by 3 is 33. When there will remain 96 tasks (assuming that no other tasks are scheduled meanwhile), an other node will be removed at the next calculation time, and so on and so forth…
refreshCycleWindow - Number of refresh cycles used to memorize scheduler load. Actual load will be computed as an average.
releaseDelay - Delay in milliseconds used before removing a free node. This parameter can be used for paid cloud instances such as Amazon, Azure, etc. when their policy define a minimum granularity on instance costs. Example for Amazon : 3600000 ms (1 hour). Default is 0 (not a paid instance, no delay).
threshold - Nodes can be released if the current time is in interval [releaseDelay - threshold, releaseDelay]. Default is 0 (not a paid instance, no delay). Related to releaseDelay. Example for amazon : 600000 ms (10 minutes).
acquisionAllowed - The time when the policy starts to work as the "scheduler loading" policy (e.g. "0 12 \* \* \*" every day at 12.00).
acquisionForbidden - The time policy removes all the nodes from the Resource Manager (e.g. "0 13 \* \* \*" every day at 13.00).
preemptive - Preemptive parameter indicates the way of releasing nodes. If it is true, nodes will be released without waiting the end of jobs running on (default is false).
allowed - If true acquisition will be immediately allowed.

7.1.9. Cron Slot Load Based Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.CronSlotLoadBasedPolicy

The "Cron slot load based" policy triggers new nodes acquisition when scheduler is overloaded (exactly like with "Scheduler loading" policy) only within a time slot defined using crontab syntax. The other time it holds all the available nodes. To use this policy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
schedulerURL - URL of the Scheduler, e.g. pnp://mymachine:64738, pamr://0, etc
schedulerCredentialsPath - Path to the credentials used for scheduler authentication.
schedulerAwarePolicyNodeSourceRecoveryDelay - Delay in ms for the Resource Manager to recover a broken node source using the CronSlotLoadBasedPolicy.
schedulerAwarePolicyNodeSourceRecoveryTrialsNumber - Number of attempts for the Resource Manager to recover to recover a broken node source using the CronSlotLoadBasedPolicy.
refreshTime - Time between each calculation of the number of needed nodes.
minNodes - Minimum number of nodes to deploy
maxNodes - Maximum number of nodes to deploy
loadFactor - number of tasks per node. Actually, if this number is N, it does not means that there will be exactly N tasks executed on each node. This factor is just used to compute the total number of nodes. For instance, let us assume that this factor is 3 and that we schedule 100 tasks. In that case, we will have 34 (= upper bound of 100/3) started nodes. Once one task finished and the refresh time passed, one node will be removed since 99 divided by 3 is 33. When there will remain 96 tasks (assuming that no other tasks are scheduled meanwhile), an other node will be removed at the next calculation time, and so on and so forth…
refreshCycleWindow - Number of refresh cycles used to memorize scheduler load. Actual load will be computed as an average.
releaseDelay - Delay in milliseconds used before removing a free node. This parameter can be used for paid cloud instances such as Amazon, Azure, etc. when their policy define a minimum granularity on instance costs. Example for Amazon : 3600000 ms (1 hour). Default is 0 (not a paid instance, no delay).
threshold - Nodes can be released if the current time is in interval [releaseDelay - threshold, releaseDelay]. Default is 0 (not a paid instance, no delay). Related to releaseDelay. Example for amazon : 600000 ms (10 minutes).
deployAllAt - Time when all nodes are deployed (crontab format) (e.g. "0 12 \* \* \*" every day at 12.00).
undeployAllAt - Time when all nodes are removed and the policy starts watching the scheduler load (e.g. "0 13 \* \* \*" every day at 13.00).
preemptive - Preemptive parameter indicates the way of releasing nodes. If it is true, nodes will be released without waiting the end of tasks running on (default is false).
acquireNow - If true, the policy will acquire all nodes immediately.

7.1.10. Native Scheduler Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.NativeSchedulerPolicy

The Native Scheduler Policy interacts with the native scheduler to request ProActive nodes deployment dynamically based on the ProActive Scheduler pending queue. This mechanism is described in Deploy ProActive Nodes dynamically via other schedulers (PBS, SLURM, …). This policy must be associated with a Native Scheduler Infrastructure and cannot be associated with any other infrastructure. To use this policy, you need to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
schedulerUrl - URL of the Scheduler, e.g. pnp://mymachine:64738, pamr://0, etc
schedulerCredentialsPath - Path to a file which contains the credentials of an administrator user which will connect to the scheduler. The ProActive Scheduler Server release contains two admin users credentials files : config/authentication/rm.cred and config/authentication/admin_user.cred
rearrangeTasks - Currently not implemented.
autoScaling - If set to true, the NativeSchedulerPolicy will scan the Resource Manager activity and Scheduler queue. If the scheduler queue is not empty and all Resource Manager nodes are busy, autoscaling will automatically start ProActive Nodes from the NativeSchedulerInfrastructure. This setting cannot be used when multiple NativeScheduler node sources are deployed.
refreshTime - The NativeSchedulerPolicy will refresh its status and observe the ProActive Scheduler queue every refreshTime milliseconds.

7.1.11. Dynamic Policy

Full name: org.ow2.proactive.resourcemanager.nodesource.policy.DynamicPolicy

The Dynamic policy is similar to the Scheduler Loading Policy in the sense that it can acquire/release nodes according to the scheduler load. The Dynamic policy is more advanced and gives a fuller range of configuration parameters which allow to:

filter the scheduler load to consider specific tasks which use the NODE_SOURCE or the (deprecated) NODESOURCENAME generic information.
delay the initial triggering of the policy.
release nodes after a configurable idle period.

This policy is very well suited to Cloud infrastructures (Azure, EC2, etc) when cloud instances need to be acquired on-demand.

Similarly to the Scheduler Loading Policy, it is important to correctly configure the maximum and minimum nodes that this policy will try to hold in accordance with the underlying infrastructure. This policy will never acquire more nodes than the infrastructure is capable of (as configured in the infrastructure parameters). Symmetrically, if an infrastructure defines a minimum number of nodes to acquire, this policy will never go below this number.

The DynamicPolicy will scale up when pending tasks are detected in the ProActive Scheduler queue and the Node Source does not have enough free nodes to assign.

In some cases, the existing free nodes will not match the requirements of these pending tasks. This can happen when the task contains a Selection Script or a NODE_ACCESS_TOKEN restriction that discards existing free nodes. In that case, the Dynamic Policy will use a best-effort strategy to handle these tasks that remain in pending state. When tasks remain in pending state longer than initDelay milliseconds, the node source will trigger a scale-up (one new node per pending task).

The loadFactor is the main parameter that allows to control the scale up/down of the DynamicPolicy. It decides how many new ProActive Nodes will be deployed when considering the current number of pending and running tasks. loadFactor is a ratio which computes the expected number of Nodes in the Node Source according to the formula:

expectedNbNodes = Ceiling((nbPendingTasks + nbBusyNodes) / loadFactor)

Two additional parameters allow to fine-tune this formula:

loadFactorRoundingUp: when false, Ceiling() is replaced by Floor() in the above formula (rounding down instead of rounding up).
loadFactorLegacyBehavior: when true, the formula is transformed to:

expectedNbNodes = Ceiling(nbPendingTasks / loadFactor + nbBusyNodes)

This means that new Nodes will be deployed as long as there are pending tasks remaining, regardless of the loadFactor value. In that case, loadFactor controls only how many new Nodes are deployed per policy cycle and not how many Nodes are present in the Node Source with regard to the number of tasks to run.

Finally, it is also possible to fully control the scale up/down formula and even link it with the state of other Node Sources. This is achieved by using the scaleNodesExpression parameter described in Scale Nodes SpEL expression.

Dynamic Policy Resource Tagging

The DynamicPolicy has the ability to add tags to resources acquired by cloud providers. The tagging mechanism depends on the infrastructure associated with the DynamicPolicy. Currently, only the Azure Scale Set Infrastructure supports resource tagging (see Scale Set Tagging for more information).

Tagging can be activated by adding the RESOURCE_TAGS Generic Information to workflows or workflow tasks that will target this node source. RESOURCE_TAGS must contain the following json key/value structure:

{
  "tagName1" :  "tagValue1",
  "tagName2" :  "tagValue2"
}

When the node source scales up due to the scheduling of a task with the above RESOURCE_TAGS generic information example, resources will be tagged with 2 tags tagName1:tagValue1 and tagName2:tagValue2.

Dynamic Policy Parameters

To use the DynamicPolicy, you have to define the following parameters:

userAccessType - see Common Parameters
providerAccessType - see Common Parameters
minNodes - Minimum number of nodes to deploy
maxNodes - Maximum number of nodes to deploy
schedulerUrl - URL of the Scheduler, e.g. pnp://mymachine:64738, pamr://0, etc
schedulerCredentialsPath - Path to the credentials used for scheduler authentication.
schedulerConnectionTimeout - Timeout in ms to establish connection with the scheduler.
schedulerConnectionTrialsNumber - Number of attempts to connect with the scheduler.
refreshTime - Time in milliseconds between each calculation of the number of needed nodes.
loadFactor - loadFactor allows to control the number of Nodes deployed in relation with the number of pending Tasks to be executed. loadFactor controls needed Nodes to execute pending Tasks according to the following formula: neededNodes = Ceiling((pendingTasks + busyNodes) / loadFactor). The Ceiling() function is used because neededNodes must be an integer. Ceiling() rounds the decimal number to the nearest integer above the value (e.g. Ceiling(3.5)=4, see https://en.wikipedia.org/wiki/Floor_and_ceiling_functions). For example, if loadFactor is 1 (the default), as soon as one task is pending and no Node is free to execute it, the Node Source will deploy one new Node. When loadFactor is 10, only 1 Node will be deployed to execute 1 to 10 pending tasks. When 10 tasks are pending, 1 Node will be deployed and execute the first pending task. After this task has finished its execution, the second task will run on the same Node, etc. If at some point 15 tasks are pending, and 1 Node is busy executing a task, a second Node will be deployed. Because, in that case neededNodes = Ceiling(15(pending) + 1(busy)/10) = 2.

The behavior of loadFactor can be further controlled using loadFactorRoundingUp and loadFactorLegacyBehavior parameters (see below).
loadFactorRoundingUp - Boolean value which decides if the Ceiling() function or the Floor() function should be used inside the loadFactor formula. When loadFactorRoundingUp is true (default), the Ceiling() function will be used. This will round up the neededNodes calculation (described in the example above). When loadFactorRoundingUp is false, the Floor() function will be used instead. This will round down the neededNodes calculation. In that case, when for example loadFactor is 10, a new Node will be deployed in the Node Source when at least 10 tasks are pending. If 9 Tasks are pending, neededNodes = Floor(9(pending) + 0(busy)/10) = 0. If 15 Tasks are pending, neededNodes = Floor(15(pending) + 0(busy)/10) = 1.
loadFactorLegacyBehavior - Boolean value which decides if the legacy loadFactor behavior should be enabled (default to false). When loadFactorLegacyBehavior is true, it modifies the loadFactor formula to: neededNodes = Ceiling(pendingTasks / loadFactor) + busyNodes. In that case, the loadFactor applies only to pendingTasks. This means that as long as pending tasks are present, new nodes will be deployed (up to the maxNodes parameter). This behavior does not really satisfy the philosophy of loadFactor which aims to deploy a limited number of nodes to handle a larger amount of short-lived tasks. loadFactorLegacyBehavior = true should be used only for backward-compatibility reasons.
initDelay - Delay in milliseconds to initialize the infrastructure (eg. in a azure cloud infrastructure this must cover the creation of Azure’s initial resources such as network, storage account, etc.). This parameter is also used as a trigger for the best-effort scale-up strategy described above.
minUptime - Minimal uptime of a free node to be candidate for deletion (in milliseconds). For example, if this parameter is set to 10 minutes, a node will not be removed from the node source unless it stays 10 minutes consecutively in state FREE, without executing any new task.
globalScope - Specifies the scope of the policy: If true, all the pending tasks will be taken into account for scaling up. If false, only the pending tasks targeting this specific Node Source will be taken into account. Pending tasks can target a Node Source by defining either the NODE_SOURCE or the (deprecated) NODESOURCENAME generic information.
scaleNodesExpression: advanced parameter allowing to fully control the formula used to scale up or scale down Nodes. The parameter expects a SpEL expression with a syntax described below.

Scale Nodes SpEL expression

scaleNodesExpression is a powerful, advanced parameter which allows full control on the scale up and scale down of a DynamicPolicy.

scaleNodesExpression uses the Spring Expression Language syntax

scaleNodesExpression must return an integer with the following semantic:

a positive integer N will require the Node Source to deploy N new Nodes.
a negative integer N will require the Node Source to remove N Nodes (only Nodes that have been in FREE state for more than minUptime milliseconds can be removed).
0 will keep the current Nodes unchanged.

Variables and Functions

The expression can use the following given variables:

loadFactor: the currently configured load factor in the DynamicPolicy (see Dynamic Policy Parameters).
globalScope: the currently configured scope of the DynamicPolicy (see Dynamic Policy Parameters).
aliveNodes : the current number of alive Nodes in the Node Source when globalScope is false. Otherwise, aliveNodes will be the current number of alive Nodes in the Resource Manager. Alive Nodes are Nodes in the following states: DEPLOYING, CONFIGURING, FREE, BUSY or TO_BE_REMOVED (see Node States).
aliveNodesInNodeSource: the current number of alive nodes in the Node Source, regardless of the globalScope configuration.
busyNodes: the current number of Nodes in BUSY state in the Node Source when globalScope is false. Otherwise, busyNodes will be the current number of Nodes in BUSY state in the Resource Manager.
busyNodesInNodeSource: the current number of Nodes in BUSY state in the Node Source regardless of the globalScope configuration.
deployingNodes: the current number of Nodes in DEPLOYING state in the Node Source (does not consider the globalScope parameter).
neededNodes: the number of Nodes needed by pending tasks. This value contains all pending tasks when globalScope is true and only tasks which define the NODE_SOURCE or the NODESOURCENAME generic information when globalScope is false. Also, this value considers multi-node pending tasks (for example, a task can require using 3 Nodes).
neededNodesPrevious: the number of Nodes that were needed at the previous refresh cycle. This value can be used to check if tasks keep staying pending.
temp, temp2, temp3, temp4: each of these variables can be set to a temporary object used in the SpEL expression. Affectations should use the z() function described below. e.g. z(temp=neededNodes+busyNodes)
tempMap: an empty Hash Map structure which can be populated and used in the SpEL expression. Affectations to the map should use the z() function described below. e.g. z(tempMap.put('key', 'value')).
nodeSources: a key-value HashMap which contain the state of all Node Sources in the Resource Manager. The key is the Node Source name and the value contains the following properties:
- acquiredNodes: the current number of acquired Nodes in this Node Source. acquiredNodes contain Nodes in CONFIGURING, FREE, BUSY or TO_BE_REMOVED states. This property is accessed using nodeSources.get('node-source-name').getAcquiredNodes().
- aliveNodes : the current number of alive Nodes in the Node Source. Alive Nodes are Nodes in DEPLOYING, CONFIGURING, FREE, BUSY or TO_BE_REMOVED states. This property is accessed using nodeSources.get('node-source-name').getAliveNodes().
- deployingNodes: the current number of Nodes in DEPLOYING state in this Node Source. This property is accessed using nodeSources.get('node-source-name').getDeployingNodes().
- busyNodes: the current number of Nodes in BUSY or TO_BE_REMOVED states in this Node Source. This property is accessed using nodeSources.get('node-source-name').getBusyNodes().
- freeNodes: the current number of Nodes in FREE state in this Node Source. This property is accessed using nodeSources.get('node-source-name').getFreeNodes().
- totalNodes: the total number of Nodes in this Node Source. This property is accessed using nodeSources.get('node-source-name').getTotalNodes().
- descriptor: a NodeSourceDescriptor object containing all the parameters of this Node Source. This property is accessed using nodeSources.get('node-source-name').getDescriptor(). The NodeSourceDescriptor object contain many methods, among which the getInfrastructureParameter(name) and the getPolicyParameter(name) will be the most useful.
  
  For example, it is possible to access the maxNodes parameter of a specific Node Source that uses a DynamicPolicy:
  T(Integer).parseInt(nodeSources.get('node-source-name').getDescriptor(). getPolicyParameter('maxNodes'))
  As getPolicyParameter returns a string, it is necessary to transform it into an integer before doing any integer comparison.
previousNodeSources: this variable contains a similar structure as nodeSources but retrieved from the previous refresh cycle. This can be used to ensure that the state of another Node Source remains stable between 2 cycles and take scale up/down decisions accordingly. Please note that on the first cycle, previousNodeSources will contain an empty HashMap. It is thus advised to ensure in the SpEL expression that returned node source information are not null. An example usage of previousNodeSources is described in Example Expressions.

scaleNodesExpression can also use the following function (in addition to the functions available by default in the SpEL language):

z(expression): evaluate the expression and return zero. This is used to allow performing affectations to the temp or tempMap variables. For example:

z(temp=neededNodes+busyNodes)+T(Math).ceil(temp/loadFactor)-aliveNodes

In this expression, the z() function prevents the affectation from adding its value to the total.

Example Expressions

Below is an example of a scaleNodesExpression that reproduces the default behavior of the DynamicPolicy:

T(Math).ceil((neededNodes + busyNodes) / loadFactor) - aliveNodes

Explanation:

T(Math).ceil((neededNodes + busyNodes) / loadFactor): this computes the expected number of Nodes. When loadFactor is 1, the expected number of Nodes is the sum of pending tasks and current Nodes in BUSY state. A loadFactor greater than 1 will reduce the amount of expected Nodes.
- aliveNodes: the number of alive Nodes is subtracted to the expected number of Nodes to return the signed difference. The returned value will either be positive (scale up) or negative (scale down).

Below is an example of a scaleNodesExpression that will scale up the Node Source when another Dynamic Policy Node Source has reached maximum capacity (this expression considers that globalScope is true for both Node Sources). The following expression must be defined on the second Node Source:

z(temp=nodeSources.get('FirstNS')) + z(temp2=previousNodeSources.get('FirstNS')) + z(temp3=(temp2!=null?T(Math).min(temp.getBusyNodes(),temp2.getBusyNodes()):0)) + (temp3 == T(Integer).parseInt(temp.getDescriptor().getPolicyParameter('maxNodes')) ? T(Math).ceil((neededNodes + busyNodesInNodeSource) / loadFactor) - aliveNodesInNodeSource : -aliveNodesInNodeSource )

Explanation:

The main structure of this expression uses the test ? case1 : case2 syntax, which evaluates case1 when test is true and case2 when test is false.

z(temp=nodeSources.get('FirstNS')): read the state of the first Node Source and store it into a temporary variable. Notice the use of the z() function and the + sign, so that this affectation does not change the value returned globally by the SpEL expression.
z(temp2=previousNodeSources.get('FirstNS')): read the previous state of the first Node Source and store it into another temporary variable.
z(temp3=(temp2!=null?T(Math).min(temp.getBusyNodes(),temp2.getBusyNodes()):0)): store into another variable the minimum amount of busy nodes that were accounted on the first Node Source in both refresh cycle. If the previous cycle is empty, store 0 into the variable.
temp3 == T(Integer).parseInt(temp.getDescriptor().getPolicyParameter('maxNodes')) : check if the number of busy Nodes in the Node Source FirstNS is equal to the maximum configured for the first Node Source for 2 consecutive refresh periods.
T(Math).ceil((neededNodes + busyNodesInNodeSource) / loadFactor) - aliveNodesInNodeSource: when the maximum number of nodes is reached on the first Node Source, the second Node Source will scale up based on the default scale up expression (see previous example). Note that we use busyNodesInNodeSource and aliveNodesInNodeSource to count only Nodes that belong to the second Node Source.
-aliveNodesInNodeSource: when the first Node Source is not at the maximum, remove all alive Nodes in the second Node Source (in reality, only FREE Nodes will be removed after minUptime milliseconds).

7.1.12. Dynamic Policy Canonical Examples

In this section we will present several canonical use cases that make use of node sources controlled by a DynamicPolicy:

A single cloud Dynamic Infrastructure
A Static Infrastructure (on-premise or cloud) + a single Cloud Dynamic Infrastructure.
A primary cloud Dynamic Infrastructure + a secondary Cloud Dynamic Infrastructure.
A Static Infrastructure (on-premise or cloud) + a primary Cloud Dynamic Infrastructure + a secondary Cloud Dynamic Infrastructure.

The last two cases are especially suitable when one is looking to optimize the cloud cost of one’s infrastructure using preemptive Spot VMs. For example, your Jobs can be executed in priority on the low cost Spot VMs or static infrastructure (that can be lower-cost pre-reserved instances). Only when Spot VMs are becoming scarce or your workload exceeds your Spot quota, you will be automatically using the Standard VMs.

Single Cloud Dynamic Infrastructure

In this use case, the Resource Manager defines a single cloud infrastructure with a DynamicPolicy attached. For the type of cloud infrastructure, we can use for example an Azure Scale Set Infrastructure, an AWS Autoscaling Infrastructure, a GCE Infrastructure, etc…

The configuration of the DynamicPolicy in this use case is fairly simple, we only need to set globalScope to true to ensure that Nodes will be dynamically deployed whenever PENDING tasks appear in the ProActive Scheduler queue. We control the cloud node source capacity using the minNodes and maxNodes parameters. And minUptimes will be adjusted to determine how long a FREE Node will remain before scaling down.

Static Infrastructure and single Cloud Dynamic Infrastructure

In this use case, the Resource Manager defines two Node Sources:

A first Node Source 'StaticNS' which will be controlled by a StaticPolicy. This Node Source can either use an on-premise infrastructure such as SSH Infrastructure V2 or a cloud infrastructure such as Azure Scale Set Infrastructure).
A second Node Source 'DynamicNS' which uses a cloud infrastructure and a DynamicPolicy.

The general configuration mentioned in the previous scenario will be applied to 'DynamicNS'. globalScope must be set to true. In addition, a scaleNodesExpression must be defined to ensure that the dynamic cloud Node Source will scale up when all Nodes of 'StaticNS' are used.

The maximum number of Nodes of 'StaticNS' cannot be easily determined by reading the parameters of its infrastructure, as it highly depends on the type of infrastructure selected (e.g. LocalInfrastructure, SSHInfrastructureV2). On the other hand, we can assume that 'StaticNS' is fully used when the number of FREE and DEPLOYING Nodes is zero and the number of BUSY Nodes is greater than zero.

scaleNodesExpression defined in 'DynamicNS':

z(temp=nodeSources.get('StaticNS')) + z(temp2=previousNodeSources.get('StaticNS')) + z(temp3=(temp2!=null?temp2.getFreeNodes()+temp2.getDeployingNodes()+temp.getFreeNodes()+temp.getDeployingNodes():-1)) + z(temp4=(temp2!=null?T(Math).min(temp2.getBusyNodes(),temp.getBusyNodes()):0)) + (temp3 == 0 && temp4 > 0 ? T(Math).ceil((neededNodes + busyNodesInNodeSource) / loadFactor) - aliveNodesInNodeSource : -aliveNodesInNodeSource )

We query 'StaticNS' info through 2 successive cycles (using nodeSources and previousNodeSources) to ensure that the observed state is stable. In temp3 we assign a number that will be zero when no DEPLOYING or FREE Node was observed. This number will be a non-zero value on the very first cycle. In temp4 we assign the minimum number of BUSY Nodes observed.

'DynamicNS' will scale up when temp3 is zero and temp4 is greater than zero. It will scale down otherwise.

Cloud Dynamic Infrastructure and Secondary Cloud Dynamic Infrastructure

In this use case, the Resource Manager defines two Node Sources:

A first Node Source 'DynamicNS' which will be controlled by a DynamicPolicy and using cloud infrastructure such as Azure Scale Set Infrastructure).
A second Node Source 'DynamicNSExtended' which uses a similar setup.

The configuration of 'DynamicNS' will be a basic configuration as described in Single Cloud Dynamic Infrastructure with globalScope true.

The configuration of 'DynamicNSExtended' will also use globalScope true and add a scaleNodesExpression to ensure that the secondary Node Source will scale up only when 'DynamicNS' is fully used.

scaleNodesExpression defined in 'DynamicNSExtended':

z(temp=nodeSources.get('DynamicNS')) + z(temp2=previousNodeSources.get('DynamicNS')) + z(temp3=(temp2!=null?T(Math).min(temp.getBusyNodes(),temp2.getBusyNodes()):0)) + (temp3 == T(Integer).parseInt(temp.getDescriptor().getPolicyParameter('maxNodes')) ? T(Math).ceil((neededNodes + busyNodesInNodeSource) / loadFactor) - aliveNodesInNodeSource : -aliveNodesInNodeSource )

We query 'DynamicNS' info through 2 successive cycles (using nodeSources and previousNodeSources) to ensure that the observed state is stable. In temp3 we assign the minimum busy nodes amount in 'DynamicNS' the two cycles.

'DynamicNS' will scale up when there are pending tasks in ProActive Scheduler queue. 'DynamicNSExtended' will scale up when the amount of BUSY Nodes in 'DynamicNS' is equal to the maximum number of Nodes configured, and will scale down otherwise.

Static Infrastructure, Primary Cloud Dynamic Infrastructure and Secondary Cloud Dynamic Infrastructure

In this use case, the Resource Manager defines three Node Sources:

A first Node Source 'StaticNS' which will be controlled by a StaticPolicy. This Node Source can either use an on-prem infrastructure such as SSH Infrastructure V2 or a cloud infrastructure such as Azure Scale Set Infrastructure, or pre-reserved instances).
A second Node Source 'DynamicNS' which will be controlled by a DynamicPolicy and using cloud infrastructure such as Azure Scale Set Infrastructure, typically Spot VMs).
A third Node Source 'DynamicNSExtended' which uses a similar setup, but typically using Standards VMs.

This use case is a combination of Static Infrastructure and single Cloud Dynamic Infrastructure and Cloud Dynamic Infrastructure and Secondary Cloud Dynamic Infrastructure:

In 'DynamicNS', we set globalScope to true and define a scaleNodesExpression as following:

z(temp=nodeSources.get('StaticNS')) + z(temp2=previousNodeSources.get('StaticNS')) + z(temp3=(temp2!=null?temp2.getFreeNodes()+temp2.getDeployingNodes()+temp.getFreeNodes()+temp.getDeployingNodes():-1)) + z(temp4=(temp2!=null?T(Math).min(temp2.getBusyNodes(),temp.getBusyNodes()):0)) + (temp3 == 0 && temp4 > 0 ? T(Math).ceil((neededNodes + busyNodesInNodeSource) / loadFactor) - aliveNodesInNodeSource : -aliveNodesInNodeSource )

In 'DynamicNSExtended', we set globalScope to true and define a scaleNodesExpression as following:

z(temp=nodeSources.get('DynamicNS')) + z(temp2=previousNodeSources.get('DynamicNS')) + z(temp3=(temp2!=null?T(Math).min(temp.getBusyNodes(),temp2.getBusyNodes()):0)) + (temp3 == T(Integer).parseInt(temp.getDescriptor().getPolicyParameter('maxNodes')) ? T(Math).ceil((neededNodes + busyNodesInNodeSource) / loadFactor) - aliveNodesInNodeSource : -aliveNodesInNodeSource )

'DynamicNS' will scale up when 'StaticNS' has reached its maximum capacity.

'DynamicNSExtended' will scale up when 'DynamicNS' has reached its max capacity.

7.2. Agents schedule

Node source policies limit ProActive Nodes utilization on the level of the ProActive Scheduler. If you need fine-grained limits on the node level ProActive Agents will help you achieve that.

The typical scenario is when you use desktop workstation for computations during non working hours.

Both linux and windows agents have an ability to:

Run ProActive Nodes according to the schedule
Limit resources utilization for these daemons (e.g CPU, memory)

Agents configuration is detailed in the section Deploy ProActive Nodes via Agents.

7.3. Locking ProActive Nodes

The Resource Manager allows to lock and unlock ProActive nodes. Locking a Node prevents new Tasks to be launched on that node. This operation is possible whatever the state of a Node is. Once locked, the Resource Manager keeps track of who has locked the Node and when.

A common use case for locking Nodes is about maintenances. You may have a long running Task executing on a ProActive Node where a maintenance is planned. Let’s say the current Task must not be interrupted and new ones not started before achieving the maintenance. A solution is to lock the Node. This way, the current Task will complete but no new Tasks are scheduled on that Node. Then, it is possible to perform the maintenance and on termination to unlock the Node. Upon unlocking, the Node becomes again eligible for Tasks execution.

Locking and unlocking a Node, or more generally a set of Nodes, is possible from the REST API, the command line client but also the Resource Manager portal.

Please note that locks are restored, by default, on Resource Manager restart and thus Scheduler restart. It is possible to disable this feature by editing the value associated to the property named pa.rm.nodes.lock.restoration in PROACTIVE_HOME/config/rm/settings.ini.

When the nodes lock restoration feature is enabled, the Resource Manager will try to lock, per Node Source, as many Nodes as there were on the previous run. The approach is best effort and Node hostname is not considered. As a consequence, Nodes are not necessarily locked on the same host after a Scheduler restart.

7.4. Undeploying Node Sources

The Resource Manager allows for Node Sources to be undeployed. In this case, the Node Source is shut down and its Nodes are removed, but the Node Source definition is kept. An undeployed Node Source can be redeployed later, using its initial configuration.

Node Source undeploying can be preemptive or not. If a Node Source undeploying is initiated with a non-preemptive requirement, then the Nodes of this Node Source which are currently running Tasks will not be removed until their Task is finished. In this case, these Nodes are displayed in the to-be-removed state. In other terms, if a Node Source is undeployed non-preemptively, it will be completely undeployed as soon as all of its Nodes have finished executing their Task. On the other hand, if a Node Source is undeployed with preemption, its Nodes are immediately removed regardless of whether they execute Tasks, and this may cause Tasks to fail.

Deploying and undeploying Node Sources is possible from the Resource Manager Web Interface, from the Resource Manager REST API, and from the Command Line client.

The deployment status of a Node Source is persistent. If the Resource Manager is restarted, then all Node Sources will be restored to their previous state.

8. Available Network Protocols

ProActive Workflows and Scheduling offers several protocols for the Scheduler and nodes to communicate. These protocols provide different features: speed, security, fast error detection, firewall or NAT friendliness but none of the protocols can offer all these features at the same time. Consequently, the selection should be made carefully. Below are introduced the available network protocols. Configuration and properties are discussed in Network Properties.

8.1. ProActive Network Protocol

ProActive Network Protocol (PNP) is the general purpose communication protocol (pnp:// scheme). Its performances are quite similar to the well known Java RMI protocol, but it is much more robust and network friendly. It requires only one TCP port per JVM and no shared registry. Besides, it enables fast network failure discovery and better scalability. PNP binds to a given TCP port at startup. All incoming communications use this TCP port. Deploying the Scheduler or a node with PNP requires to open one and only one incoming TCP port per machine.

8.2. ProActive Network Protocol over SSL

ProActive Network Protocol over SSL (PNPS) is the PNP protocol wrapped inside an SSL tunnel. The URI scheme used for the protocol is pnps://. It includes the same features as PNP plus ciphering and optionally authentication. Using SSL creates some CPU overhead which implies that PNPS is slower than PNP.

8.3. ProActive Message Routing

ProActive Message Routing (PAMR) allows the deployment of the Scheduler and nodes behind a firewall. Its associated URI scheme is pamr://. PAMR has the weakest expectations on how the network is configured. Unlike all the other communication protocols introduced previously, it has been designed to work when only outgoing TCP connections are available.

8.4. Installation on a Cluster with Firewall

When incoming connections are not allowed (ports closed, firewalls, etc.), the ProActive Scheduler allows you to connect nodes without significant changes in your network firewall configuration. It relies on the PAMR protocol.

This last does not expect bidirectional TCP connections. It has been designed to work when only outgoing TCP connections are available. Such environments can be encountered due to:

Network address translation devices
Firewalls allowing only outgoing connections (this is the default setup of many firewalls)
Virtual Machines with a virtualized network stack

When PAMR is activated, the ProActive Scheduler and nodes connect to a PAMR router. This connection is kept open, and used as a tunnel to receive incoming messages. If the tunnel goes down, it is automatically reopened by nodes.

The biggest drawback of PAMR is that a centralized PAMR router is in charge of routing message between all the PAMR clients. To soften this limitation PAMR can be used with other communication protocols. This way, PAMR is used only when needed.

By default, PNP is enabled. PNP is the default protocol for better performance and nodes can also use PAMR if needed. The PAMR Router is started by default along with the ProActive Scheduler.

For a ProActive Node to connect to the ProActive Scheduler using PAMR, the following ProActive configuration file can be used (PROACTIVE_HOME/config/network/node.ini). The properties tell the ProActive Node where to find the PAMR router. The ProActive Node will then connect to pamr://0 where 0 is the PAMR id of the Scheduler (0 by default).

proactive.communication.protocol=pamr
proactive.pamr.router.address=ROUTER_HOSTNAME
proactive.pamr.router.port=33647

This sample configuration requires to open only one port 33647 for incoming connections on the router host and all the ProActive Nodes will be able to connect to the Scheduler.

PAMR communication can be tunneled using SSH for better security. In that case, the ProActive node will establish a SSH tunnel between it and the ProActive Scheduler and use that tunnel for PAMR traffic.

See PAMR Protocol Properties reference for a detailed explanation of each property.

9. Task Termination Behavior

9.1. ProActive Node Graceful Task Termination (SIGTERM) at Killing

The task termination timeout is a cleanup timeout for each task (executed on this Node) after it was killed (through the Scheduler REST API or the Scheduler web portal). By default the task termination timeout is set to 10 seconds but it can be setup on each ProActive Node individually. It can be set at node startup by setting the "proactive.node.task.cleanup.time" property. Example: add "-Dproactive.node.task.cleanup.time=30" (set to 30 seconds as example) to the node startup command.

Find an overview of termination behavior by task and language in section Termination behavior by language.

9.1.1. Step by Step Killing Example

With a 30 seconds termination timeout following steps happen.

First

The kill request is received by the Scheduler Server.

Second

The ProActive Node which executes the task receives the kill request. The behavior depends if the Scheduler Server is configured to launch tasks in Forked or Non-Forked Mode.

Forked Mode:

On Linux, a SIGTERM signal is send to the JVM which executes the task. On Windows, the TerminateProcess method of the JVM process which executes the task is executed. That will finally lead to the Task Process being interrupted by:

A SIGTERM signal on Linux
TerminateProcess method called, on Windows

The task can deal with the killing event and start a cleanup procedure. If the task has started sub-processes it is responsible to initiate graceful termination of the sub-processes. Handling a graceful termination event looks different in each task type and is dependent on the capabilities of the language used. As an example Bash and Jython (Python task) can catch a SIGTERM signal. Java and Groovy task need to register shutdownHooks to achieve similar behavior.

Non-Forked Mode:

In Non-Forked Mode, the task runs in a separate thread in the same JVM as the ProActive Node. Therefore, the thread running the task will be interrupted.

Bash tasks, in Non-Forked Mode, do receive SIGTERM. Java and Groovy tasks will receive an InterruptedException, which can be used to handle the graceful termination of that task and sub-processes. But, Jython (Python tasks) will not receive any notification. If the task has started sub-processes it is responsible to initiate graceful termination of the sub-processes, if possible.

Third

The ProActive Node waits as many seconds as setup in the proactive.node.task.cleanup.time property or until the task is stopped. Example: cleanup timeout is 30 seconds and the task takes 3 seconds to terminate, then the ProActive Node will wait 3 seconds only.

Fourth

The ProActive Node initiates the task process and sub-process removal. This will forcefully kill the whole tree of processes. Another SIGTERM might be send shortly before the forcefully killing (SIGKILL).

Additional information

On Windows the termination procedure is similar. But the Windows SIGTERM equivalent is executing the TerminateProcess method of the process. Whereas the SIGKILL is equivalent to the forceful removal of the running process.

Termination behavior by language

Table 2. Termination Behavior for a Few Languages
Language/Execution type	Bash	Python	Java	Groovy
Forked Mode	Handle SIGTERM. Task waits cleanup timeout, before being killed.	Handle SIGTERM. Task waits cleanup timeout, before being killed.	Add a shutdown hook. Task waits cleanup timeout, before being killed.	add a shutdown hook. Task waits cleanup timeout, before being killed.
Forked Mode Run As Me	Handle SIGTERM. Task waits cleanup timeout, before being killed.	Handle SIGTERM. Task waits cleanup timeout, before being killed.	Add a shutdown hook. Task waits cleanup timeout, before being killed.	Add a shutdown hook. Task waits cleanup timeout, before being killed.
Non-Forked Mode	Handle SIGTERM. Task waits cleanup timeout, before being killed.	Terminates immediately.	Catch an InterruptedException. Task waits cleanup timeout, before being killed.	Catch an InterruptedException. Task waits cleanup timeout, before being killed.

10. Monitor the cluster state

Cluster monitoring typically means checking that all ProActive Nodes that were added to ProActive Scheduler are up and running. We don’t track for example the free disk space or software upgrade which can be better achieved with tools like Nagios.

In the Resource Manager Web Interface you can see how many ProActive Nodes were added to the Resource Manager and their usage.

The same information is accessible using the command line:

$ PROACTIVE_HOME/bin/proactive-client --listnodes

10.1. ProActive Node States

When you look at your cluster, ProActive Nodes can be in one of the following states:

Deploying - The deployment of the node has been triggered by the ProActive Resource Manager but it has not yet been added.
Lost - The deployment of the node has failed for some reason. The node has never been added to the ProActive Resource Manager and won’t be usable.
Configuring - Node has been added to the ProActive Resource Manager and is being configured.
Free - Node is available for computations.
Busy - Node has been given to user to execute computations.
To be removed - Node is busy but requested to be removed. So it will be removed once the client will release it.
Down - Node is unreachable or down and cannot be used anymore.
Needed - Metric shows how many additional nodes the Scheduler needs to schedule all pending tasks.

The state of a ProActive Node is managed by the Resource Manager. However, each Node has a user managed lock status.

10.2. JMX

The JMX interface for remote management and monitoring provides information about the running ProActive Resource Manager and allows the user to modify its configuration. For more details about JMX concepts, please refer to official documentation about the JMX architecture.

The following aspects (or services) of the ProActive Scheduler are instrumented using MBeans that are managed through a JMX agent.

Server status is exposed using the RuntimeDataMBean
- The Resource Manager status
- Available/Free/Busy/Down nodes count
- Average activity/inactivity percentage
The Accounts Manager exposes accounting information using the MyAccountMBean and AllAccountsMBean
- The used node time
- The provided node time
- The provided node count
Various management operations are exposed using the ManagementMBean
- Setting the accounts refresh rate
- Refresh all accounts
- Reload the permission policy file

MBean server can be accessed by remote applications using one of the two available connectors

The standard solution based on Remote Method Invocation (RMI) protocol is the RMI Connector accessible at the following URL: service:jmx:rmi:///jndi/rmi://HOSTNAME:PORT/JMXRMAgent where
- HOSTNAME is the hostname on which the Resource Manager is started
- PORT (5822 by default) is the port number on which the JMX RMI connector server has been started. It is defined by the property pa.rm.jmx.port .
The ProActive Remote Objects Connector provides ProActive protocol aware connector accessible at the following URL: service:jmx:ro:///jndi/PA_PROTOCOL://HOSTNAME:PORT/JMXRMAgent where
- PA_PROTOCOL is the protocol defined by the proactive.communication.protocol property
- HOSTNAME is the hostname on which the Resource Manager is started
- PORT is the protocol dependent port number usually defined by the property proactive.PA_PROTOCOL.port

The name of the connector (JMXRMAgent by default) is defined by the property rm.jmx.connectorname.

The JMX URL to connect to can be obtained from the Authentication API of the Resource Manager or by reading the log file located in PROACTIVE_HOME/logs/RM.log. In that log file, the address you have to retrieve is the one where the JMX RMI connector server has been started

[INFO 2010-06-17 10:23:27,813] [RM.AbstractJMXHelper.boot] Started JMX RMI connector server at service:jmx:rmi:///jndi/rmi://kisscool.inria.fr:5822/JMXRMAgent

Once connected, you’ll get an access to Resource Manager statistics and accounting.

For example, to connect to the ProActive Scheduler JMX Agent with JConsole tool, just enter the URL of the standard RMI Connector, as well as the username and the password.

Then depending on the allowed permissions browse the attributes of the MBeans.

10.3. Accounting

The users of ProActive Scheduler request and offer nodes for computation. To keep track of how much node time was consumed or contributed by a particular user, ProActive Scheduler associates a user to an account.

More precisely, the nodes can be manipulated by the following basic operations available to the users

The ADD operation is a registration of a node in the Resource Manager initiated by a user considered as a node provider. A node can be added, through the API, as a result of a deployment process, through an agent or manually from the command line interface.
The REMOVE operation is the unregistration of a node from the Resource Manager. A node can be removed, through the API, by a user or automatically if it is unreachable by the Resource Manager.
The GET operation is a node reservation, for an unknown amount of time, by a user considered as a node owner. For example, the ProActive Scheduler can be considered as a user that reserves a node for a task computation.
The RELEASE operation on a reserved node by any user.

The following accounting data is gathered by the Resource Manager

The used node time: The amount of time other users have spent using the resources of a particular user. More precisely, for a specific node owner, it is the sum of all time intervals from GET to RELEASE.
The provided node time: The amount of time a user has offered resources to the Resource Manager. More precisely, for a specific node provider, it is the sum of all time intervals from ADD to REMOVE.
The provided node count: The number of provided nodes.

The accounting information can be accessed only through a JMX client or the ProActive Resource Manager command line.

10.4. Metric to monitor Scheduler load

Needed Nodes metric shows how many additional nodes the Scheduler needs to schedule all pending tasks, e.g. there are 4 nodes and all of them are busy, but Needed Nodes shows 10, which means if you would have 14 nodes instead of 4, all of these nodes would be busy. Please notice, that Needed Nodes shows how many more nodes could be used right now. However, it does not give any indication about the kind of nodes needed, e.g. Selection Script of the task could require some specific nodes, thus Needed Nodes could be positive despite still having free nodes (they can be inappropriate for the current task).

You can find this metric in multiple places. In RM portal you can see it in Nodes History:

In Nodes State:

And you can find it in RM portal detailed statistics:

Also, you can find Needed Nodes in the Scheduler portal where the Scheduler status is displayed:

11. Run Computation with a user’s system account

ProActive workflow Tasks can be configured to run under a user’s system account by ticking the Run as me checkbox in the task configuration.

The RunAsMe mode can also be configured globally on the ProActive server. In that case, all workflow tasks from all users will be executed in RunAsMe mode. Global RunAsMe can be configured in file PROACTIVE_HOME/config/scheduler/settings.ini, by enabling the following property:

# If true tasks are always ran in RunAsMe mode (impersonation). This automatically implies pa.scheduler.task.fork=true (other setting is ignored)
pa.scheduler.task.runasme=true

By default, authentication is done through a password, but can also be configured to use a SSH key or password-less sudo.

The impersonation mode is configurable for each ProActive Node, using the java property pas.launcher.forkas.method.

For example:

./bin/proactive-node -Dpas.launcher.forkas.method=pwd # start a node using password impersonation
./bin/proactive-node -Dpas.launcher.forkas.method=key # start a node using SSH key impersonation
./bin/proactive-node -Dpas.launcher.forkas.method=none # start a node using password-less sudo impersonation

This property can also be specified in ResourceManager Node Source Infrastructures through parameters that allow adding extra arguments to the java virtual machine.

Find a step by step tutorial here.

11.1. System Configuration

For proper execution, the user’s system account must have:

Execution rights to the PROACTIVE_HOME directory and all it’s parent directories.
Write access to the PROACTIVE_HOME directory.

Logs are written during task execution, as a matter of fact, the executing user needs write access to the PROACTIVE_HOME directory. ProActive does some tests before executing as a different user, those need full execution rights, including the PROACTIVE_HOME parent directories.

Create a ProActive group which owns the PROACTIVE_HOME directory and has write access. Add every system user, which executes tasks under its own system account, to the ProActive group.

On Mac OS X, the default temporary folder is not shared between all users. It is required by the RunAsMe feature. To use a shared temporary folder, you need to set the $TMPDIR environment variable and the Java property java.io.tmpdir to /tmp before starting the ProActive Node.

the same applies on Windows. To use a shared temporary folder, create a folder shared by all ProActive users e.g. C:\TEMP and define the Java property java.io.tmpdir to C:\TEMP before starting the ProActive Node.

Example:

proactive-node -Djava.io.tmpdir=C:\TEMP

Extra configurations are also needed depending on the authentication method below.

11.2. Using password

The ProActive Node will try to impersonate the user that submitted the task when running it. It means the username and password must be the same between the ProActive Scheduler and the operating system.

11.3. Using SSH keys

To enable this authentication method, set the java system property -Dpas.launcher.forkas.method=key when starting a ProActive Node.

Example:

proactive-node -Dpas.launcher.forkas.method=key

A SSH key can be tied to the user’s account and used to impersonate the user when running a task on a given machine (using SSH). Additionally:

The .ssh/authorized_keys files of all machines must be configured to accept this SSH key.
The SSH key must not contain a passphrase.
Using this method on Windows machines is not recommended, as it requires the installation and configuration of a SSH Server.

When login into the scheduler portal, the private key of the user must be provided, this can be done by selecting on the login dialog: More options > Use SSH private key.

Alternatively, a user can also add its SSH private key to Third Party Credentials under the name SSH_PRIVATE_KEY. This method allows the user to login normally to the ProActive portals, without the need to enter its private key each time.

Finally, users can also authenticate to the portals with credential files. The user must first create a credential file containing the user login, password and SSH key.

Run the following command on the ProActive server to create a credential file:

$ <PROACTIVE_HOME>/tools/proactive-create-cred -F <PROACTIVE_HOME>/config/authentication/keys/pub.key -l username -p userpwd -k path/to/private/sshkey -o myCredentials.cred

This command will create an encrypted file with username as login, userpwd as password, using Scheduler public key at config/authentication/keys/pub.key for credentials encryption and using the user private SSH key at path/to/private/sshkey. The new credential will be stored in myCredentials.cred.

Once created, the user must connect to the ProActive portals using this credential file. For example, in the studio portal below:

11.4. Using password-less sudo

This configuration, only available on linux or unix Nodes, allows the impersonation to be performed using password-less sudo.

To enable it at the system level, edit the /etc/sudoers file to allow password-less sudo from the account running the ProActive node to any users which require impersonation. Password-less sudo should be enabled for any command.

For example, the following line will allow the proactive account to impersonate to any user:

proactive ALL=(ALL) NOPASSWD: ALL

To enable this configuration on the ProActive node, start it with the system property -Dpas.launcher.forkas.method=none

Example:

proactive-node -Dpas.launcher.forkas.method=none

12. Configure Web applications

The ProActive Scheduler deploys automatically several web applications in an embedded Jetty web server. The binaries and sources associated to the web applications can be found in PROACTIVE_HOME/dist/war.

To configure Web applications, for instance the HTTP port to use, edit the PROACTIVE_HOME/config/web/settings.ini file.

Web applications are deployed on the host using the HTTP port 8080 by default. The local part of the URL is based on the WAR files found in PROACTIVE_HOME/dist/war. For instance, the Scheduler web portal is available at http://localhost:8080/scheduler.

12.1. Enable HTTPS

ProActive Workflows and Scheduling provides support for HTTPS. Similarly to other web settings, HTTPS related properties are defined in PROACTIVE_HOME/config/web/settings.ini and any update would require a restart of the Scheduler instance.

Enabling HTTPS requires to set web.https property to true but also to define a path to a valid keystore through web.https.keystore along with its associated password by using web.https.keystore.password. A default keystore is provided for testing purposes, however it must not be used in production.

When you are using the default keystore or a custom keystore with a self-signed certificate you need to enable web.https.allow_any_certificate and optionally web.https.allow_any_hostname if your certificate Common Name (CN) does not match the fully qualified host name used for the machine hosting the Web applications.

Port 8443 is used for listening to HTTPS connections. This port number prevents to have root access for deploying web applications. However, you can change it by editing web.https.port value.

When HTTPS is set up, you can automatically redirect any HTTP request from users to the secured version of the protocol to make sure exchanged information is protected. It requires to enable web.redirect_http_to_https.

Additionally, you will need to update the port number used in the property jp.url.

Likewise, it is necessary to manually update the port number and enable HTTPS in the configuration and properties files of the web applications, as follows (example using default port 8443):

PROACTIVE_HOME/dist/war/job-planner/WEB-INF/classes/application.properties

pa.scheduler.rest.url=https://localhost:8443/rest
pa.catalog.url=https://localhost:8443/catalog
pa.pca.service.base.url=https://localhost:8443/cloud-automation-service
pa.scheduling.api.url=https://localhost:8443/scheduling-api/v1/graphql

PROACTIVE_HOME/dist/war/catalog/WEB-INF/classes/application.properties

pa.scheduler.url=https://localhost:8443

PROACTIVE_HOME/dist/war/cloud-automation-service/WEB-INF/classes/application.properties

pa.catalog.url=https://localhost:8443/catalog
pa.scheduler.rest.url=https://localhost:8443/rest/scheduler
pa.resource_manager.rest.url=https://localhost:8443/rest/rm
pa.cloud_automation_service.url=https://localhost:8443/cloud-automation-service
server.port=8443
pa.web.https.allow_any_certificate=true
pa.web.https.allow_any_hostname=true

PROACTIVE_HOME/dist/war/notification-service/WEB-INF/classes/application.properties

scheduling.url=https://localhost:8443

PROACTIVE_HOME/dist/war/proactive-cloud-watch/WEB-INF/classes/application.properties

pca.rm.url=https://localhost:8443/rest/rm
pca.service.url=https://localhost:8443/cloud-automation-service
pa.catalog.url=https://localhost:8443/catalog
pa.scheduler.rest.url=https://localhost:8443/rest

PROACTIVE_HOME/dist/war/scheduling-api/WEB-INF/classes/application.properties

pa.scheduler.rest.url=https://localhost:8443/rest

PROACTIVE_HOME/dist/war/rm/rm.conf

rm.rest.url=https://localhost:8443/rest
web.https.allow_any_hostname=true
web.https.allow_any_certificate=true

PROACTIVE_HOME/dist/war/scheduler/scheduler.conf

jobplanner.rest.url=https://localhost:8443/job-planner/planned_jobs
pa.scheduling.api=https://localhost:8443/scheduling-api
sched.catalog.url=https://localhost:8443/catalog
sched.rest.url=https://localhost:8443/rest
web.https.allow_any_hostname=true
web.https.allow_any_certificate=true

If you are familiar with Nginx or a similar web server, you can also use it to enable HTTPS connection to Web applications. It has the advantage to make it possible to have ProActive web applications and additional ones to coexist.

12.1.1. Creating a valid keystore

Java is bundled with some utility binaries such as keytool. This last allows to create and manage a keystore but also to generate keys and certificates.

With a self-signed certificate

Generating a new self-signed certificate imported in a new keystore with a custom password is as simple as executing the following command:

$JAVA_HOME/bin/keytool -keystore keystore -alias jetty -genkey -keyalg RSA -validity 365

This command prompts for information about the certificate and for a password to protect both the keystore and the keys within it. The only mandatory responses are to provide a password and the Fully Qualified Domain Name of the server. Once done, the value for properties web.https.keystore and web.https.keystore.password must be adapted based on the information you have entered.

For more information, please look at the Jetty documentation.

With a trusted certificate

The first step is to obtain a trusted certificate that is valid for your Fully Qualified Domain Name (FQDN). This process differs from a Certification Authority (CA) to another. In the following, it is assumed that Let’s Encrypt is used. By way of illustration, Digital Ocean provides examples for CentOS and Ubuntu.

Assumming your FQDN is example.org, then after obtaining a certificate you will get the following PEM-encoded files in /etc/letsencrypt/live/example.org:

cert.pem: Your domain’s certificate
chain.pem: The Let’s Encrypt chain certificate
fullchain.pem: cert.pem and chain.pem combined
privkey.pem: Your certificate’s private key

Before loading the key and the certificate into a new keystore that is recognized by ProActive Workflows and Scheduling, you need to combine them into a PKCS12 format file. You can achieve this action by means of the following OpenSSL command:

openssl pkcs12 \
    -inkey /etc/letsencrypt/live/example.org/privkey.pem \
    -in /etc/letsencrypt/live/example.org/cert.pem \
    -export -out jetty.pkcs12

Then, you can load the resulting PKCS12 file into a JSSE keystore with keytool:

keytool -importkeystore -srckeystore jetty.pkcs12 \
    -srcstoretype PKCS12 -destkeystore keystore

Both commands prompt for a password. You need to use the same and set it as a value to property web.https.keystore.password. The resulting keystore file corresponds to the file whose path must be used for property web.https.keystore.

Starting from update 101, Java 8 trusts Lets Encrypt certificates. However, if you are using a less recent version of Java for running JVMs related to ProActive Workflows and Scheduling, you will need to update the truststore of your Java installation.

12.1.2. Updating Java truststore for accepting Lets Encrypt certificates

sudo keytool -trustcacerts \
    -keystore $JAVA_HOME/jre/lib/security/cacerts \
    -storepass changeit \
    -noprompt \
    -importcert \
    -file /etc/letsencrypt/live/example.org/chain.pem

12.2. Enable VNC remote visualization in the browser

The REST API web application embeds a proxy to allow the remote display visualization of a ProActive Node running a given task via a VNC server from the browser. To enable this feature, the scheduler has to be configured as follows:

configure the proxy: edit the PROACTIVE_HOME/config/web/settings.ini file and set novnc.enabled to true in order to start the proxy.
configure the scheduler portal to use the proxy when opening a new tab browser that shows the remote visualization: edit the PROACTIVE_HOME/dist/war/scheduler/scheduler.conf.
- Set sched.novnc.url to the public address of the proxy. This public address is the public address of the host where the sheduler is started and the port specified by the option novnc.port in the file PROACTIVE_HOME/config/web/settings.ini
- Set sched.novnc.page.url to the public address of the VNC client to be executed in the browser. The client is located in the REST API with the file novnc.html.

For more information, see https://github.com/ow2-proactive/scheduling/blob/master/rest/README-novnc.txt

12.3. Catalog

The ProActive Catalog is a component responsible for storing various objects of ProActive Workflows and Scheduling and in particular ProActive Workflows.

The ProActive Catalog features:

a REST API.
a comprehensive Catalog Portal.

ProActive Catalog is organized into buckets. Each bucket has a unique name and stores zero, one or more versioned ProActive Objects.

By default, ProActive objects are persisted on disk using the embedded HSQL database. The data is located in PROACTIVE_HOME/data/db/catalog.

The Catalog web service binaries are stored in PROACTIVE_HOME/dist/war/catalog. This directory contains a configuration file located in PROACTIVE_HOME/dist/war/catalog/WEB-INF/classes/application.properties.

More information regarding the Catalog configuration can be found in the catalog properties section.

A complete documentation of the Catalog REST API is available on the following link.

This documentation is automatically generated using Swagger.

12.3.1. Catalog security and groups management

The Catalog authenticates REST request using a sessionID header. This sessionID is used to share session information between various ProActive Workflows & Scheduling microservices and is returned when authenticating to a ProActive Scheduler REST API..

Every request to the Catalog must have a valid sessionID inside the request header to authenticate the user. A role-based access control mechanism can also be defined on catalog buckets. When creating a bucket, it can either be made public or restrained to a specific group.

Creating users and groups at the ProActive Scheduler level is explained in the User Authentication section.

A public bucket is automatically assigned to the public-objects group.

For example, a user which is part of the interns group (GROUP:interns) can see and change buckets and their contents which belong to GROUP:interns. This user can only access to the following buckets list:

All buckets that the user created.
All buckets belonging to the interns group (GROUP:interns).
All public buckets (GROUP:public-objects).

12.4. Notification service

12.4.1. Housekeeping

Notifications in the notification service have a global life span defined in the application.properties located in * PROACTIVE_HOME/dist/war/notification-service/WEB-INF/classes/application.properties Past that time, they will be removed from the database.

Two parameters with default values may be updated:

1- The call frequency of the housekeeping defined by a CRON expression:

# This cron expression determines the housekeeping call frequency.
# Default value is 10 minutes: this will invoke the housekeeping mechanism
# to remove every notifications whose scheduled removal time is up.
notifications.housekeeping.cronexpression=*/10 * * * *

2- The life span of the notifications in seconds:

# Automatic remove notification delay (in seconds).
# The time at which notifications will be stored in the database
# Set this time to 0 if you don't want the notifications to be removed.
# Default value is 604800 seconds = 1 week
notifications.housekeeping.removedelay=604800

12.4.2. User parameter confirmation

Confirmation requests can be enabled for notification methods other than Portal in order to ask the user to confirm the parameter he provided (Email address, phone number, etc).

For Email notification method, the user won’t be able to activate it in his subscriptions before the email is confirmed. However, Third-party notification method parameter is not mandatory and users will still be able to activate the notification method before confirming it but the field that holds the parameter inside the script will be empty.

To activate confirmation requests, go to the same application.properties file located here and set to true the properties:

For Email confirmation

# Activate or deactivate email confirmation request to be used as recipient for notifications
mail.confirmation.enabled=

For Third-party confirmation

# Activate or deactivate third-party confirmation request to be used as recipient for notifications
third-party.confirmation.enabled=

Confirmation requests have a life span of 15 minutes. Once that delay is passed, expired pending confirmations are automatically removed by the housekeeping. Users will have to resend a new confirmation request in order to confirm the parameter.

12.4.3. Email notification method configuration

The Email notification method uses the Simple Mail Transfer Protocol to send emails. The notification service requires a valid SMTP configuration in order to send emails. These settings are specified in the properties file located at: PROACTIVE_HOME/dist/war/notification-service/WEB-INF/classes/email.properties

All available SMTP properties are available at https://javaee.github.io/javamail/docs/api/com/sun/mail/smtp/package-summary.html

By default Transport Layer Security is deactivated, change to true to activate it:

mail.smtp.starttls.enable=false

Authentication to the SMTP server is enabled by default.

mail.smtp.auth=true

Provide the server port used to connect

mail.smtp.port=

The credentials used to connect

mail.smtp.username=
mail.smtp.password=

The host of the SMTP server

mail.smtp.host=

A single email address is used by the notification service to broadcast emails. It is defined in the application.properties file located at PROACTIVE_HOME/dist/war/notification-service/WEB-INF/classes/email.properties and is the email that will be displayed in the "From" field of the sent emails.

notification.email.from=

12.4.4. Third party notification method configuration

The Third Party Notification method executes a script when an event occurs. There can be a single script execution, regardless of how many users are notified, or mutliple executions.

Script location

There are two ways to specify the location of the script.

Using path: relative to the Scheduler or absolute

notification.script.path=dist/war/notification-service/WEB-INF/classes/exampleScript.sh

Or a URL

notification.script.url=

Note that to the notification.script.url= property will be used by the Notification Service only if the notification.script.path= property is not provided. If provided the notification.script.url= property is ignored. It means that the notification.script.path= property will always have the upper hand to indicated the location of the script to execute.

Script engine

An optional parameter can be provided to specify the engine to be used to execute the script if not provided the extension of the file will determine the appropriate engine to use

#optional parameter for specifying Script Engine
notification.script.engine=bash

Execution modes

Single execution mode

By default Third party notification method is configured in single execution mode. In single execution mode, only one script execution will occur for a given notification regardless of how many users are notified.

For example if 5 users are subscribed to the Scheduler Frozen event with the Third party notification method active then only one script execution will take place.

#if false execute the script one time per event even if there are several subscribers
#if true execute the script for each subscribers for each event
notification.script.multiple.execution=false

Multiple executions mode

To enable the multiple execution mode and access every notified users parameter inside the script, simply change to property to true. This execution mode will trigger one execution for each notified user and, for each execution, inject the user parameter in the script as an additional Script binding.

For example, if 5 users are subscribed to the Scheduler Frozen event with the Third party notification method active and multiple execution enabled, then each time the ProActive Scheduler freezes the script will be executed one time per notified user so, in this example 5 times.

Script bindings

Notification bindings

Some relevant notification data is automatically passed to the script as bindings when executed. It is simply needed to read the field as you would do with any other field in the used language.

In some cases, bindings might not have a value as it is not relevant. For instance BUCKET_NAME, PROJECT_NAME, WORKFLOW_NAME are empty for Scheduler events as they don’t depend on job execution.

Example in groovy:

println "Reading the notification message: " + MESSAGE

Those bindings are passed to the script regardless of the selected execution mode

Third-party user parameter binding

When multiple execution mode is activated, the user which is notified can add an additional argument to the script. This letter parameter will be used inside the script as a new binding named args[]. To access the user parameter in the script, like for notification bindings, simply read the array.

Example in groovy:

println "User parameter equals: " + args[0];

The image below shows where a user may provide a parameter in the notification-portal.

Figure 15. Third-party user parameter input

Table of bindings

In the following table are specified all available fields that can be read in the script and if they have a value depending on the event origin.

Table 3. Binding holds a value regarding event type (Y = Yes , N = No)
Field name	Job	Task	Scheduler	Job planner
USER_NAME	Y	Y	Y	Y
EVENT_TYPE	Y	Y	Y	Y
JOB_ID	Y	Y	N	Y
TASK_ID	N	Y	N	N
TASK_NAME	N	Y	N	N
CREATED_BY	Y	Y	Y	Y
MESSAGE	Y	Y	Y	Y
CREATION_DATE	Y	Y	Y	Y
JOB_OWNER	Y	Y	N	Y
BUCKET_NAME	Y	Y	N	Y
PROJECT_NAME	Y	Y	N	Y
WORKFLOW_NAME	Y	Y	N	Y
SEVERITY	Y	Y	Y	Y
JOB_LINK	Y	Y	N	N
args[] - see multiple executions mode	Y	Y	Y	Y

13. Configure script engines

Most script engines do not need any configuration. This section talks about the script engines which can be configured individually.

13.1. Docker Compose (Docker task)

The Docker Compose script engine is a wrapper around the Docker Compose command. It needs to have Docker Compose and Docker installed, in order to work. The Docker Compose script engine has a configuration file which configures each ProActive Node individually. The configuration file is in PROACTIVE_HOME/config/scriptengines/docker-compose.properties.
docker-compose.properties has the following configuration options:

docker.compose.command=[docker-compose executable example:/usr/local/bin/docker-compose] --- Defines which Docker Compose executable is used by the script engine.

docker.compose.sudo.command=[sudo executable example:/usr/bin/sudo] --- Defines the sudo executable which is used. The sudo property is used to give the Docker Compose command root rights.

docker.compose.use.sudo=false --- Defines whether to execute Docker Compose with sudo [true] or without sudo [false]

docker.host= --- Defines the DOCKER_HOST environment variable. The DOCKER_HOST variable defines the Docker socket which the Docker client connects to. That property is useful when accessing a Docker daemon which runs inside a container or on a remote machine. Further information about the DOCKER_HOST property can be found in the official Docker documentation.

13.2. Perl scripts execution (Perl task)

Perl should be installed on your machine. By default the perl engine is installed in Linux and Mac Os. Please install the perl engine on Windows.
The execution mechanism of perl tasks is based on running perl files on your machine.

If you encountered the issues, please check if you have installed Perl properly on you machine. In order to verify this you can follow the next steps:

For example you should to be able to launch the command 'perl -V'.
Then from command line you should be able to launch a simple perl file (perl yourPerlFile.pl).

13.3. Python Script Engine (Python task)

Python should be installed on your machine. By default Python2 is installed on Linux and Mac OS. If another version of Python is required, it should be installed on your machine in advance.

More over, the native Python Script Engine depends on the py4j library in order to communicate with our Java classes. Please follow the following step to install it:

With Python2: pip install py4j
With Python3: pip3 install py4j

Further information can be found in the official py4j documentation.

14. Extending ProActive Scheduling Policy

In order to decide which pending Tasks will be scheduled for execution, the ProActive Scheduler uses a Scheduling Policy.

This policy can:

Sort pending tasks according to a specific order.
Exclude pending tasks until some conditions are met.

The default scheduling policy, called ExtendedSchedulerPolicy:

Excludes pending tasks when the START_AT generic information is set, and the configured time is still in the future.
Sort Pending tasks according to their corresponding Job Priority, and then by _First-In-First-Out order.

The Scheduling Policy is configured inside PROACTIVE_HOME/config/scheduler/settings.ini:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.ExtendedSchedulerPolicy

The default policy can be extended to perfectly fit an organization scheduling constraints.

In order to change the policy used by the ProActive Scheduler, simply replace the existing setting with the fully qualified class name of the alternate policy.

In the following sections, we show several alternate policies that can be used. Each of these policies inherits from the ExtendedSchedulerPolicy, which means that they apply the default policy yet may override the default behavior in some cases.

14.1. Earliest deadline first (EDF) policy

Early Deadline First Policy sorts jobs based on (by decreasing importance):

job priority (from highest to lowest).
presence of a deadline (those that have a deadline overtake those without a deadline).
job started or not (jobs already started overtake those not already started), and jobs started first overtake those started last.
For pending jobs, sort them according to the configured deadline (earliest deadlines overtake others).

To use the earliest deadline first (EDF) policy, edit the file PROACTIVE_HOME/config/scheduler/settings.ini and change the following line:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.ExtendedSchedulerPolicy

to:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.edf.EDFPolicyExtended

Each workflow can configure its deadline by using the JOB_DDL and JOB_EXEC_TIME generic information.

14.2. License Policy

The License Policy allows limiting concurrent access to external resources such as software licenses (but can also be used in other scenarios, e.g. external database connections).

To use the LicenseSchedulingPolicy, edit the file PROACTIVE_HOME/config/scheduler/settings.ini and change the following line:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.ExtendedSchedulerPolicy

to:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.license.LicenseSchedulingPolicy

The LicenseSchedulingPolicy requires an extra configuration: the maximum number of licenses per software/resource must be specified into PROACTIVE_HOME/config/scheduler/license.properties like this

software_A = 3
software_B = 2

Users declare their usage of software licenses by adding the REQUIRED_LICENSES generic information at job level or task level.

REQUIRED_LICENSES expects a comma-separated list of software names defined in the above configuration file, e.g.:

softwareA,softwareB

The behavior of REQUIRED_LICENSES is different when declared at job-level or task-level.

At task-level, if 10 tasks requiring software_A,software_B licenses are pending, they will be executed 2 by 2, according to the software with the fewest licenses.

At job-level, the License Policy considers that the whole job consumes one software license (no matter how many tasks this job contains). In the above example, it means that when 10 such jobs are pending, only 2 jobs will be allowed to start, the other jobs will remain pending until the first 2 jobs complete.

It is also possible to combine - in the same workflow - REQUIRED_LICENSES at both levels.

14.3. Node Usage Policy

The Node Usage Policy allows limiting parallel usage of Nodes within workflows. This can be used for workflows that contain a large number of tasks when one wants to limit the number of Nodes that will be allocated to each workflow. This way, several large workflows can run concurrently instead of one workflow consuming all resources.

To use the NodeUsageSchedulingPolicy, edit the file PROACTIVE_HOME/config/scheduler/settings.ini and change the following line:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.ExtendedSchedulerPolicy

to:

pa.scheduler.policy=org.ow2.proactive.scheduler.policy.limit.NodeUsageSchedulingPolicy

In order to configure the number of Nodes a workflow is allowed to use in parallel, set the MAX_NODES_USAGE generic information at job-level.

15. Addons

15.1. Get Notification on Job Events Configuration

In order to enable this functionality the following properties should be set in config/scheduler/setting.ini:

pa.scheduler.notifications.email.enabled = true
pa.scheduler.notifications.email.from = username@example.com

To configure the email sender on Scheduler, the configuration file config/scheduler/emailnotification.properties should be filled in, please refer to SMTP configuration examples for examples.

15.2. Email notification

The email notification addons includes an utility Java class but also a Groovy task (accessible from the Studio Web app) that allow to send an email from a ProActive Workflow.

The task assumes that configuration for connecting to an SMTP server is done using third-party credentials. It requires to define some values as key/value pairs in the third-party credentials associated to the user that runs the task.

This configuration can be achieved from the Scheduler Web portal, the ProActive client or even through the REST API. Please see Third-party credentials for more information.

15.2.1. SMTP configuration examples

Free

Key

Value

mail.smtp.host

smtp.free.fr

mail.smtp.username

user@free.fr

mail.smtp.password

user_password

Gmail

Password authentication to Google servers requires extra configuration.

See https://www.google.com/settings/security/lesssecureapps.

Key

Value

mail.smtp.host

smtp.gmail.com

mail.smtp.starttls.enable

true

mail.smtp.ssl.trust

smtp.gmail.com

mail.smtp.username

user@gmail.com

mail.smtp.password

user_password

Outlook

Password authentication to Microsoft servers requires extra configuration.

Key

Value

mail.smtp.host

smtp-mail.outlook.com

mail.smtp.starttls.enable

true

mail.smtp.ssl.trust

smtp-mail.outlook.com

mail.smtp.username

user@gmail.com

mail.smtp.password

user_password

If the above settings do not succeed, check in the scheduler logs any errors related to email sending.

15.3. Statistics on ProActive Jobs and Resource Usage

This addon generates a gantt chart to visualize executions of jobs and tasks, per node and over the time.

15.3.1. Installation

To enable this addon, a Node connected to the Resource Manager must have access to the PA_PLOT application configured in it.

Requirements

Linux, Windows, or MacOSX.
Python 2.7 or 3.5.
Bokeh, dateutil, requests, pytest, sphinx.

Requirements setup with Anaconda

Anaconda is the fastest and easiest way to install all requirements. You do not need internet access on the installation host.

Download and install Anaconda
Linux/MacOSX::
```
source <anaconda root>/bin/activate
```
Windows:
```
Run "Anaconda Prompt"
```

Requirements setup with Miniconda

Miniconda is a stripped down version of Anaconda, much smaller in size. The downside is that it normally requires internet access on the machine where it is installed.

Download and install Miniconda
If behind a proxy, compile condarc

Linux/MacOSX::

source <anaconda root>/bin/activate

Windows:

Run "Anaconda Prompt"

Execute::

conda install -y python-dateutil bokeh requests

Addional optional dependencies to run the unit tests:
Python 2.7
conda install -y pytest mock
Python 3.5
conda install -y pytest
Additional optional dependencies to regenerate the documentation::
```
conda install -y sphinx
```

pa_plot installation

Source Anaconda environment (see previous steps)
Unpack :file:`pa_plot.tar.gz` and move to the :file:`pa_plot` directory
Execute::
```
python setup.py install
```

15.4. ServiceNow integration

Product Name: ServiceNow
Product Version: Quebec
Latest Release compatibility: Tokyo

15.4.1. Overview

The ServiceNow integration provides means to establish a bidirectional communication between a ProActive server, and a ServiceNow instance. Here we describe how to install the integration in a ServiceNow instance and how to communicate with a ProActive server.

To learn how to communicate from ServiceNow to ProActive using ServiceNow’s REST API, please refer to the ServiceNow user documentation.

15.4.2. Objectives

The goal is to provide templates and coding elements to help ServiceNow and ProActive users to easily integrate both solutions in their pipelines.

It boils down to:

Provide reusable code in ServiceNow to query ProActive’s REST APIs.
Provide workflows in ProActive’s catalog to query ServiceNow’s REST APIs.

15.4.3. Installation

This section describes the available processes to set up the required files in ServiceNow to communicate with ProActive. The preferred installation process is dictated by the architecture of your application ecosystem in ServiceNow. Either ProActive’s integration will have a dedicated ServiceNow application, or it will be combined with an existing one. ServiceNow being an application-based solution, the natural approach is to create a dedicated ProActive application. If you want to incorporate this integration in another application that already exists, you will have to manually create files in the host application, which means copying code blocks and ServiceNow files that will be presented in the next sections.

If the latter process is chosen, you can always install the ProActive application in a ServiceNow test instance to look at the application’s files and use them as templates as well as copy their code blocks.

Import from source control

Working from an online repository is the most convenient and durable way to proceed. It greatly eases the backup, update and distribution of the application’s versions across ServiceNow instances. This method requires you to have git installed on your local system.

Here are the steps to get started from source control and lay down the application:

Define the online repository
1. In your preferred git hosting service, create a new private repository
2. Make sure that all users acting on the repository have the write and read permissions
Add the application’s files
1. Download the Application’s files as a zip archive
2. git clone the newly created repository on your local system
3. Add the applications file to the project and add them in git with git add
4. git commit the changes
5. git push origin master to push the changes to the online repository. (You will be able to create and navigate through branches directly from ServiceNow)
Import the application in ServiceNow

Connect to ServiceNow as an administrator and:

Create a new Credential record that ServiceNow will use to connect to the online repository:
- Click on the Credentials item of the Connections & Credentials menu
- Click on New to start creating a new record
- Click on Basic Auth Credentials
- Fill in the fields and click on Submit
  
  The remote repository Credential is now created.
Go to the Studio, under System Applications → Studio
Click Import from source control
Fill at least the required URL and Credential fields
Click Import. When the application import is completed, click the Select Application button.
In the Select Application dialog, click the link to the new application to open it for editing in Studio.

If you want to customize the application, it is better to create a new branch and start working from there.

Create a new branch
1. In Studio, open the Source Control menu and select the Create Branch menu item.
2. Configure the branch.
  Branch Name: branchName Create from Tag: -- None --
3. Click the Create Branch button.
4. Click the Close button.

To load the application files included in the tag, return to the main ServiceNow browser tab (not Studio) and click the browser’s reload button to refresh the page.

Upload an Update Set

Although it is not recommended by ServiceNow, you can download the following Update Set and import it into a ServiceNow instance to create an application. An Update Set describes and contains all application’s files and is specific to ServiceNow.

Download the Update Set.
Go to Retrieved Update Sets under System Update Sets
Click on Import Update Set from XML
Select the Update Set you just downloaded
Click on Upload and wait for the upload to complete

The Retrieved Update Set table should be displayed with the freshly uploaded file named ProActive Workflows & Scheduling

Click on the Update Set Name to open it.
Click on Preview Update Set and wait for the preview to finish.
This step is important as it compares an update set retrieved from a remote instance to updates on the local instance to detect potential conflicts. You must preview an update set and address all problems before you can commit the update set.
Once finished, click on Close
Problems will appear because some tables and records don’t exist in your instance and ServiceNow doesn’t like that. That’s because we are installing an application and not updating it. We can ignore them and commit the changes.
1. Select all errors by clicking on the top left checkbox (this selects all the displayed errors)
2. Click on Accept remote changes on the top right
3. Repeat step a. and b. if errors remain
Once all problems have been addressed, Click Commit Update Set
Click Confirm data loss and wait for the commit to finish
Once again, and for the same reason ServiceNow will inform us of errors
Click on Show Commit Log and check that all records are of "Info" status

Set a filter as shown and click on Run to apply it.

Result should be empty.

15.4.4. Usage

Once the application is imported in your ServiceNow instances, the two-way communication between ProActive and ServiceNow is ready.

Please refer to the User documentation to look at:

The available ServiceNow workflows in ProActive’s catalog
How to use ProActive client in a ServiceNow instance

16. Scheduler start tuning

Scheduler start can be customized by automatically triggering user’s scripts. A list of scripts can be specified in PROACTIVE_HOME/config/scheduler/settings.ini following this

pa.scheduler.startscripts.paths=/your/script1/path;/your/script2/path;/your/script3/path

For example, load-examples.groovy which is executed by default at scheduler start, has in charge the full deployment of proactive-examples by: populating the running Catalog, exposing as templates specific workflows and copying workflows dependencies into dataspaces.

17. Add custom third-party libraries to ProActive server and Nodes classpath

Some workflows may need to use third-party libraries, so they can be executed properly. In order to do that, the first option is to add those libraries to the global classpath of ProActive server or Nodes. Depending on how Nodes are started, libraries can be added in the ProActive server or the Nodes classpath in three ways:

If Nodes are running on the same host as ProActive server or on a different host with a shared Proactive installation, the libraries can be placed under the addons directory of the server (i.e., $PROACTIVE_HOME/addons);
If Nodes are running on different hosts than ProActive Server and do not share a common proactive installation (such as NFS), the libraries can be placed under the addons directory of the Node (i.e., $PROACTIVE_NODE/addons);
If Nodes are running on different hosts than ProActive Server, and such Nodes are started using the executable jar 'node.jar' (usually used for cloud infrastructure), the libraries can be placed inside the lib folder of the node.jar archive itself, which is located on the server installation under $PROACTIVE_HOME/dist/war/rest (the node.jar is downloaded dynamically by the node).

The second option is to add those libraries dynamically for a given task execution. Please refer to the User guide documentation to learn how to add additional classpath.

18. Performance tuning

18.1. Database

By default, ProActive Scheduler is configured to use HSQLDB embedded database. This last is lightweight and does not require complex configuration. However, it cannot compete with more conventional database management systems, especially when the load increases. Consequently, it is recommended to setup an RDBMS such as MariaDB, Postgres or SQL Server if you care about performance or notice a slowdown.

ProActive Workflows and Scheduling distribution is provided with several samples for configuring the Scheduler and Resource Manager to use most standard RDBMS. You can find these examples in PROACTIVE_HOME/config/rm/templates/ and PROACTIVE_HOME/config/scheduler/templates/ folders.

18.1.1. Indexes

The Scheduler and RM databases rely on Hibernate to create and update the database schema automatically on startup. Hibernate is configured for adding indexes on frequently used columns, including columns associated to foreign keys. The goal is to provide a default configuration that behaves in a satisfactory manner with the different features available in the Scheduler. For instance, the Jobs housekeeping feature requires indexes on most foreign keys since a Job deletion may imply several cascade delete.

Unfortunately, each database vendor implements its own strategy regarding indexes on foreign keys. Thus, some databases such as MariaDB and MySQL automatically add indexes for foreign keys whereas some others like Postgres or Oracle do not add such indexes by default. Besides, Hibernate does not offer control over this strategy. As a consequence, this difference of behaviour, once linked to our default configuration for Hibernate, may lead to duplicate indexes (two indexes with different names for a same column) that are created for columns in RM and Scheduler tables.

In case you are using in production MySQL, MariaDB or a similar database that automatically adds indexes on foreign keys, it is strongly recommended to ask your DBA to identify and delete duplicate indexes created by the database system since they may hurt performances.

18.1.2. Database impact on job submission performance

The choice of the database provider can influence the Scheduler and Resource Manager performance overall. In the following figure, we see the job submission time between different database providers:

On a 1000-jobs submission basis across ProActive’s embedded database, the in-memory database and a standalone PostgreSQL server, we can see that the in-memory database is in fact faster and more consistent in speed than the default embedded database. However the in-memory database has no persistence of the data on disk, as opposed to the embedded database. A standalone database is a good tradeoff between speed and data resilience.

The embedded database is used by default in ProActive. To switch to an in-memory database, you need to modify the following files:

PROACTIVE_HOME/config/scheduler/database.properties:

Replace the default value of hibernate.connection.url to:

hibernate.connection.url=jdbc:hsqldb:mem:scheduler;hsqldb.tx=mvcc;hsqldb.lob_file_scale=1

PROACTIVE_HOME/config/rm/database.properties:

Replace the default value of hibernate.connection.url to:

hibernate.connection.url=jdbc:hsqldb:mem:rm;hsqldb.tx=mvcc;hsqldb.lob_file_scale=1

To switch to a standalone database, see Database.

18.1.3. Database impact on task creation performance

The choice of the database provider influences task creation time. In the following figure, we see the task creation time for different database providers:

Each experiment, consists of submitting one replicated job with some number of replicated tasks, and measuring time to create all these replicated tasks.

18.2. Tuning Linux for 15K nodes

It is necessary to tune underlying Linux system, in case there will be plenty of nodes in the Resource Manager.

It is mandatory to allow having a large number of running processes and open files.

For example, on 15.000 nodes, the recommended ulimit settings for running processes and open files are:

nproc soft/hard 100000
nofile soft/hard 65536

In addition, we recommend to have at least 16GB RAM for 15.000 nodes.

18.3. Housekeeping

The Scheduler provides a housekeeping mechanism that periodically removes finished jobs from the Scheduler and Workflow Execution portals. It also removes associated logs and all job data from the database to save space. This mechanism has two phases:

Once a job is finished, it sets its scheduled time for removal (the job expiration date)
Actual cleaning of expired jobs from the scheduler and/or the database will be periodically triggered and performed in a bulk operation

You can configure housekeeping with the following parameters located in the scheduler config file:

pa.scheduler.core.automaticremovejobdelay : delay in seconds to remove a terminated job.
pa.scheduler.core.automaticremove.errorjob.delay : delay in seconds to remove a job terminated with errors.
pa.scheduler.core.removejobdelay : instead of using job termination time to initiate the delay, use the result retrieval time (for example, when a user opens the job results in the scheduler portal).

Additionally, you can configure a remove delay for individual job execution using the REMOVE_DELAY or REMOVE_DELAY_ON_ERROR generic information.

If the pa.scheduler.core.automaticremovejobdelay setting is set to 0 (the default), the Housekeeping mechanism is disabled (no occurrence of housekeeping).

If the pa.scheduler.core.automaticremove.errorjob.delay setting is set to 0 (the default), removal of finished jobs containing errors uses the pa.scheduler.core.automaticremovejobdelay configuration.

The errorjob.delay setting is generally used in production environment for audit purpose. It allows keeping jobs containing errors longer in the database.

Please note that the housekeeping mechanism is not applied to the jobs which were already in the database before housekeeping was switched on. Similarly, a modification of the automaticremovejobdelay, automaticremove.errorjob.delay or removejobdelay settings will not be retroactively applied to already finished jobs.

The housekeeping mechanism is periodically triggered through the cron expression pa.scheduler.core.automaticremovejobcronexpression. As the boolean pa.scheduler.job.removeFromDataBase property is set to true by default, a query is executed in the scheduler database to remove all jobs that qualify for the bulk removal.

The support of pa.scheduler.job.removeFromDataBase=false has been discontinued in ProActive Scheduler version 10.1.0.

19. Nodes and Task Recovery

The Scheduler can be started in recovery mode. If so, the Scheduler will try to restore Resource Manager nodes and ProActive tasks that were running in the previous execution of the Scheduler. The nodes and task recovery feature makes sure you do not lose on-going computations if the Scheduler experiences a failure.

19.1. Use Case and Scope of the Feature

Suppose that at an instant T the Scheduler is running tasks. Without the nodes and task recovery feature, if the Scheduler crashed at instant T, then the tasks that were on-going would be re-scheduled on any node and re-executed from the beginning when the Scheduler restarts. With the nodes and task recovery feature enabled, when the Scheduler restarts it will retrieve the previous RM nodes and the on-going tasks, and the execution of the tasks will continue and finish as if there was no failure of the Scheduler.

The nodes and task recovery feature is not applicable when the Scheduler is cleanly stopped or exits normally. This is a mechanism that is only applicable upon failure of the Scheduler (machine crash, abrupt exit).

19.2. How Does Nodes and Task Recovery Work?

At startup, the Scheduler checks the nodes and the tasks that were previously persisted in database to be able to restore their state. Here are the different cases that can occur at recovery time:

If a node that is found in database is still alive, the node’s information is restored in the Resource Manager. The same restoration mechanism is applied for running tasks in the Scheduler.
If however, no node is found in database for a given node source, then the node recovery will not take place for this node source. Instead, the nodes will be re-deployed like for the first time.
If a node that is found in database is not alive at recovery time, then the node will also be recreated automatically.

A recovery is not a restart, and as such a clean restart of the Scheduler will not preserve on-going tasks and running nodes. Indeed, in the case of a regular shutdown of the Scheduler, the nodes are cleaned and removed from the Resource Manager before the Scheduler exits. Thus, there will be no nodes and no running tasks to recover when the Scheduler is started again.

To handle the particular situation in which the Scheduler is down when a running task terminates, a proper configuration of the Scheduler allows the task to hold the result until the Scheduler is up and running again.

19.3. Nodes and Task Recovery Configuration

19.3.1. Restrictions

The nodes recovery is not available for Scheduler settings that use the PAMR protocol. This is because this communication protocol relies on router endpoint identifiers that are regenerated when connections are reestablished.

The nodes recovery is also not available on the Nodes that are launched at the Scheduler startup by default. However you get the choice to activate or deactivate the feature when you create your own Node Source. By default, the nodes recovery is activated for any new Node Source.

19.3.2. How to make sure that nodes and task recovery is enabled

The nodes and task recovery feature requires a proper configuration to ensure that nodes and tasks are kept alive during the down time of the Scheduler.

Resource Manager configuration (PROACTIVE_HOME/config/rm/settings.ini)

In order to enable the nodes and task recovery feature, make sure that the pa.rm.nodes.recovery property of the Resource Manager is set to true. This property is set to true by default.
Node configuration

The ProActive nodes should be started with a configuration that keeps them alive if the Scheduler fails. Node properties can be set at node startup:
- Use -Dproactive.node.ping.delay to specify how often the node will try to communicate with the Resource Manager. The default for this property is 30 seconds.
- Use -Dproactive.node.reconnection.attempts to specify how many times the node will try to reconnect with the Resource Manager. The default for this property is to attempt 10 times.
These two properties define the total delay for the node to reconnect to the Resource Manager. By default, the total delay of reconnection is 5 minutes. When this total delay is consumed, the node shuts itself down, and the node recovery will not be applicable any more for this node.
Scheduler configuration (PROACTIVE_HOME/config/scheduler/settings.ini)

In order to make sure that a task will attempt several times to send back its result to the Scheduler, you must set the properties:
- pa.scheduler.core.node.ping.attempts to specify the number of times a finished task will try to contact the Scheduler. The default for this property is to attempt only once.
- pa.scheduler.core.nodepingfrequency property to define a delay (in seconds) in between two attempts. The default for this property is 20 seconds.
If the Scheduler is still not up and running after the total delay defined by these properties (which is 20 seconds by default), then the task’s result will be lost forever. Note that these two properties are also used by the scheduler to decide when to re-schedule tasks.

19.3.3. How to disable nodes and task recovery

You can completely disable the nodes and task recovery feature by setting the pa.rm.nodes.recovery property of the Resource Manager to false.

In addition, the nodes and task recovery can be configured per Node Source at creation time. This configuration will be applied only if the global pa.rm.nodes.recovery property is set to true. The specification of nodes recovery per Node Source can be done through the Resource Manager Web Interface, through the Resource Manager REST API, and through the Command Line client (CLI).

Through the Web Interface, the form to create a new Node Source contains a checkbox that controls nodes recoverability.
Through the REST API, the nodesRecoverable parameter of the rm/nodesource/create and rm/nodesource REST endpoints specify whether the ProActive Nodes of a new Node Source will be recoverable. Nodes recovery will be disabled if the given value does not match any case of the "true" string.
Through the CLI, the createns and definens commands take an optional parameter to specify whether the ProActive Nodes of a new Node Source will be recoverable. Nodes recovery will be disabled if the optional parameter is provided and does not match any case of the "true" string.

19.4. Nodes and Task Recovery on Cloud Platforms

The nodes and task recovery feature of ProActive is also available for the deployments on Microsoft Azure and Amazon EC2 clouds.

For these platforms, when the Scheduler is restarted after a failure, it will try to contact the nodes that are deployed on the cloud instances:

If the nodes are alive then the recovery proceeds like with non-cloud infrastructures.
If the nodes cannot be contacted, the recovery mechanism will try to re-deploy them by asking the instances to run a node deployment script.
If the script fails, it means that the cloud instances do not exist anymore, so the recovery mechanism will trigger a re-deployment including the re-creation of the instances.

19.5. Nodes and Task Recovery Performance Tuning

By default, the persistence of the node states is slightly delayed in order to batch several database operations in the same transaction. This is done in order to minimize the execution time overhead of the nodes and task recovery feature. However, this default batching mechanism can be overriden to adapt the delay, or to disable batching.

The properties that allow performance tuning are the following:

pa.rm.node.db.operations.delay defines the delay in milliseconds that is applied when the RM requests the persistence of a node. By default the delay is set to 100 milliseconds. If the value of this property is set to 0, then all database operations related to the persistence of nodes are executed immediately and synchronously.
pa.rm.nodes.db.operations.update.synchronous defines whether the node updates are persisted synchronously (with no delay) whenever it is possible. By default this property is set to true.

20. Troubleshooting

20.1. Logs

If something goes wrong the first place to look for the problem are the Scheduler logs. By default all logs are in PROACTIVE_HOME/logs directory.

Users submitting jobs have access to server logs of their jobs through the Scheduler Web interface

20.2. Common Problems

20.2.1. 'Path too Long' Errors When Unzipping Windows Downloads

When you unzip a Windows package using the default Windows compression utility, you might get errors stating that the path is too long. Path length is determined by the Windows OS. The maximum path, which includes drive letter, colon, backslash, name components separated by backslashes, and a terminating null character, is defined as 260 characters.

Workarounds:

Move the Zip file on the root level of the system drive and unzip from there.
Use a third-party compression utility. Unlike the default Windows compression utility, some third-party utilities allow for longer maximum path lengths.

21. Reference

21.1. Scheduler Properties

Scheduler Properties are read when ProActive Scheduler is started therefore you need to restart it to apply changes.

The default configuration file is SCHEDULER_HOME/scheduler/settings.ini.

# INFORMATION : each file path must be absolute, OR relative to the Scheduler_Home path
#-------------------------------------------------------
#-------------   SCHEDULER PROPERTIES   ----------------
#-------------------------------------------------------

# Scheduler home directory (this default value should be proper in most cases)
pa.scheduler.home=.

# HSQLDB location, use this property to store the database in a different folder than scheduler_home/data/db
# you can use an absolute location or a relative location from scheduler home
# pa.hsqldb.location=data/db

# Scheduler rest url. If not defined, it is set automatically when starting the server.
# When the server has a public endpoint, different from the hostname available on the machine, this property should be used to correctly set the url
# Accessed inside workflows and tasks with the PA_SCHEDULER_REST_URL variable
#pa.scheduler.rest.url=

# Catalog rest url. If not defined, it is set automatically when starting the server.
# Accessed inside workflows and tasks with the PA_CATALOG_REST_URL variable
#pa.catalog.rest.url=

# Cloud Automation rest url. If not defined, it is set automatically when starting the server.
# Accessed inside workflows and tasks with the PA_CLOUD_AUTOMATION_REST_URL variable
#pa.cloud-automation.rest.url

# Job Planner rest url. If not defined, it is set automatically when starting the server.
# Accessed inside workflows and tasks with the PA_JOB_PLANNER_REST_URL variable
#pa.job-planner.rest.url

# Notification Service rest url. If not defined, it is set automatically when starting the server.
# Accessed inside workflows and tasks with the PA_NOTIFICATION_SERVICE_REST_URL variable
#pa.notification-service.rest.url

# Set this property to define the public url of the scheduler (e.g. if behind a reverse proxy or accessed through a domain)
# Set automatically by the server to the value of pa.scheduler.rest.ur if not set explicitly.
# Accessed inside workflows and tasks with the PA_SCHEDULER_REST_PUBLIC_URL variable.
# Example https://myserver.mydomain.com/rest
#pa.scheduler.rest.public.url=

# Set this property to define the public url of the catalog (e.g. if behind a reverse proxy or accessed through a domain)
# Set automatically by the server to the value of pa.catalog.rest.url if not set explicitly.
# Accessed inside workflows and tasks with the PA_CATALOG_REST_PUBLIC_URL variable.
# Example https://myserver.mydomain.com/catalog
#pa.catalog.rest.public.url=

# Set this property to define the public url of the cloud automation service (e.g. if behind a reverse proxy or accessed through a domain)
# Set automatically by the server to the value of pa.cloud-automation.rest.url if not set explicitly.
# Accessed inside workflows and tasks with the PA_CLOUD_AUTOMATION_REST_PUBLIC_URL variable.
# Example https://myserver.mydomain.com/cloud-automation-service
#pa.cloud-automation.rest.public.url=

# Set this property to define the public url of the job planner (e.g. if behind a reverse proxy or accessed through a domain)
# Set automatically by the server to the value of pa.job-planner.rest.url if not set explicitly.
# Accessed inside workflows and tasks with the PA_JOB_PLANNER_REST_PUBLIC_URL variable.
# Example https://myserver.mydomain.com/job-planner
#pa.job-planner.rest.public.url=

# Set this property to define the public url of the notification service (e.g. if behind a reverse proxy or accessed through a domain)
# Set automatically by the server to the value of pa.notification-service.rest.url if not set explicitly.
# Accessed inside workflows and tasks with the PA_NOTIFICATION_SERVICE_REST_PUBLIC_URL variable.
# Example https://myserver.mydomain.com/notification-service
#pa.notification-service.rest.public.url=

# Timeout for the scheduling loop (in millisecond)
pa.scheduler.core.timeout=10000

# Auto-reconnection to the Resource Manager default reconnection attempt every 10 seconds for 1 hour
pa.scheduler.core.rmconnection.autoconnect = true
pa.scheduler.core.rmconnection.timespan = 10000
pa.scheduler.core.rmconnection.attempts = 360

# Maximum number of threads used to execute client requests
# (e.g. change Job priority, kill Job, etc.)
pa.scheduler.core.clientpoolnbthreads=100

# Maximum number of threads used to execute internal scheduling operations
# (handle task termination, restart task, etc.)
pa.scheduler.core.internalpoolnbthreads=100

# Maximum number of threads used to ping tasks regularly to get its progress and detect node failures
pa.scheduler.core.taskpingerpoolnbthreads=100

# Number of threads used to delay operations which are NOT related to housekeeping
# (e.g. scheduler shutdown, handle task restart on error, etc.)
pa.scheduler.core.scheduledpoolnbthreads=20

# Number of threads used to handle scheduled operations with the housekeeping feature
pa.scheduler.core.housekeeping.scheduledpoolnbthreads=5

# Maximum number of threads in the thread pool that serves to recover running tasks in parallel at scheduler start up
pa.scheduler.core.parallel.scheduler.state.recover.nbthreads=100

# Check for failed node frequency in second
# Also used by the node to ping the scheduler after finishing a task
pa.scheduler.core.nodepingfrequency=20

# The scheduler will decide to restart a task, after a given tolerated number of failed attempts.
# A value of zero means that the scheduler will restart a task after the first failure.
# Also used by a node to retry to send the result of a task to the scheduler
pa.scheduler.core.node.ping.attempts=1

# Time in milliseconds before sending a kill request to the scheduler
pa.scheduler.core.killdelay=2000

# Scheduler default policy full name
pa.scheduler.policy=org.ow2.proactive.scheduler.policy.ExtendedSchedulerPolicy

# Alternate scheduling policies (uncomment the appropriate following line and comment the one above)

# pa.scheduler.policy=org.ow2.proactive.scheduler.policy.license.LicenseSchedulingPolicy
# pa.scheduler.policy=org.ow2.proactive.scheduler.policy.limit.NodeUsageSchedulingPolicy
# pa.scheduler.policy=org.ow2.proactive.scheduler.policy.edf.EDFPolicy
# pa.scheduler.policy=org.ow2.proactive.scheduler.policy.edf.EDFPolicyExtended

# Defines the maximum number of tasks to be scheduled in each scheduling loop.
pa.scheduler.policy.nbtaskperloop=10

# If set to true (default), the scheduling loop will deploy tasks only on nodes which are free at the beginning of the loop.
# This is mandatory in order to respect strict task fifo order or priorities, but it can reduce the task throughput.
# Change this setting if a lot a short lived tasks are scheduled per minute and enforcing strict fifo priority is not mandatory
pa.scheduler.policy.strict.fifo=true

# Path of the license properties file
pa.scheduler.license.policy.configuration=config/scheduler/license.properties

# location of the jdbm database for persistent license registrations
pa.scheduler.license.policy.db=data/licenses

# Name of the JMX MBean for the scheduler
pa.scheduler.core.jmx.connectorname=JMXSchedulerAgent

# Port of the JMX service for the Scheduler.
pa.scheduler.core.jmx.port=5822

# Accounting refresh rate from the database in seconds
pa.scheduler.account.refreshrate=180

# RRD data base with statistic history
pa.scheduler.jmx.rrd.name=data/scheduler_statistics.rrd

# RRD data base step in seconds
pa.scheduler.jmx.rrd.step=4

# User session time. User is automatically disconnect after this time if no request is made to the scheduler. 8 hours by default.
# negative number indicates that session is infinite (value specified in second)
pa.scheduler.core.usersessiontime=28800

# Timeout for the start task action. Time during which the scheduling could be waiting (in millis)
# this value relies on the system and network capacity
pa.scheduler.core.starttask.timeout=20000

# Maximum number of threads used for the start task action. This property define the number of blocking resources
# until the scheduling loop will block as well.
# As it is related to the number of nodes, this property also define the number of threads used to terminate taskLauncher
pa.scheduler.core.starttask.threadnumber=100

# Maximum number of threads used to send events to clients. This property defines the number of clients
# than can block at the same time. If this number is reached, every clients won't receive events until
# a thread unlock.
pa.scheduler.core.listener.threadnumber=100

# List of the scripts paths to execute at scheduler start. Paths are separated by a ';'.
pa.scheduler.startscripts.paths=tools/LoadPackages.groovy

# Size of parsed workflow cache, used to optimize workflow submission time
pa.scheduler.stax.job.cache=1000

# Size of the cache used by the scheduling policy to ensure that delayed jobs or tasks are scheduled at the precise date (without skipping seconds)
pa.scheduler.startat.cache=5000

# optimization cache used used by the scheduling policy to avoid parsing the same pending jobs continuously
pa.scheduler.startat.value.cache=30000

# Expiration period in seconds of cache used to download workflows
pa.scheduler.download.cache.expiration=60

# Expiration period in seconds of cache used to store session ids
pa.scheduler.method.session.cache.expiration=300

# Period of the HSQLDB monitoring thread (in seconds)
pa.scheduler.hsqldb.monitor.period=10

#-------------------------------------------------------
#----------------   JOBS PROPERTIES   ------------------
#-------------------------------------------------------

# Remove job delay (in seconds). (The time between getting back its result and removing it from the scheduler)
# Set this time to 0 if you don't want the job to be removed.
pa.scheduler.core.removejobdelay=0

# Automatic remove job delay (in seconds). (The time between the termination of the job and removing it from the scheduler)
# Set this time to 0 if you don't want the job to be removed automatically.
pa.scheduler.core.automaticremovejobdelay=0

# Automatic remove error job delay (in seconds). (The time between the termination of a job which contains errors and removing it from the scheduler)
# A job is considered with errors if its number of failed or faulty tasks is greater than 0.
# This setting is ignored if automaticremovejobdelay is set to 0
# Set this time to 0 if you want to apply the same configuration as automaticremovejobdelay.
pa.scheduler.core.automaticremove.errorjob.delay=0

# Remove job in database when removing it from the scheduler.
# This housekeeping feature can be replaced by a stored procedure
# that runs at the desired period of time (e.g. non-business hours)
# Such an example is available in samples/scripts/database/postgres/
# Changing this setting is strongly not recommended as the support for pa.scheduler.job.removeFromDataBase=false has been discontinued
pa.scheduler.job.removeFromDataBase=true

# This cron expression determines the housekeeping call frequency.
# Default value is 10 minutes: this will invoke the housekeeping mechanism
# to remove every jobs which are set to be removed and has their scheduled time for removal reached.
pa.scheduler.core.automaticremovejobcronexpression=*/10 * * * *

# Specific character encoding when parsing the job xml file
pa.file.encoding=UTF-8

# This property defines size of LRU cache which stores finished jobs* in memory.
# * by finished jobs, we mean finished jobs which were finished when scheduler started
#pa.scheduler.finishedjobs.lru.cache.size=1000

#-------------------------------------------------------
#---------------   TASKS PROPERTIES   ------------------
#-------------------------------------------------------
# Initial time to wait before the re-execution of a task. (in millisecond)
pa.scheduler.task.initialwaitingtime=1000

# Maximum number of execution for a task in case of failure (node down)
pa.scheduler.task.numberofexecutiononfailure=2

# If true tasks are ran in a forked JVM, if false they are ran in the node's JVM.
# When it's not set, each task can specify its execution mode through the task property "fork".
#pa.scheduler.task.fork=true

# If true tasks are always ran in RunAsMe mode (impersonation). This automatically implies pa.scheduler.task.fork=true (other setting is ignored)
pa.scheduler.task.runasme=false

# Maximum number of tasks in a tasks page
pa.scheduler.tasks.page.size=100

# Maximum duration time of task. If configured, the tasks will be terminated after the walltime exceeded
# Format is ss, mm:ss, or hh:mm:ss
#pa.scheduler.task.walltime=

# if the following property is set to a non-empty value, the scheduler will be able to execute only forkenvironment or clean scripts contained
# in the provided directory. All other scripts will be rejected.
#pa.scheduler.script.authorized.dir=

# The pa.scheduler.script.authorized.dir is browsed every refreshperiod time to load authorized scripts.
pa.scheduler.script.authorized.dir.refreshperiod=60000

# Refresh time to reload the security policy file (security.java.policy-server) 
pa.scheduler.auth.policy.refreshperiod.seconds=30


#-------------------------------------------------------
#-------------   DATASPACES PROPERTIES   ---------------
#-------------------------------------------------------

# Default INPUT space URL. The default INPUT space is used inside each job that does not define an INPUT space.
# Normally, the scheduler will start a FileSystemServer on a default location based on the TEMP directory.
# If the following property is specified, this FileSystemServer will be not be started and instead the provided dataspace
# url will be used
#pa.scheduler.dataspace.defaultinput.url=

# The following property can be used in two ways.
# 1) If a "pa.scheduler.dataspace.defaultinput.url" is provided, the defaultinput.path property
#   tells the scheduler where the actual file system is (provided that he has access to it). If the scheduler does not have
#   access to the file system where this dataspace is located then this property must not be set.
#       - On windows, use double backslash in the path, i.e. c:\\users\\...
#       - you can provide a list of urls separated by spaces , i.e. : http://myserver/myspace file:/path/to/myspace
#       - if one url contain spaces, wrap all urls in the list between deouble quotes :
#               "http://myserver/myspace"  "file:/path/to/my space"
# 2) If a "pa.scheduler.dataspace.defaultinput.url" is not provided, the defaultinput.path property will tell the scheduler
#   to start a FileSystemServer on the provided defaultinput.path instead of its default location

### the default location is SCHEDULER_HOME/data/defaultinput
#pa.scheduler.dataspace.defaultinput.localpath=

# Host name from which the localpath is accessible, it must be provided if the localpath property is provided
#pa.scheduler.dataspace.defaultinput.hostname=

# The same for the OUPUT (see above explanations in the INPUT SPACE section)
# (concerning the syntax, see above explanations in the INPUT SPACE section)
#pa.scheduler.dataspace.defaultoutput.url=
### the default location is SCHEDULER_HOME/data/defaultoutput
#pa.scheduler.dataspace.defaultoutput.localpath=
#pa.scheduler.dataspace.defaultoutput.hostname=

# The same for the GLOBAL space. The GLOBAL space is shared between each users and each jobs.
# (concerning the syntax, see above explanations in the INPUT SPACE section)
#pa.scheduler.dataspace.defaultglobal.url=
### the default location is SCHEDULER_HOME/data/defaultglobal
#pa.scheduler.dataspace.defaultglobal.localpath=
#pa.scheduler.dataspace.defaultglobal.hostname

# The same for the USER spaces. A USER space is a per-user global space. An individual space will be created for each user in subdirectories of the defaultuser.localpath.
# Only one file server will be created (if not provided)
# (concerning the syntax, see above explanations in the INPUT SPACE section)
#pa.scheduler.dataspace.defaultuser.url=
### the default location is SCHEDULER_HOME/data/defaultuser
#pa.scheduler.dataspace.defaultuser.localpath=
#pa.scheduler.dataspace.defaultuser.hostname=

# Is userspace based on impersonation (true) or path (false)
#  Examples:
#    sftp/vsftp is based on impersonation, a user space url will be sftp://user@server
#    file is based on path, a user space url will be file://user_space_root/user/
# Uncomment the following property to enable impersonation if the userspace protocol is sftp or vsftp (default is false)
# pa.scheduler.dataspace.defaultuser.impersonation=true

# When using userspace impersonation, internal accounts of the ProActive scheduler need to be discarded
# For example, in a sftp server, there will be no account matching internal accounts such as rm, scheduler, etc
# For these internal accounts, a user space will not be mounted.
pa.scheduler.dataspace.defaultuser.internal.accounts=admin,rm,scheduler,watcher,user,demo,guest,nsadmin,nsadmin2,provider,radmin,test,test_executor

#-------------------------------------------------------
#----------------   LOGS PROPERTIES   ------------------
#-------------------------------------------------------
# Logs forwarding method
# Possible methods are :
# Simple socket : org.ow2.proactive.scheduler.common.util.logforwarder.providers.SocketBasedForwardingProvider
# SSHTunneled socket : org.ow2.proactive.scheduler.common.util.logforwarder.providers.SocketWithSSHTunnelBasedForwardingProvider
# ProActive communication : org.ow2.proactive.scheduler.common.util.logforwarder.providers.ProActiveBasedForwardingProvider
#
# set this property to empty string to disable log forwarding alltogether
pa.scheduler.logs.provider=org.ow2.proactive.scheduler.common.util.logforwarder.providers.ProActiveBasedForwardingProvider

# Location of server job and task logs (comment to disable job logging to separate files).
# Can be an absolute path or a path relative to the scheduler home.
# If you are interested in disabling all outputs to 'logs/jobs' you must
# also have a look at the property 'pa.rm.logs.selection.location' in 'PROACTIVE_HOME/config/rm/settings.ini'
# Please note that disabling job logging will prevent Jobs and Tasks Server logs to be retrieved
# from the REST API and thus the Scheduler portal.
pa.scheduler.job.logs.location=logs/jobs/

# Size limit for job and task logs in bytes
pa.scheduler.job.logs.max.size=10MB

# Format pattern for the task output logs
pa.scheduler.job.task.output.logs.pattern=[%X{job.id}t%X{task.id}@%X{host};%X{task.name};%d{HH:mm:ss}] %m %n

# The following parameters are to monitor the quantity of jobs and the impact on memory and DB.
# This feature is disabled by default. The following log settings (in the config/log folder)
# need to be uncommented to enable the polling:
# log4j.logger.org.ow2.proactive.scheduler.core.helpers.TableSizeMonitorRunner
# log4j.logger.org.ow2.proactive.scheduler.core.helpers.JobsMemoryMonitorRunner
#
# Each polling from TableSizeMonitorRunner will print the following information into the logs:
# - JobData (All)
# - JobData (Finished)
# - JobDataVariable
# - JobContent
# - TaskData (All)
# - TaskData (Finished)
# - SelectorData
# - EnvironmentModifierData
# - ScriptData
# - SelectionScriptData
# - TaskDataVariable
# - TaskResultData
# - ThirdPartyCredentialData

# Each polling from JobsMemoryMonitorRunner will print the following information into the logs:
# - pendingJobs
# - runningJobs
# - finishedJobs
# - allJobsActual
# - AllJobsComputed
# - deleteCount
# - updateCount
# - insertCount
# - fetchCount
# - loadCount
# - flushCount

# The last 6 metrics are fetched from the Hibernate Statistics layer.

# Modify the polling frequency for the memory metrics. The default value is 1 minute.
# pa.scheduler.mem.monitoring.freq=* * * * *

# Define verbosity of job description when submitted
# If true, Job and Tasks details are logged (can slow down processes for jobs with many (>500) tasks)
# If false, only Job metadata are logged
pa.scheduler.job.submission.detailed.logging=true

#-------------------------------------------------------
#-----------   AUTHENTICATION PROPERTIES   -------------
#-------------------------------------------------------

# Path to the Jaas configuration file which defines what modules are available for internal authentication
pa.scheduler.auth.jaas.path=config/authentication/jaas.config

# Path to the private key file which is used to encrypt credentials for authentication
pa.scheduler.auth.privkey.path=config/authentication/keys/priv.key

# Path to the public key file which is used to encrypt credentials for authentication
pa.scheduler.auth.pubkey.path=config/authentication/keys/pub.key

# Uncomment the following line to configure a global domain for scheduler users in active directory environments
# pa.scheduler.auth.global.domain=mydomain

# LDAP Authentication configuration file path, used to set LDAP configuration properties
# If this file path is relative, the path is evaluated from the Scheduler dir (ie application's root dir)
# with the variable defined below : pa.scheduler.home.
# else, (if the path is absolute) it is directly interpreted
pa.scheduler.ldap.config.path=config/authentication/ldap.cfg

# Keycloak Authentication configuration file path, used to set keycloak configuration properties
# If this file path is relative, the path is evaluated from the Scheduler dir (ie application's root dir)
# with the variable defined below : pa.scheduler.home.
# else, (if the path is absolute) it is directly interpreted
pa.scheduler.keycloak.config.path=config/authentication/keycloak.cfg

# Multi-domain LDAP configuration
# a comma-separated list of the pair domain_name:configuration_path
# see comment above regarding file paths locations
# pa.scheduler.multi.ldap.config=domain1:config/authentication/ldap-domain1.cfg,domain2:config/authentication/ldap-domain2.cfg

# Login file name for file authentication method
# If this file path is relative, the path is evaluated from the Scheduler dir (ie application's root dir)
# with the variable defined below : pa.scheduler.home.
# else, the path is absolute, so the path is directly interpreted
pa.scheduler.core.defaultloginfilename=config/authentication/login.cfg

# Group file name for file authentication method
# If this file path is relative, the path is evaluated from the Scheduler dir (ie application's root dir)
# with the variable defined below : pa.scheduler.home.
# else, the path is absolute, so the path is directly interpreted
pa.scheduler.core.defaultgroupfilename=config/authentication/group.cfg

# Tenant file name for file authentication method
# If this file path is relative, the path is evaluated from the Scheduler dir (ie application's root dir)
# with the variable defined below : pa.scheduler.home.
# else, the path is absolute, so the path is directly interpreted
pa.scheduler.core.defaulttenantfilename=config/authentication/tenant.cfg

# If enabled, filter jobs according to user tenant
pa.scheduler.core.tenant.filter=false

# List of domain names that can be used during a login (can be a list of windows domain names or a list of tenants in Multi-LDAP configuration)
# This setting is interpreted by ProActive portals to show a selectable list of domains
# Domain names should be lowercase
# It is possible to allow both named domain and empty domain using the syntax pa.scheduler.allowed.domains=,domain1,domain2
# pa.scheduler.allowed.domains=domain1,domain2

# Property that define the method that have to be used for logging users to the Scheduler
# It can be one of the following values:
#	- "SchedulerFileLoginMethod" to use file login and group management
#	- "SchedulerLDAPLoginMethod" to use LDAP login management
#	- "SchedulerMultiLDAPLoginMethod" to use LDAP login management across multiple domains
#	- "SchedulerPAMLoginMethod" to use PAM login management
#	- "SchedulerKeycloakLoginMethod" to use Keycloak login management
pa.scheduler.core.authentication.loginMethod=SchedulerFileLoginMethod

# Creates a credential file (username.cred) for each successful login in the authentication folder
pa.scheduler.create.credentials.when.login=true

#-------------------------------------------------------
#------------------   RM PROPERTIES   ------------------
#-------------------------------------------------------
# Path to the Scheduler credentials file for RM authentication
pa.scheduler.resourcemanager.authentication.credentials=config/authentication/scheduler.cred

# Use single or multiple connection to RM :
# (If true)  the scheduler user will do the requests to rm
# (If false) each Scheduler users have their own connection to RM using their scheduling credentials
pa.scheduler.resourcemanager.authentication.single=false

# Set a timeout for initial connection to the RM connection (in ms)
pa.scheduler.resourcemanager.connection.timeout=120000

#-------------------------------------------------------
#--------------   HIBERNATE PROPERTIES   ---------------
#-------------------------------------------------------
# Hibernate configuration file (relative to home directory)
pa.scheduler.db.hibernate.configuration=config/scheduler/database.properties

# Drop database before creating a new one
# If this value is true, the database will be dropped and then re-created
# If this value is false, database will be updated from the existing one.
pa.scheduler.db.hibernate.dropdb=false

# This property is used to limit number of finished jobs loaded from the database
# at scheduler startup. For example setting this property to '10d' means that
# scheduler should load only finished jobs which were submitted during last
# 10 days. In the period expression it is also possible to use symbols 'h' (hours)
# and 'm' (minutes).
# If property isn't set then all finished jobs are loaded.
pa.scheduler.db.load.job.period=0m

# Defines the maximum number of times a transaction that fails and rollbacks
# will be retried. Each retry is performed after a given amount of time.
# The default value is 1 and any value below 1 is replaced by the default value.
pa.scheduler.db.transactions.maximum.retries=5

# Initial delay to wait in ms before the first retry in case of a transaction
# failure. This delay is multiplied by `pa.scheduler.db.transactions.damping.factor`
# after each retry.
pa.scheduler.db.transactions.sleep.delay=1000

# Defines the factor by which the sleep delay is multiplied after each retry.
pa.scheduler.db.transactions.damping.factor=2

# Batch size to load Jobs from database when scheduler is restarted
pa.scheduler.db.recovery.load.jobs.batch_size=100

# maximum number of items passed as parameters to some database queries (jobid list, etc)
pa.scheduler.db.items.max.size=1000

# Batch size to fetch parent tasks'results in a merge task
pa.scheduler.db.fetch.batch_size=50

#-------------------------------------------------------
#-------  VARIABLES & GENERIC INFO PROPERTIES  ---------
#-------------------------------------------------------

# file containing variables which can be used in all workflows
pa.scheduler.global.variables.configuration=config/scheduler/global_variables.xml

# refresh period, in minutes, for the global variables configuration
pa.scheduler.global.variables.refresh=5

#-------------------------------------------------------
#----------  EMAIL NOTIFICATION PROPERTIES  ------------
#-------------------------------------------------------

# Change emailnotification.properties file to set up the From address for notifications
# and its smtp servers etc.
pa.scheduler.notification.email.configuration=config/scheduler/emailnotification.properties

# Set to true to enable email notifications about finished jobs. Emails
# are sent to the address specified in the generic information of a
# job with the key EMAIL; example:
#    <genericInformation>
#        <info name="EMAIL" value="user@example.com"/>
#    </genericInformation>
pa.scheduler.notifications.email.enabled=false
# From address for notifications emails (set it to a valid address if
# you would like email notifications to work)
pa.scheduler.notifications.email.from=example@username.com

#-------------------------------------------------------
#----------  SYNCHRONIZATION STORE PROPERTIES  ---------
#-------------------------------------------------------

# location of the jdbm database for persistent channels
pa.scheduler.synchronization.db=data/synchronization

#-------------------------------------------------------
#--------------  SIGNAL API PROPERTIES  ----------------
#-------------------------------------------------------

# name of the channel of workflow signals (STOP, CONTINUE, etc)
pa.scheduler.signals.channel=PA_SIGNALS_CHANNEL_

#-------------------------------------------------------
#----------------  PORTAL PROPERTIES  ------------------
#-------------------------------------------------------
pa.scheduler.portal.configuration=config/portal/scheduler-portal-display.conf

#-------------------------------------------------------
#----------------  LABEL PROPERTIES  ------------------
#-------------------------------------------------------
pa.scheduler.label.regex=^[a-zA-Z0-9_/-]*$
pa.scheduler.label.max.length=20

21.2. Resources Manager Properties

Resource Manager Properties are read when ProActive Scheduler is started therefore you need to restart it to apply changes.

The default configuration file is SCHEDULER_HOME/rm/settings.ini.

#-------------------------------------------------------
#-------------   RMCORE PROPERTIES   ----------------
#-------------------------------------------------------


# definition of all java properties used by resource manager
# warning : definition of these variables can be override by user at JVM startup,
# using for example -Dpa.rm.home=/foo, in the java command

# name of the ProActive Node containing RM's active objects
pa.rm.node.name=RM_NODE

# number of local nodes to start with the Resource Manager
# if value is -1, then the number of local nodes is max(2, numberOfCoreAvailableLocally-1)
pa.rm.local.nodes.number=-1

# ping frequency used by node source for keeping a watch on handled nodes (in ms)
pa.rm.node.source.ping.frequency=45000

# Periodic down and lost nodes removal attempts (cron expression)
# If not set, the down and lost nodes will never be removed automatically
pa.rm.nodes.unavailable.removal.frequency=*/30 * * * *
# Time (in minutes) after which a down or lost node is eligible to periodic removal
# If not set, or if not greater than 0, the down and lost nodes will never be removed automatically
pa.rm.nodes.unavailable.maxperiod=1440

# ping frequency used by resource manager to ping connected clients (in ms)
pa.rm.client.ping.frequency=45000

# The period of sending "alive" event to resource manager's listeners (in ms)
pa.rm.aliveevent.frequency=300000

# timeout for selection script result
pa.rm.select.script.timeout=60000

# number of selection script digests stored in the cache to predict the execution results
pa.rm.select.script.cache=10000

# The time period when a node has the same dynamic characteristics (in ms).
# It needs to pause the permanent execution of dynamic scripts on nodes.
# Default is 5 mins, which means that if any dynamic selection scripts returns
# false on a node it won't be executed there at least for this time.
pa.rm.select.node.dynamicity=300000

# The full class name of the policy selected nodes
pa.rm.selection.policy=org.ow2.proactive.resourcemanager.selection.policies.ShufflePolicy

# Timeout for remote script execution (in ms) 
pa.rm.execute.script.timeout=180000

# If set to non-empty value the resource manager executes only scripts from this directory.
# All other selection scripts will be rejected.
# pa.rm.select.script.authorized.dir=

# The pa.rm.select.script.authorized.dir is browsed every refreshperiod time to load authorized scripts.
pa.rm.select.script.authorized.dir.refreshperiod=60000

# timeout for node lookup
pa.rm.nodelookup.timeout=60000

# GCM application (GCMA) file path, used to perform GCM deployments
# If this file path is relative, the path is evaluated from the Resource manager dir (ie application's root dir)
# defined by the "pa.rm.home" JVM property
# else, the path is absolute, so the path is directly interpreted
pa.rm.gcm.template.application.file=config/rm/deployment/GCMNodeSourceApplication.xml

# java property string defined in the GCMA defined above, which is dynamically replaced
# by a GCM deployment descriptor file path to deploy
pa.rm.gcmd.path.property.name=gcmd.file

# Resource Manager home directory
pa.rm.home=.

# Lists of supported infrastructures in the resource manager
pa.rm.nodesource.infrastructures=config/rm/nodesource/infrastructures

# Lists of supported node acquisition policies in the resource manager
pa.rm.nodesource.policies=config/rm/nodesource/policies

# Timeout (ms) for the resource manger to recover a broken node source in scheduler aware policy
pa.rm.scheduler.aware.policy.nodesource.recovery.timeout=10000

# Number of trials for the resource manager to recover a broken node source in scheduler aware policy
pa.rm.scheduler.aware.policy.nodesource.recovery.trial.number=10

# Max number of threads in node source for parallel task execution
pa.rm.nodesource.maxthreadnumber=75

# Max number of threads in selection manager
pa.rm.selection.maxthreadnumber=50

# Max number of threads in monitoring
pa.rm.monitoring.maxthreadnumber=5

# Number of threads in the node cleaner thread pool
pa.rm.cleaning.maxthreadnumber=5

# Maximum node and user history period in seconds (Default, disabled, uncomment to enable 7 days max history)
#pa.rm.history.maxperiod=604800

# Frequency of node history removal (cron expression)
pa.rm.history.removal.cronperiod=*/10 * * * *



# Max number of lines stored from the infrastructure processes output
pa.rm.infrastructure.process.output.maxlines=2000

#Name of the JMX MBean for the RM
pa.rm.jmx.connectorname=JMXRMAgent

#port of the JMX service for the RM.
pa.rm.jmx.port=5822

#Accounting refresh rate from the database in seconds (0 means disabled)
pa.rm.account.refreshrate=180

# RRD data base with statistic history
pa.rm.jmx.rrd.name=data/rm_statistics.rrd

# RRD data base step in seconds
pa.rm.jmx.rrd.step=4

# path to the Amazon EC2 account credentials properties file,
# mandatory when using the EC2 Infrastructure
pa.rm.ec2.properties=config/rm/deployment/ec2.properties

# Defines if the lock restoration feature is enabled on RM startup.
# When set to {@code true}, the RM will try to lock per Node Source
# as many Nodes as there were on the previous run.
#
# The approach is best effort and Node hostname is not considered.
# As a result, Nodes are not necessarily locked on the same host.
pa.rm.nodes.lock.restoration=true

# Defines if the node restoration feature is enabled.
# When set to {@code true}:
# - on RM startup the RM tries to look up the nodes that were present
#   before the scheduler crashed
# - the RM persists node information
pa.rm.nodes.recovery=true

# Insert a delay before a database node source update or any node operation
# is executed. If set to 0, all database operation are executed synchronously.
pa.rm.node.db.operations.delay=500

# If set to {@code true}, and if {@link pa.rm.node.db.operations.delay} is not
# set to 0, then node updates will be executed synchronously as much as possible.
# In this case, node updates can still be postponed if the node creation is still
# pending.
#
pa.rm.nodes.db.operations.update.synchronous=true

# maximum length of in expressions in the database
pa.rm.db.items.max.size=1000

# Defines if  the runtime (RT) have to be killed when the resource manager (RM) is shutdown.
pa.rm.shutdown.kill.rt=true

# Defines the maximum number of RMEvents which can be sent to the client in one request.
pa.rm.rest.monitoring.maximum.chunk.size=100

# Enable asynchronous logging
pa.log4j.async.appender.enabled=true

# Enable asynchronous logging appender cache (store opened file appender into a cache structure)
pa.log4j.async.appender.cache.enabled=false

# Defines the buffer size used in asynchronous appenders
pa.log4j.async.appender.buffer.size=10000

# Defines the AsynchFileAppender flush timeout
pa.log4j.async.appender.flush.timeout=50

# Defines the log4j pattern used for all file appenders (used by the scheduler for job/task log files
pa.log4j.file.appender.pattern=%d{ISO8601} %-5p [%c{1.}] %m%n

#-------------------------------------------------------
#---------------   AUTHENTICATION PROPERTIES   ------------------
#-------------------------------------------------------

# path to the Jaas configuration file which defines what modules are available for internal authentication
pa.rm.auth.jaas.path=config/authentication/jaas.config

# path to the private key file which is used to encrypt credentials for authentication
pa.rm.auth.privkey.path=config/authentication/keys/priv.key

# path to the public key file which is used to encrypt credentials for authentication
pa.rm.auth.pubkey.path=config/authentication/keys/pub.key

# LDAP Authentication configuration file path, used to set LDAP configuration properties
# If this file path is relative, the path is evaluated from the resource manager dir (ie application's root dir)
# with the variable defined below : pa.rm.home.
# else, (if the path is absolute) it is directly interpreted
pa.rm.ldap.config.path=config/authentication/ldap.cfg

# Keycloak Authentication configuration file path, used to set keycloak configuration properties
# If this file path is relative, the path is evaluated from the resource manager dir (ie application's root dir)
# with the variable defined below : pa.rm.home.
# else, (if the path is absolute) it is directly interpreted
pa.rm.keycloak.config.path=config/authentication/keycloak.cfg

# Multi-domain LDAP configuration
# a comma-separated list of the pair domain_name:configuration_path
# see comment above regarding file paths locations
# domain names must be lowercase and must also be configured in pa.rm.allowed.domains

# pa.rm.multi.ldap.config=domain1:config/authentication/ldap-domain1.cfg,domain2:config/authentication/ldap-domain2.cfg

# Login file name for file authentication method
# If this file path is relative, the path is evaluated from the resource manager dir (ie application's root dir)
# with the variable defined below : pa.rm.home.
# else, the path is absolute, so the path is directly interpreted
pa.rm.defaultloginfilename=config/authentication/login.cfg

# Group file name for file authentication method
# If this file path is relative, the path is evaluated from the resource manager dir (ie application's root dir)
# with the variable defined below : pa.rm.home.
# else, the path is absolute, so the path is directly interpreted
pa.rm.defaultgroupfilename=config/authentication/group.cfg

# Tenant file name for file authentication method
# If this file path is relative, the path is evaluated from the resource manager dir (ie application's root dir)
# with the variable defined below : pa.rm.home.
# else, the path is absolute, so the path is directly interpreted
pa.rm.defaulttenantfilename=config/authentication/tenant.cfg

# List of domain names that can be used during a login (can be a list of windows domain names or a list of tenants in Multi-LDAP configuration)
# This setting is interpreted by ProActive portals to show a selectable list of domains
# Domain names should be lowercase
# It is possible to allow both named domain and empty domain using the syntax pa.rm.allowed.domains=,domain1,domain2
# pa.rm.allowed.domains=domain1,domain2

#Property that define the method that have to be used for logging users to the resource manager
#It can be one of the following values :
#	- "RMFileLoginMethod" to use file login and group management
#	- "RMLDAPLoginMethod" to use LDAP login management
#	- "RMMultiLDAPLoginMethod" to use LDAP login management across multiple domains
#	- "RMPAMLoginMethod" to use PAM login management
#	- "RMKeycloakLoginMethod" to use Keycloak login management
pa.rm.authentication.loginMethod=RMFileLoginMethod

# Path to the rm credentials file for authentication
pa.rm.credentials=config/authentication/rm.cred

# Refresh time to reload the security policy file (security.java.policy-server) 
pa.rm.auth.policy.refreshperiod.seconds=30

#-------------------------------------------------------
#--------------   HIBERNATE PROPERTIES   ---------------
#-------------------------------------------------------
# Hibernate configuration file (relative to home directory)
pa.rm.db.hibernate.configuration=config/rm/database.properties

# Drop database before creating a new one
# If this value is true, the database will be dropped and then re-created
# If this value is false, database will be updated from the existing one.
pa.rm.db.hibernate.dropdb=false

# Drop only node sources from the data base
pa.rm.db.hibernate.dropdb.nodesources=false

#-------------------------------------------------------
#--------------   TOPOLOGY  PROPERTIES   ---------------
#-------------------------------------------------------
pa.rm.topology.enabled=true

# By default, the computation of distances between nodes is disabled,
# as it implies a very slow node acquisition time. Activate it only if mandatory
pa.rm.topology.distance.enabled=false
# Pings hosts using standard InetAddress.isReachable() method.
pa.rm.topology.pinger.class=org.ow2.proactive.resourcemanager.frontend.topology.pinging.HostsPinger
# Pings ProActive nodes using Node.getNumberOfActiveObjects().
#pa.rm.topology.pinger.class=org.ow2.proactive.resourcemanager.frontend.topology.pinging.NodesPinger

# Location of selection scripts' logs (comment to disable logging to separate files).
# Can be an absolute path or a path relative to the resource manager home.
# If you are interested in disabling all outputs to 'logs/jobs' you must
# also have a look at the property 'pa.scheduler.job.logs.location' in 'PROACTIVE_HOME/config/scheduler/settings.ini'
# Please note that disabling Job logging will prevent Jobs and Tasks Server logs to be retrieved
# from the REST API and thus the Scheduler portal.
pa.rm.logs.selection.location=logs/jobs/

# Size limit for selection scripts' logs in bytes
pa.rm.logs.selection.max.size=10MB

21.3. Network Properties

Configuration files related to network properties for the Scheduler and nodes are located respectively in:

PROACTIVE_HOME/config/network/server.ini
PROACTIVE_HOME/config/network/node.ini

If you are using ProActive agents, then the configuration file location differs depending of the Operating System:

On Unix OS, the default location is /opt/proactive-node/config/network/node.ini
On Windows OS, the default one is C:\Program Files (x86)\ProActiveAgent\schedworker\config\network\node.ini.

21.3.1. Common Network Properties

The protocol to use by the Server and the nodes can be configured by setting a value to the property proactive.communication.protocol. It represents the protocol used to export objects on remote JVMs. At this stage, several protocols are supported: PNP (pnp), PNP over SSL (pnps), ProActive Message Routing (pamr).

The Scheduler is only able to bind to one and only one address. Usually, this limitation is not seen by the user and no special configuration is required. The Scheduler tries to use the most suitable network address available. But sometimes, the Scheduler fails to elect the right IP address or the user wants to use a given IP address. In such case, you can specify the IP address to use by using theses properties: proactive.hostname, proactive.net.interface, proactive.net.netmask, proactive.net.nolocal, proactive.net.noprivate.

IPv6 can be enabled by setting the proactive.net.disableIPv6 property to false . By default, the Scheduler does not use IPv6 addresses.

If none of the proactive.hostname, proactive.net.interface, proactive.net.netmask, proactive.net.nolocal , proactive.net.noprivate properties is defined, then the following algorithm is used to elect an IP address:

If a public IP address is available, then use it. If several ones are available, one is randomly chosen.
If a private IP address is available, then use it. If several ones are available, one is randomly chosen.
If a loopback IP address is available, then use it. If several ones are available, one is randomly chosen.
If no IP address is available at all, then the runtime exits with an error message.
If proactive.hostname is set, then the value returned by InetAddress.getByName(proactive.hostname) is elected. If no IP address is found, then the runtime exits with an error message.

If proactive.hostname is not set, and at least one of the proactive.net.interface , proactive.net.netmask , proactive.net.nolocal , proactive.net.noprivate is set, then one of the addresses matching all the requirements is elected. Requirements are:

If proactive.net.interface is set, then the IP address must be bound to the given network interface.
If proactive.net.netmask is set, then the IP address must match the given netmask.
If proactive.net.nolocal is set, then the IP address must not be a loopback address.
If proactive.net.noprivate is set, then the IP address must not be a private address.
proactive.useIPaddress: If set to true, IP addresses will be used instead of machines names. This property is particularly useful to deal with sites that do not host a DNS.
proactive.hostname: When this property is set, the host name on which the JVM is started is given by the value of the property. This property is particularly useful to deal with machines with two network interfaces.

21.3.2. PNP Protocol Properties

PNP allows the following options:

proactive.pnp.port: The TCP port to bind to. If not set PNP uses a random free port. If the specified TCP port is already used, PNP will not start and an error message is displayed.
proactive.pnp.default_heartbeat: PNP uses heartbeat messages to monitor the TCP socket and discover network failures. This value determines how long PNP will wait before the connection is considered broken. Heartbeat messages are usually sent every default_heartbeat/2 ms. This value is a trade-off between fast error discovery and network overhead. The default value is 30000 ms. Setting this value to 0 disables the heartbeat mechanism and client will not be advertised of network failure before the TCP timeout (which can be really long).
proactive.pnp.idle_timeout: PNP channels are closed when unused to free system resources. Establishing a TCP connection is costly (at least 3 RTT) so PNP connections are not closed immediately but after a grace time. By default the grace time is 60 000 ms. Setting this value to 0 disables the autoclosing mechanism, connections are kept open forever.

21.3.3. PNP over SSL Properties

PNPS support the same options as PNP (in its own option name space) plus some SSL specific options:

proactive.pnps.port: same as proactive.pnp.port
proactive.pnps.default_heartbeat: same as proactive.pnp.default_heartbeat
proactive.pnps.idle_timeout: same as proactive.pnp.idle_timeout
proactive.pnps.authenticate: By default, PNPS only ciphers the communication but does not authenticate nor the client nor the server. Setting this option to true enable client and server authentication. If set to true the option proactive.pnps.keystore must also be set.
proactive.pnps.keystore: Specify the keystore (containing the SSL private key) to use. The keystore must be of type PKCS12. If not set a private key is dynamically generated for this execution. Below is an example for creating a keystore by using the keytool binary that is shipped with Java:
```
$JAVA_HOME/bin/keytool -genkey -keyalg RSA -keystore keystore.jks \
       -validity 365 -keyalg RSA -keysize 2048 -storetype pkcs12
```
proactive.pnps.keystore.password: the password associated to the keystore used by PNPS.

When using the authentication and ciphering mode (or to speed up the initialization in ciphering only mode), the option proactive.pnps.keystore must be set and a keystore embedding the private SSL key must be generated. This keystore must be accessible to ProActive services but kept secret to others. The same applies to the configuration file that contains the keystore password defined with property proactive.pnps.keystore.password.

21.3.4. PAMR Protocol Properties

PAMR options are listed below:

proactive.pamr.router.address: The address of the router to use. Must be set if message routing is enabled. It can be FQDN or an IP address.
proactive.pamr.router.port: The port of the router to use. Must be set if message routing is enabled.
proactive.pamr.socketfactory: The Socket Factory to use by the message routing protocol
proactive.pamr.connect_timeout: Sockets used by the PAMR remote object factory connect to the remote server with a specified timeout value. A timeout of zero is interpreted as an infinite timeout. The connection will then block until established or an error occurs.
proactive.pamr.agent.id: This property can be set to obtain a given (and fixed) agent ID. This id must be declared in the router configuration and must be between 0 and 4096.
proactive.pamr.agent.magic_cookie: The Magic cookie to submit to the router. If proactive.pamr.agent.id is set, then this property must also be set to be able to use a reserved agent ID.

PAMR over SSH protocol properties

To enable PAMR over SSH, ProActive nodes hosts should be able to SSH to router’s host without password, using SSH keys.

proactive.pamr.socketfactory: The underlying Socket factory, should be ssh to enable PAMR over SSH
proactive.pamrssh.port: SSH port to use when connecting to router’s host
proactive.pamrssh.username: username to use when connecting to router’s host
proactive.pamrssh.key_directory: directory when SSH keys can be found to access router’s host. For instance /home/login/.ssh
proactive.pamrssh.address: Correspond to the host that actually runs a PAMR router on the remote side of the SSH tunnel. It may be used to point to a different host than the SSH server itself - making the SSH server to act as a gateway. The parameter is also useful when the PAMR router is running inside a cloud based VM which doesn’t support its public IP address. In that case it can be used to enforce the usage of the remote loopback interface.

21.3.5. Enabling Several Communication Protocols

The next options are available to control multiprocol:

proactive.communication.additional_protocols: The set of protocol to use separated by commas.
proactive.communication.benchmark.parameter: This property is used pass parameters to the benchmark. This could be a duration time, a size, … This property is expressed as a String.
proactive.communication.protocols.order: A fixed order could be specified if protocol’s performance is known in advance and won’t change. This property explain a preferred order for a subset of protocols declared in the property proactive.communication.additional_protocols. If one of the specified protocol isn’t exposed, it is ignored. If there are protocols that are not declared in this property but which are exposed, they are used in the order choose by the benchmark mechanism.
```
Example:
Exposed protocols: http,pnp,rmi
Benchmark Order: rmi > pnp > http
Order: pnp
This will give the order of use: pnp > rmi > http
```
proactive.communication.protocols.order: A fixed order could be specified if protocol’s performance is known in advance and won’t change. This automatically disabled RemoteObject’s Benchmark.

21.4. REST API & Web Properties

REST API Properties are read when ProActive Scheduler is started therefore you need to restart it to apply changes.

The REST API documentation can be found at SCHEDULER_URL/rest/doc.

The REST API documentation for our try platform is available at https://try.activeeon.com/doc/rest/

The PROACTIVE_HOME/tools/configure-http command can be used to configure the main parameters of the web server such as the protocol (http/https) and port. See Configure Http Server Command.

The default configuration file is SCHEDULER_HOME/web/settings.ini.

### Web applications configuration ###

# web applications in dist/war are deployed by default
web.deploy=true

# the maximum number of threads in Jetty for parallel request processing
web.max_threads=400

# timeout on http requests, default to 2 minute, can be increased to handle long requests
web.idle_timeout=240000

# Maximum request and response header sizes. These values can be increased in case of HTTP 414 errors.
web.request_header_size=16384
web.response_header_size=16384

# Configuration of the quality of service filter.
web.qos.filter.enabled=true
# Context of the filter, defaults to the /rest microservice
web.qos.filter.context=/rest
# A comma-separated list of path specification, relative to the context microservice, on which a quality of service filter will be applied
# The path specification accepts wildcards.
web.qos.filter.paths=/node.jar,/node-amd-64.jar,/node-arm-v7.jar,/node-arm-v8.jar
# No more than web.qos.filter.max.requests requests can run in parallel at the same time for each element of the list.
# Pending requests will wait for web.idle_timeout milliseconds. After this timeout, an error 503 Service Unavailable with be triggered.
web.qos.filter.max.requests=2

# port to use to deploy web applications
web.http.port=8080

# define whether HTTP requests are redirected to HTTPS
# this property has effect only if web.https is enabled
web.redirect_http_to_https=false

# HTTPS/SSL configuration
web.https=false
web.https.port=8443

# WARNING: the following HTTPS default values are for testing purposes only!
# do not use them in production but create your own keystore, etc.

# path to keystore, can be absolute or relative to SCHEDULER_HOME
web.https.keystore=config/web/keystore
web.https.keystore.password=activeeon

# path to truststore, can be absolute or relative to SCHEDULER_HOME
#web.https.truststore=config/web/truststore
#web.https.truststore.password=activeeon

# define whether hostname checking is performed or not when HTTPS
# is used to communicate with the REST API
#web.https.allow_any_hostname=true

# define whether all kind of certificates (e,g. self-signed) are allowed
# or not when HTTPS is used to communicate with the REST API
#web.https.allow_any_certificate=true

### additional ssl-related properties ###
# See https://www.eclipse.org/jetty/documentation/jetty-9/index.html#configuring-sslcontextfactory
web.https.protocols.included=TLSv1.2,TLSv1.3
# web.https.protocols.excluded=SSLv2Hello,SSLv3
# web.https.cyphers.included.add=
web.https.cyphers.excluded.add=TLS_DHE.*,TLS_EDH.*
# web.https.renegotiation.allowed=true
# web.https.secure.random.algorithm=
# web.https.key.factory.algorithm=SunX509
# web.https.trust.factory.algorithm=SunX509
# web.https.max.cert.path=
# web.https.cert.alias=
# web.https.enable.crldp=true
# web.https.crl.path=
# web.https.enable.ocsp=true
# web.https.ocsp.responder.url=
# web.https.session.caching=true
# web.https.session.cache.size=
# web.https.session.timeout=

### additional http header-related properties ###
### (in order to disable one setting, uncomment it and set it empty) ###
# web.x_frame_options=SAMEORIGIN
# web.x_xss_protection=1
# web.x_content_type_options=nosniff
# web.strict_transport_security=max-age=63072000; includeSubDomains; preload
# web.expect_ct=
# web.referrer_policy=strict-origin-when-cross-origin

# Optional configuration that will enable a Content-Security-Policy or Content-Security-Policy-Report-Only response headers
# see https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
# web.content.security.policy=default-src 'self' example.com *.example.com
# web.content.security.policy.report.only=default-src 'self' example.com *.example.com

# Can be used when the server is receiving http requests relative to a base path. In that case incoming requests will be rewritten
# e.g. /base/path/resource/ will be rewritten to /resource/
# web.base.path=/base/path

# Enable PCA Proxy rewriter to route requests coming from proxyfied applications
web.pca.proxy.rewrite.enabled=true

# PCA Proxy cache size to store referer information
web.pca.proxy.rewrite.referer.cache.size=10000

# Paths that will be excluded from rewrite rules in the PCA proxy, this is to protect ProActive portals from
# being redirected by the rules, which would make them unreachable.
web.pca.proxy.rewrite.excluded.paths=/automation-dashboard/,/studio/,/scheduler/,/rm/
web.pca.proxy.rewrite.excluded.paths.exact=/automation-dashboard,/studio,/scheduler,/rm

# Uncomment and set the following settings if resource downloading must pass through a proxy
#resource.downloader.proxy=127.0.0.1
#resource.downloader.proxy.port=8080
#resource.downloader.proxy.scheme=http

### REST API configuration ###

# will be set by JettyStarter, you will need to set it if you run REST server in standalone mode
#scheduler.url=rmi://localhost:1099

# scheduler user that is used as cache
scheduler.cache.login=watcher
scheduler.cache.password=w_pwd
#scheduler.cache.credential=

# cache refresh rate in ms
rm.cache.refreshrate=3500

# will be set by JettyStarter, you will need to set it if you run REST server in standalone mode
#rm.url=rmi://localhost:1099

# rm user that is used as cache
rm.cache.login=watcher
rm.cache.password=w_pwd
rm.cache.credential=

scheduler.logforwardingservice.provider=org.ow2.proactive.scheduler.common.util.logforwarder.providers.SocketBasedForwardingProvider

#### noVNC integration ####

# enable or disable websocket proxy (true or false)
novnc.enabled=false
# port used by websocket proxy (integer)
novnc.port=5900
# security configuration SSL (ON or OFF or REQUIRED)
novnc.secured=ON
# security keystore for SSL
# to create one for development: keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass password -validity 360 -keysize 2048
novnc.keystore=config/web/keystore
# security keystore password
novnc.password=activeeon
# security keystore key password
novnc.keypassword=activeeon

#  Url of the NoVNC proxy, created automatically when the NoBNC proxy is started.
# the property can be defined manually if the ProActive server is behind a reverse proxy
# novnc.url=

studio.workflows.user.dir=data/defaultuser/

#### Job Planner REST URL
jp.url=http://localhost:8080/job-planner/planned_jobs

# path to the jetty log file, disabled by default as retention is broken in the version of jetty we are using
#jetty.log.file=./logs/jetty-yyyy_mm_dd.request.log

# retention period (days) for jetty logs
jetty.log.retain.days=5

# session cleaning period in seconds, default to 5 minutes
session.cleaning.period=300

# session timeout in seconds, default to one hour
session.timeout=3600

21.5. Catalog Properties

Catalog Properties are read from its WAR file.

The default configuration file is located at WEB-INF/classes/application.properties. It can be extended with additional Spring properties.

# Configure logging level
logging.level.org.hibernate=warn
logging.level.org.hibernate.SQL=off
logging.level.org.ow2.proactive.catalog=info
logging.level.org.springframework.web=info
# warning, some loggers levels can only be changed in the log4j2.xml configuration


# Embedded server configuration
server.compression.enabled=true
server.contextPath=/

######################
#    HIKARI & JPA    #
######################

# Hibernate ddl auto (create, create-drop, update)
spring.jpa.hibernate.ddl-auto=update

spring.jmx.unique-names=true
spring.datasource.pool-name=catalog
spring.jmx.default-domain=catalog

spring.datasource.type=com.zaxxer.hikari.HikariDataSource
# The classname of a custom org.hibernate.connection.ConnectionProvider which provides JDBC connections to Hibernate
spring.jpa.hibernate.connection.provider_class=org.hibernate.hikaricp.internal.HikariCPConnectionProvider
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl

# JDBC connection pool configuration
# https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby
spring.datasource.connection-timeout=120000
spring.datasource.maximum-pool-size=40
spring.datasource.transaction-isolation=TRANSACTION_READ_COMMITTED
spring.datasource.leak-detection-threshold=600000
spring.datasource.validation-timeout=20000

# Enable Hibernate's automatic session context management
spring.jpa.properties.hibernate.current_session_context_class=thread

# Prevent warning about deprecated naming strategy
# https://github.com/spring-projects/spring-boot/issues/2763
# Should be changed once Spring Boot 1.4 is used
spring.jpa.properties.hibernate.implicit_naming_strategy=org.hibernate.boot.model.naming.ImplicitNamingStrategyJpaCompliantImpl
spring.jpa.properties.hibernate.ejb.naming_strategy_delegator=
spring.jpa.properties.hibernate.id.new_generator_mappings=false

# Show or not log for each sql query
spring.jpa.show-sql=false

##############
# DATASOURCE #
##############

# The default settings are using hsqldb
######################## HSQL DB #######################################################
spring.datasource.driverClassName=org.hsqldb.jdbc.JDBCDriver
spring.datasource.url=jdbc:hsqldb:hsql://localhost:9001/catalog
spring.jpa.database-platform=org.hibernate.dialect.HSQLDialect

spring.datasource.username=catalog
# Use tools/encrypt to create an encrypted password
spring.datasource.password=

######################## MYSQL DB ########################################################
#spring.datasource.driverClassName=com.mysql.jdbc.Driver
#spring.datasource.url=jdbc:mysql://localhost:8008/catalog
#spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect

#spring.datasource.username=user
#spring.datasource.password=user

######################## ORACLE DB  version=12.1.0.2 #####################################
#spring.datasource.driverClassName=oracle.jdbc.driver.OracleDriver
#spring.datasource.url=jdbc:oracle:thin:@localhost:8000:xe
#spring.jpa.database-platform=org.hibernate.dialect.Oracle12cDialect

#spring.datasource.username=system
#spring.datasource.password=oracle

######################### PostgreSQL #####################################################
#spring.datasource.driverClassName=org.postgresql.Driver
#spring.datasource.url=jdbc:postgresql://localhost:5000/catalog
#spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect

#spring.datasource.username=admin
#spring.datasource.password=admin

######################### SqlServer #####################################################
#spring.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver
#spring.datasource.url=jdbc:sqlserver://sqlserver-proactive.database.windows.net:1433;database=catalog;
#spring.jpa.database-platform=org.hibernate.dialect.SQLServer2012Dialect

#spring.datasource.username=activeeon@sqlserver-proactive
#spring.datasource.password=xxxxxxx

##########################################################################################

# Disable Spring banner
spring.main.banner_mode=off


pa.scheduler.url=http://localhost:8080
# Used to perform authentication since identity service is not yet available
pa.scheduler.rest.url=${pa.scheduler.url}/rest

# Used to perform filtering based on job-planner associations status
pa.job-planner.rest.url=${pa.scheduler.url}/job-planner

pa.job-planner.planned_objects.path=/planned_jobs/buckets
pa.job-planner.planned_object.status.path=/planned_jobs/buckets/{bucketName}/{objectName}

# duration of job-planner association cache in seconds
pa.job-planner.cache.timeout=60

# Separator used in kind string, like workflow/pca
kind.separator=/

# Optional catalog security features
pa.catalog.security.required.sessionid=true

# Optional ttf fonts absolute paths to use when generating the pdf report. This is required when catalog objects contains Asian characters
pa.catalog.pdf.report.ttf.font.path=
pa.catalog.pdf.report.ttf.font.bold.path=
pa.catalog.pdf.report.ttf.font.italic.path=
pa.catalog.pdf.report.ttf.font.bold.italic.path=

# Session id cache timeout value in minutes
pa.catalog.sessionId.timeout.minutes = 1

# the maximum number of items that can be used in a SQL IN expression (default to Oracle limit)
pa.catalog.db.items.max.size=1000

21.6. Scheduler Portal Properties

Scheduler Portal Properties are read when Portal main page is loaded.

The Configuration file named scheduler-portal-display.conf is located in the folder $PROACTIVE_HOME/config/portal/.

21.6.1. The list of Columns property

It is possible to specify a list of Variables or Generic Information to display in new columns of the Execution List table (Job Centric View). The defined columns will be added after the default columns of the view, following the order given in the configuration file. The list of columns is defined as a JSON array, for example:

execution-list-extra-columns: [{ \
	"title": "start at", \
	"information": { \
		"type": "generic-information", \
		"key": "START_AT"}, \
	"hide": false }, \
	{ \
	"title": "My var", \
	"information": { \
		"type": "variable", \
		"key": "MY_VAR"}, \
	"hide": false }]

The property execution-list-extra-columns contains the JSON array. If the array is written on several lines, each line except the last one should end with the caracter \. Each element of the array should contain the following fields:

title: the header of the column containing the value of the Variable or Generic Information
information: the information about the value displayed in the column. This field contains 2 fields:
- type: the type, either variable or generic-information
- key: the key of the value in the Variable or Generic Information map
hide: whether the column should be hidden by default

21.7. Command Line

The ProActive client allows to interact with the Scheduler and Resource Manager. The client has an interactive mode started if you do not provide any command.

The client usage is also available using the -h parameter as shown below:

$ PROACTIVE_HOME/bin/proactive-client -h

21.7.1. Command Line Examples

Deploy ProActive Nodes

In non-interactive mode

$ PROACTIVE_HOME/bin/proactive-client -cn 'moreLocalNodes' -infrastructure 'org.ow2.proactive.resourcemanager.nodesource.infrastructure.LocalInfrastructure' './config/authentication/rm.cred' 4 60000 '' -policy org.ow2.proactive.resourcemanager.nodesource.policy.StaticPolicy 'ALL' 'ALL'

In interactive mode

$ PROACTIVE_HOME/bin/proactive-client
> createns( 'moreLocalNodes', ['org.ow2.proactive.resourcemanager.nodesource.infrastructure.LocalInfrastructure', './config/authentication/rm.cred', 4, 60000, ''], ['org.ow2.proactive.resourcemanager.nodesource.policy.StaticPolicy', 'ALL', 'ALL'])

Install ProActive packages

To install a ProActive package, you can use the ProActive CLI by providing a path to your package. It can be a local directory, a ZIP file or can be a URL to a web directory, a direct-download ZIP file, a GitHub repository or ZIP or a directory inside a GitHub repository. Please note that URL forwarding is supported.

In non-interactive mode

To install a package located in a local directory or ZIP

$ PROACTIVE_HOME/bin/proactive-client -pkg /Path/To/Local/Package/Directory/Or/Zip

To install a package located in a web folder (Supports only Apache Tomcat directory listing)

$ PROACTIVE_HOME/bin/proactive-client -pkg http://example.com/installablePackageDirectory/

To install a package with a direct download ZIP URL:

$ PROACTIVE_HOME/bin/proactive-client -pkg https://s3.eu-west-2.amazonaws.com/activeeon-public/proactive-packages/package-example.zip

To install a package located in a GitHub repository (either in the repository root or in a sub-folder within a repository):

$ PROACTIVE_HOME/bin/proactive-client -pkg https://github.com/ow2-proactive/hub-packages/tree/master/package-example

In interactive mode

To install a package located in any of the aforementioned possible locations

$ PROACTIVE_HOME/bin/proactive-client
> installpackage(PackagePathOrURL)

21.8. Cloud Node Source Infrastructure Startup Script

Most cloud node source infrastructures allows the user to customize the VM startup script. To run the ProActive nodes, this script is also expected to download and install the Java Runtime Environment, then download and and execute ProActive node.jar, if they are not already provided by the VM image. The script can use the following arguments:

%nodeJarUrl%: the value of the infrastructure parameter nodeJarURL, it specifies the full URL path for downloading node.jar file.
%protocol%: the network protocol (pnp or pamr) to be used by the ProActive Node for incoming communications.
%jythonPath%: the path of jython library on the AWS instances. Its default value is /tmp/node/lib/jython-standalone-2.7.0.jar/Lib.
%rmHostname%: the value of the infrastructure parameter rmHostname, it specifies the hostname or refers an IP address of the Resource Manager which is accessible from deployed nodes.
%instanceIdNodeProperty%: it specifies the infrastructure use which property to identify the instance. It could be instanceId or instanceTag.
%instanceId%: the instance identification value corresponding %instanceIdNodeProperty%.
%rmUrl%: the URL to access the Resource Manager from the nodes.
%nodeSourceName%: the name of the node source.
%nodeNamingOption%: the Java command option specifying the node naming provided by the infrastructure. It’s expected to be added in the Java command of starting node.jar.
%credentials%: the credential of the Proactive Node required for Resource Manager authentication.
%numberOfNodesPerInstance%: the value of the infrastructure parameter numberOfNodesPerInstance, it specifies how much nodes should be run on each instance.
%additionalProperties%: the value of the infrastructure parameter additionalProperties, it’s supposed to be added in the Java command of starting node.jar

These arguments will be interpreted by cloud node source Infrastructure later to replace these arguments by its effective value before the script execution.

Here is an example startup script for Linux OS images which installed wget (e.g., Debian, Ubuntu):

mkdir -p /tmp/node && cd /tmp/node
if ! type -p jre/bin/java; then wget -nv -N https://s3.amazonaws.com/ci-materials/Latest_jre/jre-8u382b05-linux-x64.tar.gz; tar -xf jre-8u382b05-linux-x64.tar.gz; mv jre1.8.0_382b05/ jre; fi
wget -nv %nodeJarUrl%
nohup jre/bin/java -jar node.jar -Dproactive.communication.protocol=%protocol% -Dpython.path=%jythonPath% -Dproactive.pamr.router.address=%rmHostname% -D%instanceIdNodeProperty%=%instanceId% -r %rmUrl% -s %nodeSourceName% %nodeNamingOption% -v %credentials% -w %numberOfNodesPerInstance% %additionalProperties% &

An example startup script for Linux OS images which installed curl instead of wget (e.g., CentOS, RHEL):

mkdir -p /tmp/node && cd /tmp/node
if ! type -p jre/bin/java; then curl -sL https://s3.amazonaws.com/ci-materials/Latest_jre/jre-8u382b05-linux-x64.tar.gz --output jre.tar.gz; tar -xf jre.tar.gz; mv jre1.8.0_382b05/ jre; fi
curl -sL %nodeJarUrl% --output node.jar
nohup jre/bin/java -jar node.jar -Dproactive.communication.protocol=%protocol% -Dpython.path=%jythonPath% -Dproactive.pamr.router.address=%rmHostname% -D%instanceIdNodeProperty%=%instanceId% -r %rmUrl% -s %nodeSourceName% %nodeNamingOption% -v %credentials% -w %numberOfNodesPerInstance% %additionalProperties% &

An example startup script for Windows OS images:

$download=New-Object System.Net.WebClient;
$download.DownloadFile('http://javadl.oracle.com/webapps/download/AutoDL?BundleId=238729_478a62b7d4e34b78b671c754eaaf38ab', 'c:\jreInstall.exe');
$procInstall=Start-Process -FilePath 'c:\jreInstall.exe' -ArgumentList '/s REBOOT=ReallySuppress INSTALLDIR=c:\jre' -Wait -PassThru;
$procInstall.waitForExit();
$download.DownloadFile('%nodeJarUrl%', 'c:\node.jar');
Start-Process -NoNewWindow 'c:\jre\bin\java' -ArgumentList '-jar', 'c:\node.jar', '-Dproactive.communication.protocol=%protocol%', '-Dproactive.pamr.router.address=%rmHostname%', '-D%instanceIdNodeProperty%=%instanceId%', '-r', '%rmUrl%', '-s', '%nodeSourceName%', '-v', '%credentials%', '-w', '%numberOfNodesPerInstance%', '%additionalProperties%'

Legal notice

For more information, please contact contact@activeeon.com.

Version

ProActive Workflows & Scheduling — Admin Guide

PWS Modules