ProActive Workflows & Scheduling (PWS)

  • PWS User Guide

     (Workflows, Workload automation, Jobs, Tasks, Catalog, Resource Management, Big Data/ETL, …​)

  • PWS Modules

  • PWS Admin Guide

     (Installation, Infrastructure & Nodes setup, Agents,…​)

1. Main principles of PCA

ProActive Cloud Automation (PCA) allows to automate service deployment, together with their life-cycle management. Services are instantiated by workflows (executed as a Job by the Scheduler), and related workflows allow to move instances from a state to another one.

At any point in time, each Service Instance has a specific State (RUNNING, ERROR, FINISHED, etc.).

Attached to each Service Instance, PCA service stores several information such as: Service Instance Id, Service Id, Service Instance State, the ordered list of Jobs executed for the Service, a set of variables with their values (a map that includes for instance the service endpoint), etc.

1.1. Definitions and Concepts

The following terms are used as ProActive Cloud Automation (PCA) definitions and main concepts:

Service

A set of workflows representing Actions and having the same Service ID.

PCA introduction
Activated Service or Service Instance

An Activated Service is an Instance of a Service. It is characterized by a Service ID and an Instance ID.

PCA activated services
Action

An Action will launch a workflow on a Service Instance modifying its State. An action is characterized by <a Service ID + a workflow ID + a state>. The Service ID and the State are respectively named « pca.service.id » and « pca.states » in the list of the workflow Generic Information.

An Action goes from an initial State and conducts to a target State after the Transition is completed. As an Action must be deterministic, for an given initial State it can conduct to a unique target state.

There are three specific kinds of Actions:

  • Activation: to start a Service Instance (from the State VOID to a defined State)

  • Finishing: to stop a Service Instance (from a defined State to the State FINISHED)

  • Killing: PCA is trying to force a Service Instance to terminate (from a defined State to the State KILLED)

States

The specific States that are pre-defined in ProActive Cloud Automation are VOID, FINISHED, ERROR, KILLED.

  • VOID Implicit State before a Service is activated

  • FINISHED Implicit State after a Service is terminated/no longer activated

  • ERROR Implicit State after something wrong happened

  • KILLED When PCA abnormally forces an Activated Service to terminate.

Please note that users are free to define their own custom States (e.g. RUNNING, SUSPENDED, …​)
Endpoints

Endpoints are a list of (Key,Value) defined in the Variables e.g. [(Swarm, <smarmendpoint>), (HDFS, <hdfs-endpoint>), (Spark, <spark-endpoint>)]. The endpoint URL should contain the protocol, e.g. www.activeeon.com will not work, but https://www.activeeon.com will.

To run multiple PCA services that share an endpoint, the same ENDPOINT_ID value should be given to all of the PCA services that, when started, require this variable.
Service_Name, Infrastructure_Name

Service_Name, Infrastructure_Name are Service specific. If a Service wants/needs them, they can be stored in the data associated to the service to give a name to the service, to identify the infrastructure where the Service Instance will be deployed.

1.2. Service Transitions

This paragraph describes how an Activated Service goes from one State to another. For instance how the Activated Service S1 transitions from the State RUNNING to the State SUSPENDED.

States and Transition Syntax

The States as well as the Transitions from one State to another are provided in workflows Generic Information named « pca.states » with the following syntax:

pca.states=(a,b) or pca.states=(a,b)(c,d)…(e,f)

meaning that the current workflow, for the current Service ID, when activated in State a, will transition the Service Instance in State b; or if activated in State c, the target state will be d.

An Activation Workflow (allowing to start a Service Instance) will have pca.states=(VOID,x).

A Finishing Workflow (allowing to terminate a Service Instance) can have for instance pca.states=(RUNNING,FINISHED)(SUSPENDED,FINISHED)

Service Transitions

Actions that can be applied on Activated Services depend on their State. If they are in the State « CUR_STATE », all the workflow actions that have the Generic Information named « pca.states » equal to (CUR_STATE, xyz_state) or (ALL, xyz_state) can be executed. After executing the workflow actions, if everything goes right, the Activated Services will be in the state « xyz_state ».

Transitioning

An Activated Service can be in a specific Transitioning State. In that case the State is labeled with the string "INITIAL_STATE -→ DESTINATION_STATE", for instance "RUNNING -→ FINISHED". No new Action can be started on a Transitioning Activated Service, except the Kill action.

  • When a PCA Workflow deploys a Service, it keeps the Nodes needed to execute the service Busy (with a Sleep on one or several Tasks). One can use those Tasks to stream output (log) of the Service Instance such that it does appear in the Scheduler. We add to that stream all outputs of the Service Instance (e.g. getting all the logs from Swarm, HDFS, Spark at the same time in the Scheduler portal). The Nodes being used can be taken from an existing Node Source (e.g. ServiceNodeSource), or launched specifically at the beginning of the deployment (e.g. starting a NodeSource on Azure Cloud).

Please note that the Service must contain at least two workflows: a start workflow which will create the Service Instance and a finish workflow which will delete the Service Instance. Moreover the start workflow must have a task which will reserve the nodes. One of the strategy is to create a task with a sleep that will loop until an action change the Service Instance State.
PCA void to running
End of Transitioning

There is no implicit detection of the end of a Transitioning State. A workflow that carries on a transition has to inform Cloud Automation Service that the transition is actually successfully finished. Each service workflows will contain a task that will update the Service Instance State after finishing its Actions.

PCA running to finished
Workflows for Service Deployment

A workflow that deploys a Service Instance will wait (Sleep Loop) for a State change and terminate in a correct manner. There are 2 possibilities:

  • Wait until the beginning of the Transition: e.g. no longer State RUNNING. (The state can be "from RUNNING to FINISHED".)

  • Wait until the end of the Transition: e.g. FINISHED (The state cannot be Transitioning, neither RUNNING, but has to have reached rather "FINISHED".) This is the preferred choice to avoid the service nodes being given to another Task/Service before a Delete Action actually remove the Service Instance artifacts

Submission of a Workflow by PCA

To execute an Action, PCA will submit a Workflow to the scheduler, using as Variable Map all the Variables that have been collected and updated so far for that Service. A first Task in the Wf is expected to collect all the values, even those not explicitly being Wf Variables.

Information of Service Instance

A Cloud Automation Workflow and third parties can get access to information about a Service Instance through Cloud Automation service. It allows for instance a Delete Action to get the URLs of the Nodes where the Service Instance is deployed, or the Docker Containers that are being used and need to be killed to terminate the Service Instance. E.g. A Cloud Automation workflow that starts Docker containers on the nodes is expected to store the Docker Ids in Cloud Automation Service for that Instance.

Variables stored in a Service Instance

From the beginning of its Activation, when an Action is executed, the Variables/Values of the action Workflows are stored and append within the PCA service for that service instance. When a Variable already existed in the list, its value is updated with the last workflow value (AddAll).

Workflows executed by a Service Instance

From the beginning of its Activation, until it reaches the FINISH state, a Service Instance has in PCA service (and accessible through the API) the ordered list of Job Id that has been executed as Action on it. Besides the Job ids, Cloud Automation service does not duplicate and store any information about the Job. The Cloud Automation portal will get those information from the Scheduler.

Endpoints

Endpoints are a list of Key/Value defined in the Variables e.g. [(Swarm, <smarmendpoint>), (HDFS, <hdfs-endpoint>), (Spark, <spark-endpoint>)]

2. PCA Portal

2.1. Overall View

PCA overall view

The ProActive Cloud Automation (PCA) Portal has a view with four main sections:

  • The Cloud Automation workflows

  • The list of activated Services Snstances

  • The list of finished or killed Services Instances

  • The monitoring items

2.2. Detailed View

PCA portail

The ProActive Cloud Automation (PCA) Portal allows to have access on several tabs.

  • Service Activation shows the list of workflow that allows activating a Service Instance. Only creation workflows with VOID as origin State are displayed

PCA service activation
  • Full Services View gives all Cloud Automation workflows sorted by Service. Only creation workflows can be submitted from this tab to create Service Instances.

PCA full service view
  • Service Jobs shows all jobs related to activated Service Instances grouped by Service Instances.

PCA service jobs
  • Activated Services gives the list of Services that are not in State ‘FINISHED’ or ‘KILLED’.

PCA activated services
  • Finished Services shows the list of Services that are in State ‘FINISHED’ and ‘KILLED’. The CLEAN action removes the Service Instance from the database.

PCA finished services
  • Monitoring Items

PCA monitoring

2.3. Usage examples

In this section, we show how to start a PCA service and how to use actions that manage its life cycle. To do that, we choose MongoDB and Spark/Hdfs PCA services as running examples.

2.3.1. MongoDB

  • Start MongoDB

To start the mongoDB service, you have to go to the cloud automation section in the automation dashboard portal and choose MongoDB service from the service activation tab. At this point, you can provide values for variables that help configure this service then press execute.

start mongo

Once the MongoDB service is deployed, the state of the instance becomes RUNNING. At this stage, you can interact with the database provided by this PCA service. For this, you might need a host and a port number that can be retrieved from the endpoint displayed in the info window as shown bellow:

endpoint mongo

Two actions are possible at this step; pause or finish the instance.

actions mongo
  • Pause MongoDB

The running instance can be paused by executing the pause action when the instance is in the RUNNING state. The pause action suspends only processes in the specified containers to release resources without losing data/container.

pause mongo

From the pause state, two actions are possible; resume or finish the instance.

actions1 mongo
  • Resume MongoDB

The resume action allows to restart a paused instance while preserving previous state of the database. The instance is the RUNNING state.

resume mongo
  • Finish MongoDB

When the user does not need this PCA service anymore, he can delete it to release resources using the finish action. This action can be applied when the instance is in the RUNNING or PAUSED state.

finish mongo

2.3.2. Spark/Hdfs

In this example, we show how to deploy through PCA Portal a HDFS and Spark platform that be used to run big data applications.

This deployment is done in several step

  • Reserve Node

This step allows to reserve the ressource to use to host the platform.

PCA reserve node
  • Deploy Swarm

After the node reservation, two actions are possible from the Actions tab. You an release the ressource or deploy swarm to have the network and cluster of docker containers.

PCA deploy swarm

Choose swarm and click on execute Action to deploy it.

PCA execute swarm1
PCA execute swarm2

At the end, the State of the Service Instance is SWARM_DEPLOYED. Three Actions are available at this step and you can access to the cluster of docker containers through the endpoint (consul_UI).

PCA execute swarm3

The next step is to deploy HDFS

  • Deploy HDFS

PCA execute hdfs

At the end, the State of the Service Instance is HDFS_DEPLOYED. Two Actions are available at this step.

PCA execute hdfs1

You can access to the HDFS interface or the cluster of docker containers through the endpoints (HDFS_UI, consul_UI).

PCA execute hdfs2

The next step is to deploy Spark.

  • Deploy Spark

PCA execute spark

You can access to HDFS or SPARK interfaces (HDFS_UI, SPARK_UI).

PCA execute spark1

3. Reference

In this section, we provide a quick reference about how to configure and start some PCA services from our Catalog.

3.1. Cassandra

This service allows to deploy through ProActive Cloud Automation (PCA) Portal a Cassandra Database server. The service can be started using the following variables.

Variables:

Table 1. Cassandra PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

cassandra-server

ENDPOINT_ID

Id of the service endpoint

Yes

String

cassandra-endpoint

ENV_VARS

List of the environment variables. Each environment variable should be preceded by -e.

No

String

e.g. -e CASSANDRA_CLUSTER_NAME="cluster1"

More details about the variable description can be found here.

3.2. Docker

This service allows to deploy through ProActive Cloud Automation (PCA) Portal any docker image. It serves as a generic template that can be used to create and start any docker image ON DEMAND simply by providing the following variables.

Variables:

Table 2. Docker PCA service variables

Variable name

Description

Required?

Type

Default/Examples

ENDPOINT_ID

Id of the service endpoint

Yes

String

docker-endpoint

DOCKER_IMAGE

Docker image name. It can include a tag as well.

Yes

String

e.g. postgres

DOCKER_PORT

The main image port. Please note that it will be forwarded to a random port that will be returned by this service.

Yes

Integer

e.g. 5432

DOCKER_CONTAINER

If you desire to stop this container and restart it later.

Yes

String

e.g. postgres-container

DOCKER_OPTIONS

options like environment variables, etc

No

String

e.g. -e POSTGRES_USER=mlos -e POSTGRES_PASSWORD=proactive -e POSTGRES_DB=activeeon

DOCKER_COMMAND

Password for the root user

No

String

e.g. -c "listen_addresses=0.0.0.0"

3.3. Elasticsearch

This service allows to deploy through ProActive Cloud Automation (PCA) Portal a Elasticsearch Database server. The service is started using the following variable.

Variables:

Table 3. Elasticsearch PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

elasticsearch-server

ENDPOINT_ID

Id of the service endpoint

Yes

String

elasticsearch-endpoint

More details about the variable description can be found here.

3.4. Jupyterlab

This service allows to deploy through ProActive Cloud Automation (PCA) Portal an instance of JupyterLab server (https://jupyterlab.readthedocs.io/en/stable/). The service is started using the following variable.

Variables:

Table 4. Jupyterlab PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

jupyterlab-server-$PA_JOB_ID

ENDPOINT_ID

Id of the service endpoint

Yes

String

jupyterlab-endpoint-$PA_JOB_ID

PROXYFIED

Allows access to the endpoint through Cloud Automation Http(s) Proxy

Yes

Boolean

false

HTTPS_ENABLED

Start JupyterLab in https mode (generates a self-signed certificate or use fullchain.pem and privkey.pem if present in the ProActive Global Space).

Yes

Boolean

false

SERVICE_PORT

Controls the port used to start the JupyterLab server. -1 for random port allocation. The port range will be chosen either from $PA_SCHEDULER_HOME/config/pca_services_port_range or /proc/sys/net/ipv4/ip_local_port_range

Yes

Integer

-1

JUPYTERLAB_TOKEN

Protects the JupyterLab service with token access. This variable must not be empty if the PROXYFIED=true mode is used.

No

String

/shared/$INSTANCE_NAME

ENGINE

Container engine

Yes

Choice(docker,singularity)

docker

DOCKER_IMAGE

Docker image used to start JupyterLab

Yes

String

activeeon/tensorboard

SINGULARITY_IMAGE_PATH

Location of the singularity image on the node file system (this path will be used to either store the singularity image or the image will be directly used if the file is present)

Yes

String

/tmp/tensorboard.sif

BUILD_IMAGE_IF_NOT_EXISTS

Pull and build the singularity image if the sif file is not present

Yes

Boolean

true

NATIVE_SCHEDULER

Name of the Native Scheduler node source to use (if the service must be deployed inside a cluster such as SLURM, LSF, etc)

No

String

Slurm

NATIVE_SCHEDULER_PARAMS

Parameters given to the native scheduler (Slurm, LSF, etc) while requesting a ProActive node used to deploy the PCA service.

No

String

--gpus=1

3.5. Kafka

This service allows to deploy through ProActive Cloud Automation (PCA) Portal an instance of Apache Kafka publish/subscribe system (https://kafka.apache.org/). The deployed instance consists in a single publish/subscribe broker that is based on Apache Zookeeper coordination server. The service is started using the following variables.

Variables:

Table 5. Kafka PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

kafka-server-1

ENDPOINT_ID

Id of the service endpoint

Yes

String

kafka-endpoint

ZOOKEEPER_INSTANCE_NAME

Name of the Zookeeper instance that coordinates the storm deployment.

Yes

String

zookeeper-server-1

ZOOKEEPER_SERVICE_ID

Zookeeper service identifier needed to start this service if it does not already exist.

Yes

String

Zookeeper

More details about the variable description can be found here.

3.6. MongoDB

This service allows to deploy through ProActive Cloud Automation (PCA) Portal a MongoDB Database server. The service can be started using the following variables.

Variables:

Table 6. MongoDB PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

mongodb-server-$PA_JOB_ID

ENDPOINT_ID

Id of the service endpoint

Yes

String

mongodb-endpoint-$PA_JOB_ID

USER

Username for the root user

No

String

e.g. my_user

PASSWORD

Password for the root user

No

String

e.g. my_password

SERVICE_PORT

Controls the port used to start the JupyterLab server. -1 for random port allocation. The port range will be chosen either from $PA_SCHEDULER_HOME/config/pca_services_port_range or /proc/sys/net/ipv4/ip_local_port_range

Yes

Integer

-1

ENGINE

Container engine

Yes

Choice(docker,singularity)

docker

DOCKER_IMAGE

Docker image used to start MongoDB

Yes

String

activeeon/mongo

SINGULARITY_IMAGE_PATH

Location of the singularity image on the node file system (this path will be used to either store the singularity image or the image will be directly used if the file is present)

Yes

String

/tmp/mongo_db.sif

BUILD_IMAGE_IF_NOT_EXISTS

Pull and build the singularity image if the sif file is not present

Yes

Boolean

true

MOUNT_DB_PATH

When using singularity, this folder will be used to store the database data

Yes

String

/tmp/mongodb-server-$PA_JOB_ID

NATIVE_SCHEDULER

Name of the Native Scheduler node source to use (if the service must be deployed inside a cluster such as SLURM, LSF, etc)

No

String

Slurm

NATIVE_SCHEDULER_PARAMS

Parameters given to the native scheduler (Slurm, LSF, etc) while requesting a ProActive node used to deploy the PCA service.

No

String

--gpus=1

Note that the USER and PASSWORD variables are used in junction. They should be either both entered or both blank. More details about the variable description can be found here.

3.7. MySQL

This service allows to deploy through ProActive Cloud Automation (PCA) Portal a MySQL Database server. The service can be started using the following variables.

Variables:

Table 7. MySQL PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

mysql-server

ENDPOINT_ID

Id of the service endpoint

Yes

String

mysql-endpoint

DATABASE

Name of a database to be created on start

No

String

e.g. my_database

USER

Username for the root user

No

String

e.g. my_user

PASSWORD

Password for the root user

No

String

e.g. my_password

Note that the USER and PASSWORD variables are used in junction. They should be either both entered or both blank. In addition, if DATABASE is assigned a value, then the defined USER will be granted superuser access (corresponding to GRANT ALL) to this database. More details about the variable description can be found here.

3.8. PostgreSQL

This service allows to deploy through ProActive Cloud Automation (PCA) Portal a PostgreSQL Database server. The service can be started using the following variables.

Variables:

Table 8. PostgreSQL PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

postgres-server

ENDPOINT_ID

Id of the service endpoint

Yes

String

postgresql-endpoint

DATABASE

Name of a database to be created on start

No

String

e.g. my_database

USER

Username for the root user. change it if you want a different root username . Default "postgres" is used if left empty.

No

String

e.g. my_user

PASSWORD

Password for the root user

Yes

String

e.g. my_password

More details about the variable description can be found here.

3.9. Storm

This service allows to deploy through ProActive Cloud Automation (PCA) Portal a cluster of Apache Storm stream processing system (http://storm.apache.org). The service is started using the following variables.

Variables:

Table 9. Storm PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

storm-cluster-1

ENDPOINT_ID

Id of the service endpoint

Yes

String

storm-gui

NUMBER_OF_SUPERVISORS

Number of worker nodes

Yes

Integer

2

ZOOKEEPER_INSTANCE_NAME

Name of the Zookeeper instance that coordinates the storm deployment.

Yes

String

zookeeper-server-1

ZOOKEEPER_SERVICE_ID

Zookeeper service identifier needed to start this service if it does not already exist.

Yes

String

Zookeeper

More details about the variable description can be found here.

3.10. Tensorboard

This service allows to deploy through ProActive Cloud Automation (PCA) Portal an instance of Tensorboard server (https://www.tensorflow.org/tensorboard). The service is started using the following variable.

Variables:

Table 10. Tensorboard PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

tensorboard-server-$PA_JOB_ID

ENDPOINT_ID

Id of the service endpoint

Yes

String

tensorboard-gui-$PA_JOB_ID

PROXYFIED

Allows access to the endpoint through Cloud Automation Http(s) Proxy

Yes

Boolean

false

SERVICE_PORT

Controls the port used to start the Tensorboard server. -1 for random port allocation. The port range will be chosen either from $PA_SCHEDULER_HOME/config/pca_services_port_range or /proc/sys/net/ipv4/ip_local_port_range

Yes

Integer

-1

MOUNT_LOG_PATH

Path on the ProActive node file system used to store Tensorboard data

Yes

String

/shared/$INSTANCE_NAME

ENGINE

Container engine

Yes

Choice(docker,singularity)

docker

DOCKER_IMAGE

Docker image used to start Tensorboard

Yes

String

activeeon/tensorboard

SINGULARITY_IMAGE_PATH

Location of the singularity image on the node file system (this path will be used to either store the singularity image or the image will be directly used if the file is present)

Yes

String

/tmp/tensorboard.sif

BUILD_IMAGE_IF_NOT_EXISTS

Pull and build the singularity image if the sif file is not present

Yes

Boolean

true

NATIVE_SCHEDULER

Name of the Native Scheduler node source to use (if the service must be deployed inside a cluster such as SLURM, LSF, etc)

No

String

Slurm

NATIVE_SCHEDULER_PARAMS

Parameters given to the native scheduler (Slurm, LSF, etc) while requesting a ProActive node used to deploy the PCA service.

No

String

--gpus=1

3.11. Visdom

This service allows to deploy through ProActive Cloud Automation (PCA) Portal an instance of Visdom server (https://github.com/facebookresearch/visdom). The service is started using the following variable.

Variables:

Table 11. Visdom PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

visdom-server-$PA_JOB_ID

ENDPOINT_ID

Id of the service endpoint

Yes

String

visdom-gui-$PA_JOB_ID

PROXYFIED

Allows access to the endpoint through Cloud Automation Http(s) Proxy

Yes

Boolean

false

SERVICE_PORT

Controls the port used to start the Visdom server. -1 for random port allocation. The port range will be chosen either from $PA_SCHEDULER_HOME/config/pca_services_port_range or /proc/sys/net/ipv4/ip_local_port_range

Yes

Integer

-1

VISDOM_ENABLE_LOGIN

Enable Visdom authentication

Yes

Boolean

false

VISDOM_USERNAME

Optional username used to authenticate with Visdom

No

String

admin

VISDOM_PASSWORD

Optional password used to authenticate with Visdom

No

String

some_password

ENGINE

Container engine

Yes

Choice(docker,singularity)

docker

DOCKER_IMAGE

Docker image used to start Visdom

Yes

String

activeeon/visdom_server

SINGULARITY_IMAGE_PATH

Location of the singularity image on the node file system (this path will be used to either store the singularity image or the image will be directly used if the file is present)

Yes

String

/tmp/visdom_server.sif

BUILD_IMAGE_IF_NOT_EXISTS

Pull and build the singularity image if the sif file is not present

Yes

Boolean

true

NATIVE_SCHEDULER

Name of the Native Scheduler node source to use (if the service must be deployed inside a cluster such as SLURM, LSF, etc)

No

String

Slurm

NATIVE_SCHEDULER_PARAMS

Parameters given to the native scheduler (Slurm, LSF, etc) while requesting a ProActive node used to deploy the PCA service.

No

String

--gpus=1

3.12. Model_Service

This service allows to deploy through ProActive Cloud Automation (PCA) Portal an instance of Model_Service.

More details about the related actions of this service can be found in the MLOS Doc.

The service is started using the following variable.

Variables:

Table 12. Model_Service PCA service variables

Variable name

Description

Required?

Type

Default/Examples

BUILD_IMAGE_IF_NOT_EXISTS

Pull or/and build the singularity image if the sif file is not present.

Yes

Boolean

false

DEBUG_ENABLED

If True, the user will be able to examine the stream of output results of each task.

Yes

Boolean

true

DOCKER_IMAGE

Specifies the name of the Docker image that will be used to run the different workflow tasks.

Yes

String

activeeon/model_as_a_service

DRIFT_ENABLED

True if a detector is needed to check for drifts in the input datasets compared to the training datasets.

Yes

Boolean

true

DRIFT_THRESHOLD

The level or point at which the data drift is detected and the user is notified.

Yes

Float

1.9

ENDPOINT_ID

The endpoint_id that will be used if PROXYFIED is set to True.

Yes

String

maas-gui-$PA_JOB_ID

ENGINE

Container engine.

Yes

List

docker

HTTPS_ENABLED

True if the protocol https is needed for the defined model-service.

Yes

Boolean

false

INSTANCE_NAME

The name of the service that will be deployed.

Yes

String

model-server-$PA_JOB_ID

NODE_NAME

The name of the node where the service will be deployed. If empty, the service will be deployed on an available node selected randomly.

No

String

Empty

PROXYFIED

True if a proxy is needed to protect the access to this model-service endpoint.

Yes

Boolean

false

PYTHON_ENTRYPOINT

This entry script starts the service and defines the different functions to deploy the model, scores the prediction requests based on the deployed model, and returns the results. This script is specific to your model. This file should be stored in the Catalog under the model_as_service_resources bucket. More information about this file can be found in the [Customize the Service] section.

Yes

String

ml_service

SERVICE_PORT

Controls the port used to start the Model Service from the Cloud Automation Portal. -1 for random port allocation.

Yes

Integer

-1

SINGULARITY_IMAGE_PATH

Location of the singularity image on the node file system (this path will be used to either store the singularity image or the image will be directly used if the file is present).

Yes

String

/tmp/model_service.sif

TRACE_ENABLED

True if the user wants to keep a trace on the different changes occurring in the service.

Yes

Boolean

true

YAML_FILE

A YAML file that describes the OpenAPI Specification ver. 2 (known as Swagger Spec) of the service. This file should be stored in the catalog under the model_as_service_resources bucket. More information about the structure of this file can be found in the section [Customize the Service].

Yes

String

ml_service-api

NATIVE_SCHEDULER

Name of the Native Scheduler node source to use (if the service must be deployed inside a cluster such as SLURM, LSF, etc)

No

String

Slurm

NATIVE_SCHEDULER_PARAMS

Parameters given to the native scheduler (Slurm, LSF, etc) while requesting a ProActive node used to deploy the PCA service.

No

String

--gpus=1

3.13. Zookeeper

This service allows to deploy through ProActive Cloud Automation (PCA) Portal an instance of Apache Zookeeper coordination server (https://zookeeper.apache.org/). The deployed instance consists in a single Zookeeper broker. The service is started using the following variable.

Variables:

Table 13. Zookeeper PCA service variables

Variable name

Description

Required?

Type

Default/Examples

INSTANCE_NAME

Service instance name

Yes

String

zookeeper-server-1

ENDPOINT_ID

Id of the service endpoint

Yes

String

zookeeper-endpoint

More details about the variable description can be found here.

Activeeon SAS, © 2007-2019. All Rights Reserved.

For more information, please contact contact@activeeon.com.