A Distributed Execution Manager (DEM) runs the business logic of custom models, interacting with the database and with external databases and systems as required. DEMs also manage cloud and physical machines.

Each DEM instance acts in either a Worker role or in an Orchestrator role. The Worker role is responsible for running workflows. The Orchestrator role is responsible for monitoring DEM Worker instances, preprocessing workflows to run, and scheduling workflows.

The DEM Orchestrator performs these tasks.

Monitors the status of DEM Workers and ensures that if a Worker instance stops or loses its connection to the Model Manager, its workflows are put back in the queue for another DEM Worker to pick up.

Manages scheduled workflows by creating new workflow instances at the scheduled time.

Ensures that only one instance of a particular scheduled workflow is running at a given time.

Preprocesses workflows before they are run, including checking preconditions for workflows, used in the implementation of the RunOneOnly feature, and creating the workflow execution history.

One DEM Orchestrator instance is designated as the active Orchestrator that performs these tasks. Because the DEM Orchestrator is essential to run workflows, install at least one additional Orchestrator instance on a separate machine for redundancy. The additional DEM Orchestrator monitors the status of the active Orchestrator so that it can take over if the active Orchestrator goes offline.