Dynamic resource configuration in Algoreus

• Background

• Solution

  • Configuring Compute Profile

  • Configuring Compute Resources

Users might want to create dynamic axons in Algoreus for improved reusability and simplified operations. This guide explains how to configure compute resources for axon execution at runtime.

---

Background

Batch Axon can be orchestrated using various engines like MapReduce or Spark. The resources for these engines (CPU and Memory) are typically configured during design time but can be altered during runtime by changing the settings in the Resources tab. Users can also alter the compute profile from the UI.

For dynamic Axons, these resources should be configured via runtime arguments. The following sections show you how to do this.

---

Solution

Configuring Compute Profile

Compute profile can be configured at runtime using the runtime argument (preferences). The value for the profile name should include the scope and profile name separated by a colon 'scope:profileName'.

For instance, to start the axon named 'BQMerge' on a profile called 'dp10' in the system scope:

Configuring Engine Resources Engine resources like CPU (cores) and memory can be configured using runtime arguments (preferences). To configure resources for Spark Driver and Executor, use the following options:

• 'task.driver.system.resources.memory' for configuring the memory for Spark Driver. Memory is set in Megabytes. For instance, setting 'task.driver.system.resources.memory' to 2048 sets the driver memory resources to 2 GB (2048 MB).

• 'task.driver.system.resources.cores' for configuring the CPU (cores) for Spark Driver. By default, the driver CPU is set to 1 core. For instance, setting 'task.driver.system.resources.cores' to 2 sets the driver cores to 2.

• 'task.executor.system.resources.memory' for configuring the memory for Spark Executors. Memory is set in Megabytes. For instance, setting 'task.executor.system.resources.memory' to 2048 sets the executor memory resources to 2 GB (2048 MB).

• 'task.executor.system.resources.cores' for configuring the CPU (cores) for Spark Executors. By default, the driver CPU (cores) is set to 1 core. For instance, setting 'task.executor.system.resources.cores' to 2 configures 2 cores for all executors.

Configuring Compute Resources

The following settings are available for configuring compute resources:

• 'system.profile.properties.serviceAccount' for specifying the service account for the cluster.

• 'system.profile.properties.masterNumNodes' for setting the number of master nodes.

• 'system.profile.properties.masterMemoryMB' for setting the memory per master node.

• 'system.profile.properties.masterCPUs' for setting the number of CPUs for the master.

• 'system.profile.properties.masterDiskGB' for setting the disk in GB per master node.

• 'system.profile.properties.workerNumNodes' for setting the number of worker nodes.

• 'system.profile.properties.workerMemoryMB' for setting the memory per worker node.

• 'system.profile.properties.workerCPUs' for setting the number of CPUs per worker node.

• 'system.profile.properties.workerDiskGB' for setting the disk in GB per worker node.

• 'system.profile.properties.stackdriverLoggingEnabled' set to true to enable Stackdriver logging for the Axons.

• 'system.profile.properties.stackdriverMonitoringEnabled' set to true to enable Stackdriver monitoring for the Axons.

• 'system.profile.properties.imageVersion' for configuring the image version.

• 'system.profile.properties.network' for configuring the network for the cluster.

---