Home
Turium Algoreus
Turium Algoreus
  • Turium Algoreus Documentation
    • Turium Algoreus Overview
      • How to Guides
        • Axons (Pipeline) User Guide
          • Algoreus Genesis
          • Algoreus Node
          • Steps for a simple batch Axon in Algoreus
          • Configuring Axon in Algoreus
          • Deploying an Axon in Algoreus
          • Running an Axon in Algoreus
          • Viewing and downloading logs in the Genesis in Algoreus
          • Scheduling an Axon in Algoreus
          • Reusable Axons in Algoreus
          • Using Triggers in Algoreus
          • Working with multiple versions of the same node in Algoreus
          • Modifying a draft Axon in Algoreus
          • Editing a deployed Axon in Algoreus
          • Duplicating An Axon in Algoreus
          • Deleting an Axon in Algoreus
          • Deploying nodes from the Algoreus Hub
          • Using node templates in Algoreus
          • Exporting and importing Axons in Algoreus
          • Dynamic resource configuration in Algoreus
          • Working with namespaces in Algoreus
        • Soma (Transformation) User Guide
          • Algoreus Soma Overview
          • Algoreus Soma Concepts
          • Algoreus Soma UI components
          • Working with multiple datasets
          • Navigating between Soma and Algoreus Genesis
          • Editing a transformation created in the Soma
          • Soma data types
          • Working with connections in Soma
          • Parsing a CSV file
          • Strings Formatting
          • Sending records to error
          • Working with numbers in Soma
          • Working with Decimal types in Soma
          • Performing date transformations in Soma
          • Filtering records
          • Finding and replacing values in a column
          • Filling null or empty cells
          • Copying, deleting, and keeping columns
          • Renaming a column
          • Joining two columns
          • Swapping two column names
          • Extracting fields to retrieve values
          • Exploding fields
          • Masking data
          • Encoding records to store or transfer data
          • Decoding records to store or transfer data
          • Applying a Hashing algorithm to a column
          • Upgrading the Soma transformation node version
          • Viewing and downloading a schema in Soma
          • Viewing Soma Service logs
        • Cerebellum (Operations and Monitoring) User Guide
          • Logging and Monitoring
          • Metrics
          • Dashboard and Reports
          • Preferences and Runtime Arguments
          • Transaction Service Maintenance
        • Engram (Metadata) User Guide
          • System Metadata
          • Discovery and Lineage
          • Audit Logging
          • Metadata Management
          • Accessing Metadata Programmatically
          • Metadata Field-Level Lineage
        • Clone (Replication) User Guide
          • Cloning overview
          • Clone Concepts
          • Adding Transformations to a Cloning Job
          • Deleting a Cloning Job
          • Tutorial: Cloning data from Oracle Database to BigQuery
        • Algology (Visualisation) User Guide
          • Dashboards
            • Using Dashboards
            • Building Dashboards
            • Manage dashboards
            • Publishing Dashboard
            • Playlist
            • Create and manage reports
            • Share dashboards and panels
            • Access Dashboard Usage
            • Search Dashboards
          • Panel Editor
            • Configure Panel Options
            • Configure standard options
          • Visualisations
            • Alert List
            • Bar Chart
            • Bar Gauge
            • Candlestick Panel
            • Canvas
            • Dashboard List
            • Flame Graph
            • Gauge
            • Heatmap
            • Histogram
            • Logs
            • Node Graph
            • Traces Panel
            • Pie Chart
            • State Timeline
            • Stat Panel
            • Time series
            • Trend Panel
            • Text Panel
            • Table
            • GeoMap
            • Datagrid Panel
            • Status history
            • Annotations
          • Explore
            • Logs in Explore
            • Queries in Explore
            • Tracing in Explore
            • Inspector in Explore
    • Turium Algoreus Connectors
Powered by GitBook
On this page
  • Background
  • Solution

Was this helpful?

  1. Turium Algoreus Documentation
  2. Turium Algoreus Overview
  3. How to Guides
  4. Axons (Pipeline) User Guide

Dynamic resource configuration in Algoreus

PreviousExporting and importing Axons in AlgoreusNextWorking with namespaces in Algoreus

Last updated 1 year ago

Was this helpful?

Users might want to create dynamic axons in Algoreus for improved reusability and simplified operations. This guide explains how to configure compute resources for axon execution at runtime.

Background

Batch Axon can be orchestrated using various engines like MapReduce or Spark. The resources for these engines (CPU and Memory) are typically configured during design time but can be altered during runtime by changing the settings in the Resources tab. Users can also alter the compute profile from the UI.

For dynamic Axons, these resources should be configured via runtime arguments. The following sections show you how to do this.

Solution

Configuring Compute Profile

Compute profile can be configured at runtime using the runtime argument (preferences). The value for the profile name should include the scope and profile name separated by a colon 'scope:profileName'.

For instance, to start the axon named 'BQMerge' on a profile called 'dp10' in the system scope:

Configuring Engine Resources Engine resources like CPU (cores) and memory can be configured using runtime arguments (preferences). To configure resources for Spark Driver and Executor, use the following options:

  • 'task.driver.system.resources.memory' for configuring the memory for Spark Driver. Memory is set in Megabytes. For instance, setting 'task.driver.system.resources.memory' to 2048 sets the driver memory resources to 2 GB (2048 MB).

  • 'task.driver.system.resources.cores' for configuring the CPU (cores) for Spark Driver. By default, the driver CPU is set to 1 core. For instance, setting 'task.driver.system.resources.cores' to 2 sets the driver cores to 2.

  • 'task.executor.system.resources.memory' for configuring the memory for Spark Executors. Memory is set in Megabytes. For instance, setting 'task.executor.system.resources.memory' to 2048 sets the executor memory resources to 2 GB (2048 MB).

  • 'task.executor.system.resources.cores' for configuring the CPU (cores) for Spark Executors. By default, the driver CPU (cores) is set to 1 core. For instance, setting 'task.executor.system.resources.cores' to 2 configures 2 cores for all executors.

Configuring Compute Resources

The following settings are available for configuring compute resources:

  • 'system.profile.properties.serviceAccount' for specifying the service account for the cluster.

  • 'system.profile.properties.masterNumNodes' for setting the number of master nodes.

  • 'system.profile.properties.masterMemoryMB' for setting the memory per master node.

  • 'system.profile.properties.masterCPUs' for setting the number of CPUs for the master.

  • 'system.profile.properties.masterDiskGB' for setting the disk in GB per master node.

  • 'system.profile.properties.workerNumNodes' for setting the number of worker nodes.

  • 'system.profile.properties.workerMemoryMB' for setting the memory per worker node.

  • 'system.profile.properties.workerCPUs' for setting the number of CPUs per worker node.

  • 'system.profile.properties.workerDiskGB' for setting the disk in GB per worker node.

  • 'system.profile.properties.stackdriverLoggingEnabled' set to true to enable Stackdriver logging for the Axons.

  • 'system.profile.properties.stackdriverMonitoringEnabled' set to true to enable Stackdriver monitoring for the Axons.

  • 'system.profile.properties.imageVersion' for configuring the image version.

  • 'system.profile.properties.network' for configuring the network for the cluster.

Background
Solution
Configuring Compute Profile
Configuring Compute Resources