Home
Turium Algoreus
Turium Algoreus
  • Turium Algoreus Documentation
    • Turium Algoreus Overview
      • How to Guides
        • Axons (Pipeline) User Guide
          • Algoreus Genesis
          • Algoreus Node
          • Steps for a simple batch Axon in Algoreus
          • Configuring Axon in Algoreus
          • Deploying an Axon in Algoreus
          • Running an Axon in Algoreus
          • Viewing and downloading logs in the Genesis in Algoreus
          • Scheduling an Axon in Algoreus
          • Reusable Axons in Algoreus
          • Using Triggers in Algoreus
          • Working with multiple versions of the same node in Algoreus
          • Modifying a draft Axon in Algoreus
          • Editing a deployed Axon in Algoreus
          • Duplicating An Axon in Algoreus
          • Deleting an Axon in Algoreus
          • Deploying nodes from the Algoreus Hub
          • Using node templates in Algoreus
          • Exporting and importing Axons in Algoreus
          • Dynamic resource configuration in Algoreus
          • Working with namespaces in Algoreus
        • Soma (Transformation) User Guide
          • Algoreus Soma Overview
          • Algoreus Soma Concepts
          • Algoreus Soma UI components
          • Working with multiple datasets
          • Navigating between Soma and Algoreus Genesis
          • Editing a transformation created in the Soma
          • Soma data types
          • Working with connections in Soma
          • Parsing a CSV file
          • Strings Formatting
          • Sending records to error
          • Working with numbers in Soma
          • Working with Decimal types in Soma
          • Performing date transformations in Soma
          • Filtering records
          • Finding and replacing values in a column
          • Filling null or empty cells
          • Copying, deleting, and keeping columns
          • Renaming a column
          • Joining two columns
          • Swapping two column names
          • Extracting fields to retrieve values
          • Exploding fields
          • Masking data
          • Encoding records to store or transfer data
          • Decoding records to store or transfer data
          • Applying a Hashing algorithm to a column
          • Upgrading the Soma transformation node version
          • Viewing and downloading a schema in Soma
          • Viewing Soma Service logs
        • Cerebellum (Operations and Monitoring) User Guide
          • Logging and Monitoring
          • Metrics
          • Dashboard and Reports
          • Preferences and Runtime Arguments
          • Transaction Service Maintenance
        • Engram (Metadata) User Guide
          • System Metadata
          • Discovery and Lineage
          • Audit Logging
          • Metadata Management
          • Accessing Metadata Programmatically
          • Metadata Field-Level Lineage
        • Clone (Replication) User Guide
          • Cloning overview
          • Clone Concepts
          • Adding Transformations to a Cloning Job
          • Deleting a Cloning Job
          • Tutorial: Cloning data from Oracle Database to BigQuery
        • Algology (Visualisation) User Guide
          • Dashboards
            • Using Dashboards
            • Building Dashboards
            • Manage dashboards
            • Publishing Dashboard
            • Playlist
            • Create and manage reports
            • Share dashboards and panels
            • Access Dashboard Usage
            • Search Dashboards
          • Panel Editor
            • Configure Panel Options
            • Configure standard options
          • Visualisations
            • Alert List
            • Bar Chart
            • Bar Gauge
            • Candlestick Panel
            • Canvas
            • Dashboard List
            • Flame Graph
            • Gauge
            • Heatmap
            • Histogram
            • Logs
            • Node Graph
            • Traces Panel
            • Pie Chart
            • State Timeline
            • Stat Panel
            • Time series
            • Trend Panel
            • Text Panel
            • Table
            • GeoMap
            • Datagrid Panel
            • Status history
            • Annotations
          • Explore
            • Logs in Explore
            • Queries in Explore
            • Tracing in Explore
            • Inspector in Explore
    • Turium Algoreus Connectors
Powered by GitBook
On this page
  • Clone Entities
  • Clone Actions
  • Clone states
  • Table states
  • Metrics
  • Components
  • Connectivity

Was this helpful?

  1. Turium Algoreus Documentation
  2. Turium Algoreus Overview
  3. How to Guides
  4. Clone (Replication) User Guide

Clone Concepts

PreviousCloning overviewNextAdding Transformations to a Cloning Job

Last updated 1 year ago

Was this helpful?

Clone Entities

Concept
Description

Cloning

A feature of Algoreus that allows for continuous low-latency data replication from operational data stores to analytical data warehouses.

Source

A node that reads changes from a database, table, or column and makes them available for processing in a Cloning job.

Target

A node that writes changes received from a source into a target database.

Source properties

Configurations for the source, including connection details, source database and table names, credentials, and other properties.

Target properties

Configurations for the target, including connection details, target database and table names, credentials, and other properties.

Cloning Job properties

Configurations for Cloning job including failure thresholds, staging areas, notifications, and validation settings.

Transformations

Implicit transformations such as column filtering and in-flight transformations such as masking fields and renaming columns.

Draft

A saved, partially completed Cloning job.

Events

Changes in the source that are to be cloned to the target.

Insert

Addition of new records in the source.

Update

Updates to existing records in the source.

Delete

Removal of existing records in the source.

Logs

The operational logs of a Cloning job.

Cloning Job detail

A detail page with Cloning job information.

Dashboard

A page that lists the state of all change data capture activities.


Clone Actions

Concept
Description

Deploy

Creating a new Cloning job by specifying a source, target, and their configuration.

Refresh

Validating a Cloning transformation.

Save

Saving a partially created Cloning job.

Delete

Deleting an existing Cloning job.

Start

Starting a Cloning job.

Stop

Stopping a Cloning job.

View logs

Viewing logs of a Cloning job.

Search

Searching for a Cloning job by its name, description, or other Cloning job metadata.

Assess

Assessing the impact of Cloning prior to starting Cloning.


Clone states

Concept
Description

Deployed

The Cloning job is deployed, but not started.

Starting

The Cloning job is initializing, and is not ready to Clone changes.

Running

The Cloning job is started, and is Cloning changes.

Stopping

The Cloning job is stopping.

Stopped

The Cloning job is stopped.

Failed

The Cloning job failed due to critical errors.


Table states

Concept
Description

Snapshotting

The Cloning job is taking a snapshot of the current state of the table prior to clone changes.

Cloning

The Cloning job is cloning changes from the source table into the destination table.

Failing

The Cloning job is failing to clone changes from the source table due to error.


Metrics

Concept
Description

Inserts

The number of inserts applied to the target in the selected time period.

Updates

The number of updates applied to the target in the selected time period.

Deletes

The number of deletes applied to the target in the selected time period.

Throughput

The number of events and the number of bytes cloned to the target in the selected time period.

Latency

The delay at which data is cloned to the target in the selected time period.


Components

Component
Description

Service

Supervises the end-to-end orchestration of Cloning jobs.

State Management

Manages the state of each Cloning job in a Cloud Storage bucket.

Execution

Clusters provide the execution environment of Cloning jobs.

Source Database

Your production operational database that clones into your target database.

Change Tracking Mechanism

Algoreus relies on a change tracking solution to read changes in the source database.

Target Database

The destination location for Cloning and analysis.

Authentication

Varies according to the source database or change tracking software.


Connectivity

From
To
Optional
Protocol
Network
Auth Security
Purpose

Cloning Service

Source DB

Yes

Depends on Source. JDBC for direct DB connection

Peering + Firewall rules + VPN/Interconnect + Router

DB Login

Needed at design time only, not execution time.

Delta Service

Cloud

No

Cloud API

VPC-SC

IAM

State Management

Cluster

Source DB

No

Depends on Source. JDBC for direct DB connection

Peering + Firewall rules + VPN/Interconnect + Router

DB Login

Needed at execution time, for reading changes from source DB to clone them to target

Cluster

Cloud

No

Cloud API

VPC-SC

IAM

State Management

Cluster

BigQuery

No

Cloud API

VPC-SC

IAM

Needed at execution time, for applying changes from the source DB to the target


Clone Entities
Clone Actions
Clone states
Table states
Metrics
Components
Connectivity