|
Apply Engine |
Scroll |
Connect CDC SQData provides two complementary types of Engines, the Replicator and the Apply Engine. This document is a reference manual for the configuration and operation of the Apply Engine.
The Apply Engine is a multi-faceted and multi-functional component that can read from and write to virtually any type of datastore be it a file, hierarchical or relational database, VSAM file, message queue or TCP/IP port. It can interpret virtually any type of data structure including all forms of relational DDL, COBOL copybooks, JSON and comma delimited files. The most common function performed by an Engine however, is to process data published by a Change Data Capture (CDC) component, applying business rules, if necessary to transform that data and efficiently write AKA Apply that data to a Target datastore of any type on any operating platform.
The Apply Engine is controlled by an SQL like scripting language with a very rich syntax providing a wide range of operations ranging from replication of identical source and target structures using a single command to complex business rule based transformations. Connect CDC SQData Apply Engine commands and functions provide full procedural control of data filtering, mapping and transformation including manipulation of data at its most elemental level if required.
There are two Quickstarts providing step by step guides to the installation, configuration, testing and operation of both the Apply and Replicator Engine components for Kafka and HDFS Hadoop:
The following sections provide a detail level Reference to the development and operation of Apply Engines.
•Engine Features and Functions An overview of Apply Engine concepts including cross-platform operation, data propagation, data normalization and exception processing.
•Script Development Detailed guide to the creation of Apply Engine scripts, including preparation of the development environment, naming conventions the definition of datastores, performing source to target data mapping, using variables, creating cursors, and how to parse and test a script.
• Connect CDC SQData Commands Describes the syntax and use of Commands.
• Connect CDC SQData Functions Introduction to the use of functions, described in detail in the Function Reference.
•Use Case Scenarios Describe commonly encountered requirements are satisfied using Commands and Functions in Apply Engine scripts.
•Engine Execution Describes how the Engine is initiated both at the command line and under the sqdaemon in each of the supported environments.
•Engine Troubleshooting Provides assistance for diagnosing common problems encountered during the operation of Apply Engines.
