Connect CDC SQData

Apache's Distributed Streaming Platform, Kafka, is a robust tool for building real-time streaming data pipelines on Open Systems platforms. Its functionality is similar to IBM's MQ and more robust than Java Messaging Service (JMS) however it is not natively supported on z/OS. Connect CDC SQData bridges the gap between z/OS and Kafka with its DB2 and IMS Change Data Capture on Z/OS, Replicator Engine running on Linux and Apply Engine Kafka Consumers running on Linux. This provides a hybrid solution that effectively supports three of Kafka's four core API's without modification of z/OS business applications in heterogeneous z/OS and Open Systems environments:

1.The Producer API which allows a business application to publish a stream of events to one or more Kafka topics.

2.The Connector API that can connect Kafka topics to existing source databases. For example, a connector to a relational database that captures every change to a table.

3.The Consumer API that can connect the Apply Engine to a partitioned stream of CDC Data to facilitate Parallel Apply processing to Relational databases and other Targets.

Connect CDC SQData's high performance Change Data Capture can be used to detect business events that "Change Data" in real time and/or capture all changes that occur in a DB2 source database. The resulting CDC data is processed by the Replicator Engine running on Linux that automatically generates JSON Schemas, translates all source data to UTF-8 and writes that to Kafka Topics using the Producer API.

Connect CDC SQData also seamlessly supports Confluent's proprietary Schema Registry by automating the generation of AVRO Schema's for new Tables and updating those schemas when they change in the Relational source database.

Operating in Distributor mode, streaming IMS CDC data can be partitioned for parallel Apply Engines running as Kafka Consumers facilitating higher performance replication of non-relational data to relational and streaming targets than otherwise possible.

HDFS is a distributed Java-based file system typically used to store large volumes of data. HDFS is one of several components that make up the framework called Hadoop; which consists of several subsystems that provide for parallel and distributed computation on large datasets. Similar to Kafka, HDFS operates only on commodity Linux based hardware. Connect CDC SQData bridges the gap between z/OS and HDFS with its DB2 Change Data Capture on Z/OS and Replicator Engine running on Linux. This provides a hybrid solution that effectively supports the HDFS distributed file system that utilizes a cluster of machines to provide high-throughput access for Big Data applications.

Connect CDC SQData's high performance Change Data Capture can be used to detect business events that "Change Data" in real time and/or capture all changes that occur in a DB2 source database. The resulting CDC data is processed by the Replicator Engine running on Linux that automatically generates JSON Schemas, translates all source data to UTF-8 and writes that to HDFS using native API's.

Connect CDC SQData also seamlessly supports AVRO formatted data by automating the generation of AVRO Schema's for new Tables and updating those schemas when they change in the Relational source database.