Connect CDC SQData

Replication to Kafka in AVRO format using the Confluent convention and the Confluent Schema Registry is the most efficient and highest performance Kafka option. The Confluent AVRO convention prefixes each AVRO message with a small header providing the schema id used to encode the AVRO formatted message. Automatic maintenance of Schemas in the Registry provides for near automatic schema evolution.

Prerequisites

1.The url of the Confluent Repository to use must be present in the OPTIONS statement.

2.Connect CDC SQData assumes that support for command line url (cURL) and the Confluent Schema Registry has been installed and configured somewhere in your environment for use by both Apply and Replicator Engines. The URL and port number of the Registry will be required in order to read and register schema information.

3.The external library libcurl is required to support communication with the Confluent Schema Registry. Libcurl can be installed using distribution provided package if available or built from source which can be downloaded from https://curl.se/download.html.

4.The Confluent Platform is available at https://www.confluent.io/download/.

5.Topic names, Schema Subjects and Metadata Objects will all be generated by default. Those defaults can be overridden for one or all source CDC records using MAPPINGS sections, either Static or Dynamic, that provide:

•A "subject", which is effectively a key to the Confluent Schema Registry, for each source object to be replicated. The Replicator will use the description of the source object provided by the Publisher to generate an appropriate schema, and will use the subject to determine the confluent topic id associated with the object, using the provided 'subject' name. If the schema is not registered, the Replicator will register a new one under the indicated "subject".

• A "Topic"  which will be used to name the Kafka topic for a specific source object name (i.e. table name). A given topic can be used for one or more object tables, even all of them. If only one topic is to be used, it can be encoded in the target url. If more than one topic is to be used, the target url must be a generic url, of nature Kafka, the substitution parameter "*" being replaced by the topic value for each Source object.

•Ignore or Terminate exception handling for source CDC records not already explicitly configured.

Example

Simple AVRO Confluent formatted Kafka where every source object name (i.e. table name) is written to a specified Topic with a common prefix and suffix and using an AVRO schema managed by the Confluent Registry.

REPLICATE

  DB2 cdc://<host_name>:<sqdaemon_port>/<publisher_name>/<subscription_name>

  TO AVRO CONFLUENT kafka:///<prefix>_*_<suffix>/key

;

OPTIONS

CONFLUENT REPOSITORY 'http://<host_name>:<host_port>',

;

 

Notes:

1.The relationship between the DESCRIPTION Alias, Topic and Subject are matters determined by the planners and architects of the organization's Confluent Schema Registry. The choice of SUBJECT may be made based on the default supported by the Confluent Control Center which requires the SUBJECT to be the same as the TOPIC with the addition of "-value".

2.The Confluent Schema Registry supports multiple Topic naming strategies. All are supported by the Replicator but they may or may not be compatible with other tools including Confluent's own Control Center.