Connect CDC SQData

The typical Kafka Topic Target "url" consists of the host address and port number and in the case of the Replicator Engine only dynamically generated Topics that have been automatically registered in the Confluent Schema Registry by the Replicator Engine.

Syntax

kafka://[<hostname>[:<port_number>]]

kafka://[<hostname>[:<port_number>]]/[<kafka_topic_id>][/<partition> |/key |/root_key |/]

 

Keyword and Parameter Descriptions

<hostname>:<port_number> Optionally identify specific Kafka Broker Hostname and TCP/IP port. Precisely recommends the dynamic specification of the Kafka Cluster including its host name using the sqdata_kafka_producer.conf file located in the Working Directory of the Apply Engine at launch. The file may actually contain all the configuration options documented by Librdkafka https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md. Typically however, only a small subset of those options are specified, including producer specific security information and a list of Kafka Broker hosts for example:

 

security.protocol=SASL_SSL

sasl.kerberos.service.name=kafka

sasl.kerberos.principal=<kafka_app_user@domain>

sasl.kerberos.keytab=/app/kafkalib/<kafka_app_user>.keytab

ssl.ca.location=/app/certificates/dev/SAI_RBC_G2_Root_CA.cer

metadata.broker.list=<broker_host_01>:<port>,<broker_host_02>:<port>,<broker_host_03>:<port>

 

<kafka_topic_id>   Specify a unique Kafka Topic ID. This is particularly useful when creating many topics or unwieldy long topic ID's. A topic containing an "*" indicates that the url is dynamic and the "*" will be replaced with the alias name of the source DESCRIPTION by default or the TOPIC <name> specified as part of the DESCRIPTION. The "*" may be preceded and followed by a string of characters to complete the full Topic name. Whether topics need to be defined to Kafka in advance depends on how Kafka has been configured.

 

[/<partition> | /key  | /root_key | / ]  While an Optional parameter, in its absence random partitioning amongst the available partitions for a topic with be used. While a specific valid partition number may be specified, Precisely strongly advises not using partition numbers as it become an additional point of maintenance.
 
The keyword "key" is used by Kafka to determine the target partition for Relational, VSAM and Keyed File sources. This is required to insure that successive changes to the same row/record are sent to the same partition, ensuring they will be processed by the consumer in the order of capture. The default functionality of /key for Relational sources is to use the full concatenated list of source key columns while VSAM and Keyed File sources must specify a KEY IS clause on each source DESCRIPTIONS. A KEY IS clause may also be specified for Relational sources to override the default with a specific set of column names from the source description to be used by Kafka for partitioning.
 
The keyword "root_key" is used only for IMS sources and by default specifies that the only the root key of any captured IMS Segments will be used used by Kafka to determine the target partition. Using the root key for all the segments captured in the hierarchy ensures that they will be processed by the consumer in the order of capture and together with all segments updated under a particular root segment.
 
"/" is required as a placeholder if the SETURLKEY function is to be used to create a custom partitioning Key.

Example

Once you have determined the content of the sqdata_kafka_producer.conf required by your site, the Kafka URL will be simply specified as follows with Topic ID's retrieved from the Confluent Schema Registry:

kafka:///*