Connect CDC SQData

The open source Kcat utility is a generic command line non-JVM Apache Kafka producer and consumer utility. While it's best known for operating as both a Producer and a Consumer, it provides another important function when starting to test a Connect CDC SQData Apply or Replicator Engine; Kcat is written in C rather than Java and therefore exercises the same libraries and configuration prepared for the Engine.

Kcat like Connect CDC SQData is sometimes referred to as a Swiss Army Tool, in this case a quick and simple tool for inspecting and creating data in Kafka. You can find Kcat on github at https://github.com/edenhill/kcat.  You may find others are already using it inside your organization or that it is already installed on your system.

Among it's many functions, Kcat can be used to quickly confirm your configuration without having to connect to a publisher for test data or wonder what has happened to all those Kafka topics you think you have just written:

•In Producer mode Kcat reads messages from stdin, delimited with a configurable delimiter (-D, defaults to newline), and produces them to the provided Kafka cluster (-b), topic (-t) and partition (-p).

•In Consumer mode Kcat reads messages from a topic and partition and prints them to stdout using the configured message delimiter.

•Kcat features a Metadata list (-L) mode to display the current state of the Kafka cluster and its topics and partitions.

•Most importatnly It can also be configured to use the sqdata_kafka_producer.config file

Finally, It also supports Avro message deserialization using the Confluent Schema-Registry when you are ready to move up to AVRO from JSON, and generic primitive deserializers.

Examples:

1.Connect as a Consumer) to "my_broker", read the last 10 messages from the topic  named "my_sqdata_test_topic" and then exit.
 
kcat -C -b my_broker -t my_sqdata_test_topic -p 0 -o -10 -e

2.Connect as a Consumer to the default broker, read and display all the mesages from the topic "my_sqdata_test_topic" with the output in hex
 
kcat -C -b 0 -t my_sqdata_test_topic -o 0 -e | hexdump

3.Connect as a Producer using the same configuration used by the Apply or Replicator Engine as specified in the sqdata_kafka_producer.config file. Continue to run until interrupted by CTL-C.
 
kcat -P -F ./<path_to>/sqdata_kafka_producer.config  -t my_sqdata_test_topic  -o 0

4.Decode Avro key (-s key=avro), value (-s value=avro) or both (-s avro) to JSON using schema from the Schema-Registry.
 
kcat -b my_broker -t my_sqdata_test_topic -s avro -r http://schema-registry-url:8081

Note, Kcat is the new name for the former Kafkacat project. It was renamed to kcat in August 2021 to adhere to the Apache Software Foundation's (ASF) trademark policies. Other than the name, nothing else was changed.