|
Semi-Automatic AVRO Schema Registration |
Scroll |
While Automatic AVRO Schema registration and management is clearly preferred, sometimes for procedural or technical reasons there is a requirement to modify schemas in some way or otherwise intervene in their registration including:
1.A restriction on the automatic update of the "Production" Confluent Schema Registry. Highly discouraged by Precisely, the worse reason for disabling automatic registration due to the inevitable complication of procedures and potential for errors that will impact production.
2.A requirement to modify the "Default" transformation of source field/column names from their original form to the JSON standard. Precisely highly recommends using the default, all lower case words separated by under-score characters.
3.A requirement to modify the datatype for one or more fields/columns.
Switching to semi-automatic schema registration is not complicated but must be accompanied by either a very effective manual or automated procedure:
1.Remove or do not include the CONFLUENT REGISTRY and it's url in the Apply Engine OPTIONS parameter list.
2.Create a section in the script that lists the Topics and Topic ID for every Description to be replicated. Many prefer to create a new Engine "Part" file that will be included when the script is parsed just like Descriptions and Procedures.
3.Parse the Apply Engine script to create a full set of JSON topic descriptions in the Working directory
4.Modify any of the generated description files as required either manually or using automation.
5.Register any schema that is modified and record the TOPIC ID for each schema in the new "Topic ID" section of the script or the Topic/ID include file either manually or using automation.
Note, Apply engines that utilize the REPLICATE function, for DB2, IMS and VSAM source data still require manual intervention to add/update the source DESCRIPTION parts that correspond to new and altered RDBMS schemas and IMS or VSAM "copybooks". Once that has been accomplished however, the Apply Engine need only be Parsed and the procedure above followed when registration of the updated AVRO schemas will not be performed automatically. Even Version 4 Apply Engines that have "customized" target DESCRIPTIONS and mapping PROCEDURES will benefit because the Target DESCRIPTIONS used to create the AVRO schemas will be automatically validated before registration when the Engine is Started.
FIXIT
Example 1 Db2 to AVRO
Review the Db2 to Kafka AVRO example in Automatic AVRO Schema Registration. Switching to Semi-Automatic schema registration is straight forward and simple.
Only a few elements of the Apply to Kafka AVRO script need to be added or altered as identified by a green bar in the first character of modified lines. Items in RED are removed/Replaced:
----------------------------------------------------------------------
-- Name: DB2TOKAF: Z/OS DB2 To Kafka AVRO on Linux
...
-- Change Log:
----------------------------------------------------------------------
-- 2019-02-01 INITIAL RELEASE using JSON
-- 2019-02-02 Switch to Confluent AVRO Schema Registry
...
OPTIONS
CDCOP('I','U','D') -- Set CHANGE OP Constants
,USE AVRO COMPATIBLE NAMES -- Required for AVRO Targets
,CONFLUENT REPOSITORY 'http://schema_registry.precisely.com:8081'
;
----------------------------------------------------------------------
-- Source Descriptions
----------------------------------------------------------------------
BEGIN GROUP DB2_SOURCE;
DESCRIPTION DB2SQL ./DB2DDL/EMP.ddl AS EMPLOYEE
KEY IS EMP_NO
TOPIC IVP_HR_EMPLOYEE
SUBJECT IVP_HR_EMPLOYEE-value;
DESCRIPTION DB2SQL ./DB2DDL/DEPT.ddl AS DEPARTMENT
KEY IS DEPT_NO
TOPIC IVP_HR_DEPARTMENT
SUBJECT IVP_HR_DEPARTMENT-value;
END GROUP;
----------------------------------------------------------------------
-- Target Datastore(s)
----------------------------------------------------------------------
DATASTORE kafka:///*/key -- specify dynamic topic
OF AVRO -- specify AVRO format
FORMAT CONFLUENT -- use Confluent Schema Registry
AS TARGET
DESCRIBED BY GROUP DB2_SOURCE -- use source for REPLICATE
;
