|
Create HDFS Apply Engine Script |
Scroll |
HDFS files can contain virtually any type of data. In the case of HDFS files created by the SQData Apply Engine they generally represent either the complete before and after image of the source data from the SQData Capture Agent or only selected fields from the Captured data that might represent a particular business event. The default Payload Format for HDFS is JSON.
The HDFS Apply Engine script generated in this step is a no-map replication script where the source datastore and target topics contain the identical data.
If you have a use case that involves replication of non-identical source/target datastores/topics, you will need to create Hadoop target descriptions before creating the Apply Engine.
This section guides you through the following tasks:
1.Add Source Data Structures
2.Determine the Hadoop HDFS Output Format
3.Determine the Hadoop HDFS Datastore specifications
4.Construct the HDFS Apply Engine Script
5.Parse Engine
6.Run Engine
7.Sample JSON Output
