I have requirement to implement ETL pipeline between RDBMS(DB2) and azure mongoDB, there would be continues sync between these two Database, meaning whenever there is writes in MongoDB, the data will be sync in DB2 and vice versa. Planning to use following design using Kafka connect.
- Deployment of JDBC source connector(Kafka connect) from DB2 to Kafka topic
- Implementation of Kafka Consumer service, which will consume the records from Kafka topic and transform into Document format(json)and this service will also connect to MongoDb for writing the documents.
I can also use Kafka stream + sink connectors here instead of writing consumer service, so my question is what is the pros/cons of using Kafka stream over Kafka consumer service for my use case?