Please see the attached 2 diagrams.
I have a single compressed/json file coming out each our, nested in the directory structure as per image, that contains a json array with “documents” that have to be loaded each into a collection.
How would I be able to do this, This will be from a Confluent/Kafka in AWS account onto Atlas.
or would I maybe need to use a Lamda function t pickup the file and decompile it into individual docs and insert them.
Is it possible to have a connector consume from a nested directory structure ?
Is it possible for a connector to take a single message on a topic and split the contents into the array of documents. - Realise i might be able to do this on the Kafka topic using SMT.
ignore the nested source folder structure, just realised that sits with my source connector, irrelevant for discussion here…
Each doc as a individual doc in the destination collection
The mongo sing connector can
- monitor/consume from multiple topics
- sink individual messages into specific databases or collections based on a key or a value in the document.
only uncertainty at the moment is if this version is available on the AWS Confluent cloud deployment.