Hi all,
Please see the attached 2 diagrams.
I have a single compressed/json file coming out each our, nested in the directory structure as per image, that contains a json array with “documents” that have to be loaded each into a collection.
How would I be able to do this, This will be from a Confluent/Kafka in AWS account onto Atlas.
or would I maybe need to use a Lamda function t pickup the file and decompile it into individual docs and insert them.
G
Anyone ?
Is it possible to have a connector consume from a nested directory structure ?
Is it possible for a connector to take a single message on a topic and split the contents into the array of documents. - Realise i might be able to do this on the Kafka topic using SMT.
G
Expanding…
ignore the nested source folder structure, just realised that sits with my source connector, irrelevant for discussion here…
{
[
{doc1},
{doc2},
{doc3},
{doc4},
{doc5}
]
}
Each doc as a individual doc in the destination collection
G
Answer.
The mongo sing connector can
only uncertainty at the moment is if this version is available on the AWS Confluent cloud deployment.
G
