Hey Mike,
Nice to meet you.
This seems like a very interesting use case. I have a couple ideas about how you might be able to rearchitect this process to be a bit more full proof. Have you looked at our Atlas Data Federation and Atlas Data Lake offerings at all?
If you’d be interested in chatting, put some time on my calendar here and I can step you through what I’m thinking to see if it’s a good fit. Calendly - Benjamin Flast
Best,
Ben