A way to interact more faster with huge databases

Hello Everybody!

How are you doing today? I hope that everybody is safe and sound! I have a necessity to interact over a large database, we are talking about 16 MM records. I’m using python with mongoengine to interact over inside my application. I’m using a simple queryset interacting using a for loop.

But it is costing many hours to process all queryset and it is causing a bottleneck in my system

I would like know if mongo (through) pymongo has alternatives to interact more faster where it can be reduced from hours to minutes.

This is the code:

contacts = Contact.objects.filter(__raw__=query).only('id', 'email', 'lead_score').no_cache().batch_size(10000)

# query is a simple dictionary where it is place the search criteria

Estou iterando pelo for abaixo:

for index, contact in enumerate(contacts, 1):
       print contact.name

Thank you so much for your help guys!

Rogério Carrasqueira