Updating 100s of Millions of Documents

Working with Data Drivers
,
3

Hi Maxime,

Thanks for your reply!

Upserting is something I considered, but the challenge I have is that when I’m processing documents, the assumption is the IP hasn’t been seen before, so it is initialised with a default object:

{   
   "ip": str,
   "ports": Array[int],
   "banners": Array[Object],
   "certificates": Array[Object],
   "first_scanned": ISODate,
   "last_scanned": ISODate,
   "headers": Array[Object]
}

If a document with the provided IP exists (unique index), we then have to merge the documents. This is because inserting elements into banners, certificates or headers almost always requires modifying the ports array. If I did an upsert, it would set the new array element but not allow me to merge the ports.

I hope this makes sense.

1 Like