Unable to add 1TB JSON file to MongoDB

Hi,
I am trying to import 1.12TB of a JSON file to the mongoDB installed on my local machine using the setup file- mongodb-windows-x86_64-6.0.5-signed.

When I use the below command:

mongoimport --db MY_DB --c MY_COLLECTION_NAMEe --batchSize 1 --file C:\myfile.json

it gives me error as below:

Failed: an inserted document is too large
0 document(s) imported successfully. 0 document(s) failed to import.

Please note that I am not using --jsonArray as the json file data is not in array.

Can anyone guide/help me resolving this problem as this makes pointless to use MongoDB as it does solves the purpose of using NoSQL database for unstructured large data.

Thanks in advance
Lokesh

If you have a single document in a 1.12TB JSON then you are looking for trouble.

The JSON file is not an array. Could it be that it is a list of smaller documents that is not syntactically enclosed with square brackets?

Even if I would be very surprised that your 1.12TB is a single JSON document there are ways to brake it down into more manageable parts and use references to the smaller parts.

1 Like

You need to break it up into 10 to 20 GB files, don’t do a full terabyte, that’s insanity lol.

@lokesh_Chandra

Take the bottom script and setup an if/else go through all of the JSON files and upload them one at a time until they are all uploaded.

It’ll automate this whole process and make your life easier after breaking up the JSON file into smaller chunks.