to be honest, JQ is the only tool I know to use confidently at this stage…
But now that I’ve finally finished the whole process, I can definitely say that JQ is the culprit (even after doing exactly as @steevej has suggested). It is doing something in the background while converting the files. To be fair, maybe my methods are also wrong.
What worked for me is:
- I filtered the large file into 5 JSON files that contains 1 key:value only per file
- Then I uploaded each file to the same collection one by one.
- I successfully uploaded 46900 documents in mongoDB ready for analysis
The JSON filtering with JQ was done in terminal of Kali (my ubuntu is in a login loop and needs to be fixed lol). I really think this process is making the mess. A few things I noticed:
1.If I filtered the key:value pair (5 total) that I need from the main JSON file and save it to one file, it saves to a very large file - 1.7GB. While the original file was only 120MB which contains at least 20 key:value pairs per object.
2. If I took just one key:value pair and save to a file (separately)… it only saved as a 1MB files. So with 5 files they are only 5MB total.
Any way… I hope this helps someone out, that is probably as ‘noob’ as me and are having the same issue.
Thanks again @kevinadi … the things I’ve learned along the way… is so gratifying… it made the whole frustration in the beginning so… worth it!