I belong to a group working on an open data voting project. We have settled on Mongo for the database after looking at a multitude of others for the project, mainly because each of the formats for voting data we get from different states and/or counties is in a different format. So my question is this- we currently have 29 data sources (state voting files) each one with an average of 6.7 million entries. What is the fastest and most accurate way to import this data into Mongo? We have one team member that has some experience with mongo and it is taking forever to import the FL voting record (7.7 million entries). It has been importing for 11 days now and we only have 4.5 million records. The server is a 16 core Xeon with 128GB of RAM and two 2TB SSD drives and is only running at about 18% capacity. The server is running Ubuntu 20 server.
Sure could use some input on if we are doing this right and again, if this is the wrong place please excuse my mistake and point me to the right one?