Yes at this point you’re getting into trickier territory of figuring out the working set size 
I can’t imagine that workload running in a O.S. with 16 GB.
It appears that that specific workload basically demands at minimum a 32GB machine. Anything less than that, then the working set won’t fit in memory anymore, and you’re faced with high frequency of swapping data in and out of disk. Since disk is typically the slowest part of a machine, everything just stops and wait until the disks has finished their job.
I have a lot of other workloads with 2 TB running in a OS with 64 GB and the daemon doesn’t take over so many memory
It is also possible that even though your total data size is large, the actual working set (data that is being worked on by the server the vast majority of time) is actually much smaller, and thus you’ll have great performance even with a smaller machine. Hence, usually there is little correlation between data size and the required RAM size for good performance.
For the best performance, you’ll need your working set to fit comfortably in RAM, irrespective of the total data you have stored in the server. This is very much workload dependent, as you have observed. However, as the first step, at least you should ensure your indexes fit in RAM.
As a side note, the MongoDB University offers a free course M201: MongoDB Performance that touches on many subjects about performance, should you be interested in learning more.
Best regards
Kevin