Load huge data from MongoDB

Working with Data
,
1

Hi everyone, currently I facing one issue.

My boss wants me to show all the data on the dashboard and all the data must be the latest data. So, when I try to load all the data, it is very very slow (need a few minutes to load).

Situation

  • Each time need to load 6000++ data from the database.
  • Each of the documents has 46 fields.
  • Share server: 4 CPU + 8 RAM
  • If the system fetches the data from the database, it is very slow

The thing that I did:

  • I have added Redis to the system. If don’t have new data coming, it will fetch the data from Redis.
  • System only picks up the field that will be used on the dashboard

So far I don’t have any idea that can do the enhancement. Hope your guys can provide a new idea to me.

Thank you

2

Hi @alan_tang,

Try MongoDB Charts to build your dashboard. It’s directly built on top of MongoDB Atlas and you don’t have to ETL the data elsewhere to use it in your dashboard.

Here is a public dashboard I made: MongoDB Charts

Here is the amount of data I’m using to produce these COVID-19 Charts:

image
image1138×535 33.5 KB

Some of my charts are using these entire collection to render the data… So it’s not 6000+ here but 2.4M for the biggest.

Also, do you have the correct indexes to support your queries? You shouldn’t need to cache data in Redis. Sounds like extra work and more synchronisation issues.

Cheers,
Maxime.

2 Likes
3

Hi @MaBeuLux88, thanks for your suggestion. I will try it. Thank you

1 Like