Map Reduce - shard crashes with out of memory exception

Ops and Admin Sharding
,
1

Hi

We face an issue when we do many map reduce operations on a sharded cluster.
The used memory increases more and more and when it is on the limit of the host memory, the mongod process crashes with an out of memry exception.

Environment:

  • 24 shards, 80 GByte RAM each
  • Default value for storage.wiredTiger.engineConfig.cacheSizeGB (no value set in the config)
  • Continuous parallel running map reduce operations on different input and output collections (Java applicaton with 10 threads)

We believe that the memory can be used by mongo up to the max availabe RAM.
But the mongod process should never crash.

Many thank for a feedback!

2

@Hermann_Jaun What specific version of MongoDB server are you using?

Can you share an excerpt of the log lines with the out of memory exception and any associated stack trace?

Thanks,
Stennie

1 Like
3

@Stennie_X
The additional inofos:

  • MongoDB server version 4.0.16
  • We use 1 mongod process only on each server
  • No other applications on the server (standard Linux installation)
4

@Hermann_Jaun Can you share the log lines with the out of memory exception and any associated stack trace?

You originally mentioned the mongod process crashes, so I’m looking for some extra context on the reason for shutdown. Unexpected shutdown usually has some information in the logs followed by a stack trace.

Can you also confirm the distro version of Linux used and whether you are running a package installed from repo.mongodb.org or an alternative source?

Thanks,
Stennie

1 Like
5

Hi @Stennie_X

The logs are unfortunately already deleted, we have too much transactions to keep them so long. But we will save the next time the respective log file.

I would like to mention, that we set in the meantime on 3.3.2020 the parameter storage.wiredTiger.engineConfig.cacheSizeGB to 35 GByte.
It looks like since then the memory consumption stays much lower than before (same load). It is since more than 2 weeks quite stable, growing slowly from the 35 GByte to 62 GByte.

We are wondering what happens when the memory utilization reache 99%. Will the server crash again?

Therefore we would like also to understand how MongoDB manage the memory. Even if we are not running any operation the memory is never full released. I.e it does not go down to the 35 GByte per shard. Is it normal?

Thank you,
Hermann