Those are two separate things.
storage.wiredTiger.engineConfig.cacheSizeGB governs the WiredTiger cache size. This is the “working memory” of WiredTiger.
maxIndexBuildMemoryUsageMegabytes governs the maximum memory usage when building an index. This is separate from WiredTiger’s memory usage.
Currently there is no setting to limit the overall memory usage of a
mongod. What I mean by
mongod memory usage is memory used by anything other than low level WiredTiger work, e.g. for index builds, aggregation, in-memory sort, connection handling, etc. Generally, this means that WiredTiger reserves about half of RAM for its use, and the rest is to be used by the
mongod process (and also all other processes in the server, including the OS). This is the reason why it’s not recommended to run multiple
mongod processes in the same server: all of them will compete for memory access.