After the ‘accumulate’ stage has run against each document on a shard it would be extremely useful to run a script on the shard against the state to reduce the state down to something smaller before passing the final state over the network back to mongos for the ‘merge’. (Over in the Elastic camp they call this the ‘combine_script’).
I have a customer orders database sharded by the Customer ID (meaning all data relating to any specific customer is kept on the same shard - so customers don’t need to be ‘merged’ between shards). I can write an accumulator to perform processing across all customer orders on the shard and the wish to perform further processing to reduce the state (containing data spanning various customer orders) down to a minimal state (mostly aggregated customer stats) to then be aggregated by mongos. I wish to avoid copying multiple large state objects (from each shard) across network to the ‘merge’ script on mongos.
An observed side effect a $group by shard key $accumulator stage is a ‘merge’ operation with the ‘init’ state (as there presumably is no other useful state from another shard to merge with and the merge script needs to be run by mongos at least once per shard). This behaviour doesn’t appear to be documented anywhere. I figure this ‘merge’ is run on mongos, but the documentation isn’t clear.
The documentation also mentions ‘merge’ being run due to memory being exhausted doesn’t explain where and when this can happen? Can this happen on the shard between document ‘accumulates’? (effectively this would be merging different ‘groups’ on the shard). Can memory related merge happen on the mongos aggregation node?
thanks in advance. Mark