Find dot in the key value in collection with 1M records

arvind_toorpu · April 13, 2022, 9:38pm

I am trying to find the query to find all the fields/keys in collection with 1M records. Buti end up getting this error with output any suggestion on how to overcome this ??

mr = db.runCommand({
  "mapreduce" : "MyCollectionName",
  "map" : function() {
    var f = function() {
      for (var key in this) {
        if (this.hasOwnProperty(key)) {
          emit(key, null)
          if (typeof this[key] == 'object') {
            f.call(this[key])
          }
        }
      }
    }
    f.call(this);
  },
  "reduce" : function(key, stuff) { return null; },
  "out": "MyCollectionName" + "_keys"
})
print(db[mr.result].distinct("_id"))

lmdb> print(db[mr.result].distinct("_id"));
MongoServerError: distinct too big, 16mb cap

I am fairly new to mongodb, so forgive my ignorance…

MaBeuLux88 · April 15, 2022, 4:00pm

Hi @arvind_toorpu,

Map / Reduce has been deprecated for year now. I wouldn’t recommend to use this but I would use the aggregation framework instead.

Here is my solution. It only works for the top level fields but maybe it’s enough for you?

I used the sample_mflix.movies from Atlas for this example:

[
  {
    '$replaceRoot': {
      'newRoot': {
        'doc': {
          '$objectToArray': '$$ROOT'
        }
      }
    }
  }, {
    '$unwind': {
      'path': '$doc'
    }
  }, {
    '$group': {
      '_id': null, 
      'keys': {
        '$addToSet': '$doc.k'
      }
    }
  }
]

Result:

{ _id: null,
  keys: 
   [ 'fullplot',
     'tomatoes',
     'languages',
     'metacritic',
     'released',
     'awards',
     'year',
     'type',
     'genres',
     'directors',
     'poster',
     'cast',
     'plot',
     'runtime',
     'countries',
     'num_mflix_comments',
     'title',
     'rated',
     'lastupdated',
     'writers',
     'imdb',
     '_id' ] }

If you want more details, I would look at Variety which is a Schema Analyzer for MongoDB. It doesn’t look like it’s still maintained though…

Or more simply, you could just use MongoDB Compass and get a schema analysis directly in there like this:

Cheers,
Maxime.