I’m catching all errors in node.js code and write them to log. Recently I start receiving this error:
MongoServerError: Erroneous index key found with reference to non-existent record id. Consider dropping and then re-creating the index and then running the validate command on the collection.
The problem here is that error doesn’t contain any information about database and collection where this error happened. I have about 92 databases and 3864 collections. As there is no information about collection, now it’s not possible for me to understand where is the problem.
Is it possible to add information about DB and collection where this error happened?
For information, I found db, collection and index names in mongo logs:
{“t”:{“$date”:“2021-10-06T12:45:37.288-04:00”},“s”:“E”, “c”:“QUERY”, “id”:4615603, “ctx”:“conn9798”,“msg”:“Erroneous index key found with reference to non-existent record id. Consider dropping and then re-creating the index and then running the validate command on the collection.”,“attr”:{“namespace”:“database_name.actions_throttle”,“recordId”:{“RecordId”:5259780},“indexKeyData”:[{“key”:{“”:“index_name”,“”:“6138e51a4a2fa2f1ef3325b5”},“pattern”:{“type”:1.0,“key”:1.0}}],“indexKeyPattern”:{“type”:1.0,“key”:1.0}}}
But wonder if it’s possible to show this information in error message from the driver.
Are you printing the entire error in your try/catch of just the error.msg? Looks like you are missing some context around the error indeed that might be in the other fields.
@MaBeuLux88
I have big codebase and I saw that in logs, so actually now it’s hard to tell how (code) exactly the error printed in stderr, but I have stack of the error, may be it will be useful:
MongoServerError: Erroneous index key found with reference to non-existent record id. Consider dropping and then re-creating the index and then running the validate command on the collection.
at MessageStream.messageHandler (/home/user/project/node_modules/mongodb/lib/cmap/connection.js:467:30)
at MessageStream.emit (node:events:390:28)
at MessageStream.emit (node:domain:537:15)
at processIncomingData (/home/user/project/node_modules/mongodb/lib/cmap/message_stream.js:108:16)
at MessageStream._write (/home/user/project/node_modules/mongodb/lib/cmap/message_stream.js:28:9)
at writeOrBuffer (node:internal/streams/writable:389:12)
at _write (node:internal/streams/writable:330:10)
at MessageStream.Writable.write (node:internal/streams/writable:334:10)
at Socket.ondata (node:internal/streams/readable:754:22)
at Socket.emit (node:events:390:28)
at Socket.emit (node:domain:537:15)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at Socket.Readable.push (node:internal/streams/readable:228:10)
at TCP.onStreamRead (node:internal/stream_base_commons:199:23)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
Any idea what could have caused this issue?
We tried to migrate to MongoDB v5 from v4 but found that our CPUs don’t support AVX.
Anyway somehow our admin team managed to compile and run v5 without it but soon we saw it was not stable
and crashing, so we downgraded to v4. I’m not sure what they were doing exactly with database
files as I was not involved in the work, may be because v5 was crashing and somehow we got one
index corrupted because of that.
Sounds like you could use MongoDB Atlas and avoid all this !
As you have the namespace now database_name.actions_throttle, I would rebuild all the indexes on this collection - just to be on the safe side and repeat the operation for each new error you get like this. Something was probably corrupted during the failed upgrade / downgrade operations.
Yes, I know about index rebuilding, I did it already. But I created this ticket to suggest to add DB/collection/index information to that error message in code of node.js driver, so if somebody has got error it will be clear right away which index corrupted and in which coollection.
Hi Dmitry. You mentioned you are running MongoDB 4.4.9. Is there an upgrade history here? For example, were you running with earlier patch versions of MongoDB 4.4?
We ask because there is an issue in earlier patch releases of MongoDB 4.4 that could have caused this issue to be persisted to your index.
Yes, we started from MongoDB 3.2.? and were upgrading it periodically during last 5 years. But as the error appeared after attempt to migrate to v5 and then back to 4.4.9, I suppose this is the reason. But anyway, main issue here is that from that error throws by driver I can’t understand in which collection and which index got corrupted. To get this info I have to ssh server and check mongo log in file system. So, I wonder if it’ possible to add this information right inside error message thrown by node.js driver?
!