Fatal assertion 28782 at src/mongo/db/catalog/index_catalog_impl.cpp

Ops and Admin
#1

Hi all, we are trying to migrate the MongoDB from a Virtual Server to a Physical Server and in that process when we do the import of the data its failing on one of the index with the following issue.

I have searched in Google and found no help so reaching out the forum here to see if any one else has faced the same issue.

We are currently using Version 3.6 and running on a server with 300gb ram and 32 cores.

Pl let me know if you have any ideas of how to get this fixed.

#2

Hi @Ravi_Thotapalli welcome to the community.

Can you post the whole error message? There should be more information regarding the cause of the error in the full message.

Please also post:

  1. What MongoDB version you’re migrating to and from?
  2. How are you migrating them? Are you doing a mongodump/mongorestore or are you doing something else?
  3. If you traced the error to one of the indexes, have you tried dropping the index before migrating? Does it still come up with the same error?

Best regards,
Kevin

#3

Kevin, thanks for your email and here is the info.

  1. DB version of the Mongo we are running is 3.6.16

  2. We did follow the same process of mongodump/mongorestore. We are doing this to migrate our DB from a VM to a Physical server.

  3. I dont think we dropped the Index as this is related to a Product and we did not want to mess with it.

Here is what we see on the error msgs after we enabled the logging level to debug mode.

COMMAND [conn3894] command admin.$cmd command: isMaster { ismaster: true, $db: “admin” } numYields:0 reslen:208 locks:{} protocol:op_msg 0ms
2020-09-06T12:51:11.308-0700 F INDEX [conn4488] Found an invalid index { v: 2, key: { _id: 1 }, name: “id”, ns: “bigid-server.tmp.agg_out.46” } on the bigid-server.tmp.agg_out.46 collection: CannotCreateIndex: use of v0 indexes is only allowed with the mmapv1 storage engine
2020-09-06T12:51:11.308-0700 F - [conn4488] Fatal Assertion 28782 at src/mongo/db/catalog/index_catalog_impl.cpp 176
2020-09-06T12:51:11.308-0700 F - [conn4488]

thanks
ravi

#4

That is a peculiar error message, and typically seen on a failed upgrade/downgrade process. You shouldn’t see this message using a straightforward mongodump/mongorestore process. I also tried many different ways to try to induce this message, but haven’t been successful so far.

Could you provide more information:

  • What is the topology of your deployment? Is it a standalone, replica set, sharded cluster?
  • A detailed description of the dump & restore process that you did
  • At what point did you see this message
  • Was the VM deployment originally on an older version of MongoDB? If yes, how was it upgraded?
  • Is the physical server a brand new server?

One last option to try is to do mongorestore --noIndexRestore. However this would mean that the indexes would need to be recreated manually.

Best regards,
Kevin

#5

Kevin, here is the info

  1. This is a Standalone server ( specs are like 300 gb ram, 32 cores )
  2. We have been running the MongoDB on a Virtual Machine and ran into some Swap space issues and hence decided to move to a Physical box. As I mentioned earlier we tried Mongoexport/Mongorestore earlier and infact did it like three times. So far no luck as such. We even tried the dbrepair option which kind of recreated all the indexes but still its like in the middle of the application execution the DB goes down with the error message I sent you.
  3. As I said we are using this as part of an application meant for data governance so when we do scans of a source system be it an Oracle backend, SQL Server or a flat file based source the scans keep running and all of a sudden would fail abruptly with the message of index creation failed and the bizzare point being, though we are using storage engine wildtiger in the error msgs it talks about mmapv1 or something.
  4. No on the VM we had the same version of Mongo DB and it was not upgradeded
  5. The Physical server is a GPU based system that is not brand new but had a different application running. However we formatted it and reinstalled the Linux ( RHEL 7 ) from scratch and then did the installation of Mongo DB

I will have to work with my team on the restore. however we are thinking of actually creating a new instance of DB on the same server and try it out if it works atleast we will not be dead in the water. As this is a prod system we have been down over 2 weeks now. Working with our vendor who supplied this community edition is working slowly.

thanks
ravi

#6

Kevin I believe we have tried the noIndexRestore option and even with that we are seeing the same issue.

#7

Hi Ravi, let me recap what I understand so far:

  1. The assert happens only in the physical server. It never happened in the VM deployment, but you’re running out of resources there so you have to move.
  2. It happened during the restore. The data was dumped from the VM using mongodump, and was restored using mongorestore.
  3. This implies that the physical server was never in an operational state, because it keeps asserting on that exact index during restore.
  4. The MongoDB server version never changes in both the VM and the physical server. There never was a different MongoDB version used at any point.
  5. You tried the dump+restore 3 times, and each time it will fail with exactly the same message on exactly the same index.

Do the points above reflect your experience?

Can I ask you to provide the output of mongod --version as well? Also how did you install MongoDB in the physical server? Did you use the steps outlined in Install MongoDB Community Edition on Red Hat or CentOS, or are you using some other method?

Out of curiosity, have you tried restoring the dump file into another VM and not the physical server?

Best regards,
Kevin