Shard Upgrade from 4.4.17 to 5.0.13, and then user database was missing

[Zhang_Eddie]

After upgrading 4.4.17 to 5.0.13, the customer database was missing, i’m pretty sure that the config files is the original one.

Running transaction
Updating : mongodb-org-database-tools-extra-5.0.13-1.el7.x86_64 1/16
Installing : mongodb-mongosh-1.6.0-1.el8.x86_64 2/16
Updating : mongodb-org-server-5.0.13-1.el7.x86_64 3/16
Updating : mongodb-org-shell-5.0.13-1.el7.x86_64 4/16
Updating : mongodb-database-tools-100.5.0-1.x86_64 5/16
Updating : mongodb-org-tools-5.0.13-1.el7.x86_64 6/16
Updating : mongodb-org-mongos-5.0.13-1.el7.x86_64 7/16
Installing : mongodb-org-database-5.0.13-1.el7.x86_64 8/16
Updating : mongodb-org-5.0.13-1.el7.x86_64 9/16
Cleanup : mongodb-org-4.4.17-1.el7.x86_64 10/16
Cleanup : mongodb-org-tools-4.4.17-1.el7.x86_64 11/16
Cleanup : mongodb-org-database-tools-extra-4.4.17-1.el7.x86_64 12/16
Cleanup : mongodb-org-server-4.4.17-1.el7.x86_64 13/16
Cleanup : mongodb-org-shell-4.4.17-1.el7.x86_64 14/16
Cleanup : mongodb-org-mongos-4.4.17-1.el7.x86_64 15/16
Cleanup : mongodb-database-tools-100.4.1-1.x86_64 16/16
Verifying : mongodb-org-5.0.13-1.el7.x86_64 1/16
Verifying : mongodb-org-mongos-5.0.13-1.el7.x86_64 2/16
Verifying : mongodb-database-tools-100.5.0-1.x86_64 3/16
Verifying : mongodb-org-tools-5.0.13-1.el7.x86_64 4/16
Verifying : mongodb-org-shell-5.0.13-1.el7.x86_64 5/16
Verifying : mongodb-org-database-5.0.13-1.el7.x86_64 6/16
Verifying : mongodb-org-database-tools-extra-5.0.13-1.el7.x86_64 7/16
Verifying : mongodb-org-server-5.0.13-1.el7.x86_64 8/16
Verifying : mongodb-mongosh-1.6.0-1.el8.x86_64 9/16
Verifying : mongodb-org-mongos-4.4.17-1.el7.x86_64 10/16
Verifying : mongodb-org-tools-4.4.17-1.el7.x86_64 11/16
Verifying : mongodb-org-server-4.4.17-1.el7.x86_64 12/16
Verifying : mongodb-org-shell-4.4.17-1.el7.x86_64 13/16
Verifying : mongodb-org-database-tools-extra-4.4.17-1.el7.x86_64 14/16
Verifying : mongodb-org-4.4.17-1.el7.x86_64 15/16
Verifying : mongodb-database-tools-100.4.1-1.x86_64 16/16

Installed:
mongodb-mongosh.x86_64 0:1.6.0-1.el8 mongodb-org-database.x86_64 0:5.0.13-1.el7

Updated:
mongodb-database-tools.x86_64 0:100.5.0-1 mongodb-org.x86_64 0:5.0.13-1.el7 mongodb-org-database-tools-extra.x86_64 0:5.0.13-1.el7 mongodb-org-mongos.x86_64 0:5.0.13-1.el7
mongodb-org-server.x86_64 0:5.0.13-1.el7 mongodb-org-shell.x86_64 0:5.0.13-1.el7 mongodb-org-tools.x86_64 0:5.0.13-1.el7

Complete!
[ezhang@prod-mgdb-pos-shard1-data-101e MongoDB5.0.13]$ systemctl status mongod
● mongod.service - MongoDB Database Server
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2022-11-03 01:26:12 UTC; 28s ago
Docs: https://docs.mongodb.org/manual
Process: 27605 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=0/SUCCESS)
Process: 27602 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
Process: 27599 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
Process: 27598 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
Main PID: 27611 (mongod)
CGroup: /system.slice/mongod.service
└─27611 /usr/bin/mongod -f /etc/mongod.conf
[aaaaaaa@xxxxxxxxx MongoDB5.0.13]$ mongosh localhost
Current Mongosh Log ID: 636318dcc319989d556c419a
Connecting to: mongodb://127.0.0.1:27017/localhost?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.6.0
Using MongoDB: 5.0.13
Using Mongosh: 1.6.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (Privacy Policy | MongoDB).
You can opt-out by running the disableTelemetry() command.

Warning: Found ~/.mongorc.js, but not ~/.mongoshrc.js. ~/.mongorc.js will not be loaded.
You may want to copy or rename ~/.mongorc.js to ~/.mongoshrc.js.
rs01 [direct: primary] localhost> db.getSiblingDB(“admin”).auth(“xxxxxxx”, “xxxxxxxx” )
{ ok: 1 }
rs01 [direct: primary] localhost> show dbs;
admin 156.00 KiB
config 512.00 KiB
local 147.47 MiB

There should be a customer database, but now we can not see it.

even through I rollbacked the server to 4.4.17 with the data backup folder, we also can not see the customer database.

[kevinadi (Kevin Adistambha)]

Hi @Zhang_Eddie welcome to the community!

How did you perform the upgrade? Did you follow the recommendations in Upgrade a Sharded Cluster to 5.0? I ask because even though you mention that this is a sharded cluster, it looks like this is upgraded using some type of package manager?

Could you elaborate on the OS, your deployment details (how many shards, the output of sh.status(), etc.), and where you are in the upgrade process?

Also please provide the logs of the mongod in question, especially when it starts up, until it’s ready to accept connections.

There should be a customer database, but now we can not see it.

Typically this was due to the new server not using the old dbpath of the previous server. Could you double check if they’re using the same dbpath? If yes, could you provide the content of that folder?

Best regards
Kevin

[Zhang_Eddie]

@kevinadi , many thanks for your help, we found the root cause, this is our new cluster which is not in use currently, and there is no shard collections in this customer DB. , but your suggestion is also precious for us, we checked this cluster according to your suggestion.