Filesystem turned into WiredTiger and went crazy

Working with Data
1

I have a nodebb platform on the server, which of course ran on mongodb.
It all started after I decided to install another platform on mysql, I ran it, and it worked fine, I connected it to nginx, and everything is fixed, both sites work! Both the Mongo and the Mysql…
Today I opened the nodebb site and I was very surprised to see that it is not connected (502 Unable to access this site)…
As a first step, I accessed the IP address of the server, and there a surprise awaited me… an Apache screen! Apparently while installing or something, I inadvertently installed Apache. Now everything was understandable, I entered ssh, turned off Apache, turned on nginx. And whoop, refresh.
Brother, trouble comes in bunches! The 502 Bad Gateway screen usually indicates an error in the database, according to my experience, I didn’t wait long, I returned to ssh and ran ./nodebb setup knowing that it would soon connect to the database, and it would start running…
but what? Suddenly an error appeared, that it cannot connect to a database!
I ran mongo and instead of inserting into the shell, this is the response:

MongoDB shell version v3.6.8connecting to: mongodb://127.0.0.1:270172023-06-03T23:20:30.600+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused2023-06-03T23:20:30.602+0200 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :connect@src/mongo/shell/mongo.js:257:13@(connect):1:6

I thought it was related to a mysql conflict with mongodb, so I turned it off, and it still didn’t work…
I restarted the server… and it didn’t help!
When I run sudo service mongodb status
This is the output:

● mongodb.service - An object/document-oriented database
     Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-06-06 18:43:29 CEST; 26min ago
       Docs: man:mongod(1)
    Process: 702081 ExecStart=/usr/bin/mongod --unixSocketPrefix=${SOCKETPATH} --config ${CONF} $DAEMON_OPTS (code=exit>
   Main PID: 702081 (code=exited, status=100)

יונ 06 18:43:28 vmi693782.contaboserver.net systemd[1]: Started An object/document-oriented database.
יונ 06 18:43:29 vmi693782.contaboserver.net systemd[1]: mongodb.service: Main process exited, code=exited, status=100/n>
יונ 06 18:43:29 vmi693782.contaboserver.net systemd[1]: mongodb.service: Failed with result 'exit-code'.

I don’t know if the material has been deleted, or it’s simply impossible to access it, but in the /var/lib/mongodb folder I have the following material:

image
image842×764 22.9 KB

Apparently, from what I understood, these are database materials, I thought that if I deleted mongodb and reinstalled, and then put in the materials that were then it would be backed up automatically.
Need and would appreciate help! Thanks in advance!
Of course, I’m attaching some files here to diagnose the problem:

root@vmi693782:~# mongo
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
2023-06-06T19:14:37.152+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2023-06-06T19:14:37.155+0200 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
   dbPath: /var/lib/mongodb
   journal:
      enabled: true
   engine: mmapv1
#  engine:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:
security:
  authorization: enabled

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

I changed some details in conf according to chatgpt because I thought it would help me, but nothing helps at all

All permissions on all mongodb folders are set to the mongodb user

root@vmi693782:~# mongod
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] MongoDB starting : pid=713911 port=27017 dbpath=/data/db 64-bit host=vmi693782.contaboserver.net
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T19:18:21.801+0200 I CONTROL  [initandlisten] options: {}
2023-06-06T19:18:21.837+0200 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2023-06-06T19:18:21.837+0200 I STORAGE  [initandlisten]
2023-06-06T19:18:21.837+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2023-06-06T19:18:21.837+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-06-06T19:18:21.837+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3466M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-06-06T19:18:22.701+0200 I STORAGE  [initandlisten] WiredTiger message [1686071902:701276][713911:0x7f8c930d9ac0], txn-recover: Main recovery loop: starting at 10/5376
2023-06-06T19:18:22.770+0200 I STORAGE  [initandlisten] WiredTiger message [1686071902:770670][713911:0x7f8c930d9ac0], txn-recover: Recovering log 10 through 11
2023-06-06T19:18:22.817+0200 I STORAGE  [initandlisten] WiredTiger message [1686071902:817148][713911:0x7f8c930d9ac0], txn-recover: Recovering log 11 through 11
2023-06-06T19:18:22.847+0200 I STORAGE  [initandlisten] WiredTiger message [1686071902:847397][713911:0x7f8c930d9ac0], txn-recover: Set global recovery timestamp: 0
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten]
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten]
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server.
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2023-06-06T19:18:22.922+0200 I CONTROL  [initandlisten]
2023-06-06T19:18:23.046+0200 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2023-06-06T19:18:23.047+0200 I NETWORK  [initandlisten] waiting for connections on port 27017

On the last line he stays stuck like that the whole time, until I get out of him

And of course, mongod.log
(I exceeded the character limit, I will post the file in the next post…)
And if anyone wants mongodb.log too:

root@vmi693782:~# tail -n 100 /var/log/mongodb/mongodb.log
2023-06-06T18:35:47.666+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T18:35:47.666+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T18:35:47.666+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T18:35:47.666+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T18:35:47.666+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T18:35:47.666+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T18:35:47.666+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1", unixDomainSocket: { pathPrefix: "/run/mongodb" } }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2023-06-06T18:35:47.667+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2023-06-06T18:35:47.667+0200 I STORAGE  [initandlisten]
2023-06-06T18:35:47.667+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2023-06-06T18:35:47.667+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-06-06T18:35:47.667+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3466M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-06-06T18:35:48.434+0200 E STORAGE  [initandlisten] WiredTiger error (-31802) [1686069348:434250][699248:0x7f0dc841aac0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2023-06-06T18:35:48.434+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069348:434313][699248:0x7f0dc841aac0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2023-06-06T18:35:48.434+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069348:434320][699248:0x7f0dc841aac0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2023-06-06T18:35:48.434+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069348:434326][699248:0x7f0dc841aac0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2023-06-06T18:35:48.437+0200 E -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 421
2023-06-06T18:35:48.437+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2023-06-06T18:35:48.437+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T18:35:48.437+0200 I NETWORK  [initandlisten] removing socket file: /run/mongodb/mongodb-27017.sock
2023-06-06T18:35:48.437+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T18:35:48.437+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T18:35:51.852+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] MongoDB starting : pid=699277 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T18:35:51.856+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1", unixDomainSocket: { pathPrefix: "/run/mongodb" } }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2023-06-06T18:35:51.856+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2023-06-06T18:35:51.856+0200 I STORAGE  [initandlisten]
2023-06-06T18:35:51.856+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2023-06-06T18:35:51.856+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-06-06T18:35:51.856+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3466M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-06-06T18:35:52.521+0200 E STORAGE  [initandlisten] WiredTiger error (-31802) [1686069352:521386][699277:0x7ff357be0ac0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2023-06-06T18:35:52.521+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069352:521440][699277:0x7ff357be0ac0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2023-06-06T18:35:52.521+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069352:521446][699277:0x7ff357be0ac0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2023-06-06T18:35:52.521+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069352:521452][699277:0x7ff357be0ac0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2023-06-06T18:35:52.523+0200 E -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 421
2023-06-06T18:35:52.523+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2023-06-06T18:35:52.523+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T18:35:52.523+0200 I NETWORK  [initandlisten] removing socket file: /run/mongodb/mongodb-27017.sock
2023-06-06T18:35:52.523+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T18:35:52.523+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T18:43:23.075+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] MongoDB starting : pid=702037 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T18:43:23.081+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1", unixDomainSocket: { pathPrefix: "/run/mongodb" } }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2023-06-06T18:43:23.081+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2023-06-06T18:43:23.082+0200 I STORAGE  [initandlisten]
2023-06-06T18:43:23.082+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2023-06-06T18:43:23.082+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-06-06T18:43:23.082+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3466M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-06-06T18:43:23.826+0200 E STORAGE  [initandlisten] WiredTiger error (-31802) [1686069803:826652][702037:0x7f09748f8ac0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2023-06-06T18:43:23.826+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069803:826709][702037:0x7f09748f8ac0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2023-06-06T18:43:23.826+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069803:826716][702037:0x7f09748f8ac0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2023-06-06T18:43:23.826+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069803:826721][702037:0x7f09748f8ac0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2023-06-06T18:43:23.828+0200 E -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 421
2023-06-06T18:43:23.828+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2023-06-06T18:43:23.828+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T18:43:23.828+0200 I NETWORK  [initandlisten] removing socket file: /run/mongodb/mongodb-27017.sock
2023-06-06T18:43:23.828+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T18:43:23.828+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T18:43:28.448+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] MongoDB starting : pid=702081 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T18:43:28.453+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1", unixDomainSocket: { pathPrefix: "/run/mongodb" } }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2023-06-06T18:43:28.454+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2023-06-06T18:43:28.454+0200 I STORAGE  [initandlisten]
2023-06-06T18:43:28.454+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2023-06-06T18:43:28.454+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-06-06T18:43:28.454+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3466M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-06-06T18:43:29.196+0200 E STORAGE  [initandlisten] WiredTiger error (-31802) [1686069809:196286][702081:0x7fcb204a6ac0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2023-06-06T18:43:29.196+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069809:196409][702081:0x7fcb204a6ac0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2023-06-06T18:43:29.196+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069809:196426][702081:0x7fcb204a6ac0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2023-06-06T18:43:29.196+0200 E STORAGE  [initandlisten] WiredTiger error (0) [1686069809:196434][702081:0x7fcb204a6ac0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2023-06-06T18:43:29.198+0200 E -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 421
2023-06-06T18:43:29.199+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2023-06-06T18:43:29.199+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T18:43:29.199+0200 I NETWORK  [initandlisten] removing socket file: /run/mongodb/mongodb-27017.sock
2023-06-06T18:43:29.199+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T18:43:29.199+0200 I CONTROL  [initandlisten] shutting down with code:100

I would really appreciate the help. I’ve already tried to ask in several forums, no one has a clue. Or no one has the power to help.
I have very important materials in it.

Thanks in advance!

2

mongod.log:

root@vmi693782:~# tail -n 100 /var/log/mongodb/mongod.log
2023-06-06T19:20:10.505+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T19:20:10.505+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2023-06-06T19:20:10.505+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T19:20:10.505+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T19:20:15.743+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] MongoDB starting : pid=714589 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T19:20:15.753+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", engine: "mmapv1", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-06T19:20:15.754+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
2023-06-06T19:20:15.754+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T19:20:15.754+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2023-06-06T19:20:15.754+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T19:20:15.754+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T19:20:20.991+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] MongoDB starting : pid=714621 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T19:20:21.004+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", engine: "mmapv1", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-06T19:20:21.005+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
2023-06-06T19:20:21.005+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T19:20:21.005+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2023-06-06T19:20:21.005+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T19:20:21.005+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T19:20:26.248+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] MongoDB starting : pid=714649 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", engine: "mmapv1", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-06T19:20:26.257+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
2023-06-06T19:20:26.257+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T19:20:26.257+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T19:20:26.257+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T19:20:31.495+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] MongoDB starting : pid=714680 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T19:20:31.505+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", engine: "mmapv1", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-06T19:20:31.506+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
2023-06-06T19:20:31.506+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T19:20:31.506+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2023-06-06T19:20:31.506+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T19:20:31.506+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T19:20:36.743+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] MongoDB starting : pid=714703 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T19:20:36.751+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", engine: "mmapv1", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-06T19:20:36.752+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
2023-06-06T19:20:36.752+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T19:20:36.752+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2023-06-06T19:20:36.752+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T19:20:36.752+0200 I CONTROL  [initandlisten] shutting down with code:100
2023-06-06T19:20:41.997+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] MongoDB starting : pid=714731 port=27017 dbpath=/var/lib/mongodb 64-bit host=vmi693782.contaboserver.net
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] db version v3.6.8
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] modules: none
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] build environment:
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", engine: "mmapv1", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-06T19:20:42.007+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
2023-06-06T19:20:42.007+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2023-06-06T19:20:42.007+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] now exiting
2023-06-06T19:20:42.007+0200 I CONTROL  [initandlisten] shutting down with code:100

From this log I understand that it moved to the wiredtiger filesystem, but I have no idea why, and since then it has been completely screwed up…

3

Look like wiredtiger can’t recognize the data file. Maybe data corrupted, or incompatible format, whatever.

If you don’t have critical/important data, (or if you have a full backup), you can try deleting the db files and start from a “clean state”

4

I don’t have a backup, what I do have is the files of the db that I copied from /var/lib/mongodb after it was knocked out…
My question is why did it happen, why did it get screwed?
The data is very, very important!

5

There is a difference between starting mongod with and without config file
In your first screenshot you have started mongod issuing just mongod
This brought up your mongod successfully on default port 27017 and default dirpath /data/db
It appears to be hung but it is not.Since your mongod was started on foreground (without fork) the session appears to be hung but log clearly shows waiting for connections
You have to open another session and connect by issuing mongo

But above is not what you want
Your data was in /var/lib/mongod
Second attempt you tried starting your service but there is a mismatch in your config file and the data files that are existing in your dbpath
Your log clearly says wired tiger data files detected but config file is having mmapv1(incompatibility between what you have under that directory vs what is mentioned in config file)
What all changes made to your config file?
Did you take a backup of it before making changes
May be putting back the original values you can start your mongod and get back your data
Hopefully there is no corruption to the files

6

I didn’t take a backup.
What things do I need to change in conf for it to be good?

7

How can I best change the file system to xfs and prevent wiredtiger storage?
I got very confused, but this is probably my solution.
Do I need to create a new partition and set it to xfs?
I would appreciate it if someone could help me with this!
Thanks!

8

What changes you made to your config file?
Do you have a snapshot of it?
in initAndListen: Location28662: Cannot start server. Detected data files in /var/lib/mongodb created by the ‘wiredTiger’ storage engine, but the specified storage engine was ‘mmapv1’., terminating
2023-06-06T19:20:15.754+0200 I NETWORK [initandlisten] shutdown: going to close listening sockets…
All files ending with .wt are WT files
Why the storage was changed to mmapv1 if the .wt files already existed in the dbpath directory?
comment/disable this parameter in your config file and enable wiredtiger and see if you can start your mongod