Exited with Error 51 (Lab - Logging to a Different Facility)

I’m having trouble with loging the mongod into a external log file. The error I am receiving is:

"ERROR: child process failed, exited with error number 51
To see additional information in this output, start without the “–fork” option."

When I set the fork option to false, I receive the following message:

"2020-03-12T19:19:19.638+0000 I CONTROL [main] log file “/var/mongodb/db/mongod1.log” exists; moved to “/var/mongodb/db/mongod1.log.2020-03-12T19-19-19”.
Aborted (core dumped)
"

I then remove the existing file as I know that it will create a new one if I restart the process. I run the mongod again and receive this message:

"vagrant@m103:~$ sudo rm /var/mongodb/db/mongod.log
rm: cannot remove ‘/var/mongodb/db/mongod.log’: No such file or directory
"

I have tried resetting the permissions to vagrant again but with no luck… I have also tried reading the file with the tale command and I was also promted that the file did not exists.

Here is my config file:

This confuses me because it previously told me that this file already existed. Is there something I am missing here?

The “I” next to “CONTROL” is “informational” meaning, it’s not an error (E). Look into the logAppend option to suppress this message.

To find the culprit, the first thing you’d want to do is to view the log file by using the cat command.

PS: Suggest you redact the config file from your post to keep in-line with forum guidelines (i.e. not posting potential answers to labs).

1 Like

Thank for pointing out that the “I” represents informational. I’ll have to re-watch that video on the log message formatting…

The cat command gave me a large output. How can I best interpret this?

vagrant@m103:~$ cat /var/mongodb/db/mongod.log
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] MongoDB starting : pid=2 dbpath=/var/mongodb/db 64-bit host=m103
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] db version v3.6.17
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] git version: 3d6953c3612b274ce592edafe6
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL2014
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] modules: enterprise
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] build environment:
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten]     distmod: ubuntu1404
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-03-12T19:30:15.199+0000 I CONTROL  [initandlisten] options: { config: "/shaf", net: { bindIp: "192.168.103.100,127.0.0.1", port: 27000 }, operationProfilinowOp", slowOpThresholdMs: 50 }, processManagement: { fork: true }, security: { a"enabled" }, storage: { dbPath: "/var/mongodb/db" }, systemLog: { destination: "/var/mongodb/db/mongod.log" } }
2020-03-12T19:30:15.200+0000 I STORAGE  [initandlisten] exception in initAndListe: Unable to lock the lock file: /var/mongodb/db/mongod.lock (Resource temporari). Another mongod instance is already running on the /var/mongodb/db directory,
2020-03-12T19:30:15.200+0000 F -        [initandlisten] Invariant failure globalsrc/mongo/db/service_context_d.cpp 272
2020-03-12T19:30:15.200+0000 F -        [initandlisten]

***aborting after invariant() failure


2020-03-12T19:30:15.214+0000 F -        [initandlisten] Got signal: 6 (Aborted).

 0x7f109d4a9ba1 0x7f109d4a8db9 0x7f109d4a929d 0x7f10981fc330 0x7f1097e59c37 0x7ff109ba6ef58 0x7f109bc5ecc8 0x7f109d325271 0x7f109d321357 0x7f109baec5d3 0x7f109dba70131 0x7f109ba0370c 0x7f109baf3f5a 0x7f109ba71089 0x7f1097e44f45 0x7f109badc0
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F109B0A0000","o":"2409BA1","s":"_ZN5mongo15printStackTraceE109B0A0000","o":"2408DB9"},{"b":"7F109B0A0000","o":"240929D"},{"b":"7F10981EC000,{"b":"7F1097E23000","o":"36C37","s":"gsignal"},{"b":"7F1097E23000","o":"3A028","b":"7F109B0A0000","o":"9CEF58","s":"_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_9B0A0000","o":"BBECC8","s":"_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6Clien109B0A0000","o":"2285271","s":"_ZN5mongo14ServiceContext20makeOperationContextEP{"b":"7F109B0A0000","o":"2281357","s":"_ZN5mongo6Client20makeOperationContextEv"0A0000","o":"A4C5D3"},{"b":"7F109B0A0000","o":"2404F65"},{"b":"7F109B0A0000","o""_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE"},{"b":"7F109B0A0000",s":"_ZZN5mongo13duration_castINS_8DurationISt5ratioILl1ELl1000EEEES2_ILl1ELl1EEEEENKUlvE_clEv"},{"b":"7F109B0A0000","o":"A53F5A","s":"_ZN5mongo11mongoDbMainEiPPF109B0A0000","o":"9D1089","s":"main"},{"b":"7F1097E23000","o":"21F45","s":"__lib,{"b":"7F109B0A0000","o":"A3C0CF"}],"processInfo":{ "mongodbVersion" : "3.6.17",: "3d6953c361213c5bfab23e51ab274ce592edafe6", "compiledModules" : [ "enterprise"{ "sysname" : "Linux", "release" : "3.13.0-170-generic", "version" : "#220-Ubunt9 12:40:49 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "7F109B0A0000"3, "buildId" : "6ACAF269F5944FAB5A293A46CBA9DEA6B6F74E3A" }, { "b" : "7FFCCCCF50 : 3, "buildId" : "8BEB43CA1BD4CFDB90BC2BFCE0EC3042E4090782" }, { "b" : "7F109AC : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "4ACB081CEF491FB4724336AC04" }, { "b" : "7F109A7A7000", "path" : "/usr/lib/x86_64-linux-mibs.so.30", "elfType" : 3, "buildId" : "931606F6B9E4CCB05C628D0525E564CE5E7A2DB"7F109A598000", "path" : "/usr/lib/x86_64-linux-gnu/libsensors.so.4", "elfType"  : "859FDBFDD82F0EFDEB44A433D9D8020A232A35E2" }, { "b" : "7F109A394000", "path" -linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "1B38A86853776548628FA4090913}, { "b" : "7F109A12B000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpagent.se" : 3, "buildId" : "AD37989D875C4B4DAAF96CD26387B8BFBAA9EE4F" }, { "b" : "7F109h" : "/lib/x86_64-linux-gnu/libwrap.so.0", "elfType" : 3, "buildId" : "54FCBC5B046F23E7DA7F7FE75B" }, { "b" : "7F1099C47000", "path" : "/usr/lib/x86_64-linux-gno.30", "elfType" : 3, "buildId" : "2A8B93895720278AAE3C1C44E38A217A4A83E134" }, 986A000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "b0E61DA2B4291F7CE5512101F7AE23C93236D4" }, { "b" : "7F109964F000", "path" : "/usrnux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "666B276BD134B0E9579B67D4EE33}, { "b" : "7F10993FE000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.: 3, "buildId" : "372822D0E17BF7C615B6345E5ECEFB1B27BCA57B" }, { "b" : "7F10991E: "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "B39B1523E5F03459BC3E2D3205" }, { "b" : "7F1098FA8000", "path" : "/usr/lib/x86_64-linpi_krb5.so.2", "elfType" : 3, "buildId" : "F53E78DECA2C22259B2FD54DC32C9E3B010BB: "7F1098CA2000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "bu8F3B11CB908F03FA568752126AD1AE3C6D1E" }, { "b" : "7F1098A87000", "path" : "/lib/nu/libresolv.so.2", "elfType" : 3, "buildId" : "9FCED6C1BB3F783375497F9C98FF2CF0 "b" : "7F1098828000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfTypId" : "5BB10EACF0B497C21806AACAAF45C36328E831A3" }, { "b" : "7F1098620000", "pat_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "D27A253ACFC83E639AE80A606A" }, { "b" : "7F109840A000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "ebuildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F10981EC000",b/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C4D728AC02A328326492273FCD" }, { "b" : "7F1097E23000", "path" : "/lib/x86_64-linux-gnu/libc.so.: 3, "buildId" : "5A49BF8DEF435AC3FE9208DF3C6B5622FE347A97" }, { "b" : "7F109AE7: "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2C5922FE5D8F6A77F42E17C591" }, { "b" : "7F1097BF0000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.se" : 3, "buildId" : "A4CF3D2F3AD65050A8199AFC54BD29893EE88902" }, { "b" : "7F109h" : "/usr/lib/x86_64-linux-gnu/librtmp.so.0", "elfType" : 3, "buildId" : "B194D321CA687678D82B712994" }, { "b" : "7F10977BD000", "path" : "/lib/x86_64-linux-gn"elfType" : 3, "buildId" : "61ECB1C9E746126B3CCCC7E82705E539ECAEC3AB" }, { "b" :", "path" : "/usr/lib/libperl.so.5.18", "elfType" : 3, "buildId" : "3B7B9ECBAD55DFF1A9D1C0B8C2" }, { "b" : "7F109721A000", "path" : "/lib/x86_64-linux-gnu/libnsype" : 3, "buildId" : "313621B833833F024B472A4ADDB794FE9AE20188" }, { "b" : "7F1ath" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "367E390FC2443D7EF749ADAA4D" }, { "b" : "7F1096D1D000", "path" : "/usr/lib/x86_64-nutls.so.26", "elfType" : 3, "buildId" : "31C8FF1B8CFAA077ECE92C00C11FCAB72272C5 "7F1096A9D000", "path" : "/lib/x86_64-linux-gnu/libgcrypt.so.11", "elfType" : 3"75E1DDBDFDD5DB837EC6E83928DB65A0A3CE4084" }, { "b" : "7F10967D2000", "path" : "4-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "35C054BECC0C5FB1AE9CDF7CD0" }, { "b" : "7F10965A3000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so : 3, "buildId" : "0788D6F3B7675F5373F8D2F66D8284ADB7D2B7B7" }, { "b" : "7F10963 : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "8D56938AD8E48A131BE1C61F6A" }, { "b" : "7F1096194000", "path" : "/usr/lib/x86_64-linux-gort.so.0", "elfType" : 3, "buildId" : "3D4F9028A10CC566F8CDC9FEFB09683B8A20FB92"F1095F5B000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "bu54C84B882A57EAA28D8F8D92F8D2C8B21A79" }, { "b" : "7F1095D52000", "path" : "/usr/ux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "F284B367B83FC07B7309FA086D" }, { "b" : "7F1095ACA000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26",, "buildId" : "7CC32240A00456FA57B74BFB922E31BA8EEF57C2" }, { "b" : "7F109582900/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "7CF4C34552BC4EE4906A90DE3C" }, { "b" : "7F10955F6000", "path" : "/usr/lib/x86_64-linux-gnu/4", "elfType" : 3, "buildId" : "5F0EF0E1DDE5070F686668B93E6A9BEC44D83220" }, { "1000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buil739A9F5E6A9850B519C95D8A811B63B8EF" }, { "b" : "7F10951CD000", "path" : "/usr/li-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "1477FEC6F18A279343616F89650A273{ "b" : "7F1094F8B000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "ebuildId" : "D4B5C925023E4142D335EEFB6106F47245A3F97C" }, { "b" : "7F1094D86000",b/x86_64-linux-gnu/libgpg-error.so.0", "elfType" : 3, "buildId" : "38CA3EE1AE3841A17FAC217CF7" }, { "b" : "7F1094B82000", "path" : "/lib/x86_64-linux-gnu/libkeyelfType" : 3, "buildId" : "0F03635F97B93D3DACD84F0ED363C56BD266044F" }, { "b" : , "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : F0D0BF62F07347AEF28E040E1A5" }, { "b" : "7F109474B000", "path" : "/usr/lib/x86_6bheimbase.so.1", "elfType" : 3, "buildId" : "F4FDBD38788250E843523FFEA869A4DA933" : "7F1094502000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfTyped" : "C03FF77D3A35A40589C712C74A8597FF532C8ED7" }, { "b" : "7F1094249000", "pathx86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "183703AF20E0C5BC58F79564DB" }, { "b" : "7F1094041000", "path" : "/usr/lib/x86_64-linux-gnu/libffipe" : 3, "buildId" : "C114D2C23BD2F3B1705F37FBF9CA06163C8B89A6" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f109d4a9ba1]
 mongod(+0x2408DB9) [0x7f109d4a8db9]
 mongod(+0x240929D) [0x7f109d4a929d]
 libpthread.so.0(+0x10330) [0x7f10981fc330]
 libc.so.6(gsignal+0x37) [0x7f1097e59c37]
 libc.so.6(abort+0x148) [0x7f1097e5d028]
 mongod(_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j+0x0) [0x7f109ba6ef58]
 mongod(_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj+0x158) [0x7f109b
 mongod(_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE+0x41) [0x7f
 mongod(_ZN5mongo6Client20makeOperationContextEv+0x27) [0x7f109d321357]
 mongod(+0xA4C5D3) [0x7f109baec5d3]
 mongod(+0x2404F65) [0x7f109d4a4f65]
 mongod(_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE+0x364) [0x7f109
 mongod(_ZZN5mongo13duration_castINS_8DurationISt5ratioILl1ELl1000EEEES2_ILl1ELlT0_EEENKUlvE_clEv+0x0) [0x7f109ba0370c]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x87A) [0x7f109baf3f5a]
 mongod(main+0x9) [0x7f109ba71089]
 libc.so.6(__libc_start_main+0xF5) [0x7f1097e44f45]
 mongod(+0xA3C0CF) [0x7f109badc0cf]
-----  END BACKTRACE  -----
vagrant@m103:~$

2020-03-12T19:30:15.200+0000 I STORAGE [initandlisten] exception in initAndListe: Unable to lock the lock file: /var/mongodb/db/mongod.lock (Resource temporari). Another mongod instance is already running on the /var/mongodb/db directory

2020-03-12T19:30:15.200+0000 F - [initandlisten] Invariant failure globalsrc

The way to read is to start from the bottom (ignoring the “BEGIN BACKTRACE” section altogether) and you’ll begin to see a few "F"s (Fatal), afterwhich the next log component is “I”… this is where you focus your efforts. Above is the “I” and the consecutive “F” lines, it should now be easy to see what’s wrong.

PS: Sometimes it may be “E” (Error) instead of “F”.

Thank you for the detailed reply. There is definitely a learning curve to using command line. I’ll apply what you’ve said going forward. Are there any commands I can use to shutdown the mongod instance that is currently running in the background and are there any commands that can be used to verify that nothing is running?

… to verify that something is running you mean? :slight_smile:

The command to find running processes was given in the “Launching Mongod” lab.
If you can’t login, the command to kill running processes was also given in the “Launching Mongod” lab.
If you can login, the command to shutdown the server was discussed in the first lecture, “The Mongod”.

Suggest you review those.

Great! I will look into those. Going through each of these labs is quick. It can be hard to keep up…

Thanks for the great help!

:slightly_smiling_face: :+1:

By the way, suggest you put localhost first in your bindIp option. Happy learning!

1 Like