ServerSelectionTimeoutError using pymongo, mongod.service crashes

Hi everyone. I’m new here, and fairly new to MongoDB, but have been enjoying working with it a lot recently.
I’m running the instance that this question refers to on a DigitalOcean server, using the CommunityEdition 4.4.9 on Ubuntu 21.04. Any help would greatly be appreciated!

I’m running a fairly lengthy Python3 script which:

  • connects to a remote redshift PSQL server
  • iterates through a number of IDs for target data to collect from said server
  • uses a bunch of heuristics to pull in all the linked data for this object sitting in different tables in the PSQL. This has to be done at the ID-level and can’t be batched, unfortunately
  • pulls this data into memory on my machine (DigitalOcean Ubuntu 21.04 droplet, 2 CPU cores, 2 GB Ram), using pandas as a mid-way point to turning the data into a list of dicts.
  • does some basic processing on this data to bring it into the right format (list and dict comprehensions)
  • runs the update_one method on a collection in a MongoDB instance running on the DigitalOcean machine (MongoDB 4.4.9 Community). The average size of a document (once inserted into Mongo) is around 100kb.

For most of the time, this works seamlessly. It’s averaging about ~10 inserts per minute, which, given all the connections and costly data pulls it needs to perform, is fine by me. So, I was thinking I will just leave this alone and complete its ~25k iterations (the number of unique IDs in the PSQL I want to collect) by running the Python script calling all this wrapped in nohup.

However, the script invarariably fails, and every time it does, 2 things happen:

  • The log file output of the script shows the following output:
 File "/root/neu-pipeline/data_collection/", line 317, in <module>
    result = mycoll.update_one(the_filter, {'$set': out_dict}, upsert=True)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1019, in update_one
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 868, in _update_retryable
    return self.__database.client._retryable_write(
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1497, in _retryable_write
    with self._tmp_session(session) as s:
  File "/usr/lib/python3.9/", line 117, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1829, in _tmp_session
    s = self._ensure_session(session)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1816, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1766, in __start_session
    server_session = self._get_server_session()
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1802, in _get_server_session
    return self._topology.get_server_session()
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 496, in get_server_session
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 215, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, 
Timeout: 30s, Topology Description: <TopologyDescription id: 61759f937123a7ca47f993bd,
 topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type:
 Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>
  • In addition to this, this will invariably mean that running
    systemctl status mongod
    returns failed.

There is no way to reproduce the exact conditions that lead to this error, and there is no particular configuration of input data that causes this. Much rather, I think that mongod is hitting the ceiling of resources available on this machine. Specifically, I think this is a memory issue. I arrived at this suspicion based on viewing top while running the script.

Now, here’s my questions:

  • Is there a good way of handling this exception from within the python script (i.e. a try-except block which, upon encountering this exception, e.g. flushes the memory, restarts mongod and tries again?
  • Is there anything that I can do in terms of the configuration of my MongoDB server (beyond moving to Atlas or another managed (paid-for) version of the server), such as set limits to memory usage etc. etc. ?
  • Are there external factors (e.g. third parties accessing the MongoDB through SSH tunnels, guis, or other scripts also hitting it) which may impact upon this occurring?

Thanks in advance for your help!

And, in addition to the above thing, it is now crashing much faster, and giving the following error message:

Traceback (most recent call last):
  File "/root/neu-pipeline/data_collection/", line 317, in <module>
    result = mycoll.update_one(the_filter, {'$set': out_dict}, upsert=True)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1019, in update_one
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 868, in _update_retryable
    return self.__database.client._retryable_write(
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1498, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1384, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 1416, in _retry_internal
    return func(session, sock_info, retryable)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 860, in _update
    return self._update(
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 829, in _update
    result = sock_info.command(
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 699, in command
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 683, in command
    return command(self, dbname, spec, slave_ok,
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 150, in command
    reply = receive_message(sock_info, request_id)
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 195, in receive_message
    _receive_data_on_socket(sock_info, 16, deadline))
  File "/usr/local/lib/python3.9/dist-packages/pymongo/", line 292, in _receive_data_on_socket
    raise AutoReconnect("connection closed")
pymongo.errors.AutoReconnect: connection closed

Any updates on this? I am facing the same error with pymongo==3.7.2

This sounds like a server issue. Please post the server’s log at the time the mongod crashes so that someone can advise.

You could also try upgrading to the latest pymongo release, 4.1.1, or the latest pymongo 3.X release, 3.12.3, but this is unlikely to resolve the issue.