Hi everyone. I’m new here, and fairly new to MongoDB, but have been enjoying working with it a lot recently.
I’m running the instance that this question refers to on a DigitalOcean server, using the CommunityEdition 4.4.9 on Ubuntu 21.04. Any help would greatly be appreciated!
I’m running a fairly lengthy Python3 script which:
- connects to a remote redshift PSQL server
- iterates through a number of IDs for target data to collect from said server
- uses a bunch of heuristics to pull in all the linked data for this object sitting in different tables in the PSQL. This has to be done at the ID-level and can’t be batched, unfortunately
- pulls this data into memory on my machine (DigitalOcean Ubuntu 21.04 droplet, 2 CPU cores, 2 GB Ram), using
pandasas a mid-way point to turning the data into alist of dicts. - does some basic processing on this data to bring it into the right format (list and dict comprehensions)
- runs the
update_onemethod on a collection in a MongoDB instance running on the DigitalOcean machine (MongoDB 4.4.9 Community). The average size of a document (once inserted into Mongo) is around 100kb.
For most of the time, this works seamlessly. It’s averaging about ~10 inserts per minute, which, given all the connections and costly data pulls it needs to perform, is fine by me. So, I was thinking I will just leave this alone and complete its ~25k iterations (the number of unique IDs in the PSQL I want to collect) by running the Python script calling all this wrapped in nohup.
However, the script invarariably fails, and every time it does, 2 things happen:
- The log file output of the script shows the following output:
File "/root/neu-pipeline/data_collection/an_pull_email_data.py", line 317, in <module>
result = mycoll.update_one(the_filter, {'$set': out_dict}, upsert=True)
File "/usr/local/lib/python3.9/dist-packages/pymongo/collection.py", line 1019, in update_one
self._update_retryable(
File "/usr/local/lib/python3.9/dist-packages/pymongo/collection.py", line 868, in _update_retryable
return self.__database.client._retryable_write(
File "/usr/local/lib/python3.9/dist-packages/pymongo/mongo_client.py", line 1497, in _retryable_write
with self._tmp_session(session) as s:
File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.9/dist-packages/pymongo/mongo_client.py", line 1829, in _tmp_session
s = self._ensure_session(session)
File "/usr/local/lib/python3.9/dist-packages/pymongo/mongo_client.py", line 1816, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.9/dist-packages/pymongo/mongo_client.py", line 1766, in __start_session
server_session = self._get_server_session()
File "/usr/local/lib/python3.9/dist-packages/pymongo/mongo_client.py", line 1802, in _get_server_session
return self._topology.get_server_session()
File "/usr/local/lib/python3.9/dist-packages/pymongo/topology.py", line 496, in get_server_session
self._select_servers_loop(
File "/usr/local/lib/python3.9/dist-packages/pymongo/topology.py", line 215, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused,
Timeout: 30s, Topology Description: <TopologyDescription id: 61759f937123a7ca47f993bd,
topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type:
Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>
- In addition to this, this will invariably mean that running
systemctl status mongod
returnsfailed.
There is no way to reproduce the exact conditions that lead to this error, and there is no particular configuration of input data that causes this. Much rather, I think that mongod is hitting the ceiling of resources available on this machine. Specifically, I think this is a memory issue. I arrived at this suspicion based on viewing top while running the script.
Now, here’s my questions:
- Is there a good way of handling this exception from within the python script (i.e. a
try-exceptblock which, upon encountering this exception, e.g. flushes the memory, restartsmongodand tries again? - Is there anything that I can do in terms of the configuration of my MongoDB server (beyond moving to Atlas or another managed (paid-for) version of the server), such as set limits to memory usage etc. etc. ?
- Are there external factors (e.g. third parties accessing the MongoDB through SSH tunnels, guis, or other scripts also hitting it) which may impact upon this occurring?
Thanks in advance for your help!