We have a Flask + MongoDB web application based on MongoEngine. We are trying to enforce a user-level limit in a thread safe manner, meaning we need to prevent multiple concurrent calls to an endpoint by the same user.
Looking at older threads and MongoEngine reference, we came up with this solutions. Relevant threads we found:
It’s not possible to lock a mongodb document. What if I need to?
Our solution is based on a boolean field in the user document which is used as a mutex. The lock field is set to True with an atomic modify, operations are performed by the endpoint and then the lock is released (False).
To enforce this we have a lock attribute in the User model:
User
- ...
- plans
- execution_plan
- ...
- lock (bool)
The following code decorates the endpoint.
def execution_plans_decorator(f):
@wraps(f)
def decorated_function(*args, **kwargs):
user: User = User.objects.get(_id=get_jwt_identity())
execution_plan = user.plans.execution_plan
timeout = time.time() + current_app.config['PLAN_LOCK_TIMEOUT']
modified = False
while time.time() < timeout:
# acquire lock with an atomic query
modified = user.modify(
query={f'plans__execution_plan__lock': False},
**{f'set__plans__execution_plan__lock': True}
)
if modified:
logger.info(f'lock acquired for execution_plan')
break
time.sleep(0.5)
if not modified:
logger.info(f'could not acquire lock for execution_plan')
return jsonify({'msg': 'Too many concurrent requests'}), 429
# if a condition is not met, block the endpoint
if not condition:
execution_plan.lock = False
user.save()
return jsonify({
'msg': f'Max number of executions reached. Upgrade execution plan'
}), 402
# execute the decorated function
try:
rv = f(*args, **kwargs)
except Exception:
# re-raise exception
raise
finally:
# always release lock
execution_plan.lock = False
user.save()
return rv
return decorated_function
We believe the code is formally correct and proved to work without any issue. However, on rare occasions in production (once every ~10.000 requests), the lock is not released and is marked as true. This locks all subsequent requests to the endpoint for a particular user, requiring manual intervention.