Mongo Realm 3rd Party Services denial of wallet protection / mitigation / prevention

MongoDB Atlas App Services & Realm
#1

Hi Mongo

(recreated topic as more info and could not edit original)

I have been experimenting with Realm 3rd Party Services to create incoming webhooks, Based on the scalability I am particularly interested in preventing attacks on the endpoint and how to prevent an attacker requesting the endpoint over and over and causing a “denial of wallet” attack.

I expected that using the ‘Can Evaluate’ expression combined with a list of abusive IP’s would mitigage basic attacks by blocking the request from being procesed; however from testing this does not seem to be the case - I can prevent the funtion from running so reducing Data Transfer and Compute Runtime but I cannot prevent the request from being billed.

Can you answer the following:

  1. Is there any way for me to completely deny a request to an endpoint and not be charged for the request?

  2. Does Mongo have any recomended practices for mitigating endpoint attacks (for example you provide a json expression showing allow if in set of ip but this is useless if the request is still charged)?

  3. What protection does Mongo provide at the Realm platform level to mitigate platform level / large scale ddos attacks on endpoints and how does this impact our billing if our endpoints are targeted?

  4. What protection does Mongo provide at the Realm platform level to mitigate targetted ddos attacks
    on our endpoints and how does this impact our billing?

For example Firebase provide a platform level ddos protection that prevents billing of clients if there was a platform level ddos attack; however this is retrospective and would only apply in case 3, they provide support for case 2 but very little for case 1 which can result in unessecary billing for clients.

Thank you,

I have included case studies and test setup below.

Case studies:

Case 1: A basic attacker requesting an endpoint over and over again using a basic load testing tool running 50req/s which would be processed by Realm and result in an additional 4.3M requests per day:

    ( ( 50 req/s * 86400 s/day ) / 1000000 ) * 0.3 = $1.29 / day

In this case I could identify the attacker and prevent the function being run for them but I cannot completely deny the requests; this would result in a $40/month cost when normal load might only run < 5$

Case 2: A basic distributed attack could request an endpoint over and over again lets say a cluster of 100 bots requesting the endpoint at 50 req/s which would be processed by Realm and result in ~430M requests per day:

    ( (  100 bots * 50 req/s * 86400 s/day  ) / 1000000 ) * 0.3 = $129+/day

Here again the IPs could be identified but not completely denied and we would have to terminate the app to prevent a bill of $4000+ for the month.

Case 3: A sophisticated attacker could distribute the attack using multiple rotating bots and target all of the apps endpoints - assuming the app has 10 endpoints and using a sustained 200 bot attack:

    ( ( 10 endpoints * 200 bots * 50 req/s * 86400 s/day ) / 1000000 ) * 0.3 = $2592 /day

Test setup:

My testing setup using the UI for setup and leaving all defaults unless specified:

  1. Create an Atlas cluster
    => AWS, Frankfurt (eu-central-1), 4.4, M0, Name = cluster
  2. Create a Realm app connected to the cluster
    => Name = app, link to cluster, set global
  3. Create a 3rd Party Service
    => 3rd Party Services => Add a Service => Type = http, Name = test
  4. Add an incoming webhook to the service
    => Add Incoming Webhook => Name = test, HTTP Method = GET

Test 1 - Setup endpoint to return ‘Hello World’, expect request to be processed and request billed.
Outcome: Endpoint returns and 1 request is billed

% curl https://webhooks.mongodb-realm.com/api/client/v2.0/app/APP/service/test/incoming_webhook/test
    "Hello World!"

Test 2 - Set Can Evaluate to always false expect request to be denied
=> Edit Webhook => Can Evaluate = { “%%true”: false }
Outcome: Endpoint returns error but 1 request is still billed

% curl https://webhooks.mongodb-realm.com/api/client/v2.0/app/APP/service/test/incoming_webhook/test
    {"error":"incoming webhook evaluation blocked by CanEvaluate","link" ... }
1 Like
#2

I am actually interested in the answer to this question as well.

Just to clarify, this attack is only possible using a Webhook, which I would have to activate first? So, as long as I only use Realm Sync with Atlas, this does not need to concern me, right?

Now if i do use webhooks, I was expecting that any billable service would require some form of authentication or built in security. The webhook is intended to link to a third party service and the documentation describes authentication:

Is it really possible to get billed for webhooks without authentication or without some sort of security token?

#3

Christian_Wagner, short answer - yes that is correct.

Longer answer - we need Mongo to comment and confirm but from my testing I beleive the following is the case:

  • To be venerable you would need to create / enable a webhook - if you do not have any then there are no endpoints to exploit therefore no excessive requests can be made.

  • When I ran my tests I left the defaults in place - my endpoint was using Authentication = System because I wanted the function to be able to access all data and the endpoint to be plubic then setup my own logic within the function; however I also want to block obviously abusing users for example by IP, ideally I was expecting the following:

guest => endpoint => function runs and responds with redacted data
user => endpoint => function runs and responds with full data
abusive ip => endpoint => denied by can evaluate / other mechanism, request is denied and function does not run

So far I can only get the function to not run, not the request to be denied, I have tried using both service rules and webhook can evaluate to deny the request.

I will run a quick test using an Authentication = Application Authentication and see if I get different results.

1 Like
#4

UPDATE: When setting Authentication = Application Authentication Realm returns a 400 for a straight request with no auth specified but the requests are still charged; therefore it appears that setting up any incoming webhook is a potential vector for abuse regardless of the authentication or can evaluate settings anyone sending a request to your endpoint regardless of if they are authenticated cause you to be billed for one request.

1 Like
#5

Hey @mba_cat - these are really good points. Some additional things to consider with rate-limiting/throttling Realm API requests -

  1. At the moment Realm will rate limit requests/s if they exceed a certain threshold to prevent requests, and case 2 / case 3 are improbable due to that rate limiting.

More configuration around rate limiting can also be an added on with an API gateway or a proxy today, but configurable rate limiting in Realm sn’t out of the question. You can vote for the feature here - Configure rate limit – MongoDB Feedback Engine

  1. Answered above, we do have set rate-limiting in place and will stop accepting requests after that.

  2. (and 4) - We’re covered under AWS Shield Advanced which should help mitigate DDoS attacks and that we’re in the process of adding more network security features (IP Access List, PrivateLink).

1 Like