Atlas API is not accepting request from terraform cloud IaC runners

Dear MongoDB community
I am deploying an atlas mongo cluster from terraform, and recently I got this issue in a couple of our terraform cloud runs by interaction with atlas mongodb API:

Error: error getting Team information: GET https://cloud.mongodb.com/api/atlas/v1.0/orgs/{ORG_ID}/teams/{TEAM_ID}: 403 (request "IP_ADDRESS_NOT_ON_ACCESS_LIST") IP address 3.230.120.28 is not allowed to access this resource.

The first thing came up to my mind was that atlas API is not accepting incoming connections from terraform cloud hosted runners where I am running the pipelines to setup/update atlas mongodb infrastructure. This due to the IP address from terraform runner is not allowed on atlas API access list.

That I tried is keeping in mind the API range list, I’ve added the range gotten from this curl request to my mongo API access list, without success, since the range is wide, sometimes the IP I got from terraform cloud run is not included on the api, notifications, sentinel or vcs

> curl \
  --request GET \
  -H "If-Modified-Since: Tue, 26 May 2020 15:10:05 GMT" \
  https://app.terraform.io/api/meta/ip-ranges
{
	"api": ["75.2.98.97/32", "99.83.150.238/32"],
	"notifications": ["52.86.200.106/32", "52.86.201.227/32", "52.70.186.109/32", "44.236.246.186/32", "54.185.161.84/32", "44.238.78.236/32"],
	"sentinel": ["52.86.200.106/32", "52.86.201.227/32", "52.70.186.109/32", "44.236.246.186/32", "54.185.161.84/32", "44.238.78.236/32"],
	"vcs": ["52.86.200.106/32", "52.86.201.227/32", "52.70.186.109/32", "44.236.246.186/32", "54.185.161.84/32", "44.238.78.236/32"]
}

Not sure if this is something related from Atlas Mongodb API side, the thing is until two days ago (and for long time) this process on terraform cloud was working well getting connections with Atlas API and I was not getting the 403 (request "IP_ADDRESS_NOT_ON_ACCESS_LIST") error until now.

I know the terraform IP range list is variable, from time to time, and also I already post a question on terraform hashicorp community and people there says here if it was working and now it doesn’t, perhaps it has to do with some change from MongoDB side

This seems like a change in your MongoDB organization or the Atlas MongoDB API then, if it used to work and now requires a specific IP allowlist. The error is returned in Terraform but is a response from that service, not Terraform itself.

I am really confused about the origin of this error, it looks like Atlas API is not accepting incoming connections from terraform cloud runner (it is a public runner hosted on terraform side) and I have to whitelist the IP address or range of them in Atlas API Access list, but the thing is this range is variable and is not included on the ranges gotten with my curl command.

I just wanted to come up here just for the record and know your thoughts about that.
Has something changed on ATLAS API for incoming connections? Why it was working well and suddenly it doesn’t?

I will appreciate your thoughts.

Update:

The same is happening with Github action runners

Error: error creating Team information: POST cloud.mongodb.com/api/atlas/v1.0/orgs/5e85b39c70ff62663a3df63a/teams: 403 (request "IP_ADDRESS_NOT_ON_ACCESS_LIST") IP address 40.65.198.72 is not allowed to access this resource.

However, for the Github actions case, the IP address range used is the same for Azure cloud.
I can figure it out here for all Github ecosystem, and I can get a quite extensive IP addresses array list for Github actions by doing this request

curl \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/meta

The IP address shown in the error is changing every time, I could add all the IP ranges on Atlas API access list via API (the CIDR blocks are 1639 items), but again, not sure why this situation comes up now and not before when it was working well.

Hi @Bernardo_Garcia,

I’ve managed to replicate a similar error response, although not from terraform directly.

I have attempted a simple curl request to retrieve a list of database users from my IP address which is NOT on the API key access list in Atlas (I have redacted the IP address information in the below response):

{
  "detail" : "IP address xyz.xy.xyz.xyz is not allowed to access this resource.",
  "error" : 403,
  "errorCode" : "IP_ADDRESS_NOT_ON_ACCESS_LIST",
  "parameters" : [ "xyz.xy.xyz.xyz" ],
  "reason" : "Forbidden"
}

The error appears to be caused when the requesting client’s IP address is not on the API access list. The documentation linked should provide instructions on how to view the details of an API key including the access list. From your example, can you ensure the IP 3.230.120.28 is on the Access List of the API key being used to perform the request?

You can perhaps add a IP range using CIDR notation to the Access List in Atlas for that specific API key that includes the possible IP’s of the client performing the request if you wish to as well.

Hi @Jason_Tran, thanks for your reply.

Indeed, is clear for me the IP ranges for terraform and Github hosted runners are not being whitelisted at the Atlas API access list, and in this case they should.
I have enabled the API access list for my organization and even I added a bunch of those ranges arrays on it. And of course, I would have to add the entire ranges to make them works, but the thing is, I want to share with you the following particular situation:

Until some days ago I didn’t need to whitelist the terraform and Github runners IP ranges on the Atlas API access list. My pipelines were working well for quite a time in that way, without atlas asking me for this.

Why suddenly Atlas API is denying me access now to my terraform and Github runners, but before it was allowing my runners to interact with it in a normal way?

I think it should be some internal stuff from Atlas API (?)

I have a hypothesis that I would like to share with you:

Last March 1, I accidentally delete my atlas cluster from an automatic process doing some tests from my local machine.
My atlas cluster is created via terraform, and I am running a terratest process to verify my infrastructure (atlas mongoDB cluster included) is being created in a proper way and after that, terratest delete the test environment created for this tests.
My mistake was to point from terratest to the current cluster in use and not to the test environment created in runtime, it was unfortunate indeed.

Then my hypothesis is:

Is Atlas MongoDB team/platform/process blacklisting the terraform hosted runners IP ranges and Github actions hosted runners IP ranges I have been using as a protection measure?
I mean perhaps they could be thinking this unfortunate situation from my side was an attack and not something from me as an owner of the cluster?

I am wondering this since from more than 1 year ago I never did need to whitelist terraform and GitHub runners IP ranges on the Atlas API access list for my organization. And after this situation, Atlas is simply asking me for doing it.

I wanted to tell you this, perhaps you can help me to think about it.
I mean for me is quite strange, why Atlas API does not demanded to whitelist the Ip ranges from the beginning and now it does it, just after this incident?

If this situation (my hypothesis) ending up affirmative … is there a chance to go with the support team, explain the situation, and perhaps reverts this behavior?

Actually, my atlas cluster is within an M10 plan, but the support for it is within the basic Free plan.
Just in case, what could you suggest to me to proceed in this case?

Hi @Bernardo_Garcia,

The IP address shown in the error is changing every time

I was presuming due to the above statement that there was a chance that the outgoing IP’s the client was changing were not on your Atlas API access list. This would also line up with the 403 error you are getting as a response.

I have enabled the API access list for my organization and even I added a bunch of those ranges arrays on it.

I understand you have stated you have added a bunch of the ranges arrays onto the Atlas API access list. However, I cannot see 3.230.120.28 in the response within any of the arrays. Please note, I am checking the api , notifications , sentinel or vcs arrays but please let me know if I am looking at the wrong array.

I wanted to tell you this, perhaps you can help me to think about it.
I mean for me is quite strange, why Atlas API does not demanded to whitelist the Ip ranges from the beginning and now it does it, just after this incident?

To better troubleshoot the issue, please provide the following information:

  1. Whether the particular IP 3.230.120.28 is currently existing in your Atlas API access list (for the particular API key you’re using) or had it existed when you had performed the initial request to Atlas?
  2. Confirm the usage of server / client with outgoing 3.230.120.28. Is the host with outgoing IP 3.230.120.28 a machine you’re currently managing on your own infrastructure? Is it an outgoing address from Terraform Cloud that may be performing the request to Atlas?
  3. Whether or not the IP 3.230.120.28 in the response has changed each time you attempt to perform a request from the same client.

Look forward to hearing from you.
Jason

Hi @Jason_Tran, Thanks for getting back to me. I will try to provide as much info I can.

Indeed the ip ranges terraform runners use are not within the request results on api , notifications, sentinel, or vcs arrays.
Terraform guys also highlight this fact

I tried to add it but it does not work in that way, as we have been checking, terraform runners IPs range change every run to a different IP address. So does not make sense to add just that particular IP 3.230.120.28 in my Atlas API access list, since the next run the client will use another one with a different range even.

It is an outgoing address from Terraform Cloud hosted runners, its perform the request to Atlas. Those runners are the public runners people use when people don’t pay Terraform selff-hosted runners in an enterprise plan

The IP for the terraform runner change every attempt I run my checks from there.

I wanted to tell you about the deletion situation because I am still wondering, why Atlas was not requiring me to whitelist my terraform runners before (almost along 1 year) and now it does?

Thank you for providing me with that information @Bernardo_Garcia,

One thing that I can think of that could be a possible reason for why it was working at one point (without requiring IP whitelisting against the IP) is that the Require IP Access List for Public API setting within your Organization Settings page was toggled to On (from Off).

Now, we can check this by going to the Organization Activity Feed section of your Atlas Org since you have stated this is an Organization API key being used.

You can filter for the following events:

  1. Require IP Access List for Public API Enabled
  2. IP Access List for Public API Not Required

Check out the below example for the above events:

There should then be a column on the right hand side called Creation Info where you can see when these or if these settings were changed and by whom.

Look forward to hearing from you.
Jason

Indeed I had to enable IP access list for public API (it was off) because of this error. I mean by the time I got this error from terraform and github actions runners it was off. Are you telling me if I put it to off again, the public API will accept any incoming connections (including my terraform runners)?

Hi @Bernardo_Garcia,

Indeed I had to enable IP access list for public API (it was off) because of this error. I mean by the time I got this error from terraform and github actions runners it was off.

Thanks for getting back to me with that info. I have done some testing with an Organization API key. Please see my below test cases and results:

  1. With the Require IP Access List for Public API setting configured to OFF and having 0 entries in the API Access List, I am able to perform the request as per normal.

  2. With the Require IP Access List for Public API setting configured to OFF and having an entry in the API Access List where the IP does not include the outgoing IP of the client performing the request, I get a response with the following error code & error:
    "error" : 403,
    "errorCode" : "IP_ADDRESS_NOT_ON_ACCESS_LIST"

  3. With the Require IP Access List for Public API setting configured to ON and have 0 entries in the API Access List, I get a response with the following error code & error:
    "error" : 403,
    "errorCode" : "ORG_REQUIRES_ACCESS_LIST"

Are you telling me if I put it to off again, the public API will accept any incoming connections (including my terraform runners)?

From my number 2 test case, if it were to be turned off I would assume you currently have some entries against the Access List so you would still hit the same error. If you have a test organization and environment where you can have this setting off with 0 entries, it would probably be best to try it out with this before attempting to make any changes on production.

Do you know if there were any entries in the Access List when the setting was OFF? If there were 0 entries an an entry was then added to the Access List with the Setting OFF, It would line up with 2. in my test case above.

You can filter to see if the Access List for a particular key was added to by filtering for the API Key Whitelist Entry Added event in your Organization Activity Feed.

An example of this entry in the feed would look like:
175.30.100.20 has been added to the access list for the API Key with public key ABCDEFGH.

Best Regards,
Jason

1 Like

@Jason_Tran It makes sense that you’ve mentioned. I was playing around last March 9th with the API in order to create restore jobs via curl command from my local machine, (such as this interaction question from my side can proof it).
Was in that time when I had to enable API access list (not because the error with terraform runners - I was wrong, sorry - ) in order to allow my home IP address to execute the curl request to create restore jobs.
So after that when I executed my pipelines from terraform and github actions, indeed according to the three cases you’ve mentioned, that is why the system is requiring from me to allow the IPs ranges, because from that time my API Access list has been activated and with entries.

I am going to disable it, but now I have 1700 entries approx … Do you know how can I delete them in a bulk way via API?
I just found the example request to delete one entry

UPDATE:

@Jason_Tran It works now.

  • I removed the access list entries I had with a loop like this
#!/bin/bash

ip_addresses=("13.64.0.0%2F16" "13.65.0.0%2F16" "...")

len=${#ip_addresses[@]}
for ((i=0; i<=$len; i++))
do
    echo ${ip_addresses[$i]}
    curl --user "{PUBLIC_KEY}:{PRIVATE_KEY}" --digest --include \
     --header "Accept: application/json" \
     --header "Content-Type: application/json" \
     --request DELETE "https://cloud.mongodb.com/api/atlas/v1.0/orgs/{ORG_ID}/apiKeys/{API_KEY_ID}/accessList/${ip_addresses[$i]}"
done
  • I toggled off atlas api access list from my organization settings and now terraform and Github actions runners are able to contact Mongo Atlas API. Thanks for the hints in this situation
2 Likes

Really glad to hear that it works now @Bernardo_Garcia.

Thanks for also posting your own solution to the removal of IP entries using the loop.

2 Likes

Thanks for your support @Jason_Tran, :slight_smile:

2 Likes

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.