Unauth Error on Moving Data from DataLake to S3

Hello, im new to Mongo but was tasked with creating a DataLake and getting that Data into S3. Im following along with the guide How to Automate Continuous Data Copying from MongoDB to S3

When I try to test the Export to S3 trigger I get the following error.

I tried contacting support and they just suggested I add

    "Effect": "Allow",
    "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
    ] 

To the role policy in aws but its already there. It was in the initial policy that was generated on the DataLake creation.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "S3 Bucket",
        "S3 Bucket*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "S3 Bucket",
        "S3 Bucket"
      ]
    }
  ]
}

Not sure where the error lies on the AWS or Mongo side. Any help would be greatly appreciated. Thanks!

1 Like

Hi @JoeKarlsson – someone having permission issues following along with your post.

2 Likes

Thank you @Andrew_Morgan @JoeKarlsson

1 Like

Hey @Chase_Russell! First of all, thanks for coming to the MongoDB Community and asking this great question! Let’s see if I can help you get this working.

I’m guessing the issue lies somewhere with the AWS Integration with Atlas. Which isn’t too surprising since AWS auth can get pretty confusing. Could you send me a screenshot of your AWS IAM Role Access page? I just want to make sure it’s setup and pointing to the right place. Here’s mine:

1 Like

Hi Joe! Thank you very much for your reply. Great article by the way!

Here are my settings:

Can you show me the linked data sources on your Atlas Trigger? It should be linked to your Atlas Data Lake.

Good Morning Joe, yep

Interesting. :thinking::thought_balloon: What happens if you rerun the permissions script that Atlas gives you through the AWS CLI? Does it give any errors? Can you show the AWS IAM profile?

Hey @Chase_Russell apologies for the delayed response here!

Based on the error message you got, it’s not actually the IAM user that’s an issue. It’s the permissions on the Database User that you’re connecting to your Data Lake with. Since you’re using an Atlas Trigger for this, it’s actually using a system user to connect to the Data Lake, which had the wrong permissions set.

This was a bug and should have been resolved a while ago though. Can you confirm it’s working now?

1 Like

Hi Benjamin, thank you for the reply. We still are experiencing this error

Ah, I see @Chase_Russell . I think that error was a red herring, we’ll fix that.

So in your Realm Trigger you are connecting to your Data Lake, so the name spaces that you would reference to access the data in your cluster are the ones you’ve defined in your Data Lake to reference the cluster. So in this example, to access the name space you’re trying to you should be using the namespace you’ve defined in the data lake (not the name of the db and collection in the cluster). Also the realm trigger will be pulling data from all the sources referenced under the virtual data lake collection so if you have multiple sources of data (e.g. S3 + Atlas) the data is going to be coming from both.

Does that make sense? If not, we can setup a call and I can walk you through it? Calendly - Benjamin Flast

Hi Benjamin, sorry for the delayed reply. Here are my DataLake settings. Based on the naming I believe I am calling them properly in the trigger but could be wrong:

Hey @Chase_Russell, based on your Storage Config, in the trigger should be specifying “MessageCenter” and “MessageInfo” (not sample_airbnb.listingAndReviews)

Following up. You may also need to create a new Data Lake to take advantage of the change we made to resolve the initial issue. But I can confirm, setup with a new Data Lake you will no longer receive this error.

Thank you again for the reply Benjamin. I made the changes you recommended and its still showing the error. I think ill start from scratch. The initial go was months ago and the steps are kind of foggy. Itd be good to start over again for the practice alone.

@Chase_Russell sorry for the delay here. Do you mind sending me an email at benjamin.flast@mongodb.com? I’d like to double check on the Data Lake and see what’s going on.

Thanks!