Garbled text from context.http.get as source of AWS SES emails

[andrefelipe (André Felipe)]

Yes, here are the results:

The console.log is indeed fine when looking at the Realm UI:

But very curious is this test, sending that paragraph inline to SES:

The email gets received correctly!

And again, getting going back to the case, if I get the HTML from here:

const emailHtml = await context.http.get({
    url: 'https://www.typographicposters.com/newsletters/atlas-test'
    // created this new URL with only that thank you note
})
const body = emailHtml.body.text()

const command = new SendEmailCommand({
    FromEmailAddress: from,
    Destination: { ToAddresses: [to] },
    Content: {
      Simple: {
        Subject: {
          Data: 'Test from Realm',
          Charset: 'UTF-8',
        },
        Body: {
          Html: {
            Data: body, // <---------------
            Charset: 'UTF-8',
          },
        },
      },
    },
  })
  await client.send(command)

The email gets received with all the garbled characters again:

And yes, that log at the Realm UI looks fine:

OK, now, what could be problem?

The source HTML is fine, and as I said, have being sending these emails for years. Just got the errors suddenly since last week.

(I will reference this source HTML from now on, it’s shorter: Atlas Test - typo/graphic posters)

[andrefelipe (André Felipe)]

To close another possibility, tried removing all CSS and HTML head.

But the result is the same:

—
Now take a look on here, this is the email source which was sent inline, where the characters were correct:

So the question is, who is fiddling with the characters? AWS or Atlas Functions?

My understanding it’s not AWS, because from the test above, sending the HTML inline, the email got received just fine.

Would be the Atlas Functions dependencies? Something at the transport level?
Please understand that it’s my guess, but makes sense somehow.

Here is the final test. Sending the exact same HTML of the thank you note, which was inlined, but now getting with the context.http.get

[Try_Catch_Do_Nothing]

Is context.http.get supposed to be used as a general purpose HTTP client like you’re using it?
Based on review of the docs, I’m not so sure. In all the examples, BSON.Binary is the expected response.

Context — Atlas App Services

exports = async function() {
  const response = await context.http.get({ url: "https://www.example.com/users" })
  // The response body is a BSON.Binary object. Parse it and return.
  return EJSON.parse(response.body.text());
};

Also, look here:

http.get() — Atlas App Services

body: The binary-encoded body of the HTTP response.

[Try_Catch_Do_Nothing]

Can you try to use axios or fetch instead of context.http.get?

[andrefelipe (André Felipe)]

Atlas functions doesn’t have fetch.

Have Axios 1.3.6 is installed, but just by requiring it error out with:

const axios = require('axios');

failed to execute source for 'node_modules/axios/index.js': FunctionError: failed to execute source for 'node_modules/axios/lib/axios.js': FunctionError: failed to execute source for 'node_modules/axios/lib/core/Axios.js': FunctionError: failed to execute source for 'node_modules/axios/lib/core/dispatchRequest.js': FunctionError: failed to execute source for 'node_modules/axios/lib/adapters/adapters.js': FunctionError: failed to execute source for 'node_modules/axios/lib/adapters/http.js': FunctionError: failed to execute source for 'node_modules/axios/lib/helpers/formDataToStream.js': TypeError: Value is not an object: undefined
	at node_modules/axios/lib/helpers/formDataToStream.js:41:19(118)

Like I said, I have being using context.http.get to get texts just fine, for more than 3 years. response.body.text() gets the body string.

[Try_Catch_Do_Nothing]

The latest version of Axios doesn’t appear to work with Atlas functions, per the below post. Version 1.2.0 appears to work fine.

Use axios in Functions. Throws errors

Hi @ruben_martin_acebedo, The following works: exports = async function(arg) { const axios = require('axios').default; const response = await axios.get('https://jsonplaceholder.typicode.com/posts/1', { responseType: 'json', headers: { "Accept": "application/json", "Accept-Encoding": "deflate" } }); return response.data; };

[andrefelipe (André Felipe)]

OK, installed Axios 1.2.0, but the response is coming as binary.

Of course I tried many different params and methods and researched on Github too, but we are losing the point here. I don’t see the reason to start testing Axios now. The issue is in MongoDB Atlas Functions and as I said have being working for 3+ years.

Now we need the MongoDB team to jump in this issue.

Here is Axios code:

const url = 'https://www.typographicposters.com/newsletters/atlas-test'
const axios = require('axios').default;

const response = await axios.get(url);
return response.data;

And the binary response, tried response.data.toString() also responseType: 'text':

@Try_Catch_Do_Nothing if you find real solutions, tested inside the Atlas Functions, let me know.

[Try_Catch_Do_Nothing]

@andrefelipe

Now we need the MongoDB team to jump in this issue.

So submit a support request then.
If it is an issue that was introduced within Atlas Functions, then I highly doubt a fix will be made for your specific use case (which is why I suggested an alternative like axios).

The bottomline is, it’s not working now, so something changed to break the existing functionality. Was it context.http.get? Was it the aws library? You need to test different scenarios to pinpoint where the issue occurs.

[Kushagra_Kesav]

Hello @andrefelipe,

Thank you for raising the concerns and discussing this. I’m looking into this. Please allow us some time.

We appreciate your patience and understanding.

Regards,
Kushagra

[John_Page]

Everything right up to the email render is correct - the Web page us using UTF-8 and returning the ™ symbol as three bytes (Unicode character inspector: ™) E2 84 A2

In your final email you can see these three bytes as â ¢ - This is Those same three bytes being rendered using either ISO 8859-1 or ISO 8859-15 the single byte representation of non ASCII characters we used before Unicode was a common thing. E2 is â and A2 is ¢ - 84 is not defined in there. (ISO/IEC 8859-15 - Wikipedia)

The issue is that the Email is not correctly defining its contents as utf8 and so the rendered is falling back tio 8 bitl. At least that’s how it looks - if you look at the source of the email (a) is it really these three bytes or has it done some off translation to three unicode characters and (b) what charset is actually defined in the raw email content.

[John_Page]

It’s possible the issue is that when calling the SES endpoint the Request is using ISO8859-1 rather then UTF 8, then the web service would think you were sending it three characters. That seems the most likely issue (we will see this is the internal email contents are quite different) .

Wh you send your Explicit test - you arent sending al the HTML you are only sending part of it (the text part) so it’s possible that somwhere in all the rest of the HTML is a problem.

[John_Page]

I just tested this using the deprecated built-in SES client and it worked correctly.

exports = async function (emailAddress, Content, Subject) {


    const ses = context.services.get('email').ses();
    const emailHtml = await context.http.get({
    url: 'https://www.typographicposters.com/newsletters/atlas-test'
    // created this new URL with only that thank you note
})
const body = emailHtml.body.text()

    const result = await ses.SendEmail({
        Source: "ps-noreply@mongodb.com",
        Destination: {ToAddresses: ["john.page@mongodb.com"]},
        Message: {
            Body: {
                Html: {
                    Charset: "UTF-8",
                    Data: body
                }
            },
            Subject: {
                Charset: "UTF-8",
                Data: "trst Email"
            }
        }
    }).then(r => console.log(EJSON.stringify(r))).catch(e => console.log(e));

};

[andrefelipe (André Felipe)]

Thank you very much @John_Page for narrowing the issue and sharing your knowledge.

Yes, using the deprecated built-in SES client solved the issue and I already changed my App to use that for now.

But it will be deprecated on August 1st. What’s the solution going forward? I understand when you said that when calling the SES endpoint the Request is using ISO8859-1. But how to overcome this?

Note:
— I’ve tried both AWS SDK JS v2 and v3, both have the same issue. Is this an App Service’s Dependency issue?
— Sorry for saying again, but I used to send emails just fine using the solution I shared above (with AWS SDK v3 as dependency with @aws-sdk/client-sesv2 package) why suddenly it stoped working?

Thank you very much again.

[John_Page]

I suspect this is an issue with a dependancy - Looks like in the config for the SDK you can set an optional _HttpHandler value which AWS describe as “Fetch in browser and Https in Nodejs.” - I’ve asked the app services development team to comment on why / if that might be passing the wrong content-type charset in the request.

If you can see a way to explicity set it to @aws-sdk/node-http-handler | AWS SDK for JavaScript v3 it might help.

[andrefelipe (André Felipe)]

I quickly browsed the @aws-sdk/node-http-handler but didn’t find a charset setting yet, will try to give attention later today.

Again, thanks for your effective support.

[John_Page]

Quicjk update - can 100% reproduce your issue unsure where it is happening though, will debug now.

[andrefelipe (André Felipe)]

Thank you very much @John_Page

[andrefelipe (André Felipe)]

Hello @John_Page do you have news on this case?

[andrefelipe (André Felipe)]

Hello @John_Page do you have news on this case? The 3rd Party Services will be deprecated in just 3 weeks.

[John_Page]

Hi André,
I replied to you about this on the 12th of June by email. We can reproduce the issue but as that context.http service is being deprecated it won’t get fixed. I gave you a workaround for it but also now the latest version of Axios works and so we woudl expect you to move to that. If you are still having issues using Axios instead of http.get then reach out and I can help again.