Garbled text from context.http.get as source of AWS SES emails

Atlas functions doesn’t have fetch.

Have Axios 1.3.6 is installed, but just by requiring it error out with:

const axios = require('axios');

failed to execute source for 'node_modules/axios/index.js': FunctionError: failed to execute source for 'node_modules/axios/lib/axios.js': FunctionError: failed to execute source for 'node_modules/axios/lib/core/Axios.js': FunctionError: failed to execute source for 'node_modules/axios/lib/core/dispatchRequest.js': FunctionError: failed to execute source for 'node_modules/axios/lib/adapters/adapters.js': FunctionError: failed to execute source for 'node_modules/axios/lib/adapters/http.js': FunctionError: failed to execute source for 'node_modules/axios/lib/helpers/formDataToStream.js': TypeError: Value is not an object: undefined
	at node_modules/axios/lib/helpers/formDataToStream.js:41:19(118)

Like I said, I have being using context.http.get to get texts just fine, for more than 3 years. response.body.text() gets the body string.

The latest version of Axios doesn’t appear to work with Atlas functions, per the below post. Version 1.2.0 appears to work fine.

OK, installed Axios 1.2.0, but the response is coming as binary.

Of course I tried many different params and methods and researched on Github too, but we are losing the point here. I don’t see the reason to start testing Axios now. The issue is in MongoDB Atlas Functions and as I said have being working for 3+ years.

Now we need the MongoDB team to jump in this issue.

Here is Axios code:

const url = 'https://www.typographicposters.com/newsletters/atlas-test'
const axios = require('axios').default;

const response = await axios.get(url);
return response.data;

And the binary response, tried response.data.toString() also responseType: 'text':

@Try_Catch_Do_Nothing if you find real solutions, tested inside the Atlas Functions, let me know.

So submit a support request then.
If it is an issue that was introduced within Atlas Functions, then I highly doubt a fix will be made for your specific use case (which is why I suggested an alternative like axios).

The bottomline is, it’s not working now, so something changed to break the existing functionality. Was it context.http.get? Was it the aws library? You need to test different scenarios to pinpoint where the issue occurs.

Hello :wave: @andrefelipe,

Thank you for raising the concerns and discussing this. I’m looking into this. Please allow us some time.

We appreciate your patience and understanding.

Regards,
Kushagra

Everything right up to the email render is correct - the Web page us using UTF-8 and returning the ™ symbol as three bytes (Unicode character inspector: ™) E2 84 A2

In your final email you can see these three bytes as â ¢ - This is Those same three bytes being rendered using either ISO 8859-1 or ISO 8859-15 the single byte representation of non ASCII characters we used before Unicode was a common thing. E2 is â and A2 is ¢ - 84 is not defined in there. (ISO/IEC 8859-15 - Wikipedia)

The issue is that the Email is not correctly defining its contents as utf8 and so the rendered is falling back tio 8 bitl. At least that’s how it looks - if you look at the source of the email (a) is it really these three bytes or has it done some off translation to three unicode characters and (b) what charset is actually defined in the raw email content.

It’s possible the issue is that when calling the SES endpoint the Request is using ISO8859-1 rather then UTF 8, then the web service would think you were sending it three characters. That seems the most likely issue (we will see this is the internal email contents are quite different) .

Wh you send your Explicit test - you arent sending al the HTML you are only sending part of it (the text part) so it’s possible that somwhere in all the rest of the HTML is a problem.

I just tested this using the deprecated built-in SES client and it worked correctly.


exports = async function (emailAddress, Content, Subject) {


    const ses = context.services.get('email').ses();
    const emailHtml = await context.http.get({
    url: 'https://www.typographicposters.com/newsletters/atlas-test'
    // created this new URL with only that thank you note
})
const body = emailHtml.body.text()

    const result = await ses.SendEmail({
        Source: "ps-noreply@mongodb.com",
        Destination: {ToAddresses: ["john.page@mongodb.com"]},
        Message: {
            Body: {
                Html: {
                    Charset: "UTF-8",
                    Data: body
                }
            },
            Subject: {
                Charset: "UTF-8",
                Data: "trst Email"
            }
        }
    }).then(r => console.log(EJSON.stringify(r))).catch(e => console.log(e));

};

Thank you very much @John_Page for narrowing the issue and sharing your knowledge.

Yes, using the deprecated built-in SES client solved the issue and I already changed my App to use that for now.

But it will be deprecated on August 1st. What’s the solution going forward? I understand when you said that when calling the SES endpoint the Request is using ISO8859-1. But how to overcome this?

Note:
— I’ve tried both AWS SDK JS v2 and v3, both have the same issue. Is this an App Service’s Dependency issue?
— Sorry for saying again, but I used to send emails just fine using the solution I shared above (with AWS SDK v3 as dependency with @aws-sdk/client-sesv2 package) why suddenly it stoped working?

Thank you very much again.

I suspect this is an issue with a dependancy - Looks like in the config for the SDK you can set an optional _HttpHandler value which AWS describe as “Fetch in browser and Https in Nodejs.” - I’ve asked the app services development team to comment on why / if that might be passing the wrong content-type charset in the request.

If you can see a way to explicity set it to @aws-sdk/node-http-handler | AWS SDK for JavaScript v3 it might help.

I quickly browsed the @aws-sdk/node-http-handler but didn’t find a charset setting yet, will try to give attention later today.

Again, thanks for your effective support.

Quicjk update - can 100% reproduce your issue unsure where it is happening though, will debug now.

1 Like

Thank you very much @John_Page

Hello @John_Page do you have news on this case?

Hello @John_Page do you have news on this case? The 3rd Party Services will be deprecated in just 3 weeks.

Hi André,
I replied to you about this on the 12th of June by email. We can reproduce the issue but as that context.http service is being deprecated it won’t get fixed. I gave you a workaround for it but also now the latest version of Axios works and so we woudl expect you to move to that. If you are still having issues using Axios instead of http.get then reach out and I can help again.

Workaround

const emailHtml = await context.http.get({
    url: 'https://www.typographicposters.com/newsletters/atlas-test'
    // created this new URL with only that thank you note
})
 
  const body = emailHtml.body.text();
  var Buffer = require('buffer').Buffer;
  var str = Buffer.from(body, 'binary').toString();

Thanks @John_Page I didn’t get the email, if you can please send again.

I didn’t know the context.http was going to be deprecated, I thought it was only the 3rd party services.

OK, will move to Axios, any version of axios or something specific?

Could you please send an example? Last time I tried the response was being returned as binary from Axios. Please refer to the message, in this same thread:

Thanks again

I hadn’t realised that context.http was considered third part services at first either but the team corrected me on that. You want this page

Thanks! Will proceed to Axios.

Will do so within the next week and come back here to close this thread.

Thanks a lot for your attention and support so far.