I am using AWS kinesis firehose to stream Cloudwatch logs to my Mongo Altas database. In doing so, I created an Altas API endpoint which is exposed to AWS kinese Firehose to call for sending over the log data. The log data will be processed first by the Altas Function, which decodes them and writes to MongoDB in blukwrite. However, during the process, I keep getting error about string formatting during JSON parsing. I looked into it and found that the body text that I got inside the ALtas function with payload.body.text() has been truncated to around 760 characters. When I log also, the request data, I can see that the full body (over 4000 characters) is there, just that somehow it was truncated when the data is accessed in the ALtas function. I searched over all documentation and forums but there is no mention of this problem. Just wonder if there is any hard limit to the length of the payload body
Replication of this problem can be done via creating a https endpoint with a altas function, then send over a POST request with a large Body (over 3000 characters).
Would be great if someone has experience using HTTPs endpoint with large payload can share your method
exports = function(payload, response) {
/* Using Buffer in Realm causes a severe performance hit
this function is ~6 times faster
*/
const decodeBase64 = (s) => {
var e={},i,b=0,c,x,l=0,a,r='',w=String.fromCharCode,L=s.length
var A="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
for(i=0;i<64;i++){e[A.charAt(i)]=i}
for(x=0;x<L;x++){
c=e[s.charAt(x)];b=(b<<6)+c;l+=6
while(l>=8){((a=(b>>>(l-=8))&0xff)||(x<(L-2)))&&(r+=w(a))}
}
return r
}
console.log("data3", payload.body.toBase64());
console.log("data3", payload.body.text());
// Get AccessKey from Request Headers
const firehoseAccessKey = payload.headers["X-Amz-Firehose-Access-Key"]
// Check shared secret is the same to validate Request source
if(firehoseAccessKey == context.values.get("FIREHOSE_ACCESS_KEY")) {
// Payload body is a JSON string, convert into a JavaScript Object
const data = JSON.parse(payload.body.text())
// Each record is a Base64 encoded JSON string
const documents = data.records.map((record) => {
const document = JSON.parse(decodeBase64(record.data))
return {
...document
}
})
response.addHeader(
"Content-Type",
"application/json"
)
// Perform operations as a bulk
context.services.get("mongodb-atlas").db("monitors").collection("firehose").insertMany(documents).then(() => {
// All operations completed successfully
response.setStatusCode(200)
response.setBody(JSON.stringify({
requestId: payload.headers['X-Amz-Firehose-Request-Id'][0],
timestamp: (new Date()).getTime()
}))
return
}).catch((error) => {
// Catch any error with execution and return a 500
response.setStatusCode(500)
response.setBody(JSON.stringify({
requestId: payload.headers['X-Amz-Firehose-Request-Id'][0],
timestamp: (new Date()).getTime(),
errorMessage: error
}))
return
})
} else {
// Validation error with Access Key
response.setStatusCode(401)
response.setBody(JSON.stringify({
requestId: payload.headers['X-Amz-Firehose-Request-Id'][0],
timestamp: (new Date()).getTime(),
errorMessage: "Invalid X-Amz-Firehose-Access-Key"
}))
return
}
}
This is my Altas function. Both
console.log("data3", payload.body.toBase64());
console.log("data3", payload.body.text());
prints trucated output of my request body