Realm Trigger: Duplicate Key Error

[Fiewor_John]

@Mark_Smith

I was following through with one of the sessions and after implementing the trigger, I checked the logs to see this error:

I believe you had a similar error while working on this during the livestream but it seems yours happened because of the dummy data you used to test your updater function which you forgot to cleanup.

I didn’t do that, so I would like some help with fixing my trigger which is still showing this error

[Crist]

I had the same error and fixed with this:

Line in the repo:
const csvLines = csvData.split("/n");
the issue is in ‘/n’:
const csvLines = csvData.split("\n");

I hope that this can help you.

[Fiewor_John]

Thank you!
I have applied this fix.
Fingers crossed hoping it works.

[Fiewor_John]

Ok.

This hasn’t fixed it, but I think you should create a PR proposing this fix to the mongodb repo that also has this error.

I was going to do it, but it looks like I would be taking credit for your observation if I do so.

However, if you’re too busy to do this, let me know so I can create a PR in your stead.

[Fiewor_John]

Update: The trigger now appears to be working

I’m not entirely sure what resolved it, but I’m sure your fix helped
Thank you @Crist

[Manuel_Martin]

I also have the same error, and deleted my collections and create again but the issue was the same.

The problem is that the Realm Function gdeltUpdater in Sample Code is wrong (from the video Use Realm Functions to Keep Your GDELT Up To Date with @MarK_Smith and @Shane_McAllister)

To get an expected result I modified a bit the code to request always the same zip file:

http://data.gdeltproject.org/gdeltv2/20220518214500.export.CSV.zip

It contains the file 20220518214500.export.CSV <== 1359 Lines (The last is empty)

In the sample code the way to get the lines from this file is:

const csvLines = csvData.split("/n");
console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 BUT THE RESULT IS csvLines.length: 818

The correct way to do it is:

const csvLines = csvData.split(/\r\n|\r|\n/);
console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 AND THE RESULT IS csvLines.length: 1359

Here is the complete gdeltUpdater I used to test if the code works ok or not:

exports = async function(){
const AdmZip = require(“adm-zip”);
const http = context.http;
const csvURL = ‘http://data.gdeltproject.org/gdeltv2/20220518214500.export.CSV.zip’;

const latestCSV = (await http.get({ url: csvURL })).body;
// VSCode Warning:
// var Buffer: BufferConstructor new (str: string, encoding?: BufferEncoding) => Buffer (+5 overloads)
//  @deprecated — since v10.0.0 - Use Buffer.from(string[, encoding]) instead.
//const zip = new AdmZip(new Buffer(latestCSV.toBase64(), 'base64')); // SAMPLE CODE
const zip = new AdmZip(new Buffer.from(latestCSV.toBase64(), 'base64')); // Changed by me
const csvData = zip.getEntries()[0].getData().toString('utf-8');

//const csvLines = csvData.split("/n");
//console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 BUT THE RESULT IS csvLines.length: 818

const csvLines = csvData.split(/\r\n|\r|\n/);
console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 AND THE RESULT IS csvLines.length: 1359

if (csvLines[csvLines.length - 1] === ""){ // Remove last line
    csvLines.pop();
}
const rows = csvLines.map((line) => line.split("\t"));
console.log(`rows.length: ${rows.length}`) // IT SHOULD BE 1358 BUT THE RESULT IS rows.length: 818 (After changing for split(/\r\n|\r|\n/) is rows.length: 1358)
console.log(`rows: ${rows}`);
// await context.functions.execute("insertCSVRows", rows, downloadId); // COMENTED TO SEE THE LOGS OF THIS FILE

};

[Mark_Smith]

Thank you so much for solving this. Hopefully other people will find this helpful.

Mark