I´am fetching data from an API. I want to store that data in my DB.
Problem:
I know I get 500 objects every fetch- that is standard with this API. But when I run it with an empty DB, does not every object get added to my DB.
As my understanding, when I use async and await, the code does not continue until the code with the await is done. I have added the console output in a comment below.
Inside a crone.scheduale() i do the following with a async callback.
→ fetch the data → using .json() to make it a json.
cron.schedule("*/10 * * * * *", async () => {
console.log("10 sek har gått");
try {
let response = await fetch("https://polisen.se/api/events", {
method: "GET",
headers: {
"User-Agent": "Oscar Throedsson",
},
});
const data = await response.json();
→ Validate that the response is an array and console.log() length of the array.
console.log("Array? ", Array.isArray(data)); //output: true
console.log(`Längd: ${data.length}`); //output: 500
→ Create a for of and run every array item in to a schema. array item is objects.
for (const element of data) {
let event = createNewDocument(element);
→ Next is validating if the object is in the database. Every array item comes with a uniqe id, and I use that to confirm if it exist or not.
const checkEventExistence = await wholeColl.findOne({ _id: element.id }); //using await so the code checks the database before continuing.
console.log(
`4. ${counter}: Comparing: from API ${
element.id
} -> from DB: ${JSON.stringify(checkEventExistence._id)}`
); // output: number -> number. To see the comparision.
counter++; // output: Helps me keep track on which list item is printed out.
console.log("--------------------");
→ In an if statement, do I check if findOne() returns falsy or truly (null or object). If it is falsy (null) I add the object to my database of truly i dont add it. I use a try catch to see if the save went well or not.
if (!checkEventExistence) {
//# | save in DB
try {
added++; //! | delete when we clean up the code
await event.save();
console.log(`SAVED - > EventID: ${event.id}`);
} catch (error) {
console.error("Error while saving event:", error);
}
//
} else {
console.log(`NOT SAVED - > EventID: ${event.id}`);
//! | delete else when we clean up the code
notAdded++; //! | delete when we clean up the code
}
1. From API: 452571
server | 2. 469: Comparing: from API 454045 → from DB: {“_id”:454045,“datetime”:“2023-10-09T18:19:14.000Z”,“name”:“09 oktober 20:00, Stöld, Skellefteå”,“summary”:“Under kvällen har polisen larmats till två butiker med anledning av stöld”,“url”:“/aktuellt/handelser/2023/oktober/9/09-oktober-2000-stold-skelleftea/”,“type”:“Stöld”,“location”:{“name”:“Skellefteå”,“gps”:“64.750244,20.950917”},“__v”:0}