Writing to the database frequently

I’ve a datasource which is a streaming data source and I wish to make a streaming pipeline to write the data to the database using nodejs and mongoose. Is there any way i can write the data to the database constantly using streams or any other mechanism?

Hi @deep_jagani and welcome to MongoDB community forums!!

To understand the requirements in more details and help you provide with relevant resources, could you help me with some informations:

  1. A brief details on functioning of the application.
  2. Are you specifically using a streaming software like Kafka or something similar? in which case the blog post on MongoDB and IIoT: Data Streaming With Kafka might be a good starting point.

There has been a similar question on stackoverflow which you can refer to for your application.
However, please note that the answer posted in StackOverflow are from community experience so please ensure that it works for your use case.

Let us know if you have any further questions.

Regards
Aasawari

Sure,

  1. The application captures data coming from iot devices every X seconds into Kinesis data streams , and writes it to the database (mongodb) in real-time.
  2. Using Kinesis data streams as streaming software

But constant writing to the database isnt’t a optimal option.

DB Schema:-
Devices :- _id,name
Variables :- _id, name, deviceId
Values :- _id, value, time, metadata : {deviceId, variableId} (timeseries collection)

Hi @deep_jagani
Thank you for sharing the information.

Could you kindly help me understand your perspective on why constantly writing to the database may not be the most optimal solution?

On the other hand, by utilising MongoDB’s bulkWrite feature, you can insert the data into the collection. For example, you can collect and then bulk write data for the day into the database if you prefer.

Let us know if you have any further questions.

Regards
Aasawari

Constant writing because it needs to be a real-time iot dashboard which needs a near realtime system.