Queries with large volumes of returned data

[winterberry (Tony)]

I have a large timeseries data collection. Think 10Hz time series data sampled data running 24/7. It’s mostly jagged tabular data.
The primary index on this is time.
If you run a query to return a weeks worth of data, this is roughly 6K rows per column in each document.
The goal is to do some fairly compute intense calculations on this data.
Running the compute cycles in the database is what I’d expect for a query followed by a main CPU based execution engine. However, returning that kind of data to be processed by different computing resources is slower than I’d expect for an I/O operation. I suspect this is because the data is returned in a text format and not a binary format. I’m not sure how to change how data is returned. If this is possible, I’m not looking for the right things.

What is the best way to deal with large volumes of data being returned from a MongoDB query?

[Kushagra_Kesav]

Hi @winterberry,

Welcome to the MongoDB Community forums

@winterberry

I have a large time series data collection. Think 10Hz time series data sampled data running 24/7. It’s mostly jagged tabular data.

Can you be a little specific here with the dataset size? Like what is the collection size of your time-series collections?

Please share the sample document.

Also, share the output of

db.collection.stats()

and

db.getCollectionInfos({name:<time-series collection name>})

@winterberry

If you run a query to return a week’s worth of data, this is roughly 6K rows per column in each document.

Do you mean 6K documents? MongoDB doesn’t have the concept of rows & columns, can you clarify this?

Also, what specific query you are executing to get the result?

@winterberry

The goal is to do some fairly computationally intense calculations on this data.

Can you clarify your approach to calculating the data? Will it be done at the database level using aggregation pipelines, at the application level, or through some other method?

@winterberry

Running the compute cycles in the database is what I’d expect for a query followed by a main CPU-based execution engine. However, returning that kind of data to be processed by different computing resources is slower than I’d expect for an I/O operation.

What do you mean by “CPU-based execution engine” here? Also, kindly help me understand what specific computing resources you are referring to when you mention “different computing resources”?

@winterberry

I suspect this is because the data is returned in a text format and not a binary format.

The query returns the cursor of the Result Set in a text format, specifically in JSON after which we can iterate over the result set.

@winterberry

What is the best way to deal with large volumes of data being returned from a MongoDB query?

Could kindly help me understand what you mean by “best” and what “deal” refers to? Also, I was curious if having a lot of data returned by a query would be a problem.

Also, provide us with information about your MongoDB deployment. Specifically, please let us know the following:

1. What version of MongoDB are you currently running?
2. Is this an on-premises deployment or is it hosted deployment such as MongoDB Atlas?

Furthermore, refer to the Best Practices for Time Series Collections to read how to improve performance and data usage for time series collections.

Best,
Kushagra