Serverless RPU model

MongoDB Atlas
,
#1

Hi folks, about a year ago I asked something similar when serverless was brand new, but seems that things have evolved.
I’m still debating whether I migrate my m2 cluster to serverless.

One question I have is regarding the RPUs. If I have a query with skip(20).limit(20) from the explain plan I see I get 20 results, with 40 scanned indexes. What is the RPU for such query?

The reason for this ask, is that I expect that a lot of queries from my app will rely on pagination using limit/skip (I know I should probably be using a timestamp field, but this is my reality at the moment).

Another one is regarding aggregations, specially with groups where the entire collection sometimes needs to be scanned. Will I be billed for the size of my collection in RPUs once aggregations are executed?

Thank you

For instance take this aggregation I run on a 110k docs collection. The first stage traverses all documents, but only 26 documents are returned from the aggregation, what is the RPU cost here?

{
  "explainVersion": "1",
  "stages": [
    {
      "$cursor": {
        "queryPlanner": {
          "indexFilterSet": false,
          "parsedQuery": {},
          "queryHash": "7023421D",
          "planCacheKey": "737D18C4",
          "maxIndexedOrSolutionsReached": false,
          "maxIndexedAndSolutionsReached": false,
          "maxScansToExplodeReached": false,
          "winningPlan": {
            "stage": "PROJECTION_SIMPLE",
            "transformBy": {
              "issue_date": 1,
              "_id": 0
            },
            "inputStage": {
              "stage": "COLLSCAN",
              "direction": "forward"
            }
          },
          "rejectedPlans": []
        },
        "executionStats": {
          "executionSuccess": true,
          "nReturned": 111579,
          "executionTimeMillis": 181,
          "totalKeysExamined": 0,
          "totalDocsExamined": 111579,
          "executionStages": {
            "stage": "PROJECTION_SIMPLE",
            "nReturned": 111579,
            "executionTimeMillisEstimate": 53,
            "works": 111581,
            "advanced": 111579,
            "needTime": 1,
            "needYield": 0,
            "saveState": 116,
            "restoreState": 116,
            "isEOF": 1,
            "transformBy": {
              "issue_date": 1,
              "_id": 0
            },
            "inputStage": {
              "stage": "COLLSCAN",
              "nReturned": 111579,
              "executionTimeMillisEstimate": 21,
              "works": 111581,
              "advanced": 111579,
              "needTime": 1,
              "needYield": 0,
              "saveState": 116,
              "restoreState": 116,
              "isEOF": 1,
              "direction": "forward",
              "docsExamined": 111579
            }
          },
          "allPlansExecution": []
        }
      },
      "nReturned": 111579,
      "executionTimeMillisEstimate": 109
    },
    {
      "$group": {
        "_id": {
          "year": {
            "$year": {
              "date": "$issue_date"
            }
          }
        },
        "total": {
          "$sum": {
            "$const": 1
          }
        }
      },
      "maxAccumulatorMemoryUsageBytes": {
        "total": 1872
      },
      "totalOutputDataSizeBytes": 11908,
      "usedDisk": false,
      "nReturned": 26,
      "executionTimeMillisEstimate": 178
    },
    {
      "$project": {
        "_id": true,
        "year": "$_id.year",
        "total": "$total"
      },
      "nReturned": 26,
      "executionTimeMillisEstimate": 178
    },
    {
      "$sort": {
        "sortKey": {
          "year": 1
        }
      },
      "totalDataSizeSortedBytesEstimate": 13156,
      "usedDisk": false,
      "nReturned": 26,
      "executionTimeMillisEstimate": 178
    }
  ],
  "serverInfo": {
   
  },
  "serverParameters": {
    "internalQueryFacetBufferSizeBytes": 104857600,
    "internalQueryFacetMaxOutputDocSizeBytes": 104857600,
    "internalLookupStageIntermediateDocumentMaxSizeBytes": 16793600,
    "internalDocumentSourceGroupMaxMemoryBytes": 104857600,
    "internalQueryMaxBlockingSortMemoryUsageBytes": 33554432,
    "internalQueryProhibitBlockingMergeOnMongoS": 0,
    "internalQueryMaxAddToSetBytes": 104857600,
    "internalDocumentSourceSetWindowFieldsMaxMemoryBytes": 104857600
  },
  "command": {
    "aggregate": "Pages",
    "pipeline": [
      {
        "$group": {
          "_id": {
            "year": {
              "$year": "$issue_date"
            }
          },
          "total": {
            "$count": {}
          }
        }
      },
      {
        "$project": {
          "year": "$_id.year",
          "total": "$total"
        }
      },
      {
        "$sort": {
          "year": 1
        }
      }
    ],
    "allowDiskUse": true,
    "cursor": {},
    "maxTimeMS": 60000,

  },
  "ok": 1,
  "$clusterTime": {
    "clusterTime": {
      "$timestamp": {
        "t": 1674088070,
        "i": 22
      }
    },
    "signature": {
      "hash": {
        "$binary": {
          "base64": "A/J23GkvPsdMBnzWui0R7/rMKGk=",
          "subType": "00"
        }
      },
      "keyId": {
        "$numberLong": "7150657730754117634"
      }
    }
  },
  "operationTime": {
    "$timestamp": {
      "t": 1674088070,
      "i": 22
    }
  }
}
#3

Hi @Vinicius_Carvalho,

Unfortunately I cannot provide the RPU costs for the queries you have provided since it depends on the exact situation but hopefully the below details may help with understanding how the RPU’s are calculated.

The best database deployment for you depends on your use case and feature requirements. You may choose a serverless instance if you want to:

  • Get started quickly with minimal database configuration.
  • Have your database scale automatically and dynamically to meet your workload.
  • Run infrequent or sparse workloads.
  • Develop or test in a cloud environment.

To help you to choose the best database deployment type, you can find more details about use cases, feature support, and comparisons in our choosing database deployment type documentation.

Read Processing Units are accrued per operation and Atlas charges one RPU for each document read (up to 4 KB) or for each index read (up to 256 bytes) when covered queries (can be satisfied by an index). For more information, please see the Serverless Instance Costs documentation.

Please note that a single read operation often does not equate to 1 RPU. One read operation can result in many RPUs depending on how the query is structured, how your data is structured, how large the documents are, among other things. In the most basic terms, RPUs represent how much work the server needs to do to fulfill your query (more on this below). Atlas meters based on the units of documents and index data read that meet the criteria outlined on our Serverless Instance Costs linked above.

During a query’s execution it may need to read a lot more data to determine what documents to return. For example, If a query involves a COLLSCAN (Collection Scan), then it can charge many RPUs as it reads all document bytes in a collection regardless of how many documents are returned.

In aggregation pipelines, all necessary documents scanned to fulfill the query contribute to the RPUs. This can include documents scanned, documents sorted, documents grouped in aggregation, and documents returned.

For guidance on optimizing aggregation pipeline performance review the following documentation links:

The Storage in the serverless instances includes the logical document and index storage. This value includes the number of bytes of all uncompressed BSON documents stored in all collections, plus the bytes stored in their associated indexes.

The best practice to reduce RPUs is to use indexes. Indexes reduce the ratio of the number of documents scanned to the number of documents returned by queries.

Using indexes improves your search queries and reduce the RPUs performed on your Atlas instance. If your application is running infrequent loads but you notice that you still incur high RPU costs, this is an indication that your queries are not optimized or your application is not using efficient indexes. See the Indexing Strategies documentation for improving the use of your indexes.

For more reading and examples about Serverless costs, see the blog post Serverless Instances Billing 101: How to Optimize Your Bill with Indexing.

Regards,
Jason

1 Like
#4

Thanks for the detailed explanation. I guess my concerns have been confirmed. The model is very similar to other vendors such as google firebase/datastore, which made me reconsider them in first place.

I think I may end up with a hefty bill at the end of the month, even if I only have a few thousand queries per day, I may end up having several million RPUs per day. I guess I’ll stick with the M2 instance and see if it can hold its breath, as we get more traffic, we move to larger clusters.

Thank you

1 Like
#5

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.