Large language models increasingly sit between people and their data, whether answering questions in a chat interface or acting as agents that query databases directly. Either way, the model must reliably translate a request into the query language the database speaks.
For SQL, progress on this task has been driven by large benchmarks such as BIRD1 and Spider2. MQL, the query language of MongoDB, has received less attention despite the fact that MongoDB is one of the world’s most popular and widely used databases.
In this blog post, we introduce AptMQL-Bench (Access-Patterns-Based Text-to-MQL), a Text-to-MQL benchmark obtained by converting existing Text-to-SQL datasets. We also present the agent-driven conversion pipeline that produced it.
We're releasing:
AptMQL-Bench: the benchmark dataset. It consists of 21 new MongoDB databases along with 3,186 natural-language requests, migrated from BIRD1, spanning domains such as commerce, science, sports, and finance.
A technical report describing our proposed conversion pipeline for access-pattern schema design, data-preserving migration, and SQL-to-MQL translation.
Evaluation script, including our proposed soft execution accuracy (Soft-EX).
The challenges
Existing Text-to-MQL approaches take one of two routes. The NL-to-mongosh dataset3 starts from native MongoDB data, pairing 766 synthetically generated requests with the eight MongoDB Atlas sample databases. Other works convert existing Text-to-SQL datasets mechanically: DocSpider4 maps each relational table to its own collection one-to-one, while TEND-v25 and its multilingual extension MultiTEND6 embed tables along foreign keys (FK).
However, doing these conversions well is not trivial. Mechanical denormalizations along foreign keys, in particular, can be fragile. In our experiments, three out of 24 databases failed to migrate due to infinite recursion on self-referential keys, while another three produced individual documents so large that they didn't fit in MongoDB's 16 MB document limit. Additionally, on eight other databases, rows whose foreign keys resolved to no surviving parent were silently dropped, reaching 25.9% row loss in the worst case.
Finally, as we show below, these conversions also lead to schemas with poor scalability characteristics.
Our proposed conversion pipeline
As Figure 1 shows, we treat schema design as a first-class step and then revise that design while migrating the data. The pipeline is driven by coding agents (e.g., Claude Code) with human review at designated quality gates, and runs in three stages.
Figure 1. The conversion pipeline: document schema design, migration and validation, and SQL-to-MQL translation, each driven by a coding agent with human review at the quality gates.
1. Access-pattern schema design. Given the source SQLite database and its column descriptions, an agent first infers a real-world application the database would serve and the access patterns it would generate. The agent then designs a high-level conceptual schema (which collections exist and how they relate), and a low-level logical schema, specifying each collection’s fields, data types, and mappings from the source database. Both follow a design guidebook we curated from three MongoDB University courses. A reviewer then confirms each modeling choice and verifies that every source table remains retrievable from the design.
This approach ensures that the resulting schemas adhere to MongoDB's golden rule for schema design: data that is accessed together should be stored together.
2. Migration and validation. To catch errors or omissions in the schema design and migration code, the agent runs three automated checks after each migration: (1) queries over each MongoDB collection retrieve the same data as corresponding queries over the original SQL tables, (2) references between collections point to existing documents, and (3) nothing on the SQLite side was left behind. Anything that fails is surfaced to a human reviewer, who then works with the agent to fix the design or the migration code. We repeat this until every check passes.
3. SQL-to-MQL translation, then optimization. Each reference SQL query from the source benchmark is translated into MQL by an agent, which revises the translation until automated checks confirm that both queries return identical results. Translations that pass are correct but often still relational in shape. For instance, they'll contain redundant $lookup stages that fetch already-embedded documents. A second pass rewrites them to use embedded fields and precomputed aggregates directly; 1,444 MQL queries (45.32%) were optimized this way, and a reviewer confirmed each one is equivalent to the original translation. BIRD's external knowledge evidence is rewritten in the same spirit, so its field references point to MongoDB paths rather than SQL columns, and each rewritten evidence is checked by a reviewer against the ground-truth MQL.
The importance of schema design
We compare the characteristics of schemas produced by our access-pattern design approach against the two mechanical strategies on the same source data.
Table 1. DB migration results. 24 databases: the 21 BIRD databases in AptMQL-Bench plus 3 Spider databases used in the scaling experiments.
One-to-one is lossless by construction: every row becomes its own document, but the result merely mirrors the relational schema, at a cost we measure below. Foreign key embedding fails outright on six out of 24 DBs, while in 8 other cases the migration is lossy.
On execution time, we find that the performance of the three designs diverges as the size of the underlying DBs is increased. To measure this, we took three DBs from Spider (bike_1, car_1, and pets_1) and 11 DBs from BIRD. We then expanded each DB by appending synthetic rows (see our paper for more details on this procedure), obtaining different versions of these DBs with sizes up to ~200 MB.
Figure 2. Execution time of the ground-truth MQLs as database size grows (log-log, averaged across databases), for each conversion strategy.
Figure 2 shows the resulting execution times. At 200 MB, access-pattern MQLs run about 11× faster than foreign key embedding and roughly two orders of magnitude faster than a one-to-one migration on the Spider datasets. On BIRD, the speedup over one-to-one grows from around 40× on the original databases to 77× once they reach the largest sizes investigated.
These results demonstrate the advantages of the workload-driven schema design advocated in this work.
Benchmarking LLMs
We evaluated eight models in total (five open-weight, three proprietary), each with and without BIRD's external knowledge evidence.
Table 2. Soft execution accuracy (Soft-EX) on AptMQL-Bench. Full results, including execution success rate and Strict-EX, are in the paper.
The results show that the benchmark is challenging for every model. The strongest model, Claude Opus 4.5, reaches only 57.38% Soft-EX without external knowledge and 70.34% with it. External knowledge helps every model, adding 12–19 points.
Some of the latest models (Gemma 4 and the Claude family of models) produce executable MQL about 98–99% of the time, confirming that invalid MQL is more and more rare and that the remaining difficulty is semantic rather than syntactic.
Why we score with Soft-EX
In metrics borrowed from Text-to-SQL, strict execution accuracy (Strict-EX) requires the returned result to equal the gold result field-for-field, so a prediction that hands back the whole document instead of a single projected field is marked wrong, even though it contains the right answer, and even though returning the document is the idiomatic thing to do in MongoDB.
We therefore propose soft execution accuracy (Soft-EX): a prediction is correct if it returns the same rows as the gold result, with matching values for every ground-truth field. Extra fields are allowed, but extra or missing rows and incorrect values are not.
Table 3. The following examples illustrate the difference between Strict-EX and Soft-EX.
With external knowledge, Opus 4.5 scores 50.44% under Strict-EX but 70.34% under Soft-EX. To confirm that this gap is due only to extra fields, we manually reviewed all 634 queries where the two metrics disagree, which covers every prediction that Soft-EX accepts but Strict-EX rejects.
In every case, the prediction returned the same rows as the gold result, with matching values for every gold field, and differed only by including extra fields. Strict-EX therefore marks these predictions wrong even though they contain the complete gold answer.
Next steps
Try it and learn more!
Benchmark and databases: Hugging Face dataset
Conversion pipeline prompts and evaluation scripts are here
References
1 Jinyang Li et al. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. Advances in Neural Information Processing Systems, 36:42330–42357.
2 Tao Yu et al. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
3 MongoDB Education AI. 2025. Natural Language to MongoDB Shell (mongosh) Benchmark. Hugging Face dataset. https://huggingface.co/datasets/mongodb-eai/natural-language-to-mongosh
4 Arif Görkem Özer, Recep Firat Cekinel, Ismail Hakki Toroslu, and Pinar Karagoz. 2025. DocSpider: A Dataset of Cross-Domain Natural Language Querying for MongoDB. Natural Language Processing, 31(6):1367–1398.
5 Jinwei Lu, Yuanfeng Song, Zhiqian Qin, Haodi Zhang, Chen Zhang, and Raymond Chi-Wing Wong. 2025. Bridging the Gap: Enabling Natural Language Queries for NoSQL Databases through Text-to-NoSQL Translation. Preprint, arXiv:2502.11201.
6 Zhiqian Qin, Yuanfeng Song, Jinwei Lu, Yuanwei Song, Shuaimin Li, and Chen Jason Zhang. 2025. MultiTEND: A Multilingual Benchmark for Natural Language to NoSQL Query Translation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24632–24657, Vienna, Austria. Association for Computational Linguistics.