Time series collections - data storage in buckets

Ops and Admin
,
1

Hi all,
We are migrating the data from regular collections to timeseries collections in mongodb.
we checked how data looks like in system.buckets.<collection_name>

You can see that 577 documents are stored in one bucket but why data is not readable here ( Binary(Buffer.from)?

If we read the data from time series collection directly, which is basically view, it shows correct data.

Is it that, mongodb stores compressed/hashed data in system.buckets.<collection_name> ?

Thanks in advance.

 {
    _id: ObjectId("62a0384c5eefb9223069dd01"),
    control: {
      version: 2,
      min: {
        _id: ObjectId("62a03868cb0ba41ed02aeba3"),
        param: 1,
        timeStamp: ISODate("2022-06-08T05:49:00.000Z"),
        isOk: false,
        _class: 'test.class',
        hs: 'test.host1'
      },
      max: {
        _id: ObjectId("62a03aa2cc14743b3525d572"),
        param: 89,
        timeStamp: ISODate("2022-06-08T05:58:54.000Z"),
        isOk: true,
        _class: 'test.class',
        hs: 'test.host2'
      },
      count: 577
    },
    meta: { zone: 1, internalID: 1810 },
    data: {
      timeStamp: Binary(Buffer.from("0900482bdc4181010000870a7d0000000000001f0000000000000004000000000000000a00000000d0f77c01000000000000001f0000000000000006000000000000000e0000000000000000", "hex"), 7),
      isOk: Binary(Buffer.from("0800008f02000000000000209240200180040020121200000000000002000000008001000200000000000000622009098404000002986160000000000200001860801900010000000000000082810400004800000260000000060000020000000000000002000000062609094208210118000600026060809909006102000000000000008282040000000600801200000060928101090000000000000000", "hex"), 7),
      param: Binary(Buffer.from("01000000000000002e409f758440e131e39aec95a42c098807098a45dcc32e0d9854e8752507e999618252c68481b469c9120c861ed3a04757840c9648d32036889145851b43c180a8923235e650095f30c240062802f50c4ea3b0b66ddcc13c1f408835a3116aa4e2147d950d44abc92493532644d4835583e02e35753995f357b68106d58ba2404364559f36204301788bf51c0676116245c9611c46868a0344c40055651348e2d9334831071c4425c1a958c726150138415d20241502e0cd00b50a5b655b8866c054ec5c6610e8b61009c20cc52f8a453052222666389d030d91350e358c8071681e4fd0055ed92c88f0782a869043d2211180253605881040eaf620762c62900805b2109fc66d0982084c880c262e0fb00c482160954121431d2648314506be0fb0c7581966448640044c300c7698d00818daa300070f8ce633810140083c810512031227f6d84426940a806ed65097c1a0675310263040102008931ae678951109ea8835c64c1cc1152123e4c670d764144601281681c05140cb003c458e5bd01416952996a5054e2da34a0e14060482a344091a11561485c08c856326751641d108d908190618863549c5a705760458ab2ccca220fc00000c0000000000", "hex"), 7),
      _class: Binary(Buffer.from("02002b000000636f6d2e616b616d61692e70726f6d6f6e2e6473732e646f63756d656e742e52656c6561726e4461746100833f0000000000000001000000000000000200000000000000090000000000000000", "hex"), 7),
      hs: Binary(Buffer.from("02002c00000070726f642d6a69726164632d61707030312d7531362e64667730322e636f72702e616b616d61692e636f6d00833f0000000000000001000000000000000200000000000000090000000000000000", "hex"), 7),
      _id: Binary(Buffer.from("070062a03868cb0ba41ed02aeba38107044000000000044e00000000000000070062a03871cc14743b35259e3e8149608002040080001c00000200100000070062a0387bcb0ba41ed02aeca28129000000000000813c1f000f00000000070062a03885cc14743b3525a13e806c00000000300000070062a03886cb0ba41ed02aef20802a00000000000000070062a03890cc14743b3525a1ba81070200000000300e0e00000000000000070062a0389acb0ba41ed02af0138207020000000008002b00c0a7ec5300002b00280020000600070062a038a4cc14743b3525a2b4802c00000200100000070062a038aecb0ba41ed02af117800702000000000000070062a038b8cc14743b3525a5a8804614460000801008070062a038c2cb0ba41ed02af3fb822b000000000000000ce01ffffd0100000d00000050540000070062a038cccc14743b3525a69c800702000000000000070062a038d6cb0ba41ed02af6f281070280a0600000047d00000020f80700070062a038e0cc14743b3525a99a80070241c110000000070062a038eacb0ba41ed02af7f48107020000200800002d00000000f80700070062a038f4cc14743b3525aa9b8207020000000008020c0000d810720d210c00000200500000070062a038fdcb0ba41ed02af8de832c00000000800000cd0220004025f8073d980020080000006e02000000000000070062a03907cc14743b3525ad7d802a000000000ed000070062a03911cb0ba41ed02afbc981070200401080b5001e00000000000000070062a0391bcc14743b3525ae788107840040105024000e00000000000000070062a03925cb0ba41ed02afec88107020000000008001e00000000000000070062a0392fcc14743b3525b16e8107028160000010060e00000000000000070062a03939cb0ba41ed02affd98107020000202818020e00000000000000070062a03944cc14743b3525b279804a00000000010000070062a03944cb0ba41ed02b0052804c2000ff01100000070062a0394ecc14743b3525b3d082298000040c1080112cfc1f05000200004e2d210000000000070062a03958cb0ba41ed02b03358149400000107080014d20000014080000070062a03962cc14743b3525b5dc81070440c1100000000e00000000000000070062a0396ccb0ba41ed02b04338007028120a1180802070062a03976cc14743b3525b8cb8107020040503004008e20000000000000070062a03980cb0ba41ed02b0736810802000200050601ad20000024080000070062a0398acc14743b3525b9e281290000000000e07f0c00000200100000070062a03994cb0ba41ed02b08278169c0000008008083dc20000200300000070062a0399ecc14743b3525bc29802980000100008001070062a0399ecb0ba41ed02b09fb822c000000006025415c254100000000000a0000000000a020070062a039b2cc14743b3525bdd7804e00000000000000070062a039b2cb0ba41ed02b0b9b8329000000006060838c00000600c00f211c102102005000008c00000700600000070062a039bdcc14743b3525be44812c00000400002f413c2f410000200000070062a039c6cb0ba41ed02b0e89802a00000000044021070062a039d1cc14743b3525c12e8107044040000004000c00000015b04f01070062a039dbcb0ba41ed02b0f83800804010000040300070062a039e5cc14743b3525c2328107044040100000002e00000000000000070062a039efcb0ba41ed02b12898107860161000000000e00000000000000070062a039f8cc14743b3525c5288308020000020001000d210000340800004d0ce11f0c43f8071a00000000043000070062a03a01cb0ba41ed02b136b808d20000074070000070062a03a04cc14743b3525c5c8822c00009c0eb2e9200900000e342040016e4e410000000000070062a03a16cb0ba41ed02b1659800702000000200c00070062a03a20cc14743b3525c9098107020120001000085e00000000000000070062a03a2acb0ba41ed02b1777802900000000000000070062a03a2acc14743b3525c994822c00000000003021fc2f2100000000000a000eb000010000070062a03a34cb0ba41ed02b17e981ac2e61e512060000aa000b00000a9000070062a03a3ecc14743b3525cc82814c00000413523021078202e2c0581800070062a03a52cb0ba41ed02b1b69802e00000000000000070062a03a52cc14743b3525cd7b070062a03a52cb0ba41ed02b1b6c070062a03a52cc14743b3525cd7c824a000140000300006c342143130200001e00000000000000070062a03a5ccb0ba41ed02b1bf2804a00000000089000070062a03a66cc14743b3525d06a81080200001a1900000d200000fc070000070062a03a70cb0ba41ed02b1ed28108020000020104036d21000054080000070062a03a7acc14743b3525d183802a00000000000000070062a03a7acb0ba41ed02b1f55822b000010fc0708263b300100080000005c00000600100000070062a03a85cc14743b3525d1f8826c0000cc10f60c610980000110300088fd21000000000000070062a03a98cb0ba41ed02b22c6804a00012000027000070062a03a99cc14743b3525d4e78267120004200160843c102100000000000c0000000000000000", "hex"), 7)
```
3

Hey @Yogesh_Sonawane1,

Thank you for reaching out to the MongoDB Community forums.

In MongoDB, the time-series collection follows a bucketing pattern to store the data in an optimized format. So, when you create a time-series collection, it creates 3 collections within the same database out of which 2 are internal collections:

For example, in your case, you have created a time-series collection. The collection acts as a view that allows you to interact with all the documents and perform operations.

Regarding your question about the readability of the data using Binary(Buffer.from) , the actual data is stored in a bucketing pattern within the db.system.buckets.<collection_name> collection. This means that the individual documents are merged together into a single document known as a “bucket” based on timestamp and metafield.

It’s worth noting that the time series collection was primarily designed to capture a stream of data and aggregate those data, where the aggregation result is typically the end result of the workflow, instead of looking at individual data points. Hence, the current implementation of system.bucket is one way to achieve this goal, and we should expect that the underlying storage pattern will change for the better over time. Therefore, I don’t think you can depend on the current pattern being what it is today. In the future, the system.bucket collection may not exist or may not resemble what it is today. To ensure that your workflow still works with future time series implementations, I encourage you not to depend on its current structure.

Hope it answers your questions, in case you have any further questions or concerns feel free to reach out.

Best regards,
Kushagra

4

Thank you @Kushagra_Kesav for detailed explanation. I have got answer to my question.

This statement confused me, because currently, shard zone key range can be added to timeseries collections only using system.buckets.<collection_name> collection. Otherwise zone key range is not working if added to time series collection directly.
If system.buckets.<collection_names> does not exist in future, how shard zone key range will work for time series collection.
we are using Mongodb 6.06

Thanks,
Yogesh S