Darshan Hiranandani : Automating Purge of Stale Temporary Collections in MongoDB 6.0

Ops and Admin Replication
,
1

Hi everyone,

I’m Darshan Hiranandani, working with MongoDB 6.0 in a shared replica set environment, where the application creates temporary collections to stage intermediate data before finalizing it into destination collections. These temporary collections are voluminous and must be retained for a specified time (per compliance), after which they need to be purged to save storage space.

I’m looking for an automated way to drop these collections after they exceed a specific time threshold. Here’s what I’ve considered so far:

  1. TTL indexes: These help remove documents after a set period, but they don’t drop the entire collection, which could become a performance issue for large collections.
  2. ObjectId.getTimestamp(): Checking the timestamp of the temporary collection and dropping it once it exceeds the time threshold.
  3. mongostat: A DBA activity tool, so I’m avoiding this option.
  4. Date field in collections: Introducing a date field to reflect the last modified timestamp to identify idle collections. However, this doesn’t fully capture inactivity, as collections still undergo read operations.
  5. $collStats: Not particularly helpful for my case since it requires manual metrics collection and cross-verification.
  6. top command: Similarly not ideal, as it requires separate monitoring for changes.
  7. db.collection.watch(): Using change streams to monitor changes, but still need to figure out how to track read operations effectively.
  8. MongoDB Atlas: The application isn’t subscribed to Atlas, so this is not an option.

Has anyone faced a similar challenge or implemented an effective strategy for automating the purge of stale temporary collections in MongoDB? Any suggestions or resources on this would be greatly appreciated!

Thanks in advance for your help!
Regards
Darshan Hiranandani

1 Like