Can't write aggregatiopn output to file

Hello.

I’m trying to dump the output from an aggregation pipeline to a file for further inspection since I’m expecting over 9000 documents.

I’ve followed the example here Saving the results of an agregation pipeline into a text file but my scenario doesn’t fits into any of those errors handled there.

I’ve tried initially with Compass but got ReferenceError: stream is not defined so I moved to mongo shell (v 1.1.9) and executed the same commands:

pvprod> {
cursor = db.offers.aggregate([ { $group: { _id: "$id", offers: { $push: { _id: "$$ROOT._id", offerId: "$$ROOT.id", title: "$$ROOT.title", status: "$$ROOT.status", createdAt: "$$ROOT.createdAt" } }, count: { $sum: 1 } } }, { $match: { count: { $gt: 1 } } }]); 
cursorAsStream = stream.Readable.from(cursor.map((entry) => EJSON.stringify(entry) + '\n'));
cursorAsStream.pipe(fs.createWriteStream('c:\\doublets.txt')); 
}

This is the returned message:

WriteStream {
  _writableState: WritableState {
    objectMode: false,
    highWaterMark: 16384,
    finalCalled: false,
    needDrain: false,
    ending: false,
    ended: false,
    finished: false,
    destroyed: false,
    decodeStrings: true,
    defaultEncoding: 'utf8',
    length: 0,
    writing: false,
    corked: 0,
    sync: true,
    bufferProcessing: false,
    onwrite: [Function: bound onwrite],
    writecb: null,
    writelen: 0,
    afterWriteTickInfo: null,
    buffered: [],
    bufferedIndex: 0,
    allBuffers: true,
    allNoop: true,
    pendingcb: 0,
    prefinished: false,
    errorEmitted: false,
    emitClose: true,
    autoDestroy: true,
    errored: null,
    closed: false
  },
  _events: [Object: null prototype] {
    unpipe: [Function: onunpipe],
    error: [Function: onerror],
    close: [Function: bound onceWrapper] { listener: [Function: onclose] },
    finish: [Function: bound onceWrapper] { listener: [Function: onfinish] }
  },
  _eventsCount: 4,
  _maxListeners: undefined,
  path: 'c:\\doublets.txt',
  fd: null,
  flags: 'w',
  mode: 438,
  start: undefined,
  autoClose: true,
  pos: undefined,
  bytesWritten: 0,
  closed: false,
  [Symbol(kFs)]: {
    appendFile: [Function: appendFile],
    ... lots of functions here  ...
    Dirent: [class Dirent],
    Stats: [Function: Stats],
    ReadStream: [Getter/Setter],
    WriteStream: [Getter/Setter],
    FileReadStream: [Getter/Setter],
    FileWriteStream: [Getter/Setter],
    _toUnixTimestamp: [Function: toUnixTimestamp],
    F_OK: 0,
    R_OK: 4,
    W_OK: 2,
    X_OK: 1,
    constants: [Object: null prototype] {
      ... constants here ...
    },
    promises: [Getter]
  },
  [Symbol(kCapture)]: false,
  [Symbol(kIsPerformingIO)]: false
}

Also tried with await but got Uncaught: SyntaxError: 'await' is only allowed within async functions. (1:10)

I’m on MongoDB Community 5.0.6

Thanks in advance.

Hi @fernandojmartin welcome to the community!

I tried the code you posted and got the same output you have in the shell. However in my case the output file was created, containing all the expected documents. Could you confirm that the output file c:\doublets.txt exists?

A more straightforward alternative approach might be to use --eval in mongo (not mongosh), e.g.:

mongo "mongodb://localhost:27017/test" --eval "db.test.aggregate([...])" --quiet > out.txt

This should output one document per line and dump them into out.txt.

If that doesn’t work, another workaround using $out aggregation stage might be more straightforward if you don’t mind using a temporary collection. Using $out as the last stage in the pipeline would output the result of the aggregation into a new collection. Once this is done, you can mongoexport the collection, and drop the collection once it’s not needed anymore.

Best regards
Kevin

1 Like