Mongo-java-driver: $out fails in lastPipelineStage when aggregate

Working with Data Drivers
, ,
1

mongo-java-driver: $out fails in lastPipelineStage when aggregate

hi mongodb,
I wanted to use datalake to archive my data, so I verified the following query in mongo shell.

    db.getCollection('myCollection').aggregate([
    {
      '$match': {
        'start': {
          '$gte': ISODate('2017-12-31T15:00:00.000Z'),
          '$lt' : ISODate('2018-01-01T15:00:00.000Z')
        }
      }
    },
    {
      '$addFields': {
        'criterionDate': ISODate('2017-12-31T15:00:00.000Z')
      }
    },
    {
      '$out': {
        's3': {
          'bucket': 'myBucket',
          'region': 'ap-northeast-2',
          'filename': {
            '$concat': [
              {
                '$toString': '$_id'
              }, '/',
              {
                '$toString': '$criterionDate'
              }, '/'
            ]
          },
          'format': {'name': 'json.gz', 'maxFileSize': "200MiB"}
        }
      }
    }
    ])

The query worked successfully!

And then, I wanted to implement as application level, so I used spring boot application and it has mongo-java-driver 4.1.0.

But when I aggregate an above pipeline stage, I’ve encountered the following error.

java.lang.IllegalStateException: Cannot return a cursor when the value for $out stage is not a namespace document

And I was able to find the root cause below link.

As you can see, in my lastPipelineStage, which names $out, there are no keys which names ‘db’ and ‘coll’. there is only existing ‘s3’.

Currently, is there another option to use $out pipeline with ‘s3’ key in mongo-java-driver?
Or do you have a plan to add about ‘s3’ key into mongo-java-driver?

Any help will be much appreciated.

2

Hi, thanks for your question. This should work with the Java driver if you call AggregateIterable#toCollection(). Unlike AggregateIterable#cursor(), that method is agnostic as to the contents of the $out stage.

Hope this is helpful.

Regards,
Jeff

2 Likes
3

@Jeffrey_Yemin
Really thanks for a quick reply and it was the perfect answer for me!
It was a perfect choice MongoDeveloperCommunity to ask my question.

Remarks:
More contexts for someone who has the same problem as me

  • Actually in my case, spring-data-mongodb(3.0.4.RELEASE) has been using mongo-java-driver.
  • I was able to aggregate with MongoTemplate object, and it requires TypedAggregation object. Both objects are located in spring-data-mongodb
  • When I create TypedAggregation object, I added AggregationOptions(“SKIP”) in order to invoke AggregateIterable#toCollection() which is located in mongo-java-driver
1 Like