Writing pipeline output to a collection in Java

[Neil_Youngman]

I have a pipeline that seems to work fine in Java until I try to write it to a collection. I thought I could simply add
pipeline.add(new Document("$merge", “testCollection));
this doesn’t seem to create the collection. I have tried
pipeline.add(new Document(”$merge", new Document(“into”,testCollection)));
but that doesn’t work either, nor does $out.

What am I missing?

[Asya_Kamsky]

When you say “it doesn’t work” what happens? Do you get an error? What’s the version or server/driver?
Are you authenticated as a user that can write to the database?

[MaBeuLux88 (Maxime Beugnet)]

Hi @Neil_Youngman and welcome in the MongoDB Community !

Aggregations are lazy by default. Meaning that if you don’t read the result of the pipeline, the pipeline is simply not executed.

So for example:

coll.aggregate(asList(Aggregates.merge("new_coll")));

Wouldn’t be executed because the result isn’t consumed.
On the other hand:

coll.aggregate(asList(Aggregates.merge("new_coll"))).into(new ArrayList<>());

has to read the result of the pipeline and is executed.

Here is a code sample to prove my point:

import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import org.bson.Document;

import java.util.ArrayList;

import static com.mongodb.client.model.Aggregates.merge;
import static java.util.Collections.singletonList;

public class AggregationFramework {

    public static void main(String[] args) {
        String connectionString = "mongodb://localhost";
        try (MongoClient mongoClient = MongoClients.create(connectionString)) {
            MongoDatabase db = mongoClient.getDatabase("test");
            MongoCollection<Document> coll = db.getCollection("coll");
            MongoCollection<Document> newColl = db.getCollection("new_coll");
            resetCollections(coll, newColl);
            System.out.println("Merge without reading the docs...");
            coll.aggregate(singletonList(merge("new_coll")));  // lazy
            System.out.println("Docs in new_coll: " + newColl.countDocuments()); // should print zero
            coll.aggregate(singletonList(merge("new_coll"))).into(new ArrayList<>()); // not lazy
            System.out.println("Merge and consume the results this time.");
            System.out.println("Docs in new_coll: " + newColl.countDocuments()); // should print 3 this time.
        }
    }

    private static void resetCollections(MongoCollection<Document> coll, MongoCollection<Document> newColl) {
        newColl.drop();
        coll.drop();
        ArrayList<Document> docs = new ArrayList<>();
        docs.add(new Document("name", "Max").append("age", 33));
        docs.add(new Document("name", "Alex").append("age", 29));
        docs.add(new Document("name", "Claire").append("age", 25));
        coll.insertMany(docs);
    }
}

Result:

Merge without reading the docs...
Docs in new_coll: 0
Merge and consume the results this time.
Docs in new_coll: 3

Cheers,
Maxime.

[Neil_Youngman]

Thanks Maxime. That was the missing piece of the jigsaw for me. A simple .into(new ArrayList<>()), as you suggested, was all I needed.

Neil

[Neil_Youngman]

I still seem to have issues with merge.
It was working with
pipeline.add(new Document("$out", “testCollection"));
I changed a few things including changing that to
pipeline.add(new Document("$merge", “testCollection"));

that threw an exception with the error message "Value expected to be of type DOCUMENT is of unexpected type STRING"
According to the manual { $merge: <collection> } is an acceptable simplified form, but I get that exception. I don’t get an exception from pipeline.add(new Document("$merge", new Document("into", "testCollection"))); but it doesn’t create new collection.

[Neil_Youngman]

There are no errors from pipeline.add()

[MaBeuLux88 (Maxime Beugnet)]

I’d recommend to use the Aggregation Framework helper that I used in my code instead of reinventing the wheel. It’s shorter and it will work .

import static com.mongodb.client.model.Aggregates.merge;
import static java.util.Collections.singletonList;
...
coll.aggregate(singletonList(merge("new_coll"))).into(new ArrayList<>());

There is also a second parameter that you can use to pass merge options if you want any.

It looks like you have issues with curly double quotes as well in your code. I’m wondering if the issue couldn’t come from that.

Also, if you really don’t want to collect the result, I would use .first() maybe instead of .into() to avoid creating a useless list and save some memory.

Cheers,
Maxime.

[Neil_Youngman]

I believe the aggregation helper has some limitations ($lookup) which will affect the rest of the pipeline. I’m not sure if I can mix and match and if I can, it won’t help with the readability.

I guess I’d better have a play and see if I can get that to work.

I don’t seem to have curly quotes in the original code. I’m not sure where they came from. If they were in the code I was running, I would not expect it to compile.

[Neil_Youngman]

OK, I was looking in the wrong place. The source collection name had been changed accidentally, so I was no longer getting any data and merge was refusing to create an empty collection. I thought the issue was the merge stage as I hadn’t intentionally changed the source name.