UsingAggregationBuilders.aggregateSeveralStages assertion fails

When running the test without changing any code assertion fails as groupByResults is not the same as sortByCountResults (example below). When ran multiple times it may pass the test.

12:47:00.905 [main] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 31 completed successfully in 938.02 ms on connection [connectionId{localValue:7, serverValue:407354}] to server mflix-shard-00-02-9jbp6.mongodb.net:27017
Document{{_id=Leonor Silveira, count=8}}
Document{{_id=Joaquim de Almeida, count=6}}
Document{{_id=Isabel Ruth, count=6}}
Document{{_id=Joèo Cèsar Monteiro, count=5}}
Document{{_id=Maria de Medeiros, count=5}}
Document{{_id=Ricardo Trèpa, count=5}}
Document{{_id=Catherine Deneuve, count=5}}
Document{{_id=John Malkovich, count=5}}

12:47:01.006 [main] DEBUG org.mongodb.driver.protocol.command - Execution of command with request id 33 completed successfully in 49.43 ms on connection [connectionId{localValue:7, serverValue:407354}] to server mflix-shard-00-02-9jbp6.mongodb.net:27017
Document{{_id=Leonor Silveira, count=8}}
Document{{_id=Joaquim de Almeida, count=6}}
Document{{_id=Isabel Ruth, count=6}}
Document{{_id=Joèo Cèsar Monteiro, count=5}}
Document{{_id=John Malkovich, count=5}}
Document{{_id=Catherine Deneuve, count=5}}
Document{{_id=Ricardo Trèpa, count=5}}
Document{{_id=Maria de Medeiros, count=5}}

groupByResults sortByCountResults

4 Likes

Hi @vintersorg_1982,

Let me know if you are still facing any issue. If yes, it would be great if you can share more details.

Kanika

1 Like

Any idea if it’s possible to do additional sort after sortByCount?
Something similar to Sorts.descending(“count”, “_id”)

1 Like

You can accomplish this with a $sort.

1 Like

I also faced the same issue. The aggregateSeveralStages test in UsingAggregationBuilders.java test is flaky, and it doesn’t consistently pass every time when using mvn test -Dtest=UsingAggregationBuilders.

2 Likes

Screenshot of the issue you are facing when it fails.

2 Likes

Here is the dpaste link of the error: http://dpaste.com/27WB5EP

1 Like

I suspect a sort issue.

For example, _id=Mercedes Sampietro, is last in the expected list but it is not in the result your are returning.

1 Like

I am also having this issue. Same result set size but different order of documents. If I run tests in quick succession of each other sometimes it will pass sometimes it will fail.

1 Like

Hi can you let us know what the driver is doing? Why sometimes the test is passing and sometimes it is failing?

1 Like

Please provide a screenshot of

Indeed, same issue so same conclusion.

1 Like

How would you like me to show in a screenshot the test failing and test passing from back to back tests? It’s clearly an issue with sorting but what is not clear is why the sort yields different results in different runs.

I would like someone from mongodb to explain why the driver is behaving like this.

1 Like

The driver is behving correctly. Your sorting is wrong.

1 Like

How can the sorting be wrong? I didn’t change anything in the test class or test case aggregateSeveralStages? It’s not like it’s custom code I wrote for this course. Additionally, the sorting works sometimes properly and sometimes does not. So this would mean the driver is behaving differently for different test runs. @kanikasingla can you please take a look?

1 Like

No that is not the issue. Can you please describe the issue again? Also, I would love to see the code changes.

Kanika

1 Like

No code changes were made for this particular test. The code is in the aggregateSeveralStages test provided in the mflix handout

Description of issue:

  1. run aggregateSeveralStages test -> observe success
  2. run aggregateSeveralStages test again -> observe failures
  3. subsequent tests -> could be success or failure
1 Like

I’m having the same issue, but in my case no tests succeded for aggregateSeveralStages
I have not changed the handout code. Like @Jacob_35779 said:

The code is in the aggregateSeveralStages test provided in the mflix handout

The code:

  @Test
  public void aggregateSeveralStages() {
    /*
    A single aggregation pipeline, and in particular a $match stage,
    could be achieve by using the find() command. So let's use something
    a bit more interesting, which is exactly what we should be using the
    aggregation framework for.
     */

    List<Bson> pipeline = new ArrayList<>();

    /*
    For all movies produced in Portugal, sum the number of times
    a particular cast member gets to visit such a lovely place. How
    many times has an individual cast member, participated in a movie
    produced in this country. And ofcourse, let's not forget to return the
    results sorted Ascending regarding the number of gigs.
    In the mongo shell this question would be answered by the following
    aggregation:
    db.movies.aggregate([
        {$match: {countries: "Portugal"}},
        {$unwind: "$cast"},
        {$group: {_id: "$cast", gigs: {$sum: 1}}}
    ])
    For that we are going to need:
    */

    /*
    - $match to find all movies produced in portugal
     */
    String country = "Portugal";
    Bson countryPT = Filters.eq("countries", country);
    Bson matchStage = Aggregates.match(countryPT);

    /*
    - $unwind the elements of the $cast array
     */
    Bson unwindCastStage = Aggregates.unwind("$cast");

    /*
    - $group based on cast name and count the number of times a cast
    member appears in the result set
     */
    // group by cast members
    String groupIdCast = "$cast";

    /*
    Group operations are in place to do some sort of accumulation
    operation.
    Operations like $sum, $avg, $min, $max ... are good candidates to be
    used along side group operations, and there is a java builder for that.
    @see com.mongodb.client.model.Accumulators handles all accumulation
    operations.
     */

    // use $sum accumulator to sum 1 for each cast member appearance.
    BsonField sum1 = Accumulators.sum("count", 1);

    // adding both group _id and accumulators
    Bson groupStage = Aggregates.group(groupIdCast, sum1);

    /*
    - $sort based on the new computed field `gigs`
     */

    // create the sort order using Sorts builder
    Bson sortOrder = Sorts.descending("count");
    // pass the sort order to the sort stage builder
    Bson sortStage = Aggregates.sort(sortOrder);

    /*
    With all these stages, we are now ready to call our aggregate method
    with a bit more complex of a pipeline than a single $match stage.
     */

    pipeline.add(matchStage);
    pipeline.add(unwindCastStage);
    pipeline.add(groupStage);
    pipeline.add(sortStage);

    AggregateIterable<Document> iterable = moviesCollection.aggregate(pipeline);

    List<Document> groupByResults = new ArrayList<>();
    for (Document doc : iterable) {
      System.out.println(doc);
      groupByResults.add(doc);
    }

    /*
    The aggregation framework also provides stages that combine
    operations that are typically expressed by several stages.
    For example, $sortByCount, combines both the $group with a $sum
    accumulator with a $sort stage.
    Don't believe me? Well, let's check it out!
     */

    List<Bson> shorterPipeline = new ArrayList<>();

    // we already have built booth $match and $unwind stages
    shorterPipeline.add(matchStage);
    shorterPipeline.add(unwindCastStage);

    // create the $sortByCountStage
    Bson sortByCount = Aggregates.sortByCount("$cast");

    // append $sortByCount stage to shortPipeline
    shorterPipeline.add(sortByCount);

    // list to collect shorterPipeline results
    List<Document> sortByCountResults = new ArrayList<>();

    for (Document doc : moviesCollection.aggregate(shorterPipeline)) {
      System.out.println(doc);
      sortByCountResults.add(doc);
    }

    /*
    Running both pipelines, the same set of results.
     */

    Assert.assertEquals(groupByResults, sortByCountResults);
  }

The error

at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at mflix.lessons.UsingAggregationBuilders.aggregateSeveralStages(UsingAggregationBuilders.java:218)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)

Line 218 is

Assert.assertEquals(groupByResults, sortByCountResults);

2 Likes

Hi, I saw the same error when I run the test without any modification of the test code.
It seems queries to MongoDB are fine and get results well.
The length of both “groupByResults” and “sortByCountResults” are the same: 347.

Then I retried the test many times and I could find it succeeds about one out of five times.
(4 of 5 are failed, in my situation.)

This is my opinion, but I think the order is different even though the content of the results is the same.
This is due to the MongoDB cluster or sharding situation.
Yes, I’m using the Atlas cluster.

I hope this is helpful.

1 Like

I’m facing this issue too, with mongo 5.0 atlas server and unmodified test code.
It seems that in that server, entries with same “count” field didn’t preserve order for each aggregation.

2 Likes