Ticket: get_comments - SOLVED

Hi,

I am getting the below error while running get_comments ticket. Could you please help me understand why?

================================== FAILURES ===================================
___________________ test_comments_should_be_sorted_by_date ____________________

client = <FlaskClient <Flask ‘mflix.factory’>>

@pytest.mark.get_comments
def test_comments_should_be_sorted_by_date(client):
    # in order from most to least recent
    movie_ids = [
        "573a1391f29313caabcd8414",
        "573a1391f29313caabcd9058",
        "573a1391f29313caabcd91ed",
        "573a1392f29313caabcd9d4f",
        "573a1392f29313caabcdae3d",
        "573a1392f29313caabcdb40b",
        "573a1392f29313caabcdb585",
        "573a1393f29313caabcdbe7c",
        "573a1393f29313caabcdd6aa"
    ]
    for id in movie_ids:
        comments = get_movie(id).get('comments', [])
        test_comments = sorted(
            comments[:],
            key=lambda c: c.get('date'),
            reverse=True
        )
        for _ in range(0, min(10, len(comments))):
            rand_int = randint(0, len(comments) - 1)
          assert comments[rand_int] == test_comments[rand_int]

E AssertionError: assert {’_id’: Objec…cdae3d’), …} == {’_id’: Object…cdae3d’), …}
E Omitting 1 identical items, use -vv to show
E Differing items:
E {‘email’: ‘kenneth_chandler@fakegmail.com’} != {‘email’: ‘michelle_fairley@gameofthron.es’}
E {‘name’: ‘Kenneth Chandler’} != {‘name’: ‘Catelyn Stark’}
E {‘text’: ‘Sequi maxime nemo amet esse consectetur et. Exercitationem saepe officiis necessitatibus. Doloremque modi itaque minima reprehenderit mollitia.’} != {‘text’: ‘Id nihil officia excepturi magnam beatae. Nesciunt illum magni aperiam enim. Occaecati maiores dolorem ipsam non illo repellendus hic. Iure nam voluptates enim vitae eos explicabo omnis illum.’}
E {’_id…
E
E …Full output truncated (3 lines hidden), use ‘-vv’ to show

tests\test_get_comments.py:51: AssertionError
============== 1 failed, 2 passed, 39 deselected in 8.01 seconds ==============

Here is my Code:

    # TODO: Get Comments
    # Implement the required pipeline.
    pipeline = [
        {
            '$match': {
                '_id': ObjectId(id)
            }
        }, {
            '$lookup': {
                'from': 'comments',
                'let': {
                    'id': '$_id'
                },
                'pipeline': [
                    {
                        '$match': {
                            '$expr': {
                                '$eq': [
                                    '$movie_id', '$$id'
                                ]
                            }
                        }
                    }
                ],
                'as': 'comments'
            }
        }, {
            '$sort': {
                'date': -1
            }
        }
    ]

    movie = db.movies.aggregate(pipeline).next()
    return movie

@ksp585

You could follow equality match syntax of $lookup.

{
   $lookup:
     {
       from: <collection to join>,
       localField: <field from the input documents>,
       foreignField: <field from the documents of the "from" collection>,
       as: <output array field>
     }
}

@nayamama: I followed the standard syntax, still getting the assertion error. Now, failing two tests compared to previous code. Please advise.

NOTE: THIS CODE IS WORKING FINE IN COMPASS.

    # TODO: Get Comments
    # Implement the required pipeline.
    pipeline = [
        {
            '$match': {
                '_id': ObjectId(id)
            }
        }, {
            '$lookup': {
                'from': 'comments',
                'localField': 'movie_id',
                'foreignField': '_id',
                'as': 'comments'
            }
        }, {
            '$sort': {
                'date': -1
            }
        }
    ]

    movie = db.movies.aggregate(pipeline).next()
    return movie 

ERROR:

================================== FAILURES ===================================
________________________ test_fetch_comments_for_movie ________________________

client = <FlaskClient <Flask ‘mflix.factory’>>

@pytest.mark.get_comments
def test_fetch_comments_for_movie(client):
    # X-Men
    movie_id = "573a139af29313caabcf0f51"
    result = get_movie(movie_id)
  assert len(result.get('comments', [])) == 432

E AssertionError: assert 0 == 432
E + where 0 = len()
E + where = <built-in method get of dict object at 0x0000025D01C27AB0>(‘comments’, )
E + where <built-in method get of dict object at 0x0000025D01C27AB0> = {’_id’: ObjectId(‘573a139af29313caabcf0f51’), ‘awards’: ‘13 wins & 25 nominations.’, ‘cast’: [‘Hugh Jackman’, ‘Patrick Stewart’, ‘Ian McKellen’, ‘Famke Janssen’], ‘comments’: , …}.get

tests\test_get_comments.py:17: AssertionError
____________________ test_fetch_comments_for_another_movie ____________________

client = <FlaskClient <Flask ‘mflix.factory’>>

@pytest.mark.get_comments
def test_fetch_comments_for_another_movie(client):
    # 300
    movie_id = "573a13b1f29313caabd36321"
    result = get_movie(movie_id)
  assert len(result.get('comments', [])) == 409

E AssertionError: assert 0 == 409
E + where 0 = len()
E + where = <built-in method get of dict object at 0x0000025D01C10F78>(‘comments’, )
E + where <built-in method get of dict object at 0x0000025D01C10F78> = {’_id’: ObjectId(‘573a13b1f29313caabd36321’), ‘awards’: ‘19 wins & 37 nominations.’, ‘cast’: [‘Gerard Butler’, ‘Lena Headey’, ‘Dominic West’, ‘David Wenham’], ‘comments’: , …}.get

tests\test_get_comments.py:25: AssertionError
============== 2 failed, 1 passed, 39 deselected in 6.94 seconds ==============

@ksp585

You messed up the localField and foreigenField. Think about it. who is local and who is foreigner?

Also, in this stage, I do not remember we need to sort in get_movie().

@nayamama: Thanks for spotting the error. I corrected it now.

Still, there is an assertion error.

================================== FAILURES ===================================
___________________ test_comments_should_be_sorted_by_date ____________________

client = <FlaskClient <Flask ‘mflix.factory’>>

@pytest.mark.get_comments
def test_comments_should_be_sorted_by_date(client):
    # in order from most to least recent
    movie_ids = [
        "573a1391f29313caabcd8414",
        "573a1391f29313caabcd9058",
        "573a1391f29313caabcd91ed",
        "573a1392f29313caabcd9d4f",
        "573a1392f29313caabcdae3d",
        "573a1392f29313caabcdb40b",
        "573a1392f29313caabcdb585",
        "573a1393f29313caabcdbe7c",
        "573a1393f29313caabcdd6aa"
    ]
    for id in movie_ids:
        comments = get_movie(id).get('comments', [])
        test_comments = sorted(
            comments[:],
            key=lambda c: c.get('date'),
            reverse=True
        )
        for _ in range(0, min(10, len(comments))):
            rand_int = randint(0, len(comments) - 1)
          assert comments[rand_int] == test_comments[rand_int]

E AssertionError: assert {’_id’: Objec…cdae3d’), …} == {’_id’: Object…cdae3d’), …}
E Omitting 1 identical items, use -vv to show
E Differing items:
E {’_id’: ObjectId(‘5a9427648b0beebeb6957ff5’)} != {’_id’: ObjectId(‘5a9427648b0beebeb6957ff1’)}
E {‘email’: ‘kenneth_chandler@fakegmail.com’} != {‘email’: ‘michelle_fairley@gameofthron.es’}
E {‘name’: ‘Kenneth Chandler’} != {‘name’: ‘Catelyn Stark’}
E {‘date’: datetime.datetime(1980, 12, 30, 14, 59, 28)} != {‘date’: datetime.datetime(1979, 9, 28, 10, 23, 39)}
E {‘text’: ‘Sequi maxime nemo amet esse consectetur et. Exercitationem saepe officiis necessitatibus. Doloremque modi itaque minima reprehenderit mollitia.’} != {'text…
E
E …Full output truncated (2 lines hidden), use ‘-vv’ to show

tests\test_get_comments.py:51: AssertionError
============== 1 failed, 2 passed, 39 deselected in 6.57 seconds ==============

1 Like

@ksp585

I just got time to try the code. For the sake of saving time, I did not run any unit test of this project before and just tried each function by monitoring status page. Therefore, I just noticed my code passes the integration test but does not pass the unit test.

Your first method is right too. It just looks a little bit complicated. When I wrote code I just followed the story provided in code comments and did not check the lab specification in course. So I ignored that the comments for each movie should be sorted by date. As it is part of lab requirement, we will have to use pipeline method like yours. Here is my solution.

    pipeline = [
        {
            "$match": {
                "_id": ObjectId(id)
            }
        },
        {
            "$lookup": {
                "from": "comments",
                "let": {"id": "$_id"},
                "pipeline": [
                    {"$sort": {"date": -1}},
                    {"$match": {
                        "$expr": {
                            "$eq": ["$movie_id", "$$id"]
                        }
                    }}
                ],
                "as": "comments"
            }
        }
    ]

Now, all unit test passed and world is a perfect world again. :wink:

8 Likes

this worked well for me…
thanks a lot:eyeglasses::+1::pray:

This works well, but I think it is much faster to do the sort after the match and this works as well.
Thanks,

I got this error even when I use the same pipeline. But I can pass the status page not the 2nd unit test.
● Get Comments › comments should be sorted by date

expect.assertions(24)

Expected 24 assertions to be called but received twelve assertion calls.

  15 |   test("comments should be sorted by date", async () => {
  16 |     // most recent to least
> 17 |     expect.assertions(24)
     |            ^
  18 |     const movieIds = [
  19 |       "573a1391f29313caabcd8414",
  20 |       "573a1391f29313caabcd9058",

Why does expr operator work but simple colon to depict equality doesn’t work. For eg:-

    pipeline:[{
                '$match': {'_id': ObjectId(id)}},
                {
                '$lookup': {
                    'from': 'comments', 
                    'let': {'id': '$_id'}, 
                    'pipeline': [
                            {'$match': {'$movie_id':'$$id'}}, 
                            {'$sort': {'date': -1}}],
                    'as': 'comments'
                    }
                }, 
            ]

Above code with colon doesn’t work.

This thread is over a year old… it would have been much better if you’d created a new thread.

The first thing to do is to look at the documentation:

Then you ask yourself, does the $eq (query operator) interpret the value '$$id' as String or does it resolve to a field?

Hi,

I don’t think I quite get you when you say field vs string. Could you please elaborate. Please pardon my ignorance.

Thanks in advance!

Regards,
Harpreet

Hi @KaurPreet, please remove the solution from your post. As a reminder, it’s against forum guidelines to post potential lab solutions.

Have you managed to look through all three documentation and all the examples?

Hi,

I don’t think I still quite get the string and field thing that you mentioned. However, what I understand after reading the documentation is that $eq operator and colon are equivalent when used in a normal query but when used in aggregation pipelines only $eq should be used when comparing two items for equality.
Please validate my understanding.

The other thing I don’t understand is the rationale behind comparing two things differently when used in a normal query vs aggregation pipeline. Why the inconsistency? It makes the syntax a confusing. Maybe you could elaborate something that I am missing here.

And a feedback. It will be better if the distinction is clearly pointed out in the documentation. Had it not been for the lab, I would have never come to know the differences in regards to when it is mandatory and when it is not. I mean on one page it does mention that $eq and colon can be used conversely but under aggregation section it is not explicitly mentioned the problem with using them conversely and it can easily escape notice, since the first thing someone learns is using colon when expressing equality in mongodb.

Sincerely,
Harpreet

I’ll try to disect it:

find()

  1. Is of the form find({ <query> }, { <projection> })
    • The query and projection operators in the links above are specifically for find()
    • The projection operators are specific to find() and should not be confused with $project aggregation stage. The former is very limited in its usage.
    • You may find an overlap in names between the operators in query and projection, e.g. $elemMatch

aggregate()

  1. The match stage in aggregate({$match: { <query> }}) is equivalent to find({ <query> })

    • All <query> operators from the link above can be used in a $match stage
  2. The other stages like $project, $group, …etc, have a set of Operators that are specific to the Aggregation Pipeline

    • Some Aggregation Pipeline Operators may have the same name as the query operators in find(), however, they will have a different syntax. An example is $eq like you’ve seen.
  3. You can use Aggregation operators in find({ <query> }) and $match by encapsulating the operator in the $expr operator

Yes, this is one of my gripes with the MongoDB operators. Why they couldn’t overload the operators and get rid of $expr is beyond me and just makes for a bad user experience. In addition, the documentation does a poor job of describing these intricacies and I’ve raised this issue a couple of times already. The idea of separating Aggregation Pipeline operators from Find() operators seems like a marketing and product gimmick.

In my view, there should be:

  1. One documentation for Operators
  2. An operator like $eq would describe the different syntax and limitations
  3. A filter on the documentation that allows a reader filter down to operators that are specific to the Aggregation Pipeline or Find()
  4. Each pipeline stage will list out all the operators that are supported with a link to each operator. Or the list of operators could be those that are not supported within each stage.
  5. $expr should just not exist

I’m sure that a considerable amount of effort will be required to forego $expr and I hope that the Query Engine team are already considering this step in the right direction.

Hopefully you get the gist now. :slight_smile:

Thank you so much for taking the effort to write such a detailed explanation. It definitely helps :blush:

If you feel ok may I know your name please.

this is from the documentation: A $match stage requires the use of an $expr operator to access the variables. $expr allows the use of aggregation expressions inside of the $match syntax.