For questions like this it’s useful to understand how Charts builds the aggregation pipeline. This is documented at Backing Aggregation Pipeline. Lookup fields aren’t explicitly listed (I’ll try to get that fixed) but they are included in step 5, Calculated Fields.
Since the Lookup stage happens after the query bar, the lookup field can’t be affected by the query. To fix this you have two options:
Use the Filter pane to do your filtering, instead of the query bar (since this happens after the Lookup stage)
Do the Lookup in a Data Source pipeline instead of using Lookup fields (since this happens before the Query Bar stage).