If you use $match then $out rather than $merge, the original collection will be replace with only the documents that matches. This way you could $match out documents with empty array. With $out you need to make sure you $project all the other fields, a $set or $addFields would then be more appropriate.
I am pretty sure your $unwind and $group is kind of useless.
But i also think that you should use $merge as you did and then do a separate deleteMany to get rid of documents you do not want to keep.