You should always worry about performance. But you should not spend time optimizing before you have performance issue. Make it right then make if fast.
The server is doing that for you as it tries to keep the working set in memory.
Do not complicate your code with early optimization.
Indexing fields used in queries always help. If you need that field, and that field is a frequent use-case, then yes index it. You may always remove the index later if you find it detrimental to your other use-cases.
Strings take more space and are slower to compare (a string a compared character per character while a number is compared in 1 operation). In your case I would keep the colours as a number. An hex code is a number after all.
One smart technique I never thought before, presented in Atlas Search to return all the applicable filters for a given search query without specifying - #2 by Erik_Hatcher in a different context might be used here. You could, for example, have frequent colour schemes (the facet_attributes) represented by a single number, an _id in another collection. So you would query with $all in a much smaller table and then $lookup using the single number in the huge collection.