I’m interested in querying similarity over documents using the uniCOIL method described here: https://arxiv.org/pdf/2106.14807.pdf. Essentially, the document encoder produces a list of terms → impact scores. Here is an example:
"Professional Archive SOC 2 Type 2 + HITRUST CSF April 1, 2021 through March 31, 2022"
[{'professional': 2.4992518424987793, 'archive': 2.791024684906006, 'soc': 2.6375365257263184, '2': 1.653570294380188, 'type': 1.8308746814727783, '+': 1.7136235237121582, 'hit': 1.9101276397705078, '##rus': 1.8782016038894653, '##t': 0.9360965490341187, 'cs': 1.8214399814605713, '##f': 1.2431288957595825, 'april': 1.9671717882156372, '1': 0.18650582432746887, ',': 0.0, '2021': 2.746654748916626, 'through': 1.6950600147247314, 'march': 1.7389779090881348, '31': 0.9341773390769958, '202': 2.0317678451538086, '##2': 0.517501711845398, '[SEP]': 0.0}]
And the query encoder, using the same string produces what I believe are term frequencies/inverse document frequencies:
"Professional Archive SOC 2 Type 2 + HITRUST CSF April 1, 2021 through March 31, 2022"
{'professional': 128, 'archive': 143, 'soc': 135, '2': 151, 'type': 94, '+': 88, 'hit': 98, '##rus': 96, '##t': 48, 'cs': 93, '##f': 64, 'april': 101, '1': 10, ',': 0, '2021': 141, 'through': 87, 'march': 89, '31': 48, '202': 104, '##2': 26, '[SEP]': 0}
Every example demonstrates using the dictionary above as the input to a query over a Lucene Index. Since Mongo Atlas Search uses Apache Lucene, it seems like this type of querying on an index should be possible in mongo.
What I’m interested in is doing the above generations myself, storing the document encoder results in a mongo index and using the query encoder results as input to an efficient mongo query (versus mongo doing something similar behind the scenes for Atlas search text queries). Is this possible?