I’ve noticed an unusual scenario where, after deleting all data from my collection, the RetrievalQA system still provided a relevant answer to the query “What is Google?”. This is perplexing because, with no data in the database, I expected no results or a response like “I don’t know.”
You should try prompting your LLM with this specific instruction - ie “Answer ONLY on the basis of the context provided below and don’t use any other knowledge source” so that it can restrict to the retrieved documents. Different LLMs have a different propensity for instruction following