I think you are talking about eviction of dirty cache pages. Clean cache pages are simple to evict, and idle file handles only have clean pages in the cache (unless your checkpoints are taking longer than the idle sweep time; in such a case, lowering the idle sweep time will do nothing to improve this).
If you are having cache pressure, that typically means the number of dirty cache pages has hit the dirty cache limit, and no amount of freeing clean cache pages will do anything to alleviate that.