I have 3 nodes of replica set (P-S-S).
There are approximately 2000 connections connected to the primary.
And I used rs.stepDown() to make Secondary Primary.
As I expected, I thought some of the connections that were connected to the existing primary should be moved to the new primary, but other than that, 2,000 connections remained in the previous primary.
How are you measuring your connection counts? Are you monitoring each member individually?
Depending on how the driver is handling it the connections may stay open or slowly go down.
My primaries tend to have slightly more connections but the whole replicaset does not have a great difference in connections. This make sense as the client will open a connection to each member to facilitate various read concerns.