Changestream TryNext is blocking when no documents are available

I am using TryNext to fetch documents as they are available in change stream feed. From the API description, if there are no documents available it should return immediately. I added latency trace around tryNext call and I am observing close to 1000ms latency if no documents are available. I looked at the go code for tryNext call

func (cs *ChangeStream) TryNext(ctx context.Context) bool {
	return cs.next(ctx, true)
}

it calls next with non-blocking set to true. After following the code loopNext call makes a Next call on cursor which is a blocking call

func (cs *ChangeStream) loopNext(ctx context.Context, nonBlocking bool) {
	for {
		if cs.cursor == nil {
			return
		}

		if cs.cursor.Next(ctx) {
			// non-empty batch returned
			cs.batch, cs.err = cs.cursor.Batch().Documents()
			return
		}

This can potentially block the call. Is there any alternative available to the non-blocking TryNext?