System variables in Aggregation Pipeline

[Laks]

Hi Team,

Can you please help me understand the difference between the below system level variables.

$$CURRENT vs $$ROOT
$$NOW vs $$CLUSTERTIME

Referenced from → https://docs.mongodb.com/manual/reference/aggregation-variables/#mongodb-variable-variable.CURRENT

[michael_hoeller (Michael Höller)]

Hi @Laks

$$NOW is always awaílable whereas $$CLUSTER_TIME is bound to the individual clusters as in replicated and sharded clusters, not for single instances.

$$ROOT References the root document
$$CURRENT References the start of the field path, which by default is ROOT

field path is the “access point” for your aggregation. When you want to do some aggregation on $item.quantity this will be the the field path. You could also write this as "$$CURRENT.quantity" where $$CURRENT defaults to the root of the current object. But you can change $$CURRENT whereas $$ROOT is constant

Regards,
Michael

[Laks]

Thanks @michael_hoeller for your response. This is helpful. One follow up question if you don’t mind, $$NOW on replica sets and shared clusters will also return the same value as $$CLUSTERTIME.

So is there any significance behind having separate system variable ? i.e., one for single instances and other for cluster setup ?

[michael_hoeller (Michael Höller)]

Hello @Laks
good question, personally I never ran into a problem using $$NOW. I don’t know the answer but will see if I can find someone who knows.
Regards,
Michael

[michael_hoeller (Michael Höller)]

Hello @Laks

On your question

So is there any significance behind having separate system variable ? i.e., one for single instances and other for cluster setup ?

I can now compile an answer, many thanks to @Asya_Kamsky for details and @Stennie_X for a very elaborate and helpful explanation.

TL;DR

The key answer to your question is to understand the difference between the local time of an environment running a MongoDB process vs. the logical time on distributed system to ensure causal consistency on distributed events.

$$NOW is wall time and returns a BSON Date value
$$CLUSTER_TIME is reflecting logical cluster clock time and returns a BSON Timestamp

The details

@Stennie_X

wall time typically refers to “elapsed time as measured on a wall clock”, but in this context it means the local datetime for the environment running a MongoDB process.

Distributed MongoDB deployments gossip a logical cluster time(Logical clock - Wikipedia) to ensure causal consistency for the order of distributed events. For example, an event that occurs at the logical timestamp T1 must always occur before an event that happens at T2 even if members of a distributed deployment are not perfectly in sync with their local times. A Timestamp is composed of a datetime value and an incrementing counter which sequences events.

In a distributed cluster $$NOW is effectively the datetime prefix from the current $CLUSTER_TIME so aggregation queries have a consistent reference when executing across multiple members of the cluster.

You will find further details/information

about vector clock and causal consistency in the MonogDB github repro:

The vector clock is used to manage various logical times and clocks across the distributed system, for the purpose of ensuring various guarantees about the ordering of distributed events (ie. “causality”).

These causality guarantees are implemented by assigning certain logical times to relevant events in the system. These logical times are strictly monotonically increasing, and are communicated (“gossiped”) between nodes on all messages (both requests and responses). This allows the order of distributed operations to be controlled in the same manner as with a Lamport clock.

about Logical clock on wikipedia:

A logical clock is a mechanism for capturing chronological and causal relationships in a distributed system. Often, distributed systems may have no physically synchronous global clock. Fortunately, in many applications (such as distributed GNU make), if two processes never interact, the lack of synchronization is unobservable. Moreover, in these applications, it suffices for the processes to agree on the event ordering (i.e., logical clock) rather than the wall-clock time. The first logical clock implementation, the Lamport timestamps, was proposed by Leslie Lamport in 1978 (Turing Award in 2013).

about Lamport Clock / timestamp (my favourite):

Use logical timestamps as a version for a value to allow ordering of values across servers

Regards,
Michael