Sync document_filters stopped working for $or

It was working before MongoDb Atlas automatically migrated the Sync rules out for Sync configuration into the common Rules configuration.

But after that, I started getting an error if I used “$or”

The sync:

{
  "type": "flexible",
  "state": "enabled",
  "development_mode_enabled": false,
  "service_name": "mongodb-atlas",
  "database_name": "MyDatabase",
  "is_recovery_mode_disabled": false,
  "queryable_fields_names": ["userId", "taskId"]
}

The rules.json:

{
  "collection": "MyCollection",
  "database": "MyDatabase",
  "roles": [
    {
      "name": "my_role_name",
      "apply_when": {},
      "document_filters": {
        "write": false,
        "read": {
          "$or": [
            { "taskId": { $in: "%%user.custom_data.ownedTasks" }},
            { "taskId": { $in: "%%user.custom_data.tempTasks" }},
          ]
        }
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    }
  ]
}

The app id deployed properly, but I get error while running a query:

Error occurred while executing findOne: cannot use 'taskId' in expression; only %% like expansions and top-level operators may be used as top-level fields

Other options I have tried:

"read": { "taskId": "someTask"  } // works
"read": { "taskId":  { "$in":  "%%user.custom_data.ownedTasks"}  } // works

Hi,

I’m taking a look into the issue right now, but in the meantime do you mind linking you app URL (looks like https://realm.mongodb.com/groups/<group-id>/apps/<app-id>)?

Jonathan

Hi @Jonathan_Lee,

sure thing, this is the app url:
https://realm.mongodb.com/groups/6192acea6e11d14b9a1454eb/apps/6192ad66a4c07eea3a528d8d

thanks

Hey @Georges_Jamous,

This is actually a known issue that existed prior to the Rules migration, when permissions in Flexible Sync were used in a non-sync context, so the migration itself shouldn’t have introduced a regression here. We have a ticket to fix this issue.

In the meantime, I would suggest doing one of the following:

  1. Create an app for Flexible Sync and an app for non-sync requests

The flexible sync app would use the permissions as you have defined above, and the non-sync app would use a nearly identical rule but using the %%root operator like this:

{
  "collection": "MyCollection",
  "database": "MyDatabase",
  "roles": [
    {
      "name": "my_role_name",
      "apply_when": {},
      "document_filters": {
        "write": false,
        "read": {
          "$or": [
            { "%%root.taskId": { $in: "%%user.custom_data.ownedTasks" }},
            { "%%root.taskId": { $in: "%%user.custom_data.tempTasks" }},
          ]
        }
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    }
  ]
}

Note that the reason we can’t use this rule in a flexible sync app is because the %%root expansion is unsupported.

  1. Define a role for non-sync requests, and one for sync requests
{
  "collection": "MyCollection",
  "database": "MyDatabase",
  "roles": [
    {
      "name": "nonSyncRole",
      "apply_when": { "_id": { $exists: true } },
      "document_filters": {
        "write": false,
        "read": {
          "$or": [
            { "%%root.taskId": { $in: "%%user.custom_data.ownedTasks" }},
            { "%%root.taskId": { $in: "%%user.custom_data.tempTasks" }},
          ]
        }
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    },
    {
      "name": "my_role_name",
      "apply_when": {},
      "document_filters": {
        "write": false,
        "read": {
          "$or": [
            { "taskId": { $in: "%%user.custom_data.ownedTasks" }},
            { "taskId": { $in: "%%user.custom_data.tempTasks" }},
          ]
        }
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    }
  ]
}

This takes advantage of the idea that apply_when expressions referencing document fields in Flexible Sync doesn’t work, but does work for non-sync requests; this is because roles are applied per-document in non-sync requests, whereas they are applied per-sync-session on flexible sync requests (at session start, no documents have been queried for yet). Hence, the first role above will always apply on non-sync requests, and the latter will always apply on Flexible Sync requests.

Note that the first role is sync incompatible because it references the %%root expansion, and will be reported as such in the UI with warning banners / modals (trying to set a sync incompatible role would actually result in an error in the CLI, but is tolerated in the UI – so this change would have to be done through the UI). This should be fine because the first role will never actually get used in flexible sync requests.

If you don’t want to deal with sync incompatible roles, then another option would be to capture the $or in the apply_when expression of a role, and then have a “catch-all” role for non-sync requests like:

{
  "collection": "MyCollection",
  "database": "MyDatabase",
  "roles": [
    {
      "name": "nonSyncRole",
      "apply_when": { "$or": [
            { "%%root.taskId": { $in: "%%user.custom_data.ownedTasks" }},
            { "%%root.taskId": { $in: "%%user.custom_data.tempTasks" }},
      ] },
      "document_filters": {
        "write": false,
        "read": true
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    },
    {
      "name": "catchAllForNonSyncRequestsRole",
      "apply_when": { "_id": { $exists: true } },
      "document_filters": {
        "write": false,
        "read": false
      },
      "read": false,
      "write": false,
      "insert": false,
      "delete": false,
      "search": false
    },
    {
      "name": "my_role_name",
      "apply_when": {},
      "document_filters": {
        "write": false,
        "read": {
          "$or": [
            { "taskId": { $in: "%%user.custom_data.ownedTasks" }},
            { "taskId": { $in: "%%user.custom_data.tempTasks" }},
          ]
        }
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    }
  ]
}

Under this configuration, the third role would apply to sync requests, whereas the first two will apply to non-sync requests.

Let me know if any of those options work for you,
Jonathan

@Jonathan_Lee, first of all thank you for this detailed answer!

I will be trying it out and let you know what we went with.

Some notes of now:

A. so, a cloud function that is being invoked from a Flexible Sync Enabled App Client (iOS) and that is executing a query to the DB, is considered a non-sync context?


B. I am not sure about this, we have tests that call cloud functions and we never saw such error, the tests have not changed.
C. What is the purpose of this apply_when here ?

Thanks

Yep. A “non-sync” context also includes things like: querying MongoDB from a Realm SDK, DataAPI, GraphQL, etc.

Hmm, I can poke around in our logs to see if I can dig anything up there. Do you happen to recall roughly when you started seeing these errors? Also, do you remember if the above usage of the $or operator existed prior to the migration in the old flexible sync permissions? (if using the UI, the “old permissions” was the permissions JSON blob in the sync config page; if using the CLI, this was the object under the “permissions” key in sync/config.json)

When referencing document fields, the apply_when expression will evaluate to false when used in flexible sync, because it is evaluated on a per-session basis (at session start, no documents have been queried for yet). Since role evaluation occurs at a per-document basis for non-sync requests, and the _id field is required for MongoDB documents, then this “trivial” apply_when will always evaluate to true for non-sync requests. In turn, a non-sync request will never have the last role be applied due to role order evaluation.

Thanks for your help, we finally went with the last option, and it seems to work fine for our use case.

{
  "name": "nonSyncRoleWrite",
  "apply_when": { "$or": [ ... ] },
  "document_filters": { "write": true, "read": false },
  ...
},
{
  "name": "nonSyncRoleRead",
  "apply_when": { .... },
  "document_filters": { "write": false, "read": true },
 ...
},
{
  "name": "catchAllForNonSyncRequestsRole",
  "apply_when": { "_id": { "$exists": true } },
  "document_filters": { "write": false, "read": false },
  ...
},
{
  "name": "syncRole",
  "apply_when": {},
  "document_filters": {
    "write": { "$or": [ .... ] },
    "read": { "$or": [ .. ] }
  },

Hmm, I can poke around in our logs to see if I can dig anything up there. Do you happen to recall roughly when you started seeing these errors? Also, do you remember if the above usage of the $or operator existed prior to the migration in the old flexible sync permissions? (if using the UI, the “old permissions” was the permissions JSON blob in the sync config page; if using the CLI, this was the object under the “permissions” key in sync/config.json )

What I recall is that $or always existed, and yes in the old permission prior to the migration as well

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.