Permission issues migrating partition-based sync to flexible sync

We are currently using partition-based sync. We are testing the migration to flexible-sync and experiencing some issues while migrating the permission logic.

In the partition-based model, we use “key=value” style _partitionKey. We have 2 basic partitions

  1. user partition, which is user=user.data.externalUserId, for instance user=fbdc4c82-3c18-4bee-9064-445b75f93cfe
  2. user metadata partition, which is userMatadata=user.data.externalUserId, for instance userMetadata=fbdc4c82-3c18-4bee-9064-445b75f93cfe

The access rules to both of these partitions are pretty trivial - the user can read and write both of those partitions when he is the owner. And that is governed by canReadPartition and canWritePartition functions that have the exact same body.

exports = function (partition) {
    console.log(`Checking if can sync a write for partition = ${partition}`);

    const user = context.user;

    let partitionKey = "";

    const splitPartition = partition.split("=");
    let partitionValue;
    if (splitPartition.length == 2) {
        partitionKey = splitPartition[0];
        partitionValue = splitPartition[1];
        console.log(`Partition key = ${partitionKey}; partition value = ${partitionValue}`);
    } else {
        console.log(`Couldn't extract the partition key/value from ${partition}`);
        return false;
    }

    switch (partitionKey) {
        case "user":
        case "userMetadata":
            console.log(`Checking if partitionValue(${partitionValue}) matches user.id(${user.data.externalUserId}) – ${partitionKey === user.data.externalUserId}`);
            return partitionValue === user.data.externalUserId;
        default:
            console.log(`Unexpected partition key: ${partitionKey}`);
            return false;
    }
};

Now, when migrating to flexible sync I need to define roles that would mimic this behavior. And that’s what I can’t figure out as after migration I get the following error

Received: ERROR "Permission denied (BIND, IDENT, QUERY, REFRESH)" (error_code=206, is_fatal=false, error_action=ApplicationBug)

The latest version of my role which I’ve named partition_owner looks like this.

{
  "roles": [
    {
      "name": "partition_owner",
      "apply_when": {
        "$or": [
          {
            "_partitionKey": "user=%%user.data.externalUserId"
          },
          {
            "_partitionKey": "userMetadata=%%user.data.externalUserId"
          }
        ]
      },
      "document_filters": {
        "write": {
          "$or": [
            {
              "_partitionKey": "user=%%user.data.externalUserId"
            },
            {
              "_partitionKey": "userMetadata=%%user.data.externalUserId"
            }
          ]
        },
        "read": {
          "$or": [
            {
              "_partitionKey": "user=%%user.data.externalUserId"
            },
            {
              "_partitionKey": "userMetadata=%%user.data.externalUserId"
            }
          ]
        }
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    }
  ]
}

What am I doing wrong here?

Hi, I suspect what is going wrong here is that "user=%%user.data.externalUserId" is being interpreted as a string value and the expansion is not actually being run on the user object, so it is looking for _partitionKey to equal the value user=%%user.data.externalUserId.

As a first question, it seems like the partition concept did not quite fit what you wanted, so you had to add a key/value storage into it. Are user and userMetadata fields on the documents themselves? If so, I suspect the ideal permissions for you would be to just define permission like this:

"read": {
          "$or": [
            {
              "user": "%%user.data.externalUserId"
            },
            {
              "userMetadata": "%%user.data.externalUserId"
            }
          ]
}

Additionally, I believe you should remove the apply_when in the statement. For device sync, this field cannot reference fields in a document as it is applied at session start (not on each individual document). See here: https://www.mongodb.com/docs/atlas/app-services/rules/roles/#how-app-services-assigns-roles

Best,
Tyler

Hey Tyler

All of the entities that I have the _partitionKey values in the format of “user=EXTERNAL_USER_ID” or “userMetadata=EXTERNAL_USER_ID”. I do not have a user or a userMetadata field.

Can we something validate that the "user=%%user.data.externalUserId" expression is not being string-interpolated and being used as is?

If that’s the case, will I have to create a new field called “OwnerId” for instance, and

  1. Create a trigger that would set the OwnerId to the EXTERNAL_USER_ID for any entity that’s inserted
  2. Write a script to set the OwnerId to the EXTERNAL_USER_ID for all existing entities.

If you think that’s the most straightforward approach, I’d go for it.

Hi @Gagik_Kyurkchyan,

I imagine creating that new OwnerId field as you described above would be your best bet, assuming you want to avoid breaking changes / modifying the existing data for _partitionKey.

The App Services Rules system isn’t really designed for handling the format of the values for _partitionKey unfortunately (it looks for the expansion identifier %% at the beginning of the string in particular). Note that it isn’t possible to call a function here in the same manner as before because the function operator is evaluated at session start – before any documents have been observed.

I imagine the role you’ll want to define will look something like this in JSON:

{
    "name": "role",
    "apply_when": {},
    "document_filters": {
        "read": { "OwnerId": "%%user.data.externalUserId" },
        "write": { "OwnerId": "%%user.data.externalUserId" }
    },
    "read": true,
    "write": true,
    "insert": true,
    "delete": true,
    "search": true
}

Let me know if that works,
Jonathan

Hey @Jonathan_Lee

Thanks for getting back.

I was trying several things before replying here. Here’s my journey

First thing I tried is the strategy I mentioned

  1. I’ve added a new “CreatedBy” field to the realm schema
  2. I’ve added a database trigger to set this field to the user’s ID every time a document was inserted or updated.
  3. I’ve adjusted the role configuration to the following
{
  "roles": [
    {
      "name": "owner",
      "apply_when": {},
      "document_filters": {
        "write": {
          "CreatedBy": "%%user.data.externalUserId"
        },
        "read": {
          "CreatedBy": "%%user.data.externalUserId"
        }
      },
      "read": true,
      "write": true,
      "insert": true,
      "delete": true,
      "search": true
    }
  ]
}

Once I did this, I decided to launch the app from the master branch without changing any code as that’s what I am trying to achieve - seamless activation of flexible sync. Unfortunately, that didn’t go well. I would receive invalid permissions errors.

I decided to add the CreatedBy field to the client code and ensure its value is set correctly. Once I did that, I was finally able to sync the data to the device. This means, we won’t be able to migrate to flexible sync unless we release a new version of the app that has CreatedBy field and everybody needs to upgrade to that latest version.

However, there are more issues that I am experiencing. When I try to create entities like before, they won’t sync. Sync times out and I see the following error in my App Service logs

ending session with error: integrating changesets failed: operation was cancelled after retrying 12 times (ProtocolErrorCode=201)

So, apparently, there are more deeper issues we will have to address.

My question is the following, are there any real-world scenarios where partition-based sync was seamlessly migrated to flexible sync?
Perhaps, a better option would simply be to deprecate the old app and start a new one that has flexible-sync built in? And suggest our customers to switch?

If updating the schema & deploying a new version of the app is unacceptable, then you could consider using the $regex operator on the “_partitionKey” field like this:

{
    "name": "role",
    "apply_when": {},
    "document_filters": {
        "read":  { "_partitionKey": { "$regex": "%%user.data.externalUserId" }, "$or": [ { "_partitionKey": { "$regex": "^user=<externalUserId-regex>$" } },  { "_partitionKey": { "$regex": "^userMetadata=<externalUserId-regex>$" } } ] }
        "write":  { "_partitionKey": { "$regex": "%%user.data.externalUserId" }, "$or": [ { "_partitionKey": { "$regex": "^user=<externalUserId-regex>$" } },  { "_partitionKey": { "$regex": "^userMetadata=<externalUserId-regex>$" } } ] }
    },
    "read": true,
    "write": true,
    "insert": true,
    "delete": true,
    "search": true
}

It’s a bit ugly, but breaking down the rule expression here for the document read/write filters:

  1. The "_partitionKey": { "$regex": "%%user.data.externalUserId" } will match for documents that contain the expanded value of "%%user.data.externalUserId" for "_partitionKey".
  2. The "$or": [ { "_partitionKey": { "$regex": "^user=<externalUserId-regex>$" } }, { "_partitionKey": { "$regex": "^userMetadata=<externalUserId-regex>$" } will match for documents that resemble the possible formats for the values of the “_partitionKey” field IIUC (the one with “user=” and the one with “userMetadata=”). Note that <externalUserId-regex> is a placeholder in this example and should be substituted in with a regex that matches the format of the values of user.data.externalUserId

Using this above configuration should hopefully tightly match the current partition-based sync permissions in a way that is compatible with flexible sync. Another thing worth mentioning here is that there is obviously some amount of performance hit with using the “$regex” operator now, and long term it may be better to continue to try and use a new field for permissions (it will definitely be faster).

Let me know what you think,
Jonathan

Hey @Jonathan_Lee sorry, I did not notice this reply.

We went forward with adding a new property for a proof of concept in a development environment. Even though the sync permissions worked as expected, there were other more substantial problems with synchronization that broke that used to work before.

As that was going to be a significant effort to fix, we decided to postpone the flexible sync migration for a while as there were other high priority items.

As for the strategy using regular expressions to match the partition sync strategy that we currently have, that’s a good idea! I am definitely going to try that once we return to the flexible sync migration feature. This at least solves the problem of data migration. and introduction of a new field.

Thanks for the information.

Hi team, any update on how to successfully perform interpolation with the %%user expansion for the rules.

e.g.

{
  "_myField": "key=%%user.id"
}

where key= is a constant. If %%user.id resolved to "sally" then, this when expanded this would look like

{
  "_myField": "key=sally"
}

@Jonathan_Lee – the below doesn’t seem to work as you suggested?

{ 
  "_myField": { "$regex": "^key=%%user.id$" }
}

Hello @toheeb,

The issue is that App Services does not support interpolation of expansions midway through a string. App Services looks for the expansion operator (%% ) at the beginning of a string, and uses that to indicate that a value should be expanded.

Thus, this is why I suggested the following:

I think I was a bit unclear on what <externalUserId-regex> means here though. <externalUserId-regex> should not be the expansion %%user.id, but instead it should be a regex capturing the values of the user IDs themselves (eg. fbdc4c82-3c18-4bee-9064-445b75f93cfe). If I’m not mistaken, this seems to be the format of a UUID.

A regex matching a UUID (assuming only lowercase letters) would look like:

[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}

I would therefore recommend trying to replace <externalUserId-regex> with [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} in the document filters read/write expressions:

    "document_filters": {
        "read":  { "_partitionKey": { "$regex": "%%user.data.externalUserId" }, "$or": [ { "_partitionKey": { "$regex": "^user=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" } },  { "_partitionKey": { "$regex": "^userMetadata=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" } } ] }
        "write":  { "_partitionKey": { "$regex": "%%user.data.externalUserId" }, "$or": [ { "_partitionKey": { "$regex": "^user=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" } },  { "_partitionKey": { "$regex": "^userMetadata=[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" } } ] }
    },

Let me know if that works,
Jonathan