Realm Data and Partitioning Strategy Behind the WildAid O-FISH Mobile Apps
Rate this tutorial
In 2020, MongoDB partnered with the to create a mobile app for officers to use while out at sea patrolling Marine Protected Areas (MPAs) worldwide. We implemented apps for , , and , where they all share the same Realm back end, schema, and sync strategy. This article explains the data architecture, schema, and partitioning strategy we used. If you're developing a mobile app with Realm, this post will help you design and implement your data architecture.
MPAs—like national parks on land—set aside dedicated coastal and marine environments for conservation. WildAid helps enable local agencies to protect their MPAs. We developed the O-FISH app for enforcement officers to search and create boarding reports while they're out at sea, patrolling the MPAs and boarding vessels for inspection.
This video gives a great overview of the WildAid Marine Program, the requirements for the O-FISH app, and the technologies behind the app:
This article is broken down into these sections:
There are three frontend applications.
The two mobile apps (iOS and Android) provide the same functionality. An officer logs in and can search existing boarding reports, for example, to check on past reports for a vessel before boarding it. After boarding the boat, the officer uses the app to create a new boarding report. The report contains information about the vessel, equipment, crew, catch, and any laws they're violating.
Crucially, the mobile apps need to allow users to view and create reports even when there is no network coverage (which is the norm while at sea). Data is synchronized with other app instances and the backend database when it regains network access.
The web app also allows reports to be viewed and edited. It provides dashboards to visualize the data, including plotting boardings on a map. User accounts are created and managed through the web app.
All three frontend apps share a common backend application. The Realm app is responsible for authenticating users, controlling what data gets synced to each mobile app instance, and persisting the data to MongoDB Atlas. Multiple "agencies" share the same frontend and backend apps. An officer should have access to the reports belonging to their agency. An agency is an authority responsible for enforcing the rules for one or more regional MPAs. Agencies are often named after the country they operate in. Examples of agencies would be Galapogas or Tanzania.
The iOS and Android mobile apps both contain an embedded Realm mobile database. The app reads and writes data to that Realm database-whether the device is connected to the network or not. Whenever the device has network coverage, Realm synchronizes the data with other devices via the Realm backend service.
The Realm database is embedded within the mobile apps, each instance storing a partition of the O-FISH data. We also need a consolidated view of all of the data that the O-FISH web app can access, and we use MongoDB Atlas for that. MongoDB Realm is also responsible for synchronizing the data with the MongoDB Atlas database.
The web app is stateless, accessing data from Atlas as needed via the Realm SDK.
- All synced collections use the same attribute name and type for the partition key.
- The key can be a
objectId, or a
- When the app provides a partition key, only documents that have an exact match will be synced. For example, the app can't specify a set or range of partition key values.
A common use case would be to use a string named "username" as the partition key. The mobile app would then open a Realm by setting the partition to the current user's name, ensuring that the user's data is available (but no data for other users).
WildAid works with different agencies around the world. Each officer within an agency needs access to any boarding report created by other officers in the same agency. Photos added to the app by one officer should be visible to the other officers. Officers should be offered menu options tailored to their agency—an agency operating in the North Sea would want cod to be in the list of selectable species, but including clownfish would clutter the menu.
We use a string attribute named
agencyas the partitioning key to meet those requirements.
As an extra level of security, we want to ensure that an app doesn't open a Realm for the wrong partition. This could result from a coding error or because someone hacks a version of the app. When enabling Realm Sync, we can provide expressions to define whether the requesting user should be able to access a partition or not.
For O-FISH, the rule is straightforward. We compare the logged-in user's agency name with the partition they're requesting to access. The Realm will be synced if and only if they're the same:
At the highest level, the O-FISH schema is straightforward with four Realms (each with an associated MongoDB Atlas collection):
DutyChangerecords an officer going on-duty or back off-duty.
Reportcontains all of the details associated with the inspection of a vessel.
Photorepresents a picture (either of one of the users or a photo that was taken to attach to a boarding report).
MenuDatacontains the agency-specific values that officers can select through the app's menus.
You might want to right-click that diagram so that you can open it in a new tab!
Let's take a look at each of those four objects.
The app creates a
DutyChangeobject when a user toggles a switch to flag that they are going on or off duty (at sea or not).
DutyChangeinherits from the Realm
Objectclass, and the attributes need to be made accessible to the Realm SDK by making them
dynamicand adding the
@objcannotation. The Kotlin app uses the
@RealmClassannotation and inheritance from
Note that there is no need to include the partition key as an attribute.
In addition to primitive attributes,
userwhich is of type
Userobjects are always embedded in higher-level objects rather than being top-level Realm objects. So, the class inherits from
Objectin Swift. The Kotlin app extends the
@RealmClassannotation to include
(embedded = true).
Whether created in the iOS or Android app, the
DutyChangeobject is synced to MongoDB Atlas as a single
DutyChangedocument that contains a
Reportobject is at the heart of the O-FISH app. A report is what an officer reviews for relevant data before boarding a boat. A report is where the officer records all of the details when they've boarded a vessel for an inspection.
In spite of appearances, it pretty straightforward. It looks complex because there's a lot of information that an officer may need to include in their report.
Starting with the top-level object -
Reportclass contains Realm
RealmListin Kotlin) to store lists of instances of classes such as
Once synced to Atlas, the Report is represented as a single
BoardingReportsdocument (the name change is part of the schema definition):
Note that Realm lists are mapped to JSON/BSON arrays.
A single boarding report could contain many large photographs, and so we don't want to embed those within the
Reportobject (as an object could grow very large and even exceed MongoDB's 16 MB document limit). Instead, the
Reportobject (and its embedded objects) store references to
Photoobjects. Each photo is represented by a top-level
PhotoRealm object. As an example,
Attachmentscontains a Realm
Listof strings, each of which identifies a
Photoobject. will step through how we implemented this.
The general rule is that it isn't the best practice to store images in a database as they consume a lot of valuable storage space. A typical solution is to keep the image in some store with plentiful, cheap capacity (e.g., a block store such as cloud storage - Amazon S3 of Microsoft Blob Storage.) The O-FISH app's issue is that it's probable that the officer's phone has no internet connectivity when they create the boarding report and attach photos, so uploading them to cloud object storage can't be done at that time. As a compromise, O-FISH stores the image in the
Photoobject, but when the device has internet access, that image is uploaded to cloud object storage, removed from the
Photoobject and replaced with the S3 link. This is why the
Photoincludes both an optional binary
pictureattribute and a
pictureURLfield for the S3 link:
Note that we include the
referencingReportIDattribute to make it easy to delete all
Photoobjects associated with a
The officer also needs to review past boarding reports (and attached photos), and so the
Photoobject also includes a thumbnail image for off-line use.
Each agency needs the ability to customize what options are added in the app's menus. For example, agencies operating in different countries will need to define the list of locally applicable laws. Each agency has a
MenuDatainstance with a list of strings for each of the customizable menus:
When MongoDB Realm Sync writes a new
Photodocument to Atlas, it contains the full-sized image in the
pictureattribute. It consumes space that we want to free up by moving that image to Amazon S3 and storing the resulting S3 location in
pictureURL. Those changes are then synced back to the mobile apps, which can then decide how to get an image to render using this algorithm:
picturecontains an image, use it.
- Else, if
pictureURLis set and the device is connected to the internet, then fetch the image from cloud object storage and use the returned image.
- Else, use the
Photodocument is written to Atlas, the
newPhotodatabase trigger fires, which invokes a function named
The trigger passes the
newPhotoRealm function the
changeEvent, which contains the new
Photodocument. The function invokes the
uploadImageToS3Realm function and then updates the
Photodocument by removing the image and setting the URL:
uploadImageToS3uses Realm's AWS integration to upload the image:
The data model is deceptively simple. There's a lot of nested information that can be captured in each boarding report, resulting in 20+ classes, but there are only four top-level classes in the app, with the rest accounted for by embedding. The only other type of relationship is the references to instances of the
Photoclass from other classes (required to prevent the
Reportobjects from growing too large).
The partitioning strategy is straightforward. Partitioning for every class is based on the name of the user's agency. That pattern is going to appear in many apps—just substitute "agency" with "department," "team," "user," "country," ...
Suppose you determine that your app needs a different partitioning strategy for different classes. In that case, you can implement a more sophisticated partitioning strategy by encoding a key-value pair in a string partition key.
For example, if we'd wanted to partition the reports by username (each officer can only access reports they created) and the menu items by agency, then you could partition on a string attribute named
partition. For the
Reportobjects, it would be set to pairs such as
partition = "firstname.lastname@example.org"whereas for a
MenuDataobject it might be set to
partition = "agency=Galapagos". steps through designing these more sophisticated strategies.