How Inviz.ai built Search Relevancy Platform with MongoDB Realm

MongoDB Realm and the Realm Mobile Database help mobile developers to build performant, resilient mobile applications in significantly less time. Apps built on Realm work when users are offline, and can sync data in realtime whenever devices are connected. When building a mobile app, MongoDB Realm’s SDKs and application development services are used alongside the Realm Mobile Database to speed development velocity.

Realm Sync: Key to MongoDB Realm is Realm Sync, an out-of-the-box data synchronization service that keeps data up-to-date between different users and devices. Realm Sync reduces the code a developer needs to write when they want to start syncing data between a mobile device and Atlas, and makes the work a lot simpler.

Realm Mobile Database: Realm Sync is used alongside the Realm Mobile Database. The Realm Mobile database is an open source, fully-featured mobile datastore that’s embedded directly on a user’s device. It allows apps to read/write locally, meaning apps can work even when a user is offline. The Realm Mobile Database is object-oriented, so it maps to the way mobile developers think and code, and it works cross-platform, for both iOS and Android.

In this blog we explain how inviz.ai built a search relevancy platform across different channels using MongoDB Atlas and Realm in a seamless way.

Problem Statement At inviz we built a crowd sourced data labeling solution. With this solution raw data gets distributed to human judges to get annotated. To minimize annotation costs, we picked judges from remote places, which presented connectivity challenges.

We saw that power failures and poor network connectivity were causing a significant loss of productivity from our human judges. There have been a high number of connection timeouts between UI or mobile applications to our cloud servers. A lack of network connectivity prevents the judges from being able to proceed with their work, and forces them to continuously check to see if the connection is strong enough to submit their assessments.

To account for this issue, we built a system where a sudden loss of connectivity would not prevent judges from continuing with their work.

Solution To reduce the issues we faced due to network hiccups, we wanted to build a solution where judges do not need to worry about being online in order to submit their judgement.

We decided the best way to solve the issue was to store the judgements locally first, directly on the judge’s device. Then, we would sync the data with the backend system whenever there was a strong enough network connection.

This idea led us to use MongoDB Realm, with Realm’s real-time data sync at the center of this new implementation.

With this new implementation every judge was assigned multiple tasks at a time. The judgement they created was first stored in the realm embedded on their mobile devices. At regular intervals, each device’s realm syncs the data to MongoDB’s cloud database, Atlas. With this approach the problems faced due to sudden drop of network connectivity have been solved to a great extent.

Implementation Details

Prerequisites

  • MongoDB Atlas account
  • Node JS >= 10.x
  • Android/iOS development tools
  • Android/iOS device or emulator

Backend Setup

  • List item
  • Login to MongoDB Atlas account and create a database (e.g loopr).
  • Add collections (products, projects, users) from MongoDB Atlas Collection Explorer, which can be done by connecting mongo shell through CLI.
  • Add seed data to collection. For reference, see the images below.

Go to the Realm Tab from the MongoDB Cloud console. Now we have to connect the realm to mongodb. Visit the Realm Tutorial to set up a barbone backend.

In Realm these steps are required:

  • Connect the mongodb database
  • Select collections that need to be synchronized
  • Define the schema, access controls, and relationships for collections
  • Start the Sync

Please refer to the below images for the schema declaration of different collections:

Projects Schema

(R-5) Users schema

Validation of collections data can be done through the Realm console to verify collections are valid with schema defined.

Writing the Client App

We built a cross platform mobile app in react-native, a framework for building native mobile apps. Realm has good support for the react-native framework. Realm made development of this app really fast since you don’t have to worry about writing back-end code. Realm takes care of it for you. React-native with MongoDB Realm makes rapid development possible, since the majority people with a UI background are familiar with JavaScript concepts.

Development Steps:

  • Set up a starter project with react-native.
  • Add npm packages for react mobile support like navigation, gesture, icons and Realm node package.
  • Define Realm entrypoint and schemas, providers, views. Refer to the below image for project structure.

Demo

This is the first screen that comes up when the app starts. Users can login with a registered email and password. enter image description here

After successful login, the app loads the project screen which lists all projects that a particular user has been assigned to. Users can scroll and click a particular project to view the Project Summary screen.

Project summary screens give a summary of annotated products like distribution of scores among products. Once users click on the Start Annotation button it takes them to the Products Annotation Screen.

The Products Annotation Screen displays a list of products added to a particular project. Products can be viewed by swiping left/right on the screen. There is a scoring group of buttons (Off-topic, Acceptable, Good, Excellent) after the product description. Users can rate the score by judging the query with the result image and description. Example: In the below images, the first query is Digital Camera but the result is memory card so the score would be Off-Topic. The second query is Digital Camera and the result is Canon digital camera so the score would be either Good or Excellent.

Realm Services and Features Used:

  • Realm Authentication: Email/Password and OAuth provider login
  • Realm Functions and Triggers: For specific tasks, like when a user is created add it to MongoDB
  • Realm Rules: For access control of data
  • Realm Mobile Database:
    • Local Application storage: lightweight and simple. Native query is available for different platforms for storing, updating and accessing the data.
    • Reactive UI: Live objects passed to views, so any change in data will directly update UI screens.
    • Realm Sync Protocol: Provides network reliability. The Realm Mobile Database is offline first meaning whatever read/write happens, it first goes to the application’s local database, then a background thread update, and retrieves changes from the remote database over the network.
← Previous

Nuxeo Achieves 11-Billion-Object Benchmark on AWS in Partnership with MongoDB Atlas

Pushing the Limits to Demonstrate Nuxeo's Ability to Manage the Largest, Most Complex Content Management Requirements Case Study Executive Summary Having already proven that their Content Services Platform could handle nearly unlimited workloads, Nuxeo set out to smash their established benchmark of 1 billion documents. The challenge: load 11 billion objects while maintaining the highest levels of system performance. MongoDB, Nuxeo’s data platform partner, helped make it possible with powerful tools capable of managing hundreds of metadata tags per object across billions of objects. In addition, Nuxeo leveraged the AWS cloud infrastructure and Amazon Elasticsearch Service – both key components of its Nuxeo Cloud offering – to achieve elastic scalability and the highest levels of indexing and search performance. Customers Expect Content Managment Without Limitations Global companies use Nuxeo to build applications that manage enormous volumes of digital content including scanned images, documents, PDFs, and even rich-media assets like high-resolution photos and video. Nuxeo supports high levels of complexity including customization, layers of security, and sophisticated metadata as well as complicated workflows and business processes. This enables Nuxeo customers to put their content to work solving complex business problems and delivering unique content-enabled solutions. David Woolston, VP Business Development at Nuxeo, describes Nuxeo as a content services platform on steroids. “One thing that drives Nuxeo’s success is our ability to scale up to handle the most complex workloads and largest repositories of documents,” explained Woolston. The secret is their relationship with AWS and MongoDB Atlas, the global cloud database service. As a disruptive player in the content services market, Nuxeo has differentiated itself through technology innovation. The Nuxeo Platform operates on the AWS Cloud, which allows for unparalleled flexibility and scalability. Nuxeo also uses the managed database service, MongoDB Atlas. Atlas is capable of managing hundreds of metadata tags across literally billions of Nuxeo objects, storing them securely and making them easily digestible and queryable in JSON-like documents. This means the Nuxeo team can focus on building new content services and platform capabilities rather than managing a database. Challenge Accepted “The biggest companies in the world are coming to us with their largest workloads and saying, ‘we know you can handle this.’ We believe our technology can scale almost endlessly with MongoDB Atlas and AWS,” said Woolston. “With a 1-billion-object benchmark already completed, we really wanted to push the limits and prove it out with a 11-billion-object benchmark.” The idea was to test Nuxeo from an optimal application and configuration perspective, not to simply throw money at more hardware. “In order to solve our customers’ complex challenges, Nuxeo provides an extremely robust platform. The deployment includes Elasticsearch, MongoDB Atlas, and all the bells and whistles of the Nuxeo Platform itself,” said Joe Quinto, Senior Program Manager for Nuxeo Cloud. “We wanted to push the boundaries of every element – stress as many components as we could. Effectively managing 11 billion documents was our yardstick. The goal was not just to hit the ceiling, but to break through it.” The biggest companies in the world are coming to us with their largest workloads and saying, ‘we know you can handle this.’ We believe our technology can scale almost endlessly with MongoDB Atlas and AWS. David Woolston, VP Business Development, Nuxeo Dynamic Testing with a Two-Phase Approach Nuxeo adopted a two-phase approach for its benchmarking exercise. In the first phase, the Nuxeo team used a single Nuxeo repository configured with MongoDB Atlas and Elasticsearch. The point of the exercise was to test the practical limits of a single-repository approach and also to illustrate the inherent advantages of a NoSQL solution like MongoDB. In the first phase, the team was able to successfully scale a single Nuxeo repository to 3 billion objects with no database sharding, a feat that’s virtually impossible with SQL-based technologies. In the second phase of the project, the Nuxeo team employed a multirepository approach and made use of multiple instances of MongoDB Atlas and the Amazon Elasticsearch Service as well as MongoDB sharding to efficiently scale to over 11 billion objects. For both phases of the benchmark the team used an actual Nuxeo Cloud deployment. This was not a highly orchestrated lab exercise. In both phases of the benchmarking project, the team continuously tested to ensure that the Nuxeo Platform and underlying technologies would scale and perform at a level that would meet enterprise customers’ high expectations. The team monitored metrics to determine if, as the repository grew, users could continue to import new objects and metadata at an extremely high rate of ingestion. Since this was a real-world exercise, Nuxeo employed its default Ingestion Pipeline throughout the benchmarking project, complete with metadata import and full-text indexing. Nuxeo also employed automated testing to address common user activities for content management. This included search (both database queries and full-text searches) and navigation as well as create, read, update, and delete (CRUD) actions. Over three months, as Nuxeo was progressing toward their 11-billion-object benchmark and continuing to load new objects and data into the system, MongoDB Atlas was managing the database. While the test was running, the MongoDB team monitored data access patterns and gave advice on best practices for such a large number of documents. It was important to identify points where they needed to increase capacity on the platform before moving to higher tiers. Using more than 100 metrics, MongoDB was able to identify tipping points and suggest ways to improve the environment before going on to the next level. “The Nuxeo team was prepared with a solid plan and they let us know what they wanted to achieve,” said Diego Burstyn, Sr. Solutions Architect at MongoDB. “Our role was to correlate that with what we were seeing under the hood.” Pushing the Limits While Maintaining Response Times and Throughput As previously mentioned, the key success indicators the Nuxeo team was looking for were response times and throughput. It was important to see that the application was responding as expected, and enormous amounts of data could be loaded in a reasonable amount of time. A critical outcome for the project was to be able to provide Nuxeo customers with real-world guidance and best practices for scaling up the Nuxeo Platform along with key services, like MongoDB Atlas. The team categorized its learnings in three specific areas: elasticity, steps, and real-life usability. Bragging Rights and Lessons Learned The 11-billion-object benchmark test proved Nuxeo’s ability to scale with AWS Cloud elasticity. Flexible scalability allowed the team to expand volume when a capacity limit was reached, or for a temporary need like re-indexing data. By implementing the test with deliberate steps, the team learned about infrastructure and configuration adjustments that needed to be made to maintain optimal performance at extreme high volumes. Testing in an actual environment, with real documents, demonstrated how the Nuxeo Platform can handle enormous document volumes in real-life applications. Traditional content management systems can’t scale the way we can,” said Woolston. “But beyond that, the power of MongoDB Atlas combined with the complete toolset and support we get from AWS allows us to scale in the most efficient and intelligent way possible. Hitting the 11-billion-object benchmark was more than a matter of bragging rights. It provided tangible proof of the unique value proposition Nuxeo offers through its partnership with MongoDB Atlas and AWS. This exercise delivered an understanding of how these different components scale and the best practices to do this efficiently. With these learnings and meaningful data, Nuxeo customers are better prepared to scale up, either in Nuxeo Cloud or in their own cloud environments. Get Started Today Try MongoDB Atlas on AWS: Redeem promo code NuxeoAtlas100 for $100 in Atlas Credits Schedule a demo of the Nuxeo Platform

October 7, 2020
Next →

Using MongoDB Skill Scanner to Build Better Training Programs

Technology leaders know that transformation is about more than just adopting modern technologies like MongoDB. The entire organization has to rally behind change — which is no easy task. The skills that modern development teams need are evolving faster than ever, and hiring to fill skills gaps can be too time-consuming and expensive of a process for many organizations. So it’s imperative that we plan for how we want to bring our people with us on our modernization journey, and proactively upskill them on the technologies we’re betting on. Because what happens if you choose MongoDB, but your developers don’t know how to use it? CIOs know that training programs are easier said than done. EY reported that 30% of CIOs acknowledge that their training programs are ineffective, and that they’re struggling to retain talent because of it. These leaders come to us to help them build and execute their MongoDB training programs , and seek advice on two extremely common yet critical challenges: How do we get away from the less effective one-size-fits-all approach? How do we measure the ROI of our training program and connect it to business impact? How we use MongoDB Skill Scanner to overcome training challenges Our Professional Services team uses a tool called MongoDB Skill Scanner to address both of these challenges. This tool helps us provide these three benefits to our customers looking to build a training program: Improve MongoDB proficiency: Teams can use Skill Scanner to quickly and easily assess the MongoDB skill gaps of their team members and gain a comprehensive understanding of their team’s MongoDB skills baseline. Increased productivity and accuracy: When team members have a comprehensive understanding of MongoDB, they are able to work more quickly and accurately on projects, leading to increased productivity and a higher quality of work. Save time and money with targeted Training: Using Skill Scanner, customers can avoid wasting time and money on trial-and-error learning. Instead, they can focus on improving their skills in a more targeted and efficient way with right-sized training plans. By leveraging this data, our customers’ engineers can engage in the right training at the right time, targeted for their job role and specific skill shortages. When a training program is built this way, engineers maximize their knowledge retention and minimize time away from their projects. Skill Scanner includes three role-based assessments, one for developers, database administrators, and DevOps respectively. Through a series of multiple choice questions, Skill Scanner provides customers with a clear understanding of their level of expertise across a set of technical skills that are critical for success in their role. After submitting the assessment, engineers will get results in each skill area outlining if they are beginner, intermediate, or advanced. Why data-driven training programs matter We’ve learned that it’s not enough to just tell teams to go watch training videos or webinars on their own, or to place everyone in the same one-size-fits-all program. Skills gaps vary from team to team, and individual to individual. The one-size-fits-all approach of some programs may not address individual learners' needs, wasting time and making it difficult for them to acquire new skills. By using Skill Scanner, we’re able to interpret this data to help determine which training courses your team should take. But we don’t only capture this data before doing training; we use Skill Scanner again after training programs are completed to see where immediate improvements have been made. This helps technology leaders prove the impact and ROI of their training, and gives them the confidence that their teams are ready to be successful with MongoDB. Developing a Precision Learning Program To go even further, our team can work with you to build a Precision Learning Program, where we use Skill Scanner data to build learning schedules that are unique to each individual. These schedules include a variety of short, blended, learning events such as classes, technical workshops, self-paced exercises, and project coaching. We’ve seen PLP lead to higher knowledge retention and of course, measurable project results. A customer who recently concluded their PLP saw a 43% increase in knowledge retention. Getting started building a personalized training program Skill gaps aren’t a novel problem IT leaders are facing. But with new digital courses, training, and technologies, the resources to close these gaps are at your fingertips. Skill Scanner and Precision Learning Program have been specifically designed to empower teams by offering targeted training that enhances their understanding of MongoDB. These short training events are carefully crafted to close skill gaps without compromising developer productivity. We’ve seen a variety of customers use this tool to help train their team’s individual needs, from needing to upskill new hires on their teams, projects with new MongoDB products, migrating to MongoDB Atlas, and more. It also saves your business the hours developers would've wasted searching for answers (and developers don’t want to spend their time that way, either). “We need help getting from point A to point B and feel MongoDB is uniquely positioned to help” — CTO at large insurance firm If you're interested in trying out MongoDB Skill Scanner or want to explore the MongoDB Precision Learning Program further, you can reach out to your account representative or contact us directly .

June 7, 2023