Jane Fine

5 results

Introducing Gap Filling for Time Series Data in MongoDB 5.3

At MongoDB we are all about letting developers innovate with data. Time series is the fastest-growing data-intensive workload, and our native time series capabilities let you build applications faster and get more insight from time series data with less cognitive load. It’s common for time series data to have gaps, such as when an IoT sensor goes offline. But in order to perform analytics and ensure correct results, time series data needs to be continuous. You may also want to create histograms or correlate data sets to enable more complex operational analytics in the context of app development. Gap filling , now available in MongoDB 5.3 Rapid Release , in combination with the densification we introduced in MongoDB 5.1, helps you better handle missing data to easily create and surface valuable insight. The two new aggregation stages create a simple, streamlined way to deal with missing data across time series and regular collections powering analytics for any use case. The $densify stage creates new documents to eliminate the gaps in the time or numeric domain at the required granularity level, and $fill sets values for the fields when a value is null or missing. Filling missing values can be done with a constant or using linear interpolation, carrying over the last observation or carrying backward the next observation. Input documents: tracking of temperature, motion, and inventory in a storage room This function produces an hourly view of the metrics for each storage room. When temperature data is missing it should be interpolated linearly, motion should default to 0, and quantity of inventory should be carried over from the last known point. Output: The second document was generated based on the two surrounding documents. Previously these types of complex analytics were possible only in specialized systems such as dedicated time series databases or data warehouses. Architecturally, technology practitioners had to make the no-win trade-off between a niche, often immature technology dedicated solely to time series workloads and disconnected from systems of record containing the full complement of enterprise data and exporting time series data into data warehouses, thereby making it hard to operationalize insights. Both involve managing multiple data silos and fragile ETL pipelines driving up complexity and cost. Workarounds to these approaches often involve developers building complex data pipelines to fill in the gaps, potentially at the application layer, leading to poor query performance or limiting analytics to small data sets. With MongoDB 5.3, developers can build rich analytics on time series data in flight and deliver operational insight to their users as part of the application experience. MongoDB 5.3 is available now. If you are running Atlas Serverless instances or have opted in to receive Rapid Releases in your dedicated Atlas cluster, then your deployment will be automatically updated to 5.3 starting today. MongoDB 5.3 is also available as a Development Release for evaluation purposes only from the MongoDB Download Center. Consistent with the new release cadence announced last year, the functionality available in 5.3 and the subsequent Rapid Releases will roll up into MongoDB 6.0, our next Major Release scheduled for delivery later this year. For a more in-depth explanation on gap filling for time series, check out our article on the MongoDB Developer Hub . Safe Harbor Statement The development, release, and timing of any features or functionality described for our products remains at our sole discretion. This information is merely intended to outline our general product direction, and it should not be relied on in making a purchasing decision. Nor is this a commitment, promise, or legal obligation to deliver any material, code, or functionality.

April 6, 2022

Building Intelligent Apps with MongoDB and Google Cloud - Part 1

Data analytics is a perpetual underachiever. Every generation of tools promises us better insight and never quite delivers. So we get stuck re-platforming and re-designing, hoping the next iteration will finally get us to the intelligence utopia. Yet modern applications must provide rich experiences, offer decision support, and continuously learn and adapt to win their users. Analytics and AI are at the heart of these Intelligent Apps . We decided to build an Intelligent App to demonstrate how easy it is to take advantage of ML and AI cloud services without hiring a team of data scientists. First, we built a simple e-commerce application - MongoDB SwagStore - using React and MongoDB Stitch with MongoDB Atlas on GCP . Stitch saved us hundreds of lines of code and our app was ready in days. But aside from implementing stock replenishment notifications with Stitch Triggers and Twilio , it wasn’t very intelligent... yet. We enabled our SwagStore with a product recommendation engine. Rather than implementing a recommendation engine from scratch, we used Google Cloud ML to train and tune a TensorFlow model that implements a WALS collaborative filtering algorithm . We then used Google Cloud Endpoints to serve up these personalized recommendations. When a user authenticates, MongoDB Stitch sends an HTTP GET request to the Google Cloud Endpoint to obtain a list of recommended products. A Stitch Function updates the recommendations array in the user document with the returned result. exports = function() { //services const gcp = context.services.get("GoogleCloudRec"); const mongodb = context.services.get("mongodb-atlas"); //my swagstore collection const users = mongodb.db("swagstore").collection("users"); const products = mongodb.db("swagstore").collection("products"); return users.findOne({user_id: context.user.id}) .then(user => { if(!user.gcpId) { return []; } //URL to GCP cloud endpoint const url = `https://jfmlrecengine.appspot.com/recommendation?userId=${user.gcpId}`; return gcp.get({ url }).then(response => { console.log("Retrieved Recommendations"); return EJSON.parse(response.body.text()); }) .then(result => { // Get the product info for the array of product ids return products.find({id: {"$in": result.articles}}, {_id:0, id:1, name:1, image:1}).toArray(); }) .then(products => { console.log(JSON.stringify(products)); // Write the products to the user document return users.updateOne({"gcpId": user.gcpId}, { $set: { "personalized_recs" : products}}) .then(() => { return products }); }); }); }; So when Jane logs into SwagStore she will see these product recommendations: And Jasper - different ones: By using MongoDB Stitch combined with powerful cloud services and APIs you can build a recommendation system like this very quickly and plug it right into you operational app getting your developers and data scientists to work together, operationalize insight, and deliver intelligence to your customers. Give it a try! Stay tuned for Part 2 where SwagStore becomes even more intelligent with an AI chatbot.

October 2, 2018