How MongoDB Atlas Helps Hakkiri Deliver Continuous Clarity to Agile Software Organizations

Using microservices to turn the noisy data that's constantly emitted by agile software development tools into actionable information that lets development teams work more effectively can be a boon to boosting developer productivity and improving the developer experience. Enter Hakkiri, a startup who are doing just that with their Continuous Clarity platform. We spoke with Hakkiri co-founders Robert Orefice and James Smith to learn more about how Hakkiri is using MongoDB to power their platform.

Read more about how Hakkiri works with MongoDB's Startup Accelerator in Startup Spotlight: Hakkiri and their plan to boost business agility.

How does MongoDB help your Continuous Clarity platform?

Hakkiri’s platform helps Agile software teams run their process with deeper insights and greater transparency. We like to equate it to a fitness tracker for your Agile teams. Attach it to your tools, enter in your planned deliverables, and the rest is taken care of for you. We’ve already thought of what to measure, how to measure it, curated how to visualize it, and added automated feedback to keep you tracking towards your goals.

MongoDB helps us solve our technical problems in a few ways:

• We needed the ability to design and implement an extensible schema that would be easy to evolve over time. Our product roadmap includes incorporating more data entities from the delivery toolchain (ex: build, test, and deployment) and MongoDB’s powerful document data model gives us confidence that we’ll be able to iterate effectively.
• Our platform pulls all the data together into a set of dashboards that enable teams to track their work more easily than ever before (ex: see Cumulative flow across teams). As such, we do a lot of business intelligence and analytics, and MongoDB makes this easy and scalable with aggregation pipelines.
• MongoDB Atlas, the fully managed database service, gives us a trusted, secure, and encrypted database environment to store data. We don’t need to spin our wheels doing database administration work.
• And finally, we needed a database that could effectively manage time-series data. Our application processes event streams coming from delivery tools such as Jira.
  • Again, an extensible schema that’s easy to evolve over time is invaluable
  • Our data needs to be aggregated at multiple levels. MongoDB enables flexible and fast calculations of summaries at multiple levels, which comes in handy when a change at the lowest level requires recalculations up the chain
  • MongoDB also provides efficient storage of time-series data for all the entities we track and perform analytics on

Which databases did you use before MongoDB?

We spent most of our career working with Microsoft SQL Server with a bit of PostgreSQL. With Hakkiri, we started from a clean slate knowing that we needed a database technology that supported both cloud hosting and on-premise deployments. We evaluated Amazon DynamoDB and MongoDB. After doing some experimentation with both, we chose MongoDB (and haven’t looked back since) for several reasons:

1. It had the on-premise support we wanted.
2. We found the online resources (i.e. technical blogs and support documentation) to be some of the best we’d seen, which helped with our learning curve.
3. The power and flexibility of aggregation pipelines to do rich and efficient calculations.
4. MongoDB Atlas was a big draw — we like that it works seamlessly with our AWS ECS environments and that it was completely separate and secure with easy-to-use backup and migration features. Currently, we use Terraform to manage our application architecture; with Atlas, it’s easy to separate the life cycle of our database environments from the rest of our microservices architecture.

Can you share your application and MongoDB stack with us?

We utilize Atlas and recently utilized its migration features to seamlessly move from MongoDB 3.4 to MongoDB 4.0. On top of MongoDB, we utilize the remainder of the MERN stack. We currently have a set of stateless, RESTFul, microservices written in NodeJS that run in AWS ECS. As our customer base grows, we plan to leverage sharding and read-only nodes in Atlas.

Can you share any best practices on scaling MongoDB or operations?

We chose Atlas so that we’d have a very easy solution for this. Atlas handles all of the management of the underlying database infrastructure so that all we have to worry about is setting our cluster size. And with the upcoming release of Atlas auto-scale, we may never have to worry about not having enough provisioned resources.

We utilize Atlas for production environments and run MongoDB locally for development. We currently do not have MongoDB deployed for production workloads on-premise but having that option is part of why we chose MongoDB originally. For monitoring, we utilize the Atlas console and alerting features. We have set alerts that notify us via email and Slack channels should performance thresholds be breached. The embedded Performance Advisor has been particularly helpful in identifying issues and providing suggested remediations.

We also utilize MongoDB Compass and the Atlas Data Explorer for manually interacting with data; most commonly for data inspection and triage. After MongoDB World 2019, we also started to utilize them heavily for building aggregation pipelines. The conference definitely showed us a very cool feature that we weren’t taking advantage of. The way that the aggregation pipeline builder shows you the flow of data in the pipeline as you build it is very helpful and powerful.

What difference has MongoDB made to your performance?

MongoDB Atlas has worked fantastically for us. As a startup, we need to prioritize our time. Having a robust and secure database system is critical to our business success and Atlas has taken ~95% of that load off of our shoulders while keeping us confident that we’ve got a strong infrastructure and security posture.

How are you securing MongoDB?

The native security features of Atlas are part of what attracted us to the database. This includes encryption for data in flight and built-in encryption for data at rest. We also secure our accounts with multi-factor authentication to the MongoDB admin console. Finally, we utilize VPC Peering to connect our Atlas cluster to the rest of our application infrastructure hosted in AWS.

How does the MongoDB query language and aggregation framework work for you?

Our application displays aggregated views of data at each level which may vary by several dimensions (by identifier, by group Identifier, and/or by time range). The flexible and powerful query syntax enables us construct queries on the fly as JavaScript objects in NodeJS and keeps our code compact and DRY (Don’t Repeat Yourself). This enables us to provide a deeper level of transparency that organizations can’t get from their existing tools.

While we haven’t started implementing it yet, we are looking forward to utilizing the materialized-views available in the MongoDB 4.2 release to make this process even more efficient.

Are you planning to use MongoDB for other applications?

Yes, we will continue to utilize Atlas underneath all our customer facing products as we build out our roadmap of capabilities. We have also begun leveraging tools like Charts for our internal operations for analytics and reporting. We’ve recently started evaluating Stitch for ad hoc and batch processes. It’s great to have such convenient and effective tools to meet our needs with minimal effort.

What's most exciting for you about MongoDB 4.2?

We were very excited about several features announced at MongoDB World 2019. We are most excited about full-text search and the inclusion of Lucene into MongoDB Atlas natively. This is going to save us a lot of time and deliver a ton of value to customers. After that, the new aggregation pipeline features are exciting, particularly the ability to pass a pipeline to an update. This will help make our calculations faster and easier to maintain.

We also plan to take a look at MongoDB Charts further. We utilized the Charts beta for looking at data for internal administrative purposes and are planning to leverage it for some application use cases.

Finally, we’re looking forward to the auto-scaling feature that was announced for Atlas. The cluster size is really the only thing we need to keep a close eye on, so this will be yet another thing we can offload.

What advice would you give someone who is considering using MongoDB for their next project?

If you are coming from a relational database world, you should do a primer tutorial on how to map the relational concepts to the document/object concepts. Expect to have a learning curve to train yourself to think in objects versus tables but you should be pleasantly surprised that you’re productive very quickly.