Oxford Nanopore Technologies & MongoDB: Powering Real-Time Genetic Analysis with Docker, MongoDB, & AWS

Mat Keep
April 14, 2016 | Updated: June 26, 2020
#Business

Genetic analysis is entering the mobile age. Earlier this year scientific journal Nature published a paper showing how Ebola researchers in Guinea were able to analyse genetic material in hours, rather than the weeks it had previously taken. This increased speed meant doctors could better understand the spread of the disease. Then quickly develop strategies to stop it.

The hardware that enabled the genetic analysis is the MinION, from UK-based Oxford Nanopore Technology. The stapler-sized MinION is the data-capture side of the analysis, but for the purposes of this article we’re interested in data processing and analysis. In particular how Oxford Nanopore has been able to build a fast, agile and powerful cloud-based platform that has the potential to deliver biological analyses to any scientist, at any time, anywhere in the world.

The applications for this genetic analysis go far beyond the medical field and disease control. Oxford Nanopore is using technologies like MongoDB, Amazon Web Services, and Docker containers in its stated goal: “to enable the real-time analysis of any living thing, by any user, in any environment.”

A Billionth of a Meter

The MinION does its genetic magic through the use of nanopores. Each nanopore is just a billionth of a meter wide. The technology in the MinION threads the genetic material through the nanopores where tiny differences in each sample can be registered as electrical disruptions. If you want a more detailed explanation of nanopores, check out Oxford Nanopore Technologies’ website.

DNA sequencing can be associated with predictive human questions alone, for example “what probability is there that this person will develop a specific disease?” But human genome research is just a part of the equation, and the portable nature of the MinION means it might be suitable for a more diverse range of questions: Is the soup I’m about to eat safe? What type of disease am I looking at? Where did this pathogen originate? How can we grow more resilient plants? Is this hospital ward clean?

Crucially, these questions need to be answered quickly, and in a range of environments – from the science lab to the middle of the jungle.

Three Billion Bases in the Cloud

The cleverest sequencing tool in the world would be worthless if we were unable to process and understand the data it created. To deal with the volume and velocity of processing billions of lines of DNA, Oxford Nanopore Technologies built analysis capabilities offered by Metrichor, on powerful software that can scale seamlessly in the cloud.

Richard Carter, Associate Director, Data Integration at Oxford Nanopore gave a presentation at MongoDB Days where he noted:

“When we began building Metrichor services, it was clear our data would not fit in the neat rows and columns of a relational database. We needed a database that could look at our complex information in more flexible and dynamic ways. It was a straightforward decision to go with MongoDB. It’s robust, best of breed, and has the data modelling and analytics flexibility we required. We also observed the technology has an incredible community behind it, coupled with extensive documentation and training. All of which enable us to get productive with the technology much faster.”

The DNA data is read locally onto the MinION and it’s then sent to an Amazon Web Services cloud. The findings are then analysed before the results are sent back to the user’s laptop or displayed in web reports. All of this is driven by, and stored in the non-relational database MongoDB. Docker containers are used to package, deploy and run the software across the cloud deployment.

Carter also noted that: “The biology and hardware is the real trick, of course, but we needed power and scalability to run cloud based services as we wished.”

There were other challenges the team had during development of their software. They had a technical goal and a number of ways they could reach it while keeping the focus on the biology. It was essential they had the freedom to experiment and make significant changes as they went along.

“Happily, MongoDB supports an evolutionary approach to development.” explained Carter. “We were spinning up instances and working on the science almost instantly. The database got out of the way.”

Carter’s team does not have a database administrator. They have found that MongoDB Cloud Manager is able to provide all the monitoring data needed to keep a deployment healthy. Features like simple, automated deployment across any cloud region, continuous backups, and telemetry visualisations also mean administration doesn’t monopolise the developers’ time.

Giant Ideas

Guinea is just one of the many places where researchers are using Nanopore’s data architecture for analysis. In fact, NASA will soon be using the MinION for testing biological molecules on the International Space Station.

Regardless of the location, the combination of rigorous science and the power of cloud computing is ushering in a new way of understanding the world.

Read more about MongoDB and its implementation on the AWS cloud platform.
MongoDB on AWS: Guidelines and Best Practices

About the Author - Mat Keep

Mat is director of product and market analysis at MongoDB. He is responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

MongoDB 3.2.5-rc1 is released

MongoDB 3.2.5-rc1 is out and is ready for testing. This is a release candidate containing only fixes since 3.2.4. The next stable release 3.2.5 will be a recommended upgrade for all 3.2 users. Fixed in this release: SERVER-23274 Aggregate with out, then stepdown, out collection dropped. SERVER-23283 RangeDeleter does not log cursor ids correctly in deleteNow() SERVER-22964 IX GlobalLock being held while wating for wt cache eviction SERVER-22937 Retry catalog operations whenever possible SERVER-22831 Low query rate with heavy cache pressure and an idle collection SERVER-21681 In-memory storage engine not reporting index size All Issues | Downloads As always, please let us know of any issues. – The MongoDB Team

April 11, 2016
Next →

Security in Government Solutions: Why Secure By Default is Essential

Data security in government agencies is table stakes at this point. Everyone knows it’s essential, both for compliance and data protection purposes. However, most government agencies are working with solutions that require frequent security patches or built-on tools to protect their data. Today, the federal government is pushing its agencies to move to modernize their solutions and improve their security posture. For example, the DHS and Cybersecurity and Infrastructure Security Agency’s recently issued technical rule for modernization of the Protected Critical Information Infrastructure program – a program that provides legal protections for cyber and physical infrastructure information submitted to DHS. “The PCII Program is essential to CISA’s ability to gather information about risks facing critical infrastructure,” said Dr. David Mussington, Executive Assistant Director for Infrastructure Security. “This technical rule modernizes and clarifies important aspects of the Program, making it easier for our partners to share information with DHS. These revisions further demonstrate our commitment to ensuring that sensitive, proprietary information shared with CISA remains secure and protected.” So how can government agencies modernize their data infrastructure and find solutions that not only protect data but also power innovation? Let’s look into a few different strategies. 1. Why secure by default is key Secure by default means that any piece of software uses default security settings that are configured for the highest possible security out of the box. CISA Director Jen Easterly has addressed how using solutions that are secure by default is critical for any organization. “We have to have [multi-factor authentication] by default. We can't charge extra for security logging and [single sign-on],” Easterly said . “We need to ensure that we're coming together to really protect the technology ecosystem instead of putting the burden on those least able to defend themselves.” “The American people have accepted the fact that they’re constantly going to have to update their software,” she said. “The burden is placed on you as the user and that’s what we have to collectively stop.” Easterly is right. Secure-by-design solutions are vital to the success of data protection. The expectation should alway be that solutions have built-in, not bolt-on security features. One approach that’s gaining traction both in the public and private sectors is zero trust environments. In a zero trust environment, the perimeter is assumed to have been breached. There are no trusted users, and no user or device gains trust simply because of its physical or network location. Every user, device, and connection must be continually verified and audited. As the creator of zero trust, security expert John Kindervag, summed it up: “Never trust, always verify.” For government agencies, that means the underlying database must be secure by default, and it needs to limit users’ opportunities to make it less secure. 2. Security isn't just on-prem anymore; cloud is secure, too Cloud can be a scary word for public sector organizations. Trusting your sensitive data to the cloud might feel risky for those who handle some of the country’s most sensitive data. But, cloud providers are stepping up to meet the security needs of government agencies. There is no need to fear the cloud anymore. Government agencies and other public sector organizations nationwide are navigating cloud modernization through the lens of increased cybersecurity requirements outlined in the 2021 Executive Order on Improving the Nation’s Cybersecurity . “The Federal Government must adopt security best practices; advance toward Zero Trust Architecture; accelerate movement to secure cloud services, including Software as a Service (SaaS), Infrastructure as a Service (IaaS), and Platform as a Service (PaaS); centralize and streamline access to cybersecurity data to drive analytics for identifying and managing cybersecurity risks; and invest in both technology and personnel to match these modernization goals.” Also, the major cloud providers are well established, purpose-built options for government users. AWS GovCloud, for example, is more than a decade old and was “ the first cloud provider to build cloud infrastructure specifically designed to meet U.S. government security and compliance needs.” This push by the federal government toward cloud modernization and increased cybersecurity will be a catalyst in upcoming years for rapid cloud adoption and greater dependence on cloud solutions designed specifically for government users. 3. Security features purpose-built for goverment needs are essential Government agencies are held to a higher standard than those in the private sector. From data used in sometimes life-or-death missions to data for students building their futures in educational institutions (and everything in between), security has real-world consequences. Today, security is non-negotiable and like we explored above, it’s especially crucial that public sector entities have built-in security measures to keep data protected. So, what built-in features should you look for? Network isolation and access It’s critical that your data and underlying systems are fully isolated from other organizations using the same cloud provider. Database resources should be associated with a user group, which is contained in its own Virtual Private Cloud (VPC), and access should be granted by IP access lists, VPC peering, or private endpoints. Encyption in flight, at rest, and in use Encryption should be the standard. For example, when using MongoDB Atlas, all network traffic is encrypted using Transport Layer Security (TLS). Encryption for data at rest is automated using encrypted storage volumes. Customers can use field-level encryption to encrypt sensitive workloads which enables you to encrypt data in your application before you send it over the network to MongoDB clusters. Users can bring their own encryption keys for an additional level of control. Granular database auditing Granular database auditing allows administrators to answer detailed questions about systems activity by tracking all commands against the database. This ensures you always know who has access to what data and how they’re using it. Multi-factor authentication User credentials should always be stored using industry-standard and audited one-way hashing mechanisms, with multi-factor authentication options including SMS, voice call, a multi-factor app, or a multi-factor device, ensuring only approved users have access to your data. MongoDB Atlas for Government: Purpose-built for public sector As we’ve discussed, solutions that are purpose-built with built-in security are ideal for government agencies, and choosing the right one is the best way to keep sensitive data protected. MongoDB Atlas for Government on AWS GovCloud recently secured its FedRAMP Moderate authorization thanks to these security measures built into the solution. FedRAMP is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services. To ensure the utmost levels of security, Atlas for Government is an independent, dedicated environment for the U.S. public sector, as well as ISVs looking to build U.S. public sector offerings. Public Sector organizations carry a heavy burden when it comes to keeping data protected. However, with the right data platform underpinning modern applications – a platform with built-in security features – progress doesn’t mean you have to compromise on security. Want to learn more about data protection best practices for public sector organizations? Attend our upcoming webinar on April 12 for deeper insight .

March 28, 2023