Leaf in the Wild: Leading Soccer Streaming Service fuboTV Scales its Business with MongoDB, Docker Containers and Kubernetes

Mat Keep
February 4, 2016 | Updated: January 11, 2024
#Business #DevOps

Leaf in the Wild posts highlight real world MongoDB deployments. Read other stories about how companies are using MongoDB for their mission-critical projects.

You would be hard pressed to find any 2016 IT industry predictions that fail to identify containers and non-relational databases as “hot market technologies,” used by the next generation of market leaders to out-innovate their legacy competitors. New York-based fuboTV is a prime example. To keep pace with business growth and an unrelenting software release schedule, fuboTV has migrated its MongoDB database to Docker containers managed by the Kubernetes orchestration system on the Google Cloud Platform.

I sat down with Dan Worth, Director of Engineering at fuboTV, and Brian McNamara, lead consultant at CloudyOps to learn more about the project.

Can you start by telling us a little bit about your company?

fuboTV aims to bring the best of live soccer to any device, at any time. We provide a streaming service to our subscribing customers here in North America that bundles live matches, entertainment channels, documentaries, news and more from soccer leagues around the world.

What role does MongoDB play in fuboTV?

MongoDB is the core database supporting our business. We use it to manage our customer data; it stores the program guide and metadata for our content catalog; and all of the API calls to our service that handle authentication, subscription management, match and channel schedules, and user device data.

^{Figure 1: Live streaming of soccer’s premier competitions}

Did you start out with MongoDB?

Our CTO has always been a big fan of MongoDB, and so we’ve used it from the very start of the company.

The speed of development and operational simplicity we get from MongoDB enables us to focus on building functionality for our rapidly expanding customer base, rather than be slowed down by the underlying database.

We initially used MongoDB on the Compose.io platform, but are now in the process of migrating to our own managed clusters on Google Container Engine (part of the Google Cloud Platform), which is built on the Kubernetes orchestration system and Docker containers.

There are several drivers for the migration to Google Container Engine:

We’ve moved the rest of our stack to Docker and Kubernetes, and wanted our MongoDB database to take advantage of all of the same benefits.
We are scaling the service out across geographic regions, and wanted the deployment flexibility that comes from Google’s cloud, and portability to other public cloud platforms in the future.
We wanted more control over the day-to-day operations and management of MongoDB to better support our rapidly evolving business requirements.

Why Docker and Kubernetes?

They bring unparalleled levels of flexibility, efficiency and uptime to our service. In previous companies, we had seen a lot of wasted resources. Using containers managed by Kubernetes, we can provision all of our environments – development, test, QA and production – to a single cluster of physical hosts. We take advantage of the Kubernetes scheduler to precisely control resource allocation across all of our apps, enabling us to maximize utilization and reduce costs.

MongoDB seamlessly integrates with Docker containers and Kubernetes to support our devops workflows.

We run a continuous integration and delivery pipeline, so we can develop, test, rollout, and if necessary, rollback with maximum speed and minimal effort.

We get zero downtime as we deploy and upgrade our applications. We can automatically reschedule containers if instances fail, with the Kubernetes replication controller ensuring that the requested number of instances are always running – enabling fault resilience for continuous availability. Data redundancy is provided by MongoDB replication within the replica set.

How is MongoDB deployed on Google Container Engine?

We provision MongoDB replica sets across Kubernetes pods in each of our Google cloud regions. Each instance is configured with 32 CPUs, 28 GB of RAM, and SSDs.

The replica sets are configured with four members for complete resilience: three are dedicated to handling operational traffic and one is configured as a hidden replica set member against which we snapshot volumes for backups.

^{Figure 2: fuboTV’s Kubernetes platform distributed across Google Container Engine regions}

How have you integrated MongoDB with Kubernetes?

Running MongoDB on Kubernetes introduces some additional considerations over many other applications:

MongoDB database nodes are stateful. In the event that a container fails and is rescheduled, it's undesirable for the data to be lost (it could be recovered from other nodes in the replica set but that would reduce the recovery speed of the fully-resilient replica set). To solve this, we make use of the Kubernetes volume abstraction to map what would otherwise be an ephemeral MongoDB data directory in the container to persistent network attached storage where the data survives container failure and rescheduling.
MongoDB database nodes within a replica set must be able to communicate with each other at all times. All of the nodes within a replica set must know the addresses of all of their peers, but when a container is rescheduled, it is likely to be restarted in a Kubernetes Pod which has a different IP Address, and so the MongoDB node would not be able to re-join the replica set. We handle this by associating a Kubernetes Service with each MongoDB node – where the Kubernetes DNS service is used to provide a hostname for the service that remains constant through rescheduling.
Once each of the individual MongoDB nodes is running (each within its own container), the replica set must be initialized and each node added. We have implemented a continuous delivery workflow using CircleCi that integrates with Kubernetes and Docker to retrieve the required configuration, and then execute the MongoDB initialization steps.

How do you scale the fuboTV Service?

The traffic profile of our streaming service is extremely “bursty”. The site handles 100x normal traffic volumes 10 minutes before the start of a big match. To handle the load, we distribute database operations across all of the replica set members. Our users are spread across North America, so to deliver a low latency experience for everyone, we use the nearest read preference to route subscribers to the closest geographic replica. MongoDB’s data center awareness is critical to ensuring the quality of our service.

Which release of MongoDB are you using?

We were very impressed with MongoDB 3.0 and the WiredTiger storage engine. Document level concurrency control and compression delivers the levels of performance and storage efficiency we need as our service has grown. For example, we achieved 20x storage reduction for one specific collection.

We upgraded to MongoDB 3.2 as soon as it was declared production ready in December last year to ensure we stay current with the latest innovations.

How are you measuring the impact of MongoDB and Kubernetes on your business?

The main benefits are development agility and speed of deployment. We can launch new services, collect feedback from users, fix problems, and iterate on features more quickly.

What advice would you give someone who is considering using MongoDB, Docker and Kubernetes for their next project?

MongoDB has a lot of nice features for resilience. To take advantage of these, make sure you fully characterize application behavior under different failure scenarios.

Docker is great to work with, but make sure your development workflows, tools and harnesses are adapted to build and run apps with it.

Kubernetes is still relatively young, but maturing quickly. You need to carefully evaluate whether you want to roll your own platform with it, or instead rely on a hosted service, as we have done with the Google Container Engine.

Brian and Dan, thanks for taking the time to share your experience with the community.

To learn more about containers and orchestration, download our new white paper. Enabling Microservices: Containers & Orchestration Explained

About the Author - Mat Keep

Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

Securing MongoDB Part 1: Data Security Requirements for Regulatory Compliance

Series Introduction The frequency and severity of data breaches continues to escalate year on year: Research from PWC identified over 117,000 attacks against information systems every day in 2014. That’s an increase of 48% over the previous year. Companies reporting losses from security breaches totaling $20m or more doubled over the same period. Updated industry regulations will increase penalties on those organizations proven to have not taken sufficient measures to protect their customers’ data against unauthorized disclosure. Its no surprise that with databases storing an organization’s most important information assets, securing them is top of mind for the business. In this 4-part blog series, we will present the best practices and controls available in MongoDB to help you create a secure, compliant database platform: In this first installment, we’ll take a look at the general requirements for data security and regulatory compliance In the 2nd part , we’ll cover enforcing access control to MongoDB In part 3 , we’ll discuss auditing and encryption In the final installment , we’ll highlight environmental control and management If you want to get a head-start and learn about all of these topics now, just go ahead and download the MongoDB Security Architecture guide . Requirements for Database Security & Compliance In light of increasing threats over the past decade coupled with heightened concern for individual privacy, industries and governments around the world have embarked on a series of initiatives designed to increase security, reduce fraud and protect personally identifiable information (PII), including: PCI DSS for managing cardholder information HIPAA standards for managing healthcare information NIST 80-53 catalogs security controls for all U.S. federal information systems except those related to national security STIG for secure installation and maintenance of computer systems, defined for the US Department of Defense European Union Data Protection Directive Asia Pacific Economic Cooperation (APEC) data protection standardization In addition to these initiatives, new regulations are being developed every year to cope with emerging threats and new demands for tighter controls governing data use. Each set of regulations defines security and auditing requirements that are unique to a specific industry or application. Compliance is assessed on a per-project basis, evaluating people and processes alongside technology. Despite the differences between each compliance standard, there are common foundational requirements across all of the directives, including: Restricting access to data, enforced via predefined privileges and security levels Measures to protect against the accidental or malicious disclosure, loss, destruction or damage of sensitive data The separation of duties when running applications and accessing data Recording the activities of users, administrative staff, and applications in accessing and processing data These requirements inform the security architecture of MongoDB, with best practices for the implementation of a secure, compliant data management environment. A holistic security architecture must cover: User access management to restrict access to sensitive data, implemented through authentication and authorization controls Logging operations against the database in an audit trail for forensic analysis Data protection via encryption of data in-motion over the network and at-rest in persistent storage Environmental and process controls The requirements for each of these elements are discussed below. **Figure 1**: MongoDB End to End Security Architecture User Access Management - Authentication Authentication is designed to confirm the identity of entities accessing the database. In this context, entities are defined as: Users who need access to the database as part of their day-to-day business function Administrators (i.e. sysadmins, DBAs, QA staff) and developers Software systems including application servers, reporting tools, and management and backup suites Physical and logical nodes that the database runs on. Databases can be distributed across multiple nodes both for scaling operations and to ensure continuous operation in the event of systems failure or maintenance. Best practices for authentication are as follows. Create Security Credentials. Create login credentials for each entity that will need access to the database, and avoid creating a single “admin” login that every user shares. By creating credentials it becomes easier to define, manage, and track system access for each user. Should a user’s credentials become compromised, this approach makes it easier to revoke the user without disrupting other users who need access to the database. Developers, Administrators, and DBAs should all have unique credentials to access the database. When logins are shared it can be impossible to identify who attempted different operations, and it eliminates the ability to assign fine-grained permissions. With unique logins, staff that move off of the project or leave the organization can have their access revoked without affecting other user accounts. Authentication should be enforced between nodes. This prevents unauthorized instances from joining a database cluster, preventing the illicit copying or movement of data to insecure nodes. Supporting In-Database and Centralized User Access Management. Databases should provide the ability to manage user authentication either within the database itself, typically via a Challenge/Response mechanism, or through integration with organization-wide identity management systems. Integrating MongoDB within the existing information security infrastructure enforces centralized and standardized control over user access. If, for example, a user’s access must be revoked, the update can be made in a single repository and enforced instantly across all systems, including MongoDB. User Rights Management - Authorization Once an entity has been authenticated, authorization governs what that entity is entitled to do in the database. Privileges are assigned to user roles that define a specific set of actions that can be performed against the database. Best practices include: Grant Minimal Access to Entities. Entities should be provided with the minimal database access they need to perform their function. If an application requires access to a logical database, it should be restricted to operations on that database alone, and prevented from accessing other logical databases. This helps protect against both malicious and accidental access or unauthorized modification of data. Group Common Access Privileges into Roles. Entities can often be grouped into “roles” such as “DBA”, “Sysadmin”, and “App server.” Permissions for a role can be centrally managed and users can be added or removed from roles as needed. Using roles helps simplify management of access control by defining a single set of rules that apply to specific classes of entities, rather than having to define them individually for each user. Control Which Actions an Entity Can Perform. When granting access to a database, consideration should be made for which specific actions or commands each entity should have permission to run. For example, an application may need read/write permissions to the database, whereas a reporting tool may be restricted to read-only permission. Some users may be granted privileges that enable them to insert new data to the database, but not to update or delete existing data. Care should be taken to ensure that only the minimal set of privileges is provided. Credentials of the most privileged accounts could compromise the entire database if they are hacked internally or by an external intruder. Control Access to Sensitive Data. To prevent the emergence of data silos, it should be possible to restrict permissions to individual fields, based on security privileges. For example, some fields of a record may be accessible to all users of the database, while others containing sensitive information, such as PII, should be restricted to users with specific security clearance. Auditing By creating audit trails, changes to database configuration and data can be captured for each entity accessing the database, providing a log for compliance and forensic analysis. Auditing can also detect attempts to access unauthorized data. Encryption Encryption is the encoding of critical data whenever it is in transit or at rest, enabling only authorized entities to read it. Data will be protected in the event that eavesdroppers or hackers gain access to the server, network or database. Encrypt Connections to the Database. All user or application access to the database should be via encrypted channels including connections established through the drivers, command line or shell, as well as remote access sessions to the database servers themselves. Internal communications between database nodes should also be encrypted, i.e. traffic replicated between nodes of a database cluster. Encrypt Data at Rest. One of most common threats to security comes from attacks that bypass the database itself and target the underlying Operating System and physical storage of production servers or backup devices, in order to access raw data. On-disk encryption of the database’s data files mitigates this threat. Sign and Rotate Encryption Keys. Encryption keys for network and disk encryption should be periodically rotated. SSL/TLS encryption channels should use signed certificates to ensure that clients can certify the credentials they receive from server components. Enforce Strong Encryption. The database should support FIPS (Federal Information Processing Standard) 140-2 to ensure the implementation of secure encryption algorithms. Environmental and Process Control The environment in which the database and underlying infrastructure is running should be protected with both physical and logical controls. These are enforced in the underlying deployment environment, rather than in the database itself, and include: Installation of firewalls Network configurations Defining file system permissions Creation of physical access controls to the IT environment As manual configuration errors are one of the largest causes of attackers bypassing security mechanisms, there are a series of operational processes that should be adopted to further promote and enforce secure operation, including: DBA and developer training Database provisioning, monitoring and backup Database maintenance, i.e. applying the latest patches Getting Started with MongoDB Security With comprehensive controls for user rights management, auditing and encryption, coupled with management controls, MongoDB can meet the best practice and requirements discussed earlier. MongoDB Enterprise Advanced is the certified and supported production release of MongoDB, with advanced security features, including Kerberos and LDAP authentication, encryption of data at-rest, FIPS-compliance, and maintenance of audit logs. These capabilities extend MongoDB’s security framework, which includes Role-Based Access Control, PKI certificates, Field-Level Redaction, and SSL/TLS data transport encryption. You can learn about all of these capabilities now by downloading the MongoDB Security Architecture guide. If you want to try them for yourself, download MongoDB Enterprise, free of charge for evaluation and development. In the second part of this blog post series , we will dive into MongoDB access control. Learn more about MongoDB security features, read our guide. MongoDB security architecture About the Author - Mat Keep Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

February 3, 2016

Next →

Top Use Cases for Text, Vector, and Hybrid Search

Search is how we discover new things. Whether you’re looking for a pair of new shoes, the latest medical advice, or insights into corporate data, search provides the means to unlock the truth. Search habits—and the accompanying end-user expectations—have evolved along with changes to the search experiences offered by consumer apps like Google and Amazon. The days of the standard of 10 blue links may well be behind us, as new paradigms like vector search and generative AI (gen AI) have upended long-held search norms. But are all forms of search created equal, or should we be seeking out the right “flavor” of search for specific jobs? In this blog post, we will define and dig into various flavors of search, including text, vector and AI-powered search, and hybrid search, and discuss when to use each, including sample use cases where one type of search might be superior to others. Information retrieval revolutionized with text search The concept of text search has been baked into user behavior from the early days of the web, with the rudimentary text box entry and 10 blue link results based on text relevance to the initial query. This behavior and associated business model has produced trillions in revenue and has become one of the fiercest battlegrounds across the internet . Text search allows users to quickly find specific information within a large set of data by entering keywords or phrases. When a query is entered, the text search engine scans through indexed documents to locate and retrieve the most relevant results based on the keywords. Text search is a good solution for queries requiring exact matches where the overarching meaning isn't as critical. Some of the most common uses include: Catalog and content search: Using the search bar to find specific products or content based on keywords from customer inquiries. For example, a customer searching for "size 10 men trainers" or “installation guide” can instantly find the exact items they’re looking for, like how Nextar tapped into Atlas Search to enable physical retailers to create online catalogs Covid-19 pandemic. In-application search: This is well-suited for organizations with straightforward offerings to make it easier for users to locate key resources, but that don’t require advanced features like semantic retrieval or contextual re-ranking. For instance, if a user searches for "songs key of G," they can quickly receive relevant materials. This streamlines asset retrieval, allowing users to focus on the task they are trying to achieve and boosts overall satisfaction. For a company like Yousician , Atlas Search enabled their 20 million monthly active users to tackle their music lessons with ease. Customer 360: Unifying data from different sources to create a single, holistic view. Consolidated information such as user preferences, purchase history, and interaction data can be used to enhance business visibility and simplify the management, retrieval, and aggregation of user data. Consider a support agent searching for all information related to customer “John Doe." They can quickly access relevant attributes and interaction history, ensuring more accurate and efficient service. Helvetia was able to achieve success after migrating to MongoDB and using Atlas Search to deliver a single, 360-degree real-time view across all customer touchpoints and insurance products. AI and a new paradigm with vector search With advances in technology, vector search has emerged to help solve the challenge of providing relevant results even when the user may not know what they’re looking for. Vector search allows you to take any type of media or content, convert it into a vector using machine learning algorithms, and then search to find results similar to the target term. The similarity aspect is determined by converting your data into numerical high-dimensional vectors, and then calculating the distance between them to determine relevance—the closer the vector, the higher the relevance. There is a wide range of practical, powerful use cases powered by vector search—notably semantic search and retrieval-augmented generation (RAG) for gen AI. Semantic search focuses on meaning and prioritizes user intent by deciphering not just what users type but why they're searching, in order to provide more accurate and context-oriented search results. Some examples of semantic search include: Content/knowledge base search: Vast amounts of organizational data, structured and unstructured, with hidden insights, can benefit significantly from semantic search. Questions like “What’s our remote work policy?” can return accurate results even when the source materials do not contain the “remote” keyword, but rather have “return to office” or “hybrid” or other keywords. A real-world example of content search is the National Film and Sound Archive of Australia , which uses Atlas Vector Search to power semantic search across petabytes of text, audio, and visual content in its collections. Recommendation engines: Understanding users’ interests and intent is a strong competitive advantage–like how Netflix provides a personalized selection of shows and movies based on your watch history, or how Amazon recommends products based on your purchase history. This is particularly powerful in e-commerce, media & entertainment, financial services, and product/service-oriented industries where the customer experience tightly influences the bottom line. A success story is Delivery Hero , which leverages vector search-powered real-time recommendations to increase customer satisfaction and revenue. Anomaly detection: Identifying and preventing fraud, security breaches, and other system anomalies is paramount for all organizations. By grouping similar vectors and using vector search to identify outliers, potential threats can be detected early, enabling timely responses. Companies like VISO TRUST and Extrac are among the innovators that build their core offerings using semantic search for security and risk management. With the rise of large language models (LLMs), vector search is increasingly becoming essential in gen AI application development. It augments LLMs by providing domain-specific context outside of what the LLMs “know,” ensuring relevance and accuracy of the gen AI output. In this case, the semantic search outputs are used to enhance RAG. By providing relevant information from a vector database, vector search helps the RAG model generate responses that are more contextually relevant. By grounding the generated text in factual information, vector search helps reduce hallucinations and improve the accuracy of the response. Some common RAG applications are for chatbots and virtual assistants, which provide users with relevant responses and carry out tasks based on the user query, delivering enhanced user experience. Two real-world examples of such chatbot implementations are from our customers Okta and Kovai . Another popular application is using RAG to help generate content like articles, blog posts, scripts, code, and more, based on user prompts or data. This significantly accelerates content production, allowing organizations including Novo Nordisk and Scalestack to save time and produce content at scale, all at an accuracy level that was not possible without RAG. Beyond RAG, an emerging vector search usage is in agentic systems . Such a system is an architecture encompassing one or more AI agents with autonomous decision-making capabilities, able to access and use various system components and resources to achieve defined objectives while adapting to environmental feedback. Vector search enables efficient and semantically meaningful information retrieval in these systems, facilitating relevant context for LLMs, optimized tool selection, semantic understanding, and improved relevance ranking. Hybrid search: The best of both worlds Hybrid search combines the strengths of text search with the advanced capabilities of vector search to deliver more accurate and relevant search results. Hybrid search shines in scenarios where there's a need for both precision (where text search excels) and recall (where vector search excels), and where user queries can vary from simple to complex, including both keyword and natural language queries. Hybrid search delivers a more comprehensive, flexible information retrieval process, helping RAG models access a wider range of relevant information. For example, in a customer support context, hybrid search can ensure that the RAG model retrieves not only documents containing exact keywords but also semantically similar content, resulting in more informative and helpful responses. Hybrid search can also help reduce information overload by prioritizing the most relevant results. This allows RAG models to focus on processing and understanding the most critical information, leading to faster, more accurate responses, and improving the user experience. Powering your AI and search applications with MongoDB As your organization continues to innovate in the rapidly evolving technology ecosystem, building robust AI and search applications supporting customer, employee, and stakeholder experiences can deliver powerful competitive advantages. With MongoDB, you can efficiently deploy full-text search , vector search , and hybrid search capabilities. Start building today—simplify your developer experience while increasing impact in MongoDB’s fully-managed, secure vector database, integrated with a vast AI partner ecosystem , including all major cloud providers, generative AI model providers, and system integrators.

September 16, 2024