FAIRTIQ secures high availability and flexible performance with MongoDB

THE CHALLENGE

The impact of early technology choices on scaling fast

Swiss-based tech product company FAIRTIQ makes public transport easier than ever by enabling users to travel without the need to pre-purchase traditional tickets. Users simply check in via the FAIRTIQ app at the start of their journey. Check-in triggers sensor data collection, the app detects when the trip ends, and a journey mapping algorithm reconstructs the route using timetables, graph algorithms, and neural networks. The pricing engine then calculates the cheapest fare before invoicing the user. Currently FAIRTIQ processes 320,000 trips daily and is available in eight European countries where it’s used actively by over 1.3 million people. Now expanding into new markets, the company’s goal is to simplify sustainable mobility internationally.

As a five-person startup in 2016, FAIRTIQ had a blank sheet when it came to its tech stack and a clear aim of 100% uptime. “As a startup, you want to be fast,” says Michel Yerly, Chief Technology Officer of FAIRTIQ. “We had only four months to bring our journey mapper from a prototype to a full product with pricing payment, customer care service, and integrate it with public transport systems.” FAIRTIQ opted for Heroku, since according to Yerly it was the simplest solution for deploying web applications to the cloud at the time. For its database, the company chose PostgreSQL, which came readily integrated with Heroku, allowing for rapid deployment—an essential advantage given the tight timeline from prototype to production.

But, PostgreSQL’s rigid schema, complex database migrations with downtime, and reliance on object-relational mapping soon proved cumbersome for FAIRTIQ’s needs. After 2 months, to maintain agility and flexibility, the company transitioned to MongoDB—its schema flexibility would allow the app to evolve without the constraints of traditional migrations. “The way I see it is that the schema sits inside the application as opposed to inside the database,” says Yerly. “A big advantage is if you release the schema upgrade, you release the microservice – you don’t need a database migration script, which will complicate your release pipeline. That’s important if you have continuous delivery, so it was the right thing for us.”

OUR SOLUTION

Adapting configuration for robust data query handling

FAIRTIQ leverages MongoDB to power its journey mapping system, microservices architecture, and analytics. Starting with three microservices – the journey mapper, the access control layer, and ‘all the rest’ - the company follows a "one microservice, one database" approach. “The microservice owns its own database and no other microservice will tap into it,” explains Yerly. This ensures clear data ownership and makes schema evolution more manageable.

FAIRTIQ initially deployed a standard MongoDB cluster with one primary node and two secondaries to ensure high availability. As the platform scaled, MongoDB grew to support over 70 microservices, with new databases added as needed for new services or to refine existing ones. As FAIRTIQ expanded, so did on-the-fly requests from non-engineering colleagues seeking insights into such data as active user numbers, trips in a given region, or invoices at a particular VAT rate. “As engineers, you probably don’t want to spend your time executing queries for others,” observes Yerly. To accommodate these requests without disrupting production, the team integrated an analytics node into the MongoDB cluster. “This did the job,” adds Yerly. “For a while.”

Then, the user service went down. “That was bad,” recalles Yerly. “Because the user service is used in the check-in process. So, no user service, no check-in, no trips - angry users.” FAIRTIQ traced the issue to an employee running aggregation queries for analytics on the wrong node, unintentionally overloading the system. This was caused by a missing specification in the connection string, which led queries to secondary nodes instead of the designated analytics node.

While secondary nodes are read-only, they still replicate primary data, creating write load. Meanwhile, the default behaviour of MongoDB Compass is to send a query on every keystroke to provide a preview of the results. This meant the employee’s aggregation query, which writes large intermediate results to disk on the secondary node, was effectively executed 300 times. Therefore hundreds of heavy analytics queries combined to overwhelm the secondaries, and exhaust IOPs.

Given FAIRTIQ’s aim of 100% uptime, a robust response was imperative to ensure the situation was both resolved quickly and would not occur again. Yerly and the team moved fast. The team upgraded the MongoDB cluster from M50 to M60, a process that replaced the older node and killed all queries. “So we were back with a fresh cluster,” says Yerly. The team then updated connection strings in documentation to ensure queries were directed to the correct nodes. It also implemented timeouts for all microservice queries to quickly identify human queries and resolve issues. To prevent accidental overloads of the production cluster, FAIRTIQ introduced a data federation layer, routing employee queries to appropriate nodes. “So, now it makes it impossible for people to connect to the wrong nodes,” says Yerly.

OUTCOME

Key learnings in ensuring a secure and flexible performance for a growing infrastructure

FAIRTIQ’s journey provided valuable learnings which it has used to strengthen its system’s reliability and efficiency. First and foremost, as Yerly puts it,” secondaries are not harmless.” By which he means that while often viewed as simple replicas of primary data, they can be elected as primaries at any time. More critically, if replication begins to lag, it can delay operations on the primary, which waits for confirmation.

FAIRTIQ’s other key take-away was that secondaries can also be mistaken as read-only nodes. While they don’t allow direct inserts or updates, secondary nodes still write to disk for replication and store intermediate aggregation queries. This behavior means that secondaries require careful monitoring to avoid unexpected performance bottlenecks. “We learned that the hard way,” says Yerly.

Correctly connecting to specific nodes proved to be tricky, too. FAIRTIQ saw that specifying a node in the connection string doesn’t necessarily guarantee a connection to that node, as the driver performs a cluster discovery before determining the best connection. To override this behavior, Yerly and the team used read preference tags, providing much greater flexibility over the read preference parameter. Meanwhile their addition of a data federation layer provides an ongoing safeguard against the risk of human error in the connection string.

Zooming out from the incident, FAIRTIQ was able to observe how its system evolved alongside its growing database needs and the importance of developing strategies to manage ad hoc queries effectively. By understanding the role and behavior of secondary nodes, Yerly and the team have ensured that the uptime and performance so crucial to FAIRTIQ’s operations and growth are maintained. This has been a must for new nationwide rollouts such as FAIRTIQ’s recent launch across the whole of Denmark.

Despite some challenges, the company remains steadfastly confident in its decision to move databases. “With MongoDB we have the flexibility we need,” says Yerly. “I have never regretted choosing MongoDB Atlas,” he added. “It's a very good choice for modern architecture - and the uptime so far has been amazing and it offers the flexibility we need.”