Lessons Learnt Building mongoengine

MongoDB
November 29, 2012
#Releases

Recently, I attended both Pycon UK and Pycon Ireland to talk about the lessons I have learnt while maintaining mongoengine. The conferences were both excellent and surprisingly different. Pycon UK had quite an “unconference” feel, with some exciting sprint rooms - I wish I had more time as by all reports the educational jam was inspirational. Pycon Ireland in contrast felt more slick with booths from DemonWare, Amazon and Facebook. If you can, I’d advise going to both conferences as they really complement each other.

Going to python conferences its great to see ever increasing numbers of developers adopting MongoDB, its such a good fit for python! Also, I can’t tell you how great it is to have people come up and thank you and 10gen for creating such a great product. As a 10gen engineer one of our core goals is to build upon the power and ease of use of mongoDB.

Back to the matter at hand, here’s a quick overview of the talk. Please check out the slides for all the detail I glazed over here and for the extra keen you can even watch the video as well!

Lessons learnt building mongoengine

  1. Dive in!
    If a project is on Github its a license to dive in. Don’t be afraid to, but do be a good citizen and feedback changes to the root repository so others can benefit.

  2. Metaclasses
    Turns out they aren’t magical! In fact how we use them in mongoengine allows us to change the class and add metadata at declaration time. This really useful trick makes creating instances of classes quick and easy.

  3. Straying from the path
    Mongoengine uses a Django style query syntax. This is bad in that its different from the native pymongo syntax. But its also good as you we add some sweet syntactic sugar.

  4. Not all ideas are good
    In mongoDB the native save operation overwrites the whole document. This catches new users out. So in mongonengine save tracks changes to the document and uses atomic updates. The jury is out whether or not this is a good thing™.

  5. Managing a community
    I’ve never done it before and its not always easy but it has been worth it. My top tip is to check out the github network graph and ask committers to forks to add pull requests back upstream. In cases where people aren’t willing to do so add a ticket with a link to the feature you’ve spotted downstream. An added benefit of bubbling up these useful features is that it will bring more contributors into the core project.

  6. Don’t be afraid to ask for help
    Cue me asking for help on any aspect of mongoengine, be it the core, the website or framework support! Please get in contact

Slides

The video is available on the PyCon Ireland channel.

← Previous

Introducing MongoClient

Today we are releasing updated versions of most of the officially supported MongoDB drivers with new error checking and reporting defaults. See below for more information on these changes, and check your driver docs for specifics. Over the past several years, it’s become evident that MongoDB’s previous default behavior (where write messages did not wait for a return code from the server by default) wasn’t intuitive and has caused confusion for MongoDB users. We want to rectify that with minimal disruption to the MongoDB apps already in production. History First, I  thought it would be interesting to share the history behind the previous default behavior, and why and how we are changing it. The old behavior goes back to the very beginning of 10gen , before Dwight and I imagined MongoDB as a stand-alone database. When we first started 10gen in the fall of 2007, we set out to build a full platform as a service stack with MongoDB as the data layer. This was a fully hosted system (still open source), that encompassed a load balancer, auto scaling application server and data tier. The application side was a full server side JavaScript environment. Every request into 10gen was an http request.  So you can imagine a controller for doing some user analytics could act like this: URL:  http://foo.10gen.com/show?sect=sports&page=blog1 CODE: db.sect_views.update( { _id : db.getParameter( “sect” ) } , { $inc : { count : 1 } } , /*upsert*/true ); db.page_views.update( { _id : db.getParameter( “page” ) } , { $inc : { count : 1 } } , true ); Writes in that system did not individually wait for a response from the database.  However, the application server itself always checked the database for any errors that occurred during the entire page load (using getLastError and getPrevError) so that the user/system would be notified of any issues.  Developers could of course also call getLastError whenever they wanted.  This worked great in the platform, as we were able to control the whole access pattern. In January of 2009, we decided for a variety of reasons to only focus on the data tier (MongoDB).  At that time, a number of people had been using MongoDB in production for almost a year as part of the full stack, and a lot more were very interested in using it standalone. Over the course of the next few months, we wrote the initial implementations of the Java, Python, Ruby and PHP drivers.  All of those drivers used the same network protocol as the original application server, which has non-synchronous write operations.  It seemed natural to us at the time given the background, but it is clear that this is not intuitive for new users of MongoDB who had never used the full stack. New Behavior Going forward, the default clearly has to be to wait for the database to acknowledge all writes; that is much more intuitive. Just flipping the default however, would be backward breaking for apps in production. The change we’re going to make is to add a new top level connection class in each driver.  For example, in Java, previously you would do: Mongo mongo = new Mongo( “mongoserver” ); That class, Mongo, will maintain the old default, and become deprecated. For new code, you will do: MongoClient mongo = new MongoClient( “mongoserver” ); which will default to a WriteConcern of 1.  Everything else will be the same. The old class will remain for quite a while (but not forever), so that we won’t break old code right now.  As another benefit, every single driver will use MongoClient so for the first time at least the top level class will have the same name across the board.  All documentation, tutorials, and sample code have been changed accordingly.  More Information For driver downloads, docs, and tutorials, please visit the drivers page on the MongoDB wiki. For bug reports and feature requests please visit jira.mongodb.org . For any other feedback, please leave a comment on this post or in our user forum . We appreciate your continued support and feedback on MongoDB. - Eliot and the MongoDB Team

November 27, 2012
Next →

MongoDB助力腾讯游戏 优化游戏开发体验

客户简介 腾讯游戏提倡超级数字场景 连接数亿游戏玩家 作为“超级数字场景”理念的倡导者和实践者，腾讯游戏致力于为用户创造高品质数字生活体验，为产业和社会发展创造更多建设性的价值。 腾讯游戏为全球知名的游戏开发与服务运营商，在全球连接超过8亿的用户。在开放发展的模式下，腾讯游戏采取自主研发和多元化的外部合作相结合的方式，在网络游戏众多细分市场领域形成专业化布局，打造覆盖全品类的产品阵营，为全球网络游戏玩家提供休闲游戏平台、大型网游、中型休闲游戏、桌面游戏、对战平台五大类。另外，腾讯游戏与全球顶级游戏开发公司建立深度合作，将国外优质的前沿产品体验带到中国，也将中国的游戏带向世界。 2021年3月，腾讯游戏针对国际业务推出了在线游戏开发平台Level Infinite PGOS（Level Infinite Game Online Service）。Level Infinite PGOS是一种游戏在线服务解决方案，旨在降低游戏后端开发和维护的难度，同时降低成本，从而使开发者专注于游戏玩法与核心逻辑开发。 业务挑战 产品力主导游戏行业竞争 游戏出海面临多重挑战 全球游戏市场规模不断扩大，游戏产业已成为一种重要的文化产业。据市场调查机构 Newzoo 最新数据显示，2023年全球游戏市场规模预计将超过1877亿美元，同比增长2.6%。 游戏市场潜力巨大、前景看好，与此同时游戏开发竞争也变得异常激烈。游戏品类越来越多、玩法越来越多，给游戏开发带来更高要求：游戏设计和架构越来越复杂，游戏开发成本水涨船高，游戏复用性较低，新项目启动门槛更高等等。 尤其对于腾讯游戏海外业务来说，研发更需要具备全球发行、全球部署的能力，直接拉升了对于底层数据架构的要求，当中包括：需要提供多租户SaaS模式；能够物理隔离每个游戏大区，满足全球各个区域的隐私保护；以及可在全球各个地区进行分布式部署、自动扩容、缩容等。 解决方案 深度契合游戏业务场景 为了应对游戏行业的最新趋势和海外市场的挑战，腾讯游戏推出为海外游戏而设的Level Infinite PGOS通用平台。Level Infinite PGOS是一套多租户SaaS游戏后台解决方案，采用全球化分布式架构，在欧洲、北美、日韩、东南亚等游戏发行热点区域部署运行。 数据库是游戏软件的核心组件，游戏玩家的各种信息、运营数据、游戏场景数据等都需要借助数据库来保存。对于数据存储系统，Level Infinite PGOS根据自身场景进行了严格的测试选型，最终采用MongoDB作为核心存储组件，成为一整套覆盖游戏各个维度、各个生命周期的解决方案。 玩家数据存储 – 与传统游戏开发有所不同，使用MongoDB去存储玩家的基础数据，不会将MongoDB直接暴露给游戏去使用，开发者无需关心底层的数据细节，即可直接灵活定义数据，例如，不同游戏可定义不同数据模板。此外，MongoDB 支持多种数据类型和数据原子运算，易于实现幂等操作；而且基于MongoDB的分片可横向扩容，对于一些爆款游戏来说，这一点很重要，可以不用担心玩家规模的快速增加。 智能对局匹配 – 在很多游戏中，都需要在平台上匹配两名玩家去进行对局竞技。以腾讯游戏为例，如果是一款全球发行的游戏，就有可能在不同游戏大区匹配到两名玩家，这种匹配看似随意，但却需要后台具备强有力的数据处理能力。技术调度要同时满足不同区域的服务器集群，也就是满足不同场景需求下的服务器扩容。腾讯游戏底层通过MongoDB实现原子化操作，经过玩家各种属性的对比，找到一个距离各个玩家最近的服务器，并进行服务器分配，最终形成一个对局。 游戏内经济系统闭环 – 假设把游戏内的经济系统理解为一个特殊的交易场景，在处理游戏交易的过程中，涉及到订单、退款、跟踪回溯等多个环节，而通过MongoDB的原子化、事务性操作可以将整个交易流程一次性完成。虽然游戏内交易是虚拟的、复杂的，但采用MongoDB可以保证交易是规范的、完整的。 数据流能力 – 游戏开发者需要跟踪玩家各种行为事件，以便形成流水日志，同时要保证玩家的所有事件是可追溯、可查询的。MongoDB的数据库实例，可将这些流水日志存储起来，并基于灵活的文档结构，让开发者不论是在开发期间、还是游戏已经发行的期间，都可以快速检索玩家的所有事件流。 客户价值 优化开发体验 拉升运维能力 根据数据显示，2023年一季度，腾讯的游戏领域收入达到483亿元，而其中132亿元来自国际市场，占游戏整体收入的27%，可见腾讯在海外市场的巨大潜力和影响力。对腾讯游戏而言，全球化协作体系已然成型。在多元化布局和全球化视野之下，中国游戏既要在内容创新、玩法创新上学习更多，也要将支持大规模玩家在线的后台技术越做越强。 回顾与MongoDB的合作历程，腾讯海外游戏Level Infinite PGOS平台负责人谢磊谈到，无论在功能还是性能上，MongoDB都很好地契合了游戏业务场景，带给腾讯游戏的不只是功能价值，还有运维价值： 简单、易用的控制台 实现全面托管服务，即时自动扩缩容的专用服务器，为实时游戏提供低延迟和高可靠性。 丰富的可视化监控 提供实时可视化日志、监控面板，研发人员、管理人员可以实时监控业务运行状态。 一键升配、降配能力 在访问量突增时，一键自动扩容保障业务的正常运行；在流量低谷，一键自动缩容以节约成本。 多维告警能力 提供运行时间、状态异常等多维度告警能力，使问题可以在最短的时间内被捕捉并通知到用户。 客户证言 腾讯海外游戏Level Infinite PGOS平台负责人 谢磊： “游戏行业的发展越来越由产品力主导。Level Infinite PGOS平台最大的使命是要能够将更新、更现代化的开发模式与腾讯已有能力结合在一起，形成开发体验良好的平台，其中尤为重要的是需要符合海外开发人员开发习惯。正是在这样的背景之下，我们的底层技术选择使用MongoDB。可以说，MongoDB让腾讯游戏Level Infinite PGOS如虎添翼。”

November 29, 2023