Mark Smith

>This is the first part of a short series of blog posts called "Rewrite it in Rust (RiiR)." It's a tongue-in-cheek title for some posts that will investigate the similarities and differences between the same service written in Python with Flask, and Rust with Actix-Web.

This post will show how I built a RESTful API for a collection of cocktail recipes I just happen to have lying around. The aim is to show an API server with some complexity, so although it's a small example, it will cover important factors such as:

-   Data transformation between the database and a JSON representation.
-   Data validation.
-   Pagination.
-   Error-handling.

## Prerequisites

-   Python 3.8 or above
-   A MongoDB Atlas cluster. Follow the "[Get Started with Atlas](https://docs.atlas.mongodb.com/getting-started/)" guide to create your account and MongoDB cluster. Keep a note of your database username, password, and [connection string](https://docs.atlas.mongodb.com/tutorial/connect-to-your-cluster/#connect-to-your-atlas-cluster) as you will need those later.

This is an *advanced* guide, so it'll cover a whole bunch of different libraries which can be brought together to build a declarative Restful API server on top of MongoDB. I won't cover repeating patterns in the codebase, so if you want to build the whole thing, I recommend checking out the source code, which is all [on GitHub](https://github.com/mongodb-developer/rewrite-it-in-rust).

It won't cover the basics of Python, Flask, or MongoDB, so if that's what you're looking for, I recommend checking out the following resources before tackling this post:

-   [Think Python](https://greenteapress.com/wp/think-python-2e/)
-   [The Python & MongoDB Quickstart Series](https://developer.mongodb.com/quickstart/python-quickstart-crud/)
-   [Flask Tutorial](https://flask.palletsprojects.com/en/1.1.x/tutorial/)
-   [Pydantic Documentation](https://pydantic-docs.helpmanual.io/)

## Getting Started

Begin by cloning the sample code source [from GitHub](https://github.com/mongodb-developer/rewrite-it-in-rust). There are four top-level directories:

-   [actix-cocktail-api](https://github.com/mongodb-developer/rewrite-it-in-rust/tree/master/actix-cocktail-api): You can ignore this for now.
-   [data](https://github.com/mongodb-developer/rewrite-it-in-rust/tree/master/data): This contains an export of my cocktail data. You'll import this into your cluster in a moment.
-   [flask-cocktail-api](https://github.com/mongodb-developer/rewrite-it-in-rust/tree/master/flask-cocktail-api): The code for this blog post.
-   [test_scripts](https://github.com/mongodb-developer/rewrite-it-in-rust/tree/master/test_scripts): A few shell scripts that use curl to test the HTTP interface of the API server.

There are more details in the [GitHub repo](https://github.com/mongodb-developer/rewrite-it-in-rust/tree/master/flask-cocktail-api), but the basics are: Install the project with your virtualenv active:

``` shell
pip install -e .
```

Next, you should import the data into your cluster. Set the environment variable `$MONGO_URI` to your cluster URI. This environment variable will be used in a moment to import your data, and also by the Flask app. I use `direnv` to configure this, and put the following line in my `.envrc` file in my project's directory:

``` shell
export MONGO_URI="mongodb+srv://USERNAME:PASSW0RD@cluster0-abcde.azure.mongodb.net/cocktails?retryWrites=true&w=majority"
```

Note that your database must be called "cocktails," and the import will create a collection called "recipes." After checking that `$MONGO_URI` is set correctly, run the following command:

``` shell
mongoimport --uri "$MONGO_URI" --file ./recipes.json
```

Now you should be able to run the Flask app from the
`flask-cocktail-api` directory:

``` shell
FLASK_DEBUG=true FLASK_APP=cocktailapi flask run
```

(You can run `make run` if you prefer.)

Check the output to ensure it is happy with the configuration, and then in a different terminal window, run the `list_cocktails.sh` script in the `test_scripts` directory. It should print something like this:

``` json
{
    "_links": {
        "last": {
            "href": "http://localhost:5000/cocktails/?page=5"
        }, 
        "next": {
            "href": "http://localhost:5000/cocktails/?page=5"
        }, 
        "prev": {
            "href": "http://localhost:5000/cocktails/?page=3"
        }, 
        "self": {
            "href": "http://localhost:5000/cocktails/?page=4"
        }
    }, 
    "recipes": [
        {
            "_id": "5f7daa198ec9dfb536781b0d", 
            "date_added": null, 
            "date_updated": null, 
            "ingredients": [
            {
                "name": "Light rum", 
                "quantity": {
                "unit": "oz", 
                }
            }, 
            {
                "name": "Grapefruit juice", 
                "quantity": {
                "unit": "oz", 
                }
            }, 
            {
                "name": "Bitters", 
                "quantity": {
                "unit": "dash", 
                }
            }
            ], 
            "instructions": [
            "Pour all of the ingredients into an old-fashioned glass almost filled with ice cubes", 
            "Stir well."
            ], 
            "name": "Monkey Wrench", 
            "slug": "monkey-wrench"
        },
    ]
    ...
```

## Breaking it All Down

The code is divided into three submodules.

-   `__init__.py` contains all the Flask setup code, and defines all the HTTP routes.
-   `model.py` contains all the Pydantic model definitions.
-   `objectid.py` contains a Pydantic field definition that I stole from the [Beanie](https://github.com/roman-right/beanie) object-data mapper for MongoDB.

I mentioned earlier that this code makes use of several libraries:

-   [PyMongo](https://pymongo.readthedocs.io/en/stable/index.html) and [Flask-PyMongo](https://flask-pymongo.readthedocs.io/en/latest/) handle the connection to the database. Flask-PyMongo specifically wraps the database collection object to provide a convenient`find_one_or_404` method.
-   [Pydantic](https://pydantic-docs.helpmanual.io/usage/exporting_models/) manages data validation, and some aspects of data transformation between the database and a JSON representations.
-   along with a single function from [FastAPI](https://fastapi.tiangolo.com/).

## Data Validation and Transformation

When building a robust API, it's important to validate all the data passing into the system. It would be possible to do this using a stack of `if/else` statements, but it's much more effective to define a schema declaratively, and to allow that to programmatically validate the data being input.

I used a technique that I learned from [Beanie](https://github.com/roman-right/beanie), a new and neat ODM that I unfortunately couldn't practically use on this project, because Beanie is async, and Flask is a blocking framework.

Beanie uses [Pydantic](https://pydantic-docs.helpmanual.io/usage/exporting_models/) to define a schema, and adds a custom Field type for ObjectId.

``` python
# model.py

class Cocktail(BaseModel):
    id: Optional[PydanticObjectId] = Field(None, alias="_id")
    slug: str
    name: str
    ingredients: List[Ingredient]
    instructions: List[str]
    date_added: Optional[datetime]
    date_updated: Optional[datetime]

    def to_json(self):
        return jsonable_encoder(self, exclude_none=True)

    def to_bson(self):
        data = self.dict(by_alias=True, exclude_none=True)
        if data["_id"] is None:
            data.pop("_id")
        return data
```

This `Cocktail` schema defines the structure of a `Cocktail` instance, which will be validated by Pydantic when instances are created. It includes another embedded schema for `Ingredient`, which is defined in a similar way.

I added convenience functions to export the data in the `Cocktail` instance to either a JSON-compatible `dict` or a BSON-compatible `dict`. The differences are subtle, but BSON supports native `ObjectId` and `datetime` types, for example, whereas when encoding as JSON, it's necessary to encode ObjectId instances in some other way (I prefer a string containing the hex value of the id), and datetime objects are encoded as ISO8601 strings.

The `to_json` method makes use of a function imported from FastAPI, which recurses through the instance data, encoding all values in a JSON-compatible form. It already handles `datetime` instances correctly, but to get it to handle ObjectId values, I extracted some [custom field](https://github.com/mongodb-developer/rewrite-it-in-rust/blob/master/flask-cocktail-api/src/cocktailapi/objectid.py) code from Beanie, which can be found in `objectid.py`.

The `to_bson` method doesn't need to pass the `dict` data through `jsonable_encoder`. All the types used in the schema can be directly saved with PyMongo. It's important to set `by_alias` to `True`, so that the key for `_id` is just that, `_id`, and not the schema's `id` without an underscore.

``` python
# objectid.py

class PydanticObjectId(ObjectId):
    """
    ObjectId field. Compatible with Pydantic.
    """

    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        return PydanticObjectId(v)

    @classmethod
    def __modify_schema__(cls, field_schema: dict):
        field_schema.update(
            type="string",
            examples=["5eb7cf5a86d9755df3a6c593", "5eb7cfb05e32e07750a1756a"],
        )

ENCODERS_BY_TYPE[PydanticObjectId] = str
```

This approach is neat for this particular use-case, but I can't help feeling that it would be limiting in a more complex system. There are many [patterns for storing data in MongoDB](https://www.mongodb.com/blog/post/building-with-patterns-a-summary). These often result in storing data in a form that is optimal for writes or reads, but not necessarily the representation you would wish to export in an API.

>**What is a Slug?**
>
>Looking at the schema above, you may have wondered what a "slug" is ... well, apart from a slimy garden pest.
>
>A slug is a unique, URL-safe, mnemonic used for identifying a document. I picked up the terminology as a Django developer, where this term is part of the framework. A slug is usually derived from another field. In this case, the slug is derived from the name of the cocktail, so if a cocktail was called "Rye Whiskey Old-Fashioned," the slug would be "rye-whiskey-old-fashioned."
>
>In this API, that cocktail could be accessed by sending a `GET` request to the `/cocktails/rye-whiskey-old-fashioned` endpoint.
>
>I've kept the unique `slug` field separate from the auto-assigned `_id` field, but I've provided both because the slug could change if the name of the cocktail was tweaked, in which case the `_id` value would provide a constant identifier to look up an exact document.

In the Rust version of this code, I was nudged to use a different approach. It's a bit more verbose, but in the end I was convinced that it would be more powerful and flexible as the system grew.

## Creating a New Document

Now I'll show you what a single endpoint looks like, first focusing on the "Create" endpoint, that handles a POST request to `/cocktails` and creates a new document in the "recipes" collection. It then returns the document that was stored, including the newly unique ID that MongoDB assigned as `_id`, because this is a RESTful API, and that's what RESTful APIs do.

``` python
@app.route("/cocktails/", methods=["POST"])
def new_cocktail():
    raw_cocktail = request.get_json()
    raw_cocktail["date_added"] = datetime.utcnow()

    cocktail = Cocktail(**raw_cocktail)
    insert_result = recipes.insert_one(cocktail.to_bson())
    cocktail.id = PydanticObjectId(str(insert_result.inserted_id))
    print(cocktail)

    return cocktail.to_json()
```

This endpoint modifies the incoming JSON directly, to add a `date_added` item with the current time. It then passes it to the constructor for our Pydantic schema. At this point, if the schema failed to validate the data, an exception would be raised and displayed to the user.

After validating the data, `to_bson()` is called on the `Cocktail` to convert it to a BSON-compatible dict, and this is directly passed to PyMongo's `insert_one` method. There's no way to get PyMongo to return the document that was just inserted in a single operation (although an upsert using `find_one_and_update` is similar to just that).

After inserting the data, the code then updates the local object with the newly-assigned `id` and returns it to the client.

## Reading a Single Cocktail

Thanks to `Flask-PyMongo`, the endpoint for looking up a single cocktail is even more straightforward:

``` python
@app.route("/cocktails/<string:slug>", methods=["GET"])
def get_cocktail(slug):
    recipe = recipes.find_one_or_404({"slug": slug})
    return Cocktail(**recipe).to_json()
```

This endpoint will abort with a 404 if the slug can't be found in the collection. Otherwise, it simply instantiates a Cocktail with the document from the database, and calls `to_json` to convert it to a dict that Flask will automatically encode correctly as JSON.

## Listing All the Cocktails

This endpoint is a monster, and it's because of pagination, and the links for pagination. In the sample data above, you probably noticed the `_links` section:

``` json
"_links": {
    "last": {
        "href": "http://localhost:5000/cocktails/?page=5"
    }, 
    "next": {
        "href": "http://localhost:5000/cocktails/?page=5"
    }, 
    "prev": {
        "href": "http://localhost:5000/cocktails/?page=3"
    }, 
    "self": {
        "href": "http://localhost:5000/cocktails/?page=4"
    }
}, 
```

This `_links` section is specified as part of the [HAL (Hypertext Application
Language)](https://tools.ietf.org/html/draft-kelly-json-hal-00) specification. It's a good idea to follow a standard for pagination data, and I didn't feel like inventing something myself!

And here's the code to generate all this. Don't freak out.

``` python
@app.route("/cocktails/")
def list_cocktails():
    """
    GET a list of cocktail recipes.

    The results are paginated using the `page` parameter.
    """

    page = int(request.args.get("page", 1))
    per_page = 10  # A const value.

    # For pagination, it's necessary to sort by name,
    # then skip the number of docs that earlier pages would have displayed,
    # and then to limit to the fixed page size, ``per_page``.
    cursor = recipes.find().sort("name").skip(per_page * (page - 1)).limit(per_page)

    cocktail_count = recipes.count_documents({})

    links = {
        "self": {"href": url_for(".list_cocktails", page=page, _external=True)},
        "last": {
            "href": url_for(
                ".list_cocktails", page=(cocktail_count // per_page) + 1, _external=True
            )
        },
    }
    # Add a 'prev' link if it's not on the first page:
    if page > 1:
        links["prev"] = {
            "href": url_for(".list_cocktails", page=page - 1, _external=True)
        }
    # Add a 'next' link if it's not on the last page:
    if page - 1 < cocktail_count // per_page:
        links["next"] = {
            "href": url_for(".list_cocktails", page=page + 1, _external=True)
        }

    return {
        "recipes": [Cocktail(**doc).to_json() for doc in cursor],
        "_links": links,
    }
```

Although there's a lot of code there, it's not as complex as it may first appear. Two requests are made to MongoDB: one for a page-worth of cocktail recipes, and the other for the total number of cocktails in the collection. Various calculations are done to work out how many documents to skip, and how many pages of cocktails there are. Finally, some links are added for "prev" and "next" pages, if appropriate (i.e.: the current page isn't the first or last.) Serialization of the cocktail documents is done in the same way as the previous endpoint, but in a loop this time.

The update and delete endpoints are mainly repetitions of the code I've already included, so I'm not going to include them here. Check them out in the [GitHub repo](https://github.com/mongodb-developer/rewrite-it-in-rust/blob/master/flask-cocktail-api/src/cocktailapi/__init__.py) if you want to see how they work.

## Error Handling

Nothing irritates me more than using a JSON API which returns HTML when an error occurs, so I was keen to put in some reasonable error handling to avoid this happening.

After Flask set-up code, and before the endpoint definitions, the code registers two error-handlers:

``` python
@app.errorhandler(404)
def resource_not_found(e):
    """
    An error-handler to ensure that 404 errors are returned as JSON.
    """
    return jsonify(error=str(e)), 404


@app.errorhandler(DuplicateKeyError)
def resource_not_found(e):
    """
    An error-handler to ensure that MongoDB duplicate key errors are returned as JSON.
    """
    return jsonify(error=f"Duplicate key error."), 400
```

The first error-handler intercepts any endpoint that fails with a 404 status code and ensures that the error is returned as a JSON dict.

The second error-handler intercepts a `DuplicateKeyError` raised by any endpoint, and does the same thing as the first error-handler, but sets the HTTP status code to "400 Bad Request."

As I was writing this post, I realised that I've missed an error-handler to deal with invalid Cocktail data. I'll leave implementing that as an exercise for the reader! Indeed, this is one of the difficulties with writing robust Python applications: Because exceptions can be raised from deep in your stack of dependencies, it's very difficult to comprehensively predict what exceptions your application may raise in different circumstances.

This is something that's very different in Rust, and even though, as you'll see, error-handling in Rust can be verbose and tricky, I've started to love the language for its insistence on correctness.

## Wrapping Up

When I started writing this post, I though it would end up being relatively straightforward. As I added the requirement that the code should not just be a toy example, some of the inherent difficulties with building a robust API on top of any database became apparent.

In this case, Flask may not have been the right tool for the job. I recently wrote a blog post about [building an API with Beanie](https://developer.mongodb.com/article/beanie-odm-fastapi-cocktails/). Beanie and FastAPI are a match made in heaven for this kind of application and will handle validation, transformation, and pagination with much less code. On top of that, they're self-documenting and can provide the data's schema in open formats, including [OpenAPI Spec](https://swagger.io/specification/) and [JSON Schema](https://json-schema.org/)!

If you're about to build an API from scratch, I strongly recommend you check them out, and you may enjoy reading Aaron Bassett's posts on the [FARM (FastAPI, React, MongoDB) Stack](https://developer.mongodb.com/how-to/FARM-Stack-FastAPI-React-MongoDB/).

I will shortly publish the second post in this series, *Build a Cocktail API with Actix-Web, MongoDB, and Rust*, and then I'll conclude with a third post, *I Rewrote it in Rust—How Did it Go?*, where I'll evaluate the strengths and weaknesses of the two experiments.

Thank you for reading. Keep a look out for the upcoming posts!

>If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

Build a RESTful API with Flask, MongoDB, and Python



<div class="introduction">

I have a MongoDB collection containing cocktail recipes that I've made during lockdown.

Recently, I've been trying to build an API over it, using some technologies I know well. I wasn't very happy with the results. Writing code to transform the BSON that comes out of MongoDB into suitable JSON is relatively fiddly. I felt I wanted something more declarative, but my most recent attempt—a mash-up of [Flask](https://flask.palletsprojects.com/en/1.1.x/), [MongoEngine](http://mongoengine.org/), and [Marshmallow](https://marshmallow.readthedocs.io/en/stable/)—just felt clunky and repetitive. I was about to start experimenting with building my own declarative framework, and then I stumbled upon an introduction to a brand new MongoDB ODM called [Beanie](https://dev.to/romanright/announcing-beanie-mongodb-odm-56e). It looked like exactly what I was looking for.

The code used in this post borrows heavily from the Beanie post linked above. I've customized it to my needs, and added an extra endpoint that makes use of [MongoDB Atlas Search](https://docs.atlas.mongodb.com/atlas-search/), to provide autocompletion, for a GUI I'm planning to build in the future.

You can find all the code [on GitHub](https://github.com/mongodb-developer/beanie-example).

>
>
>**Note**: The code here was written for Beanie 0.2.3. It's a new library, and things are moving fast! Check out the [Beanie Changelog](https://roman-right.github.io/beanie/changelog/) to see what things have changed between this version and the latest version of Beanie.
>
>

</div>

<div class="content">

I have a collection of documents that looks a bit like this:

``` json
{
   "_id": "5f7daa158ec9dfb536781b0a",
   "name": "Hunter's Moon",
   "ingredients": [
      {
         "name": "Vermouth",
         "quantity": {
            "quantity": "25",
            "unit": "ml"
         }
      },
      {
         "name": "Maraschino Cherry",
         "quantity": {
            "quantity": "15",
            "unit": "ml"
         }
      },
      {
         "name": "Sugar Syrup",
         "quantity": {
            "quantity": "10",
            "unit": "ml"
         }
      },
      {
         "name": "Lemonade",
         "quantity": {
            "quantity": "100",
            "unit": "ml"
         }
      },
      {
         "name": "Blackberries",
         "quantity": {
            "quantity": "2",
            "unit": null
         }
      }
   ]
}
```

The promise of Beanie and FastAPI—to just build a model for this data and have it automatically translate the tricky field types, like `ObjectId` and `Date` between BSON and JSON representation—was very appealing, so I fired up a new Python project, and defined my schema in a [models](https://github.com/mongodb-developer/beanie-example/blob/master/src/beaniecocktails/models.py) submodule like so:

``` python
class Cocktail(Document):
   class DocumentMeta:
      collection_name = "recipes"

   name: str
   ingredients: List["Ingredient"]
   instructions: List[str]


class Ingredient(BaseModel):
   name: str
   quantity: Optional["IngredientQuantity"]


class IngredientQuantity(BaseModel):
   quantity: Optional[str]
   unit: Optional[str]

Cocktail.update_forward_refs()
Ingredient.update_forward_refs()
```

I was pleased to see that I could define a `DocumentMeta` inner class and override the collection name. It was a feature that I thought *should* be there, but wasn't totally sure it would be.

The other thing that was a little bit tricky was to get `Cocktail` to refer to `Ingredient`, which hasn't been defined at that point. Fortunately,
[Pydantic's](https://pydanticdocs.helpmanual.io/usage/postponed_annotations/) `update_forward_refs` method can be used later to glue together the references. I could have just re-ordered the class definitions, but I preferred this approach.

The [beaniecocktails](https://github.com/mongodb-developer/beanie-example/blob/master/src/beaniecocktails/__init__.py) package, defined in the `__init__.py` file, contains mostly boilerplate code for initializing FastAPI, Motor, and Beanie:

``` python
# ... some code skipped

@app.on_event("startup")
async def app_init():
   client = motor.motor_asyncio.AsyncIOMotorClient(Settings().mongodb_url)
   init_beanie(client.get_default_database(), document_models=[Cocktail])
   app.include_router(cocktail_router, prefix="/v1")
```

The code above defines an event handler for the FastAPI app startup. It connects to MongoDB, configures Beanie with the database connection, and provides the `Cocktail` model I'll be using to Beanie.

The last line adds the `cocktail_router` to Beanie. It's an `APIRouter` that's defined in the [routes](https://github.com/mongodb-developer/beanie-example/blob/master/src/beaniecocktails/routes.py) submodule.

So now it's time to show you the routes file—this is where I spent most of my time. I was *amazed* by how quickly I could get API endpoints developed.

``` python
# ... imports skipped

cocktail_router = APIRouter()
```

The `cocktail_router` is responsible for routing URL paths to different function handlers which will provide data to be rendered as JSON. The simplest handler is probably:

``` python
@cocktail_router.get("/cocktails/", response_model=List[Cocktail])
async def list_cocktails():
   return await Cocktail.find_all().to_list()
```

This handler takes full advantage of these facts: FastAPI will automatically render Pydantic instances as JSON; and Beanie `Document` models are defined using Pydantic. `Cocktail.find_all()` returns an iterator over all the `Cocktail` documents in the `recipes` collection. FastAPI can't deal with these iterators directly, so the sequence is converted to a list using the `to_list()` method.

If you have the [Just](https://github.com/casey/just) task runner installed, you can run the server with:

``` bash
just run
```

If not, you can run it directly by running:

``` bash
uvicorn beaniecocktails:app --reload --debug
```

And then you can test the endpoint by pointing your browser at
"<http://localhost:8000/v1/cocktails/>".

![I drink too many
cocktails](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/beanie-odm-fastapi-cocktails/list_cocktails.png
"I drink too many cocktails")

A similar endpoint for just a single cocktail is neatly encapsulated by two methods: one to look up a document by `_id` and raise a "404 Not Found" error if it doesn't exist, and a handler to route the HTTP request. The two are neatly glued together using the `Depends` declaration that converts the provided `cocktail_id` into a loaded `Cocktail` instance.

``` python
async def get_cocktail(cocktail_id: PydanticObjectId) -> Cocktail:
   """ Helper function to look up a cocktail by id """

   cocktail = await Cocktail.get(cocktail_id)
   if cocktail is None:
      raise HTTPException(status_code=404, detail="Cocktail not found")
   return cocktail

@cocktail_router.get("/cocktails/{cocktail_id}", response_model=Cocktail)
async def get_cocktail_by_id(cocktail: Cocktail = Depends(get_cocktail)):
   return cocktail
```

*Now* for the thing that I really like about Beanie: its integration with MongoDB's [Aggregation Framework](https://docs.mongodb.com/manual/aggregation/). Aggregation pipelines can reshape documents through projection or grouping, and Beanie allows the resulting documents to be mapped to a Pydantic `BaseModel` subclass.

Using this technique, an endpoint can be added that provides an index of all of the ingredients and the number of cocktails each appears in:

``` python
# models.py:

class IngredientAggregation(BaseModel):
   """ A model for an ingredient count. """

   id: str = Field(None, alias="_id")
   total: int

# routes.py:

@cocktail_router.get("/ingredients", response_model=List[IngredientAggregation])
async def list_ingredients():
   """ Group on each ingredient name and return a list of `IngredientAggregation`s. """

   return await Cocktail.aggregate(
      aggregation_query=[
         {"$unwind": "$ingredients"},
         {"$group": {"_id": "$ingredients.name", "total": {"$sum": 1}}},
         {"$sort": {"_id": 1}},
      ],
      item_model=IngredientAggregation,
   ).to_list()
```

The results, at "<http://localhost:8000/v1/ingredients>", look a bit like this:

``` json
[
   {"_id":"7-Up","total":1},
   {"_id":"Amaretto","total":2},
   {"_id":"Angostura Bitters","total":1},
   {"_id":"Apple schnapps","total":1},
   {"_id":"Applejack","total":1},
   {"_id":"Apricot brandy","total":1},
   {"_id":"Bailey","total":1},
   {"_id":"Baileys irish cream","total":1},
   {"_id":"Bitters","total":3},
   {"_id":"Blackberries","total":1},
   {"_id":"Blended whiskey","total":1},
   {"_id":"Bourbon","total":1},
   {"_id":"Bourbon Whiskey","total":1},
   {"_id":"Brandy","total":7},
   {"_id":"Butterscotch schnapps","total":1},
]
```

I loved this feature so much, I decided to use it along with [MongoDB Atlas Search](https://docs.atlas.mongodb.com/atlas-search/), which provides free text search over MongoDB collections, to implement an autocomplete endpoint.

The first step was to [add a search index](https://docs.atlas.mongodb.com/reference/atlas-search/create-index/) on the `recipes` collection, in the MongoDB Atlas web interface:

![Create a new search index on the `recipes` collection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/beanie-odm-fastapi-cocktails/create_index.png "Create a new search index on the `recipes` collection")

I had to add the `name` field as an "autocomplete" field type.

![The name field must be of type "autocomplete"](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/beanie-odm-fastapi-cocktails/add_autocomplete_type.png "The name field must be of type "autocomplete"")

I waited for the index to finish building, which didn't take very long, because it's not a very big collection. Then I was ready to write my autocomplete endpoint:

``` python
@cocktail_router.get("/cocktail_autocomplete", response_model=List[str])
async def cocktail_autocomplete(fragment: str):
   """ Return an array of cocktail names matched from a string fragment. """

   return [
      c["name"]
      for c in await Cocktail.aggregate(
         aggregation_query=[
            {
               "$search": {
                  "autocomplete": {
                     "query": fragment,
                     "path": "name",
                  }
               }
            }
         ]
      ).to_list()
   ]
```

The `$search` aggregation stage specifically uses a search index. In this case, I'm using the `autocomplete` type, to match the type of the index I created on the `name` field. Because I wanted the response to be as lightweight as possible, I'm taking over the serialization to JSON myself, extracting the name from each `Cocktail` instance and just returning a list of strings.

The results are great!

Pointing my browser at
"<http://localhost:8000/v1/cocktail_autocomplete?fragment=fi>" gives me `["Imperial Fizz","Vodka Fizz"]`, and
"<http://localhost:8000/v1/cocktail_autocomplete?fragment=ma>" gives me `["Manhattan","Espresso Martini"]`.

The next step is to build myself a React front end, so that I can truly call this a [FARM Stack](https://developer.mongodb.com/how-to/FARM-Stack-FastAPI-React-MongoDB/) app.

</div>

<div class="summary">

## Wrap-Up

I was really impressed with how quickly I could get all of this up and running. Handling of `ObjectId` instances was totally invisible, thanks to Beanie's `PydanticObjectId` type, and I've seen other sample code that shows how BSON `Date` values are equally well-handled.

I need to see how I can build some [HATEOAS](https://en.wikipedia.org/wiki/HATEOAS) functionality into the endpoints, with entities linking to their canonical URLs. Pagination is also something that will be important as my collection grows, but I think I already know how to handle that.

I hope you enjoyed this quick run-through of my first experience using [Beanie](https://github.com/roman-right/beanie). The next time you're building an API on top of MongoDB, I recommend you give it a try!

If this was your first exposure to the Aggregation Framework, I really recommend you read our [documentation](https://docs.mongodb.com/manual/aggregation/) on this powerful feature of MongoDB. Or if you really want to get your hands dirty, why not check out our [free MongoDB University](https://university.mongodb.com/courses/M121/about) course?

>
>
>If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.
>
>

</div>


Build a Cocktail API with Beanie and MongoDB

## Creators
[Mehant Kammakomati](https://www.linkedin.com/in/mehant-kammakomati-1a0b41170/) and [Sai Vittal B](https://www.linkedin.com/in/saivittalb/) contributed this project.

## About the project

MergeURL is an instant URL shortening and merging service that lets you merge multiple URLs into a single short URL. You can merge up to 5 URLs within no time and share one single shortened URL. MergeURL lifts off the barriers of user registration and authentication, making it instant to use. It also provides two separate URLs to view the URLs' list and open all the browser URLs.

MergeURL is ranked #2 product of the day on ProductHunt. It is used by people across the world, with large numbers coming from the United States and India.


# Inspiration

We had this problem of sharing multiple URLs in a message or an email or via Twitter. We wanted to create a trustworthy service that can merge all those URLs in a single short one.  We tried finding out if there were already solutions to this problem, and most of the solutions we found, required an account creation or to put my credentials. We wanted to have something secure, trustworthy, but that doesn’t require user authentication. Sai Vittal worked mostly on the front end of the application, and I (Mehant) worked on the back end and the MongoDB database. It was a small problem that we encountered that led us to build MergeURL.  

We added our product to ProductHunt last August, and we became number #2 for a while; this gave us the kickstart to reach a wider audience. We currently have around 181.000 users and around 252.000 page views. The number of users motivates us to work a lot on updates and add more security layers to it. 

## Why MongoDB?
    
For MergeURL, MongoDB plays a crucial role in our URL shortening and merging algorithm, contributing to higher security and reducing data redundancy. MongoDB Atlas lifts off the burden to host and maintain databases that made our initial development of MergeURL 10X faster, and further maintaining and monitoring has become relatively easy. 

Firstly we discussed whether to go for a SQL or NoSQL database. According to the algorithms, our primary approach is that going with a NoSQL database would be the better option. MongoDB is at the top of the chart; it is the one that comes to mind when you think about NoSQL databases.  Client libraries like PyMongo make it so much easier to connect and use MongoDB. We use MongoDB Atlas itself because it’s already hosted. It made it much easier for us to work with it. We’ve been using the credits that we received from the GitHub Student Developer Pack offer. 

## How it works

The frontend is written using React, and it’s compiled into the optimal static assets.  As we know, the material is a relatively simple service; we don’t need a lot of complicated stuff in the back end. Therefore we used a microservice; we used Flask to write the backend server. And we use MongoDB. We have specific algorithms that work on the URLs, and MongoDB played a vital role in implementing those algorithms and taking control of the redundancy. 

It works relatively smoothly. You go to our website; you fill out the URLs you want to shorten, and it will give you a short URL that includes all the URLs. 

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mergeurls_shown_along_with_QR_Code_6b33977a03.png)

## Challenges and lessons learned

One of the challenges lies in our experience. We both didn’t have any experience launching a product and getting it to users. Launching MergeURL was the first time we did this, and it went very well.

MongoDB specific, we didn’t have any problems. Specifically (Mehant),  I struggled a lot with SQL databases in my freshman and sophomore years. I’m pleased that I found MongoDB; it saves a lot of stress and frustration. Everything is relatively easy. Besides that, the documents are quite flexible; it’s not restricted as with SQL. We can create many more challenges with MongoDB. 

I’ve learned a lot about the process. Converting ideas into actual implementation was the most important thing. One can have many ideas, but turning them into life is essential. 

At the moment, the project merges the URLs. We are thinking of maybe adding a premium plan where people can get user-specific extensions. We use a counter variable to give those IDs to the shortened URL, but we would like to implement adding user specific extensions. 

And we would like to add analytics. How many users are clicking on your shortened URL? Where is the traffic coming from? 

We are thrilled with the product as it is, but there are plenty of future ideas. 





Shorten multiple URLs instantly while overcoming the barriers of user registration.

MergeURL - Python Example App

Shelby Carpenter

Learn step-by-step how to build a full-stack web application to track reports of unidentified flying objects (UFOs) in your area.

Build an App With Python, Flask, and MongoDB to Track UFOs

Tom McCarthy

Pierre Petersson

The right to erasure, also known as the right to be forgotten, is a right granted to individuals under laws and regulations such as GDPR. This means that companies storing an individual's personal data must be able to delete it on request. Because this data can be spread across several systems, it can be technically challenging for these companies to identify and remove it from all places. Even if this is properly executed, there is also a risk that deleted data can be restored from backups in the future, potentially contributing to legal and financial risks.

This blog post addresses those challenges, demonstrating how you can make use of MongoDB's Client-Side Field Level Encryption to strengthen procedures for removing sensitive data.

>***Disclaimer**: We provide no guarantees that the solution and techniques described in this article will fulfill regulatory requirements around the right to erasure. Each organization needs to make their own determination on appropriate or sufficient measures to comply with various regulatory requirements such as GDPR.*

## What is crypto shredding?
Crypto shredding is a data destruction technique that consists of destroying the encryption keys that allow the data to be decrypted, thus making the data undecipherable. The example below gives a more in-depth explanation.

Imagine you are storing data for multiple users. You start by giving each user their own unique data encryption key (DEK), and mapping it to that customer. This is represented in the below diagram, where "User A" and "User B" each have their own key in the key store. This DEK can then be used to encrypt and decrypt any data related to the user in question.

![Diagram showing different data encryption keys for User A and User B](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/data_encryption_keys_602f207e54.png)

Let's assume that we want to remove all data for User B. If we remove User B's DEK, we can no longer decrypt any of the data that was encrypted with it; all we have left in our data store is "junk" cipher text. As the diagram below illustrates, User A's data is unaffected, but we can no longer read User B's data.

![Diagram showing what happens when user B's data encryption key is deleted from the key store](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/encryption_key_deleted_8c803d3e52.png)

## What is CSFLE?
With MongoDB’s [Client-Side Field Level Encryption](https://www.mongodb.com/docs/manual/core/csfle/) (CSFLE), applications can encrypt sensitive fields in documents prior to transmitting data to the server. This means that even when data is being used by the database in memory, it is never in plain text. The database only sees the encrypted data but still enables you to query it.

MongoDB CSFLE utilizes envelope encryption, which is the practice of encrypting plaintext data with a data key, which itself is in turn encrypted by a top level envelope key (also known as a "master key"). 

![Diagram showing an envelope key being used to encrypt a data key, which in turn encrypts data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/envelop_key_6a599cfd2e.png)

Envelope keys are usually managed by a Key Management Service (KMS). MongoDB CSFLE supports multiple KMSs, such as AWS KMS, GCP KMS, Azure KeyVault, and Keystores supporting the KMIP standard (e.g., Hashicorp Keyvault).

CSFLE can be used in either [automatic](https://www.mongodb.com/docs/manual/core/csfle/fundamentals/automatic-encryption/#std-label-csfle-fundamentals-automatic-encryption) mode or [explicit mode](https://www.mongodb.com/docs/manual/core/csfle/fundamentals/manual-encryption/#std-label-csfle-fundamentals-manual-encryption) — or a combination of both. Automatic mode enables you to perform encrypted read and write operations based on a defined encryption schema, avoiding the need for application code to specify how to encrypt or decrypt fields. This [encryption schema](https://www.mongodb.com/docs/manual/core/csfle/fundamentals/create-schema/#std-label-csfle-fundamentals-create-schema) is a JSON document that defines what fields need to be encrypted. [Explicit](https://www.mongodb.com/docs/manual/core/csfle/fundamentals/manual-encryption/#std-label-csfle-fundamentals-manual-encryption) mode refers to using the MongoDB driver's encryption library to manually encrypt or decrypt fields in your application.

In this article, we are going to use the [explicit encryption](https://www.mongodb.com/docs/manual/core/csfle/fundamentals/manual-encryption/#std-label-csfle-fundamentals-manual-encryption) technique to showcase how we can use crypto shredding techniques with CSFLE to implement (or augment) procedures to "forget" sensitive data. We'll be using AWS KMS to demonstrate this.

## Bringing it all together
With MongoDB as our database, we can use CSFLE to implement crypto shredding, so we can provide stronger guarantees around data privacy.

To demonstrate how you could implement this, we'll walk you through a demo application. The demo application is a python (Flask) web application with a front end, which exposes functionality for signup, login, and a data entry form. We have also added an "admin" page to showcase the crypto shredding related functionality. If you want to follow along, you can run the application yourself — you'll find the necessary code and instructions in [GitHub](https://github.com/mongodb-developer/mongodb-flask-cryptoshredding-example).

![demo application home page](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/demo_application_homepage_ad48ff5c3d.png)

When a user signs up, our application will generate a DEK for the user, then store the ID for the DEK along with other user details. Key generation is done via the  `create_data_key` method on the `ClientEncryption` class, which we initialized earlier as `app.mongodb_encryption_client`. This encryption client is responsible for generating a DEK, which in this case will be encrypted by the envelope key. In our case, the encryption client is configured to use an envelope key from AWS KMS.

```python
# flaskapp/db_queries.py

@aws_credential_handler
def create_key(userId):
    data_key_id = \
        app.mongodb_encryption_client.create_data_key(kms_provider,
            master_key, key_alt_names=[userId])
    return data_key_id
```

We can then use this method when saving the user.

```python
# flaskapp/user.py

def save(self):
        dek_id = db_queries.create_key(self.username)
        result = app.mongodb[db_name].user.insert_one(
            {
                "username": self.username,
                "password_hash": self.password_hash,
                "dek_id": dek_id,
                "createdAt": datetime.now(),
            }
        )
        if result:
            self.id = result.inserted_id
            return True
        else:
            return False
```

Once signed up, the user can then log in, after which they can enter data via a form shown in the screenshot below. This data has a "name" and a "value", allowing the user to store arbitrary key-value pairs.

![demo application showing a form to add data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/form_to_add_data_364735b756.png)

In the database, we'll store this data in a MongoDB collection called “data,” in documents structured like this:

```json
{
   "name": "shoe size",
   "value": "10",
   "username": "tom"
}
```

For the sake of this demonstration, we have chosen to encrypt the value and username fields from this document. Those fields will be encrypted using the DEK created on signup belonging to the logged in user.

```python
# flaskapp/db_queries.py

# Fields to encrypt, and the algorithm to encrypt them with
ENCRYPTED_FIELDS = {
    # Deterministic encryption for username, because we need to search on it
    "username": Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic,
    # Random encryption for value, as we don't need to search on it
    "value": Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Random,
}
```

The insert_data function then loops over the fields we want to encrypt and the algorithm we're using for each.

```python
# flaskapp/db_queries.py

def insert_data(document):
    document["username"] = current_user.username
    # Loop over the field names (and associated algorithm) we want to encrypt
    for field, algo in ENCRYPTED_FIELDS.items():
        # if the field exists in the document, encrypt it
        if document.get(field):
            document[field] = encrypt_field(document[field], algo)
    # Insert document (now with encrypted fields) to the data collection
    app.data_collection.insert_one(document)
```

If the specified fields exist in the document, this will call our encrypt_field function to perform the encryption using the specified algorithm.

```python
# flaskapp/db_queries.py

# Encrypt a single field with the given algorithm
@aws_credential_handler
def encrypt_field(field, algorithm):
    try:
        field = app.mongodb_encryption_client.encrypt(
            field,
            algorithm,
            key_alt_name=current_user.username,
        )
        return field
    except pymongo.errors.EncryptionError as ex:
        # Catch this error in case the DEK doesn't exist. Log a warning and 
        # re-raise the exception
        if "not all keys requested were satisfied" in ex._message:
            app.logger.warn(
                f"Encryption failed: could not find data encryption key for user: {current_user.username}"
            )
        raise ex
```

Once data is added, it will be shown in the web app:

![demo application showing the data added in the previous step](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/demo_application_showing_data_12dfb4df80.png)

Now let's see what happens if we delete the DEK. To do this, we can head over to the admin page. This admin page should only be provided to individuals that have a need to manage keys, and we have some choices:

![demo application admin page, showing the "delete data encryption key" button](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/demo_application_admin_page_829d227a7f.png)

We're going to use the "Delete data encryption key" option, which will remove the DEK, but leave all data entered by the user intact. After that, the application will no longer be able to retrieve the data that was stored via the form. When trying to retrieve the data for the logged in user, an error will be thrown

![demo application showing an error message when trying to retrieve data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/error_message_e04de7ff46.png)

**Note**: After we do perform the data key deletion, the web application may still be able to decrypt and show the data for a short period of time before its cache expires — this takes a maximum of 60 seconds. 

But what is actually left in the database? To get a view of this, you can go back to the Admin page and choose "Fetch data for all users." In this view, we won't throw an exception if we can't decrypt the data. We'll just show exactly what we have stored in the database. Even though we haven't actually deleted the user's data, because the data encryption key no longer exists, all we can see now is cipher text for the encrypted fields "username" and "value".

![demo application showing raw cipher text instead of decrypted data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/raw_cipher_text_8312deece3.png)

And here is the code we're using to fetch the data in this view. As you can see, we use very similar logic to the encrypt method shown earlier. We perform a find operation without any filters to retrieve all the data from our data collection. We'll then loop over our ENCRYPTED_FIELDS dictionary to see which fields need to be decrypted.

```python
# flaskapp/db_queries.py

def fetch_all_data_unencrypted(decrypt=False):
    results = list(app.data_collection.find())

    if decrypt:
        for field in ENCRYPTED_FIELDS.keys():
            for result in results:
                if result.get(field):
                    result[field], result["encryption_succeeded"] = decrypt_field(result[field])
    return results
```

The decrypt_field function is called for each field to be decrypted, but in this case we'll catch the error if we cannot successfully decrypt it due to a missing DEK.

```python
# flaskapp/db_queries.py

# Try to decrypt a field, returning a tuple of (value, status). This will be either (decrypted_value, True), or (raw_cipher_text, False) if we couldn't decrypt
def decrypt_field(field):
    try:
        # We don't need to pass the DEK or algorithm to decrypt a field
        field = app.mongodb_encryption_client.decrypt(field)
        return field, True
    # Catch this error in case the DEK doesn't exist.
    except pymongo.errors.EncryptionError as ex:
        if "not all keys requested were satisfied" in ex._message:
            app.logger.warn(
                "Decryption failed: could not find data encryption key to decrypt the record."
            )
            # If we can't decrypt due to missing DEK, return the "raw" value.
            return field, False
        raise ex
```

We can also use the `mongosh` shell to check directly in the database, just to prove that there's nothing there we can read. 

![mongosh](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mongosh_ccc9e4e3a7.png)

At this point, savvy readers may be asking the question, "But what if we restore the database from a backup?" If we want to prevent this, we can use two separate database clusters in our application — one for storing data and one for storing DEKs (the "key vault"). This theory is applied in the sample application, which requires you to specify two MongoDB connection strings — one for data and one for the key vault. If we use separate clusters, it decouples the restoration of backups for application data and the key vault; restoring a backup on the data cluster won't restore any DEKs which have been deleted from the key vault cluster.

## Conclusion
In this blog post, we've demonstrated how MongoDB's Client-Side Field Level Encryption can be used to simplify the task of "forgetting" certain data. With a single "delete data key" operation, we can effectively forget data which may be stored across different databases, collections, backups, and logs. In a real production application, we may wish to delete all the user's data we can find, on top of removing their DEK. This "defense in depth" approach helps us to ensure that the data is really gone. By implementing crypto shredding, the impact is much smaller if a delete operation fails, or misses some data that should have been wiped.

You can find more details about MongoDB's Client-Side Field Level Encryption in our [documentation](https://www.mongodb.com/docs/manual/core/csfle/). If you have questions, feel free to make a post on our [community forums](https://www.mongodb.com/community/forums/). 

Learn how to make use of MongoDB's Client-Side Field Level Encryption to strengthen procedures for removing sensitive data.

Flask

All Flask Content

Implementing Right to Erasure with CSFLE

Build an App With Python, Flask, and MongoDB to Track UFOs

MergeURL - Python Example App

Build a Cocktail API with Beanie and MongoDB

Build a RESTful API with Flask, MongoDB, and Python

Related Topics