Build a Newsletter Website With the MongoDB Data Platform
Rate this article
"This'll be simple," I thought. "How hard can it be?" I said to myself, unwisely.
Yup, that's me. You're probably wondering how I ended up in this situation.
Once upon a time, there was a small company, and that small company had an internal newsletter to let people know what was going on. Because the company was small and everyone was busy, the absolute simplest and most minimal approach was chosen, i.e. a Google Doc that anyone in the Marketing team could update when there was relevant news. This system worked well.
As the company grew, one Google Doc became many Google Docs, and an automated email was added that went out once a week to remind people to look at the docs. Now, things were not so simple. Maybe the docs got updated, and maybe they didn't, because it was not always clear who owned what. The people receiving the email just saw links to the docs, with no indication of whether there was anything new or good in there, and after a while, they stopped clicking through, or only did so occasionally. The person who had been sending the emails got a new job and asked for someone to take over the running of the newsletter.
This is where I come in. Yes, I failed to hide when the boss came asking for volunteers.
I took one look at the existing system, and knew it could not continue as it was — so of course, I also started looking for suckers er I mean volunteers. Unfortunately, I could not find anyone who wanted to take over whitewashing this particular fence, so I set about trying to figure out how hard it could be to roll my own automated fence-whitewashing system to run the newsletter back end.
Pretty quickly I had my minimum viable product, thanks to MongoDB Atlas and Stitch. And the best part? The whole thing fits into the free tier of both. You can get your own free-forever instance , just by supplying your email address. And if you ask me nicely, I might even throw some free credits your way to try out some of the paid features too.
The first hurdle of this project was unlearning bad relational habits. In the relational database world, a newsletter like this would probably use several JOINs:
- A table of issues
- Containing references to a table of news items
- Containing references to further tables of topics, authors
In the document-oriented world, we don't do it that way. Instead, I defined a simple document format:
This structure should be fairly self-explanatory. Each news item has:
- A title
- Some descriptive text
- A link to more information
- One or more topic tags
- Plus some utility fields to do things like tracking edits
Each item is part of a section and can be published simply to the web, or also to email. I don't want to spam readers with everything, so the email is curated; only items with
email: truego to email, while everything else just shows up on the website but not in readers' inboxes.
One item to point out is the updates array, which is empty in this particular example. This field was a later addition to the format, as I realised when I built the edit functionality that it would be good to track who made edits and when. The flexibility of the document model meant that I could simply add that field without causing any cascading changes elsewhere in the code, or even to documents that had already been created in the database.
So much for the database end. Now we need something to read the documents and do something useful with them.
This logic showcases the benefit of the document object model in MongoDB.
currentNewsItem.itemTitle. I don't have to create a whole separate object representation in my code and laboriously populate that with relational queries among many different tables of a database; I have the exact same object representation in the code as in the database.
In the same way, inputting a new item is simple because I can build up a JSON object from fields in a web form:
And then I can write that directly into the database:
There's a little bit more verbose feedback and error handling on this one than in some other parts of the code since people other than me use this part of the application!
So much for inserting news items into the database. What about when someone wants to, y'know, read an issue of the newsletter? The first thing I need to do is to talk to the MongoDB Atlas database and figure out what is the most recent issue, where an issue is defined as the set of all the news items with the same published date. MongoDB has a feature called the , which works a bit like piping data from one command to another in a UNIX shell. An aggregation pipeline has multiple stages, each one of which makes a transformation to the input data and passes it on to the next stage. It's a great way of doing more complex queries like grouping documents, manipulating arrays, reshaping documents into different models, and so on, while keeping each individual step easy to reason about and debug.
In my case, I used a very simple aggregation pipeline to retrieve the most recent publication dates in the database, with three stages. In the first stage, using , I get all the publication dates. In the second stage, I use to remove any null dates, which correspond to items without a publication date — that is, unpublished items. Finally, I sort the dates, using — you guessed it — to get the most recent ones.
As long as I have a list of all the publication dates, I can use the next most recent date for the navigation controls that let readers look at previous issues of the newsletter. The most important usage, though, is to retrieve the current issue, namely the list of all items with that most recent publication date. That's what the
find()command does, and it takes as its argument a simple document:
In other words, I want all the documents which are published (not the drafts that are sitting in the queue waiting to be published), and where the published date is the most recent date that I found with the aggregation pipeline above.
That reference to
orderSectionsis a utility function that makes sure that the sections of the newsletter come out in the right order. I can also catch any errors that occur, either in the aggregation pipeline or in the find operation itself.
At this point publishing a newsletter is a question of selecting which items go into the issue and updating the published date for all those items:
- The first, the filter, specifies which documents to update, which here means all the ones with an ID in the
- The second is the actual update we are going to make, which is to set the
publishedDateto today's date and mark them as published.
- The third, optional argument, is actually empty in my case because I don't need to specify any options.
I set up a simple HTTP service, which I can then access easily like this:
As is common, the REST API I want to use requires an API key. I generated the key on their website, but I don't want to leave that lying around. Luckily, Stitch also lets me define a secret, so I don't need that API key in plaintext:
And that (apart from 1200 more lines of special cases, admin functions, workarounds, and miscellanea) is that. But I wanted a bit more visibility on which topics were popular, who was using the service and so on. How to do that?
Fortunately, there's an obvious answer to my prayers in the shape of Charts, yet another part of the MongoDB Cloud platform, which let me very quickly build a visualisation of activity on the back-end.
Here's how simple that is: I have my database, imaginatively named "newsletter", and the collection, named "atf" for Above the Fold, the name of the newsletter I inherited. I can see all of the fields from my document, so I can take the
_idfield for my X-axis, and then the
createdDatefor the Y-axis, binning by month, to create a real-time chart of the number of news items submitted each month.
It was a great learning experience to build this thing, and the whole exercise gave me a renewed appreciation for the power of MongoDB, the document model, and the extended MongoDB Cloud platform - both the Atlas database and the correlated services like Stitch and Charts. There's also room for expansion; one of the next features I want to build is search, using feature.
As I mentioned at the beginning, one of the nice things about this project is that the whole thing fits in the free tier of MongoDB Atlas, Stitch, and Charts. You can for your own free-forever instance and start building today, no credit card required, and no expiry date either. There's a helpful onboarding wizard that will walk you through loading some sample data and performing some basic tasks, and when you're ready to go further, the are top-notch, with plenty of worked examples. Once you get into it and want to learn more, the best place to turn is , which gives you the opportunity to learn MongoDB at your own pace. You can also get certified on MongoDB, which will get you listed on our public list of .
Calling the MongoDB Atlas Administration API: How to Do it from Node, Python, and Ruby
May 27, 2022