Full Application

Snippet

MongoDB

Student

Partner

Introductory

Advanced

Lauren Schaefer

Learn how to execute the CRUD (create, read, update, and delete) operations in MongoDB using Node.js in this step-by-step tutorial.

MongoDB and Node.js Tutorial - CRUD Operations

Mark Smith

## An Introduction to GDELT Data
### (and How to Work with It and MongoDB)

Hey there!

There's a good chance that if you're reading this, it's because you're planning to enter the MongoDB "Data as News" Hackathon! If not, well, go ahead and [sign up here](https://www.mongodb.com/world-2022/hackathon)!

Now that that's over with, let's get to the first question you probably have:

### What is GDELT?
GDELT is an acronym, standing for "Global Database of Events, Language and Tone". It's a database of geopolitical event data, automatically derived and translated in real time from hundreds of news sources in 65 languages. It's around two terabytes of data, so it's really quite big!

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_04_12_at_10_03_37_4eaba5f14f.png)

Each event contains the following data:

Details of the one or more actors - usually countries or political entities.
The type of event that has occurred, such as "appeal for judicial cooperation"
The positive or negative sentiment perceived towards the event, on a scale of -10 (very negative) to +10 (very positive)
An "impact score" on the Goldstein Scale, indicating the theoretical potential impact that type of event will have on the stability of a country.

### But what does it look like?
The raw data GDELT provides is hosted as CSV files, zipped and uploaded for every 15 minutes since February 2015. A row in the CSV files contains data that looks a bit like this:

| Field Name  | Value                                                                                                                                         |
|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| <div style="width:220px">_id</div>                | 1037207900                                                                                                                                    |
| Day                   | 20210401                                                                                                                                      |
| MonthYear             | 202104                                                                                                                                        |
| Year                  | 2021                                                                                                                                          |
| FractionDate          | 2021.2493                                                                                                                                     |
| Actor1Code            | USA                                                                                                                                           |
| Actor1Name            | NORTH CAROLINA                                                                                                                                |
| Actor1CountryCode     | USA                                                                                                                                           |
| IsRootEvent           | 1                                                                                                                                             |
| EventCode             | 43                                                                                                                                            |
| EventBaseCode         | 43                                                                                                                                            |
| EventRootCode         | 4                                                                                                                                             |
| QuadClass             | 1                                                                                                                                             |
| GoldsteinScale        | 2.8                                                                                                                                           |
| NumMentions           | 10                                                                                                                                            |
| NumSources            | 1                                                                                                                                             |
| NumArticles           | 10                                                                                                                                            |
| AvgTone               | 1.548672566                                                                                                                                   |
| Actor1Geo_Type        | 3                                                                                                                                             |
| Actor1Geo_Fullname    | Albemarle, North Carolina, United States                                                                                                      |
| Actor1Geo_CountryCode | US                                                                                                                                            |
| Actor1Geo_ADM1Code    | USNC                                                                                                                                          |
| Actor1Geo_ADM2Code    | NC021                                                                                                                                         |
| Actor1Geo_Lat         | 35.6115                                                                                                                                       |
| Actor1Geo_Long        | -82.5426                                                                                                                                      |
| Actor1Geo_FeatureID   | 1017529                                                                                                                                       |
| Actor2Geo_Type        | 0                                                                                                                                             |
| ActionGeo_Type        | 3                                                                                                                                             |
| ActionGeo_Fullname    | Albemarle, North Carolina, United States                                                                                                      |
| ActionGeo_CountryCode | US                                                                                                                                            |
| ActionGeo_ADM1Code    | USNC                                                                                                                                          |
| ActionGeo_ADM2Code    | NC021                                                                                                                                         |
| ActionGeo_Lat         | 35.6115                                                                                                                                       |
| ActionGeo_Long        | -82.5426                                                                                                                                      |
| ActionGeo_FeatureID   | 1017529                                                                                                                                       |
| DateAdded             | 2022-04-01T15:15:00Z                                                                                                                          |
| SourceURL             | https://www.dailyadvance.com/news/local/museum-to-host-exhibit-exploring-change-in-rural-us/article_42fd837e-c5cf-5478-aec3-aa6bd53566d8.html |
| downloadId            | 20220401151500                                                                                                                                |

This event encodes Actor1 (North Carolina) hosting a visit ([Cameo Code 043](http://data.gdeltproject.org/documentation/CAMEO.Manual.1.1b3.pdf))  … and in this case the details of the visit aren't included - it's an "exhibit exploring change in the Rural US." You can click through the SourceURL link to read further details.

Every event looks like this. One or two actors, possibly some "action" detail, and then a verb, encoded using the CAMEO verb encoding. CAMEO is short for "Conflict and Mediation Event Observations", and you can find the full verb listing [in this PDF](http://data.gdeltproject.org/documentation/CAMEO.Manual.1.1b3.pdf). If you need a more "computer readable" version of the CAMEO verbs, one is [hosted here](https://github.com/tenthe/CAMEO-Event-Data-Codebook).

### What's So Interesting About an Enormous Table of Geopolitical Data?
We think that there are a bunch of different ways to think about the data encoded in the GDELT dataset.

Firstly, it's a longitudinal dataset, going back through time. Data in GDELT v2 goes from the present day back to 2015, providing a huge amount of event data for the past 7 years. But the GDELT v1 dataset, which is less rich, goes back until 1979! This gives an unparalleled opportunity to study the patterns and trends of geopolitics for the past 43 years.

More than just a historical dataset, however, GDELT is a living dataset, updated every 15 minutes. This means it can also be considered an event system for understanding the world right now. How you use this ability is up to you, but it shouldn't be ignored!

GDELT is also a geographical dataset. Each event encodes one or more points of its actors and actions, so the data can be analysed from a GIS standpoint. But more than all of this, GDELT models human interactions at a large scale. The Goldstein (impact) score (GoldsteinScale), and the sentiment score (AvgTone) provide the human impact of the events being encoded. 

Whether you choose to explore one of the axes above, using ML, or visualisation; whether you choose to use GDELT data on its own, or combine it with another data source; whether you choose to home in on specific events in the recent past; we're sure that you'll discover new understandings of the world around you by analysing the news data it contains.

### How To Work with GDELT?

Over the next few weeks we're going to be publishing blog posts, hosting live streams and AMA (ask me anything) sessions to help you with your GDELT and MongoDB journey. In the meantime, you have a couple of options: You can work with our existing GDELT data cluster (containing the entirety of last year's GDELT data), or you can load a subset of the GDELT data into your own cluster.

#### Work With Our Hosted GDELT Cluster
We currently host the past year's GDELT data in a cluster called GDELT2. You can access it read-only using Compass, or any of the MongoDB drivers, with the following connection string:

```
mongodb+srv://readonly:readonly@gdelt2.rgl39.mongodb.net/GDELT?retryWrites=true&w=majority
```

The raw data is contained in a collection called "eventsCSV", and a slightly massaged copy of the data (with Actors and Actions broken down into subdocuments) is contained in a collection called "recentEvents".

We're still making changes to this cluster, and plan to load more data in as time goes on (as well as keeping up-to-date with the 15-minute updates to GDELT!), so keep an eye out for updates to this blog post!

#### How to Get GDELT into Your Own MongoDB Cluster
There's a high likelihood that you can't work with the data in its raw form. For one reason or another you need the data in a different format, or filtered in some way to work with it efficiently. In that case, I highly recommend you follow Adrienne's advice in her [GDELT Primer README](https://github.com/adriennetacke/gdelt-primer).

In the next few days we'll be publishing a tool to efficiently load the data you want into a MongoDB cluster. In the meantime, read up on GDELT, have a look at the sample data, and find some teammates to build with!

### Further Reading
The following documents contain most of the official documentation you'll need for working with GDELT. We've summarized much of it here, but it's always good to check the source, and you'll need the CAMEO encoding listing!

[GDELT data documentation](http://data.gdeltproject.org/documentation/GDELT-Event_Codebook-V2.0.pdf)

[GDELT Master file](http://data.gdeltproject.org/gdeltv2/masterfilelist.txt)

[CAMEO code guide](http://data.gdeltproject.org/documentation/CAMEO.Manual.1.1b3.pdf)

### What next? 
We hope the above gives you some insight into this fascinating dataset. We’ve chosen it as the theme, "Data as News", for this year's MongoDB World Hackathon due to it’s size, longevity, currency and global relevance. If you fancy exploring the GDELT dataset more, as well as learning MongoDB, and competing for some one-of-a-kind prizes, well, go ahead and [sign up here to the Hackathon!](https://www.mongodb.com/world-2022/hackathon) We’d be glad to have you! 


What is the GDELT dataset and how to work with it and MongoDB and participate in the MongoDB World Hackathon '22

An Introduction to GDELT Data

Jesse Hall

In this article, we’ll learn how Mongoose, a third-party library for MongoDB, can help you to structure and access your data with ease.

## What is Mongoose?

Many who learn MongoDB get introduced to it through the very popular library, Mongoose. Mongoose is described as “[elegant MongoDB object modeling for Node.js.](https://mongoosejs.com/)”

Mongoose is an ODM (Object Data Modeling) library for MongoDB. While you don’t need to use an Object Data Modeling (ODM) or Object Relational Mapping (ORM) tool to have a great experience with MongoDB, some developers prefer them. Many Node.js developers choose to work with Mongoose to help with data modeling, schema enforcement, model validation, and general data manipulation. And Mongoose makes these tasks effortless.

> If you want to hear from the maintainer of Mongoose, Val Karpov, give this episode of the [MongoDB Podcast](https://mongodb.libsyn.com/ep-22-val-karpov-and-the-mongoose-odm) a listen!

## Why Mongoose?

By default, MongoDB has a flexible data model. This makes MongoDB databases very easy to alter and update in the future. But a lot of developers are accustomed to having rigid schemas.

Mongoose forces a semi-rigid schema from the beginning. With Mongoose, developers must define a Schema and Model.

## What is a schema?

A schema defines the structure of your collection documents. A Mongoose schema maps directly to a MongoDB collection.

``` js
const blog = new Schema({
  title: String,
  slug: String,
  published: Boolean,
  author: String,
  content: String,
  tags: [String],
  createdAt: Date,
  updatedAt: Date,
  comments: [{
    user: String,
    content: String,
    votes: Number
  }]
});
```

With schemas, we define each field and its data type. Permitted types are:

* String
* Number
* Date
* Buffer
* Boolean
* Mixed
* ObjectId
* Array
* Decimal128
* Map

## What is a model?

Models take your schema and apply it to each document in its collection.

Models are responsible for all document interactions like creating, reading, updating, and deleting (CRUD).

> An important note: the first argument passed to the model should be the singular form of your collection name. Mongoose automatically changes this to the plural form, transforms it to lowercase, and uses that for the database collection name.

``` js
const Blog = mongoose.model('Blog', blog);
```

In this example, `Blog` translates to the `blogs` collection.

## Environment setup

Let’s set up our environment. I’m going to assume you have [Node.js](https://nodejs.org/) installed already.

We’ll run the following commands from the terminal to get going:

```
mkdir mongodb-mongoose
cd mongodb-mongoose
npm init -y
npm i mongoose
npm i -D nodemon
code .
```

This will create the project directory, initialize, install the packages we need, and open the project in VS Code.

Let’s add a script to our `package.json` file to run our project. We will also use ES Modules instead of Common JS, so we’ll add the module `type` as well. This will also allow us to use top-level `await`.

``` js
...
  "scripts": {
    "dev": "nodemon index.js"
  },
  "type": "module",
...
```

## Connecting to MongoDB

Now we’ll create the `index.js` file and use Mongoose to connect to MongoDB.

``` js
import mongoose from 'mongoose'

mongoose.connect("mongodb+srv://<username>:<password>@cluster0.eyhty.mongodb.net/myFirstDatabase?retryWrites=true&w=majority")
```

You could connect to a local MongoDB instance, but for this article we are going to use a free MongoDB Atlas cluster. If you don’t already have an account, it's easy to sign up for a [free MongoDB Atlas cluster here](https://www.mongodb.com/cloud/atlas/register2).

And if you don’t already have a cluster set up, follow our guide to [get your cluster created](https://www.mongodb.com/docs/atlas/getting-started/).

After creating your cluster, you should replace the connection string above with your connection string including your username and password.

> The connection string that you copy from the MongoDB Atlas dashboard will reference the `myFirstDatabase` database. Change that to whatever you would like to call your database.

## Creating a schema and model

Before we do anything with our connection, we’ll need to create a schema and model.

Ideally, you would create a schema/model file for each schema that is needed. So we’ll create a new folder/file structure: `model/Blog.js`.

``` js
import mongoose from 'mongoose';
const { Schema, model } = mongoose;

const blogSchema = new Schema({
  title: String,
  slug: String,
  published: Boolean,
  author: String,
  content: String,
  tags: [String],
  createdAt: Date,
  updatedAt: Date,
  comments: [{
    user: String,
    content: String,
    votes: Number
  }]
});

const Blog = model('Blog', blogSchema);
export default Blog;
```

## Inserting data // method 1

Now that we have our first model and schema set up, we can start inserting data into our database.

Back in the `index.js` file, let’s insert a new blog article.

``` js
import mongoose from 'mongoose';
import Blog from './model/Blog';

mongoose.connect("mongodb+srv://mongo:mongo@cluster0.eyhty.mongodb.net/myFirstDatabase?retryWrites=true&w=majority")

// Create a new blog post object
const article = new Blog({
  title: 'Awesome Post!',
  slug: 'awesome-post',
  published: true,
  content: 'This is the best post ever',
  tags: ['featured', 'announcement'],
});

// Insert the article in our MongoDB database
await article.save();
```

We first need to import the `Blog` model that we created. Next, we create a new blog object and then use the `save()` method to insert it into our MongoDB database.

Let’s add a bit more after that to log what is currently in the database. We’ll use the `findOne()` method for this.

``` js
// Find a single blog post
const firstArticle = await Blog.findOne({});
console.log(firstArticle);
```

Let’s run the code!

```
npm run dev
```

You should see the document inserted logged in your terminal.

> Because we are using `nodemon` in this project, every time you save a file, the code will run again. If you want to insert a bunch of articles, just keep saving. 😄

## Inserting data // method 2

In the previous example, we used the `save()` Mongoose method to insert the document into our database. This requires two actions: instantiating the object, and then saving it.

Alternatively, we can do this in one action using the Mongoose `create()` method.

``` js
// Create a new blog post and insert into database
const article = await Blog.create({
  title: 'Awesome Post!',
  slug: 'awesome-post',
  published: true,
  content: 'This is the best post ever',
  tags: ['featured', 'announcement'],
});

console.log(article);
```

This method is much better! Not only can we insert our document, but we also get returned the document along with its `_id` when we console log it.

## Update data

Mongoose makes updating data very convenient too. Expanding on the previous example, let’s change the `title` of our article.

``` js
article.title = "The Most Awesomest Post!!";
await article.save();
console.log(article);
```

We can directly edit the local object, and then use the `save()` method to write the update back to the database. I don’t think it can get much easier than that!

## Finding data

Let’s make sure we are updating the correct document. We’ll use a special Mongoose method, `findById()`, to get our document by its ObjectId.

``` js
const article = await Blog.findById("62472b6ce09e8b77266d6b1b").exec();
console.log(article);
```

> Notice that we use the `exec()` Mongoose function. This is technically optional and returns a promise. In my experience, it’s better to use this function since it will prevent some head-scratching issues. If you want to read more about it, check out this note in the Mongoose docs about [promises](https://mongoosejs.com/docs/promises.html).

There are many query options in Mongoose. View the [full list of queries](https://mongoosejs.com/docs/queries.html).

## Projecting document fields

Just like with the standard MongoDB Node.js driver, we can project only the fields that we need. Let’s only get the `title`, `slug`, and `content` fields.

``` js
const article = await Blog.findById("62472b6ce09e8b77266d6b1b", "title slug content").exec();
console.log(article);
```

The second parameter can be of type `Object|String|Array<String>` to specify which fields we would like to project. In this case, we used a `String`.

## Deleting data

Just like in the standard MongoDB Node.js driver, we have the `deleteOne()` and `deleteMany()` methods.

``` js
const blog = await Blog.deleteOne({ author: "Jesse Hall" })
console.log(blog)

const blog = await Blog.deleteMany({ author: "Jesse Hall" })
console.log(blog)
```

## Validation

Notice that the documents we have inserted so far have not contained an `author`, dates, or `comments`. So far, we have defined what the structure of our document should look like, but we have not defined which fields are actually required. At this point any field can be omitted.

Let’s set some required fields in our `Blog.js` schema.

``` js
const blogSchema = new Schema({
  title:  {
    type: String,
    required: true,
  },
  slug:  {
    type: String,
    required: true,
    lowercase: true,
  },
  published: {
    type: Boolean,
    default: false,
  },
  author: {
    type: String,
    required: true,
  },
  content: String,
  tags: [String],
  createdAt: {
    type: Date,
    default: () => Date.now(),
    immutable: true,
  },
  updatedAt: Date,
  comments: [{
    user: String,
    content: String,
    votes: Number
  }]
});
```

When including validation on a field, we pass an object as its value.

> `value: String` is the same as `value: {type: String}`.

There are several validation methods that can be used.

We can set `required` to true on any fields we would like to be required.

For the `slug`, we want the string to always be in lowercase. For this, we can set `lowercase` to true. This will take the slug input and convert it to lowercase before saving the document to the database.

For our `created` date, we can set the default buy using an arrow function. We also want this date to be impossible to change later. We can do that by setting `immutable` to true.

> Validators only run on the create or save methods.

## Other useful methods

Mongoose uses many standard MongoDB methods plus introduces many extra helper methods that are abstracted from regular MongoDB methods. Next, we’ll go over just a few of them.

### `exists()`

The `exists()` method returns either `null` or the ObjectId of a document that matches the provided query.

``` js
const blog = await Blog.exists({ author: "Jesse Hall" })
console.log(blog)
```

### `where()`

Mongoose also has its own style of querying data. The `where()` method allows us to chain and build queries.

``` js
// Instead of using a standard find method
const blogFind = await Blog.findOne({ author: "Jesse Hall" });

// Use the equivalent where() method
const blogWhere = await Blog.where("author").equals("Jesse Hall");
console.log(blogWhere)
```

Either of these methods work. Use whichever seems more natural to you.

You can also chain multiple `where()` methods to include even the most complicated query.

### `select()`

To include projection when using the `where()` method, chain the `select()` method after your query.

``` js
const blog = await Blog.where("author").equals("Jesse Hall").select("title author")
console.log(blog)
```

## Multiple schemas

It's important to understand your options when modeling data.

If you’re coming from a relational database background, you’ll be used to having separate tables for all of your related data.

Generally, in MongoDB, data that is accessed together should be stored together.

You should plan this out ahead of time if possible. Nest data within the same schema when it makes sense.

If you have the need for separate schemas, Mongoose makes it a breeze.

Let’s create another schema so that we can see how multiple schemas can be used together.

We’ll create a new file, `User.js`, in the model folder.

``` js
import mongoose from 'mongoose';
const {Schema, model} = mongoose;

const userSchema = new Schema({
  name: {
    type: String,
    required: true,
  },
  email: {
    type: String,
    minLength: 10,
    required: true,
    lowercase: true
  },
});

const User = model('User', userSchema);
export default User;
```

For the `email`, we are using a new property, `minLength`, to require a minimum character length for this string.

Now we’ll reference this new user model in our blog schema for the `author` and `comments.user`.

``` js
import mongoose from 'mongoose';
const { Schema, SchemaTypes, model } = mongoose;

const blogSchema = new Schema({
  ...,
  author: {
    type: SchemaTypes.ObjectId,
    ref: 'User',
    required: true,
  },
  ...,
  comments: [{
    user: {
      type: SchemaTypes.ObjectId,
      ref: 'User',
      required: true,
    },
    content: String,
    votes: Number
  }];
});
...
```

Here, we set the `author` and `comments.user` to `SchemaTypes.ObjectId` and added a `ref`, or reference, to the user model.

This will allow us to “join” our data a bit later.

And don’t forget to destructure `SchemaTypes` from `mongoose` at the top of the file.

Lastly, let’s update the `index.js` file. We’ll need to import our new user model, create a new user, and create a new article with the new user’s `_id`.

``` js
...
import User from './model/User.js';

...

const user = await User.create({
  name: 'Jesse Hall',
  email: 'jesse@email.com',
});

const article = await Blog.create({
  title: 'Awesome Post!',
  slug: 'Awesome-Post',
  author: user._id,
  content: 'This is the best post ever',
  tags: ['featured', 'announcement'],
});

console.log(article);
```

Notice now that there is a `users` collection along with the `blogs` collection in the MongoDB database.

You’ll now see only the user `_id` in the author field. So, how do we get all of the info for the author along with the article?

We can use the `populate()` Mongoose method.

``` js
const article = await Blog.findOne({ title: "Awesome Post!" }).populate("author");
console.log(article);
```

Now the data for the `author` is populated, or “joined,” into the `article` data. Mongoose actually uses the MongoDB `$lookup` method behind the scenes.

## Middleware

In Mongoose, middleware are functions that run before and/or during the execution of asynchronous functions at the schema level.

Here’s an example. Let’s update the `updated` date every time an article is saved or updated. We’ll add this to our `Blog.js` model.

``` js
blogSchema.pre('save', function(next) {
  this.updated = Date.now(); // update the date every time a blog post is saved
  next();
});
```

Then in the `index.js` file, we’ll find an article, update the title, and then save it.

``` js
const article = await Blog.findById("6247589060c9b6abfa1ef530").exec();
article.title = "Updated Title";
await article.save();
console.log(article);
```

Notice that we now have an `updated` date!

Besides `pre()`, there is also a `post()` mongoose middleware function.

## Next steps

I think our example here could use another schema for the `comments`. Try creating that schema and testing it by adding a few users and comments.

There are many other great Mongoose helper methods that are not covered here. Be sure to check out the [official documentation](https://mongoosejs.com/docs/) for references and more examples.

## Conclusion

I think it’s great that developers have many options for connecting and manipulating data in MongoDB. Whether you prefer Mongoose or the standard MongoDB drivers, in the end, it’s all about the data and what’s best for your application and use case.

I can see why Mongoose appeals to many developers and I think I’ll use it more in the future.

In this article, we’ll learn how Mongoose, a library for MongoDB, can help you to structure and access your data with ease. Many who learn MongoDB get introduced to it through the very popular library, Mongoose. Mongoose is described as “elegant MongoDB object modeling for Node.js."

Quickstarts

Featured Quickstarts

MongoDB and Node.js Tutorial - CRUD Operations

An Introduction to GDELT Data

Getting Started with MongoDB & Mongoose

All Quickstarts