Rachelle Palmer

## Creators
[Lucas De Oliveira](https://www.linkedin.com/in/lbdeoliveira/), [Chandrish Ambati](https://www.linkedin.com/in/avmchandrish/), and [Anish Mukherjee](https://www.linkedin.com/in/mukherjeeanish/) from University of San Francisco contributed this amazing project.

## Background to the Project
In 2018, Spotify organized an Association for Computing Machinery (ACM) [RecSys Challenge](https://www.recsyschallenge.com/2018/) where they posted [a dataset of one million playlists](https://www.aicrowd.com/challenges/spotify-million-playlist-dataset-challenge), challenging participants to recommend a list of 500 songs given a user-created playlist.

As both music lovers and data scientists, we were naturally drawn to this challenge. Right away, we agreed that combining song embeddings with some nearest-neighbors method for recommendation would likely produce very good results. Importantly, we were curious about how we could  solve  this recommendation task at scale with over [4 billion user-curated playlists](https://soundplate.com/how-many-playlists-are-there-on-spotify-and-other-spotify-stats/) on Spotify, where this number keeps growing. This realization raised serious questions about how to train a decent model since all that data would likely not fit in memory or a single server.

## What We Built
This project resulted in a scalable ETL pipeline utilizing
* Apache Spark
* MongoDB
* Amazon S3
* Databricks (PySpark)

These were used to train a deep learning Word2Vec model to build song and playlist embeddings for recommendation. We followed up with data visualizations we created on Tensorflow’s Embedding Projector.

## The Process
### Collecting Lyrics
The most tedious task of this project was collecting as many lyrics for the songs in the playlists as possible. We began by isolating the unique songs in the playlist files by their track URI; in total we had over 2 million unique songs. Then, we used the track name and artist name to look up the lyrics on the web. Initially, we used simple Python requests to pull in the lyrical information but this proved too slow for our purposes. We then used asyncio, which allowed us to make requests concurrently. This sped up the process significantly, reducing the downloading time of lyrics for 10k songs from 15 mins to under a minute. Ultimately, we were only able to collect lyrics for 138,000 songs.

### Pre-processing
The original dataset contains 1 million playlists spread across 1 thousand JSON files totaling about 33 GB of data. We used PySpark in Databricks to preprocess these separate JSON files into a single SparkSQL DataFrame and then joined this DataFrame with the lyrics we saved. 

While the aforementioned data collection and preprocessing steps are time-consuming, the model also needs to be re-trained and re-evaluated often, so it is critical to store data in a scalable database. In addition, we’d like to consider a database that is schemaless for future expansion in data sets and supports various data types.  Considering our needs, we concluded that MongoDB would be the optimal solution as a data and feature store.

Check out the [Preprocessing.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Preprocessing_Pipeline.ipynb) notebook to see how we preprocessed the data.

### Training Song Embeddings
For our analyses, we read our preprocessed data from MongoDB into a Spark DataFrame and grouped the records by playlist id (pid), aggregating all of the songs in a playlist into a list under the column song_list. 
Using the Word2Vec model in Spark MLlib we trained song embeddings by feeding lists of track IDs from a playlist into the model much like you would send a list of words from a sentence to train word embeddings. As shown below, we trained song embeddings in only 3 lines of PySpark code:
```
from pyspark.ml.feature import Word2Vec
word2Vec = Word2Vec(vectorSize=32, seed=42, inputCol="song_list").setMinCount(1)
word2Vec.sexMaxIter(10)
model = word2Vec.fit(df_play)
```

We then saved the song embeddings down to MongoDB for later use. Below is a snapshot of the song embeddings DataFrame that we saved:

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/song_embeddings_dataframe_c3fc7a455b.png)

Check out the [Song_Embeddings.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Song_Embeddings.ipynb) notebook to see how we train song embeddings.

### Training Playlists Embeddings
Finally, we extended our recommendation task beyond simple song recommendations to recommending entire playlists. Given an input playlist, we would return the k closest or most similar playlists. We took a “continuous bag of songs” approach to this problem by calculating playlist embeddings as the average of all song embeddings in that playlist.

This workflow started by reading back the song embeddings from MongoDB into a SparkSQL DataFrame. Then, we calculated a playlist embedding by taking the average of all song embeddings in that playlist and saved them in MongoDB.

Check out the [Playlist_Embeddings.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Playlist_Embeddings.ipynb) notebook to see how we did this.

### Training Lyrics Embeddings
Are you still reading? Whew!

We trained lyrics embeddings by loading in a song's lyrics, separating the words into lists, and feeding those words to a Word2Vec model to produce 32-dimensional vectors for each word. We then took the average embedding across all words as that song's lyrical embedding. Ultimately, our analytical goal here was to determine whether users create playlists based on common lyrical themes by seeing if the pairwise song embedding distance and the pairwise lyrical embedding distance between two songs were correlated. Unsurprisingly, it appears they are not.

Check out the [Lyrical_Embeddings.ipynb](https://github.com/lbdeoliveira/song-playlist-recommendation/blob/main/notebooks/Lyrical_Embeddings.ipynb) notebook to see our analysis.

## Notes on our Approach
You may be wondering why we used a language model (Word2Vec) to train these embeddings. Why not use a Pin2Vec or custom neural network model to predict implicit ratings? For practical reasons, we wanted to work exclusively in the Spark ecosystem and deal with the data in a distributed fashion. This was a constraint set on the project ahead of time and challenged us to think creatively.

However, we found Word2Vec an attractive candidate model for theoretical reasons as well. The Word2Vec model uses a word’s context to train static embeddings by training the input word’s embeddings to predict its surrounding words. In essence, the embedding of any word is determined by how it co-occurs with other words. This had a clear mapping to our own problem: by using a Word2Vec model the distance between song embeddings would reflect the songs’ co-occurrence throughout 1M playlists, making it a useful measure for a distance-based recommendation (nearest neighbors). It would effectively model how people grouped songs together, using user behavior as the determinant factor in similarity.

Additionally, the Word2Vec model accepts input in the form of a list of words. For each playlist we had a list of track IDs, which made working with the Word2Vec model not only conceptually but also practically appealing.

## Data Visualizations with Tensorflow and MongoDB
After all of that, we were finally ready to visualize our results and make some interactive recommendations. We decided to represent our embedding results visually using [Tensorflow’s Embedding Projector](https://projector.tensorflow.org) which maps the 32-dimensional song and playlist embeddings into an interactive visualization of a 3D embedding space. You have the choice of using PCA or tSNE for dimensionality reduction and cosine similarity or Euclidean distance for measuring distances between vectors.

Click [here](https://projector.tensorflow.org/?config=https://embedding-projector.s3.us-west-2.amazonaws.com/template_config_2M.json) for the song embeddings projector for the full 2 million songs, or [here](https://projector.tensorflow.org/?config=https://embedding-projector.s3.us-west-2.amazonaws.com/template_config.json) for a less crowded version with a random sample of 100k songs (shown below):

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/164569070_88be5d94_a291_495d_b33d_3094634d9205_afc69037e9.gif)

The neat thing about using Tensorflow’s projector is that it gives us a beautiful visualization tool and distance calculator all in one. Try searching on the right panel for a song and if the song is part of the original dataset, you will see the “most similar” songs appear under it.

## Using MongoDB for ML/AI
We were impressed by how easy it was to use MongoDB to reliably store and load our data. Because we were using distributed computing, it would have been infeasible to run our pipeline from start to finish any time we wanted to update our code or fine-tune the model. MongoDB allowed us to save our incremental results for later processing and modeling, which collectively saved us hours of waiting for code to re-run.

It worked well with all the tools we use everyday and the tooling we chose - we didn't have any areas of friction. 

We were shocked by how this method of training embeddings actually worked. While the 2 million song embedding projector is crowded visually, we see that the recommendations it produces are actually quite good at grouping songs together.

Consider the embedding recommendation for The Beatles’ “A Day In The Life”:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/165581471_420ad43d_47f9_4aee_bf0c_2de6ac301144_c656a07f0e.gif)

Or the recommendation for Jay Z’s “Heart of the City (Ain’t No Love)”:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/164567916_28f6ebd6_0473_4d7b_b60c_55af628016f7_ec16751581.png)

Fan of Taylor Swift? Here are the recommendations for “New Romantics”:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/164567889_9ccf8326_7bc8_4c8e_9bc6_f123c0acba3f_0fab2ea1db.png)

We were delighted to find naturally occurring clusters in the playlist embeddings. Most notably, we see a cluster containing mostly Christian rock, one with Christmas music, one for reggaeton, and one large cluster where genres span its length rather continuously and intuitively.

Note also that when we select a playlist, we have many recommended playlists with the same names. This in essence validates our song embeddings. Recall that playlist embeddings were created by taking the average embedding of all its songs; the name of the playlists did not factor in at all. The similar names only conceptually reinforce this fact.



## Next Steps?
We felt happy with the conclusion of this project but there is more that could be done here.

1. We could use these trained song embeddings in other downstream tasks and see how effective these are. Also, you could download the song embeddings we here: [Embeddings](https://embedding-projector.s3.us-west-2.amazonaws.com/emb_word2vec_2M.tsv) | [Meta Info](https://embedding-projector.s3.us-west-2.amazonaws.com/meta_word2vec_2M.tsv)
2. We could look at other methods of training these embeddings using some recurrent neural networks and enhanced implementation of this Word2Vec model.





Python code example application for Spotify playlist and song recommendations using spark and tensorflow

A Spotify Song and Playlist Recommendation Engine

Pavel Duchovny

Modern platforms have a wide variety of data sources. As businesses grow, they have to constantly evolve their data management and have sophisticated, scalable, and convenient tools to analyse data from all sources to produce business insights.

MongoDB has developed a rich and powerful [query language](https://docs.mongodb.com/manual/tutorial/query-documents/), including a very robust [aggregation framework](https://docs.mongodb.com/manual/aggregation/). 

These were mainly done to optimize the way developers work with data and provide great tools to manipulate and query MongoDB documents.

Having said that, many developers, analysts, and tools still prefer the legacy SQL language to interact with the data sources. SQL has a strong foundation around joining data as this was a core concept of the legacy relational databases normalization model. 

This makes SQL have a convenient syntax when it comes to describing joins. 

Providing MongoDB users the ability to leverage SQL to analyse multi-source documents while having a flexible schema and data store is a compelling solution for businesses.

## Data Sources and the Challenge

Consider a requirement to create a single view to analyze data from operative different systems. For example:

- Customer data is managed in the user administration systems (REST API).
- Financial data is managed in a financial cluster (Atlas cluster).
- End-to-end transactions are stored in files on cold storage gathered from various external providers (cloud object storage - Amazon S3 or Microsoft Azure Blob Storage store).

How can we combine and best join this data? 

[MongoDB Atlas Query Federation](https://www.mongodb.com/docs/atlas/data-federation/) connects multiple data sources using the different [data store types](https://www.mongodb.com/docs/atlas/data-federation/overview/#mongodb-expression-exp.Data-Store). Once the data sources are mapped, we can create collections consuming this data. Those collections can have [SQL schema](https://docs.mongodb.com/datalake/reference/cli/sql/sqlgenerateschema/) generated, allowing us to perform sophisticated joins and do JDBC queries from various BI tools.

In this article, we will showcase the extreme power hidden in Atlas [SQL Query](https://www.mongodb.com/docs/atlas/data-federation/query/query-with-sql/).

## Setting Up My Federated Database Instance
In the following view, I have created three main data stores: 
- S3 Transaction Store (S3 sample data).
- Accounts from my Atlas clusters (Sample data sample_analytics.accounts).
- Customer data from a secure https source.

![Query Federation Sources](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_33_52_ec0802abed.png)

I mapped the stores into three collections under `FinTech` database:

- `Transactions`
- `Accounts`
- `CustomerDL`

Now, I can see them through a  Query Federation connection as MongoDB collections.

Let's grab our Query Federation instance connection string from the Atlas UI.

![Connection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_37_55_be8464c0d1.png)

This connection string can be used with our BI tools or client applications to run SQL queries.

## Connecting and Using $sql and db.sql

Once we connect to the Query Federation instancee via a mongosh shell, we can generate a SQL schema for our collections. This is optional for the JDBC or $sql operators to recognise collections as SQL “tables”  as this step is done automatically for newly created collections, however, its always good to be familiar with the available commands.

#### Generate SQL schema for each collection:
```js
use admin;
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.customersDL"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.accounts"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.transactions"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
```
#### Running SQL queries and joins using $sql stage:
```js
use FinTech;
db.aggregate([{
  $sql: {
    statement: "SELECT a.* , t.transaction_count FROM accounts a, transactions t where a.account_id = t.account_id SORT BY t.transaction_count DESC limit 2",
    format: "jdbc",
    formatVersion: 2,
    dialect: "mysql",
  }
}])

// Equivalent command
db.sql("SELECT a.* , t.transaction_count FROM accounts a, transactions t where a.account_id = t.account_id SORT BY t.transaction_count DESC limit 2");
```

The above query will prompt account information and the transaction counts of each account.

## Connecting Via JDBC

Let’s connect a powerful BI tool like Tableau with the [JDBC driver](https://www.mongodb.com/docs/atlas/data-federation/query/sql/drivers/jdbc/connect/).

[Download JDBC Driver](https://search.maven.org/artifact/org.mongodb/mongodb-jdbc).

![Latest Driver](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_44_11_e2df7beb3c.png)

#### Connect to Tableau
You have 2 main options to connect, via "MongoDB Atlas" connector or via a JDBC general connector. Please follow the relevant instructions and prerequisites on this [documentation](https://www.mongodb.com/docs/atlas/data-federation/query/sql/tableau/connect/) page.

##### Connector "MongoDB Atlas by MongoDB"
Search and click the  “MongoDB Atlas by MongoDB” connector and provide the information pointing to our Query Federation URI. See the following example:

![Tablue Connector](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_50_53_ace30bbb42.png)

##### "JDBC" Connector

Setting `connection.properties` file.
```
user=root
password=*******
authSource=admin
database=FinTech
ssl=true
compressors=zlib
```

Click the “Other Databases (JDBC)” connector, copy JDBC connection format, and load the `connection.properties` file.

![Tableau Connector Selection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/tableau_select_other_db_1c75d7023c.png)

Once the data is read successfully, the collections will appear on the right side.

#### Setting and Joining Data

![Setting Tables](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_11_19_53_8913b663c6.png)

We can drag and drop collections from different sources and link them together.

![Joining Tables with Drag & Drop](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_15_14_36_0e2fdcae97.png)

In my case, I connected `Transactions` => `Accounts` based on the `Account Id` field, and accounts and users based on the `Account Id` to `Accounts` field.

![Unified View of Accounts, Transactions and Users data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_15_06_50_9e906438d5.png)

In this view, we will see a unified table for all accounts with usernames and their transactions start quarter. 

## Summary

MongoDB has all the tools to read, transform, and analyse your documents for almost any use-case.  

Whether your data is in an Atlas operational cluster, in a service, or on cold storage like cloud object storage, Atlas Query Federation will provide you with the ability to join the data in real time. With the option to use powerful join SQL syntax and SQL-based BI tools like Tableau, you can get value out of the data in no time.

Try Atlas Query Federation with your BI tools and SQL today.

Learn how new SQL-based queries can power your Query Federation insights in minutes. Integrate this capability with powerful BI tools like Tableau to get immediate value out of your data. 

Data Visualization

All Data Visualization Content

A Spotify Song and Playlist Recommendation Engine

Atlas Query Federation SQL to Form Powerful Data Interactions

Related Topics