Datasets
Descriptions of the datasets used in class.
US Billboard Hot 100 lyrics, 1973–2023
Collected by Foramitti et al. (2025) to study long-term trends in stress, negativity, and simplicity in popular music lyrics and how societal crises affect those.
Observations. 266,087 entries, one per song per chart week, from January 6, 1973 to December 30, 2023 (51 years)
Variables.
| Name | Description |
|---|---|
Date |
Chart week (weekly, 1973-01-06 to 2023-12-30) |
Rank |
Position on that week’s Billboard Hot 100 chart (1–100) |
Song Title |
Song title |
Artist |
Performing artist |
Lyrics_Sentiment |
VADER compound sentiment score of the lyrics, from −1 (most negative) to +1 (most positive) |
Stress |
Frequency of stress-related words in the lyrics, from LIWC |
Lyrics_Sentiment and Stress are missing for about 13,600 rows: these correspond to songs excluded from the analysis for being instrumental, having incorrect scraping returns, or having non-English lyrics.
How it was collected. The Billboard Hot 100 chart was scraped weekly with BeautifulSoup; lyrics for each charted song were retrieved via the Lyrics.ovh and Genius APIs, cleaned of metadata/inserts (e.g. “[Chorus]”), and filtered down to English-language lyrics only. Sentiment was scored with the VADER sentiment analysis tool, and stress-related language with the LIWC tool; each song’s score is the average across all its lyrics’ words.
Links.
- Data repository (OSF): https://osf.io/2k7ut/
- Article: Foramitti et al. (2025), Societal crises disrupt long-term increases in stress, negativity, and simplicity in US Billboard song lyrics from 1973 to 2023, Scientific Reports
Loading the data in pandas. Download lyrics_OSF.csv from the OSF repository above, then:
import pandas as pd
df = pd.read_csv("lyrics_OSF.csv", index_col=0, parse_dates=["Date"])
df.head()Spotify songs
Audio features and metadata for ~32,800 Spotify tracks across 6 playlist genres, compiled by researcher Kaylin Pavlik using Spotify’s API (via the R spotifyr package) to explore genre classification patterns, and distributed through the TidyTuesday project.
Observations. 32,833 rows, one per song per playlist it appears in (28,356 unique tracks; a song can appear in multiple playlists, hence multiple rows). Album release dates range from 1957 to early 2020.
Variables.
| Name | Description |
|---|---|
track_id, track_name, track_artist |
Track identifier, title, and performing artist |
track_popularity |
Popularity score (0–100) |
track_album_id, track_album_name, track_album_release_date |
Album identifier, title, and release date |
playlist_name, playlist_id |
Playlist title and identifier |
playlist_genre, playlist_subgenre |
Primary genre (edm, latin, pop, r&b, rap, rock) and subgenre |
danceability, energy, key, loudness, mode, speechiness, acousticness, instrumentalness, liveness, valence, tempo |
Spotify audio features, mostly on a 0–1 scale (see Spotify’s audio features docs) |
duration_ms |
Track length in milliseconds |
Links.
- Data source (TidyTuesday): https://github.com/rfordatascience/tidytuesday/blob/main/data/2020/2020-01-21/readme.md
- Related article: Kaylin Pavlik, “Understanding audio features and exploring genre classification”
- R package used to collect the data:
spotifyr
Loading the data in pandas. Load it directly from TidyTuesday’s raw CSV, no download needed:
import pandas as pd
url = "https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2020/2020-01-21/spotify_songs.csv"
df = pd.read_csv(url)
df.head()