Datasets

Author

Charles Truong

Descriptions of the datasets used in class.

US Billboard Hot 100 lyrics, 1973–2023

Collected by Foramitti et al. (2025) to study long-term trends in stress, negativity, and simplicity in popular music lyrics and how societal crises affect those.

Observations. 266,087 entries, one per song per chart week, from January 6, 1973 to December 30, 2023 (51 years)

Variables.

Name Description
Date Chart week (weekly, 1973-01-06 to 2023-12-30)
Rank Position on that week’s Billboard Hot 100 chart (1–100)
Song Title Song title
Artist Performing artist
Lyrics_Sentiment VADER compound sentiment score of the lyrics, from −1 (most negative) to +1 (most positive)
Stress Frequency of stress-related words in the lyrics, from LIWC

Lyrics_Sentiment and Stress are missing for about 13,600 rows: these correspond to songs excluded from the analysis for being instrumental, having incorrect scraping returns, or having non-English lyrics.

How it was collected. The Billboard Hot 100 chart was scraped weekly with BeautifulSoup; lyrics for each charted song were retrieved via the Lyrics.ovh and Genius APIs, cleaned of metadata/inserts (e.g. “[Chorus]”), and filtered down to English-language lyrics only. Sentiment was scored with the VADER sentiment analysis tool, and stress-related language with the LIWC tool; each song’s score is the average across all its lyrics’ words.

Links.

Loading the data in pandas. Download lyrics_OSF.csv from the OSF repository above, then:

import pandas as pd

df = pd.read_csv("lyrics_OSF.csv", index_col=0, parse_dates=["Date"])
df.head()

Spotify songs

Audio features and metadata for ~32,800 Spotify tracks across 6 playlist genres, compiled by researcher Kaylin Pavlik using Spotify’s API (via the R spotifyr package) to explore genre classification patterns, and distributed through the TidyTuesday project.

Observations. 32,833 rows, one per song per playlist it appears in (28,356 unique tracks; a song can appear in multiple playlists, hence multiple rows). Album release dates range from 1957 to early 2020.

Variables.

Name Description
track_id, track_name, track_artist Track identifier, title, and performing artist
track_popularity Popularity score (0–100)
track_album_id, track_album_name, track_album_release_date Album identifier, title, and release date
playlist_name, playlist_id Playlist title and identifier
playlist_genre, playlist_subgenre Primary genre (edm, latin, pop, r&b, rap, rock) and subgenre
danceability, energy, key, loudness, mode, speechiness, acousticness, instrumentalness, liveness, valence, tempo Spotify audio features, mostly on a 0–1 scale (see Spotify’s audio features docs)
duration_ms Track length in milliseconds

Links.

Loading the data in pandas. Load it directly from TidyTuesday’s raw CSV, no download needed:

import pandas as pd

url = "https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2020/2020-01-21/spotify_songs.csv"
df = pd.read_csv(url)
df.head()

Pokémon

Information on 949 Pokémon (English and Brazilian Portuguese), compiled from the {pokemon} R package by William Amorim and distributed through the TidyTuesday project (2025-04-01 dataset, curated by Frank Hull).

Observations. 949 rows, one per Pokémon.

Variables.

Name Description
id, pokemon, species_id Unique ID, name, and species ID
height, weight Height and weight
base_experience Base experience
type_1, type_2 Primary and secondary type
hp, attack, defense, special_attack, special_defense, speed Battle stats
color_1, color_2, color_f Primary, secondary, and final color
egg_group_1, egg_group_2 Primary and secondary egg group
url_icon, url_image Icon and image URLs (url_icon is missing its leading “https:”)
generation_id Generation ID

Links.

Loading the data in pandas. Load it directly from TidyTuesday’s raw CSV:

import pandas as pd

url = "https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2025/2025-04-01/pokemon_df.csv"
df = pd.read_csv(url)
df.head()

Pixar films

Release, runtime, and critical-reception data for Pixar’s feature films, from the {pixarfilms} R package by Eric Leung, distributed through the TidyTuesday project.

Observations. 27 rows, one per Pixar film (release ratings are missing for the most recent films, not yet scored by all outlets). The upstream data is missing the title for film #27 (Elemental, released 2023-06-16).

Variables.

Name Description
number Order of release
film Film title
release_date Premiere date
run_time Film length in minutes
film_rating MPA film rating
rotten_tomatoes Rotten Tomatoes score (0–100)
metacritic Metacritic score (0–100)
cinema_score CinemaScore letter grade (A–F)
critics_choice Critics’ Choice Movie Awards score (0–100)

Links.

Loading the data in pandas.

import pandas as pd

films = pd.read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2025/2025-03-11/pixar_films.csv")
response = pd.read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2025/2025-03-11/public_response.csv")
df = films.merge(response, on="film", how="left")
df.head()

Languages of Africa

Native-speaker counts by language, language family, and country across the African continent, sourced from Wikipedia’s “Languages of Africa” page and distributed through the TidyTuesday project (2026-01-13 dataset, curated by Robert Muwanga).

Observations. 796 rows, one per language–country pair (502 unique languages, many spoken natively in more than one country, across 51 countries and 17 language families).

Variables.

Name Description
language Name of the language
family Linguistic family (shared ancestry, vocabulary, phonetics, and grammar)
native_speakers Count of native speakers
country Country where the language is spoken natively

Links.

Loading the data in pandas. Load it directly from TidyTuesday’s raw CSV:

import pandas as pd

url = "https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2026/2026-01-13/africa.csv"
df = pd.read_csv(url)
df.head()

Adult census income (numerical features)

A numeric-only subset of the UCI “Adult” census income dataset, prepared by INRIA’s scikit-learn-mooc project for exercises on the basic scikit-learn .fit/.predict/.score API. Each row is a person from the 1994 US Census; the target class is whether their income is above or below $50K/year.

Observations. 39,073 rows for training (adult-census-numeric.csv) plus 9,769 held out for testing (adult-census-numeric-test.csv).

Variables.

Name Description
age Age in years
capital-gain Capital gains reported
capital-loss Capital losses reported
hours-per-week Hours worked per week
class Target: income <=50K or >50K

Links.

Loading the data in pandas. Bundled locally, no download needed (also mirrored in JupyterLite at content/data/):

import pandas as pd

adult_census = pd.read_csv("datasets/adult-census-numeric.csv")
data = adult_census.drop(columns="class")
target = adult_census["class"]

Adult census income (full features)

The same UCI “Adult” census income dataset, this time with its numerical and categorical columns together (workclass, education, marital status, occupation, and so on), for exercises on preprocessing (scaling and encoding). A 899-row sample, drawn from OpenML’s adult dataset (version 2), with the class target attached.

Observations. 899 rows (adult-census.csv).

Variables.

Name Description
age, capital-gain, capital-loss, hours-per-week Numerical features (same as above)
workclass, education, marital-status, occupation, relationship, race, gender, native-country Categorical features
fnlwgt, educational-num Additional numerical columns, not used in the preprocessing exercises
class Target: income <=50K or >50K

Links.

Loading the data in pandas. Bundled locally, no download needed (also mirrored in JupyterLite at content/data/):

import pandas as pd

adult_census = pd.read_csv("datasets/adult-census.csv")
data = adult_census.drop(columns="class")
target = adult_census["class"]

References

Foramitti, Markus, Urs Markus Nater, Claus Lamm, and Maurício Martins. 2025. “Societal Crises Disrupt Long-Term Increases in Stress, Negativity, and Simplicity in US Billboard Song Lyrics from 1973 to 2023.” Scientific Reports 15 (1). https://doi.org/10.1038/s41598-025-28327-5.