Background


My sister, Nassim, is a singer/songwriter who lives in Los Angeles. For some reason she values my feedback and sometimes she sends me her songs.
She recently sent me some of her songs and our conversation went like this (some details have been changed because I don’t remember them):

Gol: I like your sad songs about heartbreak and all the hard stuff you’ve gone through much more than the poppy, happy songs.
Nassim: Most people don’t want to hear sad songs–they want happy, upbeat music! That’s what sells.
Gol: Are you sure?
Nassim: Yes.
Gol: Well, that’s an empirical question.
Nassim: What?
Gol: I mean that we can probably look to see if that’s true or not based on some data.
Nassim: …


So, I downloaded a dataset that had the Billboard Hot-100 Songs from 1964 - 2015 and tried to answer our question: Are sad songs or happy songs more popular?

Initial Look


I used the following packages:

library("tidyverse")
library("tidytext")
library("wordcloud")
library("reshape2")


This is a tibble (a quick look) at the dataset that has the lyrics from the Billboard Top 100 Songs from 1965 - 2015.

lyrics
## # A tibble: 5,100 x 6
##     Rank Song            Artist        Year Lyrics                  Source
##    <int> <chr>           <chr>        <int> <chr>                    <int>
##  1     1 wooly bully     sam the sha…  1965 sam the sham miscellan…      3
##  2     2 i cant help my… four tops     1965 sugar pie honey bunch …      1
##  3     3 i cant get no … the rolling…  1965 <NA>                         1
##  4     4 you were on my… we five       1965 when i woke up this mo…      1
##  5     5 youve lost tha… the righteo…  1965 you never close your e…      1
##  6     6 downtown        petula clark  1965 when youre alone and l…      1
##  7     7 help            the beatles   1965 help i need somebody h…      3
##  8     8 cant you hear … hermans her…  1965 carterlewis every time…      5
##  9     9 crying in the … elvis presl…  1965 you saw me crying in t…      1
## 10    10 my girl         the temptat…  1965 ive got sunshine on a …      3
## # ... with 5,090 more rows


I added a decade column and then created a dataset called lyrics.tidy that split all the lyrics into separate words.

lyrics <- lyrics %>%
    mutate(decade =
                ifelse(lyrics$Year %in% 1965:1969, "1960",
                ifelse(lyrics$Year %in% 1970:1979, "1970",
                ifelse(lyrics$Year %in% 1980:1989, "1980",
                ifelse(lyrics$Year %in% 1990:1999, "1990",
                ifelse(lyrics$Year %in% 2000:2009, "2000",
                ifelse(lyrics$Year %in% 2010:2015, "2010",
                "NA"
                )))))))

lyrics.tidy <- lyrics %>%
    na.omit() %>%
    unnest_tokens(word, Lyrics) #splits lyrics into separate words


Now for fun, I visualized the words in a hot pink world cloud :) Not suprisingly, LOVE is a popular concept to write about!

lyrics.tidy %>%
    anti_join(stop_words) %>% # removes stop words (and, the, etc.)
    count(word) %>% # gives a count of how many times that word appears in the dataset
    with(wordcloud(word, n, max.words = 200, color = "hotpink")) # makes a word cloud

Sentiment Analysis


Okay, now let’s see what sort of emotion is present in these lyrics. Using the Bing sentiment word dictionary, I matched each word with it’s sentiment score. A higher score means more positive sentiment and a lower score means more negative sentiment.

get_sentiments("bing")
## # A tibble: 6,788 x 2
##    word        sentiment
##    <chr>       <chr>    
##  1 2-faced     negative 
##  2 2-faces     negative 
##  3 a+          positive 
##  4 abnormal    negative 
##  5 abolish     negative 
##  6 abominable  negative 
##  7 abominably  negative 
##  8 abominate   negative 
##  9 abomination negative 
## 10 abort       negative 
## # ... with 6,778 more rows
lyrics.sentiment.bing <- lyrics.tidy %>%
    inner_join(get_sentiments("bing")) %>%
    count(Song, Year, decade, sentiment) %>%
    spread(sentiment, n, fill = 0) %>%
    mutate(sentiment = positive - negative) 


Now, let’s see how sentiment has changed throughout the decades. On average, popular song lyrics have gotten more negative over time (although I’m not sure what was going on in the 1960’s…)

ggplot(lyrics.sentiment.bing, aes(decade, sentiment, fill = decade)) + 
  stat_summary(fun.y = "mean", geom = "bar")


Here is another plot by decade, but this one shows us the sentiment score for every song in each decade. So, here we can see there is more fluctuation in the later decades and definitely more extreme scores in the 2010’s.

ggplot(lyrics.sentiment.bing, aes(Song, sentiment, fill = decade)) + 
  geom_col(show.legend = FALSE) + 
  facet_wrap(~decade, ncol = 2, scales ="free_x") + 
  theme(axis.text.x = element_blank()) 


I wondered what those really high and low songs from the 2010’s were, so I pulled those out. So, Shake it off by Taylor Swift is the song with the negative spike in the 2010’s graph. And Super Bass by Nicki Minaj is the song with the positive spike 2010’s graph. One thing to consider with this analysis is how many times a word is repeated in a song. The more often a word is repeated, the higher/lower the sentiment score will be. Hmmm…

lyrics.sentiment.bing %>%
  filter(decade == 2010) %>%
  arrange(-abs(sentiment)) %>%
  slice(1:3)
## # A tibble: 3 x 6
##   Song          Year decade negative positive sentiment
##   <chr>        <int> <chr>     <dbl>    <dbl>     <dbl>
## 1 shake it off  2014 2010        123        4      -119
## 2 shake it off  2015 2010        123        4      -119
## 3 super bass    2011 2010          6      108       102


I made another word cloud of positive and negative sentiment words. Again, LOVE is very popular to write about!

lyrics.tidy %>%
    inner_join(get_sentiments("bing")) %>%
    count(word, sentiment, sort = TRUE) %>%
    acast(word ~ sentiment, value.var = "n", fill = 0 ) %>%
    comparison.cloud(colors = c("purple1" , "royalblue"))


It’s easier to visualize the most common positive and negative words in a barchart.

bing.word.counts <- lyrics.tidy %>%
    inner_join(get_sentiments("bing")) %>%
    count(word, sentiment, sort = TRUE) %>%
    ungroup() %>%
    group_by(sentiment) %>%
    top_n(15)  %>%
    ungroup() %>%
    mutate(word = reorder(word, n)) %>%
    ggplot(aes(word, n, fill = sentiment)) + 
    geom_col(show.legend = FALSE) +
    facet_wrap(~sentiment, scales="free_y") +
    labs(y = "Number of times the word is used", x = NULL) + coord_flip()

bing.word.counts


So, I wondered if I could get any more insight by looking at groups of words together. I decided to look at the most common trigrams per decade. I added a condition where I did not include trigrams with repeated words.

lyrics.grams <- lyrics %>%
    mutate_if(is.factor, as.character) %>%
    unnest_tokens(bigram, Lyrics, token ="ngrams" , n = 3) %>%
    separate(bigram, c("word1", "word2", "word3"), sep = " ") %>%
    filter(!word1 %in% stop_words$word) %>%
    filter(!word2 %in% stop_words$word) %>%
    filter(!word3 %in% stop_words$word) %>%
    filter(word1 != word2) %>%
    filter(word2 != word3) %>%
  filter(word1 != word3) %>%
    unite(bigram, word1, word2, word3, sep= " ") %>%
    inner_join(lyrics) %>%
    count(bigram, decade, sort = TRUE) %>%
    group_by(decade) %>%
    top_n(10, n) %>%
    ungroup() %>%
    arrange(decade, n) %>%
    mutate(row = row_number())

lyrics.grams %>%
    ggplot(aes(row, n, fill = factor(decade))) + 
  geom_col(show.legend= FALSE) + facet_wrap(~decade, scales = "free_y") +    
  scale_x_continuous(breaks = lyrics.grams$row, labels = lyrics.grams$bigram) + 
  coord_flip()


Even with the condition of no repeating word within trigrams, we can see words are repeated a lot in different orders. Also, some of the most frequently occuring trigrams belonged to just one song. For example, “force don’t stop” is the fifth most frequently occuring trigram from the 1970’s and is from one Michael Jackson song. So, clearly repeating words are affecting the analysis, but at this point it’s unclear how much of an effect it has on the results. So, a future analysis could identify unique words in each song and rerun the sentiment analysis on only the unique words.

Punchline

There are plenty more analyses that can be done (for example: we could look beyond lyrics and examine the tempo or minor/major chords to determine sad vs. happy), but from this, it seems popular songs tend to use positive/happy lyrics although there has been a bit of a trend towards more sad/negative since the 1970’s.