Natural Language Processing

Executive Summary

In our project, the primary focus was on analyzing word usage within Taylor Swift discussions, exploring topics related to Taylor Swift, and investigating potential relationships between the mood of the post and the attributes of the song. Our exploration of the language and specific vocabulary used in discussions of Taylor Swift reveals patterns in how fans and the public express appreciation and criticism of her work. Through NLP methods, we processed large amounts of text data and generated two comprehensive word clouds. These visual tools encapsulate words and phrases commonly used in Reddit comment sections when discussing Taylor Swift’s music and live performances. This non-technical overview highlights universal terms and concepts associated with her, allowing us to understand her cultural influence. The results indicate a strong appeal of Taylor Swift’s albums and live performances, with exceptionally high discussion volumes. Taylor Swift’s influence is notably manifested in concert engagements, particularly associated with albums such as “Lover” and “Red”. Interestingly, the frequency of discussions about albums on Reddit correlates with the valence of the songs within those albums. However, the number of album discussions on Reddit seems to be independent of the overall popularity of the album on other platforms. This study reveals that the users on different applications have different preferences for songs, or people may be influenced by the happiness of the song when commenting or posting.

Basic Data Text Check

See the code

See the code

The majority of text length falls below 3000 characters, reflecting the text length of Reddit users on the platform to some extent. This observation provides insights into the nature of communication on Reddit as a social platform, as we can see from Figure 1. Our team also applied word tokenization and stopwords remover to extract common words related to Taylor Swift, and the top 30 words are listed below at Table 1. Words like song, post, and album are notable, since these common phrases may hint at what Taylor Swift is popular for.

Figure 1: Text Length Distribution Bar Plot
Table 1: Top 30 Words Related with Taylor Swift
Word Count
0 taylor 240,164
1 post 143,038
2 like 127,640
3 song 106,361
4 think 87,866
5 plea 86,875
6 remov 81,127
7 love 67,554
8 album 61,911
9 peopl 53,512
10 realli 48,580
11 question 46,620
12 know 45,963
13 make 45,732
14 perform 44,590
15 also 42,823
16 moder 42,701
17 stay 41,751
18 swift 41,445
19 need 40,513
20 feel 40,433
21 time 39,707
22 music 38,930
23 action 38,573
24 want 37,326
25 listen 36,434
26 even 36,026
27 thread 35,486
28 modmail 33,655
29 messag 33,320

The External Dataset

See the code

The Spotify data of Taylor Swift was utilized as the external data source for addressing specific topics in this project. The dataset compasses all albums and songs released by Taylor Swift starting from 2006 to 2023. Variables within this data frame include name, album, release_date, track_number, id, uri, acousticness, danceability, energy, instrumentalness, liveness, loudness, speechiness, tempo, valence, popularity, and duration_ms. Refer to the the Kaggle website for detailed information. The data has a dimension of 530 * 17, with no missing values observed. Records younger than the reddit data have been excluded. In the album column, regular expressions were applied to remove the strings batween parentheses and brackets, thus, preserving only the key names.

The following Table 2 is the summary table from Spotify data. For each distinct album, the average of other variables were calculated. The variables like popularity, valence, and energy will be considered to merge with the Reddit dataset, then we can see if there is more information we can figure out.

Table 2: Spotify Summary Table
album acousticness danceability energy instrumentalness liveness loudness speechiness tempo valence popularity duration_ms
0 1989 0.1912 0.6398 0.655 0.0008 0.1835 -7.2389 0.1299 125.837 0.4576 68.1875 220422
1 Fearless 0.2014 0.5649 0.6385 0 0.1568 -5.8897 0.0362 125.237 0.4094 65.9487 246291
2 Fearless Platinum Edition 0.2024 0.5759 0.6011 0 0.1598 -5.7082 0.0319 123.089 0.3801 45.9474 250875
3 Live From Clear Channel Stripped 2008 0.5512 0.5481 0.5989 0 0.1441 -5.067 0.0451 137.639 0.4721 39.375 209469
4 Lover 0.3337 0.6582 0.5452 0.0007 0.1152 -8.0133 0.0991 119.973 0.4814 82.6111 206188
5 Midnights 0.4031 0.6313 0.435 0.0259 0.1537 -10.7038 0.0958 118.857 0.2593 76.3636 206594
6 Red 0.1535 0.606 0.5991 0.0012 0.1274 -7.1596 0.0396 116.358 0.4557 59.75 252181
7 Speak Now 0.2248 0.5517 0.6542 0.0001 0.1694 -4.6875 0.0359 137.401 0.4183 55.5882 280364
8 Speak Now World Tour Live 0.2338 0.4413 0.6502 0 0.7749 -7.1298 0.0409 122.616 0.2902 49 297121
9 Taylor Swift 0.183 0.5453 0.6643 0.0001 0.1608 -4.7317 0.0327 126.054 0.4265 63.1333 213971
10 evermore 0.7947 0.5231 0.4928 0.0219 0.1142 -9.8008 0.0594 119.194 0.4275 73.3438 243440
11 folklore 0.7517 0.553 0.3965 0.0007 0.1187 -10.4592 0.0388 117.071 0.3541 66.1045 237720
12 reputation 0.1385 0.6579 0.5829 0 0.1522 -7.6724 0.0951 127.54 0.2934 82.9333 223020
13 reputation Stadium Tour Surprise Song Playlist 0.1604 0.5964 0.6532 0.0004 0.1444 -5.7775 0.0409 120.421 0.4465 39.1739 242956
Figure 2: Grouped Bar Plot of Scaled Album Count and Popularity
Figure 3: Grouped Bar Plot of Scaled Album Count and Popularity

We merged the spotify dataframe with reddit data to analyze the frequency of album mentions in comparison to the average album popularity on Spotify. The plots at Figure 2 and Figure 3 indicate that albums such as “Red”, “Midnights”, and “1989” are popular on Reddit, because of the relatively higher number of time the albums were mentioned. However, in reality, albums like “Reputation,” “Evermore,” and “Lover” exhibit higher popularity on Spotify. This difference may bacause of variations of the fans between Reddit and Spotify. Alternatively, it could be inferred that Reddit users express more interest in albums like "Red," "Midnight," and "1989," whereas Spotify users do not share the same preference. By identifying users on different platforms and understanding their preferences, targeted recommendations for song that suitable for users on each platform’s can be provided.

Table 3: Album Frequency Table Under Taylor Swift Subreddit
Album Count Popularity Source
0 1989 1,107 68.19 submissions
1 Fearless 1,206 65.95 submissions
2 Lover 1,174 82.61 submissions
3 Midnights 1,855 76.36 submissions
4 Red 1,769 59.75 submissions
5 Speak Now 706 55.59 submissions
6 Speak Now World Tour Live 8 49.0 submissions
7 evermore 730 73.34 submissions
8 folklore 958 66.1 submissions
9 reputation 489 82.93 submissions
10 1989 1,662 68.19 comments
11 Fearless 1,003 65.95 comments
12 Lover 1,451 82.61 comments
13 Midnights 674 76.36 comments
14 Red 1,894 59.75 comments
15 Speak Now 762 55.59 comments
16 evermore 935 73.34 comments
17 folklore 1,082 66.1 comments
18 reputation 667 82.93 comments
Table 4: Album Frequency Table Under Music-related Subreddit
Album Popularity Count in Submissions Count in Comments
0 1989 68.19 8 169
1 Fearless 65.95 20 33
2 Lover 82.61 9 64
3 Midnights 76.36 34 66
4 Red 59.75 37 182
5 Speak Now 55.59 1 24
6 Speak Now World Tour Live 49.0 1 1
7 evermore 73.34 7 18
8 folklore 66.1 9 68
9 reputation 82.93 2 21

It is also worth notice that is the substantial variation in the frequency of Taylor’s albums across different subreddits, see Table 3 and Table 4. In music-related subreddits, the mention is relatively infrequent, with the most common being the “Red” album, referenced nearly 200 times. However, in the Taylor Swift subreddit, this number increases to 1800 times. This indicates the gathering patterns of Taylor fans on Reddit: the majority of them are concentrated in the Taylor Swift subreddit rather than music-related subreddits. Analyzing the popularity of Taylor Swift’s albums across other platforms is crucial for future expansion and increasing influence.

Language and Word Usage in Taylor Swift Discussions

See the code

The goal of this topic is to uncover language trends in Taylor Swift’s discourse on social media. For Reddit comments, we implemented Spark NLP to systematically deconstruct comments mentioning Taylor Swift, using a series of processes including tokenization, normalization, lemmatization, and stopword cleaning. A data set was obtained that was ready for in-depth linguistic analysis.

Figure 4: Wordcould Associated with Taylor Swift and Concert

Figure 5: Wordcould Associated with Taylor Swift and Music

We constructed two main word clouds. The first word cloud Figure 4 stems from a discussion focusing on Taylor Swift’s musical output such as albums and songs, while the second Figure 5 focuses on her live performances. Notably, the word "album" appears frequently, indicating that her record collection generated significant engagement. Meanwhile, the words "tour" and "tickets" crop up frequently in concert-related discussions, highlighting the logistical aspects of her performances that elicit widespread fan engagement.

To ensure that the word cloud was not only informative but also visually compelling, we followed visualization best practices. Word clouds are presented with high contrast and clear boundaries for easy interpretation of results. These details ensure that our analysis is communicated effectively.

Our NLP-based analysis actually provides the basis for future in-depth sentiment analysis or topic trend analysis. As we parsed the data, we refined our goals, transitioning into focused examinations of specific topics that resonated most with our listeners and fans in our community. This methodological adaptability is a strategic decision in response to the evolving narratives in the data.

Reddit User Sentiment and Taylor Swift Album Popularity

See the code

We employed a pipeline with a pretrained model from John Snow Labs for sentiment analysis. The text cleaning procedure included tokenization, normalization, stemming, lemmatization, and the removal of stop words. Finally, a VivekNSentimentModel was applied. This model takes the document and tokenized words as input and outputs the sentiment of the sentence, categorizing it as either positive or negative.

Figure 6: Sentiment on Reddit Over Time (submissions)

Figure 7: Sentiment on Reddit Over Time (comments)

Time-based bar plots Figure 6 and Figure 7 represents the frequency of sentiment over the months was employed to visualize the trends of sentiment within music subreddits. There are always more blue than red, indicating that the atmosphere in music subreddits appears to be more positive than negative. Additionally, the bar plot indicates an increase in submissions and posts during the winter seasons. This observation allows us to infer the forum activity level over the past few years.

Figure 8: Submissions Sentiment and Album Popularity/Energy/Valence
Figure 9: Comments Sentiment and Album Popularity/Energy/Valence

The text data containing sentiment output was merged with album data to analyze potential relationships between music-related variables and sentiment derived from album-related posts. Two bar-line plots Figure 8, Figure 9 were created, one illustrating users’ sentiment on different albums, and the other representing the popularity/energy/valence of the albums.

It appears that energy has minimal influence on the frequency of positive and negative comments. However, albums with a higher valence (around 0.5), such as "Lover" and "Red," exhibit higher positive comment and post counts compared to albums with a lower valence rate, like "Midnights" and "Reputation." Interestingly, songs with lower valence remain popular on Spotify. This suggests that the happiness or style of the album may influence people's expressions in posts or comments, but it might have a limited effect on their choices of what to listen to.