In this vignette we show how the quanteda package can be used to replicate the analysis from Matthew Jockers’ book Text Analysis with R for Students of Literature (London: Springer, 2014). Most of the Jockers book consists of loading, transforming, and analyzing quantities derived from text and data from text. Because quanteda has built in most of the code to perform these data transformations and analyses, it makes it possible to replicate the results from the book with far less code. Throughout this vignette, we name objects based on Jockers’ book, but follow the quanteda style guide.

In what follows, each section corresponds to the respective chapter in the book.

1 R Basics

Our closest equivalent is simply:

install.packages("quanteda")
install.packages("readtext")

But if you are reading this vignette, than chances are that you have already completed this step.

2 First Foray

2.1 Loading the first text file

library(quanteda)

We can load the text from Moby Dick using the readtext package, directly from the Project Gutenberg website.

library(readtext)
data_char_mobydick <- texts(readtext("http://www.gutenberg.org/cache/epub/2701/pg2701.txt"))
names(data_char_mobydick) <- "mobydick"

The readtext() function from the readtext package loads the text files into a data.frame object. We can access the text from a data.frame object (and also, as we will see, a corpus class object), using the texts() or the View() method. Here we will display just the first 75 characters, to prevent a massive dump of the text of the entire novel. We do this using the stri_sub() function from the stringi package, which shows the 1st through the 75th characters of the texts of our new object data_char_mobydick. Because we have not assigned the return from this command to any object, it invokes a print method for character objects, and is displayed on the screen.

library(stringi)
stri_sub(data_char_mobydick, 1, 75)
## [1] "The Project Gutenberg EBook of Moby Dick; or The Whale, by Herman Melville\n"

2.2 Separate content from metadata

The Gutenburg edition of the text contains some metadata before and after the text of the novel. The code below uses the regexec and substring functions to separate this from the text.

# extract the header information
(start_v <- stri_locate_first_fixed(data_char_mobydick, "CHAPTER 1. Loomings.")[1])
## [1] 23018
(end_v <- stri_locate_last_fixed(data_char_mobydick, "orphan.")[1])
## [1] 1216393

Here, we found the character index of the beginning and end of the novel, rather than counting the lines as in the book, but the result will be very similar. If we want to verify that “orpan.” is the end of the novel, we can use the kwic() function:

# verify that "orphan" is the end of the novel
kwic(data_char_mobydick, "orphan")
##                                                            
##  [mobydick, 255352] children, only found another | orphan |
##                              
##  . End of Project Gutenberg's

If we want to count the number of lines, we can do so by counting the newlines in the text.

stri_count_fixed(data_char_mobydick, "\n")
## [1] 22107

To measure just the number lines in the novel itself, without the metadata, we can subset the text from the start and end of the novel part, as identified above.

stri_sub(data_char_mobydick, from = start_v, to = end_v) %>%
    stri_count_fixed("\n")
## [1] 21206

To trim the non-book content, we use stri_sub() to extract the text between the beginning and ending indexes found above:

novel_v <- stri_sub(data_char_mobydick, start_v, end_v)
length(novel_v)
## [1] 1
stri_sub(novel_v, 1, 94) %>% cat()
## CHAPTER 1. Loomings.
## 
## 
## Call me Ishmael. Some years ago--never mind how long precisely--having

2.3 Reprocessing the content

We begin processing the text by converting to lower case. quanteda’s *_tolower() functions work like the built-in tolower(), with an extra option to preserve upper-case acronyms when detected. For character objects, we use char_tolower():

# lowercase text
novel_lower_v <- char_tolower(novel_v)

quanteda’s tokens() function splits the text into words, with many options available for which characters should be preserved, and which should be used to define word boundaries. The default behaviour works similarly to splitting on the regular expression for non-word characters (\W as in the book), but it much smarter. For instance, it does not treat apostrophes as word boundaries, meaning that 's and 't are not treated as whole words from possessive forms and contractions.

We will skip straight to getting the character vector of tokens:

moby_word_v <- tokens(novel_lower_v, remove_punct = TRUE) %>% as.character()
(total_length <- length(moby_word_v))
## [1] 210000
moby_word_v[1:10]
##  [1] "chapter"  "1"        "loomings" "call"     "me"       "ishmael" 
##  [7] "some"     "years"    "ago"      "never"
moby_word_v[99986] 
## [1] "in"
moby_word_v[c(4,5,6)]
## [1] "call"    "me"      "ishmael"
# check positions of "whale"
which(moby_word_v == "whale") %>% head()
## [1] 2030 2060 2203 2415 4048 4211

2.4 Beginning the analysis

The code below uses the tokenized text to the occurrence of the word whale. To include the possessive form whale’s, we may sum the counts of both forms, count the keyword-in-context matches by regular expression or glob. A glob is a simple wildcard matching pattern common on Unix systems – asterisks match zero or more characters.

Note that the counts below do not match those in the book, due to differences in how the book has split on any non-word character, while quanteda’s tokenizer splits on a more comprehensive set of “word boundaries”. quanteda’s tokens() function by default does not remove punctuation or numbers (both defined as “non-word” characters) by default. To more closely match the counts in the book, we have removed punctuation.

length(moby_word_v[which(moby_word_v == "whale")])
## [1] 908
# total occurrences of 'whale' including possessive
length(moby_word_v[which(moby_word_v == "whale")]) + length(moby_word_v[which(moby_word_v == "whale's")])
## [1] 1028
# same thing using kwic()
nrow(kwic(novel_lower_v, "whale"))
## [1] 908
nrow(kwic(novel_lower_v, "whale*")) # includes words like "whalemen"
## [1] 1565
(total_whale_hits <- nrow(kwic(novel_lower_v, pattern = "^whale('s){0,1}$", valuetype = "regex")))
## [1] 1028

What fraction of the total words (excluding punctuation) in the novel are “whale”?

total_whale_hits / ntoken(novel_lower_v, remove_punct = TRUE)  
##       text1 
## 0.004895238

With ntype() we can calculate the size of the vocabulary – includes possessive forms, but excludes punctuation, symbols and numbers.

# total unique words
length(unique(moby_word_v))
## [1] 18524
ntype(char_tolower(novel_v), remove_punct = TRUE)
## text1 
## 18524

To quickly sort the word types by their frequency, we can use the dfm() command to create a matrix of counts of each word type – a document-frequency matrix. In this case there is only one document, the entire book.

# ten most frequent words
moby_dfm <- dfm(novel_lower_v, remove_punct = TRUE)
head(moby_dfm, nfeature = 10)
## Warning in head.dfm(moby_dfm, nfeature = 10): Argument nfeature not used.
## Document-feature matrix of: 1 document, 18,524 features (0% sparse).

Getting the list of the most frequent 10 terms is easy, using textstat_frequency().

textstat_frequency(moby_dfm, n = 10) 
##    feature frequency rank docfreq
## 1      the     14173    1       1
## 2       of      6446    2       1
## 3      and      6311    3       1
## 4        a      4605    4       1
## 5       to      4511    5       1
## 6       in      4071    6       1
## 7     that      2950    7       1
## 8      his      2495    8       1
## 9       it      2394    9       1
## 10       i      1976   10       1

Finally, if we wish to plot the most frequent (50) terms, we can supply the results of textstat_frequency() to ggplot() to plot their frequency by their rank:

# plot frequency of 50 most frequent terms 
library(ggplot2)
theme_set(theme_minimal())
textstat_frequency(moby_dfm, n = 50) %>% 
  ggplot(aes(x = rank, y = frequency)) +
  geom_point() +
  labs(x = "Frequency rank", y = "Term frequency")

For direct comparison with the next chapter, we also create the sorted list of the most frequently found words using this:

sorted_moby_freqs_t <- topfeatures(moby_dfm, n = nfeature(moby_dfm))

3 Accessing and Comparing Word Frequency Data

3.1 Accessing Word Data

We can query the document-frequency matrix to retrieve word frequencies, as with a normal matrix:

# frequencies of 'he' and 'she' - these are matrixes, not numerics
sorted_moby_freqs_t[c("he", "she", "him", "her")]
##   he  she  him  her 
## 1758  112 1058  330
# another method: indexing the dfm
moby_dfm[, c("he", "she", "him", "her")]
## Document-feature matrix of: 1 document, 4 features (0% sparse).
## 1 x 4 sparse Matrix of class "dfm"
##        features
## docs      he she  him her
##   text1 1758 112 1058 330
sorted_moby_freqs_t[1]
##   the 
## 14173
sorted_moby_freqs_t["the"]
##   the 
## 14173
# term frequency ratios
sorted_moby_freqs_t["him"] / sorted_moby_freqs_t["her"]
##      him 
## 3.206061
sorted_moby_freqs_t["he"] / sorted_moby_freqs_t["she"]
##       he 
## 15.69643

Total number of tokens:

ntoken(moby_dfm)
##  text1 
## 210000
sum(sorted_moby_freqs_t)
## [1] 210000

3.2 Recycling

Relative term frequencies:

sorted_moby_rel_freqs_t <- sorted_moby_freqs_t / sum(sorted_moby_freqs_t) * 100
sorted_moby_rel_freqs_t["the"]
##      the 
## 6.749048
# by weighting the dfm directly
moby_dfm_pct <- dfm_weight(moby_dfm, type = "relfreq") * 100
moby_dfm_pct[, "the"]
## Document-feature matrix of: 1 document, 1 feature (0% sparse).
## 1 x 1 sparse Matrix of class "dfm"
##        features
## docs         the
##   text1 6.749048

Plotting the most frequent terms, replicating the plot from the book:

plot(sorted_moby_rel_freqs_t[1:10], type = "b",
     xlab = "Top Ten Words", ylab = "Percentage of Full Text", xaxt = "n")
axis(1,1:10, labels = names(sorted_moby_rel_freqs_t[1:10]))

Plotting the most frequent terms using ggplot2:

textstat_frequency(moby_dfm_pct, n = 10) %>% 
  ggplot(aes(x = reorder(feature, -rank), y = frequency)) +
  geom_bar(stat = "identity") + coord_flip() + 
  labs(x = "", y = "Term Frequency as a Percentage")

4 Token Distribution Analysis

4.1 Dispersion plots

A dispersion plot allows us to visualize the occurrences of particular terms throughout the text. The object returned by the kwic function can be plotted to display a dispersion plot. The quanteda textplot_ objects are based on ggplot2, so you can easily change the plot, for example by adding custom title.

# using words from tokenized corpus for dispersion
textplot_xray(kwic(novel_v, "whale")) + ggtitle("Lexical dispersion")

To produce multiple dispersion plots for comparison, you can simply send more than one kwic() output to textplot_xray():

textplot_xray(
     kwic(novel_v, "whale"),
     kwic(novel_v, "Ahab")
) + ggtitle("Lexical dispersion")

4.2 Searching with regular expression

# identify the chapter break locations
chap_positions_v <- kwic(novel_v, phrase(c("CHAPTER \\d")), valuetype = "regex")$from
head(chap_positions_v)
## [1]     1  2581  4287 11236 13169 14043

4.2 Identifying chapter breaks

Splitting the text into chapters means that we will have a collection of documents, which makes this a good time to make a corpus object to hold the texts. Initially, we make a single-document corpus, and then use the char_segment() function to split this by the string which specifies the chapter breaks.

chapters_corp <- 
    corpus(data_char_mobydick) %>%
    corpus_segment(pattern = "CHAPTER\\s\\d+.*\\n", valuetype = "regex")
summary(chapters_corp, 10)
## Corpus consisting of 136 documents, showing 10 documents:
## 
##         Text Types Tokens Sentences                       pattern
##   mobydick.1   917   2575       101        CHAPTER 1. Loomings.\n
##   mobydick.2   651   1700        60  CHAPTER 2. The Carpet-Bag.\n
##   mobydick.3  1744   6943       264 CHAPTER 3. The Spouter-Inn.\n
##   mobydick.4   681   1927        54 CHAPTER 4. The Counterpane.\n
##   mobydick.5   405    869        29       CHAPTER 5. Breakfast.\n
##   mobydick.6   466    942        44      CHAPTER 6. The Street.\n
##   mobydick.7   519   1082        40      CHAPTER 7. The Chapel.\n
##   mobydick.8   478   1080        29      CHAPTER 8. The Pulpit.\n
##   mobydick.9  1260   4284       166      CHAPTER 9. The Sermon.\n
##  mobydick.10   658   1804        66 CHAPTER 10. A Bosom Friend.\n
## 
## Source:  /Users/kbenoit/tmp/quanteda/docs/articles/pkgdown/replication/* on x86_64 by kbenoit
## Created: Fri Dec  8 16:38:41 2017
## Notes:   corpus_segment.corpus(., pattern = "CHAPTER\\s\\d+.*\\n", valuetype = "regex")

The titles are automatically extracted into the pattern document variables, and the text of each chapter becomes the text of each new document unit. To tidy this up, we can remove the trailing \n character, using stri_trim_right(), since the \n is a member of the “whitespace” group.

docvars(chapters_corp, "pattern") <- stringi::stri_trim_right(docvars(chapters_corp, "pattern"))
summary(chapters_corp, n = 3)
## Corpus consisting of 136 documents, showing 3 documents:
## 
##        Text Types Tokens Sentences                     pattern
##  mobydick.1   917   2575       101        CHAPTER 1. Loomings.
##  mobydick.2   651   1700        60  CHAPTER 2. The Carpet-Bag.
##  mobydick.3  1744   6943       264 CHAPTER 3. The Spouter-Inn.
## 
## Source:  /Users/kbenoit/tmp/quanteda/docs/articles/pkgdown/replication/* on x86_64 by kbenoit
## Created: Fri Dec  8 16:38:41 2017
## Notes:   corpus_segment.corpus(., pattern = "CHAPTER\\s\\d+.*\\n", valuetype = "regex")

For better reference, let’s also rename the document labels with these chapter headings:

docnames(chapters_corp) <- docvars(chapters_corp, "pattern")

4.4.5 barplots of whale and ahab

With the corpus split into chapters, we can use the dfm() function to create a matrix of counts of each word in each chapter – a document-frequency matrix.

# create a dfm
chap_dfm <- dfm(chapters_corp)

# extract row with count for "whale"/"ahab" in each chapter
# and convert to data frame for plotting
whales_ahabs_df <- as.data.frame(chap_dfm[, c('whale', 'ahab')])
    
whales_ahabs_df$chapter <- 1:nrow(whales_ahabs_df)

ggplot(data = whales_ahabs_df, aes(x = chapter, y = whale)) + 
    geom_bar(stat = "identity") +
    labs(x = "Chapter", 
         y = "Frequency",
         title = "Occurrence of 'whale'")

ggplot(data = whales_ahabs_df, aes(x = chapter, y = ahab)) + 
    geom_bar(stat = "identity") +
        labs(x = "Chapter", 
         y = "Frequency",
         title = "Occurrence of 'ahab'")

The above plots are raw frequency plots. For relative frequency plots, (word count divided by the length of the chapter) we can weight the document-frequency matrix. To obtain expected word frequency per 100 words, we multiply by 100. To get a feel for what the resulting weighted dfm (document-feature matrix) looks like, you can inspect it with the head function, which prints the first few rows and columns.

rel_dfm <- dfm_weight(chap_dfm, type = 'relfreq') * 100
head(rel_dfm)
## Document-feature matrix of: 6 documents, 18,629 features (95.9% sparse).
# subset dfm and convert to data.frame object
rel_chap_freq <- as.data.frame(rel_dfm[, c('whale', 'ahab')])
rel_chap_freq$chapter <- 1:nrow(rel_chap_freq)
ggplot(data = rel_chap_freq, aes(x = chapter, y = whale)) + 
    geom_bar(stat = "identity") +
    labs(x = "Chapter", y = "Relative frequency",
         title = "Occurrence of 'whale'")

ggplot(data = rel_chap_freq, aes(x = chapter, y = ahab)) + 
    geom_bar(stat = "identity") +
    labs(x = "Chapter", y = "Relative frequency",
         title = "Occurrence of 'ahab'")

5 Correlation

5.2 Correlation Analysis

Correlation analysis (and many other similarity measures) can be constructed using fast, sparse means through the textstat_simil() function. Here, we select feature comparisons for just “whale” and “ahab”, and convert this into a matrix as in the book. Because correlations are sensitive to document length, we first convert this into a relative frequency using dfm_weight().

dfm_weight(chap_dfm, type = "relfreq") %>% 
    textstat_simil(selection = c("whale", "ahab"), method = "correlation", margin = "features") %>%
    as.matrix() %>%
    head(2)
##            whale       ahab
## whale  1.0000000 -0.2495543
## ahab  -0.2495543  1.0000000

With the ahab frequency and whale frequency vectors extracted from the dfm, it is easy to calculate the significance of the correlation.

5.4 Testing Correlation with Randomization

cor_data_df <- dfm_weight(chap_dfm, type = "relfreq")[, c("ahab", "whale")] %>%
    as.data.frame

# sample 1000 replicates and create data frame
n <- 1000
samples <- data.frame(
    cor_sample = replicate(n, cor(sample(cor_data_df$whale), cor_data_df$ahab)),
    id_sample = 1:n
)

# plot distribution of resampled correlations
ggplot(data = samples, aes(x = cor_sample, y = ..density..)) +
    geom_histogram(colour = "black", binwidth = 0.01) +
    geom_density(colour = "red") +
    labs(x = "Correlation Coefficient", y = NULL,
         title = "Histogram of Random Correlation Coefficients with Normal Curve")

6 Measures of Lexical Variety

6.2 Mean word frequency

# length of the book in chapters
ndoc(chapters_corp)
## [1] 136
# chapter names
docnames(chapters_corp) %>% head()
## [1] "CHAPTER 1. Loomings."        "CHAPTER 2. The Carpet-Bag." 
## [3] "CHAPTER 3. The Spouter-Inn." "CHAPTER 4. The Counterpane."
## [5] "CHAPTER 5. Breakfast."       "CHAPTER 6. The Street."

Calculating the mean word frequencies is easy:

# for first few chapters
ntoken(chapters_corp) %>% head()
##        CHAPTER 1. Loomings.  CHAPTER 2. The Carpet-Bag. 
##                        2575                        1700 
## CHAPTER 3. The Spouter-Inn. CHAPTER 4. The Counterpane. 
##                        6943                        1927 
##       CHAPTER 5. Breakfast.      CHAPTER 6. The Street. 
##                         869                         942
# average
(ntoken(chapters_corp) / ntype(chapters_corp)) %>% head()
##        CHAPTER 1. Loomings.  CHAPTER 2. The Carpet-Bag. 
##                    2.808070                    2.611367 
## CHAPTER 3. The Spouter-Inn. CHAPTER 4. The Counterpane. 
##                    3.981078                    2.829662 
##       CHAPTER 5. Breakfast.      CHAPTER 6. The Street. 
##                    2.145679                    2.021459

6.3 Extracting Word Usage Means

Since the quotient of the number of tokens and number of types is a vector, we can simply feed this to plot() using the pipe operator:

(ntoken(chapters_corp) / ntype(chapters_corp)) %>%
    plot(type = "h", ylab = "Mean word frequency")

For the scaled plot:

(ntoken(chapters_corp) / ntype(chapters_corp)) %>%
    scale() %>%
    plot(type = "h", ylab = "Scaled mean word frequency")

6.4 Ranking the values

mean_word_use_m <- (ntoken(chapters_corp) / ntype(chapters_corp))
sort(mean_word_use_m, decreasing = TRUE) %>% head()
## CHAPTER 135. The Chase.--Third Day.    CHAPTER 54. The Town-Ho's Story. 
##                            4.169544                            4.150151 
##               CHAPTER 16. The Ship.         CHAPTER 3. The Spouter-Inn. 
##                            4.051861                            3.981078 
##               CHAPTER 32. Cetology.         CHAPTER 64. Stubb's Supper. 
##                            3.542857                            3.468721

6.5 Calculating the TTR

Measures of lexical diversity can be estimated using textstat_lexdiv(). The TTR (Type-Token Ratio), a measure used in section 6.5, can be calculcated for each document of the dfm.

dfm(chapters_corp) %>% 
    textstat_lexdiv(measure = "TTR") %>%
    head(n = 10)
##        CHAPTER 1. Loomings.  CHAPTER 2. The Carpet-Bag. 
##                   0.3332039                   0.3688235 
## CHAPTER 3. The Spouter-Inn. CHAPTER 4. The Counterpane. 
##                   0.2376494                   0.3435392 
##       CHAPTER 5. Breakfast.      CHAPTER 6. The Street. 
##                   0.4453395                   0.4702760 
##      CHAPTER 7. The Chapel.      CHAPTER 8. The Pulpit. 
##                   0.4528651                   0.4314815 
##      CHAPTER 9. The Sermon. CHAPTER 10. A Bosom Friend. 
##                   0.2789449                   0.3481153

7 Hapax Richness

Another measure of lexical diversity is Hapax richness, defined as the number of words that occur only once divided by the total number of words. We can calculate Hapax richness very simply by using a logical operation on the document-feature matrix, to return a logical value for each term that occurs once, and then sum these to get a count.

# hapaxes per document
rowSums(chap_dfm == 1) %>% head()
##        CHAPTER 1. Loomings.  CHAPTER 2. The Carpet-Bag. 
##                         609                         437 
## CHAPTER 3. The Spouter-Inn. CHAPTER 4. The Counterpane. 
##                        1084                         464 
##       CHAPTER 5. Breakfast.      CHAPTER 6. The Street. 
##                         276                         343
# as a proportion
hapax_proportion <- rowSums(chap_dfm == 1) / ntoken(chap_dfm)
head(hapax_proportion)
##        CHAPTER 1. Loomings.  CHAPTER 2. The Carpet-Bag. 
##                   0.2365049                   0.2570588 
## CHAPTER 3. The Spouter-Inn. CHAPTER 4. The Counterpane. 
##                   0.1561285                   0.2407888 
##       CHAPTER 5. Breakfast.      CHAPTER 6. The Street. 
##                   0.3176064                   0.3641189

To plot this:

barplot(hapax_proportion, beside = TRUE, col = "grey", names.arg = seq_len(ndoc(chap_dfm)))

8 Do it KWIC

For this, and the next chapter, we simply use quanteda’s excellent kwic() function. To find the indexes of the token positions for “gutenberg”, for instance, we use the following, which returns a data.frame with the name from indicating the index position of the start of the token match:

gutenberg_kwic <- kwic(data_char_mobydick, pattern = "gutenberg")
head(gutenberg_kwic$from, 10)
##  [1]      3     52    108 255375 255538 255650 255697 255891 256007 256105

9 Do it KWIC (Better)

This is going to be super easy since we don’t need to reinvent the wheel here, since kwic() already does all that we need.

Let’s create a corpus containing Moby Dick but also Jane Austen’s Sense and Sensibility.

data_char_senseandsensibility <- texts(readtext("http://www.gutenberg.org/files/161/161.txt"))
litcorpus <- corpus(c(data_char_mobydick, austen = data_char_senseandsensibility))

Now we can use kwic() to find out where in each novel this occurred:

(dogkwic <- kwic(litcorpus, "dog"))
##                                                                 
##        [mobydick, 17213]    all over like a Newfoundland | dog |
##        [mobydick, 31814]        was seen swimming like a | dog |
##        [mobydick, 59904]                       .-- Down, | dog |
##        [mobydick, 60005]          not tamely be called a | dog |
##        [mobydick, 60528]             didn't he call me a | dog |
##        [mobydick, 86935]   sacrifice of the sacred White | Dog |
##       [mobydick, 125588]            life that lives in a | dog |
##       [mobydick, 125684]   the sagacious kindness of the | dog |
##       [mobydick, 159443] " The ungracious and ungrateful | dog |
##       [mobydick, 159485]           Give way, greyhounds! | Dog |
##       [mobydick, 170812]             to the whale that a | dog |
##       [mobydick, 194020]             the Ram-- lecherous | dog |
##       [mobydick, 197596]                  .( Bunger, you | dog |
##       [mobydick, 198082]              die in pickle, you | dog |
##       [mobydick, 198711]                Ahab, and like a | dog |
##       [mobydick, 240998]       air as a sagacious ship's | dog |
##  [austen.161.txt, 78321]        fellow! such a deceitful | dog |
##                            
##  just from the water,      
##  , throwing his long arms  
##  , and kennel!"            
##  , sir.""                  
##  ? blazes! he called       
##  was by far the holiest    
##  or a horse. Indeed        
##  ? The accursed shark alone
##  !" cried Starbuck;        
##  to it!""                  
##  does to the elephant;     
##  , he begets us;           
##  , laugh out! why          
##  ; you should be preserved 
##  , strangely snuffing;"    
##  will, in drawing nigh     
##  ! It was only the

We can plot this easily too, as a lexical dispersion plot. By specifying the scale as “absolute”, we are looking at absolute token index position rather than relative position, and therefore we see that Moby Dick is nearly twice as long as Sense and Sensibility.

textplot_xray(dogkwic, scale = "absolute")

10 Text Quality, Text Variety, and Parsing XML

11 Clustering

Chapter 11 describes how to detect clusters in a corpus. While the book uses the XMLAuthorCorpus, we describe clustering using U.S. State of the Union addresses included in the quantedaData package. We trim the corpus with dfm_trim() by keeping only those words that occur at least five times in the corpus and in at least three speeches.

library(quantedaData)
pres_dfm <- dfm(corpus_subset(data_corpus_SOTU, Date > "1980-01-01"), 
               stem = TRUE, remove_punct = TRUE,
               remove = stopwords("english")) %>% 
    dfm_trim(min_count = 5, min_docfreq = 3)

# hierarchical clustering - get distances on normalized dfm
pres_dist_mat <- textstat_dist(dfm_weight(pres_dfm, "relfreq"))

# hiarchical clustering the distance object
pres_cluster <- hclust(pres_dist_mat)

# label with document names
pres_cluster$labels <- docnames(pres_dfm)

# plot as a dendrogram
plot(pres_cluster, xlab = "", sub = "", main = "Euclidean Distance on Normalized Token Frequency")

12 Classification

13 Topic Modeling

Finally, Jockers’ book introduces topic modeling of a corpus and the visualisation through wordclouds. In our example, we use the State of the Union addresses again. We can easily apply functions from the topicmodels package by using quanteda’s convert() function. In our example, we use the Irish budget speeches from 2010 and classify 20 topics using Latent Dirichlet Allocation.

dfm_speeches <- dfm(data_corpus_irishbudget2010, 
                remove_punct = TRUE, remove_numbers = TRUE, remove = stopwords("english")) %>% 
    dfm_trim(min_count = 4, max_docfreq = 10)

library(topicmodels)
LDA_fit_20 <- convert(dfm_speeches, to = "topicmodels") %>% 
    LDA(k = 20)

# get top five terms per topic
get_terms(LDA_fit_20, 5)
##      Topic 1     Topic 2       Topic 3  Topic 4        Topic 5  
## [1,] "welfare"   "alternative" "fianna" "system"       "benefit"
## [2,] "child"     "citizenship" "fáil"   "welfare"      "child"  
## [3,] "education" "wealth"      "side"   "government's" "day"    
## [4,] "taxation"  "adjustment"  "level"  "taken"        "society"
## [5,] "fianna"    "breaks"      "third"  "fáil"         "fianna" 
##      Topic 6        Topic 7    Topic 8      Topic 9     Topic 10    
## [1,] "reduction"    "care"     "million"    "failed"    "million"   
## [2,] "spending"     "system"   "welfare"    "strategy"  "system"    
## [3,] "per"          "fáil"     "support"    "needed"    "increases" 
## [4,] "contribution" "families" "back"       "ministers" "european"  
## [5,] "level"        "hit"      "employment" "system"    "investment"
##      Topic 11    Topic 12   Topic 13    Topic 14  Topic 15  
## [1,] "welfare"   "measures" "taoiseach" "context" "fianna"  
## [2,] "per"       "scheme"   "employees" "stated"  "fáil"    
## [3,] "million"   "pension"  "rate"      "million" "national"
## [4,] "allowance" "spending" "referred"  "lower"   "irish"   
## [5,] "benefit"   "welfare"  "debate"    "esri"    "support" 
##      Topic 16       Topic 17    Topic 18     Topic 19  Topic 20  
## [1,] "society"      "taoiseach" "investment" "levy"    "welfare" 
## [2,] "enterprising" "fine"      "support"    "million" "fairness"
## [3,] "create"       "gael"      "measures"   "carbon"  "child"   
## [4,] "kind"         "may"       "hospital"   "change"  "less"    
## [5,] "sense"        "irish"     "providing"  "welfare" "national"