Digital text mining helps scholars analyze themes in nearly 1,400 Civil War-era newspapers

November 9, 2012

News reports from the Civil War contain an abundance of themes and text. Due to the enormity of the coverage during the four years of the war, it is nearly impossible for individual historians to analyze – until now.

Changing this is “Mining the Dispatch,” a project developed at the University of Richmond's Digital Scholarship Lab (DSL). Mining applies computational techniques to the study of Civil War news, giving historians new ways to see both broad and subtle patterns in the way the war was reported.

Developed by Robert K. Nelson, DSL director, the project uses a text-mining technique called topic modeling to analyze themes that dominated the nearly 1,400 issues of the Richmond Daily Dispatch from 1861–65. 

“Topic modeling is a probabilistic, statistical method that can uncover themes and categories in amounts of text so large that they cannot be read by any individual human being,” said Nelson.

Mining visualizes the ups-and-downs of war reportage, court proceedings, fugitive slave ads and even humor amid the 24 million words written in the paper during the war. “Applied to the Dispatch for the entirety of the war, topic modeling enables us to see both broad and subtle patterns in the Civil War news that we would otherwise be unable to detect, said Nelson.

Nelson introduced the project during the 2010 Society of Civil War Historians conference. “People were intrigued about a project that uses algorithmic text analysis to make an interpretive argument,” said Nelson. 

Nelson continues to use the mining process in his current research analyzing the multiple functions of the nationalistic and patriotic rhetoric that appeared in Civil War-era newspapers, including arguments calling for men to die and kill for their country and to vote for particular political parties.

Mining has received positive reviews in the Journal of American History and the popular magazine, Civil War Times. Nelson has lectured about the process and his research at several universities and colleges, including Emory, University of Virginia, University of Wyoming and the University of Maryland’s Institute for Technology in the Humanities. He also has presented the technique at meetings of the Society for the Study of Southern Literature, American Historical Association and Society of Civil War Historians, as well as the Digital Humanities’ 2012 conference in Hamburg, Germany.

For more information about the project, see dsl.richmond.edu/dispatch/pages/home.

# # #


Director, Digital Scholarship Lab