Credits

Many articles on this site come from a collection that other people made and share under a licence that asks for credit. This page gives it.

American Stories

When the Source section of an article names American Stories, the article comes from this collection:

Melissa Dell, Jacob Carlson, Tom Bryan, Emily Silcock, Abhishek Arora, Zejiang Shen, Luca D'Amico-Wong, Quan Le, Pablo Querubin and Leander Heldring. American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers. 2023. arXiv:2308.12477. DOI 10.57967/hf/0757.

The collection: dell-research-harvard/AmericanStories, on Hugging Face.

Licence: Creative Commons Attribution 4.0 International (CC BY 4.0). The text of the licence.

What we changed

We changed the text of the collection. We use its division of the pages into articles: for each article we publish, a language model reads the article once, headline and text, from the scan of the page. That reading is the text we publish.

The collection’s recognition and a second one, made by the Library of Congress (ALTO), serve as a check: a word that agrees with neither of them, or that the model is not sure of, is marked uncertain, and nothing we write repeats it. The model reads what is printed and completes nothing; a word it cannot read is shown as “[illegible]”. The spelling, the wording, the factual errors and the opinions of the period stay as they were printed.

When the reading from the image cannot be made, the article keeps the collection’s text, corrected as before: a word broken by a hyphen at the end of a line is joined, and figures and names are compared with the second recognition.

The division of the pages into articles is the collection’s work; the reading of the text is ours.