How to use AI tools for fact-checking
The debate over AI tools is still raging in the writing community (and I have a longer post in the works about my own opinions), but I think there are some uses for these tools that sidestep the ethical debates and just help all of us—authors and editors—produce better work. Fact-checking quotations is one of these, and that’s what I’m going to explore in this post, walking you through my experiments and what does and doesn’t seem to work. (I’ll date the post and include any updates at the end, but note that the tools are changing so rapidly, I can’t promise that the specifics are still valid, though the underlying principles won’t change.)
But why bother?
Before we get into the how, let’s take a minute to talk about why copyeditors fact-check quotations—and why authors and readers should care.
Copyeditors are trained to make sure texts they work on are clear, correct, and consistent. Checking the accuracy of author names and titles is standard. Does Brené Brown’s first name have one e or two? Does that e have an accent? What’s the full subtitle of her latest book, Atlas of the Heart? These are standard checks for most copyeditors. As an author, you also want these things to be right, especially when you are citing the work of someone who is a collaborator or an inspiration! These details are also very easy to verify with a quick search.
Checking the details of any direct quotes included in a manuscript takes a bit more work—sometimes a lot more work. This isn’t a big deal if there are just one or two quoted passages in a book. However, if a manuscript is full of chapter epigraphs drawn from other works, as well as quoted passages sprinkled throughout, it’s probably worth discussing up front whether checking those is part of the copyeditor’s job because it will take time to do. An author might prefer to do those checks on their own in exchange for a slightly lower fee—and they may have ready access to all of the sources necessary to verify the accuracy of the quotes.
In my experience, nine times out of ten, a quotation checks out. But it’s that tenth time that keeps me motivated to keep checking because sometimes I’ll discover a quotation that is apocryphal or misidentified or taken out of context. How many readers would know or care? Likely not many. But I don’t want authors I work with to be called out for a mistake if I can help it, and I don’t want inaccurate information to continue circulating.
Here’s an example from my own work. In my book All the Words, I write about a visit to the Dickens Museum in London. Posted on the wall near Dickens’s writing desk is a prominent quotation: “Happiness is a gift and the trick is not to expect it but to delight in it when it comes.” I snapped a photo of it and used it in my blog post about the visit, thinking that it was a wonderful metaphor for the writing process too. All we can do is sit down, again and again, to the blank page and try to get the words out. Sometimes they’ll just be regular old words, but every once in a while you’ll be able to tap into a magical vein of inspiration.
When I sat down to turn that collection of blog posts into a book, I tried to find which of Dickens’s novels this was from, only to discover they aren’t Dickens’s words at all: They are taken from the 2002 film adaptation of Nicholas Nickleby. Despite blogger Sue Brewton’s valiant efforts to stamp out this strain of what she calls the “quote virus”, it continues to proliferate, especially on the kind of quote collection websites that make a fact-checker’s job so difficult.
Screenshot of a couple dozen exceedingly cheesy images with the misattributed Dickens quote.
If you’ve decided that verifying your quotations is important to you or to authors whose work you are editing, how do you go about it? And how can AI tools make the process faster and easier? Let me walk you through what I’ve learned through trial and error.
Start with Google Books
The absolute fastest way to check a quote is to search it on Google Books. Select some of the text, add quotation marks around it if needed, and send it to Google. (I have a dedicated key on my keyboard programmed to send highlighted text into a Google search, which saves me approximately a million keystrokes a year.) Once there, click the “Books” option in the search tools menu. If you are very lucky, Google Books will bring up the quotation in the original source, alongside the book cover, allowing you to verify accuracy and check the title and author spelling at the same time. You can go a step further, if needed, and click the “Preview” button to see the quotation in context to make sure it fits the author’s point.
Disappointingly, Google’s own AI tool, called Bard, does not seem to have access to Google Books, which would have been a real advantage for this kind of search. (However, given that so much of Google Books repository is copyrighted material, it certainly is smart from a lawsuit-avoidance standpoint.)
For older works, try ChatGPT
If Google Books doesn’t immediately deliver what you need, what do you do next? If you are dealing with a well-known quote, trying to hunt down the original source can feel like trying to find a needle in a haystack. For example, what Google Books and plain Google search turned up when I tried to verify the accuracy of the Heinrich Heine quote, “Wherever they burn books, in the end they will also burn human beings” was a long list of books and websites that mention the quote but don’t lead me to its source.
In addition to the fact that the original quote is in German, and thus English translations may be slightly different in their wording, this one is especially difficult to track down because it is part of a striking memorial by the artist Micha Ullman in Berlin at the location where Nazis burned 20,000 books in 1933.
But when I asked ChatGPT to help me find the source of the quotation, it put me on the right track immediately:
The quote you’re referring to is attributed to the German poet and journalist Heinrich Heine. The original quote, in German, is “Dort, wo man Bücher verbrennt, verbrennt man am Ende auch Menschen.” The English translation is “Where they burn books, they will ultimately also burn people.”
This quote originates from Heine’s play “Almansor”, which was written in 1821. The line can be found in the first scene of the second act. The context of the quote is a discussion about the burning of the Quran by the Spanish Inquisition.
Please note that the quote is prophetic in nature. Nearly a century after Heine wrote these words, the Nazi regime in Germany did burn books - including Heine’s - and later committed the Holocaust, leading to the deaths of millions of people.
Amazing, right? If I spoke German (I decidedly do not), I could verify the translation as well. Note, too, that ChatGPT did provide me a slightly different version than the one I searched, which is also impressive.
That said, the problem of AI tools making up facts is well known at this point, and you still must verify. However, adding the title of the original source to my Google search got me to the right place immediately. An author working with this quotation might also be inspired by those details about the context of the quote to add more commentary. As the writer Shlomo Avineri points out in the Jewish Review of Books, it’s remarkable that “in a play aimed at a German, mainly Christian, audience, Heinrich Heine, born to a Jewish family in Düsseldorf, criticizes Christian Spain for the burning of the Qur’an.”
For recent works, try Bing
ChatGPT is based on training data that does not include any information after 2021, so for more recent works, you need to use an AI tool that is connected to the internet. At the time of writing, that means Microsoft’s Bing, which will provide linked sources as footnotes.
For example, here’s what it provided when I asked it to verify the same Heine quote:
The quote "Wherever they burn books, in the end they will also burn human beings" is indeed attributed to Heinrich Heine. The original source of this quote is Heine's play "Almansor", which was first published in 1821 (1). The line is a reference to the burning of the Quran during the Spanish Inquisition in an effort to eradicate the Moors from the Iberian Peninsula, which had been a major center of medieval Islamic culture (2). This quote has been widely cited in modern history, particularly in reference to instances of censorship and destruction of intellectual property (3).
Source: Conversation with Bing, 9/22/2023
(1) Wherever they burn books, in the end will also burn human beings.. https://www.bookbrowse.com/quotes/detail/index.cfm/quote_number/390/wherever-they-burn-books-in-the-end-will-also-burn-human-beings.
(2) The Tale of Two Book Burnings: Heine’s Warning in Context. https://www.ceu.edu/article/2014-03-13/tale-two-book-burnings-heines-warning-context.
(3) ‘The Lorelei’ by Heinrich Heine (1797-1856) - Classical Poets. https://classicalpoets.org/2020/01/25/the-lorelei-by-heinrich-heine/.
When I click on that second citation, it takes me to an article on the Central European University newsroom website (a trustworthy source!), reporting on a lecture by visiting professor Shlomo Avineri—apparently an early version of the article in the Jewish Review of Books I cited above.
I’ve found that for more obscure quotations, Bing will often link to the full text on the Internet Archive, which is helpful if you need to access a digital copy to verify the quotation. (If I need to go so far as to check out an ebook edition, I often will just add a query to the manuscript noting that I’ve been able to verify the author name and book title, and likely the source, and then recommending that the author verify the accuracy of the wording itself.)
How to access and prompt AI tools
If you are an editor and want to learn more about how to use AI tools in your work, I recommend Erin Servais’s excellent course AI For Editors. (Guess what? You can use AI tools to write Microsoft Word macros!)
You can access an older version of ChatGPT for free or subscribe to access the most current at OpenAI.com or Poe.com (which will also give you access to a number of other AI tools). You can access Microsoft’s Bing for free via their Edge web browser.
I use the following prompt: “Please verify the accuracy and provide the original source of the following quotation. The original source is the place where the material was first published.”
I have the prompt saved in TextExpander with a keyboard shortcut, so it takes me only a few taps to add the prompt and paste the copied text into the search box of whatever tool I’m using.
It’s also worth noting that the website Quote Investigator is still an excellent source, especially for quotes or figures (Winston Churchill, notoriously) that are frequently misattributed. But the fine folks there can only investigate so many quotes, and I’m grateful to now have additional tools in my toolbox to carry out this important copyediting task.
Published September 22, 2023
Check out our Resources page for more in-depth articles on writing, revising, polishing, and publishing your novel. Sign up for our weekly newsletter for fresh content!