http://www.tpmmuckraker.com/archives/002809.php
http://editthis.info/tpmmuckraker1/ (wiki)
http://www.docstrangelove.com/gonzopedia/index.php/Main_Page (wiki)

by intranets on Tue Mar 20, 2007 at 08:24:31 PM EST
[ Parent ]

There are 93 United States Attorneys stationed throughout the United States, Puerto Rico, the Virgin Islands, Guam, and the Northern Mariana Islands.

the Administration is committed to having a Senate-confirmed United States Attorney for all 94 federal districts.

But..what if there is a 94th?

by susie dow on Tue Mar 20, 2007 at 08:26:52 PM EST
[ Parent ]

Yes, of course, I'd be delighted if the files were hosted here. As far as I'm concerned, anyone who would like to host them should grab them and host them.

by jukeboxgrad on Wed Mar 21, 2007 at 12:15:16 AM EST
[ Parent ]

Feel free to grab the files and host them here. But the text in the diary is helpful because it explains how the 3 links provide the data in different forms. Move that text here, if you like. Whatever it takes to get the material out there so people can use it.

by jukeboxgrad on Wed Mar 21, 2007 at 12:18:33 AM EST
[ Parent ]

"We wanted to: ... provide a much higher percentage of 'correct' character recognition, which you get by retaining a high resolution prior to the text overlay"

I understand that's what you wanted, but it's not what you got. In fact, it's the opposite of what you got. Upon examination I quickly notice that your files have a much lower percentage of 'correct' character recognition (compared with the alternate version). I could give you a very long list of examples (which I derived by using a text-comparison program), but at the moment I'll just point out one very glaring and troublesome example.

Please open your first file (DOJDocsPt1-1070319_epm.pdf), and look at the bottom of the first page. You'll see this page number: DAG000000425. Double-click on it to select it. You'll find that you can't. That's because this text was not recognized as text by your OCR program.

Here's another way you can prove that text wasn't recognized: try searching for it. Type the page number ("DAG000000425") into Adobe Reader's "Find" field. Reader will tell you "no matches were found."

Is this a problem only on the first page? Definitely not. Try searching for this string: ''DAG0000". Notice how many are found: 2. This despite the fact that page numbers beginning "DAG0000" appear on almost every single one of the 49 pages in this particular file.

Is this a problem only in the first file? Definitely not. Try searching for the string ''DAG0000" in all your files (Adobe Reader's "Search" feature allows you to search all pdfs in a given folder). This is how many you'll find: 39. This is how many you should find: approximately 1557. How do I know? Because that's how many you'll find if you do the exact same search in the 28mb consolidated pdf (which I've referred to as "the alternate version").

(Incidentally, the search that finds 1557 items is about 20 times faster than the search which finds 39 items. This is the opposite of what one would expect, but it works this way because for technical reasons that are not hard to imagine, Acrobat Reader can search much more efficiently in one file that's 28mb in size, as compared with a folder of 61 files that have an aggregate size about 7 times greater.)

(Also, the number is 1557 instead of 3000 mostly because the prefix "DAG" is not always used. For example, another common page-number prefix is "OAG.")

In other words, your OCR program ignored roughly 97% of the page numbers (which the program used to create the alternate file did not ignore). Page numbers are very important, for obvious reasons. What else did your program ignore? I don't know, but if I spent more time analyzing your files I could probably prove the answer is a lot.

"We wanted to: ... 'preserve the original documents' with all of the handwritten notes, images, etc."

The alternate pdf also preserves most of that material. In a handful of places (I estimate less than 1% of the pages) where handwriting is not well-preserved in the alternate pdf, I think the original files do a nice job of filling that very small gap.

More importantly, the alternate pdf contains much more accurate text (which is obviously important because these documents are mostly text). This is aside from the fact that it's 86% smaller (and is correspondingly that much quicker to download and move around), and also enormously more manageable by virtue of being one file instead of 61.

Other than that, your approach is obviously preferable.

By the way, here's one indication the alternate pdf is working well: it's been downloaded over 300 times since it was first posted less than a day ago, and no complaints have been posted.

I have some other comments but this is all I have time for at the moment.

by jukeboxgrad on Wed Mar 21, 2007 at 10:16:41 PM EST
[ Parent ]

This brings up another serious problem with your files. Aside from a very high error rate, a major compounding problem is that "the text overlay" is invisible (on many pages it's also badly misaligned, which makes copying text a very awkward procedure). That means it's nearly impossible to detect the presence of "suspect characters," unless you use somewhat exotic techniques. (This is fundamentally different than the situation in the alternate pdf, where errors in the scanning are almost always immediately obvious to the naked eye.)

This is easily understood with an example. Use Adobe Reader to search your folder of ePM pdfs. Search for "protcgt". Yes, "protcgt". Adobe reader will then take you to page 44 of your file DOJDocsPt2-7070319_epm.pdf. Of course, you won't see "protcgt," because it's hidden in an invisible text overlay. What you'll see is a poorly aligned rectangle in the general vicinity of the word "protege." In other words, your OCR program thinks that "protege" is spelled "protcgt" (in a cubicle somewhere is a programmer who enjoys covertly acting out his hostility for the French by keeping French-sounding words out of an OCR dictionary). So you'll never find this instance of the word "protege" by searching, unless you realize that the proper way to spell it is "protcgt." And you'll most likely never realize this, and you'll never know what you're missing, because "protcgt" is invisible.

Here's another example of the exact same phenomenon. Do another search in your folder of 61 pdfs, this time looking for the string "karl row." Adobe Reader will take you to page 5 of your file DOJDocsPt3-7070319_epm.pdf. You'll see that misaligned highlight again, but you won't see "karl row." You'll see "Karl Rove."

Is it possible that someone might want to find all the places where Rove is mentioned? Yes, but that search won't be successful, and you won't know that it wasn't successful, unless you know the right way to misspell "Rove."

Is it possible that "Rove" is sometimes scanned incorrectly in the alternate pdf? Yes, but it's less likely, because the overall error rate is much lower. And if there is such an error, a reader will have at least a fighting chance of being able to spot it, because the error is not hidden in an invisible text overlay.

Another interesting example. If you search in the alternate pdf for the string "sampson," you'll get 1149 instances. If you search in the ePM files, you'll only get 946. What happened to the other 103 "sampsons?" Your program decided to liven things up by secretly using alternate spellings, in those instances. Here's one alternate spelling your program likes a lot: "Sarnpson." Here's how many times your program thought that was the right way to spell "Sampson:" 75.

Does that problem ever crop up in the alternate pdf? Yes: twice.

In a very similar manner, you're missing about 25% of the occurrences of "iglesias" and "chiara." You're missing about 30 instances of "carol lam." You're missing "charlton" about 35 times. Names are an acid test for OCR because the program doesn't get to rely on a dictionary. But obviously names are crucial in a project like this. Your program is getting names wrong roughly 10-25% of the time. This error rate is consistently much lower in the alternate pdf. After trying many names, I can't find a single instance where the count isn't lower in the ePM files.

By the way, I agree that it's a good idea to have "the ability to always read the original scan if there are suspect characters." This is very easily accomplished by opening the original files. Along with the alternative pdf, which is a compilation, I also created a batch of 61 small files that correspond to the original files, specifically as a cross-reference for the purpose of making it easy to quickly locate any original page. I'd be glad to help anyone who wants to understand how to do this (there's also some explanation about this in the diary where all these files are hosted). The steps are pretty simple. My email address is posted at dKos.

by jukeboxgrad on Thu Mar 22, 2007 at 12:02:08 AM EST
[ Parent ]