{"id":87,"date":"2013-12-21T05:46:49","date_gmt":"2013-12-21T05:46:49","guid":{"rendered":"http:\/\/sonny..ogi.edu\/~kgorman\/blog\/?p=87"},"modified":"2013-12-21T05:46:49","modified_gmt":"2013-12-21T05:46:49","slug":"gigaword-english-preprocessing","status":"publish","type":"post","link":"https:\/\/www.wellformedness.com\/blog\/gigaword-english-preprocessing\/","title":{"rendered":"Gigaword English preprocessing"},"content":{"rendered":"

I recently took a little time out to coerce a recent version of the LDC’s Gigaword English corpus<\/a>\u00a0into a format that could be used for training conventional\u00a0n<\/em>-gram models. This turned out to be harder than I expected.<\/p>\n

Decompression<\/h2>\n

Gigaword English (v. 5) ships with 7 directories of gzipped SGML data, one directory for each of the news sources. The first step is, obviously enough, to decompress these files, which can be done with gunzip<\/code>.<\/p>\n

SGML to XML<\/h2>\n

The resulting files are, alas, not XML files, which for all their verbosity can be parsed in numerous elegant ways. In particular, the decompressed Gigaword files do not contain a root node: each story is inside of\u00a0<DOC><\/code>\u00a0tags at the top level of the hierarchy. While this might be addressed by simply adding in a top-level tag, the files also contain a few \u00a0“entities” (e.g., &amp;<\/code>) which ideally should be replaced by their actual referent.\u00a0Simply inserting the Gigaword Document Type Definition<\/a>, or DTD, at the start of each SGML file was sufficient to convert the Gigaword files to valid SGML.<\/p>\n

I also struggled to find software for SGML-to-XML conversion; is this not something other people regularly want to do? I\u00a0ultimately used an ancient library called OpenSP (open-sp<\/code> in Homebrew), in particular the command osx<\/code>. This conversion throws a small number of errors due to unexpected presence of UTF-8 characters, but these can be ignored (with the flag -E 0<\/code>).<\/p>\n

XML to text<\/h2>\n

Each Gigaword file contains a series of <DOC><\/code> tags, each representing a single news story. These tags have four possible type<\/code> attributes; the most common one, story<\/code>, is the only one which consistently contains coherent full sentences and paragraphs. Immediately underneath <DOC><\/code> in this hierarchy are two tags:\u00a0<HEADLINE><\/code> and <TEXT><\/code>. While it would be fun to use the former for a study of Headlinese<\/a>, <TEXT><\/code>\u2014the tag surrounding the document body\u2014is generally more useful. Finally, good old-fashioned <p><\/code> (paragraph) tags are the only children of <TEXT><\/code>. I serialized “story” paragraphs using the lxml<\/code> library in Python. This library supports the\u00a0elegant XPath<\/a> query language. To select paragraphs of “story” documents, I used the XPath query\u00a0\/GWENG\/DOC[@type=\"story\"]\/TEXT<\/code>, stripped whitespace, and then encoded the text as UTF-8.<\/p>\n

Text to sentences<\/h2>\n

The resulting units are paragraphs (with occasional uninformative line breaks), not sentences. Python’s NLTK<\/a>\u00a0module provides an interface to the Punkt sentence tokenizer. However, thanks to\u00a0this Stack Overflow post<\/a>, I became aware of its limitations.\u00a0Here’s a difficult example from Moby Dick, <\/i>with sentence boundaries (my judgements) indicated by the pipe character (|<\/code>):<\/p>\n

A clam for supper? | a cold clam; is THAT what you mean, Mrs. Hussey?” | says I, “but that’s a rather cold and clammy reception in the winter time,\u00a0ain’t it, Mrs. Hussey?”<\/p><\/blockquote>\n

But, the default sentence tokenizer insists on sentence breaks immediately after both occurrences of “Mrs.”. To remedy this, I replaced the space after titles like “Mrs.”
\n(the full list of such abbreviations was adapted from GPoSTTL<\/a>) with an underscore so as to “bleed” the sentence tokenizer, then replaced the underscore with a space after tokenization was complete. That is, the sentence tokenizer sees word tokens like\u00a0“Mrs._Hussey”; since sentence boundaries must line up with word token boundaries, there is no chance a space will be inserted here.\u00a0With this hack, the sentence tokenizer does that snippet of\u00a0Moby Dick<\/em> just right.<\/p>\n

Sentences to tokens<\/h2>\n

For the last step, I used NLTK’s word tokenizer (nltk.tokenize.word_tokenize<\/code>), which is similar to the (in)famous Treebank tokenizer<\/a>, and then case-folded<\/a>\u00a0the resulting tokens.<\/p>\n

Summary<\/h2>\n

In all, 170 million sentences, 5 billion word tokens, and 22 billion characters, all of which fits into 7.5 GB (compressed). Best of luck to anyone looking to do the same!<\/p>\n","protected":false},"excerpt":{"rendered":"

I recently took a little time out to coerce a recent version of the LDC’s Gigaword English corpus\u00a0into a format that could be used for training conventional\u00a0n-gram models. This turned out to be harder than I expected. Decompression Gigaword English (v. 5) ships with 7 directories of gzipped SGML data, one directory for each of … Continue reading “Gigaword English preprocessing”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","footnotes":""},"categories":[3,4],"tags":[],"class_list":["post-87","post","type-post","status-publish","format-standard","hentry","category-dev","category-language"],"_links":{"self":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/87","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/comments?post=87"}],"version-history":[{"count":0,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/87\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/media?parent=87"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/categories?post=87"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/tags?post=87"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}