The included data set contains 137,362 aligned sentences extracted by pairing Simple English Wikipedia with English Wikipedia. A complete description of the extraction process can be found in "Simple English Wikipedia: A New Simplification Task", William Coster and David Kauchak (2011). In Proceedings of ACL (short paper). The data set contains those sentences with a similarity above 0.50. Higher precision alignments may be obtained by TF-IDF thresholding at higher levels.Two files are included: wiki.normal and wiki.simple. Each file contains 137,362 lines and corresponds to a sentence. The nth line/sentence in wiki.normal corresponds to the nth line/sentence in wiki.simple. Some minimal tokenization has been done to treat most punctuation characters as separate words/tokens.For questions regarding the data set set, contact David Kauchak at Pomona College.
