Research Statement

Figure 1: An example sentence pair for the RTE task. In order for a system to conclude that the premise (top) does not entail the hypothesis (bottom), it should recognize that sparked implies caused but that in Denmark precludes in Jordan. These phrase-level entailment relationships are modeled by natural logic.

Table 1: Examples of different types of entailment relations appearing in PPDB.

Figure 2: Distribution of entailment relations in different sizes of PPDB. Distributions are estimated from our manual annotations of randomly sampled pairs. PPDB-XXXL contains over 77MM paraphrase pairs (where the majority type is independent), compared to only 700K in PPDB-S (where the majority type is equivalent).

Figure 3: Summary of features extracted for each phrase pair ⟨p1,p2⟩. Full descriptions of the features used are given in the supplementary material.

Table 2: Column 1 gives the semantics of each label under MacCartney’s Natural Logic. Column 2 gives the notation we use throughout the remainder of this paper. Column 3 gives the description that was shown to Turkers.

Table 3: Top scoring pairs (x/y) according to various similarity measures, along with their manually classified entailment labels. Column 1 is cosine similarity based on dependency contexts. Column 2 is based on Lin (1998), column 3 on Weeds (2004), and column 4 is a novel feature. Precise definitions of each metric are given in the supplementary material.

Table 4: Top paths associated with the ¬ class.

Table 5: F1 measure (×100) achieved by entailment classifier using 10-fold cross validation on the training data.

Figure 4: Confusion matrices for classifier trained using only monolingual features (distributional and path) versus bilingual features (paraphrase and translation). True labels are shown along rows, predicted along columns. The matrix is normalized along rows, so that the predictions for each (true) class sum to 100%.

Table 7: F1 measure (×100) achieved by entailment classifier on the held out phrase pairs from the sentences in SICK test.

Table 6: Example misclassifications from some of the most frequent and most interesting error categories.

Figure 5: ENTAILMENT

Figure 6: CONTRADICTION

Figure 7: NEUTRAL

Figure 8: F1 measures achieved by Nutcracker on SICK test data when using various KBs. Baselines are in gray, this work in blue, human references in gold. PPDB-XL refers to a run in which every pair which appears in PPDB is assumed to be equivalent. PPDB-H refers to a run in which manual labels were used to generate axioms. PPDB+ refers to runs in which the automatic classifications were used to generate axioms. In some cases, better proof coverage causes NC to find incorrect proofs, illustrated by the decreased performance on CONTRADICTION when using PPDB-H. For example, using PPDB-H, NC finds an inconsistency for the pair Someone is not playing piano./A person is playing a keyboard. Using the PPDB+, in which piano/keyboard is falsely classified as

Table 8: Nutcracker’s overall system accuracy and proof coverage when using different sources of axioms. Coverage is measured as the percent of sentence pairs for which NC’s theorem prover or model builder is able to find a complete logical proof of either entailment or contradiction. When NC fails to find either type of proof, it guesses the most frequent class, NEUTRAL. NC alone uses no axioms. PPDB+ refers to the axioms generated automatically using the classifier described in this paper. PPDB-H refers axioms generated using the human labels on which the classifier was trained.

Table 9: Examples of T/H pairs for which the system’s prediction differed when using PPDB+ vs. WN.

Table 1: Example sentence pairs (NORM-SIMP) aligned between English Wikipedia and Simple English Wikipedia. The breakdown in percentages is obtained through manual examination of 200 randomly sampled sentence pairs in the Parallel Wikipedia Simplification (PWKP) corpus.

Table 2: The vocabulary size of the Parallel Wikipedia Simplification (PWKP) corpus and the vocabulary difference between its normal and simple sides (as a 2×2 matrix). Only words consisting of the 26 English letters are counted.

Table 3: Example of sentences written at multiple levels of text complexity from the Newsela data set. The Lexile readability score and grade level apply to the whole article rather than individual sentences, so the same sentences may receive different scores, e.g. the above sentences for the 6th and 7th grades. The bold font highlights the parts of sentence that are different from the adjacent version(s).

Figure 1: Manual classification of aligned sentence pairs from the Newsela corpus. We categorize randomly sampled 50 sentence pairs drawn from the Original-Simp2 and 50 sentences from the Original-Simp4.

Table 4: Basic statistics of the Newsela Simplification corpus vs. the Parallel Wikipedia Simplification (PWKP) corpus. The Newsela corpus consists of 1130 articles with original and 4 simplified versions each. Simp-1 is of the least simplified level, while Simp-4 is the most simplified. The numbers marked by * are slightly different from previously reported, because of the use of different tokenizers.

Table 5: This table shows the vocabulary changes between different levels of simplification in the Newsela corpus (as a 5×5 matrix). Each cell shows the number of unique word types that appear in the corpus listed in the column but do not appear in the corpus listed in the row. We also list the average frequency of those vocabulary items. For example, in the cell marked *, the Simp-4 version contains 583 unique words that do not appear in the Original version. By comparing the cells marked **, we see about half of the words (19,197 out of 39,046) in the Original version are not in the Simp-4 version. Most of the vocabulary that is removed consists of low-frequency words (with an average frequency of 2.6 in the Original).

Table 6: Top 50 tokens associated with the complex text, computed using the Monroe et al. (2008) method. Bold words are shared by the complex version of Newsela and the complex version of Wikipedia.

Table 7: Top 50 tokens associated with the simplified text.

Table 8: Frequency of example words from Table 6. These complex words are reduced at a much greater rate in the simplified Newsela than they are in the Simple English Wikipedia. A smaller odds ratio indicates greater reduction.

Table 9: Top 30 syntax patterns associated with the complex text (left) and simplified text (right). Bold patterns are the top patterns shared by Newsela and Wikipedia.

Figure 2: Distribution of document-level compression ratio, displayed as a histogram smoothed by kernel density estimation. The Newsela corpus is more normally distributed, suggesting more consistent quality.

Figure 3: A radar chart that visualizes the odds ratio (radius axis) of discourse connectives in simple side vs. complex side. An odds ratio larger than 1 indicates the word is more likely to occur in the simplified text than in the complex text, and vice versa. Simple cue words (in the shaded region), except “hence”, are more likely to be added during Newsela’s simplification process than in Wikipedia’s. Complex conjunction connectives (in the unshaded region) are more likely to be retained in Wikipedia’s simplifications than in Newsela’s.

Table 1: Sizes (in

Table 2: Partition statistics for the Callhome Egyptian Arabic corpus, supplements and evaluation datasets. Column 2,3 and 4 represent number of utterances, numbers of words and average number of words per utterance respectively.

Table 3: A sample of the special symbols using in the Arabic transcripts. These represent non-conventional speech segments such as non-verbal vocalizations, disfluencies, background noise and distortion.

Table 4: The results of the translation task described in section 4. Each utterance in the original partitions has about four redundant translations. The number of utterances in column 2 has hence effectively been multiplied by 4. The last column represents the number of words per utterance in the translations.

Table 5: A sample of the translations obtained using the translation task described in section 4. The translations are lower-cased, tokenized and punctuation has been normalized.

Figure 1: The number of workers per country. This map was generated based on geolocating the IP address of 4,983 workers in our study. Omitted are 60 workers who were located in more than one country during the study, and 238 workers who could not be geolocated. The size of the circles represents the number of workers from each country. The two largest are India (1,998 workers) and the United States (866). To calibrate the sizes: the Philippines has 142 workers, Egypt has 25, Russia has 10, and Sri Lanka has 4.

Table 1: Self-reported native language of 3,216 bilingual Turkers. Not shown are 49 languages with 20 speakers. We omit 1,801 Turkers who did not report their native language, 243 who reported 2 native languages, and 83 with

Table 2: A list of the languages that were used in our study, grouped by the number of Wikipedia articles in the language. Each language’s code is given in parentheses. These language codes are used in other figures throughout this paper.

Figure 3: An example of the Turkers’ translations of a Hindi sentence. The translations are unedited and contain fixable spelling, capitalization and grammatical errors.

Figure 2: Days to complete the translation HITs for 40 of the languages. Tick marks represent the completion of individual assignments.

Figure 4: Translation quality for languages with at least 50 Turkers. The dark blue bars indicate the proportion of translations which exactly matched gold standard translations, and light blue indicate translations which were judged to be correct synonyms. Error bars show the 95% confidence intervals for each language.

Figure 5: (a) Individual workers’ overlap with Google Translate. We removed the 500 workers with the highest overlap (shaded region on the left) from our analyses, as it is reasonable to assume these workers are cheating by submitting translations from Google. Workers with no overlap (shaded region on the right) are also likely to be cheating, e.g. by submitting random text. (b) Cumulative distribution of overlap with Google translate for workers and translations. We see that eliminating all workers with >70% overlap with google translate still preserves 90% of translations and >90% of workers.

Table 3: Translation quality when partitioning the translations into two groups, one containing translations submitted by Turkers whose location is within regions that plausibly speak the foreign language, and the other containing translations from Turkers outside those regions. In general, in-region Turkers provide higher quality translations. (**) indicates differences significant at p=0.05, (*) at p=0.10.

Figure 6: The total volume of translations (measured in English words) as a function of elapsed days.

Table 4: Size of parallel corpora and bilingual dictionaries collected for each language.

Table 5: BLEU scores for translating into English using bilingual parallel corpora by themselves, and with the addition of single-word dictionaries. Scores are calculated using four reference translations and represent the mean of three MERT runs.

Table 6: The green box shows the best languages to target on MTurk. These languages have many workers who generate high quality results quickly. We defined many workers as 50 or more active in-region workers, high quality as

Figure 1: The crowd-workers.com web service allows users to sort HITs based on hourly rate. The hourly rate is estimated by tracking how long it took other workers to complete the task (using our browser extension), along with the reward amount (which MTurk makes available).

Figure 2: The hourly earnings of the 65 Turkers in our pilot study who completed more than 100 HITs using the Crowd-Workers browser plugin.

Figure 1: End-to-end runtime as a function of the number of threads. Each data point is the minimum of at least fifteen different runs.

Figure 2: Decoding time alone.

Figure 3: Here, position-aware lexical and part-of-speech n-gram features, labeled dependency links, and features reflecting the phrase’s CCG-style label NP/NN are included in the context vector.

Table 1: Comparing Hadoop’s intermediate disk space use and extraction time on a selection of Europarl v.7 Hiero grammar extractions. Disk space was measured at its maximum, at the input of Thrax’s final grammar aggregation stage. Runtime was measured on our Hadoop cluster with a capacity of 52 mappers and 26 reducers. On average Thrax 2.0, bundled with Joshua 5.0, is up to 300% faster and more compact.

Figure 1: Phrasal paraphrases are extracted via bilingual pivoting.

Figure 2: Features extracted for the phrase the long term from the n-gram corpus (2a) and Annotated Gigaword (2b). (a) The n-gram corpus records the long-term as preceded by revise (43 times), and followed by plans (97 times). We add corresponding features to the phrase’s distributional signature retaining the counts of the original n-grams. (b) Here, position-aware lexical and part-of-speech n-gram features, labeled dependency links , and features reflecting the phrase’s CCG-style label NP/NN are included in the context vector.

Table 1: A breakdown of PPDB:Eng size by paraphrase type. We distinguish lexical (i.e. one-word) paraphrases, phrasal paraphrases and syntactically labeled paraphrase patterns.

Table 2: An overview of PPDB:Spa. Again, we partition the resource into lexical (i.e. one-word) paraphrases, phrasal paraphrases and syntactically labeled paraphrase patterns.

Figure 3: To inspect our coverage, we use the Penn Treebank’s parses to map from Propbank annotations to PPDB’s syntactic patterns. For the above annotation predicate, we extract VBP ! expect, which is matched by paraphrase rules like VBP ! expect | anticipate and VBP ! expect | hypothesize. To search for the entire relation, we replace the argument spans with syntactic nonterminals. Here, we obtain S ! NP expect S, for which PPDB has matching rules like S ! NP expect S | NP would hope S, and S ! NP expect S | NP trust S. This allows us to apply sophisticated paraphrases to the predicate while capturing its arguments in a generalized fashion.

Figure 4: An illustration of PPDB’s coverage of the manually annotated Propbank predicate phrases (4a) and binary relations with argument non-terminals (4b). The curves indicate the coverage on tokens (solid) and types (dotted), as well as the average number of paraphrases per covered type (dashed) at the given pruning level. (a) PPDB:Eng coverage of Propbank predicates (top), and average human judgment score (bottom) for varying pruning thresholds. (b) PPDB:Eng’s coverage of Propbank predicates with up to two arguments. Here we consider rules that paraphrase the full predicate-argument expression.

Table 1: Millions of monolingual web crawl and Wikipedia word tokens

Figure 1: Each box-and-whisker plot summarizes performance on the development set using the given feature(s) across all 22 languages. For each source word in our development sets, we rank all English target words according to the monolingual similarity metric(s) listed. All but the last plot show the performance of individual features. Discrim-All uses supervised data to train classifiers for each language based on all of the features.

Figure 2: Performance on the development set goes up as features are greedily added to the feature space. Mean performance is slightly higher using this subset of six features (second to last bar) than using all features (last bar).

Figure 3: Learning curves over number of positive training instances, up to 1250. For some languages, 1250 positive training instances are not available. In all cases, evaluation is on the development data and the number of negative training instances is three times the number of positive. For all languages, performance is fairly stable after about 300 positive training instances.

Table 2: Top-10 Accuracy on test set. Performance increases for all languages moving from the baseline (MRR) to discriminative training (Supv).

Figure 4: Millions of monolingual word tokens vs. Lexicon Induction Top-10 Accuracy

Table 1: Corpus size and cost. Counts of segments and words were computed after pre-processing (§2).

Table 2: Data splits for Fisher Spanish (top), Callhome Spanish (middle), and Europarl + News Commentary (bottom; for comparison). Words is the number of Spanish word tokens (after tokenization). The mean number of words per sentences ranges from 11.8 to 13.1.

Table 3: Lattice statistics for the three Fisher and two Callhome test sets. Word error rates correspond to the 1-best and oracle paths from the lattice, and

Table 4: BLEU scores (four references) on Fisher/Dev2. The columns vary the data used to train the MT system, and the rows alter the interface between the ASR and MT systems.

Table 5: BLEU scores (one reference) on Callhome/Evltest.

Figure 1: A subgraph of a lattice (sentence 17 of Fisher/Dev2) representing an ASR ambiguity. The oracle path is in bold. With access to the lattice, the MT system avoids the untranslatable word incorporamos, found in the 1-best output, producing a better translation. Above the line are inputs and the reference, with the Lattice line denoting the path selected by the MT system. The Google line is suggestive of the general difficulty in translating conversational speech.

Figure 2: Conversation-level WER and BLEU, for conversations found in Fisher/Dev (open points) and Fisher/Dev2 (solid points). The Pearson’s correlation coefficient is -0.72.

Figure 1: An illustration of our packed grammar data structures. The source sides of the grammar rules are stored in a packed trie. Each node may contain n children and the symbols linking to them, and m entries for rules that share the same source side. Each rule entry links to a node in the target-side trie, where the full target string can be retrieved by walking up the trie until the root is reached. The rule entries also contain a data block id, which identifies feature data attached to the rule. The features are encoded according to a type/quantization specification and stored as variable-length blocks of data in a byte buffer.

Table 1: Decoding-time memory use for the packed grammar versus the standard grammar format. Even without lossy quantization the packed grammar representation yields significant savings in memory consumption. Adding 8-bit quantization for the real- valued features in the grammar reduces even large syntactic grammars to a manageable size.

Figure 2: A visualization of the loading and decoding speed on the WMT12 French-English development set contrasting the packed grammar representation with the standard format. Grammar loading for the packed grammar representation is substantially faster than that for the baseline setup. Even with a slightly slower decoding speed (note the difference in the slopes) the packed grammar finishes in less than half the time, compared to the standard format.

Figure 3: Experimental results on the development and test sets. The x-axis is the number of iterations (up to 30) and the y-axis is the BLEU score. The three curves in each figure correspond to three classifiers. Upper row: results trained using only dense features (10 features); Lower row: results trained using dense+sparse features (1026 features). Left column: development set (MT03); Middle column: test set (MT04); Right column: test set (MT05).

Table 2: Comparison between the results given by Z-MERT and J-PRO (trained with 10 features).

Table 3: Extraction times and grammar sizes for Hiero grammars using the Europarl and News Commentary training data for each listed language pair.

Table 4: Extraction times and grammar sizes for the SAMT grammars using the Europarl and News Commentary training data for each listed language pair.

Table 5: Large paraphrase grammars extracted from EuroParl data using Thrax. The sentence and word counts refer to the English side of the bitexts used.

Figure 1: The reordering probabilities from the phrasebased models are estimated from bilingual data by calculating how often in the parallel corpus a phrase pair (f; e) is orientated with the preceding phrase pair in the 3 types of orientations (monotone, swapped, and discontinuous).

Figure 2: Accuracy of single-word translations induced using contextual similarity as a function of the source word corpus frequency. Accuracy is the proportion of the source words with at least one correct (bilingual dictionary) translation in the top 1 and top 10 candidate lists.

Figure 3: Scoring contextual similarity of phrases: first, contextual vectors are projected using a small seed dictionary and then compared with the target language candidates.

Figure 4: Temporal histograms of the English phrase terrorist, its Spanish translation terrorista, and riqueza (wealth) collected from monolingual texts spanning a 13 year period. While the correct translation has a good temporal match, the non-translation riqueza has a distinctly different signature.

Figure 5: Algorithm for estimating reordering probabilities from monolingual data.

Figure 6: Collecting phrase orientation statistics for a English-German phrase pair (“profile”, “Profils”) from non-parallel sentences (the German sentence translates as “Creating a Facebook profile is easy”).

Table 1: Statistics about the monolingual training data and the phrase table that was used in all of the experiments.

Figure 7: Much of the loss in BLEU score when bilingually estimated features are removed from a Spanish-English translation system (experiments 1-4) can be recovered when they are replaced with monolingual equivalents estimated from monolingual Europarl data (experiments 5-10). The labels indicate how the different types of parameters are estimated, the first part is for phrase-table features, the second is for reordering probabilities.

Figure 8: Performance of monolingual features derived from truly monolingual corpora. Over 82% of the BLEU score loss can be recovered.

Figure 1: One possible breakdown of spoken Arabic into dialect groups: Maghrebi, Egyptian, Levantine, Gulf and Iraqi. Habash (2010) gives a breakdown along mostly the same lines. We used this map as an illustration for annotators in our dialect classification task (Section 3.1), with Arabic names for the dialects instead of English

Table 1: The total costs for the three MTurk subtasks involved with the creation of our Dialectal Arabic-English parallel corpus.

Table 2: Statistics about the training/tuning/test datasets used in our experiments. The token counts are calculated before MADA segmentation.

Table 3: Comparison of the effect of morphological segmentation when translating MSA web text and Dialectal Arabic web text. The morphological segmentation uniformly improves translation quality, but the improvements are more dramatic for MSA than for Dialectal Arabic when comparing similarly-sized training corpora.

Table 4: A comparison of translation quality of Egyptian, Levantine, and MSA web text, using various training corpora. The highest BLEU scores are achieved using the full set of dialectal data (which combines Levantine and Egyptian), since the Egyptian alone is sparse. For Levantine, adding Egyptian has no effect. In both cases, adding MSA to the dialectal data results in marginally worse translations.

Table 5: The most frequent OOV’s (with counts ≥ 10) of our test sets against the MSA training data.

Figure 2: Examples of improvement in MT output when training on our Dialectal Arabic-English parallel corpus instead of an MSA-English parallel corpus.

Figure 3: Examples of ambiguous words that are translated incorrectly by the MSA-English system, but correctly by the Dialectal Arabic-English system.

Figure 4: Learning curves showing the effects of increasing the size of dialectal training data, when combined with the 150M-word MSA parallel corpus, and when used alone. Adding the MSA training data is only useful when the dialectal data is scarce (200k words).

Table 6: Results on a truly independent test set, consisting of data harvested from Egyptian Facebook pages that are entirely distinct from the our dialectal training set. The improvements over the MSA baseline are still considerable: +2.9 BLEU points when no Facebook data is available for tuning and +2.7 with a Facebook tuning set.

Table 7: A comparison of the effectiveness of performing Levantine-to-MSA mapping before translating into English, versus translating directly from Levantine into English. The mapping from Levantine to MSA was done manually, so it is an optimistic estimate of what might be done automatically. Although initially helpful to the MSA baseline system, the usefulness of pivoting through MSA drops as more dialectal data is added, eventually hurting performance.

Figure 1: An example of SOV word ordering in Tamil. Translation: The senator prepared her remarks.

Figure 2: An example of the morphology of the Bengali word হাত্সিলাম, meaning [I] was walking. CONT denotes the continuous aspect, while PAST denotes past tense.

Table 3: Dictionary statistics. Entries is the number of source-language types, while translations lists the number of words or phrases they translated to (i.e., the number of pairs in the dictionary). Controls for Hindi were obtained using Google translate, the only one of these languages that were available at the outset of this project.

Figure 3: The total volume of translations (measured in English words) as a function of elapsed days. For Malayalam, we collected half a million words of translations in just under a week.

Table 4: Data set sizes for each language pair: words in the first row, parallel sentences in the second. (The dictionaries contains short phrases in addition to words, which accounts for the difference in dictionary word and line counts.)

Table 5: BLEU scores translating into English (four references). BLEU scores are the mean of three MERT runs.

Table 6: Some example translations.

Table 7: BLEU scores translating into English on a quarter of the training data (plus dictionary), selected in two ways: best (result of vote), and random. There is little difference, suggesting quality control may not be terribly important. We did not collect votes for Malayalam.

Table 8: Misspellings of japanese (947) in the training portion of the Urdu-English data, along with their counts.

Table 9: Hiero translation results using Berkeley align- ments instead of GIZA++ heuristics. The gain columns denotes improvements relative to the Hiero systems in Ta- ble 5. In many cases (bold gains), the BLEU scores are at or above even the SAMT models from that table.

Figure 1: An aligned sentence pair.

Table 1: A subset of the Hiero and SAMT rules extracted from the sentence pair of Figure 1.

Table 2: Training data size after subsampling.

Table 3: Single-reference BLEU-4 scores.

Figure 2: An SAMT derivation. The shaded terminal symbols are the lexicalized part of a rule with terminals and non-terminals. The unshaded terminals are directly dominated by a nonterminal symbol.

Figure 1: Synchronous grammar rules for translation are extracted from sentence pairs in a bixtext which have been automatically parsed and word-aligned. Extraction methods vary on whether they extract only minimal rules for phrases dominated by nodes in the parse tree, or more complex rules that include non-constituent phrases.

Figure 2: An example derivation produced by a syntactic machine translation system. Although the synchronous trees are unlike the derivations found in the Penn Treebank, their yield is a good translation of the German.

Figure 3: An example of a synchronous paraphrastic derivation. A few of the rules applied in the parse are show in the left column, with the pivot phrases that gave rise to them on the right.

Table 1: A selection of meaning-preserving transformations and hand-picked examples of syntactic paraphrases that our system extracts capturing these.

Table 2: Number and distribution of rules in our para- phrase grammar. Note the significant number of identity paraphrases and rules with complex nonterminal labels.

Table 4: Human evaluation for shorter compressions and for variations of our paraphrase system. +Feat. includes the compression features from Section 6.1, +Aug. includes optional deletion rules from Section 6.4.

Table 3: Results of the human evaluation on longer compressions: pairwise compression rates (CR), meaning and grammaticality scores. Bold indicates a statistically significance difference at p < 0.05.

Table 5: Example compressions produced by the two systems in Table 3 for three input sentences from our test data.

Figure 1: A comparison of professional translations provided by the LDC to non-professional translations created on Mechanical Turk.

Figure 2: We redundantly translate each source sentence by soliciting multiple translations from different Turkers. These translations are put through a subsequent editing set, where multiple edited versions are produced. We select the best translation from the set using features that predict the quality of each translation and each translator.

Figure 3: BLEU scores for different selection methods, measured against the reference sets. Each score is an average of four BLEU scores, each calculated against three LDC reference translations. The five right-most bars are colored in orange to indicate selection over a set that includes both original translations as well as edited versions of them.

Table 1: Correlation (± std. dev.) for different selection methods, compared against the reference sets.

Figure 4: BLEU scores for the five right-most setups from Figure 3, constrained over the original translations.

Figure 5: The effect of varying the amount of calibration data (and using only the calibration feature). The 10% point (BLEU = 37.82) and the dashed line (BLEU = 39.06) correspond to the two right-most bars of Figure 3.

Figure 1: Two roughly equivalent Arabic sentences, one in MSA and one in Levantine Arabic, translated by the same MT system into English. An acceptable translation would be When will we see this group of criminals undergo trial (or tried)?. The MSA variant is handled well, while the dialectal variant is mostly transliterated

Figure 2: One possible breakdown of spoken Arabic into dialect groups: Maghrebi, Egyptian, Levantine, Gulf, and Iraqi. Habash (2010) also gives a very similar breakdown

Table 1: A summary of the different components of the AOC dataset. Overall, 1.4M comments were harvested from 86.1K articles, corresponding to 52.1M words.

Table 2: A breakdown of sentences for which ≥ 2 annotators agreed on whether dialectal content exists or not.

Table 3: Accuracy, dialect precision, and dialect recall (10-fold cross validation) for various classification tasks.

Figure 3: Dialect precision vs. recall for the classification task over Al-Ghad sentences (MSA vs. Levantine). The square point corresponds to the first line in Table 3.

Figure 1: An example of Urdu-English translation. Shown are an Urdu source document, a reference translation produced by a professional human translator, and machine translation output from a phrase-based model (Moses) without linguistic information, which is representative of state-of-the-art MT quality before the SIMT effort.

Figure 2: The evolution of a semantically informed approach to our synchronous context free grammars (SCFGs). At the start of summer the decoder used translation rules with a single generic non-terminal symbol, later syntactic categories were used, and by the end of the summer the translation rules included semantic elements such as named entities and modalities.

Table 1: The size of the various data sets used for the experiments in this paper including the training, development (dev), incremental test set (devtest) and blind test set (test). The dev/devtest was a split of the NIST08 Urdu-English test set, and the blind test set was NIST09.

Figure 3: Workflow for producing semantically-grafted parse trees. The English side of the parallel corpus is automatically parsed, and also tagged with modality and named-entity markers. These tags are then grafted onto the syntactic parse trees. The relation finder was designed for additional tagging but was not implemented in the current work. (Future work will test relations as another component of meaning that may contribute toward im- proved MT ouput.)

Figure 4: A sentence on the English side of the bilingual parallel training corpus is parsed with a syntactic parser, and also tagged with a named entity tagger. The tags are then grafted onto the syntactic parse tree to form new categories like NP-GPE and NP-weapon. Grafting happens prior to extracting translation rules, which happens normally except for the use of the augmented trees.

Figure 5: Example translation rules with named entity tags and modalities combined with syntactic categories.

Table 2: Named entity tags

Table 3: Modality tags with their negated versions

Figure 6: Results for a range of experiments conducted during the SIMT effort. Results show scores for base- line systems, which here include a phrase-based model (Moses) and a hierarchical phrase-based model (Hiero), neither of which make use of syntactic information. These also show the substantial improvements when syn- tax is introduced, along with different numbers of feature functions (FFs), and further improvements from semantic elements. The scores are lowercased Bleu calculated on the held-out devtest set.

Figure 7: An example of the improvements to Urdu-English translation before and after the SIMT effort. Output is from the baseline Hiero model, which does not use linguistic information, and from the final model, which incorporates syntactic and semantic information.

Figure 1: Time spent, HITs completed, and amount earned from a survey of 1,000 Turkers by Ipeirotis (2010).

Figure 1: Statistics for the training and test sets used in the translation task. The number of words and the number of distinct words is based on the provided tokenizer.

Table 1: Participants in the shared translation task. Not all groups participated in all language pairs.

Table 2: Participants in the system combination task.

Table 3: The number of items that were collected for each task during the manual evaluation. An item is defined to be a rank label in the ranking task, an edited sentence in the editing task, and a yes/no judgment in the judgment task.

Table 4: Inter- and intra-annotator agreement for the sentence ranking task. In this task, P(E) is 0.333.

Figure 2: This screenshot shows what an annotator sees when beginning to edit the output of a machine translation system.

Table 5: Official results for the WMT10 translation task, based on the human evaluation (ranking translations relative to each other)

Table 6: Official results for the WMT10 system combination task, based on the human evaluation (ranking translations relative to each other)

Figure 3: The percent of time that each system’s edited output was judged to be an acceptable translation. These numbers also include judgments of the system’s output when it was marked either incomprehen- sible or acceptable and left unedited. Note that the reference translation was edited alongside the system outputs. Error bars show one positive and one negative standard deviation for the systems in that lan- guage pair.

Table 7: System-level Spearman’s rho correlation of the automatic evaluation metrics with the human judgments for translation into English, ordered by average absolute value.

Table 9: Segment-level Kendall’s tau correlation of the automatic evaluation metrics with the human judgments for translation into English, ordered by average absolute value. Number of pairs included in comparison: cz-en 3575, fr-en 5844, de-en 7585, es-en 7911.

Table 8: System-level Spearman’s rho correlation of the automatic evaluation metrics with the human judgments for translation out of English, ordered by average absolute value.

Table 10: Segment-level Kendall’s tau correlation of the automatic evaluation metrics with the human judgments for translation out of English, ordered by average absolute value. Number of pairs included in comparison: en-cz 9613, en-fr 5904, en-de 10892, en-es 3813.

Table 11: Statistics for data collected on MTurk for the ranking task. In total, 55,082 rank labels were collected across the eight language pairs (145% of expert data). Each language pair had 600 sets, and we requested each set completed by 5 different workers. Since each set provides 5 labels, we could have potentially obtained 600 _ 5 _ 5 = 15,000 labels for each language pair. The Label count row indicates to what extent that potential was met (over the 30-day lifetime of our tasks), and the “Completed...” rows give a breakdown of redundancy. For instance, the right-most column indicates that, in the cz-en group, 2.0% of the 600 sets were completed by only one worker, while 67% of the sets were completed by 5 workers, with 100% of the sets completed at least once. The total cost of this data collection effort was roughly $200.

Figure 4: The effect of removing an increasing number of MTurk workers. The order in which workers are removed is by Kexp(w), the kappa agreement coefficient with expert data (excluding references).

Table 1: The uncased BLEU scores on WMT-09 French-English Task. The test set consists of 2525 segments, each with one reference translation.

Figure 1: Statistics for the training and test sets used in the translation task. The number of words is based on the provided tokenizer and the number of distinct words is the based on lowercased tokens.

Table 1: Participants in the shared translation task. Not all groups participated in all language pairs.

Table 2: Participants in the system combination task.

Table 3: The number of items that were judged for each task during the manual evaluation.

Figure 2: This screenshot shows an annotator editing the output of a machine translation system.

Figure 3: This screenshot shows an annotator judging the acceptability of edited translations.

Table 4: Inter- and intra-annotator agreement for the two types of manual evaluation

Table 6: Official results for the WMT09 translation task, based on the human evaluation (ranking translations relative to each other)

Table 5: A comparison between the best system combinations and the best individual systems. It was generally difficult to draw a statistically significant differences between the two groups, and between the combinations themselves.

Figure 6: The percent of time that each system’s edited output was judged to be an acceptable translation. These numbers also include judgments of the system’s output when it was marked either incomprehensible or acceptable and left unedited. Note that the reference translation was edited alongside the system outputs. Error bars show one positive and one negative standard deviation for the systems in that language pair.

Table 7: The system-level correlation of the automatic evaluation metrics with the human judgments for translation into English.

Table 8: The system-level correlation of the automatic evaluation metrics with the human judgements for translation out of English.

Table 9: The system-level correlation for automatic metrics ranking five English-Czech systems

Table 10: Sentence-level consistency of the automatic metrics with human judgments for translations into English. Italicized numbers do not beat the random-choice baseline. (This table was corrected after publication.)

Table 11: Sentence-level consistency of the automatic metrics with human judgments for translations out of English. Italicized numbers do not beat the random-choice baseline. (This table was corrected after publication.)

Figure 1: Non-factored translation

Figure 2: Factored translation

Figure 3: Example of graph of decoding steps

Figure 1: The Spanish word cada ́veres can be used to discover that the English phrase dead bodies can be paraphrased as corpses.

Figure 2: Barzilay and McKeown (2001) extracted paraphrases from multiple transla- tions using identical surrounding substrings

Figure 3: A phrase can be aligned to many foreign phrases, which in turn can be aligned to multiple possible paraphrases

Figure 4: In machine translation evaluation the following scales are used by judges to assign adequacy and fluency scores to each translation

Figure 5: Percent of unique unigrams, bigrams, trigrams, and 4-grams from the Europarl Spanish test sentences for which translations were learned in increasingly large training corpora

Figure 6: Scatterplot of the length of each translation against its number of possible permutations due to bigram mismatches for an entry in the 2005 NIST MT Eval

Figure 7: The decoder for the baseline system has translation options only for those words which have phrases that occur in the phrase table. In this case there are no translations for the source word votare ́.

Figure 1: Using a monolingual parallel corpus to extract paraphrases

Figure 2: Using a bilingual parallel corpus to extract paraphrases

Table 1: Phrases that were selected to paraphrase

Figure 3: Phrases highlighted for manual alignment

Figure 4: Paraphrases substituted in for the original phrase

Table 2: Paraphrases extracted from a manually word-aligned parallel corpus

Table 3: Paraphrase accuracy and correct meaning for the different data conditions