In natural language, a given meaning can generally be expressed in a number of different ways. This variability of form brings with it the interesting challenge of recognizing when the meaning expressed by a given sentence or text can be inferred from that expressed by another.

While such meaning comparison is practically relevant for a variety of tasks in computational linguistics (e.g., question answering, information extraction, machine translation evaluation, or text summarization), in the past couple of years researchers have tried to define a general notion of textual entailment underlying such real world inference needs and organized a yearly challenge around Recognizing Textual Entailment (RTE).

Our seminar will introduce the issue as well as questions which have been raised about it, before discussing the range of approaches, from deep linguistic analysis with logical inferences to shallow matching of surface features, which have been proposed to tackle the RTE challenge.

Students are expected to actively participate in the seminar understood as a research group, to investigate and present a subtopic, and write a term paper. The default topic for the term paper will be a quantitative and qualitative analysis of the performance of an RTE system (such as the freely available VENSES and Nutcracker systems) on data from the past RTE challenges. Alternatively, students interested in exploring their own RTE approach can form 2-3 people teams to develop, implement, and evaluate their approach, and they will hand in a term paper documenting this effort (which is rewarded with 5 extra CP). Successful teams might also be interested in participating the RTE 5 challenge, for which the development data is scheduled to be released in early summer, with the test data becoming available in early September.

Nature of course and my expectations: This is a research-oriented seminar, i.e., each participant is expected to take an active role in exploring the topic. More concretely, each participant is expected to

Academic conduct and misconduct: Research is driven by discussion and free exchange of ideas, motivations, and perspectives. So you are encouraged to work in groups, discuss, and exchange ideas. At the same time, the foundation of the free exchange of ideas is that everyone is open about where they obtained which information. Concretely, this means you are expected to always make explicit when you’ve worked on something as a team – and keep in mind that being part of a team always means sharing the work.

For text you write, you always have to provide explicit references for any ideas or passages you reuse from somewhere else. Note that this includes text “found” on the web, where you should cite the url of the web site in case no more official publication is available.

Class etiquette: Please do not read or work on materials for other classes in our seminar. Come to class on time and do not pack up early. When our seminar meets in the computer lab, only use the computers when you are asked to do a specific activity – do not read email or browse the web. All portable electronic devices such as cell phones should be switched off for the entire length of the flight, oops, class. If for some reason, you must leave early or you have an important call coming in, or you have to miss class for an important reason, please let me know before class.

References

A. Rodrigo, A. Penas, F. V. (2008). Towards an Entity-based Recognition of Textual Entailment. In TAC (2008). URL http://www.nist.gov/tac/publications/2008/participant.papers/UNED.proceedings.pdf.

Abeillé, A. (ed.) (2003). Treebanks: Building and using syntactically annotated corpora. Dordrecht: Kluwer Academic Publishers. URL http://treebank.linguist.jussieu.fr/toc.html.

Ageno, A., D. Farwell, D. Ferrés, F. Cruz, H. Rodríguez & J. Turmo (2008). TALP at TAC 2008: A Semantic Approach to Recognizing Textual Entailment. In Proceedings of Text Analysis Conference 2008. URL http://www.nist.gov/tac/publications/2008/participant.papers/UPC.proceedings.pdf.

Bensley, J. & A. Hickl (2008). Workshop: Application of LCC’s GROUNDHOG System for RTE-4. In TAC (2008). URL http://www.nist.gov/tac/publications/2008/participant.papers/lcc.proceedings.pdf.

Bergmair, R. (2008). Monte Carlo Semantics: McPIET at RTE-4. In Proceedings of Text Analysis Conference 2008. URL http://www.nist.gov/tac/publications/2008/participant.papers/cambridge.proceedings.pdf.

Blackburn, P. & J. Bos (2003). Computational Semantics. Theoria 18, 27–45. URL http://homepages.inf.ed.ac.uk/jbos/pubs/theoria.pdf.

Bos, J. & K. Markert (2005a). Combining Shallow and Deep NLP Methods for Recognizing Textual Entailment. In Proceedings of the PASCAL Challenges Workshop. URL http://www.cs.biu.ac.il/~nlp/RTE1/Proceedings/bos_and_markert.pdf.

Bos, J. & K. Markert (2005b). Recognising Textual Entailment with Logical Inference. In Proceedings of EMNLP 2005. URL http://aclweb.org/anthology-new/H05-1079.

Bos, J. & K. Markert (2006). When logical inference helps determining textual entailment. (and when it doesn’t). In R. Bar-Haim, I. Dagan, B. Dolan, L. Ferro, D. Giampiccolo, B. Magnini & I. Szpektor (eds.), Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. Venice, Italy. URL http://u.cs.biu.ac.il/~nlp/RTE2/Proceedings/24.pdf.

Burchardt, A. (2008). Modeling Textual Entailment with Role-Semantic Information. Ph.D. thesis, Universität des Saarlandes, Saarbrücken, Germany. URL http://www.coli.uni-saarland.de/%7Ealbu/papers/burchardt_diss.pdf. Volume 29 of Saarbrücken dissertation series in Computational Linguistics and Language Technology, ISBN 978-3-933218-21-6.

Burchardt, A. & A. Frank (2006). Approximating Textual Entailment with LFG and FrameNet Frames. In Proceedings of the second PASCAL Recognizing Textual Entailment Workshop. Venice, Italy, pp. 92–97. URL http://u.cs.biu.ac.il/~nlp/RTE2/Proceedings/15.pdf.

Burchardt, A., M. Pennacchiotti, S. Thater & M. Pinkal (to appear). Assessing the Impact of Frame Semantics on Textual Entailment. Natural Language Engineering .

Burchardt, A., N. Reiter, S. Thater & A. Frank (2007). A Semantic Approach To Textual Entailment: System Evaluation and Task Analysis. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Prague: Association for Computational Linguistics, pp. 10–15. URL www.aclweb.org/anthology-new/W07-1402.

Candela, J. Q., I. Dagan, B. Magnini & F. d’Alché Buc (eds.) (2006). Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, vol. 3944 of Lecture Notes in Computer Science. Springer.

Crouch, D., R. Saurí & A. Fowler (2006a). AQUAINT pilot knowledge-based evaluation: Annotation guidelines. URL http://www2.parc.com/istl/groups/nltt/papers/aquaint_kb_pilot_evaluation_guide.pdf. Ms. Palo Alto Research Center.

Crouch, R., L. Karttunen & A. Zaenen (2006b). Circumscribing is not excluding: A response to Manning. URL http://www2.parc.com/istl/members/karttune/publications/reply-to-manning.pdf. Ms. Palo Alto Research Center.

Dagan, I., O. Glickman & B. Magnini (2006). The PASCAL Recognising Textual Entailment Challenge. In Candela et al. (2006), pp. 177–190. URL http://www.cs.biu.ac.il/~nlp/RTE1/Proceedings/dagan_et_al.pdf.

Gardent, C. & K. Konrad (2000). Interpreting definites using model generation. Journal of Language and Computation 1(2), 193–209.

Garoufi, K. (2007). Towards a Better Understanding of Applied Textual Entailment: Annotation and Evaluation of the RTE-2 Dataset. Master’s thesis, Saarland University. URL http://www.ukp.tu-darmstadt.de/publications/details/?no_cache=1&pub_id=TUD-CS-2008-105.

Hickl, A. & J. Bensley (2007). A Discourse Commitment-Based Framework for Recognizing Textual Entailment. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Prague: Association for Computational Linguistics, pp. 171–176. URL http://www.aclweb.org/anthology-new/W/W07/W07-1428.

Hobbs, J., M. Stickel, D. Appelt & P. Martin (1993). Interpretation as Abduction. Artificial Intelligence 63(1–2), 69–142. URL https://eprints.kfupm.edu.sa/46799/1/46799.pdf.

Hobbs, J. R., M. Stickel, P. Martin & D. Edwards (1988). Interpretation as Abduction. In Proceedings of the 26th Annual Meeting of the ACL. Buffalo, NY: Association for Computational Linguistics. URL http://www.aclweb.org/anthology-new/P88-1012.

Hodges, D., C. Clark, A. Fowler & D. Moldovan (2006). Applying COGEX to Recognize Textual Entailment. In Machine Learning Challenges, Springer, pp. 427–448. URL http://dx.doi.org/10.1007/11736790_24.

Iftene, A. (2008). UAIC Participation at RTE4. In TAC (2008). URL http://www.nist.gov/tac/publications/2008/participant.papers/UAIC2008.proceedings.pdf.

Inoue, K., Y. Ohta, R. Hasegawa & M. Nakashima (1993). Bottom-up Abduction by Model Generation. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence (IJCAI-93). vol. 1, pp. 102–108. URL http://dli.iiit.ac.in/ijcai/IJCAI-93-VOL1/PDF/015.pdf.

Karttunen, L. & A. Zaenen (2005). Veridicity. In G. Katz, J. Pustejovsky & F. Schilder (eds.), Annotating, Extracting and Reasoning about Time and Events. Dagstuhl, Germany: Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany, no. 05151 in Dagstuhl Seminar Proceedings. URL http://drops.dagstuhl.de/opus/volltexte/2005/314.

Konrad, K. (2000). Model generation for natural language interpretation and analysis. Ph.D. thesis, Universität des Saarlandes, Saarbrücken, Germany. URL http://scidok.sulb.uni-saarland.de/volltexte/2007/1341/.

Lin, D. (1998). Dependency-based Evaluation of MINIPAR. In Proeedings of the Workshop on the Evaluation of Parsing Systems. Granada, Spain, pp. 317–330. URL http://www.cfilt.iitb.ac.in/archives/minipar_evaluation.pdf. Reprinted as Chapter 18 of Abeillé (2003).

Manning, C. D. (2006). Local Textual Inference: It’s hard to circumscribe, but you know it when you see it – and NLP needs it. URL http://nlp.stanford.edu/%7Emanning/papers/LocalTextualInference.pdf. Ms. Stanford University.

Monz, C. & M. de Rijke (2001). Light-Weight Entailment Checking for Computational Semantics. In Proceedings of the 3rd Workshop on Inference in Computational Semantics. URL http://www.dcs.qmul.ac.uk/~christof/html/publications/icos3.pdf.

Nielsen, R. D., L. Becker & W. Ward (2008). TAC 2008 CLEAR RTE system report: Facet-based entailment. In TAC (2008). URL http://www.nist.gov/tac/publications/2008/participant.papers/CLEAR.proceedings.pdf.

Oberlander, J. & A. Lascarides (2000). Laconic Discourses and Total Eclipses: Abduction in DICE. In H. Bunt & W. Black (eds.), Abduction, Beliefs and Context: Studies in Computational Pragmatics, John Benjamins, pp. 391–412. URL http://homepages.inf.ed.ac.uk/alex/papers/laconic.pdf.

Raina, R., A. Y. Ng & C. D. Manning (2005). Robust Textual Inference Via Learning and Abductive Reasoning. In M. M. Veloso & S. Kambhampati (eds.), Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, July 9-13, 2005, Pittsburgh, Pennsylvania, USA. AAAI Press / The MIT Press, pp. 1099–1105. URL http://www.aaai.org/Papers/AAAI/2005/AAAI05-174.pdf.

TAC (2008). Proceedings of the Text Analysis Conference, Gaithersburg, Maryland, November 17-19, 2008. National Institute of Standards and Technology. URL http://www.nist.gov/tac/publications/2008/papers.html.

Turney, P. D. (2001). Mining the Web for Synonyms: PMI-IR Versus LSA on TOEFL. In Proceedings of the Twelfth European Conference on Machine Learning. URL http://cogprints.org/1796/0/ECML2001.pdf.

Turney, P. D. (2008). A uniform approach to analogies, synonyms, antonyms, and associations. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008). Manchester, UK, pp. 905–912. URL http://cogprints.org/6181/1/turney_coling08.pdf.

Vanderwende, L. & W. B. Dolan (2006). What Syntax Can Contribute in the Entailment Task. In Candela et al. (2006), pp. 205–216. URL http://research.microsoft.com/pubs/69303/rte1.pdf.

Wang, R. & G. Neumann (2008). An Accuracy-Oriented Divide-and-Conquer Strategy for Recognizing Textual Entailment. In TAC (2008). URL http://www.nist.gov/tac/publications/2008/participant.papers/DFKI.proceedings.pdf.

Yatbaz, M. A. (2008). RTE4: Normalized Dependency Tree Alignment Using Unsupervised N-gram Word Similarity Score. In Proceedings of Text Analysis Conference 2008. URL http://www.nist.gov/tac/publications/2008/participant.papers/KUNLP.proceedings.pdf.

Zaenen, A., L. Karttunen & R. Crouch (2005). Local Textual Inference: Can it be Defined or Circumscribed? In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment. Ann Arbor, Michigan: Association for Computational Linguistics, pp. 31–36. URL http://www.aclweb.org/anthology-new/W/W05/W05-1206.