- The submission deadline for the Technical Report is the 19th October 2016, submission instructions are available here: http://www.evalita.it/2016/techreports
- 26th July 2016: the Development Data package (version 3) is available here: https://drive.google.com/open?id=0B9zwc2pyNmBkUV9UaWZzVWRLNzQ
- 20th June 2016: the Development Data package (version 2) is available here: https://drive.google.com/open?id=0B9zwc2pyNmBkUV9UaWZzVWRLNzQ
- 30th May 2016: the Development Data package (version 1) is available here: https://drive.google.com/open?id=0B9zwc2pyNmBkUV9UaWZzVWRLNzQ
- 17th May 2016: Task Guideline (version 1) are available here: https://drive.google.com/open?id=0B9zwc2pyNmBkeG1faVYtR3BCR2s
- 25th Mar 2016: Web site is on-line
Searching within the Frequently Asked Questions (FAQ) page of a web site is a critical task: customers might feel overloaded by many irrelevant questions and become frustrated due to the difficulty in finding the FAQ suitable for their problems. Perhaps they are right there, but just worded in a different way than they know.
The proposed task consists in retrieving a list of relevant FAQs and corresponding answers related to the query issued by the user.
Acquedotto Pugliese (AQP) developed a semantic retrieval engine for FAQs, called AQP Risponde, based on Question Answering (QA) techniques. The system allows customers to ask their own questions, and retrieves a list of relevant FAQs and corresponding answers. Furthermore, customers can select one FAQ among those retrieved by the system and can provide their feedback about the perceived accuracy of the answer.
AQP Risponde poses relevant research challenges concerning both the usage of the Italian language in a deep QA architecture, and the variety of language expressions adopted by customers to formulate the same information need.
The task that we propose is strongly related to the recently organized task at Semeval 2015 and 2016 about Answer Selection in Community Question Answering . This task helps to automate the process of finding good answers to new questions in a community-created discussion forum (e.g., by retrieving similar questions in the forum and by identifying the posts in the answer threads of similar questions that answer the original one as well). Moreover, the QA4FAQ has some common points with the Textual Similarity task  that received an increasing amount of attention in recent years.
AQP Risponde provides a back-end that allows to analyze both the query log and the customers' feedback to discover, for instance, new emerging problems that need to be encoded as FAQ. AQP Risponde is provided as web and mobile application for Android and iOS and is currently running in Acquedotto Pugliese customer care. AQP received about 25,000 questions and collected about 2,500 user feedbacks. We rely on these data for building the dataset for the task. In particular, we provide:
- a knowledge base of about 470 FAQs. Each FAQ is composed of a question, an answer, and a set of metadata (category and tags);
- a set of user queries. The queries are collected by analyzing the AQP Risponde system log;
- a set of pairs that are exploited to evaluate the contestants. We build these pairs by analyzing the user feedbacks provided by the real users of AQP Risponde. We manually check the user feedbacks in order to remove noisy or false feedbacks.
We will provide a little sample set for the system development and a test set for the evaluation. We will not provide a set of training data: AQP is interested in the development of unsupervised systems since AQP Risponde should be able to provide good performance without any user feedback.
Following, an example of FAQ is reported:
Question: Come posso telefonare al numero verde da un cellulare?
Answer: "E' possibile chiamare il Contact Center AQP per segnalare un guasto o per un pronto intervento telefonando gratuitamente anche da cellulare al numero verde 800.735.735. Mentre per chiamare il Contact Center AQP per servizi commerciali 800.085.853 da un cellulare e dall'estero e necessario comporre il numero +39.080.5723498 (il costo della chiamata e secondo il piano tariffario del chiamante)."
Tags: canali, numero verde, cellulare
The previous FAQ is relevant for the query: "Si può telefonare da cellulare al numero verde?". We will provide a simple baseline based on a classical Information Retrieval model.
FAQs will be provided in CSV format using ‘;’ as separator. The file will be encoded in UTF-8 format. Each FAQ is described by the following fields:
- id: a number that uniquely identifies the FAQ;
- question: the question text of the current FAQ;
- answer: the answer text of the current FAQ;
- tag: a set of tags separated by ‘,’.
Test data will be provided as a text file composed by two strings separated by the TAB character. The first string is the user query id, while the second string is the text of the user query. For example:
1 Come posso telefonare al numero verde da un cellulare?
2 Come si effettua l’autolettura del contatore?
The participants must provide results in a text file. For each query in the test data the participants can provide max 25 answers ranked according to their system. Each line in the file must contain three values separated by the TAB character:
Systems will be ranked according to the accuracy@1. We will compute the precision of the system taking into account only the first answer. This metric will be used for the final task ranking. Moreover, we will provide other measures, such as: MAP, GMAP and MRR, for further analysis.
Information about the submission of results and their format are available here https://drive.google.com/open?id=0B9zwc2pyNmBkeG1faVYtR3BCR2s.
We invite the potential participants to subscribe to our mailing list in order to be kept up to date with the latest news related to the task. Please share comments and questions with the mailing list. The organizers will assist you for any potential issues that could be raised.
Participants will be required to provide an abstract and a technical report including a brief description of their approach, an illustration of their experiments, in particular techniques and resources used, and an analysis of their results for the publication in the Proceedings of contest; guidelines are outlined on the EVALITA's 2016 web site.
- TBD 2016: on-line registration opens
- 30th May 2016: development data available to participants
- 12th September 2016: test data available, registration closes
- 19th September 2016: system results due to organizers
- 26th September 2016: assessment returned to participants
- 19th October 2016: technical reports due to organizers
- 5th-7th December 2016: final workshop (one day during CLIC-2016)
 Preslav Nakov, Lluís Márquez, Walid Magdy, Alessandro Moschitti, Jim Glass, and Bilal Randeree. SemEval-2015 Task 3: Answer Selection in Community Question Answering. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), June 2015, Association for Computational Linguistics, 269-281, http://www.aclweb.org/anthology/S15-2047.
 Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria and Janyce Wiebe. SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), June 2015, Association for Computational Linguistics, 252-263, http://www.aclweb.org/anthology/S15-2045.
- Annalina Caputo, Dipartimento di Informatica, Università degli Studi di Bari Aldo Moro
- Marco de Gemmis, QuestionCube S.r.l.
- Pasquale Lops, Dipartimento di Informatica, Università degli Studi di Bari Aldo Moro
- Francesco Lovecchio, AQP S.p.a.
- Vito Manzari, SudSistemi S.r.l.
If you have any questions, please contact us: firstname.lastname@example.org.