Unlocking Information Access
The CLEF Initiative (Conference and Labs of the Evaluation Forum) is structured in two main parts:
- a series of Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities;
a peer-reviewed Conference on a broad range of issues, including
- investigation continuing the activities of the Evaluation Labs;
- experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities;
- research in evaluation methodologies and challenges.
The CLEF Initiative promotes research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. CLEF promotes research and development by providing an infrastructure for:
- multilingual and multimodal system testing, tuning and evaluation;
- investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access;
- creation of reusable test collections for benchmarking;
- exploration of new evaluation methodologies and innovative ways of using experimental data;
- discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge.
CLEF 2023 LNCS Proceedings
The CLEF 2023 LNCS proceedings are available at https://doi.org/10.1007/978-3-031-42448-9.
CLEF 2023 CEUR-WS Working Notes
The CLEF 2023 CEUR-WS Working Notes are available at https://ceur-ws.org/Vol-3497/.
CLEF 2024 Conference
9-12 September 2024, Grenoble, France
Please, visit the CLEF 2024 site for more information.
Book on 20 years of CLEF
The Information Retrieval Series, volume 4
The book "Information Retrieval Evaluation in a Changing World. Lessons Learned from 20 Years of CLEF" is available at https://doi.org/10.1007/978-3-030-22948-1.
The CLEF Initiative publishes two series of peer-reviewed papers:
- the conference proceedings, containing conference papers, condensed lab overviews, and best of labs, under the Springer Lecture Notes in Computer Science (LNCS) series;
- the working notes, containing extended lab overviews and lab participant papers, under the CEUR Workshop Proceedings (CEUR-WS.org) series.
Moreover, over the years the CLEF Initiative has published a few books related to multilingual and multimodal information access.
According to Google Scholar Metrics CLEF is among the top 20 venues for the "Databases and Information Systems" area.
- LNCS Proceedings
- CEUR-WS Working Notes
Early Detection of Mental Health Disorders by Social Media Monitoring. The First Five Years of the eRisk Project
Studies in Computational Intelligence, volume 1018, 2022
Information Retrieval Evaluation in a Changing World. Lessons Learned from 20 Years of CLEF
The Information Retrieval Series, volume 41, 2019
CLEF started in 2000, under the name Cross-Language Evaluation Forum, as an event in conjunction with the European Conference for Digital Libraries (ECDL), now called Theory and Practice on Digital Libraries (TPDL).
Since 2010, CLEF has taken the form of an independent event, under the name Conference and Labs of the Evaluation Forum.
9-12 September 2024
18-21 September 2023
5-8 September 2022
21-24 September 2021
Bucharest, Romania (virtual)
22-25 September 2020
Thessaloniki, Greece (virtual)
9-12 September 2019
10-14 September 2018
11-14 September 2017
5-8 September 2016
8-11 September 2015
15-18 September 2014
23-26 September 2013
17-20 September 2012
19-22 September 2011
Amsterdam, The Netherlands
20-23 September 2010
30 September-2 October 2009
17-19 September 2008
19-21 September 2007
20-22 September 2006
21-23 September 2005
15-17 September 2004
21-22 August 2003
19-20 September 2002
3-4 September 2001
21-22 September 2000
The evolution of the CLEF labs over time: click on the buttons to select the lab subset you are interested in.
Details about each lab are available on the Web sites of the corresponding edition.
The main purpose of the Steering Committee is to ensure the continuity, the growth, and the quality of the CLEF Initiative.
University of Padua, ItalyChair
Universitat Politècnica de València, SpainDeputy Chair for the Conference
Zurich University of Applied Sciences (ZHAW), SwitzerlandDeputy Chair for the Labs
Democritus University of Thrace, GreeceMember
University of Bologna, ItalyMember
Evaluations and Language resources Distribution Agency (ELDA), FranceMember
Università della Svizzera italiana, SwitzerlandMember
University of Tübingen, GermanyMember
University of Duisburg-Essen, GermanyMember
Utrecht University, The NetherlandsMember
Université Grenoble Alpes, FranceMember
National Distance Education University (UNED), SpainMember
National Institute for Standards and Technology (NIST), USAMember
University Politehnica of Bucharest, RomaniaMember
University of Amsterdam, The NetherlandsMember
University of Aalborg, DenmarkMember
David E. Losada
Universidade de Santiago de Compostela, SpainMember
Field consultant, Vienna, AustriaMember
University of Copenhagen, DenmarkMember
IRIT, Université de Toulouse, FranceMember
University of Applied Sciences Western Switzerland (HES-SO), SwitzerlandMember
Université de Montréal, CanadaMember
University of Milano-Bicocca, ItalyMember
University of Avignon, FranceMember
Sapienza University of Rome, ItalyMember
Pierre and Marie Curie University (Paris 6), FranceMember
Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), GreeceMember
University of Hildesheim, GermanyMember
University of Sheffield, United KingdomMember
Radboud University, The NetherlandsMember
University of Tampere, FinlandMember
Trinity College Dublin, IrelandMember
ISTI, National Council of Research (CNR), ItalyChair (2000-2009)
Founder of CLEF
Centre for the Evaluation of Language and Communication Technologies (CELCT), ItalyMember
Maarten de Rijke
University of Amsterdam, The NetherlandsMember
University of Neuchêtel, SwitzerlandMember
Dublin City University, IrelandMember
What is the CLEF Association?
The CLEF Association is an independent no-profit legal entity, established in October 2013 as a result of activity of the PROMISE Network of Excellence which backed CLEF from 2010 to 2013.
What is the mission of the CLEF Association?
The CLEF Association has scientific, cultural and educational objectives and operates in the field of information access systems and their evaluation. Its mission is:
to foster critical thinking about advancing information access and use from a technical, economic and societal perspective;
to promote access to information and use of evaluation.
What are the objectives of the CLEF Association?
Within its areas of interest, the CLEF Association aims at a better understanding of the use and access to information and how to improve this. The areas of interest stated in the the above mission translate into the following objectives:
clustering stakeholders with multidisciplinary competences and different needs, including academia, industry, education and other societal institutions;
facilitating medium/long-term research in information access and use and its evaluation;
increasing, transferring and applying expertise.
What are the activities of the CLEF Association?
The CLEF Association pursues its mission and objectives via the following pillar activities:
CLEF: sustains and promotes the popular CLEF evaluation series as well as providing support for its coordination, organisation, and running;
collections and experimental data: fosters the adoption and exploitation of large-scale shared experimental collections, makes them available under appropriate conditions and trusted channels, and shares experimental results and scientific data for comparison with state-of-the-art and for reuse;
infrastructure: supports the adoption and deployment of software and hardware infrastructures which facilitate the experimental evaluation process, the sharing of experimental collections and results, and interaction with and understanding of experimental data;
education and knowledge transfer: organises educational events, such as summer schools, and knowledge transfer activities, such as workshops, aimed not only at spreading know-how about information access and use but also at raising awareness and stimulating alternative viewpoints about the technical, economic, and societal implications.
What is the statute of the CLEF Association?
The complete statute of the CLEF Association is available in the following PDF document.
What are the other Evaluation Initiatives?
There are other "sister" evaluation initiatives around the world in the information access area, namely:
If you need any additional information about CLEF, please email us at