Unlocking Information Access
The CLEF Initiative (Conference and Labs of the Evaluation Forum) is structured in two main parts:
- a series of Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities;
-
a peer-reviewed Conference on a broad range of issues, including
- investigation continuing the activities of the Evaluation Labs;
- experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities;
- research in evaluation methodologies and challenges.
The CLEF Initiative promotes research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. CLEF promotes research and development by providing an infrastructure for:
- multilingual and multimodal system testing, tuning and evaluation;
- investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access;
- creation of reusable test collections for benchmarking;
- exploration of new evaluation methodologies and innovative ways of using experimental data;
- discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge.
Bids to host CLEF 2027 are open
Deadline Friday, December 20th, 2024
Guidelines on submitting you bid can be found in the Template for Bids.
Submit your bid by email to the Steering Committee Chair at chair@clef-initiative.eu.
CLEF 2024 LNCS Proceedings, Part I
Volume 14958
The CLEF 2024 LNCS proceedings Part I are available at https://doi.org/10.1007/978-3-031-71736-9.
CLEF 2024 LNCS Proceedings, Part II
Volume 14959
The CLEF 2024 LNCS proceedings Part II are available at https://doi.org/10.1007/978-3-031-71908-0.
CLEF 2024 CEUR-WS Working Notes
Volume 3740
The CLEF 2024 CEUR-WS Working Notes are available at https://ceur-ws.org/Vol-3740/.
CLEF 2025 Conference
9-12 September 2025, Madrid, Spain
Please, visit the CLEF 2025 site for more information.
Book on 20 years of CLEF
The Information Retrieval Series, volume 4
The book "Information Retrieval Evaluation in a Changing World. Lessons Learned from 20 Years of CLEF" is available at https://doi.org/10.1007/978-3-030-22948-1.
Publications
The CLEF Initiative publishes two series of peer-reviewed papers:
- the conference proceedings, containing conference papers, condensed lab overviews, and best of labs, under the Springer Lecture Notes in Computer Science (LNCS) series;
- the working notes, containing extended lab overviews and lab participant papers, under the CEUR Workshop Proceedings (CEUR-WS.org) series.
Moreover, over the years the CLEF Initiative has published a few books related to multilingual and multimodal information access.
According to Google Scholar Metrics CLEF is among the top 20 venues for the "Databases and Information Systems" area.
- Recent
6 publications
- LNCS Proceedings
26 publications
- CEUR-WS Working Notes
25 publications
- Books
4 publications
- All
55 publications
Editions
CLEF started in 2000, under the name Cross-Language Evaluation Forum, as an event in conjunction with the European Conference for Digital Libraries (ECDL), now called Theory and Practice on Digital Libraries (TPDL).
Since 2010, CLEF has taken the form of an independent event, under the name Conference and Labs of the Evaluation Forum.
Labs Timeline
The evolution of the CLEF labs over time: click on the buttons to select the lab subset you are interested in.
Details about each lab are available on the Web sites of the corresponding edition.
Steering Commitee
The main purpose of the Steering Committee is to ensure the continuity, the growth, and the quality of the CLEF Initiative.
Nicola Ferro
University of Padua, Italy
Chair
Alberto Barrón-Cedeño
University of Bologna, Italy
Co-chair
Alba García Seco de Herrera
National Distance Education University (UNED), Spain
Co-chair
Paolo Rosso
Universitat Politècnica de València, Spain
Deputy Chair for the Conference
Martin Braschler
Zurich University of Applied Sciences (ZHAW), Switzerland
Deputy Chair for the Labs
Avi Arampatzis
Democritus University of Thrace, Greece
Member
Khalid Choukri
Evaluations and Language resources Distribution Agency (ELDA), France
Member
Fabio Crestani
Università della Svizzera italiana, Switzerland
Member
Carsten Eickhoff
University of Tübingen, Germany
Member
Norbert Fuhr
University of Duisburg-Essen, Germany
Member
Petra Galuščáková
University of Stavanger, Norway
Member
Anastasia Giachanou
Utrecht University, The Netherlands
Member
Lorraine Goeuriot
Université Grenoble Alpes, France
Member
Julio Gonzalo
National Distance Education University (UNED), Spain
Member
Donna Harman
National Institute for Standards and Technology (NIST), USA
Member
Bogdan Ionescu
University Politehnica of Bucharest, Romania
Member
Evangelos Kanoulas
University of Amsterdam, The Netherlands
Member
Birger Larsen
University of Aalborg, Denmark
Member
Maria Maistro
University of Copenhagen, Denmark
Member
Josiane Mothe
IRIT, Université de Toulouse, France
Member
Henning Müller
University of Applied Sciences Western Switzerland (HES-SO), Switzerland
Member
Jian-Yun Nie
Université de Montréal, Canada
Member
Gabriella Pasi
University of Milano-Bicocca, Italy
Member
Eric SanJuan
University of Avignon, France
Member
Laure Soulier
Sorbonne Université, France
Member
Theodora Tsikrika
Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH), Greece
MemberPast Members
Paul Clough
University of Sheffield, United Kingdom
Member
Djoerd Hiemstra
Radboud University, The Netherlands
Member
Jaana Kekäläinen
University of Tampere, Finland
Member
Seamus Lawless
Trinity College Dublin, Ireland
Member
David E. Losada
Universidade de Santiago de Compostela, Spain
Member
Mihai Lupu
Field consultant, Vienna, Austria
Member
Carol Peters
ISTI, National Council of Research (CNR), Italy
Chair (2000-2009)Founder of CLEF
Emanuele Pianta
Centre for the Evaluation of Language and Communication Technologies (CELCT), Italy
Member
Maarten de Rijke
University of Amsterdam, The Netherlands
Member
Giuseppe Santucci
Sapienza University of Rome, Italy
Member
Jacques Savoy
University of Neuchêtel, Switzerland
Member
Alan Smeaton
Dublin City University, Ireland
Member
Christa Womser-Hacker
University of Hildesheim, Germany
MemberAssociation
What is the CLEF Association?
The CLEF Association is an independent no-profit legal entity, established in October 2013 as a result of activity of the PROMISE Network of Excellence which backed CLEF from 2010 to 2013.
What is the mission of the CLEF Association?
The CLEF Association has scientific, cultural and educational objectives and operates in the field of information access systems and their evaluation. Its mission is:
-
to foster critical thinking about advancing information access and use from a technical, economic and societal perspective;
-
to promote access to information and use of evaluation.
What are the objectives of the CLEF Association?
Within its areas of interest, the CLEF Association aims at a better understanding of the use and access to information and how to improve this. The areas of interest stated in the the above mission translate into the following objectives:
-
clustering stakeholders with multidisciplinary competences and different needs, including academia, industry, education and other societal institutions;
-
facilitating medium/long-term research in information access and use and its evaluation;
-
increasing, transferring and applying expertise.
What are the activities of the CLEF Association?
The CLEF Association pursues its mission and objectives via the following pillar activities:
-
CLEF: sustains and promotes the popular CLEF evaluation series as well as providing support for its coordination, organisation, and running;
-
collections and experimental data: fosters the adoption and exploitation of large-scale shared experimental collections, makes them available under appropriate conditions and trusted channels, and shares experimental results and scientific data for comparison with state-of-the-art and for reuse;
-
infrastructure: supports the adoption and deployment of software and hardware infrastructures which facilitate the experimental evaluation process, the sharing of experimental collections and results, and interaction with and understanding of experimental data;
-
education and knowledge transfer: organises educational events, such as summer schools, and knowledge transfer activities, such as workshops, aimed not only at spreading know-how about information access and use but also at raising awareness and stimulating alternative viewpoints about the technical, economic, and societal implications.
What is the statute of the CLEF Association?
The complete statute of the CLEF Association is available in the following PDF document.
Other Initiatives
What are the other Evaluation Initiatives?
There are other "sister" evaluation initiatives around the world in the information access area, namely:
Contact
If you need any additional information about CLEF, please email us at chairclef-initiative.eu