Workshop on Living Labs for Information Retrieval Evaluation @ CIKM 2013

1 November, 2013, San Francisco, USA

Workshop Proceedings

The workshop proceedings are now available in the ACM Digital Library.

Workshop Presentations

Some of the presentations made at the workshop are now available, as indicated below; see Schedule for full details.


  • IR Evaluation: Perspectives From Within a Living Lab. Georg Buscher, Microsoft Bing

Demo Papers

  • Using CrowdLogger for In Situ Information Retrieval System Evaluation. Henry A. Feild, James Allan
  • FindiLike: A Preference Driven Entity Search Engine for Evaluating Entity Retrieval and Opinion Summarization.
    Kavita Ganesan, ChengXiang Zhai
  • Lerot: an Online Learning to Rank Framework.
    Anne Schuth, Katja Hofmann, Shimon Whiteson, Maarten de Rijke

Position Papers

  • Evaluation for Operational IR Applications -Generalizability and Automation.
    Melanie Imhof, Martin Braschler, Preben Hansen, Stefan Rietberger
  • Factors Affecting Conditions of Trust in Participant Recruiting and Retention.
    Catherine L Smith

Short Papers

  • A Private Living Lab for Requirements Based Evaluation.
    Christian Beutenmüller, Stefan Bordag, Ramin Assadollahi
  • A Month in the Life of a Production News Recommender System.
    Alan Said, Jimmy Lin, Alejandro Bellogín, Arjen de Vries

Best Demo Award

The best demo award (€ 750) went to Kavita Ganesan and ChengXiang Zhai for FindiLike.

The Best Demo Award winners received an award of 750 EUR, offered by the ‘Evaluating Information Access Systems’ (ELIAS) ESF Research Networking Programme. The award can be used to cover travel, accommodation or other expenses in relation to attending and/or demo’ing at LL’13.


The workshop was supported by the ‘Evaluating Information Access Systems’ (ELIAS) ESF Research Networking Programme.