Living Labs Challenge workshop

6 June, 2014, Amsterdam, The Netherlands

The aim of the workshop is to test and further develop a living labs for IR methodology and a living labs for IR platform, to be used in a community challenge, with two specific use-cases: product search and local domain search. The programme for the workshop includes (1) invited talks and discussions with invited guests to learn from their knowledge in this space, and (2) a hackathon with prospective participants for testing the required challenge components.

The documentation and details of the challenge API is available at:
Note that this API is in an alpha stage and is likely to change (possibly due to your feedback through


The workshop is free and open to all interested participants, but registration is required.
Registration closed on May 29.

Access to and use of the challenge API will be subject to agreement with the terms of usage.

Date and Location

The workshop will take place at the University of Amsterdam on the 6th of June, 2014.

Practical information on how to get there can be found on this page.


09:30-10:00 Opening
10:00-12:00 Invited talks
by CLEF NEWREEL organisers (TU Berlin and Plista Gmbh)
followed by an interactive Q&A session
12:00-13:00 Lunch
13:00-16:00 Hackathon
16:00-17:00 Wrap-up and conclusions
17:00-19:00 Dinner

Invited talks

Setting Up a Living Lab for Information Access Research

Frank Hopfgartner (Technische Universität Berlin, Germany)

Living Labs are real-life test and experimentation environments in which users and producers are brought together to develop, validate and test novel techniques and approaches. Primarily set up to observe users interacting with technology in a real-life context, they are an interesting methodology to push forward user-centric research. In this talk, I will make a case for the introduction of a living lab to promote research on interactive information retrieval. Moreover, I will introduce CLEF NewsReel, a living lab where participants are invited to evaluate news recommendation techniques in real-time by providing news recommendations to actual users that visit commercial news portals to satisfy their information needs.

Open Recommendation Platform for Researchers & Developers

Torben Brodt (plista GmbH)
Plista is a data-driven content and advertising platform. This talk will be held by Torben Brodt who developed the first plista algorithm and now leads the data-engineering department, which is responsible for server-scalability and algorithm-quality. His team is looking for the best algorithm for each context and thats why plista has a deep link to research and is open to any contribution from the research. This talk shows how researchers can easily start working together with plista, will show how the system was build and how it acts in the background.

Workshop Summary

The report summarizing the content of and discussions at the event is available here.

Workshop Organisers

Krisztian Balog, University of Stavanger, Norway (krisztian.balog (at)
Liadh Kelly, Dublin City University, Ireland (liadh.kelly (at)
Anne Schuth, University of Amsterdam, The Netherlands (anne.schuth (at)


The workshop is supported by the ‘Evaluating Information Access Systems’ (ELIAS) ESF Research Networking Programme.