Test period begins for CLEF 2015 Lab

Today, we are officially entering into the test phase of the LL4IR CLEF 2015 Lab.

There are some changes, detailed on this page, below is an executive summary:

  • Only 2 of the 3 use-cases will be evaluated: product search and web search (local-domain search will not run this year). These two use-cases are fully operational; see this page for updated technical details on the product search use-case.
  • The test period will run from May 1 to 15. Rankings for test queries may be uploaded (and changed) until Apr 30, 23:59 CET.
  • Training queries will continue to run during the test phase and feedback will be made available for them; however, it will not be possible to upload new rankings (either for training or for test queries) during the test period.

If you are planning on submitting a run, please let us know by sending us a short message.

CLEF Lab is underway

There has been a silence on the site over the last few weeks (months), but things have been progressing. The web search use case is operational and running in full swing for some weeks now. We will have more updates on the other two use cases very soon.

You can still sign up for CLEF until April 30. Note that the deadline for submitting official runs is April 15.
We have also prepared a guide specifically for CLEF participants; this should help you to get started right away.

Living Labs runs as a CLEF Lab in 2015

We are happy to announce that Living Labs for IR Evaluation (LL4IR) will run as a Lab at CLEF 2015. The first edition of the lab will focus on three specific use-cases: (1) product search (on an e-commerce site), (2) local domain search (on a university’s website), (3) web search (through a commercial web search engine). See futher details here.

“Give us your ranking, we’ll have it clicked!”

LL’14 deadline extension

The submission deadline for our LL’14 workshop at CIKM’14 has been extended to Aug 16. We are particularly interested to receive position papers (2 pages ACM) on the living labs experimental paradigm.

Our living labs challenge is underway since Aug 1. We are also soliciting “work-in-progress” challenge papers for the workshop, describing approaches, early stage ideas, and preliminary results. These may be submitted until Aug 30.
To receive updates on the challenge, please subscribe to our mailing list.

Living Labs challenge workshop in Amsterdam

In preparation for the challenge we are organising for LL’14, a dedicated Living Labs Challenge workshop will be held in Amsterdam on the 6th of June. The programme includes invited talks from CLEF NEWSREEL organisers and a “hackathon”. We are kindly inviting prospective challenge participants as well as potential industrial partners to attend this event!

We thank the ELIAS programme of ESF for their financial support, and the University of Amsterdam, and the ILPS group in particular, for hosting the event.