We are happy to announce that our living labs platform will continue to operate as the LL4IR Challenge. The second evaluation round, using the same setup and test queries as the CLEF 2015 Lab, will run from June 15 to 30. Note that late-breaking results from round #2 can still be included in the CLEF working notes papers.
Today, we are officially entering into the test phase of the LL4IR CLEF 2015 Lab.
There are some changes, detailed on this page, below is an executive summary:
- Only 2 of the 3 use-cases will be evaluated: product search and web search (local-domain search will not run this year). These two use-cases are fully operational; see this page for updated technical details on the product search use-case.
- The test period will run from May 1 to 15. Rankings for test queries may be uploaded (and changed) until Apr 30, 23:59 CET.
- Training queries will continue to run during the test phase and feedback will be made available for them; however, it will not be possible to upload new rankings (either for training or for test queries) during the test period.
If you are planning on submitting a run, please let us know by sending us a short message.
There has been a silence on the site over the last few weeks (months), but things have been progressing. The web search use case is operational and running in full swing for some weeks now. We will have more updates on the other two use cases very soon.
You can still sign up for CLEF until April 30. Note that the deadline for submitting official runs is April 15.
We have also prepared a guide specifically for CLEF participants; this should help you to get started right away.
We are happy to announce that Living Labs for IR Evaluation (LL4IR) will run as a Lab at CLEF 2015. The first edition of the lab will focus on three specific use-cases: (1) product search (on an e-commerce site), (2) local domain search (on a university’s website), (3) web search (through a commercial web search engine). See futher details here.
“Give us your ranking, we’ll have it clicked!”