The 2nd Living Labs for Information Retrieval Evaluation Workshop @ CIKM 2014
7 November, 2014, Shanghai, China
Call for Papers & Demos – Submission deadline: August 16
Challenge Track – Submission deadline: August 30
In the past few years, a new evaluation methodology known as living labs has been proposed as a way for researchers to be able to perform in-situ evaluation which involve and integrate users within the research process. The basic idea of living labs for information retrieval (IR) is that rather than individual research groups independently developing experimental search infrastructures and gathering their own groups of test searchers for IR evaluations, a central and shared experimental environment is developed to facilitate the sharing of resources in a live setting (most important of all: users).
Living labs would offer huge benefits to the community, such as: availability of, potentially larger, cohorts of real users and their behaviours, e.g. querying behaviours, for experiment purposes; cross-comparability across research centres; and greater knowledge transfer between industry and academia, when industry partners are involved. The need for this methodology is further amplified by the increased reliance of IR approaches on proprietary data; living labs are a way to bridge the data divide between academia and industry. Progress towards realising actual living labs has nevertheless been limited. There are many challenges to be overcome before the benefits associated with living labs for IR can be realised, including challenges associated with living labs architecture and design, hosting, maintenance, security, privacy, participant recruiting, and scenarios and tasks for use development.
The 1st Living Labs for Information Retrieval Evaluation (LL’13) workshop at CIKM 2013 was a first attempt to bring people, both from academia and industry, together to discuss challenges and to formulate practical next steps. The workshop was successful in identifying and documenting possible further directions. The goal of the second edition of the workshop is to continue our community building efforts around living labs for IR and to pursue the directions set out at LL’13. As part of this, a challenge with shared tasks in the e-commerce space and local domain search is planned.
Possible use cases in this space include, but are not limited to:
- The e-commerce domain (i.e., product search and recommendation)
- Local domain search (e.g., university search engines)
- The personal search space (search of personal computer files, emails, web pages looked at, etc.)
- Medical information retrieval (e.g., patients searching for medical information on the Internet)
- Searching Wikipedia (an open-source collection with fewer privacy concerns than, say, personal search)
Workshop topics include, but are not limited to:
- Privacy and security:
- Hosting data on secure server
- Gaining subjects trust
- Coping with individuals need for privacy
- Alternates when individuals will not share their data
- Legal and ethical issues:
- User consent
- Ethics approval
- Legalities regarding release of data
- Trust between parties
- Copyright issues
- Commercial sensitivity of interaction data
- Technical challenges:
- Designing and implementing living labs architecture
- Cost of implementation
- Maintenance and adoption
- Managing living labs infrastructure
- Practical challenges:
- Forming living labs for IR partners within the research community
- Obtaining commercial partners
- Alternates when commercial partners cannot be obtained
- Defining tasks and scenarios for evaluation purposes
The workshop is now accepting paper submissions. Short papers (4 pages), position papers (2 pages), and posters (2 pages) describing approaches or ideas / challenges on the topics of the workshop are invited. Submissions should be in ACM SIGS format. LaTeX and Word templates are available at http://www.acm.org/sigs/publications/proceedings-templates (for LaTeX, use the “Option 2″ style).
Papers should be anonymised for double blind review and submitted in PDF format through the EasyChair system no later than midnight Pacific Daylight Time on August 16, 2014. Submissions will be reviewed by members of the workshop program committee. Accepted papers will be included in the CIKM 2014 Living Labs for Information Retrieval Evaluation Workshop proceedings, which will be indexed in the ACM Digital Library.
We invite researchers and practitioners to present their innovative prototypes or practical developments in a dedicated demo track at the workshop. Demo submissions must be based on an implemented system that pursues one or more aspects relevant to the interest areas of the workshop.
Authors are strongly encouraged to target scenarios that are rooted in real-world applications. One way to think about this is by considering the following: as a company operating a website/service/application, what methods could allow various academic groups to experiment with specific components of this website/service/application?
In particular, we seek prototypes that define specific component(s) in the context of some website/service/application, and allow for the testing and evaluation of alternative methods for that component. One example is search within a specific vertical (such as product or travel search engine), but we encourage authors to think outside the (search) box.
Demonstration papers (max. 4 pages) must be submitted electronically through the EasyChair system, in PDF format and formatted using the ACM SIGS format (LaTeX and Word templates are available at http://www.acm.org/sigs/publications/proceedings-templates; for LaTeX, use the “Option 2″ style). Each submission should either include a URL for the web site of the system, or a URL of a shared code repository, or a URL to a 2-3 minute video demonstrating the system. The review process for demonstration submission is *not* double blind. Submissions should be made no later than midnight Pacific Daylight Time on August 16, 2014. Submissions will be reviewed by members of the workshop program committee. Accepted demo papers will be included in the CIKM 2014 Living Labs for Information Retrieval Evaluation Workshop proceedings, which will be indexed in the ACM Digital Library.
Challenge Track Submissions
The workshop also features a living labs challenge. The challenge focuses on two use-cases: product search (on an e-commerce site) and local domain search (on a university’s website). The challenge will run from August 1 – October 1. During the first month of the challenge participants will have the opportunity to tune their system through the product and local domain search websites. The second month of the challenge will examine the performance of participants’ systems on these search websites. Further details are available at: http://living-labs.net/challenge/.
Challenge papers (max. 4 pages) should describe the participant’s system design approach and any results available by the submission deadline. Papers must be submitted electronically through the EasyChair system, in PDF format and formatted using the ACM SIGS format (LaTeX and Word templates are available at http://www.acm.org/sigs/publications/proceedings-templates; for LaTeX, use the “Option 2″ style). The review process for challenge submissions is *not* double blind. Submissions should be made no later than midnight Pacific Daylight Time on August 30, 2014. Submissions will be reviewed by members of the workshop program committee. Accepted challenge papers will be included in the CIKM 2014 Living Labs for Information Retrieval Evaluation Workshop proceedings, which will be indexed in the ACM Digital Library.
August 16, 2014 – Paper & Demo Track submission deadline (midnight Pacific Daylight Time)
August 30, 2014 – Individual challenge papers due (midnight Pacific Daylight Time) [Challenge runs August 1 – October 1]
August 30, 2014 – Notification to paper & demo track authors
September 4, 2014 – Notification to challenge paper authors
September 10, 2014 – Camera-ready papers due
November 7, 2014 – Workshop
Further information is available on the workshop website (http://living-labs.net/ll14/) or by emailing the workshop organisers.
Krisztian Balog – University of Stavanger, Norway (krisztian.balog (at) uis.no)
Liadh Kelly – Dublin City University, Ireland (liadh.kelly (at) computing.dcu.ie)
Anne Schuth – University of Amsterdam, The Netherlands (anne.schuth (at) uva.nl)