Ten ideas for research projects for the practical assignment of the IR course 2019/2020:

Assignment 1 - Multiple topic variants

Investigate the effect on retrieval performance of combination of multiple representations of topics. For this assignment a system like anserini would be a good choice.

Background reading: a recent query variations paper by Shane Culpepper; the UQV 100 dataset; classic paper on rank fusion.

Features: setup your own experiment, some tuning / configuring of an IR system.

You could choose to focus your work on test collection building instead of experimentation, perhaps mixing assignment 1 and 3.

Assignment 2 - User study

Evaluate an already existing information retrieval system by doing a user study.

An example study could for example investigate the query logs of users of a retrieval system and investigate how they adjust their queries to find a specific piece of information.

Another example study is investigating search behaviour in voice-only search systems (e.g., Siri and Google now), and see how and when users use these systems for their complex informaition needs.

Background readings: Diane Kelly’s FnTIR on interactive IR evaluation, and Trippas paper.

Features: setup your own user study to test an hypothesis. Limited programming, primarily for analyzing collected data (could be just Excel).

Assignment 3 - Search in Personal Web Archives

Evaluate the WASP system and/or extensions. Install WASP, collect data, and carry out an evaluation of the system.

Background reading: the WASP paper, and, optionally, Gijs Hendriksen’s BSc thesis on extending WASP with context (repo).

Features: collect your own data, not too much programming.

Assignment 4 - DBPedia

Carry out an entity retrieval evaluation using the DBPedia Entity V2 collection, for example by making use of the Nordlys toolkit.

Dataset: tiny.cc/dbpedia-entity or here.

The dataset is described in detail in the SIGIR 2017 resource paper and presented at SIGIR 2017 using this poster.

If you are not yet familiar with RDF, you may want to glance over the W3C RDF Primer.

The topic of entity ranking is introduced in lecture 5 of the course. Other useful background is given in the RUSSIR lecture slides (courtesy Krisztian Balog), where you find a brief history of the various resources that have been creaAted in the IR community.

Features: some challenge in handling the data. (Limited) prior programming experience is handy.

Assignment 5 - Build your own IR system

Build your own information retrieval system and carry out an experiment with it.

Background reading: the UMass CIIR book is excellent.

Features: no better way to learn how things work than by doing it from scratch. Need considerable programming experience.

Assignment 6 - IR systems in different search contexts

Compare the results / suitability / usefulness / … of a single system (e.g., Google/Bing) in different search contexts. For example, compare the effects of searching on a certain device, having GPS available, with/without previous sessions, using different languages, etc.

Background reading: Diane Kelly’s FnTIR on interactive IR evaluation.

Features: Setup your own user study to test an hypothesis. Limited programming, primarily for analyzing collected data.

Assignment 7 - NLP for Information Retrieval

Incorporate your favorite NLP technique for retrieval, for example by using entity linking, query expansion, etc. Compare the results against a standard system, for example anserini.

Features: Combine/enrich your knowledge of NLP with Information Retrieval. Some programming knowledge would be nice.

Assignment 8 - Focus on efficiency

Test an information retrieval system on efficiency and/or scalability aspects. Design an experiment and investigate the effects of certain parameter setting, and test what the effect is on the efficiency of the system (when carrying out many queries quickly).

Background reading: checkout some of the references in Shane Culpepper’s keynote at DESIRES 2018.

Features: create understanding of inner workings of retrieval system, carry out a performance measurement.

Assignment 9 - NewDog (a.k.a. “impress your prof”)

Your IR professor has a long standing interest in integrating IR and databases.

Consider the OldDog blogpost and accompanying code as a starting point, also available as an easily replicable OSIRRC image. OldDog uses MonetDB, a surprisingly complete and reliable analytical database system with extensive SQL support; it builds on a SIGIR demo paper.

DuckDB is the newest project by the CWI database kernel hackers and aims to replace SQLLite for analytical workloads. If you are an excellent programmer and not afraid of a bit of C++ code, you will create the same setup but using instead of MonetDB, and compare their runtime performance differences (if any - and, see also assignment 8).

We would be especially interested in experimental work using DuckDB with the Python API.

If things go smooth, you succeed in creating NewDog and carry on with elements of assignment 7 using standard python libraries… to score the ultimate grade for this assignment.

Assignment 10 - Measuring personalization in search

Evaluate the personalization done by well-known existing information retrieval systems. For this project, you could, for instance, measure the impact of personalization on search results (for different queries) or compare the personalization of different search engines. Additionally, the project could focus on evaluating or quantifying the differences (e.g. how do you weigh the relative position of the search results in calculating the differences) or on visualizing personalization or filter bubbles.

Background reading: a study by Hannák et al. (2017) comparing personalization across different search engines. For their experimental design they use constructed user accounts; a study by Salehi, Du and Ashman (2015), that also uses constructed user accounts, to measure the difference between personalized and non-personalized search results on Google. This study specifically addressed search within the educational context, but other contexts might also be interesting (think about how the context might affect the design of your experiment); a study by Dillahunt, Brooks and Gulati (2015) that uses actual users to measure the impact of personalization. The last study also focused on visualizing the impact of personalization on search results.

Features: not too much programming, collecting your own search results data, constructing fake user accounts or doing a study with users, evaluation/quantification of differences between search results, techniques for the visualization of personalization in search.

Assignment 11 - Re-ranking for diversity of search results

With talk about filter bubbles and bias, as well as Google recently announcing that they are aiming to show results from more diverse domains, it might be interesting to devote your project on re-ranking search results from existing information retrieval systems to prioritise diversity more. Here, multiple types of diversity are possible (e.g. content diversity, domain diversity, etcetera).

Background reading: A seminal paper by Carbonell and Goldstein (1998) about promoting diversity/novelty, as a way to combat redundancy. A probabilistic re-ranking method, proposed by Huang and Hu, that promotes diversity for biomedical information retrieval. A survey article by Kunaver and Požrl (2017) about diversity methods for recommender systems. Though the article focuses on recommender systems, it could still provide some inspiration for techniques that can be applied to information retrieval.

Features: some tuning/configuring of an IR system.

Assignment X - Bring us your own proposal

It would be great if you have a cool idea of your own, we encourage any proposals that do not fall in the scope of the other assignments, but do fall in the scope of the course!

Find help on the forum, or go back to main assignment page.