Important Dates

Can I just use the data and not take part in the collaborative evaluation?
To gain access to the data you must participate in the
collaborative evaluation.

Where can I get help to learn how to work with this data?
In order to jumpstart the progress for participating organisations we provide a Jupyter/Python notebook (available upon registration). This includes fully working classification pipelines using the well known sklearn python library. Similarly it shows how to use the online results evaluation system.

What is RSVP?

RSVP (Rapid Serial Visual Presentation) involves presenting image stimuli at fast presentation rates. This technique can be used to study visual and perceptual processes, and when used in tandem with EEG as a strategy to label 'interesting' image stimuli.
What this type of approach takes advantage of: a) The speed of human visual recognition/processing i.e. we can often present these images at a very fast rate (example to follow) via RSVP (Rapid Serial Visual Presentation) and b) The on-the-fly capability of defining search tasks for users (within some constraints)

What is EEG?
EEG (Electroencephalography) involves the detection of electrical activity generated by the brain by affixing electrodes to the scalp. These signals when coupled with signal processing and machine learning techniques allow measurement of cognitive processes including attentional orientation to image stimuli. An EEG recording comprises of a discrete time-series measurement of voltage changes across the scalp (on a per electrode basis). 32 or 64 electrodes are typically used. In the NAILS task we are interested in these signals offset to image presentation events. The extraction of epochs (time periods) offset to each image presentation has already been performed to assist users of the data. Raw data (including triggers) can be made available upon request for those who wish to perform their own preprocessing/epoching on the dataset.

How were EEG hardware triggers/markers captured?
Triggers/markers were captured via a photodiode placed in the upper right corner of the screen. On each image change in the RSVP sequence, this corresponding patch of the screen changes between black and white repeatedly. As these time-locked changes to the image presentation are detected a marker is directly inserted on the EEG system via its serial port. This offers excellent time-synchronisation.

Who were the participants in the experiment used to capture the EEG?
Participants were recruited from the student and staff body of Dublin City University (ethic no: DCUREC/2016/099).

Were participants familiar with RSVP and the tasks in advance?
Participants had varying levels of familiarity with RSVP image-search tasks but had all completed a few sample trial blocks prior to the start of the experiment to ensure familiarity with the attentional/perceptual requirements of the upcoming tasks. These tasks did not use any overlapping stimuli with the main NAILS tasks i.e participants in the experiment were not familiar with any specific NAILS task prior to engaging in it.

Can I have some more information on how the experiment was run?
Yes, contact us at me@grahamhealy.com

Why were these tasks used?
These tasks were selected to fulfil three criteria:
1) task ordering carry-over effects should not be present e.g. it's not sensible to use a task say to search for 'people' in images in one task where in a later different search task images of 'people' might be naturally part of the non-relevant images potentially generating false positives.
2) to have varying levels of difficulty,
3) for it to be possible for an experiment participant to reasonably complete i.e. conducive to eliciting P300-related signal phenomena.

Some more background on EEG?
EEG (Electroencephalography) has recently become an accessible method for researchers and users to build and operate BCI (Brain-Computer Interface) applications, primarily because of the availability of a new generation of low-cost devices. While the initial use of such techniques began in clinical/rehabilitative settings for the purposes of augmenting communication and control, a recent trend has been to use such signals and methods in new domains, such as the image annotation task, which relies on the identification of target brain events to trigger labeling.

This trend is particularly relevant to the IR community as in recent years EEG has become a promising technology for the purposes of annotating multimedia content, or identifying when a user’s attention is drawn to something in the real world, or even as a source of user sensor data to be indexed for later retrieval. However, working with EEG data is challenging and contains many pitfalls for inexperienced researchers.

What is this P300? Are there other signal phenomena that can be used to classify 'target' images?
The P300 is a commonly studied endogenous-type potential that is linked with attentional-orientation of the type that arises when attention is drawn to a significant environmental stimulus. In BCI image-search applications it is typically identified as contributing the most useful discriminative information that can be used with a machine-learning strategy to detect target images in an RSVP search task, and particularly in instances where time-series features alone are used as the basis of building a machine-learning model. The response itself is difficult to directly see in the raw EEG signals in response to 'target' events due to issues like high SNR and trial-to-trial variability, however, using a variety of techniques it's possible to identify its response locked nature to target presentation events (~ 500ms after).

Other types of ERP/ERSP phenomena are often present alongside the P300 that can be typically visualised by means of using event-related potential averaging or via ERSP(Event-related Spectral Perturbation)/ITC (Inter-trial Coherence) revealing complex frequency responses that can be time-locked or non time-locked to the relevant image presentation event. For this reason, we provide accompanying time-frequency-based transformations (features) of the data to allow users of the dataset to explore more advanced features of the EEG that might not be present in the time-series/raw EEG recording.

Were padding images used in the experiment?
Yes, prior to and following each block 10 standard padding images are used.

What is some suggested further reading?

- Blankertz B, Lemm S, Treder M, Haufe S, Müller KR. Single-trial analysis and classification of ERP components — A tutorial (2011). NeuroImage Volume 56, Issue 2, 15 May 2011, Pages 814–825

- Lemma S, Blankertz B, Dickhausa T, Müllera KR. Introduction to machine learning for brain imaging (2011).
NeuroImage Volume 56, Issue 2, 15 May 2011, Pages 387–399



- Deng J, Dong W, Socher R, Li LJ, Li K and Fei-Fei L. ImageNet: A Large-Scale Hierarchical Image Database (2009). Computer Vision and Pattern Recognition (CVPR), 2009.

- Thorpe S, Fize D, Marlot C. Speed of processing in the human visual system (1996).
Nature. 1996 Jun 6;381(6582):520-2.