Scientific publications

As we engaged in designing digital interventions for intercultural dialogues around public cultural heritage sites, we saw an opportunity to surface multiple interpretations and points of view of history and shine a critical lens on current societal issues. To do so, we present the results of a collaborative auto-ethnography of alternative tours accompanied by intercultural guides, to explore sensory and embodied engagements with cultural heritage sites in a southern European capital. By focusing on the differences in how we experienced the heritage sites, we analyse the duality of discomfort, a common concept in HCI, in that it can both be deployed as a resource for designing systems that can transform people’s understanding of history or it can be a hindrance for engagement, having an unequal effect on individuals.

Recent methods for inductive reasoning on Knowledge Graphs (KGs) transform the link prediction problem into a graph classification task. They first extract a subgraph around each target link based on the -hop neighborhood of the target entities, encode the subgraphs using a Graph Neural Network (GNN), then learn a function that maps subgraph structural patterns to link existence. Although these methods have witnessed great successes, increasing often leads to an exponential expansion of the neighborhood, thereby degrading the GNN expressivity due to oversmoothing. In this paper, we formulate the subgraph extraction as a local clustering procedure that aims at sampling tightly-related subgraphs around the target links, based on a personalized PageRank (PPR) approach. Empirically, on three real-world KGs, we show that reasoning over subgraphs extracted by PPR-based local clustering can lead to a more accurate link prediction model than relying on neighbors within fixed hop distances. Furthermore, we investigate graph properties such as average clustering coefficient and node degree, and show that there is a relation between these and the performance of subgraph-based link prediction.

Re-identifying objects in a rigid scene across varying viewpoints (object Re-ID) is a challenging task, in particular when there are similar, even identical objects coexist in the same environment. Discriminative features play no doubt an essential role in addressing this challenge, while for practical deployment, real-time performance is another desired attribute. We therefore propose a novel framework, named Fast re-OBJ, that is able to improve both Re-ID accuracy and processing speed via tight coupling between the instance segmentation module and embedding generation module. The rich object encoding in the instance segmentation backbone is directly shared to the embedding generation module for training a more discriminative representation via a triplet network. Moreover, we create datasets with the segmentation outputs using real-time object detectors to train and evaluate our object embedding module. With extensive experiments, we prove that our proposed Fast re-OBJ improves the object Re-ID accuracy by 5% and the speed is 5× faster compared to the state-of-the-art methods.

Digital Storytelling can exploit numerous technologies and sources of information to support the creation, refinement and enhancement of a narrative. Research on text editing tools has created novel interactions that support authors in different stages of the creative process, such as the inclusion of crowd-generated content for writing. While these interactions have the potential to change workflows, integration of these in a way that is useful and matches users’ needs is unclear. In order to investigate the space of Assisted Storytelling, we designed and conducted a study to analyze how users write and edit a story about Cultural Heritage using an auxiliary source like Wikipedia. Through a diffractive analysis of stories, creative processes, and social and cultural contexts, we reflect and derive implications for design. These were applied to develop an AI-supported text editing tool using crowd-sourced content from Wikipedia and Wikidata.

Neural scene representations, such as Neural Radiance Fields (NeRF), are based on training a multilayer perceptron (MLP) using a set of color images with known poses. An increasing number of devices now produce RGB-D(color + depth) information, which has been shown to be very important for a wide range of tasks. Therefore, the aim of this paper is to investigate what improvements can be made to these promising implicit representations by incorporating depth information with the color images. In particular, the recently proposed Mip-NeRF approach, which uses conical frustums instead of rays for volume rendering, allows one to account for the varying area of a pixel with distance from the camera center. The proposed method additionally models depth uncertainty. This allows to address major limitations of NeRF-based approaches including improving the accuracy of geometry, reduced artifacts, faster training time, and shortened prediction time. Experiments are performed on well-known benchmark scenes, and comparisons show improved accuracy in scene geometry and photometric reconstruction,while reducing the training time by 3 - 5 times.

We solve object localisation in partial scenes, a new problem of estimating the unknown position of an object (e.g. where is the bag?) given a partial 3D scan of a scene. The proposed solution is based on a novel scene graph model, the Spatial Commonsense Graph (SCG), where objects are the nodes and edges define pairwise distances between them, enriched by concept nodes and relationships from a commonsense knowledge base. This allows SCG to better generalise its spatial inference to unknown 3D scenes. The SCG is used to estimate the unknown position of the target object in two steps: first, we feed the SCG into a novel Proximity Prediction Network, a graph neural network that uses attention to perform distance prediction between the node representing the target object and the nodes representing the observed objects in the SCG; second, we propose a Localisation Module based on circular intersection to estimate the object position using all the predicted pairwise distances in order to be independent of any reference system. We create a new dataset of partially reconstructed scenes to benchmark our method and baselines for object localisation in partial scenes, where our proposed method achieves the best localisation performance.

This chapter presents an experimental method designed to engage migrant participants with local cultural heritage. The initiative was part of an exploratory field study conducted in the context of the European-funded project MEMEX, a research effort promoting the social wellbeing of communities at risk of exclusion through the narration and collection of memories and stories related to cultural heritage. To engage members of such communities with the topic of cultural heritage, we deployed a two-stage intervention: a five-day photo-challenge, where participants were asked to photograph sites that they felt connected to, and a four-hour co-design workshop in which they explored the photos they had captured and co-created stories around specific sites, linking them to their memories. This chapter reflects how this process can benefit designers, individuals, and organizations in the cultural sector in capturing and reflecting on cultural heritage, engaging communities at risk of exclusion while supporting scientific and societal impact.

Specular highlights are commonplace in images, however, methods for detecting them and removing the phenomenon are particularly challenging. A reason for this is the difficulty in creating a dataset for training or evaluation, as in the real world, we lack the necessary control over the environment. Therefore, we propose a novel physically-based rendered LIGHT Specularity (LIGHTS) Dataset for the evaluation of the specular highlight detection task. Our dataset consists of 18 high-quality architectural scenes, where each scene is rendered with multiple views. In total, the dataset contains 2, 603 views with an average of 145 views per scene. Additionally, we propose a simple aggregation based method for specular highlight detection that outperforms prior work by 3.6% in two orders of magnitude less time on our dataset.