« Zurück

Mercier Julien

Chargé de Ra&D HES

Hauptvertrag

Chargé de Ra&D HES

Büro: S04

Haute école d'Ingénierie et de Gestion du Canton de Vaud
Route de Cheseaux 1, 1400 Yverdon-les-Bains, CH

MSc UNIGE/HES-SO en Développement territorial - HES-SO Master

augmented reality
location-based augmented reality
augmented reality for geoinformation

BSc HES-SO en Ingénierie des médias - Haute école d'Ingénierie et de Gestion du Canton de Vaud

user experience design
creative artificial intelligence
mobile eye tracking
augmented reality

Abgeschlossen

Building Location-based Augmented Reality Applications through Advanced Pedagogical Methods to Support Learning about Biodiversity

AGP

Rolle: Mitarbeiter

Requérant(e)s: MEI

Financement: FNS

Description du projet : The project focuses on the impact of using location-based augmented reality (AR) technologies for teaching and learning biodiviersity. These technologies allow for showing and revealing information that is not visible in the real world - for instance virtual objects displayed on top of real-world objects or contextual virtual information that is given about a real-world object. Applying AR technologies in the context of biodiversity education is promising, as it requires pupils to interpret and understand natural phenomena located in space such as plants. The goal of this multidisciplinary project is to improve the design, development and use of technology-based teaching practices, in particular location-based Mobile Augmented Reality (MAR) for children and young adults in the learning about biodiversity. In that sense, this overall purpose fits the second thematic cluster of the NPR77 module about 'Education, learning and digital change', that is about transformation of learning and assessment processes, and digital infrastructures, methods, instruments and environments.

Forschungsteam innerhalb von HES-SO: Ertz Olivier , Mercier Julien

Partenaires académiques: MEI; Haute Ecole pédagogique

Durée du projet: 01.10.2020 - 30.09.2024

Montant global du projet: 446'968 CHF

Statut: Abgeschlossen

Réalité augmentée, mémoire diminuée, quel avenir pour nos profils de compétences?

AGP

Rolle: Mitarbeiter

Requérant(e)s: MEI

Financement: HES-SO Rectorat

Description du projet : La réalité augmentée (RA) fait son apparition dans le monde de l'entreprise, avec des promesses de productivité et de performance accrues. Elle permet un accès en temps réel à tout type d'information, doublé d'un effort cognitif faible. Elle propose déjà de nombreuses applications d'assistance, ciblant des manipulations techniques qui nécessitaient, autrefois, la mémorisation de savoir-faire complexes. Du point de vue du travail qualifié, la RA peut-elle favoriser le processus de formation et ainsi faciliter la transition professionnelle, ou est-elle vouée à dévaloriser les profils de compétences en les rendant interchangeables et fongibles ? Nous proposons une expérience en milieu contrôlé qui mesurera les effets de l'utilisation de la RA sur l'acquisition et la stabilisation de compétences, afin d'apporter des réponses aux incertitudes liées à la transformation programmée de certains emplois qualifiés.

Forschungsteam innerhalb von HES-SO: Dozot Grégory , Rappo Daniel , Guillaume Sébastien , Chabloz Nicolas , Vaucher Dylan , Morel Kilian , Délèze Fabien , Mercier Julien , Casto Matteo

Partenaires académiques: MEI

Durée du projet: 01.11.2021 - 20.12.2023

Montant global du projet: 69'326 CHF

Statut: Abgeschlossen

2024

Quantifying dwell time with location-based augmented reality :

Wissenschaftlicher Artikel ArODES

dynamic AOI analysis on mobile eye tracking data with vision transformer

Julien Mercier, Olivier Ertz, Erwan Bocher

Journal of Eye Movement Research, 2024, 17, 3, 3

Link zur Publikation

Zusammenfassung:

Mobile eye tracking captures egocentric vision and is well-suited for naturalistic studies. However, its data is noisy, especially when acquired outdoor with multiple participants over several sessions. Area of interest analysis on moving targets is difficult because A) camera and objects move nonlinearly and may disappear/reappear from the scene; and B) off-the-shelf analysis tools are limited to linearly moving objects. As a result, researchers resort to time-consuming manual annotation, which limits the use of mobile eye tracking in naturalistic studies. We introduce a method based on a fine-tuned Vision Transformer (ViT) model for classifying frames with overlaying gaze markers. After fine-tuning a model on a manually labelled training set made of 1.98% (=7845 frames) of our entire data for three epochs, our model reached 99.34% accuracy as evaluated on hold-out data. We used the method to quantify participants’ dwell time on a tablet during the outdoor user test of a mobile augmented reality application for biodiversity education. We discuss the benefits and limitations of our approach and its potential to be applied to other contexts.

Motion Capture Data as Machine-Readable Notation to Capture Muscial Interpretation

Wissenschaftlicher Artikel

Experimenting with Movement Sonification and Synthesis

Mercier Julien, Irini Kalaitzidi

Proceedings of the International Conference on Technologies for Music Notation and Representation (TENOR 2024), 2024 , pp. 169-171

Link zur Publikation

Zusammenfassung:

Musical notation can be described as an abstract language that composers use so that performers may interprets a score. Such notation comes before interpretation, is reproducible, and although it contains hints targeted at the performer on how to interpret, the actual performance is uniquely situated in time and space. The only way to record and re-experience a sound performance is to use microphones, who transform acoustic waves into an electrical signal and usually lose at least some spatial dimension in the process. This may be hindering in the field of experimental music, where physical limits of sound material may be put to the test. In this short paper, we discuss how motion capture could be an alternative to, or an expansion of the acoustic recording of a performance involving movement. By recording the performer’s movements, some of the dimensions that make their interpretation singular (i.e. character, accentuation, phrasing, and nuance) are retained. A method capturing sound through movement may be interesting in the context of sound synthesis with deep learning and hold potential advantages over current methods using MIDI or acoustic, that either lack dimensions or are very sensitive to noisy data. We briefly discuss rationale, practical and theoretical foundations for the development of potentially innovative outputs.

In Situ Visualization of 6DoF Georeferenced Historical Photographs in Location-Based Augmented Reality

Buchkapitel

Mercier Julien, Ertz Olivier, Erwan Bocher

Dans Lotfian Maryam, Starace Luigi Libero Lucio, Web and Wireless Geographical Information Systems. 2024, CH-6330 Cham, Switzerland : Springer Nature Switzerland AG

Link zur Publikation

Zusammenfassung:

Location-based augmented reality (AR) enables in situ visualization of geolocated media in the real world, usually based on latitude and longitude referencing only. By integrating 6DoF georeferenced photographs, they can be positioned along all three rotational axes (pitch, yaw, roll) in addition to the usual perpendicular translation axes (surge, heave, sway), resulting in data registered on all six mechanical degrees of freedom of movement of a rigid body in three-dimensional space. In this AQ1 paper, we present an integration test of 6DoF georeferenced data from the Smapshot open API to the BiodivAR open authoring tool for locationbased AR. While both projects are open source, their data is not interoperable, similarly to most citizen science (CS) projects. After mapping the data and importing it in a new augmented environment, it can be visualized in the mobile AR interface, at the locations specified by the georeferenced images. Visualizing historical photographs in the AR interface from the actual location where they were originally taken allows them to align with their original context, resulting in an eye-catching and impactful visualization. This contextual visualization of historical photographs informs and enriches their meaning, and allows viewers to quickly detect patterns and anomalies at a glance. We perform a basic qualitative evaluation of the visualization made possible by the combination of 6DoF georeferenced data and our authoring system, based on our own observations. We discuss the potential of this proof-of-concept to foster participation and understanding in citizen science and education, especially with regards to biodiversity monitoring and education.

2023

BiodivAR: :

Wissenschaftlicher Artikel ArODES

a cartographic authoring tool for the visualization of geolocated media in augmented reality

Julien Mercier, Nicolas Chabloz, Gregory Dozot, Olivier Ertz, Erwan Bocher, Daniel Rappo

ISPRS International Journal of Geo-Information, 2023, vol. 12, no. 2, article no. 61

Link zur Publikation

Zusammenfassung:

Location-based augmented reality technology for real-world, outdoor experiences is rapidly gaining in popularity in a variety of fields such as engineering, education, and gaming. By anchoring medias to geographic coordinates, it is possible to design immersive experiences remotely, without necessitating an in-depth knowledge of the context. However, the creation of such experiences typically requires complex programming tools that are beyond the reach of mainstream users. We introduce BiodivAR, a web cartographic tool for the authoring of location-based AR experiences. Developed using a user-centered design methodology and open-source interoperable web technologies, it is the second iteration of an effort that started in 2016. It is designed to meet needs defined through use cases co-designed with end users and enables the creation of custom geolocated points of interest. This approach enabled substantial progress over the previous iteration. Its reliance on geolocation data to anchor augmented objects relative to the user’s position poses a set of challenges: On mobile devices, GNSS accuracy typically lies between 1 m and 30 m. Due to its impact on the anchoring, this lack of accuracy can have deleterious effects on usability. We conducted a comparative user test using the application in combination with two different geolocation data types (GNSS versus RTK). While the test’s results are undergoing analysis, we hereby present a methodology for the assessment of our system’s usability based on the use of eye-tracking devices, geolocated traces and events, and usability questionnaires.

Impact of Geolocation Data on Augmented Reality Usability: A Comparative User Test

Wissenschaftlicher Artikel

Mercier Julien, Chabloz Nicolas, Dozot Grégory, Catherine Audrin, Ertz Olivier, Erwan Bocher, Rappo Daniel

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2023 , vol. Volume XLVIII-4/W7-2023, pp. 133-140

Link zur Publikation

Zusammenfassung:

While the use of location-based augmented reality (AR) for education has demonstrated benefits on participants’ motivation, engagement, and on their physical activity, geolocation data inaccuracy causes augmented objects to jitter or drift, which is a factor in downgrading user experience. We developed a free and open source web AR application and conducted a comparative user test (n = 54) in order to assess the impact of geolocation data on usability, exploration, and focus. A control group explored biodiversity in nature using the system in combination with embedded GNSS data, and an experimental group used an external module for RTK data. During the test, eye tracking data, geolocated traces, and in-app user-triggered events were recorded. Participants answered usability questionnaires (SUS, UEQ, HARUS). We found that the geolocation data the RTK group was exposed to was less accurate in average than that of the control group. The RTK group reported lower usability scores on all scales, of which 5 out of 9 were significant, indicating that inaccurate data negatively predicts usability. The GNSS group walked more than the RTK group, indicating a partial effect on exploration. We found no significant effect on interaction time with the screen, indicating no specific relation between data accuracy and focus. While RTK data did not allow us to better the usability of location-based AR interfaces, results allow us to assess our system’s overall usability as excellent, and to define optimal operating conditions for future use with pupils.

2021

edTech, réalité augmentée et exploration

Professioneller Artikel ArODES

Julien Mercier, Laura León

Educateur (SER), 2021, no. 10, pp. 13-14

Link zur Publikation

2024

Enhancing Biodiversity Learning Through Analogical Reasoning and Augmented Reality: An Ecological and Experimental Study

Konferenz

Laura Leon Perez, Mercier Julien, Emmanuel Sander, Catherine Audrin, Ertz Olivier

Psychonomic Society 65th Annual Meeting, 21.11.2024 - 24.11.2024, New York City, USA

Link zur Konferenz

Zusammenfassung:

The complexity of biodiversity conceptions, encompassing both scientific and political dimensions, holds significant challenges. Effective cognitive approaches for enhancing biodiversity conceptual development are essential. This study explores the effect of an intervention using analogical reasoning combined with augmented reality (AR) technology to foster biodiversity learning among school pupils. Analogies were used for their help in grasping complex ideas by mobilizing familiar concepts and AR technology was employed because it facilitates nature exploration through its mobility and clarified abstract concepts through visualization. In a study involving 170 students aged 12 to 17, significant increase in knowledge acquisition and spatial exploration were observed in the AR condition compared to those using non-digital methods, emphasizing the potential of AR to improve learning. However, further research is needed to assess whether AR might cause distractions, potentially leading to disconnection from the natural environment. Continuing to improve this teaching approach shows great potential for enhancing students’ understanding of biodiversity concepts and increasing their environmental awareness.

In situ visualization of 6DoF georeferenced historical photographs in location-based augmented reality

Konferenz ArODES

Julien Mercier, Erwan Bocher, Olivier Ertz

Proceedings of the 21st International Symposium, W2GIS 2024, Web and Wireless Geographical Information Systems

Link zur Konferenz

Zusammenfassung:

Location-based augmented reality (AR) enables in situ visualization of geolocated media in the real world, usually based on latitude and longitude referencing only. By integrating 6DoF georeferenced photographs, they can be positioned along all three rotational axes (pitch, yaw, roll) in addition to the usual perpendicular translation axes (surge, heave, sway), resulting in data registered on all six mechanical degrees of freedom of movement of a rigid body in three-dimensional space. In this paper, we present an integration test of 6DoF georeferenced data from the Smapshot open API to the BiodivAR open authoring tool for location-based AR. While both projects are open source, their data is not interoperable, similarly to most citizen science (CS) projects. After mapping the data and importing it in a new augmented environment, it can be visualized in the mobile AR interface, at the locations specified by the georeferenced images. Visualizing historical photographs in the AR interface from the actual location where they were originally taken allows them to align with their original context, resulting in an eye-catching and impactful visualization. This contextual visualization of historical photographs informs and enriches their meaning, and allows viewers to quickly detect patterns and anomalies at a glance. We perform a basic qualitative evaluation of the visualization made possible by the combination of 6DoF georeferenced data and our authoring system, based on our own observations. We discuss the potential of this proof-of-concept to foster participation and understanding in citizen science and education, especially with regards to biodiversity monitoring and education.

Motion capture data as machine-readable notation to capture musical interpretation :

Konferenz ArODES

experimenting with movement sonification and synthesis

Julien Mercier, Irini Kalaitzidi

Proceedings of the Ninth International Conference on Technologies for Music Notation and Representation (TENOR)

Link zur Konferenz

Zusammenfassung:

Motion Capture Data as Machine-Readable Notation to Capture Muscial Interpretation

Konferenz

Experimenting with Movement Sonification and Synthesis

Mercier Julien

International Conference on Technologies for Music Notation and Representation (TENOR), 04.04.2024 - 06.04.2024, Zurich, Switzerland

Link zur Konferenz

Zusammenfassung:

2023

Impact of geolocation data on augmented reality usability :

Konferenz ArODES

a comparative user test

Julien Mercier, Nicolas Chabloz, Grégory Dozot, Catherine Audrin, Olivier Ertz, Erwan Bocher, Daniel Rappo

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ; Proceedings of the FOSS4G (Free and Open Source Software for Geospatial) 2023

Link zur Konferenz

Zusammenfassung:

While the use of location-based augmented reality (AR) for education has demonstrated benefits on participants’ motivation, engagement, and on their physical activity, geolocation data inaccuracy causes augmented objects to jitter or drift, which is a factor in downgrading user experience. We developed a free and open source web AR application and conducted a comparative user test (n = 54) in order to assess the impact of geolocation data on usability, exploration, and focus. A control group explored biodiversity in nature using the system in combination with embedded GNSS data, and an experimental group used an external module for RTK data. During the test, eye tracking data, geolocated traces, and in-app user-triggered events were recorded. Participants answered usability questionnaires (SUS, UEQ, HARUS).We found that the geolocation data the RTK group was exposed to was less accurate in average than that of the control group. The RTK group reported lower usability scores on all scales, of which 5 out of 9 were significant, indicating that inaccurate data negatively predicts usability. The GNSS group walked more than the RTK group, indicating a partial effect on exploration. We found no significant effect on interaction time with the screen, indicating no specific relation between data accuracy and focus. While RTK data did not allow us to better the usability of location-based AR interfaces, results allow us to assess our system’s overall usability as excellent, and to define optimal operating conditions for future use with pupils.

Errungenschaften

PEOPLE@HES-SO Verzeichnis der Mitarbeitenden und Kompetenzen

Mercier Julien

Chargé de Ra&D HES

Chargé de Ra&D HES

PEOPLE@HES-SO
Verzeichnis der Mitarbeitenden und Kompetenzen