Room 1, Building R

The humans behind the sign: shaping the profile and the intersection of sign language studies and studies on AVT

Ana Tamayo

Sign language (SL), sign language interpreting (SLI) and sign language translation (SLT) (Napier & Leeson, 2016; HBB4ALL, 2017; CNLSE, 2017; Tamayo, 2022a) have often been left out of both theoretical and more practical approaches within Translation Studies (TS), Audiovisual Translation (AVT), and Media Accessibility (MA) studies. Despite this lack of academic interest from the fields of TS, AVT and MA, it is widely acknowledged that SLI(T) is an AVT and MA mode, but we face the risk of leaving it behind, unless we start to provide solid theoretical and methodological approaches to the study of SL(I)(T) within TS, AVT, MA and beyond. 

Research on SL is not scarce (Ferreiro 2020), but its focus has been mainly on aspects such as linguistics and sociolinguistics, bilingualism and bimodality, language learning, and the cultural dimensions of sign languages and their users. Recently, more contributions can be found, that shape the profile of the signer (see López-Sánchez, 2021; Napier et al., 2021, for example) or that deal with the human nature and embodiment of the signer (see, for instance, Spooner et al., 2018; Butler, 2018). 

Research on AVT and MA is not scarce either, but a new revolution (?) seems to be taking place in this field of study. Concepts such as creative media access or creative AVT (Romero-Fresco & Chaume, 2022) are recently evolving and taking new paths in research and practice that explore, deeper than ever before in the AVT and MA history, the engagement-based approach of audiovisual translation and accessibility modes. Sadly, AVT and MA research does not seem to be tackling SL, SLI and SLT from that perspective. In this presentation, my aim is to focus on the humans behind the signs from the AVT and MA studies. I will discuss how concepts such as embodiment, empowerment, cultural/linguistic appropriation, identity, hegemony, translanguaging, visibility, creativity, enjoyment and many others can fit in the intersection of SL studies and studies on AVT and MA. With this, my hope is to encourage research on such intersection that includes views and approaches from both, very flourishing, fields of study and that focus on the human value in the signing. In that same line, I hope this presentation helps highlighting the importance of more real and practical communication between signers, translators and interpreters, and audiovisual creative teams. With this, we might be able to move towards a new perspective on audiovisual communication that can see beyond oral languages, and that can use sign language as much more than just a communication modality to provide translation and accessibility.

Ana Tamayo (presenter) teaches at the University of the Basque Country (UPV/EHU) since November 2016. She obtained her BA and MA at Universitat Jaume I (Castellón, Spain). At the same university she defended her PhD about captioning for d/Deaf and Hard of Hearing children in 2015. She has completed two international research stays, at the University of Roehampton (London, UK) and at the Universidad César Vallejo (Lima, Perú). Currently, she is a member of the research groups TRALIMA/ITZULIK (UPV/EHU) and collaborates with TRAMA (UJI) and GALMA (Universidade de Vigo). Her research interests focus on audiovisual translation and accessibility in different modalities. She is especially interested in contributing to the research on accessible filmmaking and captioning and accessible filmmaking and sign language and in looking at minority languages in audiovisual media and translation.

A study of current practices of subtitling for deaf and hard of hearing audiences in the Arab world

Ghanimeh El-Taweel

Subtitling is the preferred mediation mode in Arab Countries. However, subtitling for deaf and hard of hearing audiences (SDH) is still rarely offered in the region. Limited efforts are found in film festivals, odd television programs, and streaming services, as is the case of Netflix. Despite the growing interest in research on SDH worldwide, very little research is dedicated to SDH in the Arabic context. This may be indicative of how deafness is regarded in the region and the wider perception that society in the region has about disability. 

This presentation addresses the outcome of an ongoing study on the state of SDH in the Arab World and aims to map current practices. In a first step, an analysis is carried out over six years. During three moments in 2018, 2020 and 2022, respectively, data were collected by examining the existing offer of SDH as advertised on different content providers’ webpages and by further assessing what is truly offered by TV networks available in the region, such as OSN and BeIN. 

The information gathered in this first analysis raises issues that may be further explored. In a second step, an interview will be conducted with stakeholders involved in the provision and production of Arabic SDH (producers, distributors, subtitlers), and the potential end-users (deaf communities). The information gathered in the interviews will shed light on the reasons behind what is being offered, and the utility of what is being offered. Furthermore, it provides us with a better understanding of people’s perceptions about the service, as well as their views on the needs, rights, and abilities of this particular group of Arab viewers.

Ghanimeh El-Taweel is a PhD candidate at University of Antwerp. Her research focus is Subtitling for Deaf and Hard of Hearing audiences in the Arabic context. She received her MA in Audiovisual Translation for Hamad bin Khalifa University. She is currently an Affiliate Instructor at the College of Humanities and Social Sciences at Hamad bin Khalifa University.

Visual impairment as an asset in the process of subtitling for deaf and hard-of-hearing audiences

María Guirado Pérez, Marina Ramos Caro

Subtitling for the Deaf and the Hard-of-Hearing audiences (SDH) is a relatively new field from which numerous research possibilities have arisen, facilitating access to information for this collective and opening the doors to new ways of encouraging their integration (e.g., Tamayo, 2016; Cuéllar, 2018; Arnáiz, 2012). However, research on the SDH process is still scarce. In this sense, the present paper describes a process-oriented study to research if SDH can be improved by collaboration with visually impaired people. In the absence of sight, visually impaired people shape their cognitive environment by means of their remaining senses, a phenomenon called sensory compensation which is facilitated by the neuroplasticity of the thalamus (cf. López Bendito, 2018). Since visually impaired people tend to have a more developed sense of hearing, we hypothesized that this sensory compensation would give them an advantage when it comes to recognizing and describing the sounds of a film. 

To test this starting hypothesis, we designed an exploratory study. A group of visually impaired participants and a group of sighted participants were exposed to a 5-minute scene from the movie Saving Private Ryan and asked to identify and describe all the relevant sounds for the comprehension of the clip. Then, the auditory information identified and highlighted by each group was compared. Our analysis shows relevant differences between the sounds described by each group, since blind participants gave more detailed descriptions than the sighted, specifying certain details such as the intensity and the emotional features of the sounds. In contrast, sighted participants omitted significant sounds or highlighted unnecessary information. According to our results, visual impairment allows participants to focus entirely on what they are hearing —i.e., the information needed by the target audience—, thus making visually impaired people the perfect candidates to balance the hearing deficiency of the deaf.

María Guirado Pérez is a Master's student in legal translation at the University of Granada. She studied Translation and Interpreting at the University of Murcia and researched Subtitling for the Deaf and the Hard-of-Hearing in her final project. 

Marina Ramos Caro (presenter) is a lecturer and researcher in Translation and Interpreting at the University of Murcia, specializing in the influence of emotions and personality on the process and reception of Audio Description and Translation. She has worked in subtitling, dubbing and transcreation (DE/EN/PT-ES), and directs the Inclusive Translation Laboratory of the University of Murcia (Latrium), dedicated to accessibility in the performing arts. With Latrium, she has created the AD of various types of stage productions, from physical theatre to contemporary dance. She is currently involved in the ADance research project, which focuses on the reception of audio described contemporary dance.

Can facial animations support speech comprehension?

Floris Roelofsen, Loes Gennissen, Rob Belleman

For people with hearing loss it is often difficult to understand speech, especially in noisy environments. For this reason, they often partly rely on lip-reading and other visual cues on the speaker’s face. In the presence of substantial background noise, even people without hearing loss may experience difficulties understanding speech based on the sound signal alone and often rely on visual cues as well. But in many situations, the speaker’s face is not visible—think of radio, videos with voice-over, and phone calls. This presents a challenge for speech comprehension, and more generally for the accessibility of certain types of media. 

We have explored the respective benefits of two methods to convert speech into facial animations for the purpose of supporting speech comprehension when the speaker’s face is not directly visible. One method, implemented in NVIDIA’s Audio2Face application, is based on the audio-signal alone. The application takes a speech fragment as input and yields a corresponding animation of a virtual human face, i.e., an avatar, as output. We call this the AUDIO-based method. The second method makes use of computer vision technology. It consists in capturing the speaker’s face with a depth camera and using this data to obtain a corresponding facial animation. We implemented this method using a depth camera on an iPhone, and the LiveLink application by Unreal Engine, which allows one to create animated MetaHuman avatars based on data from an iPhone depth camera. We call this the VISION-based method. 

Participants in our study heard sentences of the form ‘The train to [destination] is leaving at [time]’, for different destinations and departure times. During the pronunciation of the departure time substantial background noise was always present. After hearing the fragment, participants were asked at what time the train was departing, checking whether they had understood the speech signal despite the background noise. There were three conditions: (a) without visual support, (b) with an AUDIO-based facial animation, and (c) with a VISION-based facial animation. The experiment was carried out online, in Dutch. In total, 38 people with different levels of hearing loss participated. 

We found that fragments with VISION-based facial animations had significantly better comprehension rates than fragments without visual support, while fragments with AUDIO-based facial animations had significantly worse comprehension rates than fragments without visual support. These effects were stronger for people with higher levels of hearing loss. We conclude from these results that, when of sufficiently high quality, facial animations can support speech comprehension, but equally importantly, when not of sufficient quality, they can also deteriorate speech comprehension. The AUDIO-based method, which is clearly the most scalable, seems not to be of sufficient quality at this time. However, we expect that further developments in this relatively young area of research will lead to further improvements and may facilitate scalable automated visual support to aid speech comprehension and increase media accessibility in the near future.

Floris Roelofsen (presenter) is a researcher at the University of Amsterdam and director of SignLab Amsterdam. He is interested in developing technologies that could make media content more accessible for deaf and hard of hearing individuals. 

Loes Gennissen (presenter) is a master student in Artificial Intelligence for Neurotechnology & Healthcare at Radboud University. She is interested in improving the quality of life and inclusion of individuals with a sensory impairment. 

Rob Belleman is a lecturer and researcher at the University of Amsterdam and director of the Visualisation Lab. He is interested in interactive graphical technologies that provide insight in complex problems.

The balance between standardisation and creativity in media accessibility

Pablo Romero Fresco, Louise Fryer

Media accessibility is undergoing momentous changes. The past months have seen scholarly (Deckert and Bogucki 2022) and mainstream (Groskop 2021) publications, as well as academic and industry-led events (Translating Europe 2022) engaging with the technologisation of AVT and MA and the impact that this is likely to have on research, training and practice. The shift to cloud dubbing and cloud subtitling, the introduction of machine translation and the new generation of AI-powered tools rolled out recently worldwide are quickly changing the way in which audiovisual material is translated and made accessible across languages and cultures. Reactions to these changes have come from, amongst others, Higher Education institutions offering new technology-based courses on media accessibility and the European Federation of Audiovisual Translators, whose recent “Machine Translation Manifesto” (AVTE 2021) highlights the negative effect that this technology is having on the quality of translated audiovisual material and on the translators’ working conditions. The manifesto criticises the one-size-fits-all approach adopted by this technology, which results in bland and homogeneous translations that are devoid of the human translators’ unique styles and of their creativity. It proposes the notion of the augmented translator as a creative force that can harness and benefit from the latest technology to produce high-quality output –the human factor, as stressed in this conference. 

This panel aims to reflect on the role that academics, as scholars and trainers, can have in this debate. At the moment, research in AVT and MA is going through the so-called cognitive turn (Chaume 2018) – a process of scientification whereby the findings and data that were once obtained through the opinion of experts are now the result of empirical (and often user-informed) studies. Some of these studies aim at obtaining findings that can lead to standardised criteria, which works well in the current model of AVT and MA in which translators and access experts do their jobs in isolation from the creators and applying specific sets of guidelines that have been informed by this empirical research. However, as access is increasingly considered from inception through the collaboration between creators, access experts and users, new, alternative, creative and non-standard practices are becoming more common. Yet, training in media accessibility does not normally account for non-standard approaches (at least not thoroughly) (Romero-Fresco and Chaume 2022, Romero-Fresco and Brown 2023). 

How can we make room for creativity in media access training and research? Do we currently have the means (ie. material, trainers, evaluation tools) to provide this type of training? Can this be compatible with standard approaches? Can empirical research, which is currently mainly serving standard approaches, be also used to explore creative/alternative media access? Can the new generation of AI tools be used for this purpose? These questions and many others will be addressed by a panel made up of a group of scholars, trainers and professionals engaged in standard and alternative approaches to media access.

Pablo Romero Fresco (presenter) is a Senior Lecturer at Universidade de Vigo (Spain) and Honorary Professor of Translation and Filmmaking at the University of Roehampton (London, UK). He is the author of the books Subtitling through Speech Recognition: Respeaking (Routledge), Accessible Filmmaking (Routledge) and Transformative Media Accessibility (Routledge, forthcoming). He is on the editorial board of JAT and is the leader of the international research group GALMA, for which he is currently coordinating several international projects on media accessibility and accessible filmmaking and where he works as a consultant for institutions and companies such as the European Parliament or Netflix. Pablo is also a filmmaker. His first short documentary, Joining the Dots (2012), was used by Netflix as well as film schools around Europe to raise awareness about audio description. He has just released his first feature-length documentary, Where Memory Ends (2022), which has been selected for the London Spanish Film Festival and the Seminci (Spain), and whose accessible version has been screened at special events in New York and Montreal. 

Dr. Louise Fryer (presenter) was a pioneer of  AD in the UK, piloting description for BBC television in the mid-1990s. For many years, she described for the National Theatre and VocalEyes and has trained describers in the UK and elsewhere. Louise has a doctorate in psychology. Her research interests include visual perception, sensory substitution and immersion in audiovisual media. From 2018 – 2020 Louise was a Senior teaching fellow  in the Centre for Translation Studies (CENTRAS) at University College London. She is the author of An Introduction to Audio Description : a Practical guide (2016, Routledge).  With the blind aerialiste and drag performer, Amelia Cavallo, she co-authored Integrated Access in Live Performance (Routledge, 2021) based on research carried out for Extant – the UK’s leading theatre company run by and for actors who are blind or partially blind.  Her current interest is in integrated access and creative approaches, whereby access provision (AD, signing etc.) is considered from the start and fully part of the creative process. For over 20 years Louise was a presenter for BBC Radio.

Architecture heritage accessibility. designing the innovative system of navigation and audio description of places not only for the visually impaired

Krzysztof Krejtz, Aneta Pawłowska, Piotr Milczarski, Anna Wendorff, Daria Rutkowska-Siuda, Izabela Krejtz

Audio Description (AD) as an additional narration describing the visual scene is a broadly used accessibility technique that benefits understanding of the content of films (Kruger, 2010; Orero, 2007), visual art pieces (Krejtz et al., 2016) theatre performances (Fryer, 2010) or museum spaces (Pawłowska & Sowińska-Heim, 2016; Szarkowska et al., 2016) for visually impaired as well sighted adults and children. There are also attempts to use AD in city spaces to increase the accessibility of architectural heritage and to enhance understanding of a built environment architectural heritage and to enhance understanding of a built environment (Boys, 2014; Pacinotti, 2022). 

Audio Description is often created by specialists in a domain (e.g., art historians) who provide deep knowledge into of the audio described matter. But at the same time, the perspective of experts is biased by their knowledge. Even expert perception of objects might be very different from the perception of non-experts (e.g., Castner, 2018). The analysis of experts' and non-experts' perception and visual attention patterns is researched mainly with the eye tracking method. The method has been widely used to study media content and cultural artifacts, e.g., films (Marchant et al., 2009; Treuting 2006), scenes (Smith and Henderson 2008), and visual art perception (Krejtz et al., 2016). There is also a growing number of eye tracking studies of architecture perception in the context of cultural heritage, urban planning, and design. 

During the panel, we will discuss the proposed novel approach of audio description for architectural sites created by trained in AD art historians but adjusted with the natural visual scan paths of the places by non-experts. This approach presumably will help make audio descriptions more in line with natural historical architectural sites viewing patterns thus easier to follow by the target audience: visually impaired but also sighted tourists and city inhabitants. During the panel, we will also present the work-in-progress for accessibility to architectural and cultural heritage to a broad audience. The system is based on location-based services providing real-time information and Audio Description (AD) of architectural sites. The uniqueness of the system is twofold. First, it aims to work in real-time adapting its content based on the user's location. Second, it provides AD of architectural monuments constructed to the needs of stakeholders, including the visually impaired, from natural visual scanpaths of non-expert viewers. Here, we present the general system description, preliminary results of qualitative interviews with the visually impaired on their needs regarding the audio description of the architecture, and an eye tracking study comparing visual attention to architecture monuments of experts and non-expert viewers. Finally, during the present panel, we will discuss good practices of the user-centric design for cultural heritage accessibility tools. As well as the advantages and disadvantages of interdisciplinary teamwork include scientists of various disciplines (art historians, psychologists, computer scientists, and engineers), art institutions, and NGOs (Association of the Blind).

Krzysztof Krejtz (presenter) is a psychologist at SWPS University of Social Sciences and Humanities in Warsaw, Poland, where he is leading the Eye Tracking Research Center. In his research, he focuses on Human-Computer Interaction, multimedia learning, and media accessibility. He gave several invited talks at e.g., Max-Planck Institute (Germany), Bergen University (Norway), Lincoln University Nebraska (USA), and Ulm University (Germany). He is a member of the ACM Symposium on Eye Tracking Research and Application (ACM ETRA) Steering Committee and Full Paper Co-Chair for ETRA’22 and ETRA’23. He is leading LEAD-ME COST Action (CA 19142) on media accessibility. 

Professor Aneta Pawłowska (PHD, hab.), is employed at the University of Łódz. Prof. Pawłowska is a historian of art. Since 2013, she and her team have been conducting research with the support of various cultural institutions and museums in the field of making art accessible to people with disabilities. She the author of more than 80 scientific publications and several books (e.g.  Audiodeskrypcja dzieł sztuki - metody, problemy, przyktady, Wydawnictwo UŁ, Łódz 2016). E-mail: aneta.pawlowska@uni.lodz.pl. 

Piotr Milczarski is conducting R&D in the field of image processing, deep learning, artificial intelligence, machine learning, and cybersecurity as well as the implementation of research in mobile applications and systems. Piotr has got more than 40 publications indexed in Scopus and Web of Science. Piotr is tutoring classes in the field of AI applications in mobile systems, mobile applications, computer networks, IT project management, etc. Currently, Piotr is the contractor in the project  “Friendly City". Piotr has been awarded several certificates: IBM Certified System Administrator - AIX 7, Cisco CCNA, Cisco CCNP, Cisco Excellence Award, Adobe Flex, Java, Python, etc. Piotr is working at the University of Lodz, Faculty of Mathematics and Computer Science, and the Lodz University of Technology in the Institute of Information Technologies. 

Anna Wendorff is an assistant professor in the Department of Spanish Philology at the University of Łódź (Poland) and visiting scholar at the Centre for Translation Studies at University College London in the years 2019-2021. During this time, she carried out the project “Accessible art for the blind and visually impaired in London museums” (The Polish National Agency for Academic Exchange, The Bekker Programme). Her research interests include Latin American literature and literary and audiovisual translations, especially the audio description of works of art; from 2019 to 2022: contractor in the grant “Art of Łódź against the backdrop of European art. Excluded/Enabled” (The National Centre for Research and Development, POWER Programme); from 2021 to present contractor in the grant "Friendly City. Supporting the independence of the visually impaired people in the use of public transport networks in Łódź, including the application of location information and local architectural monuments" (The National Centre for Research and Development, “Things are for people” Programme). 

Daria Rutkowska-Siuda, PhD, assistant professor in the Department of History of Architecture at the Institute History of Art, University of Lodz. Graduated in art history, specialising in 19th-century architecture. Graduated in pedagogical studies – education through art and editing. 

Artur Hłobaż is Assistant Professor at University of Lodz, Faculty of Mathematics and Computer Science and at Lodz University of Technology in the Institute of Information Technologies. He has PhD in Technical Sciences in the field of Computer Science. He was involved in many research and didactical projects like Polish National Grants CFOOD, Erasmus+ GENIUS, JEU, or BUSIT. He has experience in organizing didactic Intensive Programme Projects like GGULIVRR@Lodz 2014-2020 and has over 10 years of experience in teaching in the field of IT. He teaches programming, computer networks (Cisco CCNA and CCNP Instructor), and subjects related to websites designing. 

Izabela Krejtz is an associate professor of psychology at SWPS University of Social Sciences and Humanities, Warsaw, Poland. She is a recognized researcher in the field of cognitive psychopathology, educational psychology, and daily experience measured with the momentary ecological assessment. She is an author of several dozens of international scientific publications among which a large amount was based on eye tracking experiments. She regularly teaches experimental research methodology in the context of various eye tracking applications.

​

Watching and listening to dance

Arianna Carloni, Kim Starr, Dimitris Asimakoulas, Sabine Braun

Audio Description (AD) is used in various settings and for various cultural products such as films, TV programmes, sport events, visual artefacts, theatre productions and dance performances. Despite its many practical applications, most AD research to date has focused on film (Perego, 2018), and areas such as AD for visual and performing arts are still under-researched. As most recently acknowledged by Barnés-Castaño et al. (2021), this is particularly true for dance AD. Whilst dance AD is increasingly flourishing as a practice, there is no comprehensive overview of its current state of practice, and little systematic research into its specific characteristics and how they are received by users. 

This presentation reports on a PhD project that aims to address this research gap. By drawing on a purpose-built dance AD corpus as well as existing research on AD and insights from other academic disciplines that investigate the complex, multi-faceted field of dance and its reception, the project aims to analyse the current practice of dance AD from a linguistic and multimodal perspective, and to explore the extent to which the experience of listening to the AD of a dance piece can be compared to the experience of watching that same dance piece without AD. 

More specifically, the first part of the project investigates how dance is currently audio described, in an attempt to identify recurring patterns or features of the language of dance AD. This has been achieved by building a small dance AD multimodal corpus, which has then been analysed by observing lexico-grammatical and semantic features such as word frequency distribution, key words and co-occurrences, and the presence of figurative and dance-specific language. 

The second part of the project consists of a reception study with visually impaired and sighted participants, who have been shown dance clips with/without AD respectively and have been asked to comment on their experience of the clips. Data collected from the two groups has been analysed and compared to explore the extent to which the experience of watching a dance performance and that of listening to its AD can in fact be comparable - given that, as suggested by literature around dance reception (e.g. Butterworth, 2012; Rochelle, 2015) and the researcher’s experience as a dancer and dance viewer, watching dance is a complex and highly subjective experience in terms of how meaning, appreciation and enjoyment of the dance piece are derived. 

The presentation will give an overview of the rationale and methodological approaches that have been adopted in this study, and report on key findings from both parts. AD is a powerful tool for accessing entertainment but also for social and cultural inclusion, and the increased recent attention given to diversity and inclusion in the arts makes the further exploration of dance AD particularly timely. This, it is hoped, will ultimately increase the offer of high-quality AD for dance performances, thus making dance more diverse and inclusive - not only in terms of the range of performers on stage but also of audience members.

Arianna Carloni is a postgraduate research student at the University of Surrey, UK (School of Literature and Languages). In 2013 she completed a Masters in Audiovisual Translation at City University (London, UK) with a dissertation on audio description for dance. After working in the audiovisual translation industry both as a project manager and as a subtitler, she changed careers to pursue her passion for dance and started working for inclusive dance education charities. This rekindled her interest in dance accessibility and led her to start a PhD to explore the topic of audio description for dance in further depth.

Kim Starr, PhD, is a Research Fellow in the Centre for Translation Studies (CTS) at the University of Surrey (UK). She previously worked in commercial television, and has degrees in politics and law (BSc.Econ., QMUL), journalism (MA, Westminster) and audiovisual translation (MA, University of Surrey). Awarded a doctoral scholarship by the Arts and Humanities Research Council/TECHNE, she completed her PhD in audio description for cognitive diversity in December 2017. For the following three years she worked on the EU Horizon 2020 project ‘Methods for Managing Audiovisual Data’ (MeMAD). She is currently employed as part of the Expanding Excellence in England (E3) programme at the CTS, researching human-machine interaction in the field of automated video description, cognitive accessibility in cultural contexts, and audio description for the non-blind. Kim co-edited and co-authored the IATIS yearbook, ‘Innovation in Audio Description Research’ (Braun & Starr, 2021). In 2022, she was awarded the Fulbright-Smithsonian Scholarship.

Dr Dimitris Asimakoulas is a Senior Lecturer in Translation Studies at the University of Surrey. He has conducted research on translation historiography and sociology of translation (with various articles appearing in journals of translation studies, historiography and visual arts) and has written a monograph on cultural adaptation and comics translation. He has recently led a pilot project on accessible comics in The Cartoon Museum, London.

Sabine Braun is Professor of Translation Studies, Director of the Centre for Translation Studies at the University of Surrey, and a Co-Director of Surrey’s Institute for People-Centred AI. Her research explores human-machine interaction and integration in translation and interpreting, especially to improve access to critical information, media content and vital public services. For over 10 years, she has led a research programme that investigates the delivery of interpreting services via video link to improve language access in the public sector. In addition, she is investigating the feasibility of (semi-automating) audio description (video-to-text translation) to improve media access for diverse audiences.

​

A voice-driven interactive smart audio descriptive guide for museum accessibility:  addressing the human computer interaction challenges

Xi Wang

Museums are typically dominated by visual experiences. This means that blind and partially sighted (BPS) visitors tend to be excluded from several important aspects of the visitor experience; for example, photos, information panels and other digital media such as touch screens and video footage. Two relevant emerging assistive technologies are chatbot technologies and Artificial Intelligence (AI), which are opening up new possibilities to enable users to explore exhibitions based on their own preferences and interests. The use of new assistive technology for BPS people raises the need to consider the Human Computer Interaction aspects of the technology. This is especially so given the sustained growth in this field. In their review of the state of the art, Bhowmick & Hazarika (2017) conclude that assistive technology for persons with visual impairments is expected to grow at a swift pace and impact the lives of individuals and the elderly in ways not previously possible. 

This paper describes the process of creating a voice-driven interactive smart audio descriptive guide for Titanic Belfast, a world-leading tourist attraction. This smart audio descriptive guide is intended to enhance museum accessibility and visitor experience especially for BPS visitors. The process of developing an interactive smart audio descriptive guide proved to be necessarily complex because of the range of skillsets involved. 

Therefore, the paper first outlines the necessarily complex set of team functional roles and users, and then presents the main Human Computer Interaction challenges and opportunities arising from the user requirements. For example, the use of smart technology opens up new possibilities for museum management. As we shall see, it can provide greater flexibility in the layout and organisation of exhibits and exhibitions. In the case of a traditional audio descriptive guide, once it has been developed there is little involvement by museum management, except perhaps to monitor how many visitors use the device. This is largely because it is not feasible in practice to update or alter the audio descriptive guide. However, our smart audio descriptive guide can assist management in monitoring and optimising the visitor experience. 

The key design features of the smart guide which meet these HCI challenges are then described, for example, speech input and output, question and answer facility and feedback gathering. We then evaluated our working prototype with a group of BPS participants and a review from a museum management perspective. It was clear from our evaluation that the smart guide has great potential, not only to improve the visitor experience for BPS visitors, but also to support the venue management in meeting their accessibility aspirations and obligations, and with reduced investment compared to the traditional approach. From our evaluation, we have also identified several fruitful avenues to pursue in developing our smart audio descriptive guide further. To conclude, the smart audio descriptive guide has the potential to offer museums and cultural venues a new, affordable approach to providing a high-quality accessibility experience with lower design effort than traditional audio descriptive guide approaches. 

Bhowmick, A., & Hazarika, S. M. (2017). An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trends. Journal on Multimodal User Interfaces, 11(2), 149–172. https://doi.org/10.1007/s12193-016-0235-6

Xi Wang is a Lecturer in Translation in the School of Languages and Applied Linguistics at the University of Portsmouth, UK. She currently teaches translation at both undergraduate and postgraduate levels. She is an Associate Fellow of Higher Education Academy. Xi had been working at Queen’s University Belfast as a Marie-Curie Early Stage Researcher since 2018. She completed her PhD in Translation Studies at Queen’s which was funded by EU Horizon 2020 Research and Innovation Programme. Xi holds two MA degrees in Translation and Interpreting at Jilin University and Queen’s University Belfast. Her research interests are in audiovisual translation and media accessibility. Xi has worked with world leading tourist attraction Titanic Belfast and Royal National Institute of Blind People to investigate novel access options that employ new technologies to improve accessibility and visitor experience for blind and partially sighted visitors. She currently works as a principal investigator in collaboration with the D-Day Story Museum at Portsmouth to investigate customisable technologies to enhance museum accessibility. Xi was awarded Emerging Scholar Award at 13th and 14th International Conference on the Inclusive Museum. Xi has published peer-reviewed journal articles in the prestigious journals in the filed: Journal of Translation Spaces, Journal of Universal Access in the Information Society, Journal of Audiovisual Translation and Journal of New Voices in Translation Studies. She is the member of European Association for Studies in Screen Translation and European Society for Translation Studies.

A method to create audio descriptions with a synaesthetic approach

Silvia Soler Gallego, M Olalla Luque Colmenero

Researchers and practitioners, in visual art, television, theater, and film audio description (AD), such as Szarkowska (2013) and Hutchinson and Eardley (2019), among others, have called for alternative AD approaches, while others, including Bardini (2020), Walczak and Fryer (2017), and Ramos (2016), to cite a few, have demonstrated their potential benefit for users in reception studies. This article discusses a reception study of minority visual art AD styles and features identified in a number of corpus-based studies (Luque and Soler 2020; Soler 2021) and focuses on one of them, namely the synaesthetic metaphor feature. In neuroscience, the word synaesthesia is used to name a perceptual phenomenon “in which sensations or sensory qualities are evoked automatically in the absence of adequate stimulation” (Marks and Pierce 2014, 33). In art and design, synaesthetic representations are practices involving a transfer from one sense to another or various senses acting synchronously (Riccò 2014, 167-68), while linguistic synaesthesias are metaphors that recreate a sensation through a sense that, while being alien to it, helps to understand it through bodily experience (Steen et al. 2010, 175). The method used for creating the synaesthetic AD materials for the study is based on a deep observation of the work of art and the sensations elicited by it. A list of sensory categories (Bayod 1999) is used to translate the sensations produced by seeing the work into sensations of time, space, and touch. The resulting synaesthesias are compared, selected, and added to an objective audio description of the work. This process is carried out by the authors along with Rubén Rámila, the third researcher in the project, who is blind and has both training and experience as instructor in sensory and synaesthetic development for blind and partially sighted people organizations, schools, and museums. In this article, we present the elaboration of the synaesthetic ADs of two sets of antagonistic works of art, since opposition often facilitates understanding of the synaesthetic attributes. These are: “Aha Oe Feii?”, by Gauguin, and “Portrait of Empress Maria Fedorovna”, by Makovsky, together with “Panels for Edwin R. Campbell”, by Kandinsky, and “Vir Heroicus Sublimis”, by Newman. We will present different examples of both the stages of the method and the final synaesthetic AD, along with a discussion of these materials and its application in the reception study. 

Bayod, Carles. 1999. El arte de sentir: Guía práctica para el desarrollo del hemisferio derecho del cerebro. Indigo Ediciones. 

Hutchinson, Rachel S., and Eardley, Alison F. 2019. “Museum Audio Description: The Problem of Textual Fidelity.” Perspectives 27, no. 1: 42–57. 

Luque, María O., and Soler, Silvia. 2020. “Metaphor as Creativity in Audio Descriptive Tours for Art Museums: From Description to Practice.” Journal of Audiovisual Translation 3, no. 1, 64–78. 

Marks, Lawrence E., and Pierce, John B. 2014. “Synaesthesia across the spectrum.” In Synaesthesia: Theoretical, Artistic and Science Foundations, edited by María J. de Córdoba, Dina Riccò, and Sean A. Day, 32–49. Fundación Internacional Arteccità. 

Ramos, Marina. 2016. “Testing Audio-narration: The Emotional Impact of Language in AD.” Perspectives 24: 606–634. Riccò, Dina. 2014. “Scientific Production on Synaesthesia.” In Synaesthesia: Theoretical, Artistic and Science Foundations, edited by María J. de Córdoba, Dina Riccò, and Sean A. Day, 158–169. Fundación Internacional Arteccità. 

Soler, Silvia. 2021. “The Minority AD: Creativity in Audio Descriptions of Visual Art.” In Universal Access in Human-Computer Interaction. Access to Media, Learning and Assistive Environments, edited by Margherita Antona, and Constantine Stephanidis, 308–327. Cham: Springer. 

Steen, Gerard et al. 2010. A method for linguistic metaphor identification. From MIP to MIPVU. Amsterdam, Philadelphia: John Benjamins. 

Szarkowska, Agnieszka. 2013. “Auteur Description: From the Director's Creative Vision to Audio Description.” Journal of Visual Impairment and Blindness, September-October: 383–387. 

Walczak, Agnieszka, and Fryer, Louise. 2017. “Creative Description: The Impact of Audio Description Style on Presence in Visually Impaired Audiences.” British Journal of Visual Impairment 35, no. 1: 6–17.

Silvia Soler Gallego. Presenter: University professor and researcher of Translation and Interpreting. She holds a BA, MA and PhD in Translation and Interpreting Studies. Her PhD and subsequent research focus on the use of audio description as a tool for helping blind and partially sighted people access art museums, reception studies on new types of audio description, as well as training and applied research in this field. She is also a professional translator and audio describer and co-founder of Kaleidoscope, a non-for-profit organization providing accessibility through translation. 

M Olalla Luque Colmenero. Presenter: Professor of Translation and English Studies at the University of Granada. She holds a BA, MA and PhD in Translation and Interpreting Studies. Her PhD and subsequent research focus on the use of metaphor as a tool for helping blind and partially sighted people access art museums, as well as training and applied research in this field. She is also a professional translator and audio describer and co-founder of Kaleidoscope, a non-for-profit organisation providing accessibility through translation.

Enhancing the creative accessibility of film festivals: new perspectives on audio description

Maria Luisa Pensabene

This paper assesses the significance of audio description (AD) and seeks to break new ground in the use of creativity to the benefit of the visually impaired. The article follows a corpus-study design with special focus on niche film genres, exploring new ways in which experimentation can be a game changer when it comes to creating descriptions for avant-garde cinematography. Over the past few years, researchers have shown an increased interest in how creative descriptions (CRD) are used both in cinema and theatres. However, much uncertainty still exists on the use of AD in niche film genres. My work seeks to examine the changing nature of unobtrusive descriptions and to assess the impact that creativity would have on a different kind of multimodal contents. The study draws fully from independent migration-themed cinema, whose contents often receive low media coverage. These contents include migration stories of marginalized groups within the independent film festival circuit that need to be made accessible to the blind and visually impaired. Corpora have been singled out among a considerable number of documentaries from the 14th edition of Sole Luna Doc Film Festival (2019). The festival catalogue spans three main categories – The Journey, Human Rights, Shorts – offering an interesting selection of experimental niche films. Interestingly, niche documentaries tend to use artistic strategies to subvert mainstream cinematography, often featuring a combination of evocative narratives and ingenious aesthetics. This will determine the extent to which CRD is applicable to experimental works. Data collection methods will include experiments and designed surveys with film showcase programmes for selected audiences. The study shows how CRD mainly relies upon language and its functions. For this reason, Systemic Functional Grammar (SFG) was found to be useful for the classification of clauses. The impact of CRD is assessed through a qualitative analysis based on the use of the ideational metafunction. This will help pinpoint the effectiveness of verbal processes within the textual models. In terms of word choices, this work also gives an account of vocabulary-related stimuli (i.e. adjectives), if we think that recent studies have been homing in on the importance of lexical choices. Therefore, the objective of this investigation is to detect any para-textual / textual elements in which creativity becomes a word allowing multimodal translators to share their authorial expertise. This will encourage creative thinking in AD, where describers are able to go through a creative writing process based on metaphors and similes to emphasize the levels of imagination. With reference to a number of published works on CRD (Fryer, Walczak, 2017), auteur AD (Szarkowska, 2013) and descriptions applied to naturalistic drama (Styan, 2002, cfr. Fryer, Walczak, 2017:9) to name a few, CRD proves how research on media accessibility is constantly evolving, pushing the boundaries of factuality. For this reason, there is still abundant room for further progress in determining whether these criteria can be applied to unconventional documentaries, especially when these films foreground strong social and cultural messages. 

Styan J. L. (2002), Modern drama in theory practice: Volume 1, realism and naturalism, Cambridge, Cambridge University Press. 

Szarkowska A., 2013, Auteur Description - From the Director’s Creative Vision to Audio Description, Journal of Visual Impairment & Blindness, 107(5):383-387. 

Walczak A., Fryer L., (2017), Creative description: the impact of audiodescription style on presence in visually impaired audiences, British Studies in Translatology, 26, pp. 69-83.

Maria Luisa Pensabene (presenter) is a PhD student in Humanities at the University of Palermo. Her research interests revolve around Audiovisual Translation and Media Accessibility with a special focus on the role of creativity in Audio Description and Audio Description in Foreign Language Learning (FLL). Her experience teaching English spans over 10 years at the Faculty of Translation and Interpreting as well as upper secondary schools in Palermo. She is a contract lecturer of English at the University of Palermo, where she is currently responsible for supplementary teaching activities in the BA degree course in Literatures and Intercultural Studies, Department of Humanities. She also has significant experience as an audio describer, language consultant and liaison/conference interpreter in the legal, medical field and the entertainment industry.

Enhanced audio description: exploring first-person description in accessible film productions

Mariana López, Gavin Kearney, Krisztián Hofstädter

Enhanced Audio Description (EAD) is an alternative to Audio Description (AD) for visually impaired film and television audiences. EAD aims to reduce the number of verbal descriptions by focusing on sound design as a vehicle for accessibility, through the combination of three main techniques. The first technique is the use of sound effects to provide information on actions, elicit the presence of establishing shots, convey abstract scenes as well as indicate time and place, and the presence of characters. The second is the utilisation of headphone-based binaural audio to convey the position of characters and objects portrayed on screen, through 3D audio spatialisation. The third is the addition of the I-voice (Chion 1999), a first-person narration employed to portray aspects of the story that cannot be conveyed through sound effects or 3D audio, such as gestures, colours and complex sets of actions. These methods allow for the reduction of verbal descriptions, which in traditional AD often mask crucial soundtrack elements (such as sound effects and music), while also providing a form of integrated access, in which accessibility is part of the film and television creative and technical workflows. EAD methods have been shown to provide experiences that are as accessible, informative and engaging as traditional AD, while also being acknowledged for their potential to foster social inclusion (López, Kearney and Hofstädter 2020a-b). This research paper focuses on the I-voice or first-person description. The authors will explore the potential and challenges of using this method in a variety of film and television productions, including different genres and cast sizes. Past work by the authors demonstrated that the method was widely accepted and considered to provide a more poetic and organic take on accessibility (López, Kearney and Hofstädter 2020a-b). However, its implementation at the time was limited to examples of short student films with small cast sizes. The research presented in this paper originated in discussions with focus groups with visually impaired participants, in which the research team was encouraged to explore the expansion of the I-voice to other types of production to gauge its effectiveness and arrive at guidelines for implementation for the industry. As a result, a number of new examples were created in collaboration with industry partners. The process of creation as well as the results of a series of evaluation sessions with visually impaired participants will be presented. End-users and industry partners are key agents in this research process. While the former have been providing essential feedback on EAD strategies since the start of the project, the latter are central to the implementation needed for the EAD methods to reach wider audiences. 

Chion, M. (1999). The Voice in Cinema. New York: Columbia University Press. 

López, M., Kearney, G. and Hofstädter, K. (2020a). Seeing films through Sound: Sound design, spatial audio, and accessibility for visually impaired audiences. British Journal of Visual Impairment, 1-28. 

López, M., Kearney, G. and Hofstädter, K. (2020b). Enhancing Audio Description: Inclusive Cinematic Experiences Through Sound Design. Journal of Audiovisual Translation, 4(1), 157–182.

Mariana López (presenter) is based at the University of York (UK), where she is Professor in Sound Production and Post Production. Mariana is the Principal Investigator for the UK Arts and Humanities Research Council-funded project 'Enhancing Audio Description II: implementing accessible, personalised and inclusive film and television experiences for visually impaired audiences.' 

Gavin Kearney is Professor of Audio Engineering at University of York (UK).He has published over 100 publications and patents relating to immersive sound, many in collaboration with industry partners such as Abbey Road Studios, BBC, Google, Huawei and Mercedes-Benz Grand Prix. He is the Co-Investigator for the project 'Enhancing Audio Description II.' 

Krisztián Hofstädter is a creative technologist working as a researcher, lecturer, freelancer and artist. Besides working on the Enhancing Audio Description project as a postdoctoral researcher, his doctoral research developed brain-computer music interfaces for meditation, in which neurofeedback is linked to auditory entrainment in generative soundscapes inspired by shamanic journeying.

Creativity in audio description. culture-specific references and emotional states across Netflix series

Alessandra Rizzo, Cinzia Spinzi

Over the past decades, media accessibility (MA) has gained momentum and contributed to providing greater access to audiovisual media while addressing heterogeneous audiences. This has involved a shift of MA as a subarea within audiovisual translation (AVT) devoted to a specific group of users (i.e., the deaf and the hard of hearing; the blind and visually impaired) to a field intended for anyone who might have the necessity to take advantage of intralingual and interlingual practices of translation (e.g., issues connected with age, immigration status and L2 learning, among others). Given this universalist perspective and user-centred approach (Greco 2018), creativity has become a significant intercultural strategy in Audio Description (AD) by enhancing the interpretation of culture-specific references (CSRs) and stimulating the users to have more immersive and subjective experiences (Walczak and Fryer 2017). In addition to its artistic and aesthetic function within accessibility services (Romero Fresco 2021; Soler Gallego 2021), and in line with the idea that creative practices can foster a variety of experiences relevant to diverse types of intersemiotic translation (Rizzo 2018), as well as promote inclusiveness in terms of cognition, affection, and perception (Spinzi 2019; Ramos 2015), creativity in AD scripts and in the translation of AD scripts can potentially help portray intercultural issues and the emotional states of fictional characters. In this regard, creative choices in AD-script production (intersemiotic translation) and AD-script translation (interlingual translation) can strengthen the user’s participation in meaning-making processes, while adding cultural value to the end-product (Jankowska 2021). The focus is on the production of ADs for TV series streamed by Netflix in the period of the Covid-19 pandemic when the digital diffusion of films and TV series had a significant impact on the ever-increasing growth of audiovisual translation practices. The investigation is based on the qualitative and quantitative analysis of the English and Italian ADs provided for the two series, "The Irregulars" (2021) and "Bridgerton" (2020), as crime/mystery and period dramas respectively. The aim is to analyse the selected corpus of 16 audio descriptions (Seasons 1) and provide a contrasting analysis between the English and Italian versions in order to evaluate 1) the existence of creative solutions; 2) whether the existent creative (linguistic) devices are culture-bound markers and whether they adhere to the source or target audiences (foreignisation vs. domestication); 3) whether discourse sequences and markers, semantic intensity and interpersonal meanings, among others, derive from processes of interlingual translation (from English AD scripts to Italian translations) or are the result of intercultural and intersemiotic translation-related activities put into practice in the ADs produced from scratch in Italian. Drawing upon corpus linguistics techniques (Sketch Engine 2014) for retrieving statistical information, and upon cognitive grammar (Langacker 1987/2008) and functional grammar (Halliday 2004/2014) for qualitative interpretation, for the purpose of this study, CSRs and intralinguistic culture-bound references (e.g., metaphorical markers, symbolic terms, perceptual and cognitive linguistic categories, fixed expressions and idioms) are surveyed and classified by taking into account, on the one hand, the Extralinguistic Cultural References (ECRs) strategies for AD research (Jankowska et al. 2017) and, on the other hand, by exploring what translation strategies have been employed to transform creativity into an intercultural translation device. 

Greco, G.M. 2018. “The nature of accessibility studies”. Journal of Audiovisual Translation 1(1): 205-232. 

Halliday, M.A.K. et al., 2004/2014. An Introduction to Functional Grammar, London, Routledge. 

Jankowska, A., 2021. Audio describing films: A first look into the description process. The Journal of Specialised Translation 36(a): 26-52. 

Jankowska, A. et al. 2017. "Translating audio description scripts ... into English". SKASE Journal of Translation and Interpretation 10(2): 2-16. 

Langacker, R.W. 1987/2008. Cognitive Grammar: A Basic Introduction. Oxford University Press, Oxford. Ramos, M. 2015. "The emotional experience of films: does Audio Description make a difference?." The Translator 21(1): 68–94. 

Rizzo, A. 2018. “Transcreating the Myth: ‘Voiceless Voiced” Migrants in the Queens of Syria Project”. In C. Spinzi et al. (eds.), Translation or Transcreation? Discourses, Texts and Visuals, Newcastle-upon-Tyne, Cambridge Scholars Publishing: 150-179. 

Romero Fresco, P. 2021. “Creativity in Media Accessibility: A Political Issue”. Cultus 14: 162-197. Sketch Engine. 2014. Adam Kilgarriff et al. (eds.). The Sketch Engine: ten years on Lexicography 1: 7-36. 

Soler Gallego, S. 2021. “The Minority AD: Creativity in Audio Descriptions of Visual Arts”. Universal Access in Human-Computer Interaction. Access to Media, Learning and Assistive Environments: 308-327. 

Spinzi, C. 2019. “A cross-cultural study of figurative language in museum audio descriptions. Implications for Translations”. Lingue e Linguaggi 33: 303-316. Walczak, A. et al. 2017. "Creative description: The impact of audio description style on presence in visually impaired audiences”. British Journal of Visual Impairment 35(1): 6-17. 

TV Series 

The Irregulars, 2021. Season 1, Episodes 8, by Tom Bidwell, Netflix (UK) 

Bridgerton, 2020. Seasons 1, Episodes 8, by Chris Van Dusen, Netflix (USA)

Alessandra Rizzo (presenter) is Associate Professor at the University of Palermo Alessandra Rizzo is Associate Professor in the Department of Humanities at the University of Palermo, where she teaches ESP and audiovisual translation. She holds a PhD and a Master of Arts in Translation from the University of Essex. She was Visiting Scholar at the University of Roehampton and Honorary Research Fellow at the University of Westminster. She is a member of the AIA, ESIST and EST associations. Her research interests focus on creativity and authorial stances in audiovisual translation and accessibility, ELF in the context of migration and the visual arts, as well as on forms of (sub)titling as counter narratives. She has extensively published in national and international journals, and in dedicated volumes. She edited the special issue “Translation and Accessibility for All in the Creative Industries: Digital Spaces and Cultural Contexts” for Bridges, Trends and Traditions in Translation and Interpreting Studies (2020), and coedited Translation or Transcreation? (with Cinzia Spinzi e Marianna Zummo, 2018) and the special issue “Translating the margin: Lost voices in the aesthetic discourse” for InVerbis (with Karen Seago, 2018). She is currently working on the monograph New Perspectives on Translation. Aesthetics of migration in Documentaries and for the Stage (Peter Lang). Cinzia Spinzi (presenter) is Associate Professor at the University of Bergamo 

Cinzia Spinzi is Associate Professor at the University of Bergamo. She holds a PhD in English for Specific Purposes, a Master’s in Translation Studies from the University of Birmingham, and a Research Fellowship from the City University of London. Her research activity and major publications lie in Language Mediation, Cross-cultural communication and Translation, with a particular focus on the translation of tourism, accessibility and metaphors. She is member of the Research Centre on Languages for Specific Purposes (Cerlis) and of the EU funded Project TTRAILs on teaching and training in the field of Language for Specific Purposes. She is co-editor of an international journal Cultus: the journal of Intercultural Mediation and Communication. (www.cultusjournal.com), orcid.org/0000-0003-3267-6905. Her most recent publications include co-edited works: “Accessible stories within mediascapes: Voicing otherness in digital museums” (2022, con Rizzo, Alessandra and Gian Maria Greco), Journal of Audiovisual Translation (4)1; “Translation as Intercultural Mediation. The evolution of a paradigm” (2022), in Dominic Busch (ed.) The Routledge Handbook of Intercultural Mediation. London/NY: Routledge. Spinzi, C. / Agorni, M (eds.) 2019. “Mind the gap in tourism discourse. Traduzione, Mediazione, Inclusione”. Altre Modernità, Università di Milano; Spinzi C. Rizzo, A. Zummo, M. (eds.) 2018. Translation or Transcreation. Cambridge Scholars Publishing; research articles in international journals: A Cross-Cultural Study of figurative language in Museum Audio Descriptions. Implications for Translation, (Lingue e Linguaggi 2019), and a monograph, Discursive strategies in the language of Foreign Policy (Bari 2016), where she combines Corpus Linguistics and the Appraisal Theory. She has published two co-edited books: Mediazione Linguistica e Interpretariato (Bologna, Clueb 2013) and L’Interprete giuridico (Roma, Carocci 2015).