Earlier this year, we published some early findings from focus groups where viewers discussed their reactions to machine-translated subtitles. In the past few months, we have conducted another two focus groups and a questionnaire study on the same topic. One of the focus groups consisted of Finnish-speakers who watched two Swedish-language clips subtitled in Finnish, and the other group consisted of English-speakers who viewed two Finnish clips subtitled in English. The questionnaire explored English-speaking respondents’ views on two video clips translated from Finnish into English. These studies provided further insights into viewers’ opinions and expectations concerning machine-translated subtitles.
Machine-translated subtitles for important and unimportant content
The findings throughout the audience studies have been consistent. Viewers feel that machine-translated subtitles contain errors and are often difficult to read. Many viewers appreciate professional human input in subtitling, but many can also see potential uses for automatically produced subtitles.
As we discussed in the earlier blog post, Finnish-speaking study participants were less enthusiastic about using machine-translated subtitles in their daily lives than English-speakers who wanted to access Finnish content. Many English-speaking respondents mentioned news as a programme type where machine-translated subtitles would be useful. Other suggestions included light entertainment like chat shows or morning shows, music programmes with little spoken content, or traffic, weather and official announcements. It seems that there are two distinct uses for automatic subtitles: either something urgent and important that has to be accessed quickly, or something light, entertaining and unimportant, where misunderstandings or quality issues are less consequential.
Because the study was conducted during the covid-19 pandemic, many respondents had an immediate example of urgent information in mind. Announcements on covid-19 safety measures, infection figures, and guidance on lockdowns and curfews are crucially important and time-sensitive information. Even imperfect automatic subtitles would be better than not understanding the messages at all.
On the other hand, it is also easy to understand why some respondents said that machine translations are more suitable for light entertainment. If automatic subtitles are used for high-stakes information, translation errors may lead to risky misunderstandings, while it is less crucial to understand everything correctly in a piece of entertainment. Still, a significant number of respondents welcomed the possibility of using machine-translated subtitles to access news and information. The key to making that possible would be to make further improvements to the quality of the subtitles.
Distracted viewing
An important factor in improving the quality of automatic subtitles would be to make them easier to follow. One discovery from the viewer studies was that although respondents understood the video clips reasonably well, watching them sometimes felt like hard work. Machine-translated subtitles can be unclear and confusing, and their pacing and segmentation is often clumsy. Many viewers described having to invest considerable mental effort into piecing together information and figuring out what is going on in the videos.
In the questionnaire, respondents were asked to rate how difficult or easy it was to view these videos compared to a normal viewing experience. The responses were on a scale of 1 to 5, where 1 meant considerably more difficult and 5 meant no more difficult. The average score was 2.4, which indicates that watching these videos felt mentally taxing to most respondents.
Extensive mental effort makes the viewing experience less enjoyable and less comfortable. For example, it can feel burdensome to try to follow a sentence that continues across several subtitles and is not divided into easily readable, self-contained linguistic units. This screenshot from one of our video clips shows a problematic sentence fragment that is not understandable on its own. If the text also contains other problems, such as the misspelling of ‘Joensuu’ in this example, the reading process becomes even more difficult.
When viewers have to work hard to understand the subtitles, their attention is drawn away from the rest of the programme, and there is not enough time to focus on what happens on screen. Visual information such as the table of train journey times in this screenshot would not receive much attention. The subtitles, which should support understanding, become a distraction instead. Many respondents in our studies described experiences of being distracted, and some important information may have been missed because of that.
It would be beneficial for viewers if the distracting nature of machine-translated subtitles could be reduced. Useful improvements would include, among other things:
- Making the text more fluent.
- Improving segmentation, i.e. keeping a single sentence to a single subtitle or splitting sentences into two subtitles at a logical spot.
- Condensing the text so that the viewer does not have to read too much in a short time.
- Improving the synchrony of the subtitles with the rhythm of the dialogue.
Accuracy of the translation is important as well, but it is not enough on its own.
Humans and machines working together
Our viewer studies revealed both potential demand for automatic subtitling and need to improve its quality. They also 
indicated that viewers appreciate human expertise in subtitling. Many participants expressed a preference for human-made subtitles or human post-editing over fully automatic subtitles. In the questionnaire, when respondents were asked whether they would prefer immediate access to machine-translated subtitles or slightly delayed access to human-made subtitles, 56% answered that they would choose human-made subtitles either always or in most cases. The full range of responses can be seen in the image on the right. It shows that the responses favour human involvement, but also accept machine translation to some extent.
Automated subtitling and machine translation are not ready to fill all gaps in linguistic accessibility. They can, however, be one tool among many to improve access and inclusion. In fact, these viewer studies support MeMAD’s objective of “combining automatic efficiency with human accuracy”. A combination of accurate human input and machine translation could provide better quality than automation alone. With the efficiency offered by automation, humans would be better able to provide linguistic access to those who would not have it otherwise. Therefore, the most beneficial practice for now seems to be to make the most of the collaboration between humans and automation, and to continue exploring contexts that might be suited for automation in the future.
Finally, we would like to express our heartfelt thanks to everyone who took part in our viewer studies! The thoughtful comments by focus group participants and questionnaire respondents offer invaluable insight into how subtitles are used. That insight will enable further advances in subtitling technology and informed decision-making about the provision of machine-translated subtitles.