Tieteellisten seurain valtuuskunnan (TSV) syksyllä 2020 äänittämien Tiedeklubi-podcastjaksojen tekstivastineet on julkaistu. Englanninkielisessä jaksossa Experience Research – Future Views on Sound keskustellaan kokemuksista ja niiden tutkimisesta erityisesti virtuaalitodellisuuden ja äänen alueilla. Lue jakson tekstivastine alta tai kuuntele jakso Tiederadio-kanavalla SoundCloudissa tai Spotifyssa.

Kuva: Unsplash.

Tekstivastine: Experience Research – Future Views on Sound

Eero Tiainen: Welcome to the Science Club, to listen to this podcast about the research on experiences and audio experiences. This podcast is served to you by the Federation of Finnish Learned Societies, the Finnish Society for Future Studies and Aalto Universities experience platform. My name is Eero Tiainen, I am an experience designer with the special focus on the medium on virtual reality, VR. I work at the experience platform and today I have the privilege of moderating this podcast. Experience platforms goal is to ignite seraeach on experiences both in aalto university and also outside Aalto. Experience platform is one of the seven platforms in Aalto. They are founded to foster research innovations in the intersections of different disciplines. Let’s talk about experiences. Imagine, you´ve just gotten your hands on a new album of your favorite music, your favorite orchestra or your favorite band. You put on your headphones and press play. You hear the sweetest music, you close your eyes and let the music drift you away. You enter a very relaxed state of mind. All of a sudden, the record brakes and you hear a horrible noise from your headphones. You take them off and feel disappointed by the disruption. The next day you describe to your friend how awful the listening experience was. But was it really? Wasn’t your musical experience just fine until the noise came to ruin it? According to Daniel Garniman, we have two selves: the remembering self and experiencing self. When we make stories about what has happened to us or make plans to our future self, it is the remembering self that is activate. It is the remembering self that is talking, when you are describing your musical experience to someone. But the remembering self leaves out a lot of life. It only remembers the beginnings, the changes in the middle and the endings of experiences. However, your other self, the experiencing self, lives in the present moment. It has a time spand of pre-seconds which constitutes the psychological moment for human beings. These moments of experiences are what our lives consist of. We have 600 000 moments of experiences during a month. And in life, we have 600 million moments of experience, if we are lucky of course. But what actually is an experience? Experiences consist of three components: the receptions, emotions, and thoughts. He states, that it is the interplay and the combination of these three that then resolves as an experience in each moment. Experiences must be as old as homo sapiens, but experience research is a relatively new field of a science. Today we have two Aalto University professors giving us short presentations and discussing about the experiences in general: sound experiences and experiences design role in the future. Professor Virpi Roto and professor Sebastian Schlecht, welcome. Virpi, you are a professor of experience design. What are you experiencing at the moment, this moment right now?

Virpi Roto: I don’t think being an experience design professor changes my feelings this moment right now. I’m a human being so I experience what I have here in the space: like the books, your peaceful sound when you were telling to story, still shaped my emotional state. I get these cognitive experiences, like the white church we can see from the window here, very nice in the evening lighting. Or the nice furniture in this room, and of course the excitement of this recording, I’m experiencing as well. So, experiences can be approached from many different perspectives from perception to emotions and of all moods. So, I feel all of those at the moment.

Eero Tiainen: I have to say, I can resonate with that. I feel pretty much the same things and I would describe this moment very much the same way. It’s interesting that maybe these experiences are also contages. Or is it just Covid, I don’t know! Sebastian, you are a professor of sound in virtual reality. What is the most beautiful sound you have heard today?

Sebastian Schlecht: That’s a nice question. Not sure about the most beautiful but the one I really liked today…I was riding the Helsinki metro today and as Finns are very polite and quiete (hiljainen) people, you can actually listen very carefully what the sound scape is in the metro. And it struck me today, that you can really hear the mechanism of this, almost as a living being, this machine which brings you from A to B. All these details…And it tells a lot about the condition of this machine, of these places you are going through and the people around you. I enjoyed this moment a lot.

Eero Tiainen: I have to say that the noises of metro remind me of Linnanmäki, the emusement park. Kind of like on everyday rollercoster that metro. Okay, let’s move forward. Virpi Roto is a professor of practice in experience design in Aalto university, and the chair of Aalto experience platform. Now, professor Roto will be providing us an oversight for the fields of experience research and experience design. Professor Roto is working in the frontline for forming an international and multidisciplinary experience research community at the moment. And Virpi’s presentation is titled Experience Research – From Silos to Networks. Virpi, the floor is yours.

Virpi Roto: Thank you. So, I’ll start from the big picture and going towards more concrete activities of ours at Aalto University. The big picture is that the need of research has changed. We currently fight with some grand challenges, such as climate change or ageing society, or what is the future of work with all the new artificial intelligence systems coming to the work places. And these all require innovation beyond the disciplinary silos. We need research collaboration in the intersections of disciplines, so we need research networks with people from different disciplines. So, Aalto University was this at least, and is to support this kind of intersection innovation. Aalto was combined from business school, technical school, and arts and design school. This is a very fruitful innovation collaboration, and it’s what Aalto is known for. But it is hard to do the multidisciplinary collaboration, because people don’t understand what the others are talking about when we try to communicate. The terms are used in different ways, and where do you even publish that multidisciplinary work of yours. This is a dilemma why people don’t put so much effort in these collaborations and they rather stay in the silos. Aalto tries to facilitate these collaborations and bring together people who could solve these bigger problems in networks. And these platforms, that Eero already mentioned, are formed for that purpose. So, platforms are virtual communities of people interested in the same topics. Experiences are one of these seven multidisciplinary themes that Aalto has decides to facilitate.

We have brought professors from other schools in this experience platform of ours. We have been quite successful to facilitate collaborations so that new research projects have been established between different Aalto schools and different disciplines. We are also very proud of our six doctoral students and we hope to raise this cross-disciplinary generation of researches who are not maybe born to this kind of a culture, but at least they understand how to work in-between two or more disciplines. So, experience research is highly multi-disciplinary and that’s why Aalto also chose it to one of the platform themes. When thinking about outside Aalto, this research has scattered in different disciplines and we don’t have a community of experience researchers. But we think that our own field is the whole thing of experience research, but when we get to meet and see other researchers, we understand that okay, they mean that by experience, and it is totally different perspective from our own. So, it’s fascinating, but also making you feel like why didn’t I know that there are all these people working with the same concept. That’s why I wanted to find out who are working on the experience research field, or there is now field yet, but I really want to create an experience research field that would bring all the experience researchers from different disciplines together. So, we did a literature review finding out publications that mention experience in the key word. And we found tools to analyze more than 50 000 experience publications, and we --- those publications according to the key words which show us quite nicely some clusters about experience research areas. It seems to be from social sciences to medical applications, to education, to business topics, tourism, used experiences and technologies. This all has been very interesting and it seems to excite others as well. So, we brought together people to one event, this Seven experiencing summit. It created surprisingly much interest, we were able to spread the word about the event to different disciplines and there 460 people registered to this online event. We were very happy that they came from different disciplines and all over the world. I hope that by joining forces we can accelerate the experience research just by sharing the existing knowledge with each other. Even this is not an easy task because the definitions of experience are very different in different disciplines. But I still believe that experience research can grow more coherent research field of its own. So, this is my introduction and explanation of how to come from silos to the networks. We are now just in the beginning of establishing this experience research as a field where we know about each other’s works.

Eero Tiainen: Yes, indeed, it’s a process that is going on at the moment and it’s a very hot topic, for us at least. So, nice. I think we could continue discussion after Sebastian’s presentation. Let’s go forward towards that direction. Sebastian is a professor of practice for sound in virtual reality at the Aalto Media Lab and Acoustics Lab. Professor Shlecht’s research focuses on the intersections of mathematical filter design, efficient algorithms, and sound design. Today, professor S will be giving us a presentation titled Augmenting Sound Experiences. There will be practical audio examples given as well, and these will be edited to this podcast after words. I am sure we will be learning a lot of new concepts in this presentation so, enlighten us, professor S.

Sebastian Schlecht: Thank you very much for your introduction Eero. I would eventually like to talk about sound experiences in virtual reality and augmented reality and look a bit into the future. But before I do this, perhaps it’s worthwhile to look into the past and think about what we are actually trying to achieve with virtual reality and similar technologies. Essentially, there is this human censoring that we have, tools to understand and perceive the role…Virtual reality, in its relatively narrow definition, tries to computationally create some of these stimulations in order to transport us to different reality. But to better understand, I think is worthwhile to look into the past and try to understand why we actually sense the world in the way we do. We can understand the human censoring system in relation to our environment. The system is really shaped through all the stimuli which are surrounding us, especially those which are important for us. In evolutionary terms we often say that they are important to our survival, but maybe more modern way to say it is that they are emotionally important to us. So, if we jump now to the modern society where there are lots of technologies between our communications…For example, if you listen to my voice now, it’s not what my voice sounds like but it’s actually what is transmitted through this medium of a podcast. It’s a highly abstracted version of my voice. You understand what I’m saying and you can recognize me, but I think it’s also obvious that there is a certain mis-match between what you are listening now and what you would be listening if you would be in this room with us now. I believe that this mis-match gets more and more research focuse…

Maybe to guide this to experience…Many of has now, because of Covid-19, instead of having a few hours on the phone or in video conference calls, many of has now the experience of having a whole day of these video calls. There are many reasons why they can be so stressful, but at least one is that there is a sensory miss-match. We are trying to make sense and try to filter and adopt to a technology, and constantly asking for extra effort. So, because I’m coming from the sound field, I would like to focus little bit on the spacial hearing aspect, which is clearly lost in most of these technologies. I brought some sound examples to make this more tangeable. I will first play the recording with a phone. [record playing]. In this example I believe you can still hear that there is English female voice, maybe you recognize that there is a French voice. But if you compare it now to what you would actually hear if you were sitting in the same space with the voices, is this..[record playing]. So, if you’re listening this on headphones you can clearly hear that the English voice on your right head side, and the French speaking on the left and it’s quite easy to distinguish this. I would like to give another, even more extreme example. [record playing] So, in this rather challenging environment, which is in everyday not so rare, there are many sound sources around you…You can clearly hear, that it’s still possible to filter the English-speaking voice and understand what she is saying. So, what I’d like to discuss here, cause the loss of multiway-communication we have through these technologies…cause in the phone and video calls we have to impose these very strict rules in whose talking when so we don’t overlap each other…so we need to squeeze all our information through this one communication channel. And that’s one of the ideas leading now to virtual reality as a communication tool. There, we try to bring back to some of our natural hearing capability. The goal is that, if you are in a virtual reality situation, you would be able to communicate more naturally. I would like to mention here how researcher are trying to achieve this. One is through the personalization of this experience. In this example you just listen to. These recordings might or might not work well for your depending on how different your physiology is compared to mine, to which these recordings were made. ---

One more step that I would like to introduce here is that when we think about virtual reality it might be easy to think that this is mainly physical science endeavor, where you just have to understand the physical laws of sound waves and simulate them in the correct way. But it turns out that these processes are extremely hard and difficult to simulate. And it’s much more beneficial what humans can sense and what their perceptional experience of this is. --- A similar way of thinking also applies to the construction of virtual auditory scenes, where it is important to understand what human beings can actually perceive. More reason development is now to go one step further and to not ask, what is physiologically possible to sense but also, how do people understand the auditory and what are the cognitive processes involved and what are the assumptions we make when we listen. --- For instance, the are signals that are extremely subtle but because they are important, we have learned to distinguish them. For example, in the tone of voice, we clearly hear, or we are trained to hear, whether someone is telling a lie or truth. Acoustically speaking the differences are very subtle, but because they are emotionally so important to us, we are highly sensitive. So, understanding these priorities we have in our perception and construction of understanding our world, is important path towards creating a meaningful and human oriented virtual reality. I would like to conclude with a general comment on our future technologies…I would like to stress that I’m not advocating that we should replace our whole experiences with virtual reality. Especially now in these times, we realize how important and complex and subtle our face-to-face meeting and interactions are and we really should not to avoid these.  But on the other hand, many of our bigger challenges, like climate change, ask for technologies which allow remote collaboration and communication. I think we will see an increase of the use in the use of the virtual work places, virtual conferences, maybe even virtual tourism. So, making these experiences more worthwhile is, I believe, a worthwhile goal to achieve.

Eero Tiainen: Thank you Sebastian. It really feels like these examples show the multi-disciplinary nature of experiences and all the different aspect that are included and has to be taken into consideration when we talk about them, research them and designing them. Sebastian, you mentioned the Zoom calls that many of us are in the endless limbo of them. How could we make better Zoom calls? How could we measure the experience that the people that are involved with the calls, are having? What kind of a methods could there?

Sebastian Schlecht: I believe there is more and more research on the listening effort we have through technology. For instance, it can be measured physiologically. There are different ways of measuring stress and…I think it’s important that you mentioned we now also have these long term exposures, so it’s not stress that happens only for 5-10 minutes, but something which maybe evolves more during the day, or week or a month…As we are exposed to these so much.

Virpi Roto: How we usually measure experience, although I’d rather talk about evaluating since it’s very hard to give a number to experiences. But because we design experiences, we have a goal in mind: what kind of an experience we want to design. Then it’s easier to give a number and evaluate whether people wo use our design are actually feeling the kind of experience we wanted to design and create. Typically, we use some kind of a questionary to study and understand.

Eero Tiainen: I know that some people also use an on-going…during the experience but it is little bit difficult because it interrupts the experience.

Virpi Roto: Exactly. So there have been some studies where they have developed some non-verbal and as little disturbing tools to indicate your experience while you are interacting while something like Zoom.

Eero Tiainen: It definitely seems we can’t talk about experiences without talking about senses and embodied aspects as well. Or can we, can we separate them somehow? What do you say about this Virpi?

Virpi Roto: Yes, this is a big question. Like, can we verbally describe our experiences. Some researches say yes, and some no. Some experiences are very hard to describe verbally so there are many visuals tools, for example some animated gestures…and people can reflect whether they feel excitement or…

Eero Tiainen: Definitely, I think like Sebastian’s examples showed that it’s not only the raw data that matters in the understanding…there are multiple different factors involved, also the cognitive parts or understanding sound signals, the emotional states, and we can also…when we hear other person we have used to --- our emotional states and we have become very good at it. Someone has also said that maybe 10 percent of communication is through the actual words and the rest is body language, the tone of voice and so on. But do you think there is some kind of a common mis-conceptions about how we sense things?

Sebastian Schlecht: I don’t know whether there are fundamental mis-conceptions but I think, as you do research, I think what often happens is that you fall in love with your research assumptions. And they can be very limiting because your are focusing on very specific things and you try to make them mesaruble and can sometimes loose sight that this is just part of much, much bigger picture…So, I think these mis-conceptions can happen if you stay on a certain level of a viewpoint for too long.

Virpi Roto: Yeah. Just today I heard this presentation about brain activity and how to --- emotion or experience. And it’s not possible…none of the psycho-physiological measurements of today can say exactly what this person is feeling because everybody reacts a bit differently physiologically to their emotions.

Eero Tiainen: Yes, the context seems to be a very crucial part of forming experiences. Are we on the mercy, or are our experiences on the mercy of our surroundings?

Virpi Roto: Yes, I think so. What else creates the experiences than the surroundings? Well, if we imagine something then we might create those experiences just by ourselves.

Eero Tiainen: Indeed, it seems like it’s directly…or it gets the impulse from outside and then it goes to our system and resolves as a subjective phenomenon called experience.

Virpi Roto: Yes, experiences are subjective, dynamic and unique.

Eero Tiainen: Hopefully we are going towards a human-sized experience design in the future. Thank you, Virpi Roto and Sebastian Schlecht, for this discussion. And thanks for the Federation of Finnish Learned Societies and the Finnish Society for Future Studies, for having us. And thanks for the listeners for your attention. Science Club podcast will continue next month when the science-fictionalist and factualist Risto Isomäki and Jakke Holvas will be discussing about scientific knowledge and ecological sustainability. Thank you.