(How) Should We Measure Art?

Whatever exists at all, exists in some amount¹
Edward Thorndike

Anything that exists in amount can be measured²
William A. McCall

Introduction

We are currently living through the third – and likely final – VR revolution. The dynamic development of technology (particularly computers and, consequently, VR) has finally brought it fully into the mainstream.³ Digital immersion is no longer reserved for specialists in laboratories, as devices capable of running VR apps (the sheer number of which is growing rapidly) have finally reached the average consumer, following a trajectory observed earlier for PCs. When the first head-mounted display prototypes appeared in the 1960s, computers were not powerful enough to generate viable virtual environments, and the headsets themselves were reserved for academic and military use. It could be argued that this particular incarnation of virtual reality was merely an experiment, the results of which sharpened appetites and laid the roadmap for further research. When the VRLAB,⁴ set up at the Institute of Psychology of the Polish Academy of Sciences, first began its research into cinematic VR (CVR), experiences like these were still in their nascent stages in Poland. Now, however, that landscape looks starkly different, owing not just to the rapid pace of technological change, but also the range of initiatives that made it easier for creators to pursue a new paradigm for VR experiences. Some prefer to do it systematically, relying on the scientific method, while others choose a more intuitive approach. The same VR technology that makes it possible to capture 360° stereoscopic footage and create interactive experiences in computer-generated environments also gives us the ability to accurately analyze the reactions of participants to the content presented. Using that information, however, requires close, trans-disciplinary collaboration between VR creators and researchers.

Collaborations between the VRLAB and creators are a key pillar of the laboratory’s efforts. Naturally, its primary goal is to investigate the behavior (emotional state, attention, spatial presence and co-presence, locomotion, etc.) of VR experience participants, but research conducted in collaboration with filmmakers has added an applied aspect. Basic psychological inquiries were used to devise research methods (e.g. body-part motion capture, monitoring physiological reactions such as skin conductance and heart rate, eye tracking, but also novel survey methods) and a method for integrated, multilevel data analysis. These methods were then adapted to fit the exploratory nature of CVR research.

Characteristics and limitations of the psychophysiological measuring of art

The study of the effects of artworks on human behavior has a long history in psychology, and many different approaches have been developed over the years. The early foundations of scientific psychology were laid by the research of Wilhelm Wundt. Art as a subject of interest appears even then, for example in research into sensory experience.⁵ Art occupies a central position in Gustav Fechner’s investigations of the stimulus–perception relationship which is the domain of psychophysics.⁶ Edward Titchener wrote that the blending which occurs in perception “may be so complete as to give us the illusion of qualitative simplicity. But […] systematic observation always reveals that […] we may attend separately to the separate components.”⁷ But how separate are they? The process of perceiving an artwork starts with stimulus reception, after which the recipient’s nervous system begins to look for meaning in the incoming signals. At this stage, we can already observe the emergence of simple attentive effects and affective reactions that shape the final experience. At the opposite end of the complexity continuum is the approach in which we analyze the process of forming a multidimensional relationship with the artwork.

Matthew Pelowski and his co-authors⁸ reviewed empirical and psychological approaches to the study of contact with art, paying particular attention to cognitive models of information processing. Nearly all of the models described by the authors include affects and appraisals as the primary products of such contact. Other key elements include finding meaning in artworks, comparing experiences to memories, and even matching artworks to the self. Longitudinal impacts include self-adjustment, wellness, and health. Robert Solso writes:

There are as many ways of looking at art as there are viewers of art. That huge diversity is one indication that we humans are a highly distinctive lot of creative people. It does not mean, however, that there are no universal principles of perception and cognition that apply to all of us as we view and appreciate art.⁹

Is he correct? Only about the existence of universal principles. Viewers and ways of looking may be organized by identifying a finite set of types. Is it wrong that he’s incorrect? No, as we ought to be interested in what can be generalized. Outliers rarely offer such valuable information. While they may prompt new questions or attempts at explanation, they essentially cannot be told apart from random events, observational errors, etc. Our approach focuses on viewer reactions generated by contact with an artwork and immediately after it. The primary goal here is to capture, in real-time, conscious and subconscious reactions elicited by contact with the artwork, and the aggregate reactions generated in its wake. Such an approach guarantees that the observed variability stems from the impact of the artwork or its constituent elements. We understand the need to study the long-term effects of art, but such inquiries veer too closely to what Dariusz Doliński critiqued as “people in a freezer.”¹⁰ A psychologist cannot examine research subjects as if they had been put into a freezer right after experiencing an artwork, only to be removed some time later, asked to recall the experience, cross-reference it with salient memories, and then respond to a batch of survey questions. “However, real social life is nothing like a freezer,” Doliński writes; in the time between experience and recollection, countless events may occur and shape the final response. Stripped of control over variables, we lose any notion of what is and isn’t the result of contact with art. And so it ceases to be interesting. In summary, our approach is set apart by its embrace of the scientific method and control of confounding variables using the latest data collection and analysis technologies. This makes the results of our research independently verifiable.

Original research

Psychological research tends to strive for as much simplification of the research situation as is possible without detriment to psychological realism.¹¹ A psychologist would not be at all surprised by a study that measured changes in key-press rates following the appearance of a letter on a screen, depending on whether it was preceded by the appearance of a fixation point. Professor Piotr Francuz asserted that the virtual world was too busy for it to be reliably investigated.¹² However, the experiences of the Virtual Reality and Psychophysiology Laboratory at the Institute of Psychology of the Polish Academy of Sciences (VRLAB IP PAN) clearly show that it can and should be examined. Let’s take a closer look at examples of research into the behavior, attention, and emotions of virtual experience users. Some of these studies involved environments with linear storytelling, while others probed interactive, spatial experiences that can be freely explored in any direction and without time constraints. So as not to distract readers from the methods and potential application of the results, I will not go into further detail of these experiments here. Rather than on their content, let’s focus on their general nature.

Methods

The paragraphs below present a synthetic outline of the methods used in several different experiments carried out in the course of VRLAB collaborations with artists, filmmakers, creators, film scholars, and art promoters. This will not be a detailed report from a series of studies, as that is beyond the scope of this essay. Each of the results quoted below illustrates a different aspect of our approach to measuring the influence of art. Breaking them down in detail would require separate theoretical introductions, broader analyses, and discussions of their individual premises, which were slightly different in each study. This is not our goal here. Rather, we want to showcase the benefits that science offers art and artists, and argue against authors who question the feasibility of collaborations between science and art.

Participants. The studies from which the examples of the results are derived involved from 40 to 160 people, all of them adult residents of Poland (with one exception, which featured teenagers – eighth- and ninth-graders). The participants were usually divided into art creators and art recipients (based on participant declarations). The recipients were then screened to identify individuals who were interested in cultural events and attended them at least occasionally. The selection sought to eliminate the possibility of the artwork (usually a work of cinematic VR) being experienced by a person who would not normally consume such content on account of disinterest in or disdain for art. Studies of such specific groups (people experiencing art for the first time, for example) are interesting in and of themselves, but extend beyond the scope of our research objectives here. The participation of the groups was approved by the relevant Ethics Commissions. Special protocols were also in place to minimize COVID-19 transmission risks.

Materials and equipment. The virtual environments were displayed using head-mounted displays, either a HTC VIVE Pro Eye or a HTC VIVE headset with factory SMI modification. For 3DoF¹³ experiences the HTCs were wired, but 6DoF experiences¹⁴ employed a wireless HTC. The headsets were powered by PCs with Intel i9 2.8GHz or Intel Xeon E5-1630 3.7GHz CPUs, Gigabyte GeForce RTX 3070 or Nvidia GeForce GTX 1080Ti GPUs, and 64-bit Windows 10 Pro operating systems. Psychophysiological data was collected using BIONOMADIX amplifiers connected to BIOPAC MP150 data acquisition systems with ACQKNOWLEDGE software, or NEUROSCAN EEG amplifiers with CURRY software. The psychophysiological data was analyzed using MATLAB (EEGLAB and LEDALAB) and R software with relevant add-ons.

Standard survey methods were used to measure psychological variables (e.g. FCZ-KT, STAI), along with custom surveys specifically tailored to the objectives of particular studies. Open questions and structured interviews were also used.

Procedures. Participants were invited into the laboratories, where they were introduced to the research procedure and asked to sign a consent form. They were then asked to take a test run through the procedure, then to establish a baseline measurement, and finally subjected to the VR experience. Afterward, participants completed several surveys, sat down for an interview, and were debriefed.

Test station. The physical space allotted for movement in the lab was adapted to best fit the test environments. The lab’s physical dimensions were 4.8 x 8.9 meters, meaning that the safe movement space was 4 x 4 meters for 6DoF experiences. In the case of 3DoF experiences, the tests were performed in the same physical space, but participants sat on a custom swivel chair, modified to help acquire an EEG signal. The sole exception was the 3DoF experience that served as a control for the 6DoF experiences – in this case, participants stood and turned on the spot.

Results

The experience of space in free exploration

There are a growing number of virtual environments available for entertainment, training, and research purposes, in which participants can freely explore a limited or (near-) unlimited space. Analyzing participants’ movements within that space can help us determine the impact that virtual spaces have on their users. Observations can show, for example, that positively valenced objects elicit approach behaviors, while negatively valenced objects mostly elicit avoidance behaviors. The results can also suggest whether the environment stresses or relaxes participants. Fig. 1 shows routes taken during the free exploration of an interactive virtual environment, which was then modified (at random) with positive or negative stimuli, realistic or symbolic in form. Users could freely move in a space measuring 4 x 4 meters. The impact of intrusive stimuli was controlled, and individual iterations of the environment were similar, albeit not identical. The one systematic difference between iterations involved the specific arrangement of manipulated stimuli used.

Comparing the surface area walked over by the users reveals, as predicted, that positive and realistic stimuli made participants more relaxed, while negative and abstract stimuli made them more stressed. Surrounded by positive stimuli, participants covered a greater area on average than those experiencing negative stimuli.

Reactions to characters in virtual environments

What will happen if we place other characters inside a virtual environment? A virtual agent, for example, usually humanoid in form, that users can interact with and that itself could initiate interactions. Virtual agents are increasingly prevalent in virtual reality interfaces, but can also serve a variety of other functions, e.g. play a role in the narrative served by the virtual space. From an evolutionary standpoint, humanoid characters are highly significant stimuli for users. This means that they ought to elicit strong reactions, similar to those elicited by other people in our environment (e.g. maintaining interpersonal distance, respecting personal and intimate space, refraining from touching others). On the other hand, if users do not treat the agents as other people (for whatever reason), it may lead to radically different behaviors toward them (e.g. refusal to maintain interpersonal distance, inappropriate touching, shoving, destruction attempts). Fig. 2 shows a comparison of minimum distances to the agent maintained by younger and older users in virtual environments.¹⁵

Older users tended to maintain greater distance from the agent than younger users. Neither group tried to touch the virtual agent. Both groups maintained distance and did not attempt to invade the agent’s personal space. As they likely derive from intergenerational cultural differences, the disparities in distance to the virtual agent might be used as an important cue in the design of virtual environments and experiences for the elderly.

Visual attention in cinematic VR experiences

Cinematic VR (CVR) experiences are a specific subsection of virtual environments. On the one hand, they’re more rudimentary than computer-generated 3D environments, as they do not allow free movement through the presented world, interaction with objects and characters, or influence over the environment itself. These experiences usually tell a linear story, which is more or less the same for all users, has the same running time, and its key events are triggered at similar points. Naturally, because the projection is (usually) spherical and the participant is placed in its center, individual experiences might vary, even significantly so (sometimes according to creator intention, other times not). On the other hand, they can be much richer, in that they offer a level of photorealism difficult to reproduce (as of today, at least) in computer-generated environments, which promotes high levels of the illusion of spatial presence and co-presence.¹⁶

Effective storytelling in CVR depends on participants being placed in the right direction at key points in the story, which gives them the physical opportunity to follow what is displayed on the sphere. Naturally, what they are looking at matters a lot. Even commercial versions of modern HMDs often ship with eye-tracking capability. Data collected in this manner can be analyzed in a variety of ways; the results can even be fed back into the environment to modify its appearance or behavior. Analyses like these may also help us establish whether participants are paying attention to the desired elements of the environment. The convergence of visual attention can be expressed using the so-called convergence index,¹⁷ which is high when all viewers look at the same fragment of the image, and low when their gazes are scattered. Calculating the index may prove particularly useful in the case of cinematic VR, where a low convergence index might suggest that viewers are looking in different directions.

Fig. 3 shows raw eye fixation data captured while watching a spherical film. As we can see, much of the attention is aimed at the “equator” of the sphere. The picture also shows, however, that participants pay disproportionate attention to events occurring on one specific “hemisphere.” Very little happens near the top or bottom of the sphere, or behind the viewers’ backs. This means that the director actually shot what is essentially a 2D film, but with a 360° camera. Everything interesting happens on just one side of the sphere. If a participant accidentally turns in the right direction at the beginning of the experience, he or she may not turn around throughout it, staring at one point as if at a cinema screen. But if they begin the experience looking at a dead zone, they will have to be induced to look for action elsewhere. Fortunately, viewers caught in these situations are typically very active and tend to look independently for more stimulating parts of the environment (see Fig. 4 below).

The top part of Fig. 4 shows the situation just after cutting. A new scene begins and participants look for action around them (the fixations are distributed nearly uniformly along the equator of the sphere). They scan nearby subjects (the participants around them in this case), and soon find and focus on a new place of action. Here, the group’s convergence index is very low and grows over time. The bottom part of Fig. 4 shows a situation in which the scene has been going on for a few seconds. The action takes place in a single spot, with most of the viewers looking in its direction. Their attention is focused from beginning to end. Most of the gazes are fixed on that area of the sphere. The group’s convergence index is consistently high throughout. How can we achieve such a result? What do we do to move from behaviors seen in the top part of Fig. 4 to those seen in the bottom part? This can and should be examined using systematic, controlled, and repeatable experiments.

An interesting variant of the convergence factor is the individual indicator. This is high when the participant has their gaze fixed on the same part of the image within a given interval (typically 5, 10, or 15 seconds), and low when their gaze wanders. The value can be used (e.g. in combination with heart rate changes, further discussed below) to calculate the overall attention index. When the individual convergence index is calculated in real time, it can be used to identify moments when the participant is distracted, and emit suggestions or pointers that help them re-engage with the narrative. It can also be used to isolate individual differences in reception. This is how it was applied in recent research into the specific reactions of CVR developers compared to the reactions of a control group.

Measuring physiological responses – attention and emotion in (C)VR

Electrodermal activity (EDA, previously called galvanic skin response, GSR) is a simple and highly useful parameter in research into CVR and interactive virtual environments. EDA is the record of non-specific changes in electric skin conductance, which depends on the skin’s moisture level, meaning that it essentially reflects the activity of sweat glands resulting from stimulation of the autonomic nervous system. EDA may reflect the spontaneous reaction of the human body to an event – for example arousal or experiencing emotions while watching a film.

Fig. 5 shows a typical EDA feed from a single person, following preliminary analysis using Ledalab software.¹⁹ The EDA trace has a distinctive pattern – it changes relatively slowly, with intermittent peaks featuring a sharp rise and then a gentle sloping fall. The pattern reflects the nature of arousal elicited by significant stimuli – it rises quickly and falls slowly. Sometimes, the next stimulus appears before arousal returns to the baseline level. The average level of arousal then increases or, in other words, tonic activity rises. The intermittent peaks, meanwhile, denote phasic activity. The more peaks that appear following an event, the greater arousal response the event elicits. CVR makers can adopt different strategies of steering that arousal.

Some events in CVR generate strong arousal, because they resemble primal fight-or-flight triggers (e.g. sudden and loud noises, sudden flashes of light, darkness). Incorporating stimuli like these into a VR environment naturally generates an increases in arousal, but it is not necessarily linked directly to the intended content of the story. Increased tonic activity, however, may make it easier to elicit a reaction to events that are a crucial part of the narrative, but it can also make it more difficult, particularly when the creator overdoes the amount of primal stimulation, and tonic activity reaches the ceiling limit.

Situations in which the story triggers arousal rather than primal stimuli are much more interesting. These include, for example, the appearance of a character or figure, interactions between characters, spoken words, or music. Fig. 6 (below) shows a sample viewer reaction to a situation in which the heroine of the story is suffering.

EDA is not the only parameter that can provide highly useful information to VR creators. Electrocardiograms (ECG), which trace changes in electrical potential during the depolarization and repolarization of myocardial cells, offer even broader opportunities. For example, acceleration of the heart rate (HR) can be interpreted as reflecting an increase in arousal related to the processing of a given stimulus (mirroring the interpretation of a rise in EDA). HR deceleration, meanwhile, may be the result of increased attention generated by the occurrence of a given stimulus.

The appearance in the film of objects or people carrying negative connotations typically elicits a rise in arousal, reflected by an increase in HR. In turn, the appearance of significant situations or elements elicits the focus of attention, as inferred from HR deceleration. In Fig. 7, showing changes in heart rate over time, we can see a brief, four-beat-per-minute acceleration in HR triggered by a negatively charged stimulus, followed by a deceleration elicited by the main character appearing, after complete darkness, in a brightly lit scene, and a loud sound made by an object crucial to the story.

Discussion and suggestions for further research

We recently celebrated the bicentennial of the Gipsformerei in Berlin, the oldest division of the city’s state museum network. Opened in 1819, the still-operational unit continues to manufacture plaster replicas of a vast array of artworks. Many of its products are copies of relatively small objects. Perhaps the most famous of them is the iconic bust of Nefertiti, reproduced by the workshop hundreds of times since 1913.²⁰ There are also larger replicas of the same type, including one of a several-meter-tall Mayan stele.²¹ Plaster replicas made their way into museums and moved from city to city as part of traveling exhibitions, allowing audiences to experience exotic artworks (almost) without moving. Some museums went so far as to draft plans for a network that would allow them to easily exchange their plaster replicas. Today, VR performs similar roles: the Mayan stele mentioned above can now be experienced in 3D. More precisely, we can now admire the scans of a plaster replica first cast in the late nineteenth century – which means, in this case, that the modern VR experience is based on its Victorian predecessor.

Scans and virtual models allow us to admire art in new contexts. VR eliminates almost all limitations – users can freely manipulate an object, enlarge it, even touch it, basically in any place and at any time.²² The scale of reproduction has also changed significantly. Photogrammetry now allows the scanning of much larger objects and spaces, while much more democratic access to technology has enabled those other than craftspeople, artists, or scientists employed by museums to create experiences like these and share them with others. Hence the possibility of visiting Nefertiti’s tomb²³ or the Göbekli Tepe.²⁴ Both these experiences allow users to immerse themselves in the history of the places they depict. For some, the journey will be purely technical. They will see the interiors, reconstructed with laser-like precision and filled with paintings, hieroglyphs, and carvings; the interactive features will tell them about the construction and mythology of those places. Others, meanwhile, will be more interested in the stories of the sites. And it is the latter case that I find much more interesting.

We know the story of Nefertiti’s tomb. Compared to Göbekli Tepe, the burial chamber is relatively new, dating back only 3,276 years. The inscriptions guide users through the experience of Nefertiti’s final journey, and a story about her life and the love she shared with Ramesses II. Thanks to this, they can experience its uniqueness through the uniqueness of the place they have visited virtually.

Göbekli Tepe is much older – some archeologists date the site (at least the oldest of its discovered portions) back as far as 11,000 years. We don’t know what it was used for, or what the significance of the reliefs and carvings found there was. The reasons for its deliberate backfilling also remain unclear – was it the work of the original builders? Had they buried it to hide or destroy it? Users must discover the story sealed in the stone circles and figures themselves, as it was not written down in hieroglyphics (or any other script) and does not impose itself upon the experience. The very form of Göbekli Tepe, however, suggests the existence of some kind of story. Arranged in a circle, the figures (perhaps of humans) and the reliefs depicting animals and geometric shapes seem to follow a pattern that resurfaced nearer our times in temples. The difference is that a church makes it easy to follow the proper path. Large images on the walls clearly identify the beginning of the story and lead us through the narrative (and the space). Let’s take for example the pictures and sculptures portraying the Stations of the Cross, which are typically numbered to make navigation easier. Göbekli Tepe makes no such accommodations. Users don’t know whether they should move left or right, but like Catholics walking down the Way of the Cross, they may move from the crane to the fox, recreating one of (at least) two possible stories.

The virtual interiors of Nefertiti’s tomb and Göbekli Tepe have storylines built into them. The former, while interactive, ultimately unfolds along one path, and resembles a linear film, even though it allows for deeper exploration or the omission of certain plotlines. The latter, meanwhile, is multidirectional, as are the potential narratives it gives rise to. This raises questions as to whether it’s actually possible – and if so, how – to assess how these affect users. Virtual reality technologies offer researchers wholly novel means of observing and capturing behavior which are far ahead of anything we have had previously. When it comes to VR experiences, we are dealing with what is, at its core, the complete digitization of behavior. We are now able to track, with millisecond precision, the position of users and their movement within a virtual space, as well as the psychophysiological parameters that reflect the responses of their autonomic and central nervous systems (such as heart rate, skin conductance, muscle tension, respiratory depth), and even the points in space their eyes are fixed on and attendant pupil dilation. Each of these parameters has its own significance, and may be interpreted either individually or in the context of other variables. The ability to capture and record them with such precision, as well as cutting-edge data analysis suites, allow us to make inferences about the emotions and attention of VR users. Once we’re able to monitor and analyze these parameters in real-time, we will be able to feed this information back into the virtual environment and use it to modify the narrative on the fly.

This essay sought to outline our approach to investigating the impact of art. In our original research, we demonstrated the usefulness of existing psychological methods of evaluating emotional reactions, and mentioned the specific ramifications of studying VR artworks, films, and interactive environments. The listed methods by no means exhaust the catalog of available tools – both for data collection techniques and methods of data analysis. This, in turn, enables collaboration between scientists and artists, and the pursuit of new challenges. One such new objective entails providing creators with the means to verify their hypotheses in the early stages of a work. Another involves establishing a catalog of means specific to virtual reality technologies. A third, meanwhile, seeks to develop methods of using collected user response data and incorporating them in real time into the artwork. These are just a few of the emerging directions in VR research, but, at this stage, the three mentioned above seem the most interesting.

The collaboration of art and science, ensuring the interpenetration of the disciplines, may give rise to interesting research questions, new artworks to study, novel, reliable and accurate research methods, and meaningful interpretations of research results. The virtual world is growing ever richer, and science continues to offer us new means of studying it.

Acknowledgments: I would like to thank my co-workers from the VRLAB IP PAN, for the long hours spent in the lab and in front of computer screens, cleaning and analyzing data. My most heartfelt thanks to: Laura Osęka, Małgorzata Draps, Agata Feledyn, and the interns who assisted us. Many thanks to Sado Rabaudi (WorldViz) and Daniel Cnotkowski (LIT OPI PIB), who helped start this whole thing, and to Paweł Kobyliński (LIT OPI PIB), who helped the project grow into new areas.

The essay was based on research conducted for the “New Narrative Forms and Technologies” project. The project received funding from the Ministry of Science and Higher Education’s 2019–2023 Regional Excellence Project, grant no: 023/RID/2018/19, amount of funding received: 11,865,100 PLN.

Part of this essay was based on the results of a research project commissioned under National Centre for Film Culture grant no. 461/FDE/BSU/2018.

Some of the results featured in this essay were taken from a research project conducted under #eeagrants NCN IDEALAB grant no. 2019/35/J/HS6/03166.

The study was funded with a Ministry of Science and Higher Education grant for large research infrastructure, Application no. IA/SP/0165/2017.