Technology

There are many technological modalities of interaction with digital surfaces controlled remotely by speech and gesture. These functionalities allow users to interact without holding or touching a physical device. However, this distance, perhaps caused by audiovisual display devices, somehow makes the virtual world lack “physical” veracity, namely due to the lack of tactile stimulation in virtual objects. Nevertheless, despite the significant progress in the development of tactile feedback technologies, most of the patented brands on the market provide contact controllers. That is, tactile feedback devices in the form of joysticks, gloves, belts, waistcoats, suits, etc., which, for the most part, are physical extensions of a computer or wearable computers. Thus, although these devices can offer a rich stimulation in haptic sensations, they condition the sensory experience to only one user and one part of the body, an aspect that weakens the immersive component and the purposes of this PhD project. Therefore, and taking into consideration the desire of free movement of the visitors in the installation space, we understand to be primordial to produce an art installation that, not depending entirely on wearable computers, promotes a collective experience with the artistic object. On the other hand, and in a more pragmatic way, it would be counterproductive to restrict that experience to the number of available devices or, even, that the possible experience be framed only in a simple technical effect. In that sense, using this constraint as a creative elixir and a source of motivation, several haptic feedback technologies that could fit the stipulated premises were identified. Therefore, it was considered essential to focus on non-contact haptic technologies. Following this census, we verified the potential of some of these devices in loco at the European conference Eurohaptics 2018. On the other hand, some technologies related to sound installation were also census. In this way, we intend the sound device to exercise two main functions: to generate acoustic pressures and to establish, through sonic ambiences, and reorder the visitor’s perceptive sphere in the apprehension of reality. The intention is, therefore, to create a uniform dispersion of low frequencies and some sound notes in high frequencies for spatial orientation. There was also the will to objectively analyse the somatosensory behaviour of the various visitors through biometric reading. Although there is not a wide variety of professional devices that can be easily instrumented, it was possible to identify some companies that provide services compatible with the outlined objectives. However, after the change of the room and given the impediment of producing events that could raise the number of visitors for the validation of a scientific study, and in agreement with the project supervisor, this study ceased to have effect. Even so, we will now describe the various technologies identified.

Technological survey

There are many technological modalities of interaction with digital surfaces remotely controlled by speech and gesture. These functionalities allow users to interact without holding or touching a physical device. However, this distance, perhaps caused by audiovisual display devices, somehow makes the virtual world lack \”physical\” veracity, namely due to the lack of tactile stimulation in virtual objects. Nevertheless, despite the significant progress in the development of tactile feedback technologies, most of the patented brands on the market provide contact controllers. That is, tactile feedback devices in the form of joysticks, gloves, belts, waistcoats, suits, etc., which, for the most part, are physical extensions of a computer or wearable computers. Thus, although these devices can offer a rich stimulation in haptic sensations, they condition the sensory experience to only one user and one part of the body, an aspect that weakens the immersive component and the purposes of this PhD project. Therefore, and taking into consideration the desire of free movement of the visitors in the installation space, we understand to be primordial to produce an art installation that, not depending entirely on wearable computers, promotes a collective experience with the artistic object. On the other hand, and in a more pragmatic way, it would be counterproductive to restrict that experience to the number of available devices or, even, that the possible experience be framed only in a simple technical effect. In that sense, using this constraint as a creative elixir and a source of motivation, several haptic feedback technologies that could fit the stipulated premises were identified. Therefore, it was considered essential to focus on non-contact haptic technologies. Following this census, we verified the potential of some of these devices in loco at the European conference Eurohaptics 2018. On the other hand, some technologies related to sound installation were also census. In this way, we intend the sound device to exercise two main functions: to generate acoustic pressures and to establish, through sonic ambiences, and reorder the visitor\’s perceptive sphere in the apprehension of reality. The intention is, therefore, to create a uniform dispersion of low frequencies and some sound notes in high frequencies for spatial orientation. There was also the will to objectively analyse the somatosensory behaviour of the various visitors through biometric reading. Although there is not a wide variety of professional devices that can be easily instrumented, it was possible to identify some companies that provide services leading to the outlined objectives. However, after the change of the room and given the impediment of producing events that could raise the number of visitors for the validation of a scientific study, and in agreement with the project supervisor, this study ceased to have effect. Even so, we will now describe the various technologies identified.

Non-contact haptic feedback devices

AirWave

This technology developed by Microsoft uses air vortex rings as haptic feedback in the user’s interaction with a piece of software. Unlike standard air jets, which are turbulent and dissipate quickly, vortex rings can be focused to travel several metres and transmit a perceptible tactile response. Thus, based on this discovery, a group of Microsoft researchers, revisiting Lord Kelvin’s (1867) descriptions of an air vortex formation generator, explored specific parameters in order to generate acoustic pressures capable of transmitting a tactile response. Furthermore, through objective measurements, the AirWave can achieve a spatial resolution of 10 cm at a distance of 2.5 m. In these proposals, the researchers also demonstrate that these vortices can be used to direct tactile stimuli in different regions of the human body. The AirWave prototype did not manage to get beyond the prototyping phase nor to achieve a market statement. Still, we take this technology into account. However, given the premise of our project, we thought that this technology would not be suitable for it, as the special characteristics of these devices and all the necessary apparatus – i.e. the amount of devices needed –to create the desired collective experience would greatly overload all the logistical and financial aspects adjacent to the production of this art installation.

The AirWave prototype emits a vortex of smoke to prove non-contact haptic stimulation.

(a) As the air spot is pushed out of the aperture, the boundary layer starts to curl outwards as it exits, (b) causing the vortex to detach and (c) the AirWave prototype to be filled with mist to create a visual vortex ring just after exit.

AIREAL Disney Research

Combining distance reading with the great effectiveness of a 3D camera (3D Depth camera) and with small low-frequency sound speakers and mechanical motion motors, AIREAL allows, through gesture recognition in free space (Mid-air), the haptic stimulation of virtual objects in three-dimensional space. In other words, this technology allows the production of non-contact haptic stimuli based on user interaction.

Therefore, this new generation of directed air vortices aims to provide an effective tactile response in a field of view of 75 degrees and with a resolution of 8.5 cm at 1 m. Furthermore, this technology allows the integration of multiple vortex emitters, thus optimising the desired effect to the size of the intended area at relatively low cost. However, there are several limitations of our current device.

First, the AIREAL device produces an audible sound (“knock”) due to the physical emission of low frequencies but high amplitude by small loudspeakers. Second, although AIREAL does not require active user instrumentation, this device still requires passive instrumentation and needs the user’s interaction area.

Although the potentialities of this device are numerous, its disadvantages might minimize the ambition of the Hapticotopia project. As a result, having in the forefront the desire to achieve immersion, it is easy to understand why the noise generated by this device would be a disadvantage. That is, the gradual increase of devices would also be proportional to the noise emitted, and this would mean an experience exactly opposite to the intended immersion experience.

The AIREAL device emits a ring of air called a vortex, which transmits physical forces that a user can feel outdoors.

A fully assembled AIREAL device.

Continuous tactile sensations in the open air surround the user.

STRATOS Explore Ultraleap

Embedded in a grid of several small ultrasonic frequency emitters in such a way as to create through the air a tactile response, the disposition developed by the company Ultraleap (until 2019 Ultrahaptics) is able, through focal points of pressure, to induce a tactile sensation on the skin. Thus, rapidly and repeatedly updating the position of a focal point along a given trajectory can draw two-dimensional curves (referred to as patterns) on users\’ palms. Created to complement the purposes of virtual reality (VR) and augmented reality (AR), this device achieves an extreme feasibility of synchronism between visual and tactile stimuli. The latest vibrometry studies have shown that speed optimization can maximize the skin deflection caused by a focal point. They therefore produce sensations with greater acuity in the stimulation of tactile pattern design.

The small dimensions, in its modular aspect – the possibility to attach several devices in a matrix – and the success in the international market concerning the technological sector of non-contact haptic devices make this technology a strategic partner in the construction of this prototype for an art installation. Moreover, the educational and support programme for textures programming and the support to university projects by the company Ultraleap also allow to trial the technology for a period of three months without additional costs.

STRATOS Explore Toolkit

Visual simulation of the focal point of ultrasound wave emission.

STRATOS Explore device

Sound systems and loudspeaker configuration

Dolby Atmos Home

Like all the sound systems mentioned below, Dolby Atmos Home is part of what is called Spatial Sound. One of several sound distribution techniques that aim to provide an immersive experience to the user. As this system is aimed at displaying audiovisual content, its layout itself comprises the fixed position of several spectators. Thus, it is understood that there is a specific and immobile direction for an extended period of time.

So, although this is a technology that dominates the consumer and professional market (in Home Cinema), we see with some reluctance the limitation of movements of possible visitors in the art installation. We also understand as disadvantages the existence and dependence on a listening point (sweet-spot: where the fullness of the mix can be grasped with some acuity).

A traditional 5.1 speaker layout: five listener-level speakers and four Dolby Atmos speakers.

Dolby Atmos speakers allow you to reflect sound off the ceiling to produce an incredibly realistic overhead sound experience.

Soundscape d&b audiotechnik

Designed to give an audience greater immersion at events – such as concerts, theatres, operas – the latest sound system from d&b audiotechnik Soundscape proposes a different approach to common performance sound systems. By integrating at its genesis a 64 x 64 matrix and the arrangement of a continuous line of loudspeakers along the walls of the performance hall, it provides an immersive 360 degree sound experience. With different software modules, this sound system offers artists, sound designers and audio specialists new tools to express their creativity. Like the sound system mentioned above (Dolby Atmos home), the sound mixing of various elements is based on \”objects\” that group together a defined number of audio tracks. In this way, and through various algorithms, this recent technology (2018) emulates the psychic acoustic perception of listeners in event rooms.

Given the design purposes of the Hapticotopia art installation, this seems to be an excellent option. However, the logistical and financial effort involved in the implementation of this device may jeopardize the conceptual principles of the installation prototype to be produced. That is, although it is possible to reposition the various sound elements in the event room, there is still dependence on a listening reference point. This means that the arrangement of speakers obeys the positioning of a stage and, therefore, a predefined listening direction.

The graphic shows a system that can place objects throughout the audience. The side and rear loudspeakers provide positioning of larger objects as well as accurate reverb emulation.

The loudspeakers (red) provide the relevant direction information to the listener, scaling the sound intensity relative to the target point.

Positioning area with sound objects, level indicators (green) and sound dispersion (yellow).

Sound4D

In 2015, a sound system based on omnidirectional loudspeakers emerged, produced in Amsterdam, the Netherlands. In a 16 by 16 metre room with 16 metal mesh pillars each containing 6 speakers, the Sound 4D system makes it possible not only to position sound objects spatially, but also to change their size and shape. In other words, sound mixing is no longer done by mixing the various signals for two outputs (stereo: left and right) but by creating geometric shapes in space. Through software created from scratch for this purpose, the entire audience experience is controlled in a virtual sound sphere. The performer can even control the entire panoply of polygons through gesture (with Leapmotion technology, the same that is installed in the Ultrahaptics Stratos Explore kit).

Still, despite being the technology that best adapts to the premises defined for this project, its character of continuous development and the minimum size inscribed for the ideal dispersion of sound frequencies limit the adaptability of the prototype to rooms with predefined characteristics. Moreover, as we can see at the Sound 4D Institute in Budapest, there are still some deficiencies in the interconnection between software and hardware – as expected of a prototype – and in the latest version of omnidirectional loudspeakers.

Main room of the Sound 4D Institute in Budapest.

Max SP connections allow two hands and ten fingers connected by LeapMotion to stimulate and trigger virtual spatial objects and instruments.

Spatial blending visualised with 4D.animator software.

Portable biometric reading devices

E4 Empatica

Allowing researchers to perform in-depth analysis and visualisation of the human body\’s biometrics, the E4 Empatica offers real-time physiological data acquisition for medical use. This wristband has the possibility to make several measurements the blood volume pulse (BVP), from which heart rate variability can be gauged (PPG Sensor); motion-based activity capture (3-axis Accelerometer); tag events and associate them with physiological signals from the blood volume pulse (BVP), from which heart rate variability can be gauged; measure constantly fluctuating changes in certain electrical properties of the skin (EDA Sensor, GSR Sensor); and measure peripheral skin temperature, among other functions.

The easy instrumentation and the acuity of the collected data fulfil all the necessary requirements to integrate this project. However, a possible loan agreement is out of the question for the company. Therefore, one of the devices will have to be purchased at a reduced price. Thus, this factor may condition the possible integration of E4 Empatica in this project.

Assembly and instrumentation of the device.

Instant connection to other devices via Bluetooth and Cloud.

Everion Biovotion

Everion is a true multi-sensor platform that currently provides 22 measurement parameters and various data analyses: collection of vital sign parameters in a professional and clinical environment (heart rate, skin temperature, respiratory rate, blood oxygenation). Additional clinical and non-clinical parameters (blood pulse wave, movement and step intensity, energy expenditure, sleep quality, heart rate variability, Inter-beta-interval and a stress score) provide supplementary insights of analytical value. The platform is expandable and new parameters can be added to broaden the spectrum of measurements. In addition, Everion provides advanced analytics through artificial intelligence and Data Learning for predictable, accurate and personalised insight.

In line with the device previously exposed, the potentialities mentioned also fit in the premises of this project. Unfortunately, having already held the loan agreement, due to the lack of insurance viability and financial support for that purpose by UBI, which delayed the whole research plan, the tenuous bond fell into disrepute, not being possible to make a new agreement.

Everion device. There are no buttons, no cables, no tape and no need for calibration. It is comfortable to wear on the arm, where it collects data in real time, continuously and non-invasively, 24/7.

Provides data and insights, as it is only possible in a clinical environment. A truly unique application capable of informing, motivating and guiding towards better health.

PLUX – Wireless Biosignals

Biosignalsplux was born because standard biosignal sensors cannot always meet all needs and provide high quality data in extreme conditions or under special requirements. In this sense, this solution produces biosignal platform bases to accelerate the development of new custom sensors and to create a specific solution that ensures the acquisition of high quality and reliable signals. This company develops new sensor solutions for the specific needs of your applications: customised solutions to meet your requirements for signal processing, sensors and raw data analysis to convert meaningful information that would serve as a basis to support future decisions. It also produces wearable devices and Cloud applications with access to mobile devices.

Of the referenced devices, this is the one that offers the highest signal acuity. However, its time-consuming instrumentation presents the biggest limitation to the desired simplicity and speed in the course of the whole experience. Inclusively, this fact could even influence the interpretative process of the art.

Plux Professional high-performance real-time acquisition device for breathing and movement even in dynamic conditions.

Plux Professional instrumentation mode with full data logging capability and generic ports for extra sensors.

Technology Census Findings

Technology Census Findings

All the technologies previously listed were subjected to an evaluation process, which is governed by the balance between quality, size and cost. As mentioned above, many of the technologies belong to international manufacturers, and this was a factor considered when acquiring and/or applying for loans. In this way, reference should be made to the effort that needs to be spent on all logistics and coordination, allocated to transport, insurance involved in the planning and involvement of external partners in this project. However, it should be noted that throughout the journey already completed, this project has been well received, especially by foreign entities.

Regarding the three application areas of the aforementioned technology – Non-contact haptic feedback technologies, Sound System and Biosignals – it is concluded that:

1. In the non-contact haptic area, the enormous innovation and the restitution of old principles, such as the displacement of air in rings generated by vortices, stand out. However, the two proposals – AirWave and AIREAL -, due to constraints related to the prototype and the unwanted noise emitted by the air cannons, did not reach a state of maturity in the market and, therefore, also did not prescribe the desired immersion effect in the prototype. Thus, the selection fell on the company Ultraleap, namely the Stratos Explore development kit.

2. In the area dedicated to sound display systems, it is important to note the “size” index as a decisive factor in this selection. That is, there is an unconditional relationship between sound propagation and the room in question. Therefore, due to the uncertainty regarding the exhibition room and the costs related to the installation of the sound system, it was deemed necessary to opt for the adequacy of an audio system to the architectural characteristics of the exhibition room. As such, we chose to install a sound system that, through the simulation of an acoustic prediction of the room, will be adapted to the exhibition space.

3. In the area referring to biometric reading, as already mentioned, the “size” factor was central in the selection process: not only would it be necessary to have an easy, fast instrumentation that remained unknown to visitors, but it would also be indispensable to achieve extreme acuity in the collection of biodata. Therefore, the companies Empatica and Biovotion were chosen. Given that with Empatica there was only the possibility of purchasing biometric wristbands at a reduced price, in an attempt to reduce costs, the selection fell on the loan agreement with Biovotion. However, as previously mentioned, following the lack of support regarding the equipment insurance, the company gave up the previous loan agreement due to the logistic obstacles put by UBI. Ergo, and as a consequence of the set of events and postponements, also motivated by the pandemic situation installed in the world, this possibility of cooperation was cancelled.