During the first months of the TACTILITY project, the consortium mainly worked in designing and setting up the necessary tools and systems and starting the research.
The main result achieved during the first months of the TACTILITY project was the Alpha demonstrator. This is the result of the technical efforts dedicated mainly in WP2 to 7 to the development of the different components of the system and to their validation and integration into the Alpha demonstrator. Several versions of the Alpha stimulator, electrodes and hand garment were manufactured and distributed between partners. Also, a second version of the kinematic glove for hand tracking was designed. Similarly, several efforts were devoted to the research on the haptic feedback and development of the SW modules for WP3 and WP4. The first system including the Alpha electrotactile feedback and kinematic glove and the SW modules necessary for the integration of those components and scenarization in the VR environment was developed and integrated into the Alpha demonstrator.
Alpha prototype allowed starting the research on electro-tactile feedback in WP3, where some preliminary results regarding the psychometric assessment were already obtained. The work continued with the research on the moving patterns and contact combinations and also the dynamic stimulation patterns that will allow conveying intuitive tactile feedback with high resolution and communicating multiple variables simultaneously. All these results were integrated in the library of high-fidelity tactile sensations or haptic library.
Alpha prototype also allowed the integration with the VR environment in WP4. The first version of the Unity stimulator control module was finished and along with the Manus VRK SDK allowed the realization of the first prototype integrating kinematic information and electrotactile feedback. Research activities were focused mainly on enabling the interaction with virtual objects using the kinematic information provided by the glove and conducting the first experiments in virtual reality using electrotactile feedback. The results showed good acceptability of the TACTILITY technology and a remarkable performance of electrotactile feedback for contact rendering.
During the same initial period, the work on WP6 related to the telemanipulation scenario was started by implementing the first electronic skin prototype with 32 sensors. This allowed implementing passive touch experiments and to provide feedback to the human user. Research started in a controlled environment to study the relation between the applied force on the skin patch and the measured sensor charge. An electrode matrix was placed on the user forearm as first approach to the processing of tactile data and their encoding into stimulation parameters for electrotactile feedback. Also, the work related to the communication of that information to a remote receiver was started with some results presented in two conferences.
The Alpha prototype was ready around M9 and has been already used to show to selected partners in the industrial and scientific fields the capabilities of the TACTILITY system in a Mixed Reality environment. A passive touch version of the Alpha demonstrator was used for the first workshop with stakeholders (see pictures below).
With the knowledge acquired during the development and testing of the Alpha prototype, the specifications of the Beta prototype were defined and the design of the individual components for the Beta demonstrator was started. The main goal for the Beta stimulator was focused on functional improvements and increasing the number of channels while achieving a size miniaturization at the same time. For the electrodes, the work was focused on designing the first electrode to cover a big portion of the hand. The work to manufacture the first version of the electrotactile glove was also started by designing the shape and size of the glove and laminating the electrodes to the textile. All these efforts allowed having the first version of the complete Beta ES glove with 32 channels of electrotactile stimulation (pictures below).
On the software side the initial version of the Unity stimulator control was very much improved allowing complete manual control of the stimulation parameters through sliders and buttons, or at a higher level some automated routines for feedback modulation in amplitude, pulse width and/or frequency. This modulation allowed providing feedback of physical characteristics such as force, and also allowed integrating a first version of the haptic library on the Unity stimulation control. Finally, a preliminary method for calibrating ES amplitudes was also developed and included in the software module.
With these components and the outer kinematic glove used for hand and finger tracking, two different demonstrators were implemented to showcase the functionality of the system. The following picture is a screenshot of the rain forest demo, where the user can feel the rain falling in the palm of the hand. The rain ranges from a light rain causing a lighter sensation to a heavy rain causing a stronger sensation.
The second demonstrator was related to the industrial sue case proposed in the project. The major challenges of this demonstrator were a proper collision handling simulation between the user’s virtual hand and the virtual environment, the usage of the USC to control multiple pads simultaneously and the authoring of the tactile effects. The VR scene consisted in a workshop where the user could interact with different parts of the car (see pictures below).
This Beta demonstrators were showcased on the Laval Virtual 2021 event, with around 150 people visiting the booth and 50 of them testing the system with positive feedback in general.
The use case related to the remote operation of a system (tele-manipulation) also continued by developing the first e-skin prototype with 6 piezoelectric sensing arrays (electronic skin) and embedded electronics for signal acquisition and conditioning. A complete set of sensing arrays with different geometries and sensor distributions was developed to fit the fingers and palm of a Michelangelo prosthetic hand. This allowed implementing the first demonstrator of the sue case, with the system being able to recognize and provide visual feedback of the passive touch over the hand.
Again, all the development and research activities done with the Beta demonstrator and the feedback obtained from external testers, allowed gaining insights of the functionalities to be added or potential improvements of the system. Thus, the work continued on defining those as specifications for the Gamma demonstrator and the derived ones for each of the components of the system and continuing the development for improving the different components of the demonstrator during the last months of the project.
This effort allowed several of the individual components being redesigned or largely improved, and when the design was finished and the component properly tested, being integrated into the initial Beta demonstrator to improve its functionality until reaching the final Gamma demonstrator with all the components already updated.
An intermediate version with components from Beta but already several updates of the Gamma prototype was demonstrated during a workshop organized in San Sebastian on May 2022.
The final Gamma ES glove integrated the Gamma stimulator and the v6 gloves that were divided into two different parts, one for the fingertips and one for the palm and more proximal phalanges. Gamma stimulator was designed to improve a little more yet the miniaturization while keeping the 32 configurable or 64 fixed output channels and improving the output precision and other functionalities like to impedance measurement or user interface access and aesthetics. Also, the mechanics were reviewed to increase robustness and quality of contact with the electrode. The 64 pads electrodes design included a unique anode, design approach protected by a patent.
The ES glove including the electrodes was also redesigned according to the previous knowledge and information and to the two separate pieces of the electrode.
The Gamma version also included a new kinematic glove designed for improved tracking of finger position based on Electro Magnetic Field technology. The Manus Metaglove is the result of the efforts dedicated to improving this design, achieving millimeter precision, with much higher repeatability and reliability on finger movement and pose tracking.
With these last components design, manufactured and properly validated, the final version of the Gamma prototype including ES inner glove and kinematic outer glove was finished.
The same way that all the physical components of the system were redesigned and improved form Beta to Gamma version, the software components were also redesigned and improved. The Unity Stimulator Control included more functionalities to support the 64 channels’ version, the authoring capabilities extended to support the new hardware with new UIs added and the multipoint feedback by replicating the effects developed for a single finger. Finally, the capability of actuating on different pads at the same time but independently was also included in Gamma version of the USC. All those modifications with the addition of the new calibration methods and the different haptic effects, turned in the proposal of a unified architecture in order to define how interactions in the virtual reality environment were rendered using electrotactile feedback, which was called electrotactile rendering pipeline. Part of this pipeline is device agnostic and could be used with any other stimulator or any other haptic feedback device, and the other part is very much related and optimized to the ES glove design for Tactility.
With the latest components of the Gamma prototypes, several different implementations of Gamma demonstrators were developed. The automotive use case and a specific demonstrator developed to show the potential of the new kinematic glove have been shown to selected stakeholders in private meetings.
The health-related demonstrator focused on phobias to cockroaches was demonstrated to several psychologists and researchers to get their feedback, which was positive in general terms. Including comments such as it seemed a useful environment for clinical purposes and they were willing to introduce it in their regular practice and/or recommend to their patients, because it provides a comprehensive sensation adding value to the experience.
Very similarly to what has been shown above, for the telemanipulation scenario the work progressed during the last months of the project to achieve the final Gamma demonstrator. This final demonstrator includes the mean of controlling remotely hand by a joystick or EMG signals, the Michelangelo hand sensorized with the 64 channels’ electronic skin last version and the embedded electronics that read those signals and convert them in a touch signal sent through Internet to a remote PC were it is converted into an electrical stimulation feedback provided to the remote operator via the Gamma ES glove.
