Special Articles on 5G Evolution & 6G (2)—Initiatives toward Implementation and Use Cases—
Human Augmentation Platform for New Styles of Communication

Human Augmentation Motion Sharing Haptic Sharing

Syunpei Miyaura and Takayuki Amano
6G Network Innovation Department

Abstract
NTT DOCOMO is developing a platform for achieving “human augmentation” that will extend human capabilities over a network as one of the new values to be provided in the 6G era as described in “NTT DOCOMO 5G Evolution and 6G White Paper.” In the 6G era, connecting information from a person's brain and body to a network will make it possible to experience the sensation of actually being present elsewhere via the network, which is expected to provide new value and solutions for a variety of social problems. This article describes the Human Augmentation Platform that can share sensations and motions as well as personal skills with another person.

01. Introduction

• Extremely low latency-a feature of the 6th-Generation mobile ...

  Open

  Extremely low latency—a feature of the 6th-Generation mobile communications system (6G)—will enable the speed of communications in a network to exceed the response speed of the human nervous system in 6G. As a result, connecting information from a person's brain and body to the network should be able to extend human capabilities via the network [1]. Technology for enhancing human abilities is called “human augmentation” [2], but the Human Augmentation Platform (HAPF) being developed by NTT DOCOMO is more concerned with sharing experiences beyond time and space than enhancing muscular strength such as by a powered exoskeleton. There have been many studies to date on human augmentation, but these have generally been proposals related to elemental technologies focusing on the development of individual devices or services. In contrast, NTT DOCOMO is concentrating its efforts on constructing platform technology for human augmentation.

  At NTT DOCOMO, our goal is to achieve “ubiquitous body,” “sharing of skills,” “sharing of emotions,” “telepathy/telekinesis,” and “sharing of five senses” as directions of human augmentation (Figure 1). In this article, we first provide an overview of HAPF. We then describe “motion sharing” as technology developed for achieving a ubiquitous body and enabling the sharing of skills from among the above five directions and “haptic sharing” from among the five human senses as technology developed toward the sharing of the five senses.

  

02. Overview of HAPF

• The overall configuration of the HAPF system ...

  Open

  The overall configuration of the HAPF system consists of sensing devices for grasping a person's state, actuation devices^*1 for reproducing that state in another body, and HAPF for connecting these two types of devices (Figure 2). In this system, HAPF compares the sensed information with body information on the actuation-device side, applies that difference to the sensing information, and transmits the resulting sensing information to the actuation devices. For example, motions can be shared in an appropriate form according to the range of body movement, or an easy-to-recognize sense of touch can be presented according to haptic sensitivity. The magnitude of a shared motion can also be expanded or reduced. In other words, HAPF can achieve smooth and natural motion sharing between people and robots of different size, physique, etc. and can reproduce precise or delicate movements based on large movements. In addition, HAPF can be interconnected with a variety of devices from partner companies, and collaboration with partner companies having their own sensing and actuation technologies can be promoted by providing a Software Development Kit (SDK)^*2 that can be easily connected to HAPF.

  

  1. Actuation device: While a sensing device detects a physical movement in the outside world and converts that to a signal, an actuation device generally refers to equipment that generates a physical movement such as motor rotation according to an internal signal and acts on the outside world.
  2. SDK: A set of documents, tools, libraries, sample programs, etc. needed to create applications.

03. Motion Sharing

• Motion sharing means the sharing of one's own movements ...

  Open

  Motion sharing means the sharing of one's own movements with another person, robot, and avatar. For example, sharing your own motions with a robot situated at a remote location would enable you to operate the robot as if you were actually there. You could also record a series of motions as a skill and reproduce them at any time or share them with another person. In the future, we can expect motion sharing in this way to enable all sorts of physical skills to be downloaded and acquired for personal use in much the same way that various types of applications are downloaded to an information device for personal use.

  3.1 System Configuration

  The basic system for motion sharing consists of a sensing device that detects a person's motions, an actuation device that reproduces the detected state of that person at another person, robot, etc., and HAPF that connects both sides.

  (1) Sensing device

  Photo 1 shows FirstVR [3] developed by H2L, Inc., an NTT DOCOMO business partner, as a device for detecting motions. This device uses a muscle displacement sensor^*3 and acceleration sensor^*4 to detect motions at the part of the body wearing the device.

  (2) Actuation device

  Photo 2 shows PossessedHand, an actuation device also from H2L. This device uses technology that can induce any motion by electrically stimulating muscles [4].

  (3) HAPF

  HAPF compares the range of motion of each of the bodies sharing a motion and shares that motion within an appropriate range. For example, it can be expected that motion sharing with a person having a narrow range of motion will be conducted within a range generating little physical burden on that person. HAPF can also store information on a series of motions obtained from sensing devices as a skill and share those motions via actuation devices at any time. This process would also take the range of motion of the target body into account as described above.

  

  3.2 HAPF Operation

  When sharing a motion, the data linkage control section of HAPF controls actuation based on operation information stored in the operation database that registers sensed operation information and on body information stored in the body database that registers body information. Specifically, the data linkage control section controls actuation using sensing data in the operation database based on the range of motion indicated by body information registered beforehand in the body database for the other person targeted for operation sharing.

  For example, given two persons (A and B) designated for motion sharing, let the range of inward rotation of the right arm be 0 – 120° for A and 0 – 90° for B. In the event that control information for motion sharing in the range 60 – 110° is delivered from A to B, past technology would attempt to move the arm as far as 110° beyond the range of motion of B. In contrast, the data linkage control section of HAPF will modify the control information to 60 – 90°, the range of motion of B, so as to limit that movement. Therefore, this motion can be shared within a range reasonable for the target body by controlling the arm's range of motion.

  When sharing a skill, the data linkage control section performs control based on a skill database that registers a time sequence of motions in addition to the configuration used when sharing a motion. The skill database records motion information from the sensing device from the operation at the beginning of the recording to the operation at the end of recording. This recording process links the device used in sensing to the target body information. The data linkage control section then obtains (plays back) this motion information from the skill database at any time and controls actuation while taking into account body information the same as when motion sharing.

  1. Muscle displacement sensor: A sensor that detects the state of a muscle in relation to input force. It is also used to freely operate a robot, etc. by the movement of one's own arms or hands.
  2. Acceleration sensor: A sensor that detects attitude (inclination) or extent of acceleration in the linear direction. There is also an angular acceleration sensor that detects acceleration in the rotational direction. They can be used for navigation, camera shake compensation, and input into sensory video games.

04. Haptic Sharing

• Focusing on touch as one of the five human senses, ...

  Open

  Focusing on touch as one of the five human senses, haptic sharing aims to share a “haptic sensation” in addition to “seeing” and “hearing” achieved by existing media. NTT DOCOMO has developed FEEL TECH^*5 technology that shares the sense of touch experienced when touching an object via HAPF tailored to the other person's degree of feeling. The aim here is to share a haptic sensation of an object that cannot be conveyed by video, audio, or text. In the future, we can expect this technology to be applied to such fields as medical treatment and traditional crafts. It could be used, for example, to enable a physician to perform palpation on a remotely located patient via a robot.

  4.1 System Configuration

  The FEEL TECH system consists of a sensing device that detects the sense of touch when a person touches an object, an actuation device that reproduces that detected sense of touch in another person, and HAPF that connects both sides.

  (1) Sensing device

  Photo 3 shows a sensing device developed by the Haptics Lab at the Graduate School of Engineering, Nagoya Institute of Technology, which is an NTT DOCOMO partner, as a means of detecting the sense of touch. This device uses a piezoelectric element to sense vibrations when a person touches an object.

  (2) Actuation device

  Photo 4 shows an actuation device developed by the Graduate School of Media Design, Keio University, which is also an NTT DOCOMO partner, as a means of haptic sharing. This device reproduces a haptic sensation using a transducer.

  (3) HAPF

  HAPF converts information taking into account individual differences in haptic sensitivity between the sensing device and actuation device. This function enables even a person with low haptic sensitivity to perceive differences in haptic sensations felt by a person with high haptic sensitivity.

  

  4.2 HAPF Operation

  When haptic sharing, the data linkage control section of HAPF controls actuation based on operation information stored in the operation database that registers sensed haptic information and on body information stored in the body database that registers individual sensitivity characteristics as body information. Specifically, the data linkage control section controls actuation using sensing data in the operation database while taking into account haptic sensitivity characteristics registered beforehand in the body database. This process begins by measuring individual sensitivity characteristics beforehand by presenting a user with vibrations at three different frequencies and measuring the minimum amplitude that can be perceived at each frequency. As shown by the graph in Figure 3, these three measurement frequencies correspond to locations that involve the Meissner's corpuscle^*6 and Pacinian corpuscle^*7, which are subcutaneous sensors related to vibration within the sense of touch. The sense of touch can now be shared based on individual sensitivity characteristics registered beforehand.

  For example, let's assume that the measurement results of two persons (A and B) targeted for sharing a haptic sensation show high sensitivity for A and low sensitivity for B. Now, if A is to share a haptic sensation with B, the data linkage control section controls the sensation to be conveyed to B by amplifying the vibration taking into account the low sensitivity of B. In short, person B, despite having low sensitivity, is able to feel differences in haptic sensations in the same way as person A having high sensitivity.

  

  1. FEEL TECH: A trademark or registered trademark of NTT DOCOMO.
  2. Meissner's corpuscle: One type of receptor located within the skin for sensing uneven vibration stimuli such as Braille characters.
  3. Pacinian corpuscle: One type of receptor located within the skin for sensing rough or smooth vibration stimuli, such as a file or silk, and detecting a sense of impact, etc.

05. Conclusion

• This article provided an overview of HAPF and described ...

  Open

  This article provided an overview of HAPF and described the technologies developed for achieving “motion sharing” and “haptic sharing.” We point out here that HAPF is equipped not only with functions for absorbing differences in body information and for recording and reproducing motions as described in this article but also with device management functions and a service user interface. In addition, various types of sensory quantities other than the data described here can be handled on the same platform. These features are expected to reduce the development, maintenance, and operation workload of device manufacturers and service providers and to facilitate the creation of even more human augmentation services.

  In addition to the above developments, NTT DOCOMO has been presenting these technologies at exhibitions and elsewhere to enhance the visibility of its achievements in this area and hunt for partner companies. In FY2022, we exhibited HAPF at docomo Open House'23 [5] and Mobile World Congress (MWC) Barcelona 2023 [6] (Photo 5). These exhibitions attracted the interest not only of leading broadcasters in Japan but also of international media outlets, so we were able to disseminate information about the attractiveness of HAPF as a platform independent of national characteristics and culture.

  Going forward, we plan to extend HAPF to the sharing of the human senses other than touch to enable the sharing of information that cannot easily be understood using only video or text. By connecting to feelings, physical memories, and experiences that people may have wanted to convey up to now but could not, NTT DOCOMO aims to create a “world of mutual understanding.”

  

• REFERENCES

  Open

  1. [1] NTT DOCOMO: “5G Evolution and 6G White Paper Version 5.0,” Jan. 2023.
     https://www.docomo.ne.jp/english/binary/pdf/corporate/technology/whitepaper_6g/DOCOMO_6G_White_PaperEN_v5.0.pdf (PDF format:4,110KB)
  2. [2] J. Rekimoto: “Internet of abilities: Human augmentation, and beyond (keynote),” 2017 IEEE Symposium on 3D User Interfaces (3DUI), Apr. 2017.
  3. [3] H2L: “FirstVR | Move the World with Your Hand—Toward a Domain of Intuitive Operation through VR” (in Japanese).
     http://first-vr.com/
  4. [4] E. Tamaki, T. Miyaki and J. Rekimoto: “PossessedHand: Techniques for controlling human hands using electrical muscles stimuli,” Proc. of the SIGCHI Conference on Human Factors in Computing Systems, pp. 543–552, May 2011.
  5. [5] NTT DOCOMO: “docomo Open House'23.”
     https://www.docomo.ne.jp/english/corporate/technology/rd/openhouse/openhouse2023/
  6. [6] GSMA MWC: “MWC Barcelona 2023.”
     https://www.mwcbarcelona.com/

VOL.25 NO.3