Special Articles on 5G Evolution & 6G (2)—Initiatives toward Implementation and Use Cases—
Innovative Technologies for Remotely Operated Robots Pioneered by Integrating Cellular Communications and RealHaptics

Remotely Operated Robot The Force/Tactile Sensation Communication URLLC

Yutaka Musaka, Ryota Horiuchi and Yoshifumi Morihiro
6G Network Innovation Department

Abstract
Social demands for cooperative robots such as remotely operated robots are further increasing, and there are high expectations for “RealHaptics^®*1,” robotics technologies that can reproduce flexible and gentle human-like movement. Because RealHaptics deals with the delicate sense of touch that humans have, it has very high latency requirements compared to other applications, and thus has not yet been put to practical use via cellular communications. Therefore, NTT DOCOMO conducted a field experiment using 5G URLLC to show the feasibility of radio remote control with RealHaptics. This article describes the one-of-a-kind world that can be achieved by integrating cellular communications and RealHaptics, and its feasibility.

1. RealHaptics^®: A registered trademark of Motion Lib, Inc. in Japan, USA and China.

01. Introduction

• In Japan, where there are concerns about future labor shortages ...

  Open

  In Japan, where there are concerns about future labor shortages due to the declining birthrate and aging population, there are growing social demands for robots that can replace humans as new workers and for robotics that enables such replacement. Looking to the future of robotics, NTT DOCOMO is focusing on “RealHaptics,” a robot control technology developed by Keio University Haptics Research Center (hereafter referred to as “Keio University”).

  RealHaptics uses the force/tactile sensation communication technology^*2 to allow the remote operator to sense the response of objects that the remotely operated robot touches. In conventional remote operation, the operator must visually judge the grip and the amount of force applied, and the operator's sense of control is required. In contrast, with remote operation using RealHaptics, the operator can operate the robot more intuitively while receiving realistic feedback of the response of objects from the remotely operated robot. In this way, RealHaptics enables flexible and gentle robot control based on the operator's intuitive force control.

  Keio University has proposed the concept of the Internet of Actions (IoA)^®*3 and has positioned RealHaptics as an important technology forming the backbone of IoA. IoA is a concept of a world in which people's actions such as cooking and cleaning are brought together (mutually provided) through remote control and network-connected robots spontaneously perform flexible actions like humans by dynamically connecting machines, people, and operation content through networks. RealHaptics, a key technology of IoA, is expected to promote co-creation between humans and robots and contribute to solving social issues and further streamlining society.

  NTT DOCOMO has been promoting R&D with Keio University, recognizing the unique innovation of RealHaptics and its affinity and developmental potential with the networks needed to achieve IoA [1]. This project focuses on the remote operation of robots using RealHaptics, which could only be achieved via cable communications, via cellular communications. In addition, NTT DOCOMO is developing a platform to further expand the world of IoA and is considering functional enhancement to RealHaptics based on human tasks. Through these R&D efforts, NTT DOCOMO aims to create new use cases in the era of the 5th-Generation mobile communications system (5G) Evolution and the 6th-Generation mobile communications system (6G) and provide highly convenient solutions and services unique to cellular communications.

  This article describes the features of RealHaptics, the world of IoA that this project aims to achieve, and the feasibility of RealHaptics via cellular communications.

  1. Force/tactile sensation communication technology: A technology that represents the human senses of force and touch as data and transmits them via telecommunications.
  2. IoA^®: A registered trademark of Motion Lib, Inc. in Japan.

02. Features of RealHaptics

• 2.1 What is RealHaptics?

  Open

  1) Overview of RealHaptics

  RealHaptics is technology that reproduces a realistic “response” generated by contacting with a real object at the operator, who is not in direct contact with the object. Responses that simultaneously evoke the tactile sensation^*4 and force sensation^*5 are difficult to measure with existing sensing technology and have conventionally been considered tacit knowledge^*6. However, RealHaptics is unique because it can quickly estimate the force on the motor based on the position, which is the angle of rotation of the motor, using a proprietary force estimation algorithm and can therefore combine the position and force data to turn the response into quantitative data as explicit knowledge^*7.

  Furthermore, with remotely operated robots, RealHaptics can instantly synchronize the position and force information acquired at high speed between the cockpit of the operator (hereinafter referred to as the “leader”) and the remotely operated robot that is in direct contact with the target (hereinafter referred to as the “follower”), thus satisfying the high standards for bilateral control described below and enabling realistic responses to be shared between both the leader and follower. Leaders can feel these responses even remotely, allowing intuitive control of robots. This one-of-a-kind feature allows RealHaptics to easily reproduce subtle movements that require delicate human senses. The responses obtained through robot operation can be saved and shared as explicit data, which holds promise for application to advanced solutions other than remote operation.

  2) Quantification of Force/Tactile Sensation

  Reference [2] states that force sensation and tactile sensation are difficult to represent objectively because they depend on sensory mechanisms that occur within humans. However, the response that humans perceive is represented as a combination of force and tactile sensation, and can be quantitatively described by the energy transfer that occurs during contact with an object. The magnitude of the force/tactile stimulus is defined as the quantity of force/tactile sensation. The instantaneous value^*8 of the quantity of the force/tactile sensation can be expressed as Equation (1), and the effective value^*9 of the quantity of the force/tactile sensation as Equation (2), as follows. Note that ƒ is the force, ∨ is the velocity, and ΔT is the time window^*10.

  

  3) Bilateral Control

  To convey correct data on force/tactile sensation, the leader and follower must simultaneously fulfill the following two functions [3].

  1. The leader and follower make the same move. In other words, their positions are synchronized.
  2. The action force generated by the follower is equal to the reaction force generated by the leader. In other words, the action-reaction law holds for the entire system.

  The above two functions are collectively called “transparency.” To satisfy transparency, both the leader and follower must be treated as equals, and data about their positions and forces must be transmitted in both directions and controlled to keep them synchronized. Such a control scheme is called “bilateral control.”

  If the transmission of position and force data is delayed, the desired transparency is not achieved, and the reproducibility of force/tactile sensation and the operability of the robot are impaired [4]. This is a major issue when implementing bilateral control via cellular communications.

  2.2 Relevance to “Haptics” Tactile Feedback Technologies

  1) Definition and Characteristics of Haptics

  Haptics is called tactile feedback technology or cutaneous sensory feedback technology. Although it is difficult to uniquely define the term, it is often described as technology that uses devices that can actuate^*11 force, vibration, and motion to artificially create and provide pseudo-tactile sensations to the device user.

  Examples of haptic applications that are familiar to many people include character manipulation in video games and text input on smartphones. In these applications, users can obtain vibrotactile feedback from actuators built into game controllers and smartphones. The former gives the user tactile feedback according to the actions of the character being controlled in the video game, while the latter vibrates a display that has no structures such as physical buttons in response to the user's motions, so as to provide the user with tactile feedback that makes it feel as if the user had pressed a button.

  Thus, haptics are pseudo-reproductions of the sense of touch by means of alternative actuator stimuli that are different from the actual stimuli and excel at creating an illusion for the user. Therefore, haptics has a very high affinity with video games and Virtual Reality (VR).

  2) Comparison of Haptic Feedback in Remotely Operated Robots

  RealHaptics is technology for improving the accuracy of bilateral control, but it can also be considered a type of haptic feedback technology. Figure 1 shows a comparison of vibrotactile feedback and force/tactile feedback in a remotely operated robot. With vibrotactile feedback, the leader can experience the presence and pseudo-texture of distant diamonds through the vibration of the actuator. However, since force level is not considered, the leader cannot obtain a response that would allow them to recognize the shape of the diamond. On the other hand, with force/tactile feedback, not only presence and texture but also realistic force level is shared between the leader and follower, so that the leader can feel the response of actually gripping the diamond, just as it is too hard to crush, even though it is not actually with the leader.

  

  1. Tactile sensation: A sensation that occurs when a human being and an object lightly touch each other. In this article, it is distinguished from force sensation.
  2. Force sensation: A sensation that occurs when an object exerts power on a human being or when a human being exerts power on an object. In this article, it is distinguished from tactile sensation.
  3. Tacit knowledge: Subjective knowledge that is difficult to define by words or numbers.
  4. Explicit knowledge: Objective knowledge that can be defined by words and numbers.
  5. Instantaneous value: An instantaneous value at a given point in time.
  6. Effective value: In this article, a numerical value derived from a time-series waveform of the instantaneous value of force/tactile quantity, based on the definition of the effective value used to express an alternating voltage or current. It is also called Root Mean Square (RMS) value because it is defined as the square, mean, and square root of a certain waveform. This expression is suitable for handling force/tactile sensation data that varies greatly with time.
  7. Time window: A portion of a time-series data from which a section has been cut off.
  8. Actuate: To have a physical effect on humans or the environment.

03. The World Achievable by Integration with Cellular Communications

• The integration of RealHaptics and cellular communications ...

  Open

  The integration of RealHaptics and cellular communications is essential to expand the aforementioned IoA world. There are two directions in this IoA world. The following describes a world that relates cellular communications to each direction.

  3.1 Expanding the Area of Remote Control Services with Responses

  The first direction in the world of IoA is to dynamically link machines, people, and operating content through networks and aim to bring people's actions together through remote control. By making use of the cableless and mobility features of the cellular environment in this direction, the following values can be provided to all outdoor areas, areas where the quality of fixed lines varies, such as homes, and special areas where network cables are physically difficult to install.

  1) Providing Specialist Skills Quickly to Any Location

  Although desk work is more often done remotely these days, there are still some tasks that need to be done on-site. Such tasks include home-visit medical care, repairing water leaks, and lifesaving activities. For tasks that involve visiting households, large amounts of time and effort are required. For tasks that require an emergency response by specialists, responses are delayed by the travel time of the specialist. To address such situations, new services are possible that provide professional skills quickly and anywhere by combining precise remote operation and cellular communications to establish systems in which specialists remotely operate robots stationed in homes.

  2) Avoiding Human Damage by Remote Control

  In Japan today, there are many on-site operations that take place in environments that are harsh to humans. For example, workers who support our daily lives are exposed to various risks, such as working near heat sources that may cause burns, working at construction sites that may cause heat stroke or fall hazards, and working at sites contaminated with substances that are harmful to the body. Enabling precise remote control in a range of environments can dramatically reduce the danger to life and health hazards that workers face.

  3.2 Extended Use of Work Data Including Responses

  The second direction in the world of IoA entails handling motion data as content so that appropriate actions are available whenever desired [5]. Combining this direction with Multi-access Edge Computing (MEC)^*12 servers makes it possible to store motion data of remote operations with responses on a server and share it with other users and to provide the following value by extending the use of the content data.

  1) Automatic Control of Robots Based on Human Tasks

  RealHaptics can formalize the human force level and logically define the control rules for force level on the robot side, thereby facilitating the development of automated robots that can perform precise tasks.

  Recently, Autonomous Mobile Robots (AMRs)^*13 have begun to appear on the market, successfully improving the efficiency of human tasks by cooperating with humans in areas such as cleaning work in office buildings and serving food in restaurants. However, not all human tasks can be automated with these robots, and some precision tasks are still performed manually. Looking toward the future, in the transitional stage, it will be necessary to accumulate data on tasks and motions when humans use remote control as teacher data, so that if control AI that mimics human intelligence is established, a future in which all simple tasks can be automated will become a reality.

  2) Reproducing Realistic Responses in Virtual Space

  By constructing a force/tactile sensation model of the response that is fed back when force is applied to an object via remote operation, realistic response can be reproduced in virtual space. Accumulating the data for sensations of touching objects remotely in MEC servers will make it available for various purposes such as creating illustrated books with force/tactile sensation for education, confirming textures when shopping online, and reproducing responses in the Metaverse^*14.

  1. MEC: A system in which servers are installed at locations near users. Standard servers are typically placed on the Internet, but MEC servers are installed within the carrier network to reduce latency, which greatly improves response speeds.
  2. AMR: A robot that can track its own path using sensors and other devices and can run autonomously. Because such robots can autonomously determine obstacles, they excel at rerouting and cooperating with humans.
  3. Metaverse: 3D virtual space built on a computer network and the services using it.

04. Feasibility with Cellular Communications

• 4.1 Issues Raised with 4G Experiments

  Open

  Because remotely operated robots utilizing RealHaptics require constant synchronization of leader and follower movements, it became clear that communications latency has a significant impact on robot control performance. RealHaptics technology can be fully utilized if robots are used in wired network environments within Japan as communications environments that do not affect robot control performance are available.

  NTT DOCOMO's goal is to achieve remote operations that demonstrate the inherent capabilities of RealHaptics in cellular communications and provide responses comparable to that of direct contact. However, experiments via cellular communication have shown that robots cannot be operated properly due to latency fluctuations^*15 that occur in the radio communications section between the base station equipment and the terminal equipment.

  One-to-one voice and video calls are catch-ball communications in which there is no need to exactly synchronize the speaker's and listener's states, and the existing 4th-Generation mobile communications system (4G) meets the acceptable latency criteria for conducting conversations without discomfort. However, force/tactile sensation transmission requires fore/tactile sensation data to be kept constantly synchronized between the leader and follower, so the acceptable latency requirements are strict. If 4G is used for force/tactile sensation transmission, the latency requirements cannot be met, and the robot gradually becomes uncontrollable. Therefore, to achieve remote operation using force/tactile sensation transmission in cellular communications, a radio communications technology that minimizes latency fluctuations occurring in the radio communications section is required.

  4.2 Field Experiment Using 5G URLLC

  NTT DOCOMO has focused on Ultra-Reliable and Low Latency Communications (URLLC)^*16 of 5G, which can suppress latency fluctuations generated in the radio communications section between base station equipment and terminal equipment as much as possible and has conducted a field experiment to verify the feasibility of RealHaptics in cellular communications. In this experiment, parameter settings that satisfy the requirements of URLLC and 5G enhanced Mobile BroadBand (eMBB)^*17 are applied to the experimental equipment equivalent to base station equipment and terminal equipment in the radio communications section, and the operability of the remotely operated robot is compared.

  1) Experiment Overview

  Network equipment was configured via experimental equipment that meets the requirements of URLLC or eMBB and that can remotely control a robot using RealHaptics. This configuration is shown in Figure 2. The packets sent by the leader and follower, respectively, are passed through the radio communications section between the terminal equipment and base station equipment to the application server. The application server controls the forwarding of packets received from the leader to the follower and packets received from the follower to the leader, so that the leader and follower can bidirectionally share force/tactile sensation data. The network emulator^*18 between the application server and the base station equipment adds a fixed latency^*19 to packets as appropriate to simulate the effects of latency caused by the length of the transmission distance.

  2) Evaluation Items

  In this field experiment, the evaluation item is the reproducibility of the force/tactile sensation generated when grasping a sponge, as shown in Figure 3. The reproducibility of the force/tactile sensation quantities is defined by the following equation on the basis of the effective value defined by Equation (2) described above.

  

  Equation (3) is the percentage of error in the position, force, and velocity acting on the leader and follower, respectively, with the closer to 100% indicating the better performance. In other words, it represents the degree of clarity of the response perceived at the hand side via communications, compared to the response perceived at the remote side.

  3) Experimental Results and Discussion

  The reproducibility of quantity of force/tactile sensation between a leader and follower is shown in Figure 4.

  When radio communications using URLLC is applied and no fixed latency is added, the reproducibility of quantity of the force/tactile sensation reaches up to 96.4%. This reproducibility is almost the same as that obtained via cable communications alone, which means that the sensation as close as possible to that of directly grasping a real object is reproduced. Furthermore, when fixed latency of 30 ms or less is added, the reproducibility exceeds 80%. These results suggest that the degree of latency fluctuation in the radio communications section is small, resulting in highly accurate bilateral control between the leader and follower and achieving high transparency. Packet log analysis confirmed that latency fluctuations are significantly suppressed in the case of URLLC compared to the eMBB case described below.

  On the other hand, when radio communications are implemented with eMBB, the reproducibility of quantity of the force/tactile sensation averages only 43%, regardless of the amount of fixed latency that is added. This result confirms that it is difficult to reproduce detailed responses by RealHaptics in high-speed, high-capacity communications technologies because the accuracy of bilateral control is impaired by communications latency and latency fluctuations in the radio communications section. When the added fixed latency was more than 40 ms, the robot became uncontrollable and data acquisition itself was difficult.

  The above results show that URLLC can reproduce the response of directly grasping an object without any real difference, thus increasing the feasibility of radio remote operation utilizing RealHaptics, which is difficult to stabilize with eMBB.

  

  1. Latency fluctuation: Latency caused mainly by scheduling, retransmission processing, etc. in a radio communications section. Divided into fixed and variable latency, the degree of change in variable latency. Also called “jitter.”
  2. URLLC: One of the requirements of 5G, a technology that enables reliable, low-latency radio communications.
  3. eMBB: One of the requirements of 5G, a technology that enables high-speed, high-capacity radio communications.
  4. Network emulator: Equipment that can reproduce various events that occur on a network, such as latency, fluctuations, and packet loss.
  5. Fixed latency: Persistent latency caused primarily by propagation latency when a radio or electric signal passes through a transmission system such as an optical cable.

05. Conclusion

• This article has described RealHaptics, its relation to widely ...

  Open

  This article has described RealHaptics, its relation to widely known haptics technologies, future use cases enabled through integration with cellular communications, and its feasibility in cellular communications. Going forward, we will focus on control functions related to low latency, which is expected to be a requirement for 5G Evolution and 6G and aim for practical application of RealHaptics through a series of feasibility studies in cellular communications. As a first step toward IoA with cellar communications, we will also pursue the uniqueness of RealHaptics and accelerate platform development to provide solutions using cellular communications.

  Finally, we would like to thank Keio University Haptics Research Center for their cooperation in this project and in writing this article.

• REFERENCES

  Open

  1. [1] M. Takahata et al.: “Haptic Communication Technology Using Real-time Robotics,” NTT DOCOMO Technical Journal, Vol. 9, No. 3, pp. 57–61, Dec. 2007.
  2. [2] K. Ohnishi et al.: “Haptics and Its Applications: Transmission, Record, Reproduction, and Display of Haptics Sensation,” CMC Publishing, 2022 (in Japanese).
  3. [3] K. Ohnishi, Y. Saito, S. Fukushima, T. Matsunaga and T. Nozaki: “Future Society Opened by Real Haptics,” Journal of the Japan Society of Applied Electromagnetics, Vol. 25, No. 1, pp. 9–16, 2017 (in Japanese).
  4. [4] A. Suzuki and K. Ohnishi: “Reproducibility and Operationality in Bilateral Teleoperation under Time Delay,” IEEJ Transactions on Industrial Applications, Vol. 133, No. 7, pp. 738–746, 2013 (in Japanese).
  5. [5] T. Nozaki and S. Hangai: “Telexistence and Real Haptics—Fusion of Humans and Machines Opens a New Avenue—,” Journal of the Robotics Society of Japan, Vol. 36, No. 10, pp. 668–672, 2018 (in Japanese).

VOL.25 NO.3