Development of AI Optical Sensor-Neuromorphic Electronic Device-Based Electronic Skin Capable of Real-Time Finger Motion and Gesture Recognition in 3D Free Space for Metaverse

The research results from Professor Wang Gunuk’s group were published in Nature Electronics, and also introduced in Nature Electronics News and Views.

▲ (From left) Cho Haein (KU, first author), Lee Inho (Ajou University, co-first author), Jang Jingon (research professor at KU, co-first author), Park Sungjun (professor at Ajou University, co-corresponding author), and Wang Gunuk (professor at KU, co-corresponding author).

Professor Wang Gunuk’s group of the KU-KIST Graduate School of Converging Science and Technology and Department of Integrative Energy Engineering and Professor Park Sungjun’s group of the Department of Electrical and Computer Engineering (main affiliation) and Department of Intelligence Semiconductor Engineering (dual affiliation) at Ajou University conducted joint research. This research successfully developed both a thin film-type organic optical sensor device that is attachable to micro-wrinkles of the skin and a neuromorphic artificial synapse device on top of an ultrathin substrate one micrometer thick (1/100 the thickness of a hair), and then combined them to fabricate an electric skin capable of recognizing finger motion.

The combination of IoT, advanced sensing, and AI technologies has allowed the real-time acquisition, recognition, and interpretation of people’s natural motions. Various commands may be carried out initiated by simple human motions. These technologies are drawing much attention because they can implement virtual realities, including metaverses, and be extended to other fields, such as diagnoses based on biological signals acquired on a real-time basis.

In particular, finger motion has the highest degree of freedom of expression among all body motions and may be used to deliver intuitive nonverbal expressions. Therefore, efforts have been made to recognize and interpret figure motions. However, the existing methods have limitations, particularly that a large and fixed sensing device should be used, the system may be constrained spatially, and the algorithm for signal recognition and processing is complicated and consumes much time and energy. These limitations make daily application difficult.

The motion recognition platform developed in this research in the form of a skin-conformable electric device was fabricated by coupling an ultrathin artificial synapse array device to a high-efficiency organic photodiode that specialized in low-power and high-efficiency signal processing. The platform is capable of converting finger motions from optical signals into electronic signals and exhibiting high accuracy recognition capabilities by learning signal patterns (Figure 1). Furthermore, a substrate one micrometer thick was used to minimize the stiffness of the device so that it could remain stable under repeated mechanical deformation, tightly attach to the skin surface, and allow for natural motion.

▲ Figure 1. (a) A schematic diagram showing the recognition process in 3D space with light, by the finger motion recognition platform using an organic optical sensor-artificial synapse device; (b) An actual photograph of the ultrathin artificial synapse array device attached to the skin surface of a finger; and (c) LTP/LTD characteristics of the ultrathin artificial synapse array device attached to a finger model.


The skin-conformable short-range optical sensor using the ultrathin organic photodiode is stably settled onto the finger skin interface, exhibiting distance-dependent photo-detecting performance at a high sensitivity of 90 mV/mm under the green light in the X-axis and red light in the Y-axis. In a two-dimensional light-receiving environment, the sensor can accurately detect time-dependent motion changes, such as digit patterns, alphabets, and simple gestures according to free finger motions following a Eulerian trail (single line drawing) (Figure 2).


▲ Figure 2. (a) A schematic diagram of the experiment determining the change of output voltage depending on the distance between the light source and the organic photodiode; (b) The output voltage change characteristics depending on the distance change under both red and green lighting; (c) An actual photograph of the ultrathin organic photo-detecting sensor attached to a finger skin surface; (d) The output voltage change characteristics in the X-axis and Y-axis for the finger-writing motion for digit 3; and (e) The 3D trajectory of the finger motion.

The ultrathin neuromorphic device may be tightly attached to a skin surface and is capable of processing 6,400 repeated input signals even under mechanical deformation of up to 60%. Furthermore, the device exhibited durability even after 1,200 bending cycles. The maximum recognition accuracy of the platform being up to 95% confirmed that the approach is commercially applicable. The authors also confirmed that the platform can recognize not only the finger motions of a single person but also the finger motions of different people under various attachment environments (Figure 3).

▲ Figure 3. (a) A schematic diagram of the experiment on the strain durability of the ultrathin artificial synapse array device; (b) An actual photograph of the ultrathin artificial synapse array device compressed from 0 to 60%; (c) Results of the operational durability test of the ultrathin artificial synapse array device obtained by performing 6,400 cycles of input signals at a 60% deformation rate; (d) Drawing of digit 3 acquired by the ultrathin organic photodiode attached to a finger moving in free space over time and the corresponding sequential image reshaping; (e) Distribution of weights (w) for finger-written digits from 0 to 9 after 10 learning epochs; (f) Finger writing motion of digit 3 in a 3D free space; and (g) Recognition accuracy according to the number of repeated cycles of the ultrathin artificial synapse array depending on the deformation rate.


This paper was authored by Cho Haein, a student in the integrated master-doctoral degree program of the KU-KIST Graduate School of Converging Science and Technology; Lee Inho, a student in the integrated master-doctoral degree program of the Department of Intelligence Semiconductor Engineering at Ajou University; and Doctor Jang Jingon at KU as co-first authors; and Park Sungjun of the Department of Electrical and Computer Engineering and Department of Intelligence Semiconductor Engineering at Ajou University; and Professor Wang Gunuk of the KU-KIST Graduate School of Converging Science and Technology as co-corresponding authors.

The research was supported by the National Research Foundation of Korea, the Market-Leading K-Sensor Technology Development Program of the Korea Evaluation Institute of Industrial Technology, the Information Technology Research Center Program, the Mid-career Researcher Program, the Nanomaterials Source Technology Development Program, the International Research Exchange Program, the Stage 4 BK 21 Program, and the Creative and Adventurous Research Program of the Ministry of Science, ICT and Future Planning, and the Commissioned Research by the Korea Institute of Science and Technology. The results were published in a globally renowned journal in the field of flexible electronics, Nature Electronics (IF=34.3 and IF%=0.362 as of 2022), on August 10.
* Title of article : Real-time finger motion recognition using skin-conformable electronics