FIELD: data processing.
SUBSTANCE: invention relates to the field of artificial intelligence. Invention comprises a method for multimodal noncontact control of a mobile information robot, consisting of capturing three-dimensional space, gesture recognition, characterized by that combined processing of video and audio information from a user, serial processing of video information from user, formation of areas with people on each three-dimensional (3D) frame of depth map, calculation of 3D 25-point models of human skeletons, tracking the nearest person, transformation of 3D 25-point skeleton model of the nearest person into 2D 25-point skeleton model, determining a face and hand shape graphic region within the formed rectangular area with a human, computation of visual signs of gesture at a certain moment of time, serial processing of audio information from user, calculation of acoustic features, recognition of speech commands.
EFFECT: technical result consists in expansion of functionality of mobile information robot due to use of multimode human-machine interaction, which is understood as control of mobile information robot through combination of audio and video modalities.
1 cl, 1 dwg
| Title | Year | Author | Number |
|---|---|---|---|
| METHOD FOR AUDIOVISUAL RECOGNITION OF PERSONAL PROTECTION EQUIPMENT ON HUMAN FACE | 2022 |
|
RU2791415C1 |
| UNIVERSAL OPERATOR INTELLIGENT 3-D INTERFACE | 2017 |
|
RU2693197C2 |
| METHOD OF DISPLAYING THREE-DIMENSIONAL FACE OF THE OBJECT AND DEVICE FOR IT | 2017 |
|
RU2671990C1 |
| METHOD OF TASKS DISTRIBUTION BETWEEN SERVICING ROBOTS AND MEANS OF CYBERPHYSICAL INTELLIGENT SPACE WITH MULTIMODAL USER SERVICE | 2016 |
|
RU2638003C1 |
| METHOD OF TRANSMITTING MULTIMODAL INFORMATION ON MISSION-CRITICAL OBJECTS | 2018 |
|
RU2696221C1 |
| METHOD, TERMINAL AND SYSTEM FOR BIOMETRIC IDENTIFICATION | 2023 |
|
RU2815689C1 |
| METHOD OF CONTROLLING DEVICE USING GESTURES AND 3D SENSOR FOR REALISING SAID METHOD | 2011 |
|
RU2455676C2 |
| SYSTEM FOR RECOGNITION AND TRACKING OF FINGERS | 2012 |
|
RU2605370C2 |
| METHOD, TERMINAL AND SYSTEM FOR BIOMETRIC IDENTIFICATION | 2022 |
|
RU2798179C1 |
| METHOD OF RECOGNIZING SPEECH EMOTIONS USING 3D CONVOLUTIONAL NEURAL NETWORK | 2023 |
|
RU2816680C1 |
