METHOD AND DEVICE FOR GENERATING VIDEO CLIP FROM TEXT DESCRIPTION AND SEQUENCE OF KEY POINTS SYNTHESIZED BY DIFFUSION MODEL Russian patent published in 2024 - IPC G06T13/00 G06F40/00 G06N20/00 

FIELD: physics.

SUBSTANCE: invention relates to the field of machine learning models implementing the synthesis of video clips based on text descriptions. A method of generating a video clip from a text description includes steps of: receiving a text description of the video clip to be generated; obtaining a vector representation e of the text description according to the vector space of the pre-trained neural network model of linking images and text descriptions based on the received text description of the video clip; vector representation 2×N×L sequence of key points of the generated video clip, which is synthesized by the trained diffusion motion model based on the vector representation e of the text description, where 2 is the number of coordinates, the first coordinate indicates the height H of the frame, and the second coordinate indicates the width W of the frame, N is the number of key points on each frame, and L is the number of frames in the video; displaying a vector representation of a sequence of key points of the generated video clip into a row of L two-dimensional images of key points of the generated video clip, wherein each two-dimensional image of key points from said row corresponds to a corresponding frame of the generated video clip, and generating a sequence of video clip frames using a pretrained stable diffusion model, wherein generation of each frame of the video clip by the stable diffusion model is additionally controlled by the controlling neural network model based on the two-dimensional image of key points of the corresponding frame from said row.

EFFECT: possibility of synthesizing a high-quality video clip from a text description without using reference video clips.

18 cl, 9 dwg