NEURAL NETWORKS WITH ATTENTION-BASED SEQUENCE TRANSFORMATION Russian patent published in 2021 - IPC G06N3/04 G06N3/08 

FIELD: computer technology.

SUBSTANCE: invention relates to a system, a data carrier, and a method for converting a sequence based on attention. The method receives an input sequence having a respective input at each of multiple input positions in the order of input; the input sequence is processed by an encoder neural network to generate a corresponding encoded representation of each of the inputs in the input sequence, with the encoder neural network containing a sequence of one or more encoder subnets, while each encoder subnet is designed to receive a respective encoder subnet input for each of the multiple input positions and generate a corresponding subnet output for each of the multiple input positions, with each encoder subnet comprising an encoder self-attention sublayer that is designed to receive a subnet input for each from multiple input positions, and for each particular input position in the order of input, apply a self-attention mechanism to the encoder subnet inputs at multiple input positions to generate the appropriate output for that particular input position, at the same time, the application of the self-attention mechanism comprises the steps at which: the request from the subnet input is determined at the specified specific input position, the keys are determined that are obtained from the subnet inputs at the multiple input positions, the values obtained from the subnet inputs at the multiple input positions are determined, and the said determined request, keys and values are used for generating the said respective output for a specific input position; and the encoded representations are processed by the decoder neural network to generate an output sequence having a respective output at each of the multiple output positions in the output order.

EFFECT: improving the speed of sequence conversion.

30 cl, 3 dwg