📋 Tender Overview
iFLYTEK has led technological breakthroughs in speech recognition speech synthesis machine translation and other areas. According to the keynote speech of iFLYTEK Research Institute scientist Pan Jia at NCMMSC 2022 iFLYTEK is poised to make additional breakthroughs in four key areas of research self-supervised pre-training speech recognition speech synthesis and sound event detection to embrace the future.As the first of our new series on iFLYTEKs research progress this article focuses on self-supervised pre-training technology.iFLYTEK has made significant efforts in incorporating self-supervised pre-training in speech processing by improving the use of information in clustered labels. Clustering is followed by four downstream tasks phoneme classification speaker classification fundamental frequency F0 and energy prediction.Based on iFLYTEKs analysis that showed the current clustering process is not fully interpretable the companys research institute explored whether clustered tags can carry the contextual semantic features related to phonemes while excluding some prosodic information and speaker information.iFLYTEK Research Institute proposes the SEAU pattern approach. This process uses labeled data from the downstream ASR to randomly initialize the training of the speech recognition model. An encoder model within the speech recognition model is then used to extract the pre-training data or in other words the representation of unlabeled data and clustering. The clustering result is then used as the self-supervised label for self-supervised pre-training. The above steps can be repeated and iterated.The approach has been successful in tests of downstream multilingual speech recognition tasks in languages such as Turkmen Portuguese and French which are prone to data labeling difficulty or missing.For the ASR task iFLYTEK Research Institute is working on Progressive Multi-Scale PMS capabilities with the hope of achieving a multi-scale or gradual self-supervised learning process.Improving this approach requires that the self-supervised learning task is set in the shallow layer of the transformer and that the number of tag categories gradually increases. The receptive field of the shallow model is small and it gradually increases with the depth of layers enabling the self-supervised model to learn more coarse-grained phoneme information and reduce the over-fitting risk of the self-supervised model.In targeting the recognition task of multi-modalityboth vision and audiothe iFLYTEK Research Institute recommends the utilization AV2vec approach.This approach combines the loss function of AV-HuBERT and data2vec and uses the supervised model in previous speech recognition to extract and cluster features as Mask prediction labels. In addition the visual ROI region of interest in video-enabled speech recognition tools is extended to analyze the whole head instead of just the lips including the neck or throat since movements in the throat area may also represent information related to phonemes.iFLYTEK Speech and Multimodal Pre-training Model CommissioningiFLYTEK has issued a demo based on multimodal pre-training which is functionally effective in challenging auditory scenarios such as subway ticket sales in crowded stations. Multi-modal correlation technology is needed to enable recognition of human speech interference coming from the same direction as a customer.The speech and multimodal pre-training models released by iFLYTEK Research Institute have achieved SOTA scores on leading open-source data such as AISHELL2 Voxceleb LRS3 DFEW and MISP.The two pre-training models have provided commissioning interfaces on the iFLYTEK open platform covering different fields and supporting multiple tasks. Although the quantity of parameters is relatively small compared with public models in the industry the performance is outstanding providing a brand-new alternative for industrial application scenarios of pre-training models.
