Transformer "Transformers" changed computer vision

Transform

Produced by Zhineng Zhixin

Transformer is an NLP classic model proposed by Google's team in 2017, the Transformer model uses the Self-Attention mechanism, does not use RNN sequential structure, so that the model can be parallelized and trained, and can have global information.

Over the past few years, Transformers have revolutionized the nature of deep learning models, revolutionizing the field of artificial intelligence. Transformers, which introduce attention mechanisms, allow models to weigh the importance of different elements when processing input sequences, and unlike traditional deep learning models that process data sequentially or hierarchically, Transformers can capture dependencies between elements in parallel, making larger scale model training possible.

Structure and application of Transformer

Although originally designed for natural language processing (NLP), Transformer is beginning to find applications in many different fields, one of which is computer vision.

Traditionally, computer vision has relied on convolutional neural networks (CNNs) as deep learning architectures, but with the growth of datasets and powerful GPU support, the rise of deep learning has changed the field. However, researchers are beginning to realize that Transformer can also be used to process image data, making it a promising option for computer vision applications.

Reasons for choosing Transformer

Computer vision tasks such as image classification, object detection, and image segmentation have traditionally relied on CNNs. However, Transformer excels at capturing remote dependencies and global contextual information in images, which is essential for handling complex visual tasks. Unlike CNNs, Transformers process all elements in parallel, eliminating the need for processing order, which speeds up the time for training and inference, making large-scale vision models more feasible.

The multimodality of the Transformer makes it suitable for tasks that require understanding and reasoning about visual and textual information, and the resulting attention map provides insight into which parts of the input are more important when making predictions, increasing the interpretability of the model.

Application of Transformer in computer vision

In object detection tasks, models such as DETR (DEtection TRansformer) excel in processing a variable number of objects in an image without the need for anchor frames, which is a major breakthrough. In semantic and instance segmentation tasks, models such as Swin Transformer and Vision Transformer provide improved spatial understanding and feature extraction. In addition, Transformer-based models, such as DALL-E, can generate highly creative and context-aware images from textual descriptions, opening up new opportunities for content generation and creative applications. On top of that, Transformer can generate descriptive titles for images.

Combine Transformer and CNN

While Transformer has advantages in recognizing complex objects, the performance advantage of CNNs in inference time cannot be ignored. Therefore, in vision processing applications, it is possible to leverage both Transformers and CNNs, taking full advantage of both, which is a growing area of research. Synopsys ARC® NPX6 NPU IP is an example that can handle CNNs and transformers, leveraging convolution accelerators and tensor accelerators to deliver superior performance and power efficiency.

● Conclusion

The rise of Transformer marks a major revolution in computer vision. Its unique features, such as attention mechanisms, parallel processing, and scalability, challenge the dominance of CNNs and bring exciting possibilities to computer vision applications. As the Transformer model continues to be refined in vision tasks, we can expect more breakthroughs that will lead to smarter, more powerful vision systems with a wider range of practical applications.