Before reading this article, please click "Follow", which is convenient for you to join the discussion and sharing, and can bring you a different reading experience, thank you for your support
Wen | Mo Qingyan
Editor|Mo Qingyan
preface
In today's world, due to the scarcity of digital automation systems, there are many industries where engineering documents are still in paper format, and in these documents, the understanding and interpretation of single-line diagrams have become the most painful problem for people.
However, these technical documents play a vital role in many areas such as electrical systems and distribution systems, so many highly skilled engineers and professionals have to rely on a lot of time to decipher these drawings.
The digitization of engineering drawings has become increasingly important in recent years, and the development of equipment monitoring, risk analysis, safety inspections and other operations all rely on this technology, and they are also influenced by computer vision and image understanding.
With the development of convolutional neural networks, they are widely used in a variety of image-related tasks, including biometric-based authentication, image classification, handwriting recognition, and object recognition.
Before the advent of volume neural networks, the technology of segmentation, classification, and object recognition of images could almost be said to be in place, however, the introduction of volume neural networks completely changed this field.
And in order to solve the problem of digitization of engineering drawings, a foreign scientist first proposed GANs in 2014 to solve the problem of imbalance of data sets, several types of symbols in drawings, and overrepresentation or insufficient service in data sets, is the second problem people have to face.
The YOLO V5 model for object recognition, a model with a special focus on single-line symbols, is a big help for scientists to overcome the second puzzle, so how does it work?
Related works
The scientific community has recently made new breakthroughs in this field, which has led to new developments in different deep learning techniques for digitizing engineering drawings, and GAN is a new product of scientists.
Digitization of engineering documentation
Engineering drawings are widely used in multiple disciplines, often containing a wide variety of symbols, solid or dashed lines, and text to depict complex engineering processes in a condensed and comprehensive manner.
In order to digitize these sketches, people began to pour more effort into machine vision, and with the significant advances in computer vision and machine learning, coupled with a lot of data that has not yet been digitized, a fully automated framework that can digitize drawings has become the most urgent need.
Relying heavily on extensive feature extraction is a major limitation of the learning method, which is highly dependent on the quality of the extracted features, and the existing literature in the field focuses primarily on addressing specific aspects of digital drawings rather than providing a comprehensive and fully automated framework.
Some studies focus on the recognition and classification of common symbols in engineering drawings, as well as the separation of chart Chinese from other graphic elements, and the use of image processing technology for line recognition and deep learning methods for symbol detection.
Or heuristics can be used in other studies to locate and classify components in drawings using specific methods to achieve a high level of accuracy.
However, when the schematic or symbolic representation changes, these methods may require the modification of heuristic rules, or the development of new heuristic rules, and the effectiveness of these methods depends largely on the balanced distribution of data in the dataset.
In recent years, there have been attempts to apply deep learning-based techniques to tasks similar to the digitization of engineering drawings, and some studies have used single-stage inspection-based techniques to identify door, window and furniture objects in floor plans and have produced good results.
But these studies used small data sets and an insufficient number of furniture items per drawing, which also led to a decrease in performance.
Symbol recognition in process and meter diagrams is a very important area, and the complexity of meter diagrams brings various challenges to symbol recognition, including adjacent lines, overlapping symbols, ambiguous regions, and similarity between symbols.
One study evaluated four classification tools using synthetic and raw plot table datasets, among which the model that showed the best performance also had significant drawbacks, relying on grouping symbols before classification and strikethrough for clearer observations.
Combining heuristics and deep learning techniques, component inspection of engineering drawings holds promise of breaking through barriers, using a two-stage process involving Euclidean metrics to connect pipe labels and symbols, and probabilistic Hough transforms for pipe inspection to localize symbols and text.
Another approach uses a fully linked convolutional neural network to develop symbolic localization and techniques, with a dataset of 672 process flow diagrams used for automated drawings, which improves performance compared to traditional methods, but does not enable accurate detection of all components.
To capture the time-varying signal generated by pen movement during a single-line freehand circuit diagram sketch, the scientists used the hidden Markov model, for which a dataset containing 100 hand-drawn sketches was examined, and it achieved a high accuracy in correctly classifying points associated with connecting lines and symbol categories.
Later, scientists developed a new circuit diagram recognition system that solved sketch recognition as a dynamic programming problem, combined with a new technique that successfully identified symbols interspersed in sketches, proving that the method was effective in identifying free-form sketches.
Advanced deep learning methods have been pioneered in music, and have achieved significant improvements, and a number of techniques have been successfully applied to recognize handwritten musical symbols, with higher performance than traditional symbol recognition and detection structured image processing methods.
In conclusion, existing research highlights the significant gap between the current state of machine learning and technical image understanding, which stems from the rapid development of the field, as well as the imbalance and incremental progress in key application areas.
Vast web
First proposed in 2014 by an American scientist, generative adversarial networks are considered generative models that can create unique and fresh content, and generators and discriminators are the two competing models that make up GAN.
The discriminator is used as a classifier that receives input from the generator and the training set, and it will learn how to distinguish between real and fake input samples during training, however, the generator is taught how to create samples that accurately reflect the fundamental properties of the original content.
So how do generators and discriminators work?
The generator is actually a network that uses the current data to generate new realistic pictures, using random noise to create images, the goal of the generator is to trick the discriminator into believing that the false image it produces is real, and when the sample generated by the generator is discriminated, it will try to reduce the accuracy of the discriminator as much as possible.
Unlike the generator, the discriminator is a network used to distinguish between images, it determines whether the input image is a fake image created by the generator or a real image that already exists, and the discriminator's job is to highlight the difference between the actual image that already exists, and the fake image generated by the generator.
Minimizing the difference between the distribution of real data and the distribution of artificial data is the main goal of this model training, the discriminator seeks to optimize accuracy when distinguishing real samples, and it seeks to maximize when separating fake samples from real samples.
The generator in this model is responsible for generating fictitious pictures, in order to determine which picture is real and which is fake, the discriminator uses fake images created by the generator or real images that already exist, and the generator and discriminator will develop in a confrontational way after thousands of iterations.
Recommended method
The scientists proposed a method for symbol recognition in line diagrams, and provided information about the datasets used for testing, including data exploration and preprocessing techniques, and proposed solutions to solve the class imbalance problem in these drawings.
SLD symbol recognition method
The experiment is divided into two parts: 1. Generate synthetic images through two sub-models; 2. Detect symbols in original and enhanced images.
These diagrams can be found in paper documents or scanned images in many departments, interpreting and analyzing these documents requires a lot of time, effort and expertise, and an accurate understanding of these drawings is critical, and misreading these papers can lead to very serious consequences.
As part of the data preparation process, scientists use the model to create pictures of artificial symbols, and the dataset is also divided into two categories, with the first set containing only the actual images.
The second set of datasets consists of the actual images created by two sub-models and the generated images, the first set includes only the original images, and the other set combines the original images with the composite images generated by the model.
SLD data analysis
In the original data, the size of the data table is 7000×5000 pixels, and the scientists divided the worksheet into 6*4 grids to speed up training, creating 24 subimage blocks, which are relatively small compared to the original image.
In order to train a deep learning model, images and schematics must be fully annotated, so scientists use a tool to annotate this image set, the process of annotating diagrams is very simple, just use this tool to record the class of the relevant symbol and its location.
The data produced by the annotation is saved in a file representing multiple unique classes, nearly two thousand multiple samples of different symbols, but such raw samples are very unbalanced.
The scientists who studied this said that in some cases, the differences between the symbols can be very large, but the symbols were excluded from the original dataset due to the unusually underrepresented appearance.
Data generation for GANs
To ensure that the model's volume neural network, backbone network can be used normally, someone provides a set of requirements to replace the pooling layer on the discriminator and generator with strangling convolution using its original settings.
Volar neural networks are often used to identify features, and secondly, in order to solve the problem of gradient vanishing, scientists use a batch normalization method, and a gradient propagator is built into each layer to ensure that the gradient reaches each layer.
While preventing the generator model from collecting all instances at the corresponding point, a major problem in this case is that various neural networks use various activation functions for activation and Adam optimization, and the results show that the model is more efficient and is generally considered the gold standard.
In this study, scientists use the model to create a composite picture of a single-line graph, and then combine the composite image with an image generated by another model and an actual image to augment the dataset and improve the symbol recognition algorithm.
It seems that solving the problem of data-based engineering drawings is just around the corner.
conclusion
The main objective of this study is to compare the quality of the composite images generated by the two models, which combines actual SLD images with composite images, with various types and quantities of images used for training purposes.
The model was able to successfully detect symbols from several different classes, and although some components differed slightly, these results demonstrated the accuracy of the detection technique in challenging tasks.
Based on the scientists' research showing that the use of a mix of real and synthetic images during the training process can enhance the ability to recognize symbols, the scientists found that the YOLO model is promising to solve the difficulty of digitizing engineering drawings.
After reading this article, please follow + comment, so that the remaining subsequent events have new developments that can be quickly followed