Reports from the Heart of the Machine
Machine Heart Editorial Department
Isn't that a data breach?
MNIST recognition is already 100% accurate? Recently, a paper in the preprint platform arXiv, "Learning with Signatures", has attracted attention.
In this work, the authors looked at the use of Signature Transform in a learning environment. The paper proposes a supervisory framework that provides state-of-the-art classification accuracy with very few labels, no credit assignment, and little overfitting. The authors take advantage of harmonic analysis tools by using Signature and log signature and use it as scoring functions RMSE and MAE Signature and log signature.
The researchers used a closed equation to calculate the best possible scale factor. The final implementation of the classification results executes on the CPU several orders of magnitude faster than other methods. The authors report results from the AFHQ dataset, Four Shapes, MNIST, and CIFAR10, achieving 100% accuracy on all tasks.
MNIST is considered to be the Hello World of machine learning, which is a dataset that everyone will use when getting started, which contains 70,000 handwritten digital images, of which 60,000 are for training and 10,000 are for testing. Images in MNIST are grayscale and have a resolution of only 28×28 pixels. Despite the "simplicity" of the problem, algorithms that achieve 100% recognition accuracy always feel unreliable, so let's see what the paper says.
Thesis link: https://arxiv.org/abs/2204.07953v1
Code: https://github.com/decurtoydiaz/learning_with_signatures
With Signature, a small number of labeled samples rival the benefits of deep learning
In the last century, making computers capable of learning has been an important research direction. In recent years, supervised and unsupervised learning using deep learning has become a proxy for SOTA solutions. The dominant field of model-based solutions has rapidly transformed into a data-driven framework with unprecedented success. However, due to the large number of hyperparameters of such models, which are difficult to interpret, and their robustness lacks convergence theory guarantees, progress in some areas has stagnated.
Good progress has been made in integrating Signature Transform into learning frameworks in recent years, primarily as a feature extractor in the ML paradigm or as a pooling layer within a deep network. Because of signature's good theoretical nature, many researchers use it as a way to construct learning problems. However, a common framework for signatures has not yet been established, largely because there is no properly defined scoring function to guide the learning mechanism.
Recently, researchers have proposed to study a new type of learning mechanism by using Signature Transform, a recently developed harmonic analysis tool that provides a compact and rich description of irregularly sampled data streams. The researchers explored the idea that by converting the data into a compact and complete domain, the study could reap the same empirical benefits as deep learning by using very few labeled samples.
In addition, signature's general nonlinear properties, unaffected by temporal reparameterization, make it a better candidate for alternative representation of computer inference knowledge. After all, humans don't need thousands of examples to learn simple concepts, but only a handful of carefully selected examples to guess quickly and correctly. The advent of Signatures achieves this goal, and computers can quickly infer information because its representation is easy to understand, rich, and complete. But this also requires a scoring function, just as loss and credit allocation to traditional learning frameworks provide the ability to direct optimization to a potentially good solution.
The study used RMSE, MAE Signature, and log-signature to assess visual similarity between image distributions to determine GAN convergence. From another perspective, RMSE, MAE Signature, and logarithmic signature are indeed correctly defined scoring functions that can be used for tasks such as classification, clustering, and so on. Under this assumption, this study aims to further investigate the behavior, nature, and generalization ability of this learning framework on a number of tasks.
In the case of Signature, the study recommends the use of signature Transform-based similarity measures. The framework works on the CPU orders of magnitude faster than deep learning methods, and avoids the tedious credit allocation of millions of hyperparameters done on the GPU at high compute and environmental costs. These metrics capture detailed visual cues and can be used for classification tasks with a very small memory footprint, fast execution, and high accuracy.
Regarding credit allocation, such as backpropagation, it has always been the basis of modern automated learning techniques; it is theoretically feasible to extract all subsequent information from data through only one pass (i.e., using an epoch). But in fact, due to the limitations of the learning mechanism, many methods use more than one training data transfer, and these characteristics cannot be attributed to their advantages. Given an appropriate scoring function, signatures provide a compact representation that computers can use to infer fine-grained information without using backpropagation, thus avoiding optimizing millions of hyperparameters.
Learning with signatures has a computational advantage over other training mechanisms, as the number of labeled examples can be greatly reduced and the training is replaced by the element-by-element mean, which gives good generalization the statistical robustness required.
Given a set of signature sequences defined as the element mean: RMSE and MAE signatures can be defined as:
Use signature for few-shot classification
The authors argue that a Few-shot sample classification can be achieved by using the signature and defined fractional functions against the test samples (after optional enhancement and calculating element averages). The number of Signatures required to achieve extremely high classification accuracy may depend on the complexity of the task, some categories may require only one, and categories with more variability may require tens of thousands to thousands of training samples.
To further investigate the effects of the multiplicity introduced by averaging multiple transform versions of the same test instance, the authors demonstrated the visualization using specific augmentation techniques, such as random comparison.
Figure 1: PCA adaptive t-SNE with signatures on 300 images of AFHQ, categories: cat (red), dog (green), and wild (blue).
Figure 2: Comparison of the eigenvalue spectrum of a given AFQH sample and its corresponding transformation with a random contrast (a)-(d).
Often, one would think that achieving 100% accuracy on a training set must be a data breach. For the study, there were more doubts in social networks.
On reddit, a netizen said: "There are several figures in the MNIST dataset that classify humans differently from labels. A test set accuracy of 100% indicates that the network is actually worse than those with an error rate of 99.7%. So as others have said, the 100% accuracy figure is very dubious."
For reference:
https://www.reddit.com/r/MachineLearning/comments/u7ouxh/r_authors_claim_to_have_solved_mnist_and_cifar/