What is an artificial intelligence "black box"?

A computer scientist explains what it means when the inner workings of ChatGPT and other artificial intelligence are hidden.

For some, the term "black box" conjures up images of recording devices on airplanes that are valuable for postmortem analysis if the unthinkable happens. And for others, it will be reminiscent of a small theater with minimal equipment. However, black box is also an important term in the field of artificial intelligence.

Artificial intelligence (AI) black boxes refer to AI systems whose inner workings are invisible to the user. You can provide them with inputs and get outputs, but you can't examine the system code or logic that produces the outputs.

Machine learning is the main branch of artificial intelligence. It is the basis for generative AI systems such as ChatGPT and the Dell-e2. Machine learning has three components: an algorithm or set of algorithms, training data, and a model. An algorithm is a set of programs in machine learning in which an algorithm learns to recognize patterns after being trained on a large number of examples (training data). Once a machine learning algorithm is trained, the result is a machine learning model. Models are what people use.

For example, machine learning algorithms can be designed to recognize patterns in images, and the training data can be images of dogs. The resulting machine learning model will be a "dog observer." You can give it an image as input and get if and where a set of pixels in the image represents a dog as output.

Any of the three components of a machine learning system can be hidden or placed in a black box. Usually, the algorithm is public, which makes it less efficient to put it in a black box. Therefore, in order to protect their intellectual property, AI developers often put models in a black box. Another approach employed by software developers is to obfuscate the data used to train the model — in other words, to put the training data in a black box.

Black-box algorithms make it difficult to understand how AI works, but the situation is not black and white.

The reverse side of the black box is sometimes referred to as a glass box. An AI glass box is a system in which algorithms, training data, and models are all available for anyone to see. But researchers also sometimes characterize some aspects of them as black boxes.

That's because researchers don't fully understand how machine learning algorithms, especially deep learning algorithms, work. The explainable field of AI is working to develop algorithms that, while not necessarily glass boxes, can be better understood by humans.

Why AI black boxes are important

In many cases, there are good reasons to be wary of black-box machine learning algorithms and models. Let's say a machine learning model has made a diagnosis of your health. Do you want this model to be a black box or a glass box? How about the treatment prescribed by the doctor? Maybe she wondered how the model made its decisions.

What if a machine learning model that decides whether you're eligible for a business loan from a bank rejects you? Don't you want to know why? If you do, you can appeal the decision more effectively, or change your circumstances to increase your chances of getting a loan next time.

Black boxes also have important implications for software system security. For years, many in the computer field believed that keeping software in a black box would prevent hackers from checking it, so it would be safe. This assumption has largely proven wrong, as hackers can reverse engineer software — that is, build a copy by closely watching how it works — and discover exploitable vulnerabilities.

If the software is in a glass box, software testers and well-meaning hackers can check it and inform the creators of the weaknesses, thus minimizing cyberattacks.

Want to know more about the future of artificial intelligence, chatbots and machine learning? Feel free to let us know in the comments!

Artificial Intelligence (Kai-Fu Lee on how to reshape the future map of individuals, business and society) Kai-Fu Lee ¥27.5 Purchase

If friends like it, stay tuned to "Know the New"!