As the new generation of artificial intelligence technology represented by machine learning continues to develop in a more advanced, complex and autonomous direction, our economic and social development has ushered in transformative opportunities.
But at the same time, the transparency and explainability of AI algorithms have also brought unprecedented challenges to many fields such as public trust and public safety.
From January 11 to 14, the "Tencent Technology for Good Innovation Week" jointly sponsored by Tencent Research Institute and Tencent Sustainable Social Value Division (SSV) was held online. "Transparent Explainable AI – The Concept and Practice of Opening a Black Box" is the first thematic forum of this year's Innovation Week, focusing on the explanatory and transparent issues of artificial intelligence.
The forum was presided over by Yang Jian, Vice President of Tencent Group and General Counsel of Tencent Research Institute. At the meeting, Zhang Qinkun, Secretary General of Tencent Research Institute, and Ding Shouhong, head of face recognition technology at Youtu Lab, first jointly released the "Interpretable AI Development Report 2022", followed by Yang Qiang, academician of the Royal Canadian Academy of Sciences & Canadian Academy of Engineering, Yao Xin, director of the Department of Computer Science and Engineering of Southern University of Science and Technology, Zhu Jing, dean of the School of Humanities of Xiamen University, Wu Baoyuan, consultant of Tencent AI Lab, He Fengxiang, algorithm scientist of Jingdong Exploration Research Institute, and Zheng Yefeng, head of Tianyan Laboratory. Participated in a roundtable discussion on explainable AI.
The following is a compilation of articles from the roundtable discussion session:
Explainable conceptual consensus of AI
Yao Xin:
When discussing the transparency and explainability of AI algorithms, we should first consider three W's problems – Who, What, and Who.
First of all, who is transparent and explainable to whom? Because from the perspective of scientific research, any research must be transparent and explainable, otherwise this paper cannot be published. So I guess that in the past, transparency and explainability may not be explainable or transparent for scientists, because transparency and explainability to scientists are not necessarily transparent and explainable to the public. The second is to explain what? To explain the results of the model is to explain how the model works. Third, explanations always have a purpose, either to hold accountable or to understand the scientific principles of the model.
Based on the different answers to these three W's, there will be very different transparency and interpretability, and the corresponding solutions may be completely different. In any case, when considering transparency and explainability, we must first have a conceptual consensus, so that we know that we are talking about the same thing, rather than using the same term, and everyone is talking about different problems at different levels of abstraction.
Wu Baoyuan:
Explainable is an important part of trusted AI and one of the prerequisites for trustworthiness, but compared to trustworthy features such as robustness and fairness, I think explainable is not a concept that exists independently. That's what Teacher Yao just mentioned, what are we explaining? Other features have their own clear mathematical definitions, such as robustness, fairness, etc., but explainability is not, because when we mention it separately, the default behind it is more likely to be the interpretability of the accuracy of the model. Perhaps this may also explain why there are so many interpretable research ideas at present, but there seems to be no clear framework, and I think the main reason is that its interpretation objects are different, and there is no way to unify them.
Based on this understanding, I personally have a small idea that it should not be called explainability, and it may be more accurate to call it explainable. Explainability, which one might mistakenly think of as a property of independent existence; explainability is a kind of explainable ability, just like we say understanding, leadership, and so on, it is a means, a behavior, an operation exists, and it needs to be tied to something else. I think that when it is mentioned later, it should be accurately described as the explainable force for what characteristics it is, rather than generally speaking about how interpretable it is.
What is the value of explainable AI?
Zhu Jing:
People's requirements for the interpretability and transparency of artificial intelligence systems are roughly four levels:
The first is aimed at direct users, who need to understand what the principles behind AI products and services are, which is an important foundation for building trustworthy AI. Interpretable AI actually underpins trusted AI.
At the second level, for policies and regulatory authorities, they hope to understand the fairness and accountability of artificial intelligence products by explaining the principle, and the process of attribution is the basis for our further accountability and accountability. Therefore, interpretable AI is also associated with responsible AI, accountable AI.
The third level is the technical engineering and science level, we want to understand why some algorithms can succeed, what is the mystery behind its success, what is its application scope, and whether it can use such algorithms or some technologies on a larger scale.
The fourth is for the public to understand AI, and if the majority of the public cares, he can also understand what the corresponding technology and system generally work in this regard.
He Fengxiang:
In the current AI system, in fact, the operating mechanism behind many algorithms is unknown and unclear, and this unknown brings unknown and unmanageable risks, including security, robustness, privacy protection, fairness and so on.
These points are related to the key and life-threatening areas of social operation, such as medical care and autonomous driving. This will bring great application difficulties and social distrust of AI. Because when the operation mechanism of AI algorithm is unknown, its risk mechanism, risk size, and risk scale are unknown, and it is difficult for us to manage risks and then control risks.
What are the challenges of explainable AI?
It turned out that one of my students had done a little bit of work on fairness with me, which was very consistent with what other literature had found, that is, the accuracy and fairness of the model were contradictory. From the perspective of fairness, the model with the best performance is not necessarily the best measured by the indicator, you have to make the model the most fair, and if it is measured by the indicator, its performance will be lost. In fact, interpretability is very similar to the various versions of the interpretability indicators now, but if you really consider these indicators, the performance of the model will always fall, and you must consider how to find a compromise solution in the actual process.
The inability and undesirableness of interpretability itself is also a question worth pondering. For example, if we study crime rates or disease transmission rates, incidence rates, etc., if we take ready-made statistics, such as data collected in different races and different regions, it is very likely that some races or certain regions have high crime rates, which is because this is the case when data are collected. In this way, if similar interpretable conclusions are made public, racial or geographical discrimination may result. But in fact, behind the data is that we did not collect other characteristics when collecting, such as why is the spread rate of this region so high? It is likely that it is insufficient government investment, or some other factor.
So this also enlightens us to explainability itself, what its credibility is, its accuracy, its fairness, whether it ignores certain features, or exaggerates certain features, its robustness, whether it changes the sample a little, its interpretability is diametrically opposed, these need us to think further.
In addition, I have talked to many experts who study interpretability, and their confusion is that the current method of interpretability is unverified, even contradictory, which raises the question of the credibility of the method of interpretability itself.
In my opinion, there are two paths to understanding the operation mechanism of deep learning algorithms. Theoretically, current research cannot fully explain why theoretically less generalized depth models have been so successful in many fields. This contradiction between theory and practice, like the dark clouds in physics, reflects people's lack of understanding of machine learning, which is now a difficult point in improving the interpretability of algorithms in theory.
From an experimental perspective, practices in many experimental disciplines can be used as inspiration for machine learning research, such as physics, chemistry, and the medical treatment just mentioned. For example, the qualification test in the drug development process should be double-blinded; in the research of physics and chemistry, there are strict requirements for the experiment of control variables. Can a similar mechanism be strictly enforced in AI research? I think that might be another path. In my opinion, many of the existing explanations of AI algorithms are heuristic, and what we need in key areas is evidence, which requires a lot of work on both theory and experimentation.
How does explainable AI work?
Many experts have pointed out that there are different goals, different objects, and different requirements for interpretation, so in fact, the interpretability problem of artificial intelligence may belong to pluralism, that is, to allow a variety of different levels of different interpretations to play a role in this, for different fields, different objects, using different interpretations.
When interpretability has its limitations or other goals and requirements, and needs to make trade-offs, we think we can also carry out alternative, or compensatory and complementary strategies from multiple levels. For example, for the regulatory authorities, its requirements for interpretability, and for the public or expert level, will be different, so this can be through several levels, such as regulatory authorities, industry, market, and communication and popularization level, for safety, robustness requirements are higher, or there is better communication and understanding at the expert level, and for the public, there needs to be some conversion, and there is a need for some authoritative departments, departments with credibility, Make some explanations and identifications to the society.
Deep neural networks can solve particularly complex problems, and I think there is a reason why people use deep networks now, that is, the problem itself may be more complex. This is a hypothesis. If this assumption is correct, then the corresponding interpretability is not particularly well understood. Because these complexities need to be dealt with, the corresponding models are necessarily complex.
So I always feel that there is an inherent contradiction between transparency, explainability and performance, if you now put the direction of technical discussion, how to find a compromise solution, according to different scenarios, explainable purposes, find different compromise solutions, so that there may be some more specific technologies, or can promote these technologies to the direction of landing.
We have tried some technically feasible solutions to quantify various trustworthy features, but it is difficult to achieve uniform quantification, such as fairness and robustness have different quantitative criteria and indicators. It is difficult to optimize when combining different features simply, because their criteria are highly misaligned and very different, which involves how to align the coordinates of these features. I think it's very difficult to find a global coordinate system. We can start from the local, for a certain scenario, such as a medical scene, first of all, privacy as a premise, in finance or automatic driving, we take robustness as a premise, and then to study other characteristics, perhaps step by step to find this coordinate system.
Explain the state of AI's technology?
Zheng Yefeng:
In general, because we still lack a very good theoretical framework, we may creatively think of some algorithms for the problem, trying to improve the interpretability of the system itself, and give you two examples to illustrate the exploration of our Tianyan laboratory in this regard.
Deep learning may have hundreds of billions, trillions of parameters, which is too complicated for the doctor to understand the underlying principles of this algorithm, and the algorithm itself may lack a global interpretability. But the accuracy of deep learning frameworks is very high, so we can't use them. The model with very good interpretability is the regression model, and the main problem of this type of model is that the accuracy rate is too low. So we did an exploration, and we wanted to combine these two models, which have a very high accuracy rate and a certain amount of interpretability, not complete explainability.
We use this hybrid model for disease risk prediction, that is, based on the patient's previous medical records, we predict the probability that the patient will get a major disease within the next 6 months, such as the probability of his stroke. Every patient's visit contains a lot of information, in which we need to extract some important information related to the prediction goal, we know that the biological learning network is best at automatic feature learning. So we use a deep learning network to compress a visit record into a characteristic vector, and then we use a regression model to synthesize the patient's multiple visit records to predict the risk of stroke in the next 6 months.
Here we use the full linear regression model, from the patient dozens, hundreds of past calendar year records, select several visits most relevant to the prediction goal, select these cases, we will give it the corresponding weight. So the interpretability of this full linear regression model is very good, because it only focuses on very few variables, and ordinary people can understand it very well, with which number of visits to record, do information weighting, and get the final risk estimate. This is a global interpretability, much better than deep learning.
Yang Qiang:
We look at the correlation between each algorithm and its corresponding interpretability, and find an interesting phenomenon, for example, in machine learning, deep learning is very efficient, but its corresponding interpretability is very poor. Similarly, the linear model is not that high, but it is relatively interpretable, and so is the tree model, and the causal model is even more so. So often we do have to make a trade-off, that is, we choose which point in this space in this space, which one of the dimensions can be explained and the dimension of efficiency, and now there is no such algorithm that is higher in both dimensions.
Explains the industry practice of AI
Different industries have different requirements for explainability and transparency, and I will share my experience and understanding with you in combination with the scene of medical AI. We all know that medical treatment is a strongly regulated field around the world, a medical product must get a medical device registration certificate to be listed, and the auxiliary diagnostic algorithm AI product belongs to the three types of medical treatment, that is, the most stringent level of supervision, so we have to disclose a lot of information, roughly including data sets and clinical algorithm verification. The former emphasizes the fair diversity and broad coverage of data sets, while the latter focuses on disclosing the performance of our algorithm when it is truly in clinical trials and when it is truly clinically applied.
In addition, our test samples also need to have a good diversity, covering different hospitals, different regions, different patient groups, manufacturers, scanning parameters and so on. Clinical trials are more stringent, first of all, we have to solidify the code of the algorithm, and we cannot change the code during the clinical trial, because you can't change the code while doing the experiment, which loses the meaning of the clinical trial.
Therefore, the supervision of medical AI is very strong, and the Food and Drug Administration needs us to disclose a lot of information to improve the transparency of medical AI products, which has very strict or even harsh written requirements. Because we know that intelligent learning networks naturally do not have good explanatoryity, although you can do some intermediate enhancements, you can improve these things to a certain extent, and regulation can also understand that this explanatoryity is a little worse, and because of poor explanatoryity, the higher the transparency of the requirements.
I think there are two paths to providing instructions for AI systems. The first path starts with the process of generating an AI system. There are now practices for this, such as open source code, that explains what data is used, how the data is used, and how it is preprocessed. This will enhance people's trust and understanding of AI, which is also like Teacher Zheng just mentioned, when applying for medical-related qualifications, we need to report the production details to the relevant institutions.
The second way is to make an algorithm specification from the predictions made by the generated AI system and the indicators of the decision. For example, do some evaluation of AI systems. For the indicators we just mentioned, including interpretability, robustness, accuracy, privacy protection, fairness, find some better quantitative indicators, find some evaluation and estimation methods, and use these indicators as instruction manuals for AI systems.
Explains the future development of AI
Yang Qiang: I look forward to the governance of artificial intelligence in the future, under the premise of the harmonious coexistence of artificial intelligence, people and machines, and jointly solving the problems we have to solve, it will become more and more mature. I'm very bullish on this area.
Zhu Jing: I look forward to further discussions in this field, and scholars in different fields can participate. For example, like what I do myself, it's mainly philosophy, the philosophy of technology. In the philosophy of science and technology, in fact, there are nearly a hundred years of accumulation and exploration of interpretation, and there should be a lot of resources that can be explored and used to participate in such an interesting and challenging topic as the present.
He Fengxiang: AI itself is an interdisciplinary field, it may use a lot of mathematics, statistics, physics, computers and other fields of knowledge, many of the points mentioned today, including privacy protection, fairness, many of which also come from the humanities, law, sociology and other aspects. So this means that the study of trusted AI and explainability and other aspects will require people from various disciplines to work together to do one thing, and it will greatly require everyone to work together to promote the development of this field.
Yao Xin: For doing research, I hope that there will be a little bit of focused discussion in the future. The 3W I just talked about, which part of transparency and explainability do we want to address, for whom? What if it's for healthcare, for the regulators, for the doctors, for the patients, or for the developers of the system? I think there's a lot of room for your imagination and ability to play.
Wu Baoyuan: I hope that future AI researchers will have comprehensive thinking ability, not only focusing on a certain feature, such as focusing on accuracy. I hope to make credibility a prerequisite, and the relationship between multiple characteristics is worth thinking about. It is hoped that AI researchers and humanities scholars will have more exchanges and broaden their horizons. For the public and the government, it is hoped that through discussion, we can also understand the current situation of development, and hope to have an inclusive mindset to understand this field.
Zheng Yefeng: For the algorithmians, of course, we hope that in the future, scientists will find very good algorithms with good interpretability and very high accuracy, and really achieve both fish and bear paws.
