This article is the original of the Translational Medicine Network, please indicate the source for reprinting
Author: Tracy
Recently, large language models (LLMs) have been recognized as having great potential to advance medical diagnosis, especially in dermatological diagnosis, which is a very important task, because skin and subcutaneous diseases are among the leading contributors to the global burden of non-fatal disease. This study introduces SkinGPT-4, an interactive dermatological diagnostic system based on a multimodal large language model. With SkinGPT-4, users can upload photos of their own skin for diagnosis, and the system can autonomously evaluate the images, identify the characteristics and categories of skin conditions, conduct in-depth analysis, and provide interactive treatment recommendations.
On July 5, 2024, Gao Xin's team from King Abdullah University of Science and Technology published a research paper titled "Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4" in the journal Nature Communications. SkinGPT-4 can play a key role in improving patient access to healthcare and improving the quality of care services around the world. It is important to emphasize that no AI system is foolproof. Therefore, SkinGPT-4 is not intended to replace dermatologists, but rather as an evolving and optimized tool that acts as an assistant to facilitate communication between patients and doctors.
https://www.nature.com/articles/s41467-024-50043-3
Background:
01
Cutaneous and subcutaneous diseases are the fourth largest cause of the global burden of non-fatal disease, affecting a significant proportion of people, with prevalence ranging from 30% to 70% across all ages and regions. However, dermatologists have been in short supply, especially in rural areas, and the cost of consultation is rising. As a result, the burden of diagnosis often falls on non-specialists, such as primary care physicians, nurse practitioners, and physician assistants, who may have limited knowledge and training, resulting in low diagnostic accuracy. The use of teledermatology has become very popular in order to expand the range of services available to medical professionals. This involves transmitting a digital image of the affected skin area, usually taken with a digital camera or smartphone, as well as other relevant medical information from the user to the dermatologist. The dermatologist then reviews the case remotely and makes recommendations for diagnosis, work-up, treatment, and follow-up. Despite this, there are three major hurdles in the field of dermatological diagnostics. First, there is a lack of dermatologists who can diagnose patients, especially in rural areas. Second, accurately interpreting dermatological images is quite a challenge. Finally, generating patient-friendly diagnostic reports is often a time-consuming and labor-intensive task for dermatologists.
Advances in technology have led to the development of various tools and techniques to help dermatologists make diagnoses. For example, recent advances in deep learning (DL) have made it possible to develop artificial intelligence (AI) tools to help diagnose skin diseases from images. Most studies focused on identifying skin lesions from dermoscopic images. Dermoscopy, however, is often not readily available outside of dermatology clinics. Several studies have explored the use of clinical photographs of skin cancer, however, these methods are triage tasks tailored to specific diagnostic goals, and their methods still require further analysis by dermatologists in order to issue reports and make clinical decisions. These methods do not allow for the automatic generation of detailed reports in natural language, nor can they have an interactive conversation with the patient. Currently, there is no such diagnostic system that allows users to submit data autonomously, and through interactive analysis, generate easy-to-understand text reports and images to diagnose skin conditions on their own.
Inspired by the current state-of-the-art multimodal large language model, the team launched SkinGPT-4, an interactive dermatology diagnostic system based on a multimodal large language model. SkinGPT-4 brings innovation in two ways. First of all, SkinGPT-4 is a multimodal large language model aligned with Llama-2-13b-chat. Secondly, SkinGPT-4 is a multimodal large language model designed for dermatological diagnosis. With SkinGPT-4, users can upload photos of their own skin for diagnosis. The system autonomously evaluates images, identifies the characteristics and categories of skin conditions, conducts in-depth analysis, and provides interactive treatment recommendations. At the same time, SkinGPT-4's local deployment capabilities and commitment to user privacy also make it an attractive option for patients seeking a reliable and accurate diagnosis of skin conditions. To demonstrate the robustness of SkinGPT-4, the team quantitatively evaluated 150 real-world cases that were independently reviewed by board-certified dermatologists. The results show that SkinGPT-4 consistently provides accurate diagnosis of skin diseases. While SkinGPT-4 is not a substitute for doctors, it greatly enhances users' understanding of their medical conditions, facilitates patient-physician communication, speeds up the dermatologist's diagnostic process, facilitates tiered care, and has the potential to promote "person-centered" care and health care equity, especially in underserved areas. In summary, SkinGPT-4 represents a major leap forward in the field of dermatology diagnosis in the era of large language models, and is also a valuable exploration of multimodal large language models in medical diagnosis.
Illustration of SkinGPT-4
Research Progress
02
SkinGPT-4 is interactive, informative, and easy to understand
Dermatological diagnosis
SkinGPT-4 brings numerous advantages to patients and dermatologists alike. A significant benefit is that it leverages comprehensive and trustworthy medical knowledge tailored specifically to skin conditions. This allows SkinGPT-4 to provide interactive diagnosis, interpretation, and recommendations for dermatological conditions, which challenges MiniGPT-4. Unlike MiniGPT-4, which lacks relevant medical knowledge and field-specific adaptation training, SkinGPT-4 overcomes this limitation and increases its proficiency in the field of dermatology. To demonstrate the advantages of SkinGPT-4 over MiniGPT-4, the team presented two real-world examples of interactive diagnostics.
For actinic keratoses cases, MiniGPT-4 recognizes features such as small bumps and red bumps and incorrectly diagnoses the skin disease as acne; Whereas, SkinGPT-4 identifies features such as plaques, nodules, pustules, and scars, and diagnoses the skin disease as actinic keratoses, a common skin condition caused by prolonged exposure to the sun's ultraviolet (UV) rays. In an interactive conversation, SkinGPT-4 also suggested that the cause of skin diseases is sun exposure, which has also been verified by board-certified dermatologists. Taking fingertip eczema as an example, MiniGPT-4 identified some characteristics, such as cracks and peeling skin, but could not accurately diagnose the condition, and blamed dry weather and excessive hand washing as the cause of the skin disease. In contrast, SkinGPT-4 identified the characteristics of a skin disease as dry, itchy, and flaky skin, and diagnosed the type of skin disease as fingertip eczema, which was also validated by a board-certified dermatologist.
In conclusion, the lack of dermatological knowledge and field-specific adaptation poses a significant challenge to MiniGPT-4 in achieving accurate dermatological diagnosis. In contrast, SkinGPT-4 successfully and accurately identified the dermatologic features shown in the images. It not only suggests the underlying disease type, but also suggests potential treatments. This further highlights the importance of domain specificity for dermatological diagnosis of SkinGPT-4.
Clinical evaluation of SkinGPT-4 by a board-certified dermatologist
To assess the reliability and robustness of SkinGPT-4, the team conducted a comprehensive study involving a large number of real-world cases and compared its diagnosis to that of a board-certified dermatologist. The results show that SkinGPT-4 consistently provides accurate diagnoses, consistent with those of board-certified dermatologists.
Of the 150 cases, a significant proportion of SkinGPT-4 diagnoses (80.63%) were assessed by board-certified dermatologists as correct or relevant. The review included strongly agreed (75.00%) and agreed (5.63%). In addition, SkinGPT-4's answers to disease causes and potential treatments were considered beneficial (82.50%) and useful (85.63%) by doctors. SkinGPT-4 proved to be an invaluable tool in the doctor's diagnosis process (87.50%), allowing patients to better understand their disease (83.70%). SkinGPT-4's ability to support local deployment and protect user privacy has been highly recognized (92.50%), which further increases the willingness to use SkinGPT-4 (77.50%).
Overall, SkinGPT-4 provides reliable diagnoses, assists physicians in the diagnostic process, promotes patient understanding, and prioritizes user privacy, making it a valuable asset in the field of dermatology.
SkinGPT-4、SkinGPT-4（仅限第 1 步）、SkinGPT-4（仅限第 2 步）、MiniGPT-4和皮肤科医生生成的诊断。
SkinGPT-4 acts as a family doctor on call around the clock
Online consultation with a dermatologist who usually has to wait a few minutes to get a response; or a face-to-face consultation with a dermatologist who usually has to wait weeks for an appointment, SkinGPT-4 offers several advantages. First, it can be used around the clock, ensuring that patients receive medical advice on an ongoing basis. In addition, SkinGPT-4 offers a fast response time, usually within a few seconds. This makes it a quick and convenient option for patients who need immediate diagnosis outside of normal office hours.
SkinGPT-4 provides the ability to make an initial diagnosis, enabling patients to make informed decisions about seeking face-to-face medical care. This feature helps reduce unnecessary doctor's office visits, saving patients time and money. The potential to improve access to health care is especially important in rural areas or areas where dermatologists are scarce. In these areas, patients often face long wait times or have to travel long distances to see a dermatologist. By leveraging SkinGPT-4, patients can get their initial diagnosis quickly and conveniently, reducing the need for in-person visits and reducing the strain on healthcare systems in these underserved areas.
Consistency of SkinGPT-4 diagnosis
GPT tends to generate results in a variety of formats based on probability, so careful consideration must be given to the risks and consistency associated with AI-generated content, especially when it comes to medical diagnosis. To demonstrate the consistency of the SkinGPT-4 results, the team randomly selected 45 samples (5 from each category). For each sample, the team performed 10 independent diagnoses. SkinGPT-4 made the same diagnosis on the same graph, with a consensus ratio of 93.73%. For inconsistent conditions, a board-certified dermatologist can observe the characteristics of a wide range of possible skin types. For example, benign tumors can be easily confused with melanoma skin cancer. Overall, the diagnosis of SkinGPT-4 is consistent and reliable.
Clinical evaluation of SkinGPT-4 by board-certified offline and online dermatologists.
Conclusions of the study
03
The team's research demonstrates the potential of using visual input in LLMs to enhance dermatological diagnoses. With the release of more advanced LLMs such as GPT-4, the accuracy and quality of diagnostics can be further improved. However, potential privacy issues related to the use of LLMs such as ChatGPT and GPT-4 as APIs must be addressed. Because it requires users to upload their private data. In contrast, SkinGPT-4 provides a solution to this privacy problem. By allowing users to deploy the model on-premises, concerns about data privacy can be effectively addressed. Users can use SkinGPT-4 independently within the scope of their own system to ensure the security and privacy of their personal information.
Deploying SkinGPT-4 in the real world can present potential challenges, especially due to the variability of images submitted by patients. Factors that contribute to this discrepancy include differences in smartphone camera quality, variations in image pre- and post-processing, different angles, and different lighting conditions. In addition, addressing the different severity of skin conditions is another challenge. During the training of SkinGPT-4, the scientific community lacked the specific data needed to enable the model to accurately identify the severity of skin diseases. SkinGPT-4 still exhibits robust and acceptable performance when presenting dermatological images taken at different angles, lighting conditions, pixel densities, and resolutions.
The diagnosis of complex skin diseases presents additional challenges to SkinGPT-4. In practice, complex dermatological diseases often occur, including a combination of dermatological diseases that exhibit a wide range of characteristics. Currently, there is a lack of datasets that contain multi-label dermatological images, as well as corresponding dermatologist diagnoses. Solving this data gap is the key point for applying SkinGPT-4 to the diagnosis of complex skin diseases in the future.
Current research on Fitzpatrick V-VI (dark skin tone) is relatively limited, and the performance of state-of-the-art dermatology AI algorithms on dark skin lesions is significantly inferior compared to the efficacy of light skin lesions, especially if confirmed by biopsy. The main challenge comes from the fact that the early features of some dark skin diseases are less pronounced, making diagnosis more challenging. As a result, people with darker skin tones often receive a diagnosis at a later stage, leading to increased morbidity, mortality, and associated costs. Compounding the problem is the scarcity of Fitzpatrick V-VI data. To address this limitation, future research work will involve the systematic collection of Fitzpatrick V-VI data and targeted training of SkinGPT-4 to enhance its diagnostic capabilities for Fitzpatrick V-VI patients.
With LLM-based applications such as SkinGPT-4 accessing more reliable medical training data, and constantly evolving and improving, the potential for significant advancements in online medical services is enormous. SkinGPT-4 can play a key role in improving patient access to healthcare and improving the quality of care services around the world. It must be emphasized that no AI system is foolproof or completely free of misinformation and misdiagnosis. Therefore, SkinGPT-4 is not intended to replace dermatologists, but rather as an evolving and optimized tool that acts as an assistant to facilitate communication between patients and doctors. The team's expectation for SkinGPT-4 is to provide patients with more information about skin conditions, while also providing valuable assistance to doctors in the diagnosis process. Therefore, the team has included clear disclaimers and guidelines on the software page, emphasizing the importance of following medical advice and strongly recommending consulting with a qualified doctor for a specific diagnosis. These precautions are designed to encourage patients to use SkinGPT-4 responsibly and to ensure that users understand the limitations of the software in a healthcare setting. The team will continue to conduct research in this area to further develop and refine this technology.
【Contributions】
The Translational Medicine Network (360zhyx.com) is the core portal of translational medicine, which aims to promote the development of basic research, clinical diagnosis and treatment, and industry, covering omics, testing, immunology, oncology, cardiovascular, diabetes, etc. If you have the latest research content published, please contact us for a free report (official account menu bar - online customer service contact), our philosophy: content creates value, transformation creates the future!
The articles published by the Translational Medicine Network (360zhyx.com) are intended to introduce the progress of cutting-edge medical research and cannot be used as a treatment plan. If you need health guidance, please visit a regular hospital.
Popular · live broadcasts/events
🕓 Beijing|07/12
▶ 2024 Symposium on the Application of Single-Cell Omics in the Research and Diagnosis and Treatment of Rheumatic Immune Diseases
🕓 Beijing|09/20-21
▶ The 5th Symposium on the Application of Single Cell Technology and the Frontier Symposium on Spatial Omics welcomes your participation!
Click on the corresponding text to view details