AI in Hidradenitis Suppurativa: Expert Evaluation of Patient-Facing Information

Introduction

Hidradenitis suppurativa (HS) is a chronic and inflammatory skin disease characterized by painful abscesses, nodules, and sinus tracts, primarily affecting the axillae, groin, and perianal areas. The global prevalence of HS is estimated to be around 1%, with higher rates in females and individuals with a family history of the condition. HS can lead to significant physical and psychological morbidity, including pain, scarring, and social isolation. Many patients hesitate to share their symptoms with healthcare professionals due to the intimate nature of the affected areas, feelings of embarrassment, and prior negative healthcare experiences, which can result in delayed diagnosis and treatment.¹ In recent years, there has been a growing trend among patients to use Artificial intelligence (AI) to search for health information online. A recent survey found that approximately 30–40% of patients use AI systems like virtual assistants or chatbots for health-related inquiries,² highlighting the increasing role of AI in patient self-management and health information gathering. AI emerges as a potential ally for patients, offering a means for them to gain better insights into their symptoms and potentially reducing the considerable time gap between initial symptoms, medical diagnosis, and treatment.¹ AI systems (AIs), such as ChatGPT (Chat Generative Pre-Trained Transformer) developed by OpenAI and released on November 30, 2022, and Bard developed by Google and released on March 21, 2023, are advanced language models capable of providing conversational responses similar to interacting with a human.³ The deployment of AI chatbots holds considerable promise for elevating patient care in dermatology and contributing to broader public health initiatives.⁴

We report on an innovative study evaluating the accuracy of AIs—ChatGPT and —in providing information on HS. Our study aims to explore the potential of AI chatbots in supporting HS patients by offering insights into symptoms, thus possibly reducing the time to diagnosis and treatment.

Material and Methods

The two most widely used AI systems were selected and studied: Chat GPT (version 3.5) and Bard.⁵ Seven questions related to HS were developed with the help of HS patient associations:

These questions were administered in an individual chat by an investigator (AC E). All questions were asked to both AI systems on the same day (December 20, 2023) in French for practicality.

The ChatGPT and Bard responses were independently assessed by 18 hS experts. All experts independently evaluated all responses using a 5-point Likert scale (1: strongly agree; 2: agree; 3: neutral; 4: disagree; 5: strongly disagree).

Statistical Analysis

We developed a web-based questionnaire Google Form to allow the experts to evaluate the AI responses. All participant responses were exported as an Excel file (Microsoft Corporation) from the Google Form website. An AI-generated response was deemed appropriate if the majority of experts strongly agreed or agreed with it; otherwise, it was classified as inappropriate.

The average points obtained for each question according to the Likert scale were calculated and presented as mean ± standard deviation. A reminder that the Likert scale used ranged from 1 (strongly agree) to 5 (strongly disagree). For comparisons where the data followed a normal distribution, an independent t-test was used. However, for comparisons where the data did not meet the normality assumption, the Mann–Whitney U-test was applied. Differences in scores between ChatGPT and Bard were specifically analyzed using the Mann–Whitney U-test, as the data did not conform to normal distribution. A statistical significance threshold was set at a p-value of < 0.05 for all analyses.

Results

The complete list of questions and answers can be found in Appendix.

Both AI systems responded to all seven questions. ChatGPT provided relatively shorter answers with a mean word count of 228±48 (range 148–292), compared to 254±77 (range 96–354) for Bard.

Responses by ChatGPT to 6 of the 7 queries were deemed “appropriate” (86%) by the experts, whereas the response to one question (14%) was rated inappropriate (Q6 “What treatments are available?”). Responses by Bard to 1 of the 7 queries were deemed “appropriate” (14%) by the experts (Q5 “Which doctor do I need to see?”), whereas responses to 86% (n=6), were rated inappropriate.

Figure 1 presents the descriptive statistics for the scores given by the 18 participants. The correlation between the scores given by all evaluators for the answers provided by the two AI systems, is presented in Table 1.

Table 1 Comparative Analysis of Scores by Evaluators for Responses from Two AIs

Figure 1 List of the seven queries submitted to ChatGPT and Bard along with their experts’ evaluations.

A significant difference in mean scores between the two AI was found by all participants, who indicated that ChatGPT was the most accurate. The response to question 6 regarding treatment options was classified as “completely incorrect” when the majority of experts rated it with the lowest possible score on the Likert scale, indicating that they found the answer to be inaccurate or inappropriate. There was no significant difference between the two AIs (p=0.086) in the evaluation of question N°6. However, ChatGPT offered better advice to improve patient quality of life.

Discussion

This study marks a pioneering attempt to assess the capability of AI in HS self-diagnosis. ChatGPT outperformed Bard, offering more precise answers, particularly on symptoms, although both struggled with treatment-related questions.

Our findings suggest a significant difference in the performance of the 2 AIs. ChatGPT was deemed accurate (appropriate responses) in 86% of cases, overshadowing Bard’s 14% accuracy rate. Notably, all experts highlighted ChatGPT’s superior performance, marking a stark contrast in reliability and the potential utility in patient support. However, both AI systems fell short in offering satisfactory advice on HS treatment.

AI’s ability to provide both accurate and inaccurate HS information has far-reaching implications for patients, healthcare professionals and the healthcare system as a whole. Accurate AI-generated information could empower patients by helping them understand their symptoms earlier and prompting them to consult healthcare professionals in a timely manner, potentially reducing delays in diagnosis. This could have a significant positive impact on patient outcomes, particularly for diseases such as HS, for which early intervention is essential. For healthcare professionals, AI systems could serve as additional tools to improve patient education and help with initial screening, reducing the workload of overburdened clinicians. However, there are serious risks if AI provides inaccurate or misleading information, particularly in the medical context. Inaccurate treatment advice or diagnostic information could delay appropriate care, leading to a deterioration in patients’ conditions and increased healthcare costs due to mismanagement or unnecessary interventions.

From a systemic perspective, if AI systems such as ChatGPT or Bard can reliably improve symptom understanding, they could potentially ease pressure on healthcare systems by reducing the need for unnecessary consultations or diagnostic procedures. However, if AI systems frequently provide incorrect information, the resulting delays in diagnosis or treatment could add to the burden on healthcare services, as patients may present with more advanced or complicated cases. It is therefore essential to improve the accuracy of AI responses, so that AI becomes a valuable asset rather than a liability in healthcare.

For Question 6, related to treatment options, 61.1% of the experts rated ChatGPT’s response as either “disagree” or “strongly disagree”, indicating their dissatisfaction with the accuracy of the information provided. This is also the question where Bard’s responses received the lowest ratings: 88.9% of the experts rated Bard’s response as either “disagree” or “strongly disagree”, indicating a high level of dissatisfaction with the accuracy of the information provided. This point is aligned with findings from a recent systematic review on ChatGPT by Sallam et al.² The tendency for incorrect responses may be attributed to the model’s primary training on diverse internet content, including articles, books, Wikipedia, news, and websites, rather than sourcing information directly from scientific societies. The responses to treatment-related questions were rated poorly because they often lacked the necessary specificity and accuracy expected from current medical guidelines. These topics tend to be more factual and within the AI systems’ capability to provide correct and consistent information, whereas treatment recommendations are more complex and require individualized, evidence-based guidance. We advocate for enhancing the accuracy of responses to medical queries by mandating that AIs consult specific scientific societies instead of relying solely on websites. The challenge stems from the potential variations in recommendations among different societies and countries.

In the initial four questions, ChatGPT outperformed Bard. Notably, in the field of diagnostics, ChatGPT has exhibited its ability in producing satisfactory results in previous studies.^6–9 It is essential to highlight that only one study has demonstrated superiority of ChatGPT over other AIs.¹⁰

However, the question arises: why is there disparity between the two AI systems? The emergence of AI technology has ushered in dialogue models that are gradually superseding traditional search engines such as Google or Bing. These models, rooted in large language models, employ deep learning techniques to generate natural language texts in various formats, including question and answer, summaries, and narratives, based on user-provided data.¹¹ The disparity between the two AI systems, ChatGPT and Bard, likely stems from differences in their training data and underlying algorithms. ChatGPT, which is designed to generate creative and coherent responses, excels in producing conversational answers that align well with patient queries about symptoms and general disease information.¹² Bard, on the other hand, focuses on providing comprehensive and informative responses but struggles with the level of precision needed for accurate medical information, particularly in complex areas like treatment recommendations. These differences in approach and data sources explain why ChatGPT performed better in this study, especially in areas requiring nuanced, human-like communication.

Despite these distinctions, when it comes to HS-related questions, this limitation did not seem to hinder ChatGPT’s performance, as it consistently outperformed Bard.

Interestingly, neither of the two AIs suggested that patients refer to reputable medical society websites, which offer scientifically validated content overseen by expert committees. Nevertheless, it is essential to underscore that while AI serves as a helpful resource, it cannot substitute a comprehensive medical assessment. Given the complex and varied nature of HS symptoms, only a qualified healthcare provider can ensure an accurate diagnosis. Self-diagnosis based solely on online information, even with AI, carries the risk of misinterpretation.

Limitation

This study has several limitations that should be considered when interpreting the results. First, the sample size of 18 hS experts, while adequate for an exploratory study, may not be fully representative of the broader dermatology community. A larger cohort would provide a more comprehensive range of expert opinions and enhance the validity of the findings. Replicability is a key consideration for future research Second, the use of only 7 predefined questions limits the scope of the AI evaluation, as these questions may not encompass the full range of HS-related patient queries. Future studies should expand the question set to assess AI performance across diverse medical contexts. Third, the reliance on expert evaluations introduces a degree of subjectivity, despite their value. Incorporating more objective measures, such as patient outcomes or alignment with established clinical guidelines, would strengthen the conclusions. Lastly, the cross-sectional design limits our ability to observe changes in AI performance over time, particularly as these systems undergo regular updates and improvements.

A longitudinal approach could also provide valuable insights into how AI performance changes over time and its potential role in medical information dissemination.

Additionally, using AI chatbots in clinical contexts is not yet advisable due to the lack of comprehensive data on their medical expertise. The ethical implications of relying on AI for medical information also warrant caution, as misinformation could have significant consequences for patient care. Finally, these AI systems do not offer clinical imaging capabilities to support diagnostic hypotheses, limiting their utility in more complex medical scenarios.

Conclusion

In conclusion, the comparison between ChatGPT and Google Bard in responding to 7 specific questions about HS suggests that AI chatbots may have the potential to assist in providing information and hopefully reducing diagnostic delays. ChatGPT demonstrated a higher accuracy rate compared to Bard, particularly in providing symptom-related information, but both systems showed significant limitations in addressing treatment-related questions. While these findings are encouraging, they are limited to HS and cannot be generalized to other medical conditions or broader healthcare contexts. Additionally, AIs tools should not be considered replacements for professional medical consultation. Further research is needed to evaluate AI’s performance across a wider range of medical conditions and questions before drawing broader conclusions about its role in healthcare. An ethical and legal framework for AIs should be collaboratively established, as the dissemination of misinformation can pose detrimental consequences for wandering patients.