Abstract

BACKGROUND/AIMS

Large language models (LLMs) such as ChatGPT-4o mini, ChatGPT-4o, Gemini 1.5 Pro and Microsoft Copilot play an important role in improving health literacy and supporting patient decision making. This study aims to evaluate the clinical appropriateness and readability of responses provided by these LLMs to frequently asked questions about retinal detachment (RD), epiretinal membrane (ERM), and macular hole (MH).

MATERIALS AND METHODS

This comparative cross-sectional study did not involve human participants. A total of 121 questions related to RD, ERM and MH were compiled. These questions were presented to LLM interfaces and were categorised into different themes. Inter-rater agreement between two retinal specialists who scored the responses based on their clinical expertise was assessed using the intraclass correlation coefficient (ICC). In addition, the readability of the responses was analysed using five readability indices.

RESULTS

Response appropriateness rates were similar for ChatGPT-4o mini (100%), ChatGPT-4o (99.2%) and Gemini 1.5 Pro (99.2%), while Microsoft Copilot (90.9%) showed comparatively lower performance. A high level of inter-rater agreement was observed among clinicians. Regarding readability, ChatGPT-4o performed best, whereas ChatGPT-4o mini performed worst. Gemini 1.5 Pro and Microsoft Copilot were intermediate between these two models. The excellent inter-rater agreement (ICC: 0.891-0.938) further supports the reliability of the expert evaluation process.

CONCLUSION

Our study showed that the evaluated LLMs provided largely appropriate and comprehensive answers to questions related to vitreoretinal surgery. ChatGPT-4o showed the highest performance in both appropriateness and readability, while ChatGPT-4o mini, despite its high appropriateness, had the lowest readability score.

INTRODUCTION

Natural language processing (NLP) techniques, particularly chatbots powered by large language models (LLMs), offer significant potential in healthcare by providing insights into disease prevention, diagnosis, and treatment.^{1, 2} Trained on vast textual datasets, these models generate human-like responses through self-supervised learning.^{3, 4}

LLMs process medical data contextually and perform semantic analysis, improving both information retrieval and patient communication.³ Unlike traditional rule-based chatbots with fixed keyword-based responses, LLMs generate more detailed and meaningful answers by analyzing queries in context.⁴ Their ability to retain prior interactions enhances response consistency, thereby benefiting patient education. Additionally, LLMs improve medical information accessibility and clarity, contributing to effective health communication. These models are now widely applied across healthcare settings.

This study evaluates the capabilities, applications, and readability levels of four prominent artificial intelligence (AI) models: OpenAI’s ChatGPT-4o mini and ChatGPT-4o, Gemini 1.5 Pro, and Microsoft Copilot. ChatGPT-4o mini, a lightweight and cost-effective model, is optimized for basic NLP tasks, while ChatGPT-4o is an advanced multimodal system that processes text, images, audio, and video.^{5, 6} Both models set a benchmark for LLM performance and enhance user interaction.

Google’s Gemini project, the company’s most advanced AI model, builds on the Bard AI programme’s core technologies. Gemini 1.5 Pro aims to enhance AI capabilities and user interaction, making it the latest version chosen for this study.⁷ Microsoft’s Copilot integrates NLP, retrieval-augmented generation (RAG), and LLMs to generate accurate responses. By retrieving diverse data, it improves clinical appropriateness and user experience, while its RAG framework enhances dynamic information processing in search-based models.^{8, 9}

LLMs are trained on extensive datasets that incorporate content with varying levels of reliability. The presence of inconsistent online information gives rise to concerns regarding the reliability of generated responses, which may contain inaccuracies or omissions. This emphasises the necessity for rigorous evaluation and verification of medical information, particularly in healthcare settings where chatbot-generated content directly impacts patient safety.^{10, 11} The challenge stems from LLMs’ limited capacity to assess the reliability and clinical appropriateness of their training data.^{3, 12} While these models continue to evolve, further in-depth assessment is required to determine their effectiveness in specific medical domains.

Vitreoretinal diseases are serious conditions that can lead to vision loss and often necessitate surgical intervention. It is imperative that patients have access to accurate, contemporary, and comprehensible information. However, ophthalmic surgical explanations frequently involve complex medical terminology, which can impede readability and comprehension, particularly for patients with limited medical knowledge. While LLMs enhance access to medical information, their clinical appropriateness and clarity are critical in facilitating effective patient communication.¹³ Therefore, this study evaluates both the clinical appropriateness and readability of LLM-generated responses to determine their accessibility for patient education.

Previous studies evaluating ophthalmology chatbots have primarily focused on individual retinal diseases, theoretical knowledge assessments, or the clinical appropriateness of a limited number of LLMs. Moreover, readability and comprehensiveness have not been evaluated across multiple contemporary LLMs for common vitreoretinal surgical conditions. Therefore, this study aimed to compare the clinical appropriateness, comprehensiveness, and readability of responses generated by four widely used LLMs for frequently asked questions regarding retinal detachment (RD), epiretinal membrane (ERM), and macular hole (MH), thereby providing a broader assessment of their potential role in patient education. We hypothesized that although contemporary LLMs would generally provide clinically appropriate responses, there would be significant differences in their comprehensiveness and readability.

MATERIALS AND METHODS

Study Design

This study was conducted on February 5, 2025, using ChatGPT-4o mini, ChatGPT-4o (OpenAI, San Francisco, CA, USA), Gemini 1.5 Pro (Google LLC, Mountain View, CA, USA), and Microsoft Copilot (Microsoft Corporation, Redmond, WA, USA). These LLMs were selected based on their performance and accessibility.

Ethics Approval and Informed Consent

This study did not involve human participants or identifiable human data. The Institutional Review Board of our hospital determined that the study met the criteria for exemption from ethical review and that it was conducted in accordance with the tenets of the Declaration of Helsinki. Therefore, informed consent was not required.

ChatGPT-4o mini is optimized for efficiency, requiring less computational power and offering faster responses, while ChatGPT-4o is designed for high-precision tasks requiring more advanced processing. Both models were included to assess their respective strengths. Similarly, Gemini 1.5 Pro, an advanced multimodal model by Google, and Microsoft Copilot, an AI-powered assistant integrated with search-engine capabilities, were evaluated for their clinical appropriateness and readability in ophthalmic contexts.

Patient questions were sourced from reputable ophthalmology databases (e.g., the American Academy of Ophthalmology: https://www.aao.org/eye-health). Frequently asked questions were identified from online patient forums and clinical encounters. Each question was submitted to the LLMs three times to assess response variability. The three generated responses were assigned sequential numbers (1-3), and one response was selected using a computer-generated random number before expert evaluation to minimize selection bias. Only one randomly selected response was included in the analysis to simulate a typical real-world patient interaction and to avoid treating multiple responses to the same question as independent observations, which could have artificially inflated the sample size and influenced the comparative analyses. The questions were categorized into six domains: pathogenesis, risk factors, clinical presentation, diagnosis, treatment/prevention, and prognosis. To minimize bias, conversation history was cleared after each response, and chatbot-specific features were removed for a blinded assessment. Figure 1 illustrates the overall study design.

Clinical Appropriateness Assessment

All recorded responses were independently reviewed by two retina specialists, one attending physician (S.E.) and one associate professor (M.K.), based on their clinical expertise. Each response was categorized as “appropriate,” “inappropriate,” or “incomplete”.

A response was classified as appropriate if it closely resembled the standard medical guidance that a clinician would provide and contained accurate information.

Responses containing incorrect or misleading information or significantly deviating from accepted clinical recommendations were considered inappropriate.

Incomplete: a response that was relevant and correct but lacked sufficient detail.

The final clinical appropriateness rating for each LLM-generated response was determined by consensus between the two evaluators. If consensus could not be reached after discussion, the corresponding question and response would have been excluded from the analysis according to the predefined study protocol. However, consensus was achieved for all evaluated responses; therefore, no responses were excluded.

Comprehensiveness Assessment

Responses that were rated as “appropriate” underwent further evaluation to assess their level of detail and completeness using a five-point Likert scale:¹⁴

Not comprehensive (1): The response failed to address the key aspects of the question or contained incorrect or misleading information, rendering it of no value to patients or clinicians.

Slightly comprehensive (2): The response provided only basic definitions with minimal detail and lacked sufficient explanation for patient education.

Moderately comprehensive (3): While covering fundamental concepts, the response remained superficial, omitting essential diagnostic or treatment details.

Comprehensive (4): The response addressed the most critical elements but lacked additional insights or alternative perspectives, making it generally useful but requiring further clarification for clinical decision-making.

Highly comprehensive (5): The response provided a detailed, scientifically accurate, and well-structured explanation, covering diagnosis, treatment, complications, and alternative approaches, making it optimal for patient education.

All responses were independently assessed by two retina specialists who were blinded to minimize bias. Inter-rater reliability was measured using the intraclass correlation coefficient (ICC) to assess consistency between evaluators. High ICC values indicated strong agreement, ensuring the reliability of the assessment process.

Readability Assessment

The readability of LLM-generated responses was assessed using the online readability tool Readable (https://app.readable.com/text/) to determine their accessibility for an average reader. Five well-established readability indices were used:^{15, 16}

Flesch-Kincaid grade level: This metric estimates the United States (US) grade level required to understand a text by analyzing sentence length and syllables per word. Lower Flesch-Kincaid grade level (FKGL) scores indicate greater readability, suggesting that the content is easier to understand. This index is commonly used in medical education and patient information materials to assess whether a text aligns with the general population’s literacy level.¹⁵

The gunning fog index evaluates readability based on sentence complexity and the proportion of polysyllabic those with three or more syllables. It is particularly relevant for medical texts, as dense terminology can significantly impact comprehension. A gunning fog index (GFI) score above 12 indicates that the text requires university-level reading proficiency.^{15, 17} In this study, GFI served as a key indicator of the ease with which LLM-generated content could be understood by general readers and patients.

Coleman-Liau index: Unlike traditional readability indices, Coleman-Liau index (CLI) assesses text complexity based on character count per word rather than word count per sentence. This makes it especially useful for medical documents, in which longer medical terms can distort traditional readability metrics. CLI provides a more precise measure of cognitive load and health literacy, offering insights into whether AI-generated medical texts align with the general public’s comprehension abilities.¹⁵

The simple measure of Gobbledygook index (SMOGI) is widely applied in public health and patient education; it evaluates readability by measuring the frequency of polysyllabic words. A score above 8 suggests that the text is difficult to read, requiring higher education levels for full comprehension.^{15, 17} In healthcare communication, SMOGI is essential for assessing the readability of medical responses and their accessibility to individuals with lower health literacy.

Flesch reading ease score: Assesses readability on a 0-100 scale, with higher scores indicating greater ease of reading. Scores between 70 and 80 correspond to an 8^th-grade reading level, making the text suitable for general patient education. A low flesch reading ease score (FRES) score suggests high text complexity, which may hinder effective communication in healthcare settings.¹⁷

Lower FKGL, GFI, CLI, and SMOGI scores indicate higher readability, making the text easier to comprehend. Conversely, a higher FRES score reflects improved accessibility. These indices correlate with the minimum education level required to understand the text in the US education system, providing a standardized measure for evaluating patient-directed medical information.¹⁸

Statistical Analysis

In this study, continuous variables were reported as mean ± standard deviation and minimum-maximum values, while descriptive statistics for categorical variables were presented as counts and percentages (%). The normality of the distributions of continuous variables was assessed using the Kolmogorov-Smirnov test. To determine statistically significant differences between readability metrics and LLM performance, a one-way analysis of variance was applied, followed by Tukey’s honest significant difference post-hoc tests to identify pairwise differences. The results were analyzed to determine the best- and worst-performing LLMs for each disease group and readability metric. Additionally, Pearson correlation analysis was conducted to evaluate the relationships between LLM-generated responses and readability indices, with results presented in Supplementary Table 1. Statistical analyses were performed using the SPSS, version 28.0 (IBM Inc., Armonk, NY, USA). A p-value <0.05 was considered statistically significant, and 95% confidence intervals were used.

RESULTS

Response Appropriateness and Consistency

A total of 121 questions were asked of four LLMs. For RD, 55 questions were categorized into pathogenesis (n=10), risk factors (n=10), clinical findings (n=10), diagnosis (n=10), treatment/prevention (n=10), and prognosis (n=5). For ERM, 31 questions were categorized into pathogenesis (n=7), risk factors (n=3), clinical findings (n=4), diagnosis (n=3), treatment/prevention (n=8), and prognosis (n=6). For MH, 35 questions were categorized into pathogenesis (n=8), risk factors (n=7), clinical findings (n=6), diagnosis (n=5), treatment/prevention (n=5), and prognosis (n=4) (Supplementary Table 1).

Independent evaluations showed similar levels of response appropriateness for ChatGPT-4o mini (121/121, 100%), ChatGPT-4o (120/121, 99.2%), and Gemini 1.5 Pro (120/121, 99.2%), whereas Microsoft Copilot achieved 110/121 (90.9%) appropriate responses, with 10 incomplete and 1 inappropriate response (Table 1). Figure 2 illustrates the distribution of questions and response appropriateness ratings. Although other LLMs showed consistency across disease groups, Microsoft Copilot had lower reliability, particularly for treatment- and prognosis-related answers. Some diagnostic responses also lacked detail and were therefore clinically incomplete. ChatGPT-4o mini, despite its high response appropriateness, sometimes produced responses that were highly technical and difficult to read. In particular, Microsoft Copilot produced a treatment response that was scientifically questionable, further highlighting limitations in the appropriateness of its responses.

Response Comprehensiveness and Inter-Rater Reliability

The Likert scale evaluations of LLM responses are summarized in Table 2. ChatGPT-4o scored highest for RD, Gemini 1.5 Pro scored highest for ERM, and both models performed best for MH. However, no statistically significant differences in Likert scores were observed among LLMs in any disease group (RD: p=0.063, ERM: p=0.532, MH: p=0.086). Inter-rater agreement, assessed using ICC, demonstrated high consistency across evaluations (Table 3). This strong agreement confirms the reliability of the assessment and supports the objectivity of the evaluation process.

Readability Performance

The readability scores of LLM-generated responses are summarized in Table 2. ChatGPT-4o exhibited the highest readability across all disease groups, while ChatGPT-4o mini exhibited the lowest readability. ChatGPT-4o mini scored higher on the FKGL, GFI, CLI, and SMOG indices and lower on the FRES, indicating a more complex and technical language style (p<0.001). Conversely, ChatGPT-4o provided the most accessible responses, making it the most suitable model for patient education.

Analysis of the relationship between response appropriateness and readability metrics revealed that models with higher readability scores, such as ChatGPT-4o, tended to have higher appropriateness rates. Despite the appropriateness of ChatGPT-4o mini’s responses, it remained the least readable model. Gemini 1.5 Pro performed similarly to ChatGPT-4o in response appropriateness but had lower readability scores in some metrics.

Figure 3 presents scatter plots of readability indices for LLM-generated responses, illustrating variation across models. Correlation analyses by disease group revealed no strong associations, indicating variability in readability metrics. Notably, ChatGPT-4o mini generated responses with higher GFI and SMOGI scores, reflecting longer and more complex sentences, whereas other models exhibited lower scores, suggesting simpler, more readable responses. These results demonstrate that LLMs do not consistently produce responses of equal readability; measured readability varies depending on the applied readability metric (Supplementary Table 2).

Response Length and Word Count

The word counts of LLM-generated responses are presented in Supplementary Table 3. Significant differences were observed across disease groups (p<0.001). ChatGPT-4o mini produced the shortest responses, likely due to its optimized architecture, which compresses information for efficiency. This design enables faster responses but results in more concise outputs compared to larger models.

DISCUSSION

Our findings indicate that LLMs provide comprehensive responses to vitreoretinal surgery-related questions, with comparable performance across models. However, Microsoft Copilot demonstrated the lowest appropriateness, particularly in diagnosis, treatment, and prevention, which was most pronounced in the RD group. This aligns with previous studies reporting Bing (now Microsoft Copilot) performing worse in similar categories.^{19, 20}

Several studies have evaluated Gemini’s appropriateness in ophthalmology, primarily focusing on theoretical knowledge in exam-based formats, without assessing readability or accessibility for patient education.²¹ Another study analyzed Gemini’s ability to detect macular diseases in OCT images but did not evaluate chatbot-generated responses.²² Similarly, research on chatbot responses to age-related macular degeneration and diabetic retinopathy was limited to two disease groups.²³ Unlike previous studies, our research assesses both the appropriateness and readability of patient-centered chatbot responses across a broader spectrum of ophthalmic conditions, providing insights into those responses.

While LLM chatbots generate grammatically correct and human-readable texts, their suitability for patient education depends on meeting specific readability criteria. ChatGPT-4o mini exhibited the lowest readability despite its high appropriateness, likely due to its use of more technical, condensed language, complex sentence structures, and greater information density. This pattern aligns with findings from a study comparing ChatGPT-4o and ChatGPT-4o mini.²⁴

The American Medical Association recommends that health-related materials should be at a 5^th to 6^th-grade reading level.²⁵ However, our results indicate that LLM-generated responses are often complex, with low readability scores. This aligns with previous research demonstrating low readability in online RD and ERM resources, as well as findings that ChatGPT-4 responses for atrial fibrillation education had lower readability scores than other chatbots.^26-28 In contrast, a study on melanoma found no significant readability differences among Bard, Bing, and ChatGPT-4.²⁹ These discrepancies suggest that some LLMs favor technical and academic language, potentially limiting accessibility for patient education. Improving readability could enhance public health communication and information dissemination.

Our readability analysis highlights that complex sentence structures may hinder patient comprehension.³⁰ Thus, LLM-generated responses should be evaluated not only for appropriateness but also for readability in medical education materials.^{31, 32}

The integration of chatbots into ophthalmic surgery education and patient follow-up is becoming increasingly relevant.³³ Preoperatively, chatbot-assisted education can provide realistic expectations about surgery, potentially reducing patient anxiety.³⁴ Additionally, these systems can enhance patient-doctor interactions by enabling patients to ask more informed questions.³⁰ In the postoperative phase, chatbots play a crucial role in improving patient adherence, monitoring for complications, tracking medication compliance, and encouraging follow-up visits.^{31, 32} Such tools can alleviate clinical workload, facilitate more effective communication, and contribute to better clinical outcomes.³³

Our findings underscore that readability is not just a technical parameter but has a direct impact on patient understanding and decision-making.³⁴ Optimizing chatbot-generated responses for readability is essential not only for patient education but also for clinical decision-support systems.³³ Improving readability is essential for effective patient communication, as high clinical appropriateness does not necessarily translate into better patient comprehension.^{31, 34}

Among all disease groups, ChatGPT-4o mini produced the shortest responses. While Gemini 1.5 Pro was previously reported to generate longer responses, our study found that ChatGPT-4o, Gemini 1.5 Pro, and Microsoft Copilot produced similar response lengths, all longer than ChatGPT-4o mini.³⁵ The shorter responses from ChatGPT-4o mini may result from its smaller, optimized architecture, designed to maximize efficiency and speed, leading to more concise output. Although shorter responses are generally expected to improve readability, ChatGPT-4o mini demonstrated the lowest readability scores despite producing the shortest responses. This finding suggests that readability was influenced more by linguistic complexity and technical terminology than by response length alone.

Study Limitations

This study has several limitations. First, we focused solely on three retinal diseases, thereby limiting the generalizability of the findings to other ophthalmologic or medical fields. Future research should be conducted across multiple specialties to enhance generalizability. Second, response appropriateness was assessed using a subjective rating system; however, 99.2% of responses were deemed appropriate or partially appropriate, supporting the reliability and validity of our evaluation method. While the subjectivity of the Likert scale is a concern, the use of two independent retina specialists and the attainment of a high ICC score enhanced test consistency. Additionally, our study design ensured that evaluators were blinded to the chatbot source, minimizing potential bias. However, in clinical practice, experts may interpret responses differently, highlighting the need for multidisciplinary validation of the appropriateness of LLMs. Another limitation is that LLMs do not provide references for their responses, making it impossible to validate the content using standardized quality assessment tools such as DISCERN or HONcode. The lack of citations poses challenges for patient education, as users cannot verify the reliability of chatbot-generated medical content. This is particularly concerning for patient decision-making and underscores the importance of reference-backed chatbot outputs in clinical applications. Furthermore, although established readability indices provide standardized measures of linguistic complexity, they do not directly assess patient comprehension or information retention. Therefore, readability scores should be interpreted as surrogate measures of text accessibility rather than direct indicators of patient comprehension. Lastly, LLMs are continuously updated, meaning that our findings reflect a specific time frame and may not remain consistent with future iterations. Consequently, the exact reproducibility of our findings may be limited as model architectures, training data, and response generation strategies continue to evolve. Given the rapid evolution of AI models, periodic reassessment is necessary to ensure the ongoing validity and clinical relevance of comparative evaluations.

CONCLUSION

ChatGPT-4o, ChatGPT-4o mini, and Gemini 1.5 Pro provided highly clinically appropriate responses for RD, ERM, and MH. However, ChatGPT-4o demonstrated superior readability, whereas ChatGPT-4o mini, despite its appropriateness, exhibited the lowest readability owing to its condensed, technical responses. These findings highlight that readability is as essential as appropriateness in patient education. Future research should focus on optimizing LLM-generated content for better readability and comprehension, ensuring their effective integration into medical education and clinical decision-making. Nevertheless, LLM-generated information should be considered a complementary educational resource and not replace professional ophthalmologic consultation or clinical judgment.