Automating fine needle aspiration and non-gynecologic quality control using a large language model: A real-world evaluation of diagnostic accuracy and efficiency

Human subjects and ethical approval

This study was conducted in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki) for experiments involving humans. All procedures were performed in compliance with relevant laws and institutional guidelines.

The proposed activity, as described in this manuscript, was reviewed by the Human Research Protection Program (Institutional Review Board [IRB] Study ID: STUDY00026725, Submission ID: STUDY00026725). The program determined that this activity does not meet the definition of human subject research as defined in 45 CFR 46.102(e) and/or (l). Therefore, IRB review and approval were not required.

All patient reports were de-identified at the time of data collection to ensure patient privacy. Given the de-identified nature of the data and the determination that the activity does not constitute human subject research, informed consent was not required. The privacy rights of human subjects were observed throughout the study.

Study design and dataset

This study evaluated the performance of an LLM in automating diagnostic categorization of cytopathology reports as part of a CLIA QC workflow. The current manual QC process in our laboratory serves as the operational baseline for this study, against which we compared the LLM-assisted workflow’s accuracy and efficiency.

Reports were retrieved from the Cerner Oracle LIS and represented all FNA and NG cytology cases accessioned between January 1^st and 31^st, 2024. Gynecologic (GYN) cytology reports were intentionally excluded from this analysis because their diagnostic categories are consistently captured in our LIS as discrete, categorical data (structured fields). Including these cases would have measured the extraction of pre-structured data rather than the interpretation of unstructured free text. We therefore focused on FNA and NG cases where NLP is operationally required in our workflow.

In total, 472 reports were included, comprising 149 FNA and 323 NG cases. All cases were included, and to simulate real-world conditions, there were no exclusion criteria.

Each report contained unstructured free-text fields authored by cytopathologists, including final diagnoses and cytologic findings. These text-based reports served as the input for the LLM pipeline.

Reports were de-identified at the time of data collection, and a local version of the model described below was deployed to ensure patient privacy.

LLM model and prompt design

We developed and tested an AI-assisted QC pipeline using the DeepSeek-R1:32B LLM for all experiments.^[10] This open-source LLM was locally deployed to process cytopathology reports and generate structured output. All inference and data processing were performed on a dedicated local workstation (Lenovo Legion 5 Pro) equipped with an AMD Ryzen 7 6800H (3.2 GHz), 32 GB RAM, and an NVIDIA GeForce RTX 3070 Ti Laptop GPU (NVIDIA, Santa Clara, California, USA), ensuring a consistent and controlled inference environment for all reports.

Each cytopathology report was submitted individually using a standardized prompt template that instructed the model to extract the following elements into JavaScript Object Notation (JSON) format:

• Final diagnosis.

• Diagnostic category (one of the five CLIA-mandated categories: Non-diagnostic, Benign, Atypical, Suspicious, or Malignant). Cases that do not fit these categories based on institutional convention were designated “Other” by the expert reviewer.

• Confidence score (see below).

For simplicity, we will abbreviate “Suspicious for Malignancy” as “Suspicious” for the remainder of the manuscript. Reports with ambiguous or low-confidence predictions were flagged for manual review to ensure diagnostic safety.

Model selection rationale

DeepSeek-R1:32B was selected because it provided a practical balance of performance and deployability for our study setting. At the time this project was initiated, it was among the strongest locally deployable open-weight models in its size class for instruction following, and it consistently adhered to a constrained output format, including reliable generation of valid JSON fields required by our pipeline. Local deployment was an explicit design requirement to support privacy adherence. The 32B parameter scale was compatible with our available workstation memory constraints, whereas larger models were not practical to run reliably in our environment. We note that the model landscape is evolving rapidly, and future work should evaluate newer lightweight models, including models with medical domain tuning, to assess whether comparable or improved accuracy and structured output reliability can be achieved with lower computational requirements.

Confidence score

The model was prompted to generate a self-assigned confidence score to quantify the certainty of its diagnostic category prediction. This was implemented through a structured JSON field within the prompt template: “diagnostic_category_confidence”: int

//This should be an integer between 0 and 10, where 10 denotes absolute certainty.

//Use this to indicate how certain you are that the diagnostic category is what is mentioned above.

The model’s confidence score was therefore self-reported, based on its internal language understanding rather than a probabilistic output. Scores ranged from 0 (no confidence) to 10 (complete confidence).

Predictions were considered low-confidence if the returned value was ≤4, and such cases were automatically flagged for manual review. Reports were also flagged if the model failed to output a valid diagnostic category (e.g., returned “Other” or “Not sure”), which indicated potential ambiguity or incompleteness in the source report.

This approach is consistent with previously published work using prompt-based confidence scoring for LLMs in clinical information extraction,^[7,11] where self-reported confidence values were used to triage uncertain cases for expert validation.

Pipeline implementation

A custom Python 3.11.5 (Python Software Foundation, Wilmington, Delaware, USA) script was created to automate the processing of cytopathology reports and generate a structured QC summary as shown in Figure 1. The pipeline performed the following steps:

Close

Figure 1: Quality control report generation artificial intelligence pipeline. LLM: Large language model, JSON: JavaScript Object Notation.

Export to PPT

1. Loaded each de-identified cytology report individually.

2. Submitted the report text, along with a standardized prompt, to the DeepSeek-R1:32B model.

3. Parsed the model’s JSON-formatted output, extracting the predicted final diagnosis, diagnostic category, and confidence score.

4. Compiled all structured results into a centralized CSV file.

5. Aggregated diagnostic data across all cases to generate a summary QC report for human review.

Reports that yielded incomplete, ambiguous, or low-confidence predictions (defined as a confidence score ≤4) were automatically flagged for manual review.

The final QC report included a table summarizing the predicted distribution of diagnostic categories across all cases. In addition to the AI-generated tallies, the table featured a column for cytotechnologists to manually record tallies from the flagged cases that they reviewed, and a final column that combined both the manual and AI-derived counts for submission as part of the official QC documentation.

Flagged cases were clearly identified within the QC report, and the full text of each flagged report was reprinted directly in the document. This allowed cytotechnologists or cytopathologists to quickly review these cases without needing to manually retrieve them from the LIS, streamlining the follow-up process and ensuring efficiency in manual validation.

Ground truth comparison and expert review

To evaluate diagnostic performance, all 472 LLM-processed reports were independently reviewed by a board-certified cytopathologist, who was blinded to the model’s outputs during initial review. The expert assigned a diagnostic category to each case based on the original text, which served as the ground truth for accuracy comparison.

After the initial review, a second validation step was conducted in which the same cytopathologist was presented with a custom report generated by the pipeline. This included the LLM’s predicted diagnostic category, associated confidence score, and metadata for each case. The goal of this second review was to assess the model’s output in context and flag any clinically concerning discrepancies.

Each LLM output was categorized into one of two performance classes based on the expert cytopathologist’s judgment:

• Correct: The model’s predicted diagnostic category was a direct match to the expert-assigned ground truth.

• Incorrect: The model’s predicted diagnostic category did not match the expert-assigned ground truth (raw misclassification).

The AI model was deemed to be ambiguous if it reported a confidence score ≤4 or if the returned category was anything other than the 5 standard diagnostic categories. These cases were flagged for manual review by the cytotechnologist.

Statistical analysis

Statistical analysis was performed using Python 3.11.5 (Python Software Foundation, Wilmington, Delaware, USA) with the following libraries: Pandas, numpy, scikit-learn (for confusion matrices), seaborn, and matplotlib. The AI system’s structured output was compared against the cytopathologist-reviewed ground truth to calculate diagnostic accuracy, which was reported overall and separately for FNA and NG cases. Confusion matrices were constructed to visualize misclassification patterns. Inter-rater agreement between the LLM and the expert ground truth was quantified using Cohen’s Kappa. The 95% confidence interval (CI) for the overall accuracy rate was calculated using the Wilson score interval method. In addition, technical time was measured for both manual and AI-assisted workflows to quantify the time savings of the automated workflow. The technical time for the AI-assisted workflow (LLM-pipeline) was calculated to be the sum of the time required for technical staff to manually review all flagged (low-confidence and ambiguous) cases and compute the final tallies.

Addressing diagnostic ambiguity

Most misclassified cases involved semantic ambiguity or missing clinical context, rather than purely syntactic complexity. For example, the model classified a multi-part specimen as Benign instead of Non-diagnostic due to institution-specific rules for handling insufficient parts, and a mucinous cyst as Benign instead of Atypical, missing an implied diagnostic concern. Other errors, such as labeling thymic cortical tissue as Suspicious instead of Atypical or a urine cytology case with squamous atypia as Benign instead of Atypical, reflect challenges in parsing nuanced diagnostic phrases or applying context-specific knowledge. Ambiguous cases often involve nuanced diagnostic phrases or implied intent, which can also pose difficulties for human reviewers, as seen in prior NLP studies.^[18] Further improvements could likely be achieved by refining the prompt to better flag ambiguous cases for review and to ensure that diagnostic classification is performed at the specimen-part level rather than the case level.

Operational benefits and workflow enhancement

Manual cytopathology QC is time-intensive and demands skilled personnel. This study shows that an LLM-based pipeline can substantially alleviate the workload by automating initial categorization and prioritizing ambiguous reports for expert review. Rather than replacing human reviewers, the model enhances their effectiveness by ensuring that attention is focused where it is most needed.

The model also provides structured outputs (e.g., CSV [comma-separated value]-formatted tables) that are compatible with existing QC workflows, potentially increasing consistency and reducing data-entry errors. Further, this approach could be extended to other data-driven QC tasks in cytopathology, such as tracking diagnostic category distributions by provider, identifying cases for retrospective review, or correlating cytology results with surgical pathology follow-up. This aligns with prior studies showing that AI-based tools can reduce review time, prioritize high-risk cases, and support more efficient pathology workflows.^[19,20]

The alternative to using LLMs for report categorization is to employ traditional NLP methods such as regular expression searches, keyword rules, or handcrafted decision trees. While these approaches can perform well when report language is highly standardized, they are brittle in the presence of negation, hedging language, multi-specimen structure, and stylistic variation across pathologists. For example, if not designed carefully, simple keyword matching can misclassify phrases such as “negative for high-grade carcinoma” because it contains a malignant keyword. Even carefully crafted rule sets require ongoing maintenance to adequately handle new data. In contrast, the LLM used in this study performed semantic interpretation of the report text and produced structured outputs in a fixed JSON schema, enabling robust downstream parsing with minimal rule engineering. For this QC task, the key operational advantage is that a single prompt-based pipeline can generalize across heterogeneous free text patterns without the need to construct and continuously update an extensive set of institution-specific rules.

Indirect educational and diagnostic improvement value

While the primary objective of this study was to demonstrate efficiency and accuracy for CLIA compliance, the resulting AI-assisted QC pipeline establishes a robust infrastructure for continuous performance improvement. By automatically and reliably categorizing every case, the system provides the fundamental data required to track and benchmark the diagnostic categorical distributions of individual cytopathologists and cytotechnologists. More importantly, the system can be adapted to systematically track inter-observer disagreements between the initial cytotechnologist screening and the final cytopathologist sign-out. This capability would allow the laboratory to easily identify and flag cases exhibiting significant diagnostic category changes (e.g., Benign to Malignant). These flagged, high-impact discrepancy cases can then be prioritized for retrospective review, which serves as a powerful, targeted tool for staff education and quality assurance, thereby indirectly leading to long-term improvements in diagnostic performance and consistency within the laboratory.

Deployment consistency and safety guardrails

To ensure reliable performance within our operational environment, we controlled for potential sources of LLM variability. Specifically, deploying a single, open-source model (DeepSeek-R1:32B) locally with a fixed prompt template provided a consistent, verifiable inference environment. This controlled design ensures that the model’s performance is stable within our environment, though continuous monitoring for drift will be essential upon deployment. Crucially, the model was designed with a builtin safety mechanism: The self-assigned confidence score. Cases that yielded low confidence or ambiguous output were automatically flagged for human intervention. This human-in-the-loop strategy acts as a critical guardrail, ensuring that any rare, non-reproducible misclassification by the LLM is intercepted and corrected by an expert, thereby maintaining diagnostic integrity while achieving operational efficiency.

Scalability and future directions

As pathology laboratories face increasing case volumes and workforce shortages,^[3] tools like this AI QC pipeline offer a scalable solution to maintain quality while improving efficiency. Although this study focused on cytopathology, similar approaches could be adapted for other subspecialties such as histopathology, surgical pathology, hematopathology, and in medical practice outside of the laboratory, where structured extraction from unstructured text is similarly valuable. In all such settings, careful attention must be made to ensure that the pipeline remains compliant with the Health Insurance Portability and Accountability Act or other local information security regulatory requirements. Possible solutions include employing local AI infrastructure or obtaining a business associate agreement to use a compliant cloud provider. The use of DeepSeek-R1:32B, an open-source model, was specifically identified to facilitate local deployment; however, other open-source LLMs could be considered.

To prepare this system for broader clinical deployment, further validation across diverse datasets, institution-specific prompt refinement, and ongoing performance monitoring will be essential. Future studies comparing model performance with human inter-observer consistency would provide additional context for interpreting model accuracy. Incorporating a human-in-the-loop approach, where technical staff, pathologists, and other physicians remain in control but are supported by AI, strikes the right balance between safety and efficiency.

Although this was a single institution evaluation, the pipeline is designed to be portable. It operates on report text alone, and the task is defined entirely in the prompt template, which can be adjusted to match local reporting conventions or category mappings without changing the core codebase. We did not have access to external multi-institution datasets and therefore did not directly assess cross-site generalizability. However, the validation approach used here, including expert adjudication, structured output checks, and manual review of low confidence or non-standard outputs, can be applied by other laboratories to evaluate performance on their own data before use.

Limitations

This study has several limitations that should be considered. First, it represents a single-center evaluation with ground truth established by a single board-certified cytopathologist. While this is consistent with internal laboratory QC practices, future multi-center studies involving multiple expert reviewers are needed to further establish the generalizability and reproducibility of the model’s performance. We further acknowledge that pathological diagnosis inherently involves subjectivity, as does the interpretation of pathology notes in cases where the diagnostic category is not explicitly stated. Inclusion of multiple expert reviewers would provide a more comprehensive assessment of model performance. Furthermore, because model performance relies on prompt engineering, its effectiveness at other sites may vary due to differences in terminology and report structure. However, the core framework presented, including local deployment, confidence-based filtering, and a human-in-the-loop pipeline, is highly portable. Other laboratories can adapt this method by validating and fine-tuning prompts using their local data before deployment. Second, the study’s scope was intentionally restricted to FNA and NG cytology reports for which manual NLP is currently required. The model’s performance on GYN specimens was not assessed. Finally, this evaluation focused strictly on the model’s NLP accuracy in correctly categorizing the finalized report text. We did not evaluate correlation with subsequent surgical pathology results, as our objective was to support CLIA regulatory QC requirements.