Impact of the Paris system for reporting urine cytopathology on predictive values of the equivocal diagnostic categories and interobserver agreement

INTRODUCTION

Urine cytology, much like the gynecologic screening Pap test prior to The Bethesda System in 1988 and thyroid fine-needle aspirations before 2007, suffers from a variable, nonstandardized, and often equivocal terminology. Many classification systems have been proposed since Papanicolaou's proposal in 1947 in an attempt to standardize the diagnostic terminology; however, none have gained widespread implementation. Urine cytology detects high-grade urothelial carcinoma (HGUC) with high specificity and is still the best and most reliable screening test for this condition. The Paris System (TPS) was developed on this principle, essentially to prioritize the detection of high-grade urothelial lesions and standardize the diagnostic terminology based on the underlying histopathology and clinical outcomes, in hopes of minimizing the use of equivocal terms such as “atypical” and “suspicious,” thus facilitating clinical decision-making.[1] For TPS to be widely implemented and for it to endure the test of time, its diagnostic utility and clinical effectiveness should be continuously validated. Since TPS was published in 2016, a number of institutional studies have been published, studying the application of all or some of TPS diagnostic criteria, some with conflicting results.[2345678910] Fewer studies have assessed interobserver agreement.[1112]

In this study, in addition to assessing the interobserver reproducibility (precision), we studied the overall predictive value and detection rate of biopsy-proven high-grade malignancy of TPS in comparison to the traditional method (TM). Predictive value or risk of malignancy by TPS diagnostic category was also studied. The impact of misclassification on clinical management was evaluated. TPS, its sensitivity, specificity, and accuracy for detecting HGUC in comparison to the TM were studied. Finally, a survey was conducted at the end of this study to evaluate the practicality of using TPS by the six participating cytopathologists.

MATERIALS AND METHODS

The study design was reviewed by the Institutional Review Board at Cedars-Sinai Medical Center (CSMC) for compliance with the hospital's national and international standards and designated as “exempt.” An extensive review of the extant literature on urine cytopathology and TPS was conducted. The pathology laboratory database (PowerPath, Sunquest Information Systems, Tucson, AZ, USA) was queried for cytopathology urine samples with concurrent or follow-up surgical biopsies with malignant (HGUC) or benign (non-HGUC) diagnoses performed within a period of up to 3 months. Only cases satisfactory for cytologic evaluation with well-fixed, well-prepared, liquid-based preparations were accepted for inclusion in this study. One hundred urinary cytology samples from 100 patients obtained over 5 years (2013–2017) were selected. All specimens were Papanicolaou stained SurePath^® preparations. Each cytology case was independently reviewed by six cytopathologists from the Department of Pathology at CSMC, blinded to the previous cytology and surgical pathology diagnoses and the diagnoses of the other participating cytopathologists. The cytopathologists had between 6 and 25 years' experience with interpreting urine cytopathology at the time of review of study cases. Each cytopathologist rendered a diagnosis according to five TPS categories: negative for HGUC (NHGUC), atypical urothelial cells (AUC), low-grade urothelial neoplasm (LGUN), suspicious for HGUC (SHGUC), and HGUC [Figure 1].[13]

Close

Figure 1

The Paris System categories. Negative for high-grade urothelial carcinoma (a), atypical urothelial cells (b), suspicious for high-grade urothelial carcinoma (c), and high-grade urothelial carcinoma (d)

Export to PPT

The six review cytopathologists had not been involved in the development of TPS; however, each cytopathologist received training on its application (one Grand Rounds presentation by Dr. Guliz Barkan, one College of American Pathologists webinar, two internal PowerPoint presentations, TPS textbook, and a handout summarizing TPS criteria). The nuclear–cytoplasmic (N/C) ratio was estimated visually using the definitions published by TPS. The category LGUN was recognized as a subcategory of NHGUC and was only used where papillary groups of urothelial cells were present with well-defined fibrovascular cores.

The agreement was assessed using kappa. Disagreements were classified as high and low impacts depending on the potential impact of a misclassification on clinical management. High impact was defined as disagreement between categories SHGUC or HGUC versus categories NHGUC, LGUN and AUC, whereas low impact was defined as disagreement among categories NHGUC, LGUN, and AUC. Sensitivity and specificity were assessed by dichotomizing the traditional and TPS diagnostic categories as benign or malignant among majority consensus agreement and comparing them to concurrent surgical biopsies. The decision to include the atypical and low-grade categories in both TM and TPS in the dichotomized benign category was based on what is currently hypothesized about the pathogenetic bases of urothelial carcinoma.[13] The biopsy diagnoses were used as the gold standard and were classified as “biopsy-proven HGUC” and “biopsy-proven non-HGUC,” the latter including biopsy-proven benign and low-grade urothelial carcinoma cases.

A consensus diagnosis for TPS among the six cytopathologists was obtained by the majority consensus diagnosis for each case. The consensus TPS diagnoses were classified as “benign,” a category that included NHGUC, LGUN, and AUC, versus “malignant,” a category that included SHGUC and HGUC.

The TM diagnoses were classified as “malignant,” a category that included HGUC and SHGUC, versus “benign,” a category that included no malignant cells identified, atypical cells present, and low-grade urothelial carcinoma.

At the end of the review, a survey was conducted to assess the ease of implementing the criteria detailed in TPS, its practical benefit, and overall satisfaction.

RESULTS

Of the 100 cases reviewed, 70 were voided and 30 were instrumented (washings/catheterized). They included 26 (26%) negative for malignant cells, 44 (44%) atypical, 11 (11%) suspicious for HGUC, and 19 (19%) HGUC cases diagnosed by TM. Fifty-three percent were concurrent surgical biopsy-proven HGUC and 47 (47%) were biopsy-proven non-HGUC (31 benign and 16 low-grade urothelial carcinoma by surgical biopsy). TPS diagnoses were 54 (54%) NHGUC, 23 (23%) AUC, 9 (9%) SHGUC, and 14 (14%) HGUC. None of the cases were cytologically diagnosed as LGUN or low-grade urothelial carcinoma or other malignancy by either system. There was a perfect agreement among 31% of the cases, with an overall kappa = 0.362 (P < 0.0001, 95% CI: 0.331–0.394). The overall kappa was 0.411 after collapsing the SHGUC and HGUC into one diagnostic category. Among cases with disagreements, 17% (12/69 cases) fell under disagreements with high clinical impact and 65% (45/69 cases) fell under disagreements with low clinical impact. Kappa values by TPS diagnostic category were only significant for the NHGUC category (kappa = 0.483, P < 0.0001, 95% CI: 0.432–0.533) and the HGUC category (kappa = 0.520, P < 0.0001, 95% CI: 0.470–0.571) [Table 1].

Both TM and TPS showed high specificity (100% for both) and high positive predictive value (PPV) (100% for both) for detecting HGUC. TPS showed slightly higher sensitivity at 43% (compared to 38% for TM) and accuracy at 70% (compared to 66% for TM). The detection rate of biopsy-proven HGUC was higher by TM at 57% versus 43% by TPS [Table 2].

Out of the 54 cases classified as NHGUC by TPS, 16 (30%) were biopsy-proven HGUC and 38 (70%) were biopsy-proven non-HGUC. The 16 biopsy-proven HGUC cases were classified as 3 negatives, 12 atypical, and 1 suspicious by TM. Of the 23 cases diagnosed as AUC by TPS, 14 (61%) were biopsy-proven HGUC and 9 (39%) were biopsy-proven non-HGUC. All the 9 cases diagnosed as SHGUC and all the 14 cases diagnosed as HGUC (100% each) by TPS were biopsy-proven HGUC [Table 3].

Fifty-four percent (54/100) of the samples were diagnosed as NHGUC by TPS compared to 26% (26/100) diagnosed as negative for malignant cells by TM (P < 0.0001). In TM, 44% (44/100) of the samples were diagnosed as “atypical cells present” versus 23% (23/100) diagnosed as AUC by TPS (P < 0.0001). No significant difference was noted in the number of cases diagnosed as suspicious by TM (11%, 11/100) versus TPS (9% (9/100) or the number of cases diagnosed as HGUC by TM (19%, 19/100) versus TPS (14%, 14/100), although a decrease in the rate of both categories using TPS is observed [Table 4].

Sixteen (30%) cases diagnosed as NHGUC by TPS were concurrent biopsy-proven HGUC. Three out of those 16 cases were also called negative by TM, 12 were called atypical, and 1 case was called suspicious. All the 16 cases were re-reviewed by the first and last authors (RB and SB). The NHGUC diagnosis was confirmed in 11/16 cases (true negative), whereas five were reclassified as atypical. Eight out of the 11 confirmed negative cases either had clusters of small cells (bland) in voided urine or clusters of basal urothelial cells (washings). The discrepancy in diagnosis of the remaining 3/11 cases (called negative by both TPS and TM) compared to biopsy was attributed to sampling effect. Reclassification of 5/16 cases from NHGUC to atypical after re-review was due to the presence of either rare cell with an only slight increase in N/C ratio or focal subtle atypical features in a background of either inflammation, crystals, or degenerative changes. Figure 2a–d shows the details on subtle atypical findings from each case.

Close

Figure 2

Focal subtle atypia in biopsy-proven high-grade urothelial carcinoma cases with initial negative high-grade urothelial carcinoma, (a) acute inflammation, crystals, reactive cells (curved arrow). Nuclear–cytoplasmic ratio >0.5, hyperchromasia, membrane irregularity (arrow), (b) acute inflammation, reactive cells (curved arrow). Nuclear–cytoplasmic ratio >0.5, hyperchromasia, nuclear membrane irregularity (arrow), (c) nuclear–cytoplasmic ratio >0.5, nuclear irregularity (arrow). Enlarged nuclei, membrane irregularity, nuclear–cytoplasmic ratio <0.5 (arrowhead), and (d) crystals, acute inflammation, nuclear hyperchromasia, irregularity (arrow). Degenerated cells with irregular nuclei, nuclear–cytoplasmic ratio ≤ 0.5 (arrowhead)

Export to PPT

Sixty-seven percent of participating cytopathologists indicated that TPS has influenced their clinical practice or helped improve their diagnostic skills and decreased the rate of AUC diagnosis in the survey taken. The overall satisfaction with TPS was 33%. Challenges that were listed by the cytopathologists included strictly applying the N/C ratio criteria. The majority found that N/C ratio cutoffs of 0.5 versus 0.4 to differentiate between AUC versus NHGUC and N/C cutoffs of 0.5 and 0.6 to differentiate between AUC versus SHGUC were quite challenging. Severe nuclear hyperchromasia and N/C ratio >0.7 as requirements to call SHGUC or HGUC were the limiting factors in some cases that met all other criteria listed in these categories [Figure 3]. Survey results are summarized in Table 5.

Close

Figure 3

Biopsy-proven high-grade urothelial carcinoma not meeting required criteria by The Paris System of nuclear–cytoplasmic ratio >0.7 (a) and severe nuclear hyperchromasia (b)

Export to PPT

DISCUSSION

The clinical utility of any classification system depends on both its precision (reproducibility) and its diagnostic accuracy. The overall agreement by TPS was fair, with a kappa value of 0.362, which is close to the overall kappa value of 0.32 previously reported by Long et al.[11] TPS demonstrated good precision among NHGUC and HGUC diagnostic categories, whereas other equivocal categories (AUC and SHGUC) demonstrated poor interobserver variability. Similar findings were recently reported by a web-based interobserver study, cosponsored by the American Society of Cytopathology (ASC) and International Academy of Cytology that agreements were highest among the NHGUC and HGUC categories and low in equivocal categories, in concordance with our results.[14] High clinical impact disagreement between the participating cytopathologists occurred in 17% of the study cases, a rate comparable to that recently reported (15%).[11] Both TPS and TM showed comparable sensitivity, specificity, positive and negative predictive values, and accuracy, although TPS showed slightly higher sensitivity and accuracy.

No significant difference in the rate of SHGUC and HGUC categories in both TPS and TM was detected in our study. Differences were significant among the negative and atypical categories. The atypical category has been challenging for both pathologists and clinicians ever since diagnostic systems were designed for reporting urine cytopathology. The definition of the inclusion criteria to use the term “atypical” was often vague and confusing in prior classifications. The frequency of use of the term “atypical” has widely ranged from 2% to 31% in the reported literature. The high frequency of using the atypical category often resulted in over investigation and excessively prolonged surveillance.[15] Furthermore, the risk of urothelial carcinoma associated with the term has varied from 8.3% to 37.5%, a wide range of variation that has only rendered this category almost clinically meaningless.[15161718192021]

An important goal for TPS was to limit the use of AUC diagnosis to cells with atypical features that are more likely to be associated with HGUC. Our results show that TPS is successful in reducing the atypical category. We observed a significant decrease in the rate of atypical diagnosis using TPS (23%) as compared to TM (44%). Recently, published studies have reported TPS AUC rates varying from 22 to 31%.[5111722] In our study, there is a significant difference in the rate of atypical diagnosis after applying TPS, where a total of 64% (28/44) of the cases that were diagnosed as “atypical cells present” by TM were then reclassified as NHGUC by TPS. In other words, TPS has decreased the rate of atypical diagnosis in benign samples by 28%. The reduction in our AUC diagnosis was associated with a significant increase in the rate of NHGUC (54% by TPS vs. 26% by TM). Survey of the study participants suggested that this resulted from the application of well-defined criteria and the change in the terminology from “negative for malignant cells” to the more specific terminology of “negative for HGUC,” thus decreasing the apprehension of missing low-grade carcinoma. In addition, the recommendation to diagnose samples with clusters of small cells and tissue fragments as negative, if no nuclear atypia is present by TPS, also helped reduce the AUC diagnosis. This shift from the AUC category to the NHGUC has also been observed in other studies.[52324252627] Recently published studies have shown that PPV of the AUC category for HGUC has increased after applying TPS (43%–83% in one study and 28%–46% in another).[2325] In our study, the PPV of the AUC category by TPS is 61% (14 of 23 were biopsy-proven HGUC) versus 43% for the atypical category by TM (19 of 44 were biopsy-proven HGUC). In addition, the relative risk (RR) of a sample ending up being biopsy-proven HGUC when called AUC by TPS is twice the risk of it being biopsy-proven non-HGUC when the sample is called NHGUC (RR = 2.054). Therefore, we can conclude that TPS has increased the clinical meaningfulness or the likelihood that the AUC diagnosis represents true HGUC cases. This can eventually result in facilitating clinical decision-making and fewer patients referred to unnecessary cystoscopy procedures, improving the rate of HGUC detection by cystoscopy.[28]

An increased risk of malignancy associated with the NHGUC is observed in our study, where 16 (30%) cases diagnosed as NHGUC by TPS were concurrent biopsy-proven HGUC. Re-review and reclassification of these cases to AUC after observing either rare cells with only slight increase in N/C ratio or focal subtle atypical features in a background of either inflammation, crystals, or degenerative changes resulted in a decrease in risk of malignancy associated with NHGUC category from 30% to 20% (11/54). While 20% risk of malignancy is still high, it has been reported to range from 13% to 37% by other prospective and retrospective studies conducted to evaluate urine samples pre- and post-TPS.[1224] It seems that strictly following the N/C ratio and nuclear hyperchromasia criteria or dismissing subtle atypical features if there is a background of inflammation/degeneration or crystals has resulted in underdiagnosing some of the cases that were biopsy-proven HGUC in this study. This decrease in the detection rate of HGUC after applying TPS was also observed in one other recently published study.[27] We hypothesize that this decrease is the result of three main challenges: the N/C ratio criteria, nuclear hyperchromasia as a required criterion for atypia of high-grade malignancy in the presence of other high-grade criteria, and the less defined guidelines for assessment of samples with degeneration or obscuring inflammatory background/crystals. The clinical impact of decreasing the rate of AUC with a secondary increase in the rate of NHGUC and a potential decrease in the rate of detection of HGUC by TPS needs to be further investigated in large prospective studies.

The diagnostic challenges observed in our study have resulted in an extensive search of the relevant existing published literature. Layfield et al. demonstrated suboptimal interobserver reproducibility of estimated of N/C ratio and Zhang et al. demonstrated that morphologists overestimated the N/C ratio.[2930] A recent study by Cowan et al. found limitations regarding N/C ratio >0.7 and number of cells meeting all HGUC criteria to qualify for this diagnosis, despite improved risk stratification.[3] Another study found that N/C ratio was predictive of malignancy only if observed in single cells, whereas other high-grade criteria such as an irregular nuclear membrane, hyperchromasia, and coarse granular chromatin were predictive in both single cells and groups. Coarse chromatin alone was more specific than other individual features.[42631] Digital image analysis has been shown to support N/C ratio cutoff of 0.5 as a criterion for AUC.[32] A recently published study utilizing digital image analysis found that the average N/C ratio for HGUC was 0.57 and for SHGUC was 0.53, proposing reduction of the N/C ratio criterion to a cutoff below 0.7.[33] HGUC cells have been shown to have different cytologic morphologies, where cells can have atypical or malignant nuclei but with abundant cytoplasm, mimicking umbrella cells, in up to 17% of samples.[34] Finally, degeneration/poor preservation and obscuring inflammation were implicated in 50% of false-negative urine samples.[35] A comprehensive approach to looking at the nuclear criteria proposed by TPS in light of these findings is needed to increase the diagnostic accuracy and detection rate of HGUC.

Hundred percent of the study cases diagnosed as SHGUC by TPS were surgically biopsy-proven HGUC. A similar rate was previously reported in one other study assessing TPS.[36] This raises the question of whether SHGUC should remain a separate category from HGUC. This may also have implications on the interobserver agreement. This prompted re-assessment of interobserver agreement in our study after collapsing the SHGUC and HGUC into one diagnostic category. The overall agreement slightly increased as a result to 0.411. Of note, all samples from our study were collected from the lower urinary tract, where the clinical implication of these two categories might not be as significant as that of samples obtained from the upper urinary tract. Additional large studies with clinical follow-up are needed to further investigate the impact of combining those two categories.

Our end of study survey showed that all cytopathologists agreed that the use of NHGUC terminology proposed by the TPS instead of the general terminology of “negative for malignant cells” traditionally used for that category was quite helpful in focusing on the detection of high-grade lesions. In addition, most of the participating cytopathologists found that the application of TPS was practical and helpful in decreasing the rate of atypical diagnosis, a category that was frequently used as a wastebasket in many cases and has lost its clinical usefulness as a result. Fifty percent found the cell number cutoff of urothelial cells with severe atypia, proposed by TPS as one of the criteria to call HGUC versus SHGUC, helpful. With that said, the overall satisfaction with TPS was 33%, which was mainly attributed to challenges resulting from strictly applying the N/C ratio criteria, especially to differentiate between NHGUC, AUC, and SHGUC, where the N/C ratio differed minimally from the cutoff criteria proposed by TPS. In addition, severe nuclear hyperchromasia and N/C ratio >0.7 as strict requirements to call HGUC were the limiting factors in some cases that met all other criteria listed in those categories. These limitations have resulted in disqualifying some cases that contain overtly malignant cells from HGUC diagnosis.

Our study is limited by the number of cases studied, its retrospective nature, and the fact that only cytology samples with concurrent surgical biopsy were included. Therefore, the reported percentage of each TPS category may not reflect all the cytology samples seen in our laboratory. While we have recently implemented TPS in our laboratory and intend to continue to report on its performance in prospective studies, multiple large prospective studies from different institutions are needed to further refine the diagnostic criteria of TPS, to define the risk related to TPS initial diagnostic categories, and to establish clear-cut clinical management guidelines based on clinical outcomes, with collaborations with urologists to include their valuable feedback.

CONCLUSIONS

The remarkable decrease in the rate of atypical diagnosis and the increased clinical significance of this category as demonstrated in our study is evidence that TPS has improved actionable clinical decision-making. While TPS shows adequate precision for categories NHGUC and HGUC, the interobserver variability for other equivocal categories remains low. Further refinement and widespread application of TPS diagnostic criteria for the atypical and malignant categories may result in higher detection rate of HGUC and improved interobserver agreement.

COMPETING INTERESTS STATEMENT BY ALL AUTHORS

The authors declare that they have no competing interests.

AUTHORSHIP STATEMENT BY ALL AUTHORS

All authors of this article declare that we qualify for authorship as defined by ICMJE.

Each author has participated sufficiently in the work and takes public responsibility.

For appropriate portions of the content of this article, each author acknowledges that this final version was read and approved.

ETHICS STATEMENT BY ALL AUTHORS

This study was conducted with approval from the Institutional Review Board of all the institutions associated with this study as applicable.

LIST OF ABBREVIATIONS (In alphabetic order)

AUC: Atypical urothelial cells

CAP: College of American Pathologists

CI: Confidence intervals

HGUC: High-grade urothelial carcinoma

LGUN: Low-grade urothelial neoplasm

NHGUC: Negative for high-grade urothelial carcinoma

N/C ratio: Nuclear to cytoplasmic ratio

PPV: Positive predictive value

SHGUC: Suspicious for high-grade urothelial carcinoma

TPS: The Paris System

TM: Traditional Method.

EDITORIAL/PEER-REVIEW STATEMENT

To ensure the integrity and highest quality of CytoJournal publications, the review process of this manuscript was conducted under a double-blind model (the authors are blinded for reviewers and vice versa) through automatic online system.