Interobserver reproducibility of The Paris System for Reporting Urinary Cytology

INTRODUCTION

The diagnosis of urothelial carcinoma and posttherapy follow-up requires examination of either histologic biopsy specimens or cytologic examination of voided urine or instrumented urinary tract specimens. The diagnosis of urothelial carcinoma by cytologic methods can be challenging, especially the recognition of low-grade urothelial carcinomas. A number of diagnostic approaches and classification schemes have been proposed over the last seven decades.[12345] Diagnostic classifications and criteria have been proposed by Papanicolaou,[1] Koss et al.,[2] Murphy et al.,[3] Ooms and Veldhuizer,[4] and The Papanicolaou Society of Cytopathology.[5] All of these systems have been highly successful for the recognition of high-grade urothelial carcinomas (HGUCs) but have had low sensitivity for the recognition of low-grade urothelial carcinomas.[12345] A number of technologies have been brought to bear to improve diagnostic sensitivity of urinary cytology,[67] but the success of these methodologies has often been insufficient to justify their cost. Fluorescence in situ hybridization analysis has been one of the more popular, but some studies have shown that its sensitivity for low-grade papillary urothelial carcinoma is not significantly superior to that of cytology alone.[7]

The recently proposed Paris System for Reporting Urinary Cytology advocates a classification to improve the sensitivity and specificity for the diagnosis of HGUC.[8] This system utilizes seven categories designated: (1) inadequate/less than optimal adequacy; (2) negative for high-grade urothelial carcinoma (NHGUC); (3) atypical urothelial cells (AUCs); (4) low-grade urothelial neoplasm (LGUN); (5) suspicious for high-grade urothelial carcinoma (SHGUC); (6) HGUC; and (7) other malignancies primary and metastatic. Each of these categories is well defined by specific criteria and is associated with a known risk for malignancy. The Paris System for Reporting Urinary Cytology also suggested management options for each diagnostic category.[8]

Currently, little published data exist documenting the interobserver reproducibility of these categories. For clinical utility, a categorization scheme must be both accurate and precise. We investigated the interobserver reproducibility of five categories used in The Paris System for Reporting Urinary Cytology. The analysis was performed by four cytopathologists who had not participated in the development of The Paris System for Reporting Urinary Cytology. Herein, we report the results of our reproducibility study for evaluation of precision of the Paris System.

METHODS

The study design was reviewed by the Institutional Review Board at the University of Missouri for compliance with university, national, and international standards. The Institutional Review Board designated the study as “exempt.” Three hundred and fifty-seven urinary cytology specimens (328 voided urines, 13 catheterized urines, 10 ureteral brushings and washings, 4 obtained during cystoscopy, and 2 obtained from the kidney) obtained over a 10-year period were selected for the study. Only cases with well-fixed, well-prepared, liquid-based preparations were chosen for inclusion in the study. The majority of cases were voided urines. All specimens were Papanicolaou-stained ThinPrep^® preparations. Each case was reviewed independently by four cytopathologists unaware of the previous diagnoses and unaware of the diagnoses given by other cytopathologists participating in the study. The cytopathologists had between 6 and 25 years’ experience with interpreting urinary cytology. The categories used for assignment were those of the Paris System and included: unsatisfactory, NHGUC, AUC, LGUN, SHGUC, HGUC, and other malignancies.[8] The four review cytopathologists had not been involved in the development of The Paris System for Reporting Urinary cytology, but each cytopathologist read the monograph entitled, The Paris System for Reporting Urinary Cytology and applied the criteria as outlined in the relevant chapters (3, 4, 5, 6, and 7).[8] The nuclear-cytoplasmic (N/C) ratio was estimated visually using the definitions proffered in the Paris System monograph [Table 1 for criteria used]. One of the cytopathologists had attended lectures at national meetings outlining the Paris System. The category LGUN was recognized as a subcategory of NHGUC and was only used where papillary groups of urothelial cells were present with well-defined fibrovascular cores.

Table 1 Criteria for assignment to the diagnostic categories of The Paris System for Reporting Urinary Cytology

Overall agreement was assessed using absolute agreement and weighted chance-corrected agreement (kappa). For weighted kappa, concordant results were given full credit (1) and discordant results were given half credit if the discordance was off by a single category [Table 2]. For kappa calculations, the categories were ordered as follows: NHGUC greater than AUC greater than LGUN greater than SHGUC greater than HGUC. The categories of “unsatisfactory” and “other malignancy” were excluded from the agreement analysis. The overall average absolute agreement and expected agreement were calculated by determining the agreement between each pair of observers and calculating the weighted average (each pair had a different number of cases due to exclusion of cases classified as unsatisfactory or other). Statistical calculations were performed using Stata 14 (StataCorp., College Station, Texas, USA). Weighted averages were calculated using the MetaProp command. The average kappa statistics were calculated using the Kapci command.

Table 2 Weighting scheme for agreement analysis

To further characterize agreement, we investigated the reliability of specific rating categories. To that end, we tabulated the results of all the rater comparisons and determined the frequency of each possible combination of ratings (NHGUC vs. AUC, NHGUC vs. SHGUC, etc.). We categorized each combination as high clinical impact, low clinical impact, or no impact according to the potential impact of a misclassification on clinical management. We determined the reliability of individual categories by determining the conditional probability of a disagreement by a second rater, given a particular rating by the first rater.

RESULTS AND DISCUSSION

The diagnoses were unevenly distributed across categories [Table 3]. NHGUC (44.5%) and AUC (22%) were the most common diagnoses. Of 1902 comparisons between rater pairs, 999 (52.5%) showed complete agreement, 451 (23.7%) differed by a single category, 228 (12%) differed by two categories, 199 (10.5%) differed by three categories, and 25 (1.3%) differed by four categories. Using the impact categories in Table 4, 14.7% of the disagreements were considered “high clinical impact.” The average absolute agreement was 65% (95% confidence interval: 63–67). The average expected agreement (due to chance alone) was 44% (95% confidence interval: 42–46). The average chance-corrected agreement (kappa) was 0.32 (95% confidence interval: 0.28–0.32).

Table 3 Tabulated scores for each observer

Table 4 Overall agreement between all pairs of observers

The agreement by category is shown in Table 5. Raters had the highest agreement (63%) on NHGUC. Raters showed poorer agreement on the other categories. For example, given a diagnosis of SHGUC by one rater, the chance of agreement was only 13% and the second diagnoses were evenly distributed over the other categories. Figures 1–4 show examples of specimens with poor interobserver agreement.

Table 5 The conditional probability of agreement

Close

Figure 1

Photomicrograph of representative field from a case evaluated as “atypical urothelial cell” by two reviewers, suspicious for high-grade urothelial carcinoma by one observer, and high-grade urothelial carcinoma by the final reviewer (Papanicolaou, ×600)

Export to PPT

Close

Figure 2

Photomicrograph of a specimen designated as atypical urothelial cell by two reviewers and as suspicious for high-grade urothelial carcinoma by two reviewers

Export to PPT

Close

Figure 3

Photomicrograph of a specimen designated as atypical urothelial cell by three observers and as negative for high-grade urothelial carcinoma by one observer

Export to PPT

Close

Figure 4

Photomicrograph of a specimen designated as atypical urothelial cell by two observers, as suspicious for high-grade urothelial carcinoma by a single observer, and as malignant by the final observer

Export to PPT

Cytologic analysis of specimens obtained from the urinary tract remains a cornerstone for the diagnosis and surveillance of patients with urothelial carcinoma. The sensitivity and specificity of cytologic analysis are excellent for HGUCs. Cytologic analysis is accurate for both superficial high-grade lesions as well as those extending deeper into the bladder wall. Unfortunately, the sensitivity of cytologic study for the diagnosis of LGUNs is significantly poorer.[910] In addition, noncarcinomatous papillary lesions exist (papilloma and papillary urothelial lesion of low malignant potential) complicating the cytologic diagnosis of low-grade urothelial carcinomas. The Papanicolaou Society of Cytopathology Task Force recommendations acknowledged this issue and included the category “atypical urothelial cells.” More recently, a number of investigators have studied the clinical significance of the atypical category and attempted to define useful morphologic criteria to further stratify malignancy risk. Published data are inconclusive as to whether specimens designated “atypical” have increased risk over those designated “negative.”[1112] Cytologic features have been identified which correlate with increased risk for low-grade urothelial carcinoma.[13]

Previously, many cytopathologists subdivide the atypical category into “atypical, suspicious for malignancy” and “atypical, favor reactive.”[1112] Such a subdivision may increase diagnostic accuracy, but the malignancy risk of these two categories and their reproducibility are incompletely understood. The relationship between the category “atypical suspicious for malignancy” and the presence of low-grade or HGUC is also poorly defined.

Recently, The Paris System for Reporting Urinary Cytology was developed to improve diagnostic accuracy and clarify the usage and implications of the atypical category or categories.[8] This system effectively has two atypical categories named as: (1) AUCs and (2) SHGUC. The categories AUCs and SHGUC are defined by cytologic features including the N/C ratio. Recently, Layfield et al.[14] demonstrated imperfect interobserver reproducibility of estimates of N/C ratio, and Zhang et al.[15] demonstrated that morphologists overestimated the N/C ratio. The category LGUN is well defined by both cellular and architectural features. The category LGUN should have limited usage and is considered a subcategory of NHGUC. While the categories of The Paris System have good estimates of malignancy risk, we are unaware of any study documenting the interobserver reproducibility of these categories. Such data are needed to estimate the precision of The Paris System for Reporting Urinary Cytology.

Our study was designed to evaluate diagnostic precision not diagnostic accuracy of The Paris System for Reporting Urinary Cytology, and thus, follow-up histologic diagnoses were not necessary. We investigated the reproducibility of category assignment by four cytopathologists reviewing 357 specimens. The clinical utility of any classification system depends on both its precision (reproducibility) and its diagnostic accuracy. Prior classifications of urinary tract specimens were often imprecise in relation to the term “atypical” and for the diagnosis of low-grade urothelial carcinomas. The frequency of use of the term “atypical” has ranged from 2% to 31%, and the risk of malignancy associated with the term has varied from 8.3% to 37.5%.[12161718192021] Hence, use of the term varies widely as does its relationship to the presence of malignancy. Recently, The Paris System for Reporting Urinary Cytology was developed to improve both precision and accuracy for the diagnosis of urothelial carcinoma. Definitive diagnostic criteria were proposed. While malignancy risk was associated with each category, little information is available on the precision of assignment of specimens to these diagnostic categories. Brimo and Auger[21] pointed out that even when the “atypical” category is precisely defined and accurately applied, variability in the incidence of the “atypical” diagnosis and its association with carcinoma will vary between laboratories and patient populations studied. Our study reported an AUC rate of 22%. While this is high in comparison to some reports,[171922] it is less than the AUC rates reported by Rosenthal et al.[23] (31%) and Brimo andAuger[21] (26%). Our finding of an AUC rate of 22% continues to demonstrate the difficulty associated with classification of some urine specimens despite the more precise definitions of The Paris System. The atypical category is often associated with poor reproducibility or precision as recognized in the Papanicolaou Society of Cytopathology guidelines for the pancreaticobiliary and respiratory systems. The atypical cells of uncertain significance (ACUS) category also presents challenges for the Bethesda system for cervical cytology. This was addressed by adding categories ACUS high and ACUS low, which do not necessarily improve precision but appear to have increased the clinical utility of the Bethesda system for cervical cytology. Despite the poor precision of the category, AUCs, this category appears to have clinical value because it stratifies malignancy risk. Because of the clinical value of the atypical category, more reproducible criteria are desirable. While the N/C ratio would appear to be measurable and reproducible, it appears imperfect in clinical practice and in a number of morphometric studies.[14] A potential problem with the Paris System for urinary cytology is not the individual criteria including N/C ratio but the propensity for cytopathologists to use atypical categories as a “wastebasket” for cases where a cytopathologist is not certain a specimen is benign but does not see sufficient changes to be comfortable with a suspicious for malignancy diagnosis. In some of these cases just calling a specimen atypical may seem like a safe diagnostic option for some cytopathologists despite the published criteria. Large linear regression studies may be helpful in developing more reproducible criteria for the atypical category.

We assessed clinical impact of disagreement for the diagnostic pairs as shown in Table 4. In the majority of diagnostic pairings, the diagnoses varied by only one category and the clinical impact of such a disagreement between observers would be low with little effect on management. However, disagreement by two or more categories could have a significant impact on clinical management. Hence, classification discrepancy pairs or “NHGUC vs. SHGUC,” “AUC vs. HGUC,” “LGUN vs. HGUC,” and “NHGUC vs. HGUC” could have a significant impact on how a patient is clinically managed. Such “high impact” disagreement occurred in approximately 15% of cases [Table 4]. The probability that a second rater would agree with the initial rater varied considerably [Table 5]. Agreement for the category NHGUC was good at 63%. Agreement for the other categories was poor, especially for the categories LGUN (6%) and SHGUC (13%). Even the categories AUC and HGUC had relatively poor agreement between raters (21%). The Paris System for Reporting Urinary Cytology demonstrates good precision for the diagnosis of NHGUC but had poor interobserver reproducibility (precision) for the other categories. This finding is similar to issues identified with prior categorization proposals.[910111213] Kassouf observed that the category “atypical urothelial cells” is frequently reported raising concerns regarding the possible presence of cancer which may lead to over investigation and excessively prolonged surveillance.[24] The new Paris System may reduce the percentage of urinary tract samples designated as “atypical.” Our series reported an AUS rate of 22%, which is lower than that found by some authors using other classification systems but is still higher than an optimal AUS rate of below 10% as reported by some authorities.[171922]

CONCLUSIONS

Clinical outcomes analysis will be necessary to determine the implications of our findings of poor interobserver agreement for the majority of the categories used in The Paris System for Reporting Urinary Cytology. Our findings suggest that the Paris System suffers from an unacceptably low precision in some of its diagnostic categories. Further, large studies will be necessary to demonstrate the precision and accuracy of The Paris System for Reporting Urinary Cytology.

COMPETING INTERESTS STATEMENT BY ALL AUTHORS

The author(s) declare that they have no competing interests.

AUTHORSHIP STATEMENT BY ALL AUTHORS

All authors of this article declare that we qualify for authorship as defined by ICMJE.

Each author has participated sufficiently in the work and take public responsibility for appropriate portions of the content of this article.

TL carried out data collection and organized. LJL developed the concept and wrote the article. ME, SRF, and TG reviewed slides. RLS completed the statistical analysis. All authors read and approved the final manuscript.

ETHICS STATEMENT BY ALL AUTHORS

This study was conducted with approval from Institutional Review Board (IRB) of all the institutions associated with this study as applicable.

Authors take responsibility to maintain relevant documentation in this respect.

LIST OF ABBREVIATIONS (In alphabetic order)

ACUS - Atypical cells of uncertain significance

AUC - Atypical urothelial cells

HGUC - High grade urothelial carcinoma

LGUN - Low grade urothelial neoplasm

N/C - Nuclear/cytoplasmic

NHGUC - Negative for high grade urothelial carcinoma

SHGUC - Suspicious for high grade urothelial carcinoma

USA - United States of America.

EDITORIAL/PEER REVIEW STATEMENT

To ensure the integrity and highest quality of CytoJournal publications, the review process of this manuscript was conducted under a double-blind model (authors are blinded for reviewers and vice versa) through automatic online system.