Comparison of sensitivity and timing of early signal detection of four frequently used signal detection methods: An empirical study based on the US FDA adverse event reporting system database

Abstract

Background: There are limited published comparative data regarding the sensitivity and timing of early signal detection with commonly used signal detection methods (SDMs), including the reporting odds ratio (ROR), proportional reporting ratio (PRR), information component (IC) and gamma Poisson shrinker (GPS). Objective: To examine the sensitivity and timing of early signal detection across four SDMs using the Adverse Events Reporting System (AERS) database of the US Food and Drug Administration. Methods: The four SDMs were applied to retrospectively detect ten confirmed drug-event combinations (DECs). The sensitivity to detect adverse events was defined as the percentage of DECs detected by the respective SDMs as positive signals. The timing of early signal detection was measured by comparing the index date of withdrawal (IDW), defined as the date on which the drug was removed from the market, with the index date of detection (IDD), defined as a date on which the signal was significantly detected by the SDM. Results: The estimated sensitivity was 100% for ROR, 90% for PRR and IC and 70% for GPS. The sensitivity increased with increasing numbers of reports per DEC. Compared with the IDW, the signals were detected on average 10 quarters earlier by ROR, 9 quarters earlier by PRR, 9.9 quarters earlier by the IC and 4.7 quarters earlier by GPS. Conclusions: The sensitivity and timing of early signal detection varies across the four SDMs. Numerically, the ROR showed better performance in sensitivity and early signal detection based on ten selected DECs. Given the limited number and range of DECs selected in this study and the unavailability of specificity assessment, further large-scale prospective studies are warranted in order to provide better guidance on the selection of SDMs. © 2008 Adis Data Information BV. All rights reserved.