raises more troubling questions about the ROCKET-AF trial, which compared the novel oral anticoagulant rivaroxaban (Xarelto) to warfarin in patients with atrial fibrillation.
The controversy about the trial first unfolded last fall when it became known that the portable Alere devices used to monitor and calibrate warfarin usage in the trial were seriously defective. Since then investigators, industry, regulators, and critics have been trying to assess whether the defective devices compromised the findings of the trial.
The new development in the story is that shortly after the trial started, employees of drugmaker Johnson & Johnson became aware that questions were being raised about the monitoring device in a small number of cases. The company launched a program to investigate these concerns but, according to BMJ associate editor Deborah Cohen, failed to inform either regulators or the trial's data and safety monitoring board (DSMB) about the findings of the program.
The BMJ reported that 149 samples were sent to the company by trial investigators who had questions about the accuracy and reliability of the INR readings with the device. Of the samples, 71 were from from patients randomized to warfarin. (Fake INR readings were generated for patients who were randomized to rivaroxaban.) According to the company, there were "16 instances where the value from the point of care device and lab were... inconsistent."
Thus, in this small, non-representative sample, nearly one-quarter of the portable readings used in the trial did not match standard laboratory readings.
The BMJ quoted a member of the trial's DSMB, , of Oxford University, who said he had "no memory of the board being told" about the findings. "Clearly, if the sponsor of the trial had concerns about the validity of the point-of-care testing of INR that would have been important for the board to be made aware of [them]," he told the BMJ. Neither did the company inform either the FDA or the EMA about the findings.
One ROCKET-AF investigator, , of Helskini University General Hospital, agreed that the findings should have been shared. "I now have some doubts about the validity of ROCKET AF trial. It possibly skews the data in favour of rivaroxaban," he said.
Defending ROCKET-AF
Johnson & Johnson responded that the company behaved properly because there was no good reason for it to report this data. I spoke with Pete DiBattiste, the head of development for cardiovascular drugs at Janssen (the pharmaceutical arm of Johnson & Johnson), who told me that the company "took great care and are highly confident that we submitted all relevant information not only to the FDA but to all health authorities around the globe." He pointed out that the 149 samples submitted by investigators to the program represented less than one-twentieth of 1% of the approximately 366,000 INR measurements obtained in the trial.
The small number of samples sent by investigators "is itself a sign of reassurance," said DiBattiste. The program "was put in place as a safety check" but "ended up yielding no meaningful issue" and so there was no reason to report the findings to either the DSMB or regulators, he said. He also pointed out that a certain amount of discordance would be expected, and that the FDA standard for approval of devices does not require perfect accordance.
The company has also defended ROCKET-AF on the basis of analyses performed by Janssen and its European partner, Bayer, as well as separate analyses performed by the ROCKET-AF investigators. But these analyses have also been subject to criticism. In her article, Cohen cites numerous questions that have been raised about these analyses.
"We need to find ways to reduce uncertainty and increase clarity about the balance of benefit to harm," . Ideally, independent trials should replicate the results, they said, but "this may take several years. In the mean time, making the data available for independent scrutiny should be a mandatory regulatory requirement, particularly when there are questions about trial rigour." For now, however, "patients and clinicians must... live with the uncertainty left by the evidence currently available."
In a , , , and , discussed the larger regulatory issues involving diagnostic devices raised by this case. They noted that diagnostic devices "rarely require prospective clinical studies for clearance" and that devices that have been recalled can be used as predicates for the approval of new devices. Even the FDA has acknowledged that "there are few performance standards... based on clearly defined scientific parameters."
Perspective From Sanjay Kaul
, of Cedars-Sinai Medical Center in Los Angeles, was a member of the FDA panel that reviewed ROCKET-AF He told me that he thinks Janssen should have disclosed the data to either the DSMB or the regulators. "It was a mistake that continues to fuel the controversy and perpetuate the perception of malfeasance." Kaul does not find strong evidence in the available data suggesting that the ROCKET-AF results are invalid, but he does support "a reassessment of benefit-risk by independent academics" to perhaps "dispel any residual doubt or adjudicate remaining uncertainties."
Here is Kaul's full statement:
"The saga continues!
"I find it unacceptable that Janssen did not disclose the results of the Covance recheck program to the DSMB or the regulators. It was a mistake that continues to fuel the controversy and perpetuate the perception of malfeasance.
"I am all for data transparency and independent replication as suggested by the editorialists. However, the key question here is whether there is sufficient evidence that the benefit-risk balance of rivaroxaban was materially impacted by the faulty INR readings to revoke the approved indication. The EMA conducted an 'independent' investigation and concluded the benefit-risk balance remained unchanged. The FDA is also conducting its own investigation and so far has not made any major pronouncements. I don't want to second guess the FDA, but if there was an obvious 'fatal flaw' to render the trial data invalid, they would have acted sooner in the interest of public safety.
"Here is my own personal perspective.
Of the 767 values where lab INR>4, 172 (22%) device INR values were between 2 and 3 and 47 (6%) device INR values were less than 2. A device INR of <2 (a 2-INR category discordance) is likely to lead to dose adjustment (escalation). It is important to keep in mind that dose adjustments are seldom based on single INR values. Please note that the difference in bleeding in those with vs without discrepancy is 1.5/100 patient years to 2.0/100 patient years. This is a marginal difference in a relatively small subset (6% of cohort) which is not likely to result in a material difference in the benefit-risk balance. Even if this discrepancy was observed in 100% of the cohort (worst-case scenario), I am not convinced this would result in an undesirable benefit-risk balance. It is reassuring to learn that in data sets external to ROCKET-AF trial (where the defective INR device was not utilized), warfarin-related bleeding rates were consistent with those observed in ROCKET-AF.
"What compounds the issue is that dose adjustment resulting from a clinically-meaningful discrepancy in INR values would have [been] expected to result in lower efficacy event rates (stroke and systemic embolism) and higher safety event rates (bleeding) with warfarin. While observed bleeding rates were aligned with the expected rates, observed stroke rates were the opposite of expected (higher rates). In addition, why would rivaroxaban's (fixed doses were used) bleeding outcomes also be affected in a directionally-similar manner as with warfarin by INR discrepancy but not stroke? These data appear to suggest that observed differences in warfarin event rates cannot be primarily attributed to the adjustments made in warfarin dose based on discrepancy-related device INRs.
"Bottom line, short of conducting a new trial, perhaps, a reassessment of benefit-risk by independent academics might be able to dispel any residual doubt or adjudicate remaining uncertainties amongst the skeptics."
Previous coverage:
- (July 13, 2016)
- (February 3, 2016)
- (December 13, 2015)
- (November 30, 2015)