Hydroxychloroquine: It Works, Dammit! (?)

A pre-print published online July 1 by the International Journal of Infectious Diseases has invigorated the discussion around hydroxychloroquine as a treatment for COVID-19.  This is an analysis of patients treated at Henry Ford Health System in southeast Michigan.  Henry Ford is a large and mature health system known for its high quality care.  You could certainly expect they could produce good data on a large number of patients treated for COVID-19.

The paper, “Treatment with Hydroxychloroquine, Azithromycin, and Combination in Patients Hospitalized with COVID-19” provides several great teaching examples for evaluating journal articles.  Unfortunately none of them teach us how great hydroxychloroquine is at treating COVID-19.  It is also not an example of good writing.

The introduction reviews previous evidence that suggested hydroxychloroquine might exert anti-viral effects against SARS-CoV-2 and that this prompted implementation of a “highly protocolized” use of hydroxychloroquine ± azithromycin (an antibiotic also used to treat COVID-19) in patients at Henry Ford.  The treatment protocol, developed by a COVID-19 Task Force, gave hydroxychloroquine “400 mg twice daily for 2 doses on day 1, followed by 200 mg twice daily on days 2-5. Azithromycin was dosed as 500mg once daily on day 1 followed by 250mg once daily for the next 4 days. The combination of hydroxychloroquine+azithromycin was reserved for selected patients with severe COVID-19 and with minimal cardiac risk factors.” Hydroxychloroquine was started at a median of one day after admission, which is helpful and corrects a problem seen in other papers where it may have been given to patients late in the course of their treatment when the disease has progressed and become severe.

Patients had to have a positive reverse transcription-polymerase chain reaction (RT-PCR) test for SARS-CoV-2 based on a nasopharyngeal swab “during their hospitalization.”  All patients were admitted through the Emergency Department and were ≥18 years old.  The data came from their electronic medical record and included typical, if not detailed, information on patient demographics, co-morbidities, and hospital course.  The primary outcome was in-hospital mortality.

This retrospective cohort analysis involved 2,541 patients admitted 3/10/2020 through 5/2/2020 and included four treatment groups:  1) received neither hydroxychloroquine nor azithromycin (n=409); 2) received hydroxychloroquine only (n=1202); 3) received azithromycin only (n=147); and 4) received both hydroxychloroquine and azithromycin (n=783).  So 78% of the patients received hydroxychloroquine, which appears to align with a fairly consistent practice of offering hydroxychloroquine to their patients with COVID-19.  (There will always be some patients who have contraindications to receiving a drug.)

The crude (unadjusted) results with respect to in-hospital death: 

Looks pretty phenomenal, right?  Don’t place your orders for stock in Sanofi-Synthelabo just yet.  We need to take a closer look at these treatment groups to see if either the study design or decisions around the statistical analysis may have biased the results.

Some people have noticed that the patients who received neither hydroxychloroquine nor azithromycin were on average about five years older than those in the other groups.  [I believe there is a typo in Table 1, since the median age for the hydroxychloroquine group falls outside the interquartile range, which represents the 25%ile through the 75%ile, so it is not 53 as reported].  Yes, that could account for some of the increase in mortality in the “neither medication” group, but: 1) how important is a five year age difference clinically; and 2) the investigators controlled for age in their statistical analysis and the hydroxychloroquine group still showed lower mortality (although controlling for age by splitting them into age <65 vs. ≥65 was suboptimal). 

Remember the protocol the COVID-19 Task Force developed?  Hydroxychloroquine plus azithromycin was reserved for patients with “severe COVID-19.”  That means the vast majority of patients receiving hydroxychloroquine only did not have severe disease.  None of the other treatment groups got an advantage like that.  So by definition, the hydroxychloroquine only group has a lower likelihood of death at the start.  Controlling for other factors like age or co-morbidities will not remove that effect.

What about the “neither hydroxychloroquine nor azithromycin” group – what would be considered the referent or control group.  You might recall from previous posts that when the principle of “the control group should be boring” is violated it can cause some crazy results.  The control group in this study is 16% of the patients treated, so they were fairly rare.  Table 1 reveals some puzzling attributes of that group.  Yes, they’re a little older.  They’re less likely to be Black, their average BMI is lower, and they are less likely to have diabetes than the hydroxychloroquine groups.  There isn’t a lot of difference among the treatment groups with respect to most of the other co-morbidities.  They have a lower prevalence of “chronic lung disease” but I find this a puzzling variable, because it is very common but doesn’t appear to include the most common components of chronic lung disease, i.e., chronic obstructive pulmonary disease or asthma.  On paper the control group looks like they have a lower risk profile, but they have the highest mortality rate.

So we have patients whose underlying health looks relatively favorable (compared to the other treatment groups), but they are dying at the highest rate.  At the same time, according to Table 1, they are not as likely to be admitted to the ICU or receive mechanical ventilation compared to the hydroxychloroquine groups.  What’s up with that? (Cue Kenan Thompson.)  Not only are they not getting hydroxychloroquine, but they’re not getting other forms of more aggressive treatment.  This ain’t no boring control group.

The control group patients are a little more likely to have cancer.  This wouldn’t account for the entire difference among groups, but makes me think that the patients who received neither hydroxychloroquine nor azithromycin included those who may have had some end-stage disease process and a decision was made by either the patient or their family that extraordinary measures should not be undertaken to save their lives.  That is supposition on my part, but is suggested by the constellation of patterns just described.  Controlling for ventilator use will not remove that effect.

What about the hydroxychloroquine plus azithromycin group?  They had severe COVID-19 disease but perhaps their mortality rate wasn’t as bad as expected.  It’s hard to say without a randomized trial.  Recall that they didn’t receive both drugs if they had more than minimal cardiac risk factors.  Possibly these patients had severe COVID-19 but decent health cardiovascular-wise, and maybe pulmonary-wise as well.  They had the highest proportion of patients <65 years old.

I have a couple of other issues with this paper that will be described here.  In addition to their Cox regression to control for confounders like age, chronic kidney disease, etc., the investigators did another analysis where they matched patients according to their propensity scores. “A propensity score was created for each patient based on the set of patient characteristics used in the Cox regression model.”  This statement doesn’t make sense.  Propensity scores are developed in order to remove the effect of a differential likelihood in receiving a treatment or having an exposure of interest.  So you develop a model which predicts the likelihood of receiving, for the sake of argument given there are four treatment groups, hydroxychloroquine.  These factors may or may not be the same as the factors that predict mortality.  Each patient in the “hydroxychloroquine treatment” model gets a propensity score, or model-predicted probability of receiving hydroxychloroquine based on their characteristics.  This propensity score can then be used in the model which examines the factors associated with mortality, either as a variable or in “weighting” the observations.  It has been used in the past to create groups “matched” with respect to their likelihood of receiving the exposure/treatment of interest, however, this is no longer recommended because it can actually increase bias, among other problems.  That’s what the investigators did in this analysis.  So I pay no attention to their propensity score analysis at all.

Finally, the discussion reads as if the investigators are very happy with their results.  They give lip service to the fact that their study is observational, but no consideration for other reasons why they may have seen lower mortality in their hydroxychloroquine groups.  Azithromycin seems to be an afterthought.  Their mortality rates, overall for hospitalized patients (18.1%) and for patients admitted to the ICU (45%), are similar to what has been reported previously (according to their discussion: 10-30% for hospitalized patients and one study that found 58% mortality for patients admitted to the ICU – but other studies have found rates of mortality in the ICU of 25-45%).  It seems if almost 80% of the patients received hydroxychloroquine shortly after admission, and it cut the mortality rate by two-thirds (according to Table 2), you would see mortality rates on the lower end rather than in the middle.

Bottom line: we got one hot mess in the middle of summer.