Hospitals have long sought to improve their care and outcomes through quality improvement programs. But often these programs can either rely on relatively weak evidence or be weighed down by slow-running, inflexible trials. Led by Leora Horwitz, MD, MHS, NYU Langone’s Rapid Randomized Controlled Trial (RCT) Lab, in the Center for Healthcare Innovation and Delivery Science (CHIDS), addresses this challenge.

We aim to transform NYU Langone into a learning healthcare system by using rapid-cycle randomized controlled trials to test simple, pragmatic ideas. Our goal is to identify strategies across our health system that can quickly improve healthcare practice and be scaled nationally. We are one of the first centers in the country to apply rapid RCT methodologies to routinely change our healthcare system.

Our Approach

RCTs are the gold standard of hypothesis testing because they allow us to isolate the effect of our intervention while holding all other variables constant. But typically, RCTs are slow (taking several years), resource intensive, and inflexible—making them impractical for quality improvement. In our lab, we randomize in a cycle of rapidly iterated trials, allowing us to repeatedly test, fine-tune, and adapt our interventions based on data. In this way, we combine the robustness of an RCT with the flexibility and speed of A/B testing and quality improvement techniques.

What Have We Done?

Our group has worked with 16 departments across NYU Langone and completed 15 trials, and we are currently involved in 8 ongoing trials, with 8 additional trials in the planning stage.

Because of our lab, we now know that:

• shortening our mammogram reminder messages and including a link directly to a booking page can increase the appointment rate from 35 percent to 50 percent
• changing just a few sentences in telephone outreach scripts can both shorten telephone calls and increase rates of appointments for annual examinations
• our 19,000 post-discharge phone calls did nothing to reduce risk of readmission
• targeting telephone screening reminders at patients least likely to attend on their own is more effective than targeting those who were more likely to attend (and perhaps just needed a little nudge)
• real-time prescription benefit alerts, which prompt prescribers to offer “cheaper” medications to patients, reduce patient out-of-pocket costs by an average of 11 percent, and by nearly 40 percent in high-cost drug classes.

We’re also exploring how best to encourage parents to bring their children in for routine vaccinations; whether stopping midnight vitals can improve sleep and reduce delirium without any safety issues; and what the effects of providing “hospital at home” might be.

Our Results

Some recent results from our studies include the following:

• Scheduling Reminders: Best aimed at those patients who are least likely to attend
• Shortened Scheduling Messaging: Patients who receive a shorter reminder message schedule more appointments
• Florida Call Center: Small changes to a call script can significantly increase appointment scheduling
• Patient Experience: Sending patients thank you cards does not improve survey response rates
• COVID-19 Predictive Model: Displaying patient risk scores to clinicians was safe but did not reduce length of stay
• Prescriptions: Real-time recommendations can reduce patient out-of-pocket costs
• Flu Vaccine Alerts: Optimizing the inpatient influenza alert increases the vaccination rate, reduces alert fatigue, and simplifies workflow
• Pediatric Vaccines: Text reminders improve routine pediatric vaccination rates
• BETTER CARE-Heart Failure: Best practice alert (BPA) and in-basket message both increased MRA prescribing
• MyChart Messages: Personalized messages doubled patient response rate for preventative care appointments
• BPA Optimization: Program to safely reduce VTE prophylaxis best practice alert firings
• Flu Vaccine Alerts, Fall 2023 Round: Daily firings reduced significantly with slightly lower vaccination rate?

These findings have now been integrated into routine practice.

Work with Us

Our lab is open to projects in any discipline, from any staff member, within the NYU Langone system. They should meet the following criteria:

• involve a high volume of target patients or events, so we can quickly (ideally within a month or two) see whether your intervention is working or not
• use data that are easily obtained and routinely collected, so our analysis does not add any extra work for frontline staff
• measure outcomes over short term—over a period of days or weeks—again, so we can quickly see if the intervention is working
• be randomizable, so we can create intervention and control groups, again, with minimal effort and input from frontline staff

When we work with you, we assist with randomization methods and support your team in assessing results. We provide mentorship, design expertise, project management, IT support, data extraction, statistical analysis, regulatory reporting, and manuscript support. We would typically expect our projects to last one to six months and require minimal additional input from frontline staff.

We’ve also created a toolkit, built from our experience, that provides easy-to-use guidance on designing, managing, and analyzing rapid RCTs.

If you have an idea for a project, please contact Holly Krelle, program manager, at Holly.Krelle@NYULangone.org. Our 2024 open call is now live. Please submit an application to work with us. The open call is for NYU Langone Health employees/trainees only.

Selected Publications

Austrian J … Mann, D. Applying A/B testing to clinical decision support: Rapid randomized controlled trials. J Med Internet Res. 2021. DOI.

Blecker S … Katz SD. Interrupting providers with clinical decision support to improve care for heart failure. Int. J. Med. Inform. 2019. DOI.

Desai SM … Horwitz LI. Effects of real-time prescription benefit recommendations on patient out-of-pocket costs: A cluster randomized clinical trial. JAMA Intern. Med. 2022. DOI.

Horwitz LI … Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med. 2019. DOI.

Major VJ … Aphinyanaphongs Y. Evaluating the effect of a COVID-19 predictive model to facilitate discharge: A randomized controlled trial. Appl Clin Inform. 2022. DOI.

Using Rapid Randomized Trials to Improve Health Care Systems | Annual Reviews