Rapid Randomized Controlled Trial Lab | NYU Langone Health

Skip to Main Content
Healthcare Delivery System Redesign Rapid Randomized Controlled Trial Lab

Rapid Randomized Controlled Trial Lab

Hospitals have long sought to improve their care and outcomes through quality improvement programs. But often these programs can either rely on relatively weak evidence or be weighed down by slow-running, inflexible trials. Led by Leora Horwitz, MD, MHS, NYU Langone’s Rapid Randomized Controlled Trial (RCT) Lab, in the Center for Healthcare Innovation and Delivery Science (CHIDS), addresses this challenge.

We aim to transform NYU Langone into a learning healthcare system by using rapid-cycle randomized controlled trials to test simple, pragmatic ideas. Our goal is to identify strategies across our health system that can quickly improve healthcare practice and be scaled nationally. We are one of the first centers in the country to apply rapid RCT methodologies to routinely change our healthcare system.

Our Approach

RCTs are the gold standard of hypothesis testing because they allow us to isolate the effect of our intervention while holding all other variables constant. But typically, RCTs are slow (taking several years), resource intensive, and inflexible—making them impractical for quality improvement. In our lab, we randomize in a cycle of rapidly iterated trials, allowing us to repeatedly test, fine-tune, and adapt our interventions based on data. In this way, we combine the robustness of an RCT with the flexibility and speed of A/B testing and quality improvement techniques.

Graphic Showing How A/B Testing Enables Lab Members to Identify Ideas, Adjust Based on Results, and Disseminate Resulting Best Practices
Our A/B testing workflow enables us to identify areas of need and develop best practices to optimize healthcare practice.

What Have We Done?

Our group has worked with 16 departments across NYU Langone and completed 15 trials, and we are currently involved in 8 ongoing trials, with 8 additional trials in the planning stage.

Because of our lab, we now know that:

  • shortening our mammogram reminder messages and including a link directly to a booking page can increase the appointment rate from 35 percent to 50 percent
  • changing just a few sentences in telephone outreach scripts can both shorten telephone calls and increase rates of appointments for annual examinations
  • our 19,000 post-discharge phone calls did nothing to reduce risk of readmission
  • targeting telephone screening reminders at patients least likely to attend on their own is more effective than targeting those who were more likely to attend (and perhaps just needed a little nudge)
  • real-time prescription benefit alerts, which prompt prescribers to offer “cheaper” medications to patients, reduce patient out-of-pocket costs by an average of 11 percent, and by nearly 40 percent in high-cost drug classes.

We’re also exploring how best to encourage parents to bring their children in for routine vaccinations; whether stopping midnight vitals can improve sleep and reduce delirium without any safety issues; and what the effects of providing “hospital at home” might be.

Our Results

Some recent results from our studies include the following:

These findings have now been integrated into routine practice.

Work with Us

Our lab is open to projects in any discipline, from any staff member, within the NYU Langone system. They should meet the following criteria:

  • involve a high volume of target patients or events, so we can quickly (ideally within a month or two) see whether your intervention is working or not
  • use data that are easily obtained and routinely collected, so our analysis does not add any extra work for frontline staff
  • measure outcomes over short term—over a period of days or weeks—again, so we can quickly see if the intervention is working
  • be randomizable, so we can create intervention and control groups, again, with minimal effort and input from frontline staff

When we work with you, we assist with randomization methods and support your team in assessing results. We provide mentorship, design expertise, project management, IT support, data extraction, statistical analysis, regulatory reporting, and manuscript support. We would typically expect our projects to last one to six months and require minimal additional input from frontline staff.

We’ve also created a toolkit, built from our experience, that provides easy-to-use guidance on designing, managing, and analyzing rapid RCTs.

If you have an idea for a project, please contact Holly Krelle, program manager, at holly.krelle@nyulangone.org. We will also soon be launching our 2022–23 open call for projects.

Selected Publications

Austrian J … Mann, D. Applying A/B testing to clinical decision support: Rapid randomized controlled trials. J Med Internet Res. 2021. DOI.

Blecker S … Katz SD. Interrupting providers with clinical decision support to improve care for heart failure. Int. J. Med. Inform. 2019. DOI.

Desai SM … Horwitz LI. Effects of real-time prescription benefit recommendations on patient out-of-pocket costs: A cluster randomized clinical trial. JAMA Intern. Med. 2022. DOI.

Horwitz LI … Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med. 2019. DOI.

Major VJ … Aphinyanaphongs Y. Evaluating the effect of a COVID-19 predictive model to facilitate discharge: A randomized controlled trial. Appl Clin Inform. 2022. DOI.