Random Forests as Statistical Procedures: Design, Variance, and Dependence
Variance and dependence in random forests, including the uncertainty that remains as the number of trees grows.
arXiv
Questions, methods & contributions
My growing focus on methods and theory comes from years of collaborative biomedical research. Over the last several years, much of my time has been spent thinking through and developing clinical prediction models, often using machine learning. Through that work, I’ve become increasingly bothered by how disconnected prediction modeling can feel from the ideas about uncertainty and inference I rely on as a statistician. I want to know how much confidence to place in an individual prediction, how uncertain a model’s performance estimate from cross-validation is, and how much I can trust the variables a model selects. I’ve often struggled to find satisfactory answers in the statistical literature, especially ones I can use to guide decisions in practice. These are questions I’ve been thinking about for years, and much of my current research is an effort to answer them. As I move forward in my career, I plan to spend more of my time on theory and methods, particularly addressing these issues.
I’m interested in how the decisions we make when building and evaluating a model affect its results and the uncertainty around them. In other words, my research interests lie at the intersection of prediction modeling and inference. My current projects focus on the variance of random forests and cross-validation, both developing the theory to understand that variance and finding ways to estimate it in practice. These projects grew out of wanting to better understand methods I use routinely, especially what their results can tell us and where we may be placing too much confidence in them.
I also continue to work on clinical trial design, where my methodological research began during my PhD. Building on that background in Bayesian dose-finding trials, I’m developing adaptive designs that use repeated patient outcomes to guide decisions as a study progresses, while accounting for the dependence among observations from the same patient. As with my work on prediction models, the motivation comes from problems I’ve encountered through collaboration and want better ways to address.
Variance and dependence in random forests, including the uncertainty that remains as the number of trees grows.
arXiv
How the learning procedure, performance metric, and resampling design contribute to uncertainty in cross-validation estimates.
Manuscript in preparation
Procedure-aligned synthetic resampling (PASR) for estimating uncertainty in cross-validation performance estimates.
Manuscript in preparation
Bayesian adaptive trial designs that use repeated patient outcomes while accounting for dependence within patients and changes over time.
Work in progress
A comparison of five common methods for analyzing pre–post data in clinical research. 300+ citations.
Journal of Biometrics & Biostatistics
A dose-finding method for cancer drug combinations that incorporates toxicity scores into the continual reassessment framework.
Contemporary Clinical Trials
A comparison of variable-selection methods for random-forest regression with continuous outcomes.
Briefings in Bioinformatics
An R package providing clustered datasets for evaluating and comparing statistical methods.
BMC Medical Research Methodology
No work in this category yet.
I’ve spent much of my career working collaboratively with clinical investigators and research scientists, with more than 70 coauthored papers across oncology, cardiovascular disease, pediatrics, neurology, emergency medicine, and other areas. As a collaborative biostatistician, I work with clinical colleagues to help develop and design research studies, oversee the collection of data, and conduct analyses, helping investigators connect the questions they want to answer with what we can learn from the data. It is from these collaborations that I identify areas of methodological research to pursue. I am motivated by finding statistical gaps in applied research and coming up with rigorous and novel methods to solve them.
I am Director of Biostatistics for the ARISE Cancer Consortium, a multisite pediatric oncology consortium across North and South Carolina. I lead biostatistics and data coordination across the consortium’s studies, with responsibility for trial design, protocol review, statistical analysis plans, data monitoring, and analysis.
Much of my clinical work is about understanding what patients experience during treatment and how their health changes afterward. In cancer research, I’ve worked on studies of heart function, persistent fatigue, and treatment tolerability. One issue I’ve spent time on is that calling an adverse event “low grade” doesn’t tell us how much it matters to the person experiencing it, particularly when it keeps happening or occurs alongside other symptoms. That question led to my first-author JCO paper on adverse events, treatment experience, and treatment discontinuation. Outside oncology, a few examples of my collaborations include work with emergency medicine teams on predicting deterioration in patients with pulmonary embolism, as well as pediatric studies predicting dropout from weight-management programs and testing ways to reduce missed well-child visits. I also work on migraine trials that examine both headache frequency and how much headaches interfere with patients’ lives, and on studies of how clinicians use decision-support tools in electronic health records. The studies highlighted below represent a selection of that work.
PREVENT
The PREVENT randomized trial, led by W. Gregory Hundley, evaluated atorvastatin for limiting declines in cardiac function during and after doxorubicin treatment. I served as the primary biostatistician responsible for the statistical analyses.
NEJM Evidence
UPBEAT
The UPBEAT cohort study examined persistent fatigue following breast cancer treatment and its relationships with psychological symptoms and cardiovascular function.
JAMA Network Open
CROWN
A multisite study of cardiovascular changes during ovarian suppression and aromatase inhibitor therapy in premenopausal women with breast cancer.
American Heart Journal
Migraine trial · Rebecca Wells
A randomized clinical trial with Rebecca Wells comparing mindfulness meditation with headache education for adults with migraine.
JAMA Internal Medicine
E1912 · Low- and moderate-grade adverse events
The role of low- and moderate-grade adverse events in patients’ treatment experience and treatment discontinuation in the E1912 trial.
Journal of Clinical Oncology