A (nonrandom) sample of medical statistics

Lily Koffman

3/8/24

Follow along

Link to slides: https://lilykoff.github.io/grand_rounds_2024/

Introduction

  • Tests
  • P-values
  • Study design

Tests

Four possible outcomes of a test

  • True positive (TP): have disease, test positive
  • True negative (TN): healthy, test negative
  • False positive (FP): healthy, test positive
  • False negative (FN): have disease, test negative
  • Sensitivity: how good is a test at identifying the sick?
    • Probability that test is positive if person is sick TP/(TP + FN)
    • False negative rate: 1-sensitivity
  • Specificity: how good is a test at identifying the healthy?
    • Probability test is negative if person is healthy TN/(TN + FP)
    • False positive rate: 1-specificity

Sensitivity specificity tradeoff

Sahagun, J. Med. Toxicol, 2023

Casscells, N Engl J Med, 1978

https://www.nejm.org/doi/10.1056/NEJM197811022991808

Interpreting a lab result

If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5 per cent, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?

  1. \(< 5 \%\)
  2. \(5-10\%\)
  3. \(11-50\%\)
  4. \(51-80\%\)
  5. \(> 95\%\)

https://pollev.com/lilykoffman075

OR text lilykoffman075 to 22333

18\(\%\) - four of 20 fourth-year students, three of 20 internal med residents, and four of 20 attending physicians answered correctly

Illustration

1000 individuals

1 person has disease, 999 people do not have the disease

False positive rate is 5\(\%\) (specificity is 95\(\%\))

\(0.05 \cdot 999 \approx 50\) healthy people test positive

Probability that a person who tests positive has the disease: \(\frac{1}{51} = 2\%\)

How to use tests, then?

  • PPV: how often someone who tests positive is actually sick
  • NPV: how often someone who tests negative is actually healthy
  • Both depend on prevalence
  • As prevalence increases, NPV \(\downarrow\), PPV \(\uparrow\)

Illustration with increased prevalence

Prevalence is 50/1000 instead of 1/1000

\(950\cdot0.05 = 48\) healthy people test positive

Probability person who tests positive has the disease: \(\frac{50}{50+48} = 51\%\)

How to increase PPV?

  • Repeat test
    • Probability of of healthy person testing positive twice: \(0.05\cdot0.05 = 0.0025\)
  • There will be \(999\cdot 0.0025 \approx 2-3\) false positives

Probability that a person who tests positive has the disease: \(\frac{1}{4} = 25\%\)

Panel of tests on healthy individual

Probability of false positive = \(5\%\)

Probability all tests are normal = \((0.95)^{20} = 36\%\)

P-values

https://xkcd.com/1478/

P-values in the wild

Leinweiber, J Vasc Surg, 2023

P-values in the wild

Leinweiber, J Vasc Surg, 2023

How do we get them?

  • Specify null hypothesis
    • In this case: no difference in between men, women in mean aneurysm diameter
  • Compare the groups
    • In this case: compare mean aneurysm size
  • Calculate probability of observing difference at least as large as the difference we observe, assuming there is no true difference between the groups
  • If the probability is small, reject the null and have evidence that there is a difference in means between the groups

Does sample size matter?

Which p-value would you “prefer”?

Which p-value would you “prefer”?

P-hacking

  • Selective reporting of significant results
  • Deciding whether to keep collecting data partway through experiment/trial
  • Recording many outcomes and deciding which to report post-hoc
  • Deciding whether to include/drop outliers post-hoc
  • Performing (or not performing) subgroup analyses post-hoc

Study design

Randomized (controlled) trials

Observational studies

RCTs

Farber, N Engl J Med, 2022

  • Enroll individuals and randomize to treatment or control group
    • Endovascular therapy vs. revascularization
  • Follow over time and assess outcome
    • Amputation, major limb reintervention, or death

Prospective cohort studies

Benson, Ann Surg, 2021

  • Enroll individuals free of outcome, follow and assess outcome
    • Enroll individuals before vascular surgery
    • Assess mortality

Retrospective cohort studies

Schermerhorn, N Engl J Med, 2015

  • Population of interest is identified, exposure status is retroactively determined
    • Population: Medicare patients who had abdominal aortic aneurysm repair
    • Exposure: open or endovascular repair
    • Outcome: mortality/complications

Comparison: confounding

  • RCTs
    • Randomization ensures confounders are balanced (on average) between groups
  • Cohort studies
    • Adjust for confounding with multivariable regression, matching, or stratification
    • Cannot address unmeasured confounding variables

Comparison: validity

  • RCTs
    • Internally valid, but may not generalize if target population \(\neq\) study population
    • Strict inclusion criteria (“Patients were excluded from the trial if they had excessive risk associated with open vascular surgery”)
  • Cohort studies
    • More likely to be externally valid (“real world data”)
    • Confounding, information bias (incorrect assessment of exposure, outcome, or both) can affect internal validity

Comparison: loss to follow up (LTFU) or poor compliance

  • Both RCTs and cohort studies:
    • LTFU often associated with outcome, exposure, or both
      • I.e. side effects of treatment or severity of disease
      • “Final” outcome can’t be assessed if LTFU
    • Loss of power or bias
  • Strategies to minimize
    • Study design
    • Reporting of LTFU and potential implications
  • Fong, J Am Heart Assoc, 2020 found 16\(\%\) of cardiovascular RCTs could have a change in primary outcome if “plausible assumptions are made about differential event rates of participants lost to follow up”

Comparison: selection bias

  • Both RCTs and cohort studies:
    • Possible depending on recruitment strategy/cohort selection
      • Association between peripheral vascular disease and orthopedic injuries due to selection of hospitalized individuals1
    • Exclusion of women from PAD research2

Case study: paclitaxel-coated balloons and stents1

Katsanos, J Am Heart, 2018

Risk of all-cause death \(\color{red}{1.68}\) times higher in pactlitaxel group at 2 years and \(\color{red}{1.93}\) times higher at 5 years

FDA safety alert

Cohort studies to the rescue!

  • Retrospective analysis of insurance claims-matched patients1
    • Paclitaxel-coated balloons associated with increased long term survival, lower amputation rates, lower cardiovascular event rate
  • Retrospective analysis of Medicare data2
    • Paclitaxel-coated balloons associated with better survival at 2 years, clinical outcomes at 1 year

References

  1. Mahmood Kabeil, Riley Gillette, Ethan Moore, Robert F. Cuff, Jason Chuen, Max V. Wohlauer, A primer on cohort studies in vascular surgery research, Seminars in Vascular Surgery, Volume 35, Issue 4, 2022, Pages 404-412, ISSN 0895-7967, https://doi.org/10.1053/j.semvascsurg.2022.09.004

  2. Casscells, W., Schoenberger, A., & Graboys, T. B. (1978). Interpretation by physicians of clinical laboratory results. The New England journal of medicine, 299(18), 999–1001. https://doi.org/10.1056/NEJM197811022991808

  3. Tigges, S. Papa & Papa Bear’s medical statistics short course. Radiopaedia. https://radiopaedia.org/courses/medical-statistics-short-course

  4. Sahagun, B.E., Williams, C. & Su, M.K. Biostatistics and Epidemiology Principles for the Toxicologist: The “Testy” Test Characteristics Part I—Sensitivity and Specificity. J. Med. Toxicol. 19, 40–44 (2023). https://doi.org/10.1007/s13181-022-00916-0

Articles included

  • Leinweber, M. E., Geisbuesch, P., Balzer, K., Schmandra, T., Karl, T., Popp, S., Hoffmann, J., Schmitz-Rixen, T., Jung, G., POPART Registry Collaborators, Oikonomou, K., Storck, M., Balzer, K., Kugelmann, U., Schneider, C., Engelhardt, M., Petzold, M., Weis-Mueller, B., Wortmann, M., Popp, S., … Bail, D. (2023). Sex disparities in popliteal artery aneurysms. Journal of vascular surgery, S0741-5214(23)02437-0. https://doi.org/10.1016/j.jvs.2023.12.036
  • Farber, A., Menard, M. T., Conte, M. S., Kaufman, J. A., Powell, R. J., Choudhry, N. K., Hamza, T. H., Assmann, S. F., Creager, M. A., Cziraky, M. J., Dake, M. D., Jaff, M. R., Reid, D., Siami, F. S., Sopko, G., White, C. J., van Over, M., Strong, M. B., Villarreal, M. F., McKean, M., … BEST-CLI Investigators (2022). Surgery or Endovascular Therapy for Chronic Limb-Threatening Ischemia. The New England journal of medicine, 387(25), 2305–2316. https://doi.org/10.1056/NEJMoa2207899
  • Benson, R. A., & Nandhra, S. (2021). Outcomes of Vascular and Endovascular Interventions Performed During the Coronavirus Disease 2019 (COVID-19) Pandemic. Annals of surgery, 273(4), 630–635. https://doi.org/10.1097/SLA.0000000000004722

Articles included

  • Schermerhorn, M. L., Buck, D. B., O’Malley, A. J., Curran, T., McCallum, J. C., Darling, J., & Landon, B. E. (2015). Long-Term Outcomes of Abdominal Aortic Aneurysm in the Medicare Population. The New England journal of medicine, 373(4), 328–338. https://doi.org/10.1056/NEJMoa1405778
  • Fong LCW, Ford TJ, da Costa BR, Jüni P, Berry C. Bias and Loss to Follow-Up in Cardiovascular Randomized Trials: A Systematic Review. J Am Heart Assoc. 2020 Jul 21;9(14):e015361. doi: 10.1161/JAHA.119.015361. Epub 2020 Jul 9. PMID: 32646264; PMCID: PMC7660731.
  • Katsanos, K., Spiliopoulos, S., Kitrou, P., Krokidis, M., & Karnabatidis, D. (2018). Risk of Death Following Application of Paclitaxel-Coated Balloons and Stents in the Femoropopliteal Artery of the Leg: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. Journal of the American Heart Association, 7(24), e011245. https://doi.org/10.1161/JAHA.118.011245

Articles included

  • Behrendt, C. A., Sedrakyan, A., Peters, F., Kreutzburg, T., Schermerhorn, M., Bertges, D. J., Larena-Avellaneda, A., L’Hoest, H., Kölbel, T., & Debus, E. S. (2020). Editor’s Choice - Long Term Survival after Femoropopliteal Artery Revascularisation with Paclitaxel Coated Devices: A Propensity Score Matched Cohort Analysis. European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery, 59(4), 587–596. https://doi.org/10.1016/j.ejvs.2019.12.034
  • Long, C. A., Zepel, L., Greiner, M. A., Hammill, B. G., Patel, M. R., & Jones, W. S. (2019). Use and 1-year outcomes with conventional and drug-coated balloon angioplasty in patients with lower extremity peripheral artery disease. American heart journal, 217, 42–51. ,https://doi.org/10.1016/j.ahj.2019.07.014>
  • Freisinger, E., Koeppe, J., Gerss, J., Goerlich, D., Malyar, N. M., Marschall, U., Faldum, A., & Reinecke, H. (2020). Mortality after use of paclitaxel-based devices in peripheral arteries: a real-world safety analysis. European heart journal, 41(38), 3732–3739. https://doi.org/10.1093/eurheartj/ehz698