July 22, 2024
Principles of Study Design
Medicare Administrative Contractors (MACs) develop Local Coverage Determinations (LCDs) based on clinical evidence as described in CMS’s Internet-Only Manual 100-08, Chapter 13 – Local Coverage Determinations and §13.5.3 Evidentiary Content.1
In evaluating clinical evidence, the MACs must assess the certainty and quality of evidence. The analysis also must determine if the evidence supports a benefit to the Medicare population in terms of improving health outcomes.
There are many systems of evidence analysis that are considered by the MACs. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach has national acceptance as a system to provide guidance for rating quality of evidence and developing strength of recommendations and will be utilized in this educational article.2 GRADE describes evidence analysis as a pyramid with the highest quality (but lowest volume) at the top, and lower quality (but more abundant) work at the bottom. In this model the layers from top to bottom include:
Randomized control trials (highest)
Cohort studies
Case-controlled studies
Case series and reports (lowest)
Systematic reviews and meta-analysis are used to scrutinize evidence.3
Assessing Individual Studies
Several criteria determine weaknesses and strengths of clinical research. High confidence studies share study designs that offer scientific validity that the outcome of interest is related to the intervention that there is a causal relationship between the two.
Consideration is given to each of the GRADE domains for assessing the certainty in the evidence such as risk of bias, imprecision, inconsistency, indirectness, publication bias, large effects, dose response gradients, and residual plausible opposing bias. Different terminology may be used to describe these domains especially as there are multiple tools (besides GRADE) for the assessment that provide slight variation in these terms.4
Many factors can introduce bias into a study and measures to reduce bias are essential to high-quality research. Characteristics of high-quality studies include:
- Randomization- similar subjects are randomly assigned to intervention vs. control. Adequate control group and proper randomization are necessary to determine if the outcome of interest is due to the intervention or chance.
- Well-designed prospective studies allow control of variables that cannot be accounted for when relying on historical populations and improve confidence in the study.
- Adequate sample size to determine statistical significance. The data analytics within the study should explain how the sample size was calculated and the sample must be large enough to eliminate chance as an explanation for the study findings.
- Proper masking (blinding) to ensure study participants and investigators do not know to which group the subjects were assigned to (intervention or control).
- Reducing potential bias is important in study design: types of bias that may impact an investigation include selection, performance, detection, attrition, reporting bias and others.5
- Assessing the impact of confounding variables that can distort measurements of the outcome of interest of the study, introduce bias and mislead results. It is particularly important to assess the impact of confounding in observational studies. Factors such as age, gender, co-morbidities, co-interventions, dosing, and other factors may impact the potential of the intervention to be the causal effect within the study.
- Generalizability is how well the study can be applied to a broader population and is necessary to consider how well the study can be applied to the population of interest. The applicability of the results to the Medicare population is considered.
- Clinically significant proper outcome measures. Risk and benefit assessments aid in determining if the health care outcomes experienced by patients from the intervention are high priority such as reducing disability, morbidity, mortality and improving quality of life, while balancing the risk of harm associated with the intervention.6
- Appropriate length- the length of the study follow-up is important to ensure the duration is sufficient to understand the true outcome of the intervention as well as late presenting complications.
- Accurate results- another important factor is how the outcome is measured and if the modality used to collect this data is sufficient for accurate results.
All of the above characteristics of high-quality studies and measures to reduce bias contribute to understanding if a study informs on the potential benefit of the intervention within the Medicare population.
Real world evidence (RWE) and real-world data (RWD) describe data that is collected outside of clinical trials from various sources such as medical records, claims data, product or disease registries and other data gathering sources like digital health technologies. These are often retrospective or observational studies. While comparative effectiveness research can be conducted from RWD sets there are many challenges to overcome. Without a control group, it is difficult to attribute outcomes to the product under investigation definitively. Controlling variables, accounting for missing data, reducing bias, and ensuring adequate group sizes are complex tasks. A benefit to RWE and RWD is that it can offer valuable insights into patients excluded in clinical trials. Therefore RWD/RWE may support the use of these products, fill in gaps in evidence and address real-life settings. RWE must be peer reviewed and published to be considered in the Summary of Evidence of an LCD and appropriately referenced in the Bibliography of the LCD.
Further details on analysis of evidence review can be found in this proposed CMS document at: https://www.cms.gov/medicare-coverage-database/view/medicare-coverage-document.aspx?mcdid=34&docTypeId=-1&status=all&sortBy=title&bc=16 [cms.gov].7
For definitions see the GRADE HANDBOOK: Glossary of terms and concepts.5
- 21st Century Cures Act. https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf. Accessed 6/3/24.
- GRADE. https://www.gradeworkinggroup.org/. Accessed 6/3/24.
- Kirmayr M, Quilodrán C, Valente B, Loezar C, Garegnani L, Franco JVA. The GRADE approach, Part 1: how to assess the certainty of the evidence. Medwave. 2021;21(2):e8109.
- Schünemann HJ BSea. The development methods of official GRADE articles and requirements for claiming the use of GRADE – A statement by the GRADE guidance group. J Clin Epi 2023.
- Introduction to GRADE Handbook. https://gdt.gradepro.org/app/handbook/handbook.html#h.xivvyiu1pr3v. Published 2013. Updated 10/2013. Accessed 6/3/24.
- CMS. APPENDIX A: General Methodological Principles of Study Design. CAG-00461N Web site. https://www.cms.gov/Medicare/Coverage/DeterminationProcess/downloads/id166b.pdf. Accessed 6/2/24.
- CMS. (PROPOSED) CMS National Coverage Analysis Evidence Review. https://www.cms.gov/medicare-coverage-database/view/medicare-coverage-document.aspx?mcdid=34&docTypeId=-1&status=all&sortBy=title&bc=16. Published 2023. Updated 6/22/23. Accessed 6/3/24.