The assessment starts with a set of preliminary considerations to clarify the sources of information and help define the effect modification under consideration.
Study or assessment reference: Use this section to link the completed ICEMAN form to the study, meta-analysis, or publication under consideration, or to assign a custom identifier for tracking completed forms. Examples include an author-year citation, DOI, trial acronym, study registry number. It may also be helpful to specify the comparison of interest, especially if a study includes more than two arms, or to add a link to a published protocol.
State a single outcome and time-point of interest: Because ICEMAN refers to a single outcome at a time, users must specify the outcome of interest and, if applicable, the time-point of outcome assessment (e.g. mortality at 1 year follow-up).
State a single effect measure of interest: Specify a single effect measure of interest (e.g. relative risk, risk difference, odds ratio, or hazard ratio for binary outcomes, or difference or ratio of means for continuous outcomes). The type of effect measure is a key consideration because the magnitude of effect modification typically differs between effect measures, and in particular between measures of relative versus absolute effect.1, 2, 3, 4 Therefore, the credibility rating is likely to differ depending on the chosen effect measure.
Example: An RCT showed that a lifestyle modification program can prevent diabetes.5 A subgroup analysis divided patients into four groups according to their predicted risk of developing diabetes. On the relative hazard ratio scale, the effect was consistent across risk groups (no suggestion of effect modification). On the absolute risk difference scale, however, the effect was much greater in high-risk than in low-risk patients.6
State a single potential effect modifier of interest: Specify the potential effect modifier of interest (only one effect modifier per ICEMAN form). Effect modifiers may be patient characteristics (e.g. disease severity, age, or type of tumour), intervention alternatives (e.g. different doses, co-interventions, or modes of administration), or, in a meta-analysis, methodological study characteristics (e.g. risk of bias, outcome definition, type of funding). Note that the instrument does not apply when the effect modifier is another outcome. Note that an effect modifier (e.g. sex) is different from a particular subgroup (e.g. women).
Was the effect modifier measured before randomization? The instrument applies to potential effect modifiers assessed before randomization, e.g. baseline variables or a stratification variable at randomization. If the effect modifier was measured after randomization, e.g. an intermediate outcome, the assessment of effect modification is complicated and potentially misleading.7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 Those analyses require different methods13, 17, 22 and result in less secure conclusions.
There are exceptions in which the instrument does apply to effect modifiers measured after randomization: (1) the effect modifier is a non-modifiable characteristic such as sex or age; (2) for meta-analyses: the effect modifier is a study characteristic such as risk of bias, length of follow-up, or mean received dose.
Example: An RCT testing strict or conventional management of hyperglycaemia with insulin therapy in ICU patients claimed an effect modification by length of hospital stay.23 Length of ICU stay (the apparent effect modifier), however, was shortened by the intervention. This prognostic imbalance between intervention and control group within the length-of-stay subgroups likely created the differences in mortality.
Does the analysis suggest possible effect modification? Apply ICEMAN version 1.1 only when the analysis indicates possible effect modification. Define this as an interaction test, or, in meta-analyses, an interaction or meta-regression test, with a p-value < 0.1. If the p-value is ≥ 0.1, stop the assessment. ICEMAN evaluates the credibility of suggested effect modification, not the absence of it, and its criteria do not support assessing absence. With p ≥ 0.1, the evidence for effect modification remains very weak. Use the overall effect estimate and state that you did not perform an ICEMAN assessment.
References
1. Engels EA, Schmid CH, Terrin N, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: An empirical study of 125 meta-analyses [Internet]. Stat. Med. 2000 ;19(13):1707–1728.Available from:
http://dx.doi.org/10.1002/1097-0258(20000715)19:13<1707::aid-sim491>3.0.co;2-p
2. Venekamp RP, Rovers MM, Hoes AW, Knol MJ. Subgroup analysis in randomized controlled trials appeared to be dependent on whether relative or absolute effect measures were used [Internet]. J. Clin. Epidemiol. 2014 Apr. ;67(4):410–415.Available from:
http://dx.doi.org/10.1016/j.jclinepi.2013.11.003
3. Rhodes KM, Turner RM, Higgins JP. Empirical evidence about inconsistency among studies in a pairwise meta-analysis. Res. Synth. Methods. 2016 ;7(4):346–370.
4. White IR, Elbourne D. Assessing subgroup effects with binary data: Can the use of different effect measures lead to different conclusions? [Internet]. BMC Med. Res. Methodol. 2005 ;515.Available from:
https://doi.org/10.1186/1471-2288-5-15
5. Knowler WC, Barrett-Connor E, Fowler SE, Hamman RF, Lachin JM, Walker EA, Nathan DM, Diabetes Prevention Program Research Group. Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin [Internet]. N. Engl. J. Med. 2002 ;346(6):393–403.Available from:
http://dx.doi.org/10.1056/NEJMoa012512
6. Kent DM, Steyerberg E, Klaveren D van. Personalized evidence based medicine: Predictive approaches to heterogeneous treatment effects [Internet]. BMJ. 2018 ;363k4245.Available from:
http://dx.doi.org/10.1136/bmj.k4245
7. VanderWeele TJ. On the distinction between interaction and effect modification [Internet]. Epidemiology. 2009 Nov. ;20(6):863–871.Available from:
http://dx.doi.org/10.1097/EDE.0b013e3181ba333c
8. Yusuf S, Wittes J, Probstfield J, Tyroler HA. Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials [Internet]. JAMA. 1991 ;266(1):93–98.Available from:
http://dx.doi.org/10.1001/jama.1991.03470010097038
9. Sun X, Briel M, Walter SD, Guyatt GH. Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses [Internet]. BMJ. 2010 ;340(mar30 3):c117.Available from:
http://dx.doi.org/10.1136/bmj.c117
10. Sun X, Ioannidis JPA, Agoritsas T, Alba AC, Guyatt G. How to use a subgroup analysis: Users’ guide to the medical literature: Users’ guide to the medical literature [Internet]. JAMA. 2014 ;311(4):405–411.Available from:
http://dx.doi.org/10.1001/jama.2013.285063
11. Oxman AD, Guyatt GH. A consumer’s guide to subgroup analyses [Internet]. Ann. Intern. Med. 1992 ;116(1):78–84.Available from:
http://dx.doi.org/10.7326/0003-4819-116-1-78
12. Simon R. Patient subsets and variation in therapeutic efficacy [Internet]. Br. J. Clin. Pharmacol. 1982 Oct. ;14(4):473–482.Available from:
http://dx.doi.org/10.1111/j.1365-2125.1982.tb02015.x
13. Matsuyama Y, Morita S. Estimation of the average causal effect among subgroups defined by post-treatment variables [Internet]. Clin. Trials. 2006 ;3(1):1–9.Available from:
http://dx.doi.org/10.1191/1740774506cn135oa
14. Hirji KF, Fagerland MW. Outcome based subgroup analysis: A neglected concern [Internet]. Trials. 2009 ;10(1):33.Available from:
http://dx.doi.org/10.1186/1745-6215-10-33
15. Cuzick J. The assessment of subgroups in clinical trials [Internet]. Experientia Suppl. 1982 ;41224–235.Available from:
https://www.ncbi.nlm.nih.gov/pubmed/6958512
16. Cook DI, Gebski VJ, Keech AC. Subgroup analysis in clinical trials [Internet]. Med. J. Aust. 2004 ;180(6):289–291.Available from:
http://dx.doi.org/10.5694/j.1326-5377.2004.tb05928.x
17. Desai M, Pieper KS, Mahaffey K. Challenges and solutions to pre- and post-randomization subgroup analyses [Internet]. Curr. Cardiol. Rep. 2014 ;16(10):531.Available from:
http://dx.doi.org/10.1007/s11886-014-0531-2
18. Hoorn R van, Tummers M, Booth A, Gerhardus A, Rehfuess E, Hind D, Bossuyt PM, Welch V, Debray TPA, Underwood M, Cuijpers P, Kraemer H, Wilt GJ van der, Kievit W. The development of
CHAMP: A checklist for the appraisal of moderators and predictors [Internet]. BMC Med. Res. Methodol. 2017 ;17(1):173.Available from:
http://dx.doi.org/10.1186/s12874-017-0451-0
19. Grady D, Cummings SR, Hulley SB. Chapter 11: Alternative trial designs and implementation issues. In: Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB, editor(s). Designing clinical research. Philadelphia: Lippincott Williams & Wilkins; 2007.
20. Moyé LA. 21 the multiple comparison issue in health care research [Internet]. In: Handbook of statistics. Elsevier; 2007. p. 616–655.Available from:
http://dx.doi.org/10.1016/s0169-7161(07)27021-x
21. Rosenbaum PR. The consquences of adjustment for a concomitant variable that has been affected by the treatment [Internet]. J. R. Stat. Soc. Ser. A. 1984 ;147(5):656.Available from:
http://dx.doi.org/10.2307/2981697
22. Korn EL, Othus M, Chen T, Freidlin B. Assessing treatment efficacy in the subset of responders in a randomized clinical trial [Internet]. Ann. Oncol. 2017 ;28(7):1640–1647.Available from:
http://dx.doi.org/10.1093/annonc/mdx197
23. Van den Berghe G, Wilmer A, Hermans G, Meersseman W, Wouters PJ, Milants I, Van Wijngaerden E, Bobbaers H, Bouillon R. Intensive insulin therapy in the medical
ICU [Internet]. N. Engl. J. Med. 2006 ;354(5):449–461.Available from:
http://dx.doi.org/10.1056/NEJMoa052521