The items included in ICEMAN do apply in principle to claims of effect modification made in observational studies, or meta-analyses of observational studies, but additional considerations may apply. The variability of study designs and the risk of confounding for main effects introduces additional complexity and uncertainty. Applying ICEMAN to observational studies may therefore result in over-optimistic credibility judgments.
Provide a copy of the completed ICEMAN form(s) in an appendix to your article. In the forms, don't forget to provide supporting commentaries; without those, it is difficult for readers to understand the judgements. In the methods section, state that you applied ICEMAN to potential effect modifications with an interaction p-value of 0.1 or smaller. In the results, provide interaction p-values and credibility ratings using the wording as suggested in ICEMAN. If the credibility is moderate or high, justify using ICEMAN criteria.
Do not apply ICEMAN to assess the consistency of effects across subgroups. There is currently no formal instrument that can do that.
Do not apply ICEMAN if there is very little statistical evidence; if the p-value of interaction is larger than 0.1, you can safely assume that the credibility of the effect modification is low or very low - even without applying ICEMAN.
Do not systematically apply ICEMAN to all subgroup analyses conducted for the two reasons provided above.