The reliability of systematic reviews of the effects of health interventions is variable. Consequently, policymakers and others need to assess how much confidence can be placed in such evidence. The use of systematic and transparent processes to determine such decisions can help to prevent the introduction of errors and bias in these judgements. This article suggests five questions that can be considered when deciding how much confidence to place in the findings of a systematic review of the effects of an intervention: Did the review explicitly address an appropriate policy or management question? Were appropriate criteria used when considering studies for the review? Was the search for relevant studies detailed and reasonably comprehensive? Were assessments of the studies' relevance to the review topic and of their risk of bias reproducible? Were the results similar from study to study?
Monitoring equity and research policy
A key challenge that policymakers and those supporting them must face is the need to understand whether research evidence about an option can be applied to their setting. Systematic reviews make this task easier by summarising the evidence from studies conducted in a variety of different settings. Many systematic reviews, however, do not provide adequate descriptions of the features of the actual settings in which the original studies were conducted. This article suggests questions to guide those assessing the applicability of the findings of a systematic review to a specific setting: Were the studies included in a systematic review conducted in the same setting or were the findings consistent across settings or time periods? Are there important differences in on-the-ground realities and constraints that might substantially alter the feasibility and acceptability of an option? Are there important differences in health system arrangements that may mean an option could not work in the same way? Are there important differences in the baseline conditions that might yield different absolute effects even if the relative effectiveness was the same? What insights can be drawn about options, implementation, and monitoring and evaluation?
This study assessed the methods used in the evaluation of measles vaccination coverage in 2012/3 in eastern and southern Africa, identified quality concerns and made recommendations for improvement. Of the 13 reports the authors reviewed, there were weaknesses in 10 of them for ethical clearance, 9 for sample size calculation, 6 for sampling methods, 12 for training structures, 13 for supervision structures and 11 for data analysis. The authors recommend improvements in the documentation of routine and supplementary immunisation, via home-based vaccination cards or other records. They recommend that standards be developed for report templates and for the technical review of protocols and reports. This would ensure that the results of vaccination coverage surveys are accurate, comparable, reliable and valuable for programme improvement.
The AIDS epidemic remains of global significance and there is a need to target (a) the adolescent age-groups in which most new infections occur; and (b) sub-Saharan Africa where the greatest burden of the epidemic lies. A focused systematic review of school-based sexual health interventions in sub-Saharan Africa to prevent HIV and Sexually Transmitted Infections (STI) in this age group was therefore conducted. Some 1,020 possible titles and abstracts were found, 23 full text articles were critically appraised, and 12 articles (10 studies) reviewed, reflecting the paucity of published studies conducted relative to the magnitude of the HIV epidemic in sub-Saharan Africa. Knowledge and attitude-related outcomes were the most associated with statistically significant change. Behavioural intentions were more difficult to change and actual behaviour change was least likely to occur. Behaviour change in favour of abstinence and condom use appeared to be greatly influenced by pre-intervention sexual history. There is a great need in sub-Saharan Africa for well-evaluated and effective school-based sexual health interventions.
The economic benefits of healthcare research require study so that appropriate resources can be allocated to this research, particularly in developing countries. This study took the form of a systematic review to identify the methods used to assess the economic impact of healthcare research, and the outcomes. The initial search yielded 8,416 articles, of which 18 articles were included in the analysis, as well as eleven other reports. It found that the outcomes assessed as healthcare research payback included direct cost-savings, cost reductions in healthcare delivery systems, benefits from commercial advancement, and outcomes associated with improved health status. The study found that different methods and outcomes can be used to assess the economic impacts of healthcare research. However, none of the research from low- and middle-income countries had evaluated the economic return of research. The authors recommend a consensus on practical guidelines at international level in order to build capacity, arrange for necessary informative infrastructures and promote necessary skills for economic evaluation studies in developing countries.
Policy-makers and health system managers routinely face difficult decisions around improving health and promoting equity. They must consider complex, core questions about particular programmes to implement and effective strategies for organising the overall health system. For instance, does contracting out services to the private sector improve access to health care? How could the health system best retain trained health care providers in underserved areas? Do conditional cash transfers improve the uptake of health interventions? This brief provides essential background information to systematic reviews: how they are conducted, what they entail and their theoretical roots. It discusses tools like GRADE (Grading of Recommendations, Assessment, Development and Evaluation) and organisations like the Cochrane Collaboration. The brief calls for increased funding to support systematic reviews, improved methodological development in the reviewing process, increased networking and the need to promote training of end-users. The use of knowledge translation is highlighted.
Current systematic reviews have a utilitarian bias – they tend to be concerned more with the effects on populations and average effects than with distributional effects and impacts in disadvantaged sub¬groups. the author poses that systematic reviews should routinely consider the effects of interventions on health inequities. The Cochrane Health Equity Field was set up explicitly to further this agenda, to encourage systematic review authors to explicitly assess the effects of interventions not only on the whole population, but on the disadvantaged. Evidence on interventions to improve public health is in short supply, partic¬ularly evidence on social determinants. Public health systematic reviewers need to continue to develop new methods and better frameworks to inform decision-making.
In this study, researchers aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations. They developed a health equity plausibility algorithm with three items that dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: patient characteristics; intervention delivery (i.e. implementation); and comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. The proportion agreement was 66% for patient characteristics, 67% for intervention delivery and 55% for the comparator. Users of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need, the authors conclude.
This paper presents the methodology and findings from a proof-of-concept study undertaken to explore the viability of conducting a systematic, largely qualitative synthesis of evaluation reports emanating from Community Based Rehabilitation (CBR) projects in developing countries. Computer assisted thematic qualitative analysis was conducted on recommendation sections from 37 evaluation reports, arising from 36 disability and development projects in 22 countries. Quantitative overviews and qualitative summaries of the data were developed. The application of the synthesis methodology utilised in this proof-of-concept study was found to be potentially very beneficial for future research in CBR, and indeed in any area within health services or international development in which evaluation reports rather than formal research evidence is the primary source material. The proof-of-concept study identified a number of limitations which are outlined. Based on the conclusions of 37 evaluation reports, future policy frameworks and implementation strategies in CBR should include a stronger emphasis on technical, organisational, administrative and personnel aspects of management and strategic leadership.
It is 20 years since an international consultation in Lejondal, Sweden, highlighted the need for more and better research “to understand and improve how societies organize themselves in achieving collective health goals, and how different actors interact in the policy and implementation processes to contribute to policy outcomes.” One outcome was the creation of the Alliance for Health Policy and Systems Research. There have since been several important milestones on the path towards more and better health systems research. Such encouraging developments need to be carefully balanced against areas where progress has not met expectations. The development of national capacity for financing and institutional leadership of health policy and systems research has been slow. The authors ask: how can health policy and systems research seize this favourable context and contribute more effectively to universal health coverage, greater health security, improved value in health and effective engagement of citizens? First, they propose that there is a need a comprehensive review of the progress that has been made and the progress that might be anticipated in the future. Second, there is a need to accelerate the development of capacity and opportunity for health policy and systems research in low- and middle-income countries. Lastly, they suggest that there is a need for a revitalised approach to stewardship that reflects the pluralistic reality of contemporary health systems and prioritises health policy and systems research across all countries.