Advertisement

AHRQ EPC Series on Improving Translation of Evidence into Practice for the Learning Health System: Introduction

      Learning health systems are those “in which science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the care process. . . .”
      • Institute of Medicine
      Best Care at Lower Cost: The Path to Continuously Learning Health Care in America.
      (p. 136) However, health systems often don’t have ready access to the best available evidence in an actionable form, or they struggle to put evidence into practice. To help address this, the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) program has worked to improve translation of high-quality evidence into practice by learning health systems through funding nine pilot projects. This article introduces a special series of articles summarizing this work. Future articles in the series will highlight and describe in greater detail five of the nine EPC pilot projects conducted under this initiative and provide an overall summary of lessons learned and a health systems panel’s feedback on the pilots.
      AHRQ is a federal agency in the Department of Health and Human Services. Its mission is to produce evidence to make health care safer, of higher quality, and more accessible, equitable, and affordable, and to ensure that the evidence is understood and used. AHRQ is committed to building bridges between research and practice and working with health systems to put research into practice through the dissemination and implementation of evidence. It is also committed to accelerating the spread of evidence-based best practices across learning health systems and to help systems become learning health systems.
      • Agency for Healthcare Research and Quality. AHRQ Research Summit on Learning Health Systems
      For more than 20 years, the AHRQ EPC program has produced scientifically rigorous and comprehensive evidence reviews that synthesize research to inform evidence-based care in order to improve the quality of health care in the United States. To this end, AHRQ funds 12 academic/research organizations (known as the EPCs) that review all relevant scientific literature on a wide spectrum of clinical and health services topics to produce various types of evidence reviews, including systematic reviews. Because studies on a given topic are often numerous and can have disparate findings, it is critical to summarize and synthesize findings across studies. AHRQ’s evidence reviews synthesize these studies and address the strength of the available evidence.
      The AHRQ EPC program partners with external organizations—health care systems, clinicians, guideline development groups, and others—that are committed to using the review for implementing evidence-based practices. AHRQ wants to make sure that a partner organization needs and has a plan for using the review,
      • Atkins D.
      • et al.
      Better information for better health care: the Evidence-based Practice Center program and the Agency for Healthcare Research and Quality.
      • Matchar D.B.
      • et al.
      Dissemination of Evidence-based Practice Center reports.
      before committing funds. Organizations or individuals can nominate topics through the AHRQ Effective Health Care (EHC) Program website.
      • Agency for Healthcare Research and Quality. Effective Health Care Program
      EHC Program Topic Nomination and Selection.
      The EPC program assesses every proposed topic with predefined criteria (appropriateness, importance, duplication, feasibility) to select those with the greatest potential for impact, change, and value.
      • Agency for Healthcare Research and Quality. Effective Health Care Program
      EHC Program Topic Nomination and Selection.
      • Whitlock E.P.
      • et al.
      AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program.

      Agency for Healthcare Research and Quality. Developing and Selecting Topic Nominations for Systematic Reviews. Eder M, et al. Methods Guide for Comparative Effectiveness Reviews. AHRQ Publication No. 12(13)-EHC153-EF. Nov 2012. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-guidance-topic-nomination_methods.pdf.

      Many of its reports have been used by clinical professional organizations or other federal agencies to inform and develop clinical practice guidelines, coverage decisions, quality measures, educational materials and tools, and research agendas. For example, EPC evidence reviews are used by the Center for Medicare & Medicaid Services to inform its coverage decisions, and the US Preventive Services Task Force (USPSTF) to develop its recommendations.
      A MEDLINE (PubMed) search revealed that, in 2017, nearly 35 systematic reviews, 59 randomized or controlled clinical trials, and 166 observational studies were published every day, at an increasing rate compared to previous years. Even though evidence reviews summarize and synthesize a large or heterogeneous body of evidence in attempts to facilitate decision making, the evidence is still growing in volume and complexity. In addition, competing responsibilities and priorities keep clinicians and other decision makers from rapid uptake of evidence as it becomes available.
      The AHRQ EPC program recognizes that production of high-quality evidence reviews does not guarantee or promote their uptake.

      Agency for Healthcare Research and Quality. Understanding Health-Systems’ Use of and Need for Evidence to Inform Decisionmaking. White CM, et al. Research White Paper. AHRQ Publication No. 17(18)-EHC035-EF. Oct 2017. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/health-systems-research.pdf.

      • Guise J.M.
      • Savitz L.A.
      • Friedman C.P.
      Mind the gap: putting evidence into practice in the era of learning health systems.
      As part of AHRQ’s commitment to accelerating the spread of evidence-based best practices across learning health systems, the EPC program has been engaging in efforts to help learning health systems use evidence from its reports to improve patient care and health outcomes.
      • Borsky A.E.
      • et al.
      Next steps in improving healthcare value: AHRQ Evidence-based Practice Center program—applying the knowledge to practice to data cycle to strengthen the value of patient care.
      In 2016 the EPC program began exploring the evidence needs of health systems and how they currently access evidence and incorporate it into their improvement and care processes. It developed a framework for categorizing health care system questions and needs, and examined the health care systems’ process for incorporating evidence into decision making.

      Agency for Healthcare Research and Quality. Understanding Health-Systems’ Use of and Need for Evidence to Inform Decisionmaking. White CM, et al. Research White Paper. AHRQ Publication No. 17(18)-EHC035-EF. Oct 2017. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/health-systems-research.pdf.

      This initial work and various outreach efforts to health care systems have brought to light a number of important issues. Although systematic reviews and other evidence syntheses are generally considered unbiased and of high quality, systematic reviews alone are not often seen as primary sources of evidence, particularly among health care system decision makers. There are a number of reasons for this, including the length and format of reports, which are perceived as barriers to evidence uptake.
      • Marquez C.
      • et al.
      Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study.
      • Murthy L.
      • et al.
      Interventions to improve the use of systematic reviews in decision-making by health system managers.
      • Villa L.
      • et al.
      Health care decision makers’ use of comparative effectiveness research: report from a series of focus groups.
      In addition, health care system users of systematic reviews often need some form of a translational or companion product to help them understand and use the findings from the reviews to improve patient care.

      The 2018 AHRQ EPC Pilot Projects to Translate Evidence into Practice

      AHRQ directed the EPCs to identify and engage health care systems to work cooperatively with them to develop and pilot translational products based on a recent EPC evidence review. Each of these individual pilot projects was designed to accelerate uptake and use of findings from an EPC review that was current, relevant to the quality improvement efforts of the selected health care system, and aligned with AHRQ priorities. AHRQ gave the EPCs a high degree of freedom when designing their pilots to encourage creativity and diversity in products and methods of engagement, and thus did not specify use of any particular technology, format, or audience. Although this variance complicated summarizing results across the pilots, it allowed a wider net to be cast in this information-gathering pilot study.
      The EPCs began designing their pilots in fall 2017. As part of the pilot, each EPC evaluated its own product and summarized its implementation and evaluation in a report that can be found on the AHRQ Effective Health Care Program (https://effectivehealthcare.ahrq.gov/) website and in Table 1. In addition, the AHRQ Scientific Resource Center (SRC), which provides scientific support to the program, summarized the evaluation findings across the pilots. The SRC developed a form to record common elements among the very different pilot projects, including a combination of open-ended questions and discrete ratings. Subject areas for the reporting form included potential implementation and impact, process descriptions, applicability of product to other EPC reviews or health care systems, and EPC impressions of completing the project.
      Table 1EPC Pilot Project Groupings, Pilot Project Titles, and Corresponding EPC Evidence Review Titles
      References can be found at the end of the article.
      Product CategoryEPCsPilot ProductPilot Project TitleEPC Evidence Review
      Dissemination ProductsJohns Hopkins University (JHU)Key Points, Newsletter Item, Evidence to Decision FrameworkDisseminating Findings from EPC Reports: Pilot Project of Three Products

      Agency for Healthcare Research and Quality. Disseminating Findings from EPC Reports: Pilot Project of Three Products. Robinson KA. Methods Research Report. AHRQ Publication No. 18(19)-EHC024-EF. Oct 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-disseminating-findings_0.pdf.

      “Contrast-Induced Nephropathy: Comparative Effects of Different Contrast Media”

      Agency for Healthcare Research and Quality. Contrast-Induced Nephropathy: Comparative Effects of Different Contrast Media. Eng J, et al. Comparative Effectiveness Review No. 155. AHRQ Publication No. 15(16)-EHC022-EF. Dec 2015. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/nephropathy-contrast-induced_research.pdf.

      University of Alberta (UA)1- and 3-Page SummariesDevelopment and Usability Testing of EPC Evidence Review Dissemination Summaries for Health Systems Decisionmakers

      Agency for Healthcare Research and Quality. Development and Usability Testing of EPC Evidence Review Dissemination Summaries for Health Systems Decisionmakers. Hartling L, et al. Methods Research Report. AHRQ Publication No. 18(19)-EHC027-EF. Nov 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/topics/usability-testing/methods-report.

      “Strategies to Improve Mental Health Care for Children and Adolescents”

      Agency for Healthcare Research and Quality. Strategies to Improve Mental Health Care for Children and Adolescents. Forman-Hoffman VL, et al. Comparative Effectiveness Review No. 181. AHRQ Publication No. 16(17)-EHC035-EF. Dec 2016. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/mental-health-children_research.pdf.



      “First- and Second-Generation Antipsychotics in Children and Young Adults: Systematic Review Update”

      Agency for Healthcare Research and Quality. First- and Second-Generation Antipsychotics in Children and Young Adults: Systematic Review Update. Pillay J, et al. Comparative Effectiveness Review No. 184. AHRQ Publication No. 17-EHC001-EF. Mar 2017. (Updated: Jan 2018.) Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/antipsychotics-children-update_research-2017_0.pdf.

      Kaiser Permanente Research Affiliates (KPRA), Southern California (SCEPC)CyberseminarLinking Evidence Reviews to Organizational Guideline Planning: A Pilot Test of an Interactive, Web-Based Presentation and Discussion of Evidence

      Agency for Healthcare Research and Quality. Linking Evidence Reviews to Organizational Guideline Planning: A Pilot Test of an Interactive, Web-Based Presentation and Discussion of Evidence. Lin JS, et al. Methods Research Report. AHRQ Publication No. 18(19)-EHC025-EF. Oct 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/linking-evidence-reviews-methods_0.pdf.

      “Screening for Abnormal Glucose and Type 2 Diabetes Mellitus: A Systematic Review to Update the 2008 U.S. Preventive Services Task Force Recommendation”
      • Agency for Healthcare Research and Quality. Screening for Abnormal Glucose and Type 2 Diabetes Mellitus
      A Systematic Review to Update the 2008 U.S. Preventive Services Task Force Recommendation. Selph S, et al. Evidence Synthesis No. 117. AHRQ Publication No.
      Interactive Data Visualization ProductsUniversity of Connecticut (UConn)
      Products to be highlighted by this series.
      Quality Measure IndexAssessing the Impact of Indexing Performance Measure Codes on the Perceived Value of EPC Reports to Health Systems

      Agency for Healthcare Research and Quality. Assessing the Impact of Indexing Performance Measure Codes on the Perceived Value of EPC Reports to Health Systems. White CM, et al. Methods Research Report. AHRQ Publication No. 18-EHC022-EF. Sep 2018. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-assessing-impact.pdf.

      “Nonsurgical Treatments for Urinary Incontinence in Women: A Systematic Review Update”

      Agency for Healthcare Research and Quality. Nonsurgical Treatments for Urinary Incontinence in Women: A Systematic Review Update. Balk E, et al. Comparative Effectiveness Review No. 212. AHRQ Publication No. 18-EHC016-EF. Aug 2018. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-212-urinary-incontinence-updated_1.pdf.

      Pacific Northwest (PacNW)MAGICapp and Tableau Data VisualizationImproving Access to and Usability of Systematic Review Data for HealthSystems GuidelinesDevelopment

      Agency for Healthcare Research and Quality. Improving Access to and Usability of Systematic Review Data for Health Systems Guidelines Development. Totten AM, et al. Methods Research Report. AHRQ Publication No. 19-EHC010-EF. Feb. Feb 2019. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-health-systems-guidelines-development.pdf.

      “Noninvasive, Nonpharmacological Treatment for Chronic Pain: A Systematic Review”

      Agency for Healthcare Research and Quality. Noninvasive Nonpharmacological Treatment for Chronic Pain: A Systematic Review. Skelly AC, et al. Comparative Effectiveness Review No. 209. AHRQ Publication No. 18-EHC013-EF. Jun 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/nonpharma-chronic-pain-cer-209.pdf.

      Brown University, Duke University, Minnesota
      Products to be highlighted by this series.
      Interactive Report PresentationWeb Interactive Presentation of EPC Reports and Mapping to Quality Measures

      Trikalinos T, et al. Web Interactive Presentation of EPC Reports and Mapping to Quality Measures. Draft Report. Rockville, MD: Agency for Healthcare Research and Quality, 2018.

      “Nonsurgical Treatments for Urinary Incontinence in Women: A Systematic Review Update”

      Agency for Healthcare Research and Quality. Nonsurgical Treatments for Urinary Incontinence in Women: A Systematic Review Update. Balk E, et al. Comparative Effectiveness Review No. 212. AHRQ Publication No. 18-EHC016-EF. Aug 2018. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-212-urinary-incontinence-updated_1.pdf.

      Implementation ProductMayo ClinicDecision AidsEPC Pilot Project: A Dual Approach to Facilitate Health Systems Uptake of Evidence Synthesis Reports. Anxiety in Children

      Agency for Healthcare Research and Quality. EPC Pilot Project: A Dual Approach to Facilitate Health Systems Uptake of Evidence Synthesis Reports. Anxiety in Children. Morrow AS, et al. Methods Research Report. AHRQ Publication No. 18(19)-EHC021-EF. Oct 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-report-anxiety-children_0.pdf.

      “Anxiety in Children”

      Agency for Healthcare Research and Quality. Anxiety in Children. Wang Z, et al. Comparative Effectiveness Review No. 192. AHRQ Publication No. 17-EHC023-EF. Aug 2017. (Updated: Jul 2018.) Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-192-revised-anxiety-report.pdf.

      RTI International—University of North Carolina (RTI-UNC)
      Products to be highlighted by this series.
      EHR Implementation PackageDevelopment of a Primary Care Guide for Implementing Evidence-Based Screening and Counseling for Unhealthy Alcohol Use with Epic-Based ElectronicHealth Record Tools: A Pilot Dissemination Project
      • Barclay C.
      • et al.
      AHRQ series on improving translation of evidence: implementing evidence-based screening and counseling for unhealthy alcohol use with Epic-based electronic health record tools.
      “Screening, Behavioral Counseling, and Referral in Primary Care to Reduce Alcohol Misuse”

      Agency for Healthcare Research and Quality. Screening, Behavioral Counseling, and Referral in Primary Care to Reduce Alcohol Misuse. Jonas DE, et al. Comparative Effectiveness Review No. 64. AHRQ Publication No. 12-EHC055-EF. Jul 2012. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/alcohol-misuse_research.pdf.

      ECRI Institute–Penn Medicine (ECRI-Penn)Clinical Practice Pathway
      Products to be highlighted by this series.
      Use of a Clinical Pathway to Facilitate the Translation and Utilization of AHRQ EPC Report Findings

      Agency for Healthcare Research and Quality. Use of a Clinical Pathway to Facilitate the Translation and Utilization of AHRQ EPC Report Findings. Flores E, et al. Methods Research Report. AHRQ Publication No. 19-EHC002-EF. Dec 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/clinical-pathway-methods-report.pdf.

      “Early Diagnosis, Prevention, and Treatment of Clostridium difficile: Update.”

      Agency for Healthcare Research and Quality. Early Diagnosis, Prevention, and Treatment of Clostridium difficile: Update. Butler M, et al. Comparative Effectiveness Review No. 172. AHRQ Publication No. 16-EHC012-EF. Mar 2016. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/c-difficile-update_research.pdf.

      EPC, Evidence-based Practice Center; EHR, electronic health record.
      * References can be found at the end of the article.
      Products to be highlighted by this series.

      Translational Products Developed

      The EPCs developed a broad range of translational products. We categorized the nine projects into three main types: (1) dissemination, (2) interactive data visualization, and (3) implementation. Dissemination products all aimed to provide a general overview of the evidence to inform a system-level or departmental decision. Interactive data visualizations generally provided more detailed data than dissemination products, with the intent of informing or answering very specific questions. Implementation products aimed to integrate evidence into clinical work flow. Table 1 provides the names of the pilot projects, the EPC(s) that developed them, and the title of the EPC review on which each translational product was based.
      In this article, we summarize each of the nine EPC pilot projects and characterize the chosen approach to improve uptake of EPC review findings. Figure 1 depicts the EPC pilot projects on the dissemination to implementation continuum described by Tabak and colleagues.
      • Tabak R.G.
      • et al.
      Bridging research and practice: models for dissemination and implementation research.
      Table 2 provides a detailed summary of key features and characteristics of these products. Later articles in the series will describe in detail five of the EPC pilot projects and their results: quality measure index, interactive report presentation, EHR implementation package, and clinical practice pathway (see bolded projects in Table 1).
      Figure 1
      Figure 1This graphic depicts the nine Evidence-based Practice Center (EPC) pilot project approaches mapped on the continuum from dissemination to implementation. EHR, electronic health record.
      Table 2Summaries of Pilot Projects
      Type of ProductDissemination ProductInteractive Data Visualization ProductImplementation Product
      ProductKey Points, Newsletter Item, EtD Framework1- and 3-Page SummariesCyberseminarQuality Measure IndexMAGICapp and Tableau Data VisualizationInteractive Report PresentationDecision AidsEHR Implementation PackageClinical Practice Pathway
      Product intentInform departmental decision on which contrast media should be usedProvide overview of the state of the evidenceProvide interactive forum to discuss evidence translation and inform national, system-level guideline developmentIdentify and match AHRQ EPC evidence reviews associated with quality improvement measuresInform ongoing work related to opioid prescribing guideline developmentCreate customizable information about desired comparisons and outcomesSupport systemwide implementation of a therapy and promote evidence-based decision making among patientsSupport systems-wide implementation of updated alcohol abuse screening recommendationsInform clinical decision making regarding Clostridium difficile infection treatment
      Product target audience
      System decision makers were conceptualized as those who make decisions for quality and safety across clinical specialties and throughout an entire system (for example, Kaiser Permanente guideline development team). Department decision makers make decisions for one clinical department and not necessarily across a whole system (for example, clinical directors).
      • Department decision makers• System decision makers• System decision makers

      • Researchers
      • System decision makers• System decision makers

      • Researchers
      • Clinicians

      • System decision makers

      • Researchers
      • Clinicians

      • Patients

      • System decision makers
      • Department decision makers• Clinicians

      • System decision makers
      Type of info included from systematic review• High-level summary

      • Executive summary
      • High-level summary

      • Key messages

      • Executive summary
      • High-level summary•Executive summary• Executive summary

      • Evidence tables
      • Evidence tables• Key messages

      • Executive summary
      • Executive summary• Executive summary

      • Evidence tables
      EPCJHUUAKPRA / SCEPCUConnPacific NorthwestBrown / Duke / MinnesotaMayo ClinicRTI-UNCECRI-Penn
      Health care system partnerJHU Health SystemAlberta Health ServicesKaiser Permanente Care Management InstituteDuke University Health System; Hartford HealthCare; UConn Health CareOregon Health & Science UniversityDuke University Health SystemMayo Clinic Health SystemUniversity of North Carolina General Internal Medicine ClinicPenn Medicine
      EtD, Evidence to Decision; EHR, electronic health record; JHU, Johns Hopkins University; UA, University of Alberta; KPRA/SCEPC, Kaiser Permanente Research Affiliates / Southern California Evidence-based Practice Center; AHRQ, Agency for Healthcare Research and Quality; UConn, University of Connecticut; RTI-UNC, RTI International–University of North Carolina; ECRI-Penn, Penn Medicine.
      * System decision makers were conceptualized as those who make decisions for quality and safety across clinical specialties and throughout an entire system (for example, Kaiser Permanente guideline development team). Department decision makers make decisions for one clinical department and not necessarily across a whole system (for example, clinical directors).

      Dissemination Products

      Dissemination is defined by Tabak et al. as “the active approach of spreading evidence-based interventions to the target audience via determined channels using planned strategies.”
      • Tabak R.G.
      • et al.
      Bridging research and practice: models for dissemination and implementation research.
      (p. 339) We expanded on this definition to include products that involved repackaging portions of existing EPC reviews or key messages into a more readable format or through Cyberseminars. All three new products in this category provided high-level summaries of the EPC reviews and aimed to address department or system-level decisions. These three products were as follows:
      • Key points, newsletter item, and Evidence to Decision framework. The Johns Hopkins University (JHU) EPC developed, piloted, and compared three different products: key points (developed from current EPC program guidance); a newsletter item (developed according to a recommendation from a previous work group);

        Agency for Healthcare Research and Quality. Understanding Health-Systems’ Use of and Need for Evidence to Inform Decisionmaking. White CM, et al. Research White Paper. AHRQ Publication No. 17(18)-EHC035-EF. Oct 2017. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/health-systems-research.pdf.

        and an Evidence to Decision (EtD) framework developed according to guidance previously published by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group.
        • Alonso-Coello P.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction.
        The primary source of the information in JHU’s products was the EPC review executive summary and accompanying journal manuscript.
        • Eng J.
        • et al.
        Comparative effect of contrast media type on the incidence of contrast-induced nephropathy: a systematic review and meta-analysis.
        For the EtD framework, questions pertaining to EtD elements (such as benefits/harms, outcome importance, equity, acceptability, and feasibility) were phrased to stimulate health care system partners’ thinking about these parameters.
      • 1- and 3-page summaries. The University of Alberta (UA) EPC developed 1- and 3-page summaries for two EPC reviews based on guidance from the Canadian Foundation for Healthcare Improvement and Cochrane.

        Canadian Foundation for Healthcare Improvement/Canadian Health Services Research Foundation. Communication Notes: Reader-Friendly Writing—1:3:25. Jun 1, 2010. Accessed Jun 10, 2019. http://www.cfhi-fcass.ca/migrated/pdf/communicationnotes/cn-1325_e.pdf.

        • AlFaleh K.
        • Anabrees J.
        Probiotics for prevention of necrotizing enterocolitis in preterm infants.
        Information included in UA EPC’s products primarily came from the EPC reviews’ executive summaries. The two products intentionally differed on specific design elements (use of color, placement of program logos, and others) and types of information included (qualitative vs. quantitative) for the purpose of comparison. Both summaries briefly mentioned methods, with links to the full reports; and in the summary of a qualitative EPC review, information included strength of evidence and qualitative comparative analysis figures were presented. Summary tables, strength of evidence, and network meta-analysis figures were displayed in the quantitative summary product.
      • Cyberseminar. The Southern California EPC (SCEPC) and Kaiser Permanente Research Affiliates (KPRA) EPC jointly developed and evaluated a Clinical Operations Evidence Review (COER) Cyberseminar. A Cyberseminar is a Web-based, interactive webinar during which both researchers and participants can ask questions and discuss the nuances of report findings. The interactive seminar format was developed based on investigator experiences with the USPSTF, US Department of Veterans Affairs Cyberseminar program, and SCEPC’s previous expert panel work. The Cyberseminar highlighted implementation-ready interventions and went beyond simply describing the evidence by discussing contextual and implementation considerations.

      Interactive Data Visualization Products

      We included products in this category that use technology or software to enable an interactive interface with findings in reports. Data visualization has been previously described as “encompass[ing] a broad set of techniques for representing data values and other information graphically.”
      • Hallgren K.A.
      • et al.
      Beyond path diagrams: enhancing applied structural equation modeling research through data visualization.
      (p. 75) These three products were as follows:
      • Quality measure index. The University of Connecticut (UConn) EPC developed a table for inclusion in EPC reviews that would direct readers to the page numbers where information relevant to a quality measure could be found. In electronic formats of the report, the page numbers in the quality measure index table would be hyperlinked, immediately redirecting readers to the relevant pages. Quality measure indexing was developed in response to feedback from previous methods work groups and health care systems. Developing these tables required an additional review to identify relevant quality measures, as this information is not typically contained in EPC reviews.
      • MAGICapp and Tableau data visualization. The Pacific Northwest (PacNW) EPC explored various existing software options for different data visualizations. The EPC selected and piloted two interactive data visualizations using MAGICapp and Tableau software. The MAGICapp visualization used a standardized template provided by MAGICapp, which allowed users to see summary results at a high level, as well as to drill down more deeply into specific information as needed. The Tableau visualization required more programming effort but allowed the user to slice and dice to see specific types of information.
      • Interactive report presentation. The Brown University EPC developed an open-source web-based interactive report that allowed users to compare across select populations, interventions, and outcomes of interest. Data for meta-analyzed and non-meta-analyzed outcomes were included in the product. Health care system partners provided feedback on key features and analyses in mock-ups of the interactive tool.

      Implementation Products

      Implementation is defined by Tabak et al. as “the process of putting to use or integrating evidence-based interventions within a setting.”
      • Tabak R.G.
      • et al.
      Bridging research and practice: models for dissemination and implementation research.
      (p.339) We expanded on this definition to include those that facilitated integration of evidence directly into clinical work flow. The three new implementation products were as follows:
      • Decision aids. The Mayo Clinic EPC developed two decision aids: a health care system decision aid and an encounter decision aid. The health care system decision aid was developed based on the EtD framework by GRADE
        • Alonso-Coello P.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction.
        and includes information on costs, feasibility, and other implementation considerations that are not typically included in EPC reviews. The encounter decision aid was developed based on “the principles of shared decision making and the design characteristics recommended for decision aids”

        Agency for Healthcare Research and Quality. EPC Pilot Project: A Dual Approach to Facilitate Health Systems Uptake of Evidence Synthesis Reports. Anxiety in Children. Morrow AS, et al. Methods Research Report. AHRQ Publication No. 18(19)-EHC021-EF. Oct 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-report-anxiety-children_0.pdf.

        (p. 3) and includes information pertaining to harms and effectiveness.
      • Clinical practice pathway. The ECRI-Penn EPC piloted a clinical pathway that was integrated into the EHR. The clinical practice pathway was developed based on Penn Medicine’s existing 10-step process for developing pathways, which includes a review of existing guidelines, rapid review updates, and clinical input to develop a structured pathway. Health care system partners reviewed and gave feedback on pathway prototypes.
      • EHR implementation package. The RTI International–University of North Carolina (RTI-UNC) EPC piloted an implementation package that included an EHR integrated tool (Nurses Best Practice Advisory) and instructions for use. The EHR implementation package included an overview of implementation, preliminary steps, detailed implementation steps, facilitators, and barriers to implementation.

      Conclusion

      AHRQ EPC reviews serve as a comprehensive, high-quality, trusted source of evidence. To increase the use of evidence by health systems, EPCs developed translational products that include dissemination products, interactive data visualization products, and implementation products. Each of these product types serves different purposes and different audiences within the health system. The dissemination products provide different subsets of information from the larger EPC review in concise, organized formats. The interactive data visualization products allow users to view the review evidence in as much detail as they wish; these products preserve the comprehensiveness of the reviews and make details accessible, while attempting to avoid overwhelming the user in the way a full EPC review might. Implementation products provide additional contextual information that is required to fully integrate the evidence into clinical care. All three types of translational products have some degree of usefulness for health systems, and further research could explore which types of products are most useful in different contexts.
      We hope that the articles in this series will inform health systems about how others have tried to improve the translation of evidence into practice, get ideas from these efforts, and use this information to inform their own efforts to bridge the evidence-to-practice gap going forward.

      Funding and Disclaimer

      This project was funded under Contract No. HHSA 290-2017-00003C from the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services (HHS). The authors of this manuscript are responsible for its content. Statements in the manuscript do not necessarily represent the official views of or imply endorsement by AHRQ or HHS.

      Acknowledgments

      The authors respectfully acknowledge the contributions of Anne Sales, PhD, RN, for her feedback and input on the projects. The authors also acknowledge the work of Elizabeth Stoeger, BS; Allison Schmidt, MS; and Samuel Aldape, BA.

      Conflicts of Interest

      All authors report no conflicts of interest.

      References

        • Institute of Medicine
        Best Care at Lower Cost: The Path to Continuously Learning Health Care in America.
        National Academies Press, Washington, DC2013 (Accessed Jun 8, 2019)
        • Agency for Healthcare Research and Quality. AHRQ Research Summit on Learning Health Systems
        Executive Summary. Sep 15, 2017.
        Accessed. Jun 10, 2019;
        • Atkins D.
        • et al.
        Better information for better health care: the Evidence-based Practice Center program and the Agency for Healthcare Research and Quality.
        Ann Intern Med. 2005 Jun 21; 142: 1035-1041
        • Matchar D.B.
        • et al.
        Dissemination of Evidence-based Practice Center reports.
        Ann Intern Med. 2005 Jun 21; 142: 1120-1125
        • Agency for Healthcare Research and Quality. Effective Health Care Program
        EHC Program Topic Nomination and Selection.
        Accessed. Jun 10, 2019;
        • Whitlock E.P.
        • et al.
        AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program.
        J Clin Epiemiol. 2010; 63: 491-501
      1. Agency for Healthcare Research and Quality. Developing and Selecting Topic Nominations for Systematic Reviews. Eder M, et al. Methods Guide for Comparative Effectiveness Reviews. AHRQ Publication No. 12(13)-EHC153-EF. Nov 2012. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-guidance-topic-nomination_methods.pdf.

      2. Accessed. Jun 10 2019;
      3. Agency for Healthcare Research and Quality. Understanding Health-Systems’ Use of and Need for Evidence to Inform Decisionmaking. White CM, et al. Research White Paper. AHRQ Publication No. 17(18)-EHC035-EF. Oct 2017. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/health-systems-research.pdf.

        • Guise J.M.
        • Savitz L.A.
        • Friedman C.P.
        Mind the gap: putting evidence into practice in the era of learning health systems.
        J Gen Intern Med. 2018; 33: 2237-2239
        • Borsky A.E.
        • et al.
        Next steps in improving healthcare value: AHRQ Evidence-based Practice Center program—applying the knowledge to practice to data cycle to strengthen the value of patient care.
        J Hosp Med. 2019; 14: 311-314
        • Marquez C.
        • et al.
        Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study.
        Implement Sci. 2018 Jun 22; 13: 84
        • Murthy L.
        • et al.
        Interventions to improve the use of systematic reviews in decision-making by health system managers.
        policy makers and clinicians Cochrane Database Syst Rev. 2012 Sep 12; (EPub only)CD009401
        • Villa L.
        • et al.
        Health care decision makers’ use of comparative effectiveness research: report from a series of focus groups.
        J Manag Care Pharm. 2013; 19: 745-754
        • Tabak R.G.
        • et al.
        Bridging research and practice: models for dissemination and implementation research.
        Am J Prev Med. 2012; 43: 337-350
        • Alonso-Coello P.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction.
        BMJ. 2016 Jun 28; i2016: 353
        • Eng J.
        • et al.
        Comparative effect of contrast media type on the incidence of contrast-induced nephropathy: a systematic review and meta-analysis.
        Ann Intern Med. 2016 Mar 15; 164: 417-424
      4. Canadian Foundation for Healthcare Improvement/Canadian Health Services Research Foundation. Communication Notes: Reader-Friendly Writing—1:3:25. Jun 1, 2010. Accessed Jun 10, 2019. http://www.cfhi-fcass.ca/migrated/pdf/communicationnotes/cn-1325_e.pdf.

        • AlFaleh K.
        • Anabrees J.
        Probiotics for prevention of necrotizing enterocolitis in preterm infants.
        Cochrane Database Syst Rev. 2014 Apr 10; (EPub only)CD005496
        • Hallgren K.A.
        • et al.
        Beyond path diagrams: enhancing applied structural equation modeling research through data visualization.
        Addict Behav. 2019; 94: 74-82
      5. Agency for Healthcare Research and Quality. EPC Pilot Project: A Dual Approach to Facilitate Health Systems Uptake of Evidence Synthesis Reports. Anxiety in Children. Morrow AS, et al. Methods Research Report. AHRQ Publication No. 18(19)-EHC021-EF. Oct 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-report-anxiety-children_0.pdf.

      6. Agency for Healthcare Research and Quality. Disseminating Findings from EPC Reports: Pilot Project of Three Products. Robinson KA. Methods Research Report. AHRQ Publication No. 18(19)-EHC024-EF. Oct 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-disseminating-findings_0.pdf.

      7. Agency for Healthcare Research and Quality. Contrast-Induced Nephropathy: Comparative Effects of Different Contrast Media. Eng J, et al. Comparative Effectiveness Review No. 155. AHRQ Publication No. 15(16)-EHC022-EF. Dec 2015. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/nephropathy-contrast-induced_research.pdf.

      8. Agency for Healthcare Research and Quality. Development and Usability Testing of EPC Evidence Review Dissemination Summaries for Health Systems Decisionmakers. Hartling L, et al. Methods Research Report. AHRQ Publication No. 18(19)-EHC027-EF. Nov 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/topics/usability-testing/methods-report.

      9. Agency for Healthcare Research and Quality. Strategies to Improve Mental Health Care for Children and Adolescents. Forman-Hoffman VL, et al. Comparative Effectiveness Review No. 181. AHRQ Publication No. 16(17)-EHC035-EF. Dec 2016. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/mental-health-children_research.pdf.

      10. Agency for Healthcare Research and Quality. First- and Second-Generation Antipsychotics in Children and Young Adults: Systematic Review Update. Pillay J, et al. Comparative Effectiveness Review No. 184. AHRQ Publication No. 17-EHC001-EF. Mar 2017. (Updated: Jan 2018.) Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/antipsychotics-children-update_research-2017_0.pdf.

      11. Agency for Healthcare Research and Quality. Linking Evidence Reviews to Organizational Guideline Planning: A Pilot Test of an Interactive, Web-Based Presentation and Discussion of Evidence. Lin JS, et al. Methods Research Report. AHRQ Publication No. 18(19)-EHC025-EF. Oct 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/linking-evidence-reviews-methods_0.pdf.

        • Agency for Healthcare Research and Quality. Screening for Abnormal Glucose and Type 2 Diabetes Mellitus
        A Systematic Review to Update the 2008 U.S. Preventive Services Task Force Recommendation. Selph S, et al. Evidence Synthesis No. 117. AHRQ Publication No.
        in: 13-05190-EF-1. Apr 2015
      12. Agency for Healthcare Research and Quality. Assessing the Impact of Indexing Performance Measure Codes on the Perceived Value of EPC Reports to Health Systems. White CM, et al. Methods Research Report. AHRQ Publication No. 18-EHC022-EF. Sep 2018. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-assessing-impact.pdf.

      13. Agency for Healthcare Research and Quality. Nonsurgical Treatments for Urinary Incontinence in Women: A Systematic Review Update. Balk E, et al. Comparative Effectiveness Review No. 212. AHRQ Publication No. 18-EHC016-EF. Aug 2018. Accessed Jun 8, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-212-urinary-incontinence-updated_1.pdf.

      14. Agency for Healthcare Research and Quality. Improving Access to and Usability of Systematic Review Data for Health Systems Guidelines Development. Totten AM, et al. Methods Research Report. AHRQ Publication No. 19-EHC010-EF. Feb. Feb 2019. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/methods-health-systems-guidelines-development.pdf.

      15. Agency for Healthcare Research and Quality. Noninvasive Nonpharmacological Treatment for Chronic Pain: A Systematic Review. Skelly AC, et al. Comparative Effectiveness Review No. 209. AHRQ Publication No. 18-EHC013-EF. Jun 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/nonpharma-chronic-pain-cer-209.pdf.

      16. Trikalinos T, et al. Web Interactive Presentation of EPC Reports and Mapping to Quality Measures. Draft Report. Rockville, MD: Agency for Healthcare Research and Quality, 2018.

      17. Agency for Healthcare Research and Quality. Anxiety in Children. Wang Z, et al. Comparative Effectiveness Review No. 192. AHRQ Publication No. 17-EHC023-EF. Aug 2017. (Updated: Jul 2018.) Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-192-revised-anxiety-report.pdf.

        • Barclay C.
        • et al.
        AHRQ series on improving translation of evidence: implementing evidence-based screening and counseling for unhealthy alcohol use with Epic-based electronic health record tools.
        Jt Comm J Qual Patient Saf. 2019; 45: 566-574
      18. Agency for Healthcare Research and Quality. Screening, Behavioral Counseling, and Referral in Primary Care to Reduce Alcohol Misuse. Jonas DE, et al. Comparative Effectiveness Review No. 64. AHRQ Publication No. 12-EHC055-EF. Jul 2012. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/alcohol-misuse_research.pdf.

      19. Agency for Healthcare Research and Quality. Use of a Clinical Pathway to Facilitate the Translation and Utilization of AHRQ EPC Report Findings. Flores E, et al. Methods Research Report. AHRQ Publication No. 19-EHC002-EF. Dec 2018. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/clinical-pathway-methods-report.pdf.

      20. Agency for Healthcare Research and Quality. Early Diagnosis, Prevention, and Treatment of Clostridium difficile: Update. Butler M, et al. Comparative Effectiveness Review No. 172. AHRQ Publication No. 16-EHC012-EF. Mar 2016. Accessed Jun 10, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/c-difficile-update_research.pdf.

      Biography

      Celia Fiordalisi, MS, is Research Associate, Scientific Resource Center (SRC) for the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program, Portland VA [US Department of Veterans Affairs] Research Foundation, VA Portland Health Care Systems, Portland, Oregon.
      Amanda Borsky, DrPH, MPP, is Health Scientist, US Preventive Services Task Force and Evidence-based Practice Center (EPC) Program, Center for Evidence and Practice Improvement (CEPI), AHRQ, Rockville, Maryland.
      Stephanie Chang, MD, MPH, is Director, EPC Program, CEPI, AHRQ.
      Jeanne-Marie Guise, MD, MPH, is Director, SRC for the AHRQ EHC Program, Portland VA Research Foundation.