Advertisement

AHRQ Series on Improving Translation of Evidence: Perceived Value of Translational Products by the AHRQ EPC Learning Health Systems Panel

      The Agency for Healthcare Research and Quality (AHRQ) has focused on building and supporting learning health systems. AHRQ defines a learning health system as a system in which “internal data and experience are systematically integrated with external evidence, and that knowledge is put into practice.”
      • Agency for Healthcare Research and Quality. About Learning Health Systems
      AHRQ adds that, as a result of this use of experience and knowledge, “patients get higher quality, safer, more efficient care, and health care delivery organizations become better places to work.”
      • Agency for Healthcare Research and Quality. About Learning Health Systems
      AHRQ has commissioned several agencywide initiatives to explore how its work can be more relevant to leaders, embedded researchers, and change agents in health care delivery organizations striving to become learning health systems.
      • Agency for Healthcare Research and Quality. About Learning Health Systems
      • Agency for Healthcare Research and Quality. AHRQ Research Summit on Learning Health Systems
      • Agency for Healthcare Research and Quality. Supporting the Next Generation of Learning Health Systems Researchers
      Grants Awarded Under the AHRQ-PCORI Institutional Mentored Career Development Program (K12). Sep 2018.
      Among these initiatives is the AHRQ Evidence-based Practice Center (EPC) Program.
      Despite a growing number of models to improve the dissemination and implementation of research findings,
      • Tabak R.G.
      • et al.
      Bridging research and practice: models for dissemination and implementation research.
      the amount of time it currently takes to translate evidence into practice is too long.
      • Morris Z.S.
      • Wooding S.
      • Grant J.
      The answer is 17 years, what is the question: understanding time lags in translational research.
      • Balas E.A.
      • Boren S.A.
      Managing clinical knowledge for health care improvement.
      Decision makers within health systems may have different evidence needs than other stakeholders (for example, clinical professional societies, practicing clinicians), and they likely need additional products to help them incorporate and use findings from systematic evidence reviews in their routine operations (hereafter, we refer to these types of products as translational products). For example, clinical professional organizations routinely use systematic reviews to inform the development of their guidelines. However, systematic reviews are not frequently used by health care managers and policy makers in decision making and may need to be in more succinct or alternative formats.
      • Marquez C.
      • et al.
      Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study.
      • Vogel J.P.
      • et al.
      Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey.
      Health systems are also still looking for ways to effectively combine their locally generated data with critically appraised external evidence.
      • Guise J.M.
      • Savitz L.A.
      • Friedman C.P.
      Mind the gap: putting evidence into practice in the era of learning health systems.
      The core focus of EPCs is to synthesize evidence using rigorous scientific methods,

      Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Jan 2014. Accessed Aug 14, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf.

      but the EPC Program recognizes that not everyone has time to read a full evidence review and that an evidence review alone will not change practice.
      To advance this goal, the EPC Program has taken a multipronged approach.
      • Borsky A.E.
      • et al.
      AHRQ Evidence-based Practice Center Program—applying the knowledge to practice to data cycle to strengthen the value of patient care.
      In 2018, EPCs conducted nine pilot projects to develop new and innovative translational products for health systems.
      • Agency for Healthcare Research and Quality. Effective Health Care Program
      AHRQ EPC Program Helps Health Systems Use Evidence: Series Overview. May 30, 2018.
      • Morrow A.S.
      • et al.
      Developing tools to enhance the use of systematic reviews for clinical care in health systems.
      An overview of the pilot projects
      • Fiordalisi C.
      • et al.
      AHRQ EPC series on improving translation of evidence into practice for the learning health system: introduction.
      and the results of other projects are described in detail in other articles in this series.
      • Barclay C.
      • et al.
      Implementing evidence-based screening and counseling for unhealthy alcohol use with Epic-based electronic health record tools.
      • Adam G.P.
      • et al.
      Web-based interactive presentation of systematic review reports.
      • White C.M.
      • et al.
      Linking evidence reports and performance measures to help learning health systems use new information for improvement.
      AHRQ gave the EPCs flexibility in the design and evaluation of their products. Each EPC could select its own report topic, often in consultation with its partner health system,
      • Fiordalisi C.
      • et al.
      AHRQ EPC series on improving translation of evidence into practice for the learning health system: introduction.
      and develop a product it believed would be responsive to the needs of its partner health system. Because decisions are made throughout multiple levels of the health system, the EPCs also had the flexibility to select specific partners within the health system. The products developed for the pilots ranged from dissemination products (for example, short summaries), to interactive presentations of the data, to implementation products (for example, clinical pathway). The content each product pulled from the EPC evidence reviews varied depending on the intent of the product. More information about each EPC pilot, including examples of the products, are available on the AHRQ website.
      • Agency for Healthcare Research and Quality. Effective Health Care Program
      AHRQ EPC Program Helps Health Systems Use Evidence: Series Overview. May 30, 2018.
      In a parallel effort, the AHRQ EPC Program has convened a Learning Health Systems Panel to hear directly about the needs of learning health systems clinical leaders and to guide the development, implementation, and evaluation of the use of different translational products.
      • Agency for Healthcare Research and Quality. Effective Health Care Program
      Learning Health Systems Panel: Series Overview. Jan 8, 2019.
      The purpose of this paper is to (1) describe AHRQ’s Learning Health Systems Panel, and (2) report on some of the early findings from the panel about challenges health system clinical leaders face in adopting evidence-based practices, and their feedback on the utility of the EPC pilots of the translational products. These findings can inform the EPC Program – and other organizations that produce evidence reviews – to help health systems use evidence to inform operational decisions about design and delivery of health-related services.

      Panel Organization and Composition

      AHRQ’s Learning Health Systems Panel is managed by the American Institutes for Research (AIR) and its partners from Kaiser Permanente (KP) Northwest Region and the University of California, San Francisco (UCSF). The AIR team engaged two national leaders with deep field connections to recruit a representative panel of key stakeholders internal to delivery systems who are intended users of the AHRQ evidence reports. Lucy Savitz, PhD, MBA, vice president for Health Research, KP; and Andrew Bindman, MD, professor of medicine, epidemiology, and biostatistics, UCSF, identified a group that represents a cross-section of health systems in terms of geography, size, services offered, and teaching status. Overall, these health systems have some experience using evidence to improve care, and many of them participate in collaboratives and research networks (for example, the High Value Healthcare Collaborative) to contribute to and disseminate evidence. Table 1 provides a descriptive list of the panel members together with associated organizational affiliations and attributes.

      Objectives of the First Learning Health Systems Panel Meeting

      In January 2019 AHRQ convened the Learning Health Systems Panel for its first in-person meeting. The meeting objectives were to gather feedback from the panel members on (1) challenges to implementing evidence in their health systems and (2) the utility of a set of EPC translational products that were developed through pilot projects to increase health systems’ use of evidence from systematic reviews.
      • Barclay C.
      • et al.
      Implementing evidence-based screening and counseling for unhealthy alcohol use with Epic-based electronic health record tools.
      • Adam G.P.
      • et al.
      Web-based interactive presentation of systematic review reports.
      • White C.M.
      • et al.
      Linking evidence reports and performance measures to help learning health systems use new information for improvement.
      During the meeting, panel members discussed their experiences using evidence, including evidence from systematic reviews. Through a facilitated discussion led by Savitz and Bindman, they described the key challenges they face related to embedding external evidence into—and potentially changing—their existing clinical workflows.
      Next, Jeanne-Marie Guise, MD, MPH, director, Scientific Resource Center for the EPC Program, provided a brief summary of the overall goal of each of the EPC pilot projects to develop, test, and evaluate a new translational product in collaboration with a partnering health system. The summary presentation included screenshots of the products. Guise described their intent and findings from the evaluation of each pilot, as well as overall conclusions across the pilots. Panel members discussed these products and then participated in an activity in which they ranked the top three translational products based on their expected usefulness for translating, sharing, and facilitating use of evidence from systematic reviews within their health system (see Table 2). Panel members then discussed the results of the ranking process and what they found helpful or not helpful about the products.
      Based on an audio recording and notes from the meeting, the AIR team created a meeting summary to describe the discussions. This summary, which was reviewed by AHRQ and shared with the panel members, was used as the basis for the key findings for this article.

      Key Findings

      Challenges Health Systems Face in Adopting Evidence-Based Practices

      Several issues and unmet needs emerged from the panel discussion about the challenges of adopting evidence-based practices within the panel members’ health systems:
      • Lack of evidence addressing health system operational issues to identify and implement the most effective care delivery models. A common “operational issue” shared by the panel relates to how to align internal evidence generated by the learning health system with external evidence to help inform if, how, and in what patient groups to implement an intervention or treatment. For example, internal evidence might include information gathered from a health system’s internal electronic health record (EHRs) about their patients, whereas external evidence might include synthesized information from a systematic evidence review.
      • Need for strategies to overcome barriers clinicians and other frontline staff face in implementing evidence due to two primary factors: (1) the strength and clarity of available evidence may be weak and/or scant, and (2) the perception that the evidence does not apply to their own patient population, particularly vulnerable populations (such as patients with diabetes and food insecurity).
      • Implementing evidence-based changes into existing workflows is complex, beginning with identifying who is responsible for implementing the change, how to communicate the change, and how to operationalize the change in clinical decision support systems.
      • Need to distill and communicate actionable evidence to clinicians and frontline staff that meet them where they are rather than providing broad clinical decision support. Findings from an evidence review can be complex. But health systems want to be able to easily understand and use the findings to inform and change practice.
      • Need for providing the most timely and current evidence because clinicians want the most up-to-date evidence in clinical decision support tools in EHRs. Evidence reviews, particularly systematic reviews, often take more than a year to complete. They lack the timeliness and currency that clinicians desire, and the growing number of retractions from individual studies that may not be incorporated into the product is concerning.

      Feedback on Utility of EPC Translational Products

      Overall, the panel members thought some of the EPC translational products (Table 2) would be helpful to health systems. The panel members selected a dissemination, interactive data visualization, and implementation product as the top three products prioritized through the ranking exercise—demonstrating the value of a broad range of approaches for various users within the health system. The rationale for choosing these three products is described below:
      • One- and Three-Page Summaries: The summaries used different design elements (use of color, placement of program logos, and others), and types of information included (qualitative vs. quantitative) for the purpose of comparing the benefits and harms of treatment options. Panel members liked that the summaries were short and easy to disseminate within their organization. The content was accessible and could be read and grasped in a short amount of time. One panel member shared, “That’s something I can read quickly and I can forward quickly to my cardiology service line lead or my orthopedic service lead with confidence they’d probably look at it. They wouldn’t look at the 1,400-page report. So the access and usability and the fact that I could scan that in 10 to 15 minutes and glean the key insights.”
      • MAGICapp and Tableau Data Visualization: MAGICapp and Tableau are two software programs that allow users to create visual presentations of data (for example, from systematic reviews). The specific types of visualizations vary depending on the program. MAGICapp allowed users to see summary results at a high level and to drill down into specific information as needed. The Tableau visualization allowed the user to slice and dice to see specific types of information as desired (for example, by dividing the data into subgroups corresponding to interests). The panel liked the visual, interactive nature of the product, specifically the ability to drill down to various levels of evidence. They also found the overview of the evidence to be useful and appreciated that they could easily share the data with colleagues to facilitate conversations and decision making. One panel member shared, “You can get the overview and also do the drill-down to the various levels of evidence. It can be very powerful to share with people and you can get to their level. Allows you to be very precise in how you use the data.” Finally, several panel members were familiar with Tableau, which increased its appeal.
      • Clinical Encounter and Health System Decision Aid: The health care system decision aid included information on costs, feasibility, and other implementation considerations that are not typically included in EPC reviews. The encounter decision aid included information pertaining to the benefits and harms of treatment options to facilitate shared decision making. Panel members felt that the clinical encounter aid distills information in a simple manner accessible to both the clinician and patient, and thus can be used to facilitate a discussion. They also remarked that the format of the product is visually appealing. One panel member shared, “This mental health example gives you pause to look at it. The color and plus marks [notations used to display reduction in symptoms and improvement in day-to-day function] really give you that visualization, and I can easily turn this around from the ambulatory setting to the patient. It’s not only a clinical decision-making tool but can also double as a shared decision-making tool.”
      Panel members shared the following reasons for not providing higher rankings for some of the other reviewed translation products:
      • Feeling overwhelmed and uncertain of the usefulness of the product. For example, panel members perceived one of the interactive products as having an overwhelming number of Web links, and they did not think leaders in their health system would click on any of the links. They were not sure that the leaders’ questions would be answered by clicking on the links.
      • Concern that the product could not be used in their setting. For example, with a dissemination product that relied on an in-person delivery mode, the panel members noted that it is difficult to get attendance at seminars. One panel member shared, “The only times we can do . . . seminars are early morning or after work, so the only other time we could to do it is during lunchtime. But patient care can bleed into lunch time. The only way we can get people into a lunchtime situation is if we buy and bribe with food. Also, we’ve had low attendance unless time is held.”

      Implications

      Overall, the feedback from the Learning Health Systems Panel members reinforced previous findings about challenges faced by health systems in translating evidence into practice
      • Balas E.A.
      • Boren S.A.
      Managing clinical knowledge for health care improvement.
      • Marquez C.
      • et al.
      Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study.
      • Bornstein S.
      • et al.
      Putting research in place: an innovative approach to providing contextualized evidence synthesis for decision makers.
      • Damschroder L.J.
      • et al.
      Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.

      Agency for Healthcare Research and Quality. A Framework for Conceptualizing Evidence Needs of Health Systems. Schoelles K, et al. Research White Paper. AHRQ Publication No. 18-EHC004-EF. Dec 2017. Accessed Aug 14, 2019. https://www.ncbi.nlm.nih.gov/sites/books/NBK493738/.

      Agency for Healthcare Research and Quality. Understanding Health-Systems’ Use of and Need for Evidence to Inform Decisionmaking. White CM, et al. Research White Paper. AHRQ Publication No. 17(18)-EHC035-EG. Oct 2017. Accessed Aug 14, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/health-systems-research.pdf.

      and highlighted their particular needs for timely, concise, and actionable evidence that can be easily operationalized into existing clinical workflows. Thus, although they found value in some of the translational products, even the new products developed under the pilots leave some challenges unaddressed, suggesting a need for further improvements. The feedback highlights several key considerations for developing evidence reviews and translational products that are both relevant and actionable for health systems. Specifically, the panel expressed a need for the following:
      • Clear, Concise, Actionable Data to Support Decision Making at Various Levels: Health systems need evidence that is summarized and presented in ways that support decision making for leaders, clinicians and other frontline staff, and patients.
      • Products that Combine Local Data with Findings from Evidence Reports: Health systems need translational products that allow them to integrate their own evidence—generated from their local patient population data—within the context of the findings of an evidence review.
      • Easily-Accessible Translational Products: Health systems need products that (1) can be easily shared across their system, (2) primarily include data in the form of graphs and tables, and (3) offer users the ability to interact with the data to drill down as needed.

      Next Steps

      The AHRQ EPC Program will use the panel’s feedback as part of its ongoing efforts to increase usability and accessibility of EPC evidence reviews and translational products for health systems. The EPC Program is further developing two of the products from the pilots: the one- and three-page summaries and MAGICapp and Tableau data visualization. AHRQ is exploring the use of a brief summary to highlight key findings with future reports and expanding functionality of the AHRQ website to include interactive Web-based visualizations of evidence reviews that will allow readers to select drill-down options most relevant to their information needs.
      In addition, the EPC Program plans to develop at least one new translational product that health systems can use to assess and integrate EPC evidence review findings into clinical care and routine operations. This might include a product that provides summaries specifically tailored for key users of evidence within health systems, or provides expanded contextual information (such as implementation resources and feasibility factors) that learning health systems need to assess the business case for implementing an intervention. A new product(s) will build on key components of the EPC pilots’ translational products found to be most useful to the Learning Health Systems Panel. The team will use an iterative process to develop the new product(s) to ensure that it resonates with and will be useful for the health systems.
      Each Learning Health Systems Panel member will implement and evaluate one of these products. Panel members will be able to select the product: either one of the new products developed by the AIR team or one of the existing EPC products, which includes products from these pilots. Panel members will serve as local implementation champions and receive support and guidance from the AIR team to tailor the implementation to their local context and needs. Through the Learning Health Systems Panel project, the AIR team will also conduct a process evaluation of the implementation to inform the refinement of the products and assess the acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability.
      • Damschroder L.J.
      • et al.
      Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
      These domains are based on two widely respected implementation and evaluation frameworks, combined into a refined Consolidated Framework for Implementation Research model.
      • Fisher E.S.
      • Shortell S.M.
      • Savitz L.A.
      Implementation science: a potential catalyst for delivery system reform.
      • Proctor E.
      • et al.
      Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda.
      The primary data source for the evaluation will be interviews with health system stakeholders who are involved with the implementation.

      Conclusion

      The AHRQ EPC Program is committed to helping health systems use information from evidence reviews to inform operational decisions. As the program continues these efforts, it will actively disseminate this work and make its materials publicly available
      • Agency for Healthcare Research and Quality
      Effective Health Care Program.
      so that other health systems can use the products and materials in their own settings to improve the delivery and quality of patient care.

      Funding and Disclaimers

      The Learning Health System Panel project was funded under Contract No. HHSP233201500014I from the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services (HHS).
      The authors of this manuscript are responsible for its content. Statements in the manuscript do not necessarily represent the official views of or imply endorsement by AHRQ or HHS.

      Acknowledgments

      The authors would like to thank the members of the AHRQ Evidence-based Practice Center (EPC) Learning Health Systems Panel for their ongoing feedback and commitment to helping health systems use information from evidence reviews to inform operational decisions.

      Conflicts of Interest

      All authors report no conflicts of interest.

      References

        • Agency for Healthcare Research and Quality. About Learning Health Systems
        Accessed Aug. Mar 2019; 14: 2019
        • Agency for Healthcare Research and Quality. AHRQ Research Summit on Learning Health Systems
        Executive Summary. Sep 15, 2017.
        Accessed Aug. 2019; : 14
        • Agency for Healthcare Research and Quality. Supporting the Next Generation of Learning Health Systems Researchers
        Grants Awarded Under the AHRQ-PCORI Institutional Mentored Career Development Program (K12). Sep 2018.
        Accessed Aug. 2019; : 14
        • Tabak R.G.
        • et al.
        Bridging research and practice: models for dissemination and implementation research.
        Am J Prev Med. 2012; 43: 337-350
        • Morris Z.S.
        • Wooding S.
        • Grant J.
        The answer is 17 years, what is the question: understanding time lags in translational research.
        J R Soc Med. 2011; 104: 510-520
        • Balas E.A.
        • Boren S.A.
        Managing clinical knowledge for health care improvement.
        Yearb Med Inform. 2000; 9: 65-70
        • Marquez C.
        • et al.
        Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study.
        Implement Sci. 2018 Jun 22; 13: 84
        • Vogel J.P.
        • et al.
        Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey.
        Health Res Policy Syst. 2013 May 24; 11: 19
        • Guise J.M.
        • Savitz L.A.
        • Friedman C.P.
        Mind the gap: putting evidence into practice in the era of learning health systems.
        J Gen Intern Med. 2018; 33: 2237-2239
      1. Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Jan 2014. Accessed Aug 14, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf.

        • Borsky A.E.
        • et al.
        AHRQ Evidence-based Practice Center Program—applying the knowledge to practice to data cycle to strengthen the value of patient care.
        J Hosp Med. 2019; 14: 311-314
        • Agency for Healthcare Research and Quality. Effective Health Care Program
        AHRQ EPC Program Helps Health Systems Use Evidence: Series Overview. May 30, 2018.
        Accessed Aug. 2019; : 14
        • Morrow A.S.
        • et al.
        Developing tools to enhance the use of systematic reviews for clinical care in health systems.
        BMJ Evid Based Med. 2018; 23: 206-209
        • Fiordalisi C.
        • et al.
        AHRQ EPC series on improving translation of evidence into practice for the learning health system: introduction.
        Jt Comm J Qual Patient Saf. 2019; 45: 558-565
        • Barclay C.
        • et al.
        Implementing evidence-based screening and counseling for unhealthy alcohol use with Epic-based electronic health record tools.
        Jt Comm J Qual Patient Saf. 2019; 45: 566-574
        • Adam G.P.
        • et al.
        Web-based interactive presentation of systematic review reports.
        Jt Comm J Qual Patient Saf. 2019; 45: 629-638
        • White C.M.
        • et al.
        Linking evidence reports and performance measures to help learning health systems use new information for improvement.
        Jt Comm J Qual Patient Saf. 2019; 45: 705-709
        • Agency for Healthcare Research and Quality. Effective Health Care Program
        Learning Health Systems Panel: Series Overview. Jan 8, 2019.
        Accessed Aug. 2019; : 14
        • Bornstein S.
        • et al.
        Putting research in place: an innovative approach to providing contextualized evidence synthesis for decision makers.
        Syst Rev. 2017 Nov 2; 6: 218
        • Damschroder L.J.
        • et al.
        Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
        Implement Sci. 2009 Aug 7; 4: 50
      2. Agency for Healthcare Research and Quality. A Framework for Conceptualizing Evidence Needs of Health Systems. Schoelles K, et al. Research White Paper. AHRQ Publication No. 18-EHC004-EF. Dec 2017. Accessed Aug 14, 2019. https://www.ncbi.nlm.nih.gov/sites/books/NBK493738/.

      3. Agency for Healthcare Research and Quality. Understanding Health-Systems’ Use of and Need for Evidence to Inform Decisionmaking. White CM, et al. Research White Paper. AHRQ Publication No. 17(18)-EHC035-EG. Oct 2017. Accessed Aug 14, 2019. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/health-systems-research.pdf.

        • Fisher E.S.
        • Shortell S.M.
        • Savitz L.A.
        Implementation science: a potential catalyst for delivery system reform.
        JAMA. 2016 Jan 26; 315: 339-340
        • Proctor E.
        • et al.
        Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda.
        Adm Policy Ment Health. 2011; 38: 65-76
        • Agency for Healthcare Research and Quality
        Effective Health Care Program.
        Accessed Aug. 2019; : 14