Advertisement

Developing the Safer Dx Checklist of Ten Safety Recommendations for Health Care Organizations to Address Diagnostic Errors

Open AccessPublished:August 10, 2022DOI:https://doi.org/10.1016/j.jcjq.2022.08.003

      Background

      Most health care organizations (HCOs) find diagnostic errors hard to address. The research team developed a checklist (the Safer Dx Checklist) of 10 high-priority safety practices HCOs can use to conduct a proactive risk assessment to address diagnostic error.

      Methods

      First, the team identified potential practices based on reviews of recent literature, reports by national and international organizations, and interviews with quality/safety leaders. Then a Delphi panel was conducted, followed by an online expert panel, to prioritize 10 practices. The prioritization process considered impact on safety and feasibility of practice implementation within a one- to three-year time frame. Finally, cognitive walkthroughs were conducted for a face-validity check with end users. The team also conducted content analysis in each step to look for themes that influenced prioritization or checklist implementation.

      Results

      A total of 71 practices for prioritization were identified through the Delphi panel of 28 experts; 65% of participants reached consensus on 28 practices. A multidisciplinary panel of 10 experts helped prioritize and refine the top 10 practices, which were then developed into a checklist paired with implementation guidance. Practices included themes related to creating organizational and leadership accountability for improving diagnosis, including patients in diagnostic safety work, and developing and implementing organizational infrastructure for measurement and improvement activities. Qualitative analysis revealed insights for implementation. End users at three different HCOs helped refine implementation guidance for the checklist.

      Conclusion

      The researchers identified 10 safety practices to help organizations conduct a proactive, systematic assessment of risks to timely and accurate diagnosis. The Safer Dx Checklist can enable HCOs to begin implementing strategies to address diagnostic error.
      Reduction of diagnostic errors (such as missed, delayed or wrong diagnoses) is a major challenge for health care organizations (HCOs) striving to improve patient safety.
      • Graber ML
      • et al.
      The next organizational challenge: finding and addressing diagnostic error.
      The National Academies of Sciences, Engineering, and Medicine report Improving Diagnosis in Health Care recommends HCOs develop dedicated programs to address diagnostic errors and reduce harm.

      Institute of Medicine. Improving Diagnosis in Health Care. Washington, DC: National Academies Press, 2015. Accessed Aug 15, 2022. http://iom.nationalacademies.org/Reports/2015/Improving-Diagnosis-in-Healthcare.aspx.

      Diagnostic errors share several underlying common themes with other types of patient safety problems, but multiple complex cognitive and system factors make them particularly challenging to address.
      • Graber ML
      • Wachter RM
      • Cassel CK.
      Bringing diagnosis into the quality and safety equations.
      For instance, most analyses of diagnostic errors involve discussions of clinical uncertainty, natural evolution of diagnosis, and multiple types of cognitive errors. This is exacerbated by system vulnerabilities such as time and productivity pressures.
      • Zwaan L
      • Singh H.
      The challenges in defining and measuring diagnostic error.
      The complexity of defining and measuring diagnostic errors poses challenges in developing solutions compared to other types of patient safety concerns, such as medication or surgical errors.
      • Schiff GD
      • et al.
      Diagnostic error in medicine: analysis of 583 physician-reported errors.
      Thus, HCOs need pragmatic guidance on interventions to address diagnostic errors.
      • Singh H
      • Upadhyay DK
      • Torretti D.
      Developing health care organizations that pursue learning and exploration of diagnostic excellence: an action plan.
      HCOs can prevent diagnostic errors by identifying structures and processes that are at risk and implementing interventions to address them. A proactive, systematic assessment of risks and vulnerabilities related to diagnosis can help HCOs identify diagnostic safety risks before harmful incidents occur and learn about potential pockets of excellence.
      • Carson-Stevens A
      • Donaldson L
      • Sheikh A.
      The rise of Patient Safety-II: should we give up hope on Safety-I and extracting value from patient safety incidents?.
      This approach complements existing safety improvement approaches focused on retrospective analysis of unsafe and suboptimal care. We used a multimethod approach to develop an expert consensus-based checklist of 10 high-priority safety practices HCOs can use to improve the safety of the diagnostic process. HCOs could also use this checklist to ensure that their infrastructure and processes support safe and timely diagnosis.

      Methods

      We used a multistep procedure to identify and select practices that could improve diagnostic safety in real-world settings.
      • Linstone HA
      • Turoff M
      The Delphi Method: Techniques and Applications.
      • Hasson F
      • Keeney S
      • McKenna H.
      Research guidelines for the Delphi survey technique.
      • Howell AM
      • et al.
      International recommendations for national patient safety incident reporting systems: an expert Delphi consensus-building process.
      • Jones J
      • Hunter D.
      Consensus methods for medical and health services research.
      The main steps included literature reviews, analysis of interviews, an online Delphi panel, a synchronous expert panel (virtual due to COVID-19), and a final face-validity check with end users. For literature review, we evaluated peer-reviewed literature, including (1) narrative reviews and systematic reviews on diagnostic error, including interventions to reduce them; (2) models, papers, commentaries, and perspectives focused on organizational strategies to improve diagnosis; and (3) reports from several national and international bodies, including the National Academies of Sciences, Engineering, and Medicine; the National Quality Forum; and the World Health Organization (WHO).

      Institute of Medicine. Improving Diagnosis in Health Care. Washington, DC: National Academies Press, 2015. Accessed Aug 15, 2022. http://iom.nationalacademies.org/Reports/2015/Improving-Diagnosis-in-Healthcare.aspx.

      ,
      • Singh H
      • Upadhyay DK
      • Torretti D.
      Developing health care organizations that pursue learning and exploration of diagnostic excellence: an action plan.
      ,

      ECRI. Deep Dive: Safe Ambulatory Care: Strategies for Patient Safety & Risk Reduction. 2019. Accessed Aug 15, 2022. https://www.ecri.org/landing-ambulatory-care-deep-dive.

      • Singh H
      • Sittig DF.
      Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework.

      National Quality Forum. Improving Diagnostic Quality and Safety: Final Report. Sep 19, 2017. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=85969.

      National Quality Forum. Advancing Chief Complaint-Based Quality Measurement: Final Report. Jun 24, 2019. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=90454.

      National Quality Forum. Improving Diagnostic Quality and Safety/Reducing Diagnostic Error: Measurement Considerations: Final Report. Oct 6, 2020. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=93907.

      Health Research & Educational Trust. Improving Diagnosis in Medicine: Diagnostic Error Change Package. Sep 2018. Accessed Aug 15, 2022. https://www.improvediagnosis.org/wp-content/uploads/2018/11/improving-diagnosis-in-medicine-change-package-11-8.pdf.

      Agency for Healthcare Research and Quality
      Operational Measurement of Diagnostic Safety: State of the Science.
      • McDonald KM
      • et al.
      Patient safety strategies targeted at diagnostic errors: a systematic review.
      • Abimanyi-Ochom J
      • et al.
      Strategies to reduce diagnostic errors: a systematic review.
      • Perry MF
      • et al.
      The diagnostic error index: a quality improvement initiative to identify and measure diagnostic errors.
      • Singh H
      • et al.
      System-related interventions to reduce diagnostic errors: a narrative review.
      • Graber ML
      • et al.
      Cognitive interventions to reduce diagnostic error: a narrative review.
      The Delphi panel exercise was informed by recommendations, ideas, and organizational practices based on the preceding environmental scan and qualitative data from previously conducted structured interviews with 32 quality and safety leaders across the United States.
      • Giardina TD
      • et al.
      Creating a learning health system for improving diagnostic safety: pragmatic insights from US health care organizations.
      This latter study provided insights into several foundational building blocks related to learning health systems that could inform organizational efforts to reduce diagnostic error. These building blocks included promoting an organizational culture specific to diagnostic safety, using science and informatics to improve measurement and analysis, leadership incentives to build institutional capacity to address diagnostic errors, and patient engagement in diagnostic safety.
      Members of the research team conducted content analysis of these resources to identify concepts for practice development. An iterative multistage development process helped identify meaningful recommendations to develop potential practices for the Delphi survey. Subsequent refinement identified a final list of 10 high-priority practices that were then converted into checklist items. Because the checklist development process drew heavily on a conceptual framework for diagnostic errors used for measurement and reduction of diagnostic errors (the Safer Dx framework),
      • Singh H
      • Sittig DF.
      Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework.
      we titled the checklist the Safer Dx Checklist.

      Delphi Panel

      Iterative review of multiple resources to identify potential practices and subsequent refinement and consolidation identified 71 practices that were considered pragmatic and useful for improving diagnostic safety. Based on our current and prior work, six major domains were developed for the Delphi questionnaire: (1) practices related to leadership commitment and personnel investment for diagnostic safety, (2) practices related to creating a diagnostic safety culture to encourage reporting and learning, (3) practices related to developing and implementing infrastructure for measurement and improvement activities, (4) practices related to engaging clinicians and staff in diagnostic safety activities, (5) practices related to patient engagement in diagnostic safety, and (6) practices related to specific resources and activities to improve the diagnostic process.
      We sent invitations to 32 nationally selected experts via e-mail to participate in the two-round online Delphi panel. These 32 experts included 16 physicians in various specialties; 16/32 were female; 4/32 were patient/family advocates. Experts represented the fields of quality and patient safety, diagnostic error research, clinical informatics, human factors, and clinical operations and health systems leadership.

      Round One of Delphi

      The online Delphi was administered using Qualtrics (Qualtrics LLC, Provo, Utah) from August through October 2020. The experts were asked to use a five-point Likert scale to state their agreement level for each practice. We also provided an option to add free-text comments for all practices. We asked our experts to think about health care organizations broadly, including inpatient and/or ambulatory settings, and consider the following two prioritization criteria as they rated each practice:
      • 1.
        Feasibility: Consider if it is feasible for many US health care organizations to implement this practice in the next one to three years.
      • 2.
        Potential to affect diagnostic safety: Consider if the implemented practice has potential to lead to significant improvement of diagnostic safety or harm reduction associated with diagnostic errors at the health care organization.
      On the five-point scale, both “agree” or “strongly agree” to rate a practice were classified as “agree” for analysis purposes.
      • Howell AM
      • et al.
      International recommendations for national patient safety incident reporting systems: an expert Delphi consensus-building process.
      A “neutral,” “disagree,” or “strongly disagree” response by the expert was classified as “disagree.” Using standard Delphi method criteria, 70% expert agreement was required for a practice to achieve consensus.
      • Howell AM
      • et al.
      International recommendations for national patient safety incident reporting systems: an expert Delphi consensus-building process.
      ,
      • Fink A
      • et al.
      Consensus methods: characteristics and guidelines for use.

      Round Two of Delphi

      Experts were provided feedback about Round One, such as what percentage of experts agreed/strongly agreed, were neutral, or disagreed/strongly disagreed with each practice.
      • Graber ML
      • Wachter RM
      • Cassel CK.
      Bringing diagnosis into the quality and safety equations.
      They were also provided with anonymized comments on practices from other experts as well as statistical aggregates, including mean, median, and their own ratings for individual practices from Round One. All 71 practices from the first round were included in the second round, and experts were asked to repeat the process as described in Round One. Experts were able to either keep their original ratings or change them based on the comments and ratings of the practices from their peers. The results of the second round were also analyzed as described earlier to assess consensus.
      We conducted content analysis of the comments on both rounds to evaluate themes that reflected prioritization logic or future implementation of practices.

      Expert Panel

      The aim of the synchronous expert panel was to critically review the results of the Delphi panel and select the top 10 practices to improve diagnostic safety. We selected 10 multidisciplinary experts with experience and expertise in diagnostic safety, of which 6 experts had also participated in the Delphi (for a list of experts and affiliations, see Appendix 1, available in online article). All experts were sent findings of the Delphi process and the list of practices under consideration to review prior to the meeting.
      The expert meeting was conducted virtually as two half-day sessions one week apart in May 2021. Prior to the meeting, we reviewed the feedback and comments collected during the Delphi process and highlighted duplicative statements that could potentially be merged with one another. During the expert meeting, we reviewed each practice selected after Round Two of Delphi to identify the top 10 while posing three questions for experts to consider while prioritizing these practices:
      • 1.
        Wording/comprehension: Are the items appropriately worded so that an HCO's multidisciplinary team can answer them and find them useful for improvement?
      • 2.
        Feasibility: Would HCOs be able to implement this practice in the next one to three years?
      • 3.
        Potential to affect diagnostic safety: Can the practice lead to significant reduction of diagnostic harm?
      To enable sorting into high, medium, or low priority for additional consideration, we used Trello (Atlassian Corporation PLC, New York City, NY), a freely available online program for organizing lists and “cards” on a virtual whiteboard (see Figure 1). When there was disagreement, we used majority rule to achieve consensus. Each practice listed as medium was later discussed, and those listed as low were discarded. All practices that were listed as high at the end were reviewed for overlap and verbiage. After all the practices were categorized as high priority, all were reevaluated for consideration. In addition, the final top 10 list was reassessed for comprehensiveness to ensure that they addressed previously identified risk areas in the literature or areas experts deemed high priority. We invited experts to contribute ideas or innovations from their own disciplines, sites, or experiences as examples to facilitate implementation. We also solicited expert input to develop a supplemental implementation guidance document to accompany the checklist that provided instructions on how to use the checklist.
      Figure 1
      Figure 1Shown here is a screenshot from Trello, a Web platform for sorting practices into high, medium, and low.

      Preimplementation Cognitive Walkthrough

      As a final preimplementation check with a group of likely end users, we conducted a cognitive walkthrough
      • Bradford A
      • et al.
      Development and usability testing of the AHRQ Common Formats to capture diagnostic safety events.
      ,
      • Lyon AR
      • et al.
      The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability.
      with chief quality/safety officers (all physicians; for a list of names and affiliations, see Appendix 1) from three different types of health care organizations (which had not participated in this project thus far) to determine how the checklist fits with their current practices and state of diagnostic safety. Walkthroughs were based on a short, open-ended, semistructured guide that included questions related to whether the checklist added value to their safety efforts, if it was readily implementable and easy to understand, and, finally, who would lead the implementation process. Rapid content analysis was performed with the help of notes taken during the interview. A qualitative researcher did the initial content analysis that was followed by multiple team discussions to finalize the emerging themes.
      This exercise provided unique insights into how this checklist and implementation guidance document could be used. This step was also geared toward determining whether the practices were perceived as clear and important. We used the input to clarify or refine certain operational concepts to exemplify each practice. Both the expert panel and cognitive walkthroughs were recorded on Zoom. We conducted content analysis of these data and used meeting notes to synthesize main findings.

      Results

      Delphi Panel

      We identified 71 practices for subsequent prioritization in the Delphi panel. Of the 32 experts we invited, 29 agreed to participate in the two-round online Delphi (90.6% response rate). Twenty-eight of the 29 experts completed Round One (completion rate 96.6%), and 25 of 29 experts went on to complete the second round of the Delphi survey (completion rate 86.2%). At the end of Round Two, 70% consensus was reached on 20 practices. When we used a more flexible 65% cutoff, consensus was reached on 28 practices, which were then included for further evaluation (see Table 1). Participants provided additional context for each of the practices, including comments about feasibility, importance of the practice, and redundancy or synergy with other practices.
      Table 1Post Delphi Practices Meeting 65% Agreement Criteria
      Section 1: Practices Related to Leadership Commitment and Personnel Investment for Diagnostic SafetyExpert Agreement (%)
      Practice 1. Health care organizations (HCOs) and senior leadership/C-suite consistently share (in plain language) data related to diagnostic errors to their governance boards, such as narrative patient stories related to misdiagnosis, action plans, and, when possible, additionally quantified and stratified data to measure and track diagnostic errors.72
      Practice 2.
      Practices that met 65% to 70% consensus.
      HCOs build an accountability framework for their leadership to ensure system and process changes are made in response to learnings from data analysis.
      68
      Section 2: Practices Related to Creating a Diagnostic Safety Culture to Encourage Reporting and LearningExpert Agreement (%)
      Practice 3. HCOs promote both accountability and nonpunitive culture to encourage providers to share diagnostic safety issues and concerns without fear of retribution and improve psychological safety for providers.80
      Practice 4. HCOs close the loop with providers who report cases to let them know of corrective actions or steps taken to prevent recurrence in a timely and effective manner.76
      Practice 5. HCOs use morbidity and mortality (M&M) and other interdepartmental and/or interprofessional quality conferences/reviews as sources for identifying and learning from diagnostic errors. (Some of these usually do not make their way to incident reporting systems or risk management.)76
      Practice 6. HCOs create feedback loops to referring organizations and/or clinicians on diagnostic and treatment-related outcomes of their patients. For instance, they have an established mechanism for capturing, measuring, and providing feedback to the diagnostic team when there is a significant change in diagnosis.72
      Practice 7.
      Practices that met 65% to 70% consensus.
      HCOs implement systems that allow clinicians (e.g., emergency department [ED] physicians, primary care practitioners, advanced practice nurses) to efficiently and reliably follow up on patients they cared for (e.g., follow up on admitted patients to learn if diagnosis changed or evolved).
      68
      Section 3: Practices Related to Developing and Implementing Infrastructure for Measurement and Improvement ActivitiesExpert Agreement (%)
      Practice 8. HCOs use an electronic health record (EHR) data warehouse or other EHR query tools to apply trigger algorithms to improve diagnostic safety. For instance, HCOs implement tracking programs to identify and act upon abnormal test results that have not been followed up.96
      Practice 9. HCOs’ quality and safety teams work with frontline clinicians to understand contributory factors for diagnostic errors and/or missed opportunities and to give them an opportunity to review cases undergoing safety analysis.92
      Practice 10. HCOs include multidisciplinary perspectives in analysis of diagnostic errors and consider both human factors and cognitive elements (e.g., modified use of root cause analysis and other techniques).84
      Practice 11. HCOs evaluate patient complaints to look for and address missed opportunities in diagnosis.76
      Section 4: Practices Related to Engaging Clinicians and Staff in Diagnostic Safety ActivitiesExpert Agreement (%)
      Practice 12.
      Practices that met 65% to 70% consensus.
      HCOs use online learning modules, videos, simulated case-based learning, and virtual learning platforms for competency-based training on diagnostic error recognition and reduction as well as to systematically increase awareness on diagnostic errors.
      68
      Practice 13.
      Practices that met 65% to 70% consensus.
      HCOs identify and evaluate work-system/environmental factors that place a cognitive burden on clinicians and implement measures to reduce these factors.
      68
      Section 5: Practices Related to Patient Engagement in Diagnostic SafetyExpert Agreement (%)
      Practice 14. HCOs share all test results with patients through online portals and other mechanisms if needed.88
      Practice 15. HCOs engage patients to proactively seek test results (i.e., caution patients “no news is not good news.”)80
      Practice 16. HCOs create a culture where patients are encouraged and educated on how to report when they have concerns or things are not right.76
      Practice 17. HCOs provide patients access to review clinicians’ notes about themselves to check for accuracy and errors.76
      Practice 18.
      Practices that met 65% to 70% consensus.
      HCOs engage with patients and/or patient family advisory councils in discussions of diagnostic errors and related serious safety events (e.g., include patients in root cause analysis).
      68
      Section 6: Practices Related to Improving the Diagnostic ProcessExpert Agreement (%)
      Practice 19. Organizations provide resources to facilitate patient understanding of diagnosis, including resources such as translation services and having someone to help in situations related to patients with low health literacy.92
      Practice 20. Organizations encourage and facilitate clinicians and laboratory/radiology professionals to interact directly with one another in cases that pose diagnostic challenges.76
      Practice 21. HCOs implement robust systems and processes to improve coordination and communication during handoffs and transitions (e.g., between inpatient care teams). For instance, certain patients undergoing transitions of care, including discharges from hospital and ED, are tracked.76
      Practice 22. HCOs implement standardized methods for addressing diagnosis in handoffs and transitions of care, such as the forthcoming TeamSTEPPS® program focused on diagnostic error.
      See TeamSTEPPS® for Diagnosis Improvement (Agency for Healthcare Research and Quality, https://www.ahrq.gov/teamstepps/diagnosis-improvement/index.html, accessed August 15, 2022).
      72
      Practice 23. Radiologists are available 24/7 to read urgent diagnostic imaging studies in real time.72
      Practice 24.
      Practices that met 65% to 70% consensus.
      To close the loop on communication of abnormal test results, including incidental findings, HCOs use existing resources, such as self-assessment using ONC SAFER Guide on Test Results Reporting and Follow-Up.
      See the SAFER (Safety Assurance Factors for EHR Resilience) guide, Self-Assessment: Test Results Reporting and Follow-Up (Office of the National Coordinator for Health Information Technology, https://www.healthit.gov/sites/default/files/safer_test_results_reporting.pdf, accessed August 15, 2022.
      68
      Section 7: Practices Related to EHR/InformaticsExpert Agreement (%)
      Practice 25. HCOs optimize the use of EHR for validated and accurate features that support diagnosis.84
      Practice 26. HCOs uses an interoperable and certified EHR to participate in health information exchange with outside institutions to support diagnostic quality (e.g., exchange test results and documentation related to diagnoses).72
      Practice 27.
      Practices that met 65% to 70% consensus.
      Technologies and information resources that support diagnosis, such as validated Web-based decision support tools and online knowledge reference materials, are available to all providers to aid differential diagnosis.
      68
      Practice 28.
      Practices that met 65% to 70% consensus.
      The organization has a system in place to review and correct inaccurate diagnoses in the EHR.
      68
      low asterisk Practices that met 65% to 70% consensus.
      See TeamSTEPPS® for Diagnosis Improvement (Agency for Healthcare Research and Quality, https://www.ahrq.gov/teamstepps/diagnosis-improvement/index.html, accessed August 15, 2022).
      See the SAFER (Safety Assurance Factors for EHR Resilience) guide, Self-Assessment: Test Results Reporting and Follow-Up (Office of the National Coordinator for Health Information Technology, https://www.healthit.gov/sites/default/files/safer_test_results_reporting.pdf, accessed August 15, 2022.
      A qualitative content analysis of the free-text comments from both rounds revealed several topics essential for successful implementation of practices. Frequently mentioned topics included building an institutional culture specific to diagnostic safety that involved clinicians and patients, leadership involvement to facilitate practice implementation, a well laid-out system to implement the practices that has clinical and operational support, and the need for resources—finances as well as time/personnel. Several participants indicated that these factors were essential for any change management initiative related to diagnosis in any health care system. Table 2 lists ideas or insights related to implementation captured from the Delphi.
      Table 2Selected Quotes on Ideas or Insights on Implementation of Practices from Each Step
      StagesSelected Quotes on Ideas or Insights on Implementation of Practices
      Delphi—

      Themes:

      Leadership involvement

      Institutional culture

      Clinician and patient

      System to implement—operational support
      “Crucial to have leadership support.”
      “Must include Board enthusiasm and highest-level executive to be successful.”
      “System interventions are most effective if they start at the top. If Dx errors are important to the leaders, they will be important to everyone.”
      “Culture building/setting expectations by C-suite = essential.”
      “The ability to make significant changes within the HCO requires a top-down cultural shift.”
      “Accountability to implement change and improvement is essential.”
      Delphi—

      Themes:

      Feasibility

      Resources (Time/Money)
      “Worry about [practice] feasibility, as this is an additional burden unless diagnostic quality is incorporated as one additional item on the quality spectrum.”
      “Most organizations will not have the resources to fund this.”
      “Resources, training, interprofessional staff are needed to tackle this problem.”
      “This is becoming less and less feasible—especially after COVID-19 financial impact.”
      “This requires funding and recognition by the HCOs of the need to prioritize Dx safety.”
      “Feasibility will be challenged by competing revenue-generating activities and requires training of physicians for self-directed diagnostic error improvement work.”
      “One of the biggest problems with Dx error is denial (and lack of follow-up). Most errors are never detected or appreciated. Many are covered up out of fear of litigation.”
      “I think leaders are going to be focused on COVID-19 and finances in the next one to three years.”
      “Feasibility challenging due to cost/investment in this highly specialized workforce; however, return on investment is high if embraced by organization and able to relate meaningfully to others in health care workforce to effect change.”
      Expert Panel“I think this is highly important; without an accountability framework, we can identify problems but they never get fixed.”
      “Backbone of any Q&S program is a good safety culture…how do you give voice to what I think is under-representative in any current structure and that is Dx safety.”
      “I wonder when we say ‘go implement this,’ it will not be feasible for most, because I'm not sure who's got this mastered anyway.”
      “Patients should be able to put inputs in their portals, should be able to react and flag errors, add info as well—that is critical next step—getting patients involved at all is a challenge at lot of places.”
      End User Validation“You will want someone who will have enough influence and power to make the board care about these things, but you also want the right experts to make sure the correct things happen, and the right processes are implemented.”
      “Could see this being a project that systemwide council could take on, but a lot would be implemented at each facility; each facility would have to have their own team.”
      “All are big undertakings; would need to implement across a timeline.”
      “Organizations still struggle with patient and family engagement/involvement (legal concerns).”
      “Be more prescriptive about who should use the checklist (CMO, CQO, Chief Legal Officer, etc.).”
      “Don't use word ‘standard’ practice in scoring recommendations—maybe just remove the word—well-known and well-documented practice that happens all the time.”
      “Add something about ensuring this initiative/checklist is presented to the board; you engage your board quality committee in completing the checklist, board needs to hold management accountable, e.g., ‘engage your board in a continuous journey.’”
      “Resistance to change—clinicians do not want to acknowledge that they made diagnostic errors.”
      “Need bandwidth to take this on, especially if it is competing with other organizational priorities (pandemic, who is developing operational plan—especially in smaller organizations—the time to do the work, budget cycles, surveys, expanding other services, etc.)
      “Would take a year or two to implement.”
      “The 10 items are disparate; there are a lot of different stakeholders, and a lot of people would need to own different sections.”
      HCO, health care organization; Q&S, quality and safety; CMO, chief medical officer; CQO, chief quality officer.

      Expert Panel

      The panel consisted of 10 participants (see Appendix 1) who discussed and debated 28 practices from the Delphi: 20 that had reached 70% consensus and the additional 8 with 65% consensus. Despite being virtual, our group was very engaged and used the Chat and Raise Hand features to interact in addition to conversations. We spent up to 15 minutes discussing a practice. A member of the team acted as a time manager to effectively manage time per practice. Discussions helped modify the language of each of the high-priority practices and clarified their focus. Certain practices had overlapping goals and were combined. Wording from certain practices were rephrased to clarify concepts.
      The panel recommended the checklist items be scored according to whether they were fully implemented, partially implemented, or not implemented. In addition to a final list of top 10 (see Table 3), these discussions also helped generate one to two pragmatic examples for each recommended practice to facilitate their implementation. We also used some of the discussion points to develop a one-page companion implementation guidance document to pair with the checklist (see Appendix 1).
      Table 3Ten High-Priority Practices for Diagnostic Excellence
      1.Health care organization leadership builds a “board-to-bedside” accountability framework that includes structure, capacity, transparency, time, and resources to measure and improve diagnostic safety.
      2Health care organization promotes a just culture and creates a psychologically safe environment that encourages clinicians and staff to share opportunities to improve diagnostic safety without fear of retribution.
      3.Health care organization creates feedback loops to increase information flow about patients’ diagnostic and treatment-related outcomes. These loops include clinicians and external organizations and establish mechanisms for capturing, measuring, and providing feedback to the diagnostic team about patients’ subsequent diagnosis and clinical outcomes.
      4.Health care organization includes multidisciplinary perspectives to understand and address contributory factors in analysis of diagnostic safety events. These perspectives include human factors, informatics, information technology system design, and cognitive elements.
      5.Health care organization actively seeks patient and family feedback to identify and understand diagnostic safety concerns and addresses concerns by codesigning solutions.
      6.Health care organization encourages patients to review their health records and has mechanisms in place to help patients understand, interpret, and/or act upon diagnostic information.
      7.Health care organization prioritizes equity in diagnostic safety efforts by segmenting data to understand root causes and implementing strategies to address and narrow equity gaps.
      8.Health care organization has in place standardized systems and processes to encourage direct, collaborative interactions between treating clinical teams and diagnostic specialties (e.g., laboratory, pathology, radiology) in cases that pose diagnostic challenges.
      9.Health care organization has in place standardized systems and processes to ensure reliable communication of diagnostic information between care providers and with patients and families during handoffs and transitions throughout the diagnostic journey.
      10.Health care organization has in place standardized systems and processes to close the loop on communication and follow up on abnormal test results and referrals.

      Preimplementation Validation with End Users

      We conducted the final preimplementation review with chief quality/safety officers from three HCOs that did not have active diagnostic safety programs. The findings were useful to inform implementation guidance. For instance, it helped confirm that the first implementation step for an organization should be to identify a senior leader (for example, chief quality officer, chief patient safety officer, chief medical officer, other clinician with oversight of quality) who can serve as the champion for diagnostic excellence, oversee implementation, and monitor implementation and performance of the related action plan. The users also recommended regular checkpoints for follow-up that include annual review of the checklist and periodic monitoring and revisions to the action plan to advance improvements.
      All three noted that the checklist would serve as a robust road map for HCOs that are early in their journey toward diagnostic excellence and guide HCOs that are already doing diagnostic safety work in further advancing their progress. They believed that although most checklist items would integrate well into existing patient safety systems and processes, the usual challenges should be expected, such as resistance to change, clinician uneasiness to tackle diagnostic errors without a just culture, and the need to connect the clinician's purpose to this initiative. Participants agreed with the 10 prioritized practices and provided input on how some of the examples could be modified for clarification and how to make implementation guidance more user-friendly.
      Ideas or insights related to implementation are summarized in Table 2. Several common themes on implementation were observed. All steps unequivocally highlighted the importance of engaging the top-level leadership to create a top-down cultural shift that sets expectations, maintains accountability, and helps design effective system interventions. Moreover, experts discussed the role of the board to hold the management accountable, and the management in turn holding the implementers and frontline staff accountable while supporting them with resources. Some concerns about the feasibility of these practices arose in the context of the COVID-19 pandemic, but experts suggested that despite challenges due to high cost/investment, the return on investment to an organization would be high if the organization embraced the initiative. Benefits could include better safety culture associated with acknowledging and reporting errors.
      The final list of top 10 practices is presented in Table 3. The full checklist is included as Appendix 1 and is freely available online.

      Baylor College of Medicine; Institute for Healthcare Improvement. The Safer Dx Checklist: 10 High-Priority Organizational Practices for Diagnostic Excellence.. https://www.ihi.org/resources/Pages/Tools/safer-diagnostic-checklist.aspx.

      Discussion

      Using a multimethod approach, we developed a checklist of top 10 safety practices for HCOs to implement to address diagnostic error. Each of the original six content domains included at least one practice that reached consensus. However, most of the practices fell under two domains: practices related to creating a diagnostic safety culture to encourage reporting and learning, and practices related to developing and implementing infrastructure for measurement and improvement activities. Because most HCOs are resource constrained and at any given time have several different competing priorities, this list of high-priority practices could serve as a starting point for self-assessment to guide their efforts to promote safe and timely diagnosis.
      The Safer Dx Checklist offers pragmatic evidence-based strategies to help HCOs address diagnostic safety. Despite recent reports from the National Academies of Sciences, Engineering, and Medicine

      Institute of Medicine. Improving Diagnosis in Health Care. Washington, DC: National Academies Press, 2015. Accessed Aug 15, 2022. http://iom.nationalacademies.org/Reports/2015/Improving-Diagnosis-in-Healthcare.aspx.

      ; the National Quality Forum

      National Quality Forum. Improving Diagnostic Quality and Safety: Final Report. Sep 19, 2017. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=85969.

      ,

      National Quality Forum. Improving Diagnostic Quality and Safety/Reducing Diagnostic Error: Measurement Considerations: Final Report. Oct 6, 2020. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=93907.

      ; WHO

      World Health Organization. Global Patient Safety Action Plan 2021–2030: Towards Eliminating Avoidable Harm in Health Care. 2021. Accessed Aug 15, 2021. https://apps.who.int/iris/rest/bitstreams/1360307/retrieve.

      ; and the Organisation for Economic Co-operation and Development

      Organisation for Economic Co-operation and Development. The Economics of Patient Safety: Strengthening a Value-Based Approach to Reducing Patient Harm at National Level. Slawormirski L, Auraaen A, Klazinga N. Mar 2017. Accessed Aug 15, 2022. https://www.oecd.org/els/health-systems/the-economics-of-patient-safety-march-2017.pdf.

      that highlight diagnostic errors, most HCOs either do not appreciate the significance of harm from diagnostic errors because of measurement challenges or find diagnostic errors hard to address. There are no clear guidelines or best practices to help HCOs implement measurement and learning activities. Often due to competing priorities, there may be a lack of resources and leadership support to create a program targeting improving diagnosis.
      • Newman-Toker DE
      • et al.
      Are health care provider organizations ready to tackle diagnostic error? A survey of Leapfrog-participating hospitals.
      With the Safer Dx Checklist, HCOs can evaluate and identify where to begin and how to overcome initial challenges. The checklist also aims to create a sense of shared responsibility, which may help to overcome specific barriers related to diffusion of responsibility, such as who should be in charge of addressing diagnostic safety.
      We developed accompanying guidance and plans to help disseminate the checklist widely through our existing national networks and professional patient safety organizations. Many HCOs lack specific methods or measurements that help surface diagnostic safety issues to the top of ongoing priorities. A self-assessment using the Safer Dx Checklist can serve as an organizational catalyst
      • Singh H
      • Sittig DF.
      Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework.
      to overcome challenges related to competing priorities. However, the checklist is only a starting point, and organizations need to have appropriate resources, clinician engagement, legal and culture climate, and ways to overcome various other barriers. Currently, there is a lack of incentives or a “business case” to work on diagnostic safety initiatives as opposed to working on other accountability metrics for quality and safety.

      Centers for Medicare & Medicaid Services. Linking Quality to Payment. 2021. Accessed Aug 15, 2022. https://data.cms.gov/provider-data/topics/hospitals/linking-quality-to-payment.

      ,
      • Berenson R
      • Singh H.
      Payment innovations to improve diagnostic accuracy and reduce diagnostic error.
      To overcome this issue, external stakeholders, such as payers and accreditation organizations, may need to incentivize change management initiatives that encourage the use of the Safer Dx Checklist to implement recommended practices. For instance, in the United States, the Centers for Medicare & Medicaid Services (CMS) recently implemented a payment policy that requires eligible hospitals to attest they performed an annual safety assessment checklist for electronic health records beginning in 2022.
      • Sittig DF
      • Singh H.
      Policies to promote shared responsibility for safer electronic health records.
      Similar policy incentives could be used for diagnosis. In fact, the work presented here is already informing Recognizing Excellence in Diagnosis (REDx), an initiative being led by a patient safety watchdog organization in the United States, the Leapfrog Group, to publicly report and recognize hospitals for preventing patient harm due to diagnostic errors.

      Leapfrog Group. Patient Safety Watchdog The Leapfrog Group Funded for National Initiative on Preventing Harm from Diagnostic Error. Jan 28, 2021. Accessed Aug 15, 2022. https://www.leapfroggroup.org/news-events/patient-safety-watchdog-leapfrog-group-funded-national-initiative-preventing-harm.

      ,

      Leapfrog Group. Recognizing Excellence in Diagnosis. (Updated: 2022.) Accessed Aug 15, 2022. https://www.leapfroggroup.org/influencing/recognizing-excellence-diagnosis.

      Although the use of the checklist will still require some initial resource and time investment, it could become a centerpiece of efforts to ensure that HCOs address diagnostic safety, providing long-term benefits that outweigh the initial investments. Professional societies could also promote this checklist for quality improvement activities. The checklist could also be a companion for organizations that are implementing other types of patient safety guidance documents, such as the Institute for Healthcare Improvement's National Action Plan to Advance Patient Safety

      Institute for Healthcare Improvement. Safer Together: A National Action Plan to Advance Patient Safety. 2021. Accessed Aug 15, 2022. http://www.ihi.org/Engage/Initiatives/National-Steering-Committee-Patient-Safety/Pages/National-Action-Plan-to-Advance-Patient-Safety.aspx.

      or WHO's Global Patient Safety Action Plan 2021–2030.

      World Health Organization. Global Patient Safety Action Plan 2021–2030: Towards Eliminating Avoidable Harm in Health Care. 2021. Accessed Aug 15, 2021. https://apps.who.int/iris/rest/bitstreams/1360307/retrieve.

      The checklist identifies a variety of topics that have emerged as important in recent research. For example, closing the loop on patient outcomes after a handoff and feedback are key components of diagnostic safety that have been much discussed but not acted upon.

      Shenvi EC, et al. “Closing the loop”: a mixed-methods study about resident learning from outcome feedback after patient handoffs. Diagnosis (Berl). 2018 Nov 27;5:235–242.

      • Meyer A
      • et al.
      A program to provide clinicians with feedback on their diagnostic performance in a learning health system.
      • Cifra CL
      • Sittig DF
      • Singh H.
      Bridging the feedback gap: a sociotechnical approach to informing clinicians of patients’ subsequent clinical course and outcomes.
      • Branson CF
      • et al.
      Improving diagnostic performance through feedback: the Diagnosis Learning Cycle.
      Practices 3 and 9 specifically address this concern (Table 3). Lack of follow-up of subcritical test results and referrals has also been identified as a challenge in multiple studies,
      • Darragh PJ
      • et al.
      A systematic review of interventions to follow-up test results pending at discharge.
      • Singh H
      • et al.
      Follow-up actions on electronic referral communication in a multispecialty outpatient setting.
      • Callen JL
      • et al.
      Failure to follow-up test results for ambulatory patients: a systematic review.
      • Zimolzak AJ
      • et al.
      Why test results are still getting “lost” to follow-up: a qualitative study of implementation gaps.
      and Practice 10 addresses this issue and provides additional guidance for HCOs. This is important because the current Joint Commission National Patient Safety Goal 2 addresses only critical test results. Several items have additional implications for current policy or national improvement efforts. For instance, Practice 6, on patient review of health records,
      • Giardina TD
      • et al.
      Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes.
      is timely because of the 21st Century Cures Act in the United States, which includes a provision requiring that patients can access all their diagnostic information electronically.

      Congress.gov. H.R. 34—21st Century Cures Act. 2015–2016. Accessed Aug 15, 2022. https://www.congress.gov/bill/116th-congress/house-bill/1309.

      Practice 7, focused on health equity, can provide a new lens for organizations aiming to improve diagnostic outcomes of underrepresented populations.
      One of the questions HCOs might ask is where to begin if several practices are marked as not implemented. We recommend starting with a board-to-bedside accountability framework, outlined in Practice 1, which helps create organizational and leadership accountability for improving diagnosis, including among governance bodies. For most items, senior leadership support would be essential. In addition, the multidisciplinary team assembled to conduct an ongoing assessment could assess how existing organizational priorities, strengths, and opportunities align with various types of diagnostic safety issues they encounter. If patients and family members are not yet engaged in an HCO's diagnostic safety initiatives, Practice 5 could be an excellent starting point.

      Limitations

      The development process for the Safer Dx Checklist had several limitations. First, the evidence base for some practices may be mixed and included expert opinion. However, many of the practices build on existing patient safety literature or high-reliability principles. Second, due to the pandemic, we were unable to have in-person interactions that could have led to richer discussions. However, despite our initial reservations about using a virtual platform (vs. in-person), discussions were robust and enthusiastic and provided new insights, which was confirmed by several participants through unsolicited feedback. Experts also offered to clarify their input after the meeting. Last, although all the items have some applicability to HCOs globally, the checklist was developed with a focus on health care delivery in the United States.

      Conclusion

      We developed the Safer Dx Checklist, a proactive self-assessment checklist of best practices for diagnostic safety that can be used by HCOs within and outside the United States. It addresses crucial gaps where HCOs lack infrastructure, accountability, specific methods, or measurements to help surface diagnostic safety issues to the top of ongoing priorities. The checklist identifies a variety of topics that have emerged as important in recent research and includes pragmatic practices that if implemented can help organizations begin and advance activities to reduce diagnostic error and achieve diagnostic excellence. External stakeholders, including patient advocates, public and private payers, and accrediting bodies, should encourage the use of this checklist and guide health care organizations in their journey toward diagnostic excellence.

      Funding

      This work is supported by the Gordon and Betty Moore Foundation (GBMF 8126) and partially supported by the Houston VA (Us Department of Veterans Affairs) Health Services Research and Development (HSR&D) Center for Innovations in Quality, Effectiveness and Safety (CIN 13–413). Dr. Singh is also supported by the VA HSR&D Service (IIR17-127 and the Presidential Early Career Award for Scientists and Engineers USA 14-274), the Agency for Healthcare Research and Quality (R01HS27363), the Gordon and Betty Moore Foundation (GBMF 8838 and GBMF 5498), and the CanTest Collaborative, funded by a Cancer Research UK Population Research Catalyst award (C8640/A23385). The opinions expressed are those of the authors and not necessarily of the US Department of Veterans Affairs or the US government.

      Conflicts of Interest

      All authors report no conflicts of interest.

      Appendix. SUPPLEMENTARY MATERIALS

      References

        • Graber ML
        • et al.
        The next organizational challenge: finding and addressing diagnostic error.
        Jt Comm J Qual Patient Saf. 2014; 40: 102-110
      1. Institute of Medicine. Improving Diagnosis in Health Care. Washington, DC: National Academies Press, 2015. Accessed Aug 15, 2022. http://iom.nationalacademies.org/Reports/2015/Improving-Diagnosis-in-Healthcare.aspx.

        • Graber ML
        • Wachter RM
        • Cassel CK.
        Bringing diagnosis into the quality and safety equations.
        JAMA. 2012 Sep 26; 308: 1211-1212
        • Zwaan L
        • Singh H.
        The challenges in defining and measuring diagnostic error.
        Diagnosis (Berl). 2015; 2: 97-103
        • Schiff GD
        • et al.
        Diagnostic error in medicine: analysis of 583 physician-reported errors.
        Arch Intern Med. 2009 Nov 9; 169: 1881-1887
        • Singh H
        • Upadhyay DK
        • Torretti D.
        Developing health care organizations that pursue learning and exploration of diagnostic excellence: an action plan.
        Acad Med. 2020; 95: 1172-1178
        • Carson-Stevens A
        • Donaldson L
        • Sheikh A.
        The rise of Patient Safety-II: should we give up hope on Safety-I and extracting value from patient safety incidents?.
        Int J Health Policy Manag. 2018 Jul 1; 7: 667-670
        • Linstone HA
        • Turoff M
        The Delphi Method: Techniques and Applications.
        Addison-Wesley, Boston1975
        • Hasson F
        • Keeney S
        • McKenna H.
        Research guidelines for the Delphi survey technique.
        J Adv Nurs. 2000; 32: 1008-1015
        • Howell AM
        • et al.
        International recommendations for national patient safety incident reporting systems: an expert Delphi consensus-building process.
        BMJ Qual Saf. 2017; 26: 150-163
        • Jones J
        • Hunter D.
        Consensus methods for medical and health services research.
        BMJ. 1995 Aug 5; 311: 376-380
      2. ECRI. Deep Dive: Safe Ambulatory Care: Strategies for Patient Safety & Risk Reduction. 2019. Accessed Aug 15, 2022. https://www.ecri.org/landing-ambulatory-care-deep-dive.

        • Singh H
        • Sittig DF.
        Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework.
        BMJ Qual Saf. 2015; 24: 103-110
      3. National Quality Forum. Improving Diagnostic Quality and Safety: Final Report. Sep 19, 2017. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=85969.

      4. National Quality Forum. Advancing Chief Complaint-Based Quality Measurement: Final Report. Jun 24, 2019. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=90454.

      5. National Quality Forum. Improving Diagnostic Quality and Safety/Reducing Diagnostic Error: Measurement Considerations: Final Report. Oct 6, 2020. Accessed Aug 15, 2022. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=93907.

      6. Health Research & Educational Trust. Improving Diagnosis in Medicine: Diagnostic Error Change Package. Sep 2018. Accessed Aug 15, 2022. https://www.improvediagnosis.org/wp-content/uploads/2018/11/improving-diagnosis-in-medicine-change-package-11-8.pdf.

        • Agency for Healthcare Research and Quality
        Operational Measurement of Diagnostic Safety: State of the Science.
        in: Singh H Bradford A Goeschel C. Issue Brief 1. AHRQ Publication No. 20-0040-1-EF, 2022 (Apr 2020. Accessed Aug 15)
        • McDonald KM
        • et al.
        Patient safety strategies targeted at diagnostic errors: a systematic review.
        Ann Intern Med. 2013 Mar 5; 158: 381-389
        • Abimanyi-Ochom J
        • et al.
        Strategies to reduce diagnostic errors: a systematic review.
        BMC Med Inform Decis Mak. 2019 Aug 30; 19: 174
        • Perry MF
        • et al.
        The diagnostic error index: a quality improvement initiative to identify and measure diagnostic errors.
        J Pediatr. 2021; 232: 257-263
        • Singh H
        • et al.
        System-related interventions to reduce diagnostic errors: a narrative review.
        BMJ Qual Saf. 2012; 21: 160-170
        • Graber ML
        • et al.
        Cognitive interventions to reduce diagnostic error: a narrative review.
        BMJ Qual Saf. 2012; 21: 535-557
        • Giardina TD
        • et al.
        Creating a learning health system for improving diagnostic safety: pragmatic insights from US health care organizations.
        J Gen Intern Med. Epub. 2022 Jun 1;
        • Fink A
        • et al.
        Consensus methods: characteristics and guidelines for use.
        Am J Public Health. 1984; 74: 979-983
        • Bradford A
        • et al.
        Development and usability testing of the AHRQ Common Formats to capture diagnostic safety events.
        J Patient Saf. Epub. 2022 Apr 21;
        • Lyon AR
        • et al.
        The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability.
        Implement Sci Commun. 2021 Jul 17; 2: 78
      7. Baylor College of Medicine; Institute for Healthcare Improvement. The Safer Dx Checklist: 10 High-Priority Organizational Practices for Diagnostic Excellence.. https://www.ihi.org/resources/Pages/Tools/safer-diagnostic-checklist.aspx.

      8. World Health Organization. Global Patient Safety Action Plan 2021–2030: Towards Eliminating Avoidable Harm in Health Care. 2021. Accessed Aug 15, 2021. https://apps.who.int/iris/rest/bitstreams/1360307/retrieve.

      9. Organisation for Economic Co-operation and Development. The Economics of Patient Safety: Strengthening a Value-Based Approach to Reducing Patient Harm at National Level. Slawormirski L, Auraaen A, Klazinga N. Mar 2017. Accessed Aug 15, 2022. https://www.oecd.org/els/health-systems/the-economics-of-patient-safety-march-2017.pdf.

        • Newman-Toker DE
        • et al.
        Are health care provider organizations ready to tackle diagnostic error? A survey of Leapfrog-participating hospitals.
        Diagnosis (Berl). 2017 Jun 27; 4: 73-78
      10. Centers for Medicare & Medicaid Services. Linking Quality to Payment. 2021. Accessed Aug 15, 2022. https://data.cms.gov/provider-data/topics/hospitals/linking-quality-to-payment.

        • Berenson R
        • Singh H.
        Payment innovations to improve diagnostic accuracy and reduce diagnostic error.
        Health Aff (Millwood). 2018; 37: 1828-1835
        • Sittig DF
        • Singh H.
        Policies to promote shared responsibility for safer electronic health records.
        JAMA. 2021 Oct 18; 326: 1477-1478
      11. Leapfrog Group. Patient Safety Watchdog The Leapfrog Group Funded for National Initiative on Preventing Harm from Diagnostic Error. Jan 28, 2021. Accessed Aug 15, 2022. https://www.leapfroggroup.org/news-events/patient-safety-watchdog-leapfrog-group-funded-national-initiative-preventing-harm.

      12. Leapfrog Group. Recognizing Excellence in Diagnosis. (Updated: 2022.) Accessed Aug 15, 2022. https://www.leapfroggroup.org/influencing/recognizing-excellence-diagnosis.

      13. Institute for Healthcare Improvement. Safer Together: A National Action Plan to Advance Patient Safety. 2021. Accessed Aug 15, 2022. http://www.ihi.org/Engage/Initiatives/National-Steering-Committee-Patient-Safety/Pages/National-Action-Plan-to-Advance-Patient-Safety.aspx.

      14. Shenvi EC, et al. “Closing the loop”: a mixed-methods study about resident learning from outcome feedback after patient handoffs. Diagnosis (Berl). 2018 Nov 27;5:235–242.

        • Meyer A
        • et al.
        A program to provide clinicians with feedback on their diagnostic performance in a learning health system.
        Jt Comm J Qual Patient Saf. 2021; 47: 120-126
        • Cifra CL
        • Sittig DF
        • Singh H.
        Bridging the feedback gap: a sociotechnical approach to informing clinicians of patients’ subsequent clinical course and outcomes.
        BMJ Qual Saf. 2021; 30: 591-597
        • Branson CF
        • et al.
        Improving diagnostic performance through feedback: the Diagnosis Learning Cycle.
        BMJ Qual Saf. 2021; 30: 1002-1009
        • Darragh PJ
        • et al.
        A systematic review of interventions to follow-up test results pending at discharge.
        J Gen Intern Med. 2018; 33: 750-758
        • Singh H
        • et al.
        Follow-up actions on electronic referral communication in a multispecialty outpatient setting.
        J Gen Intern Med. 2011; 26: 64-69
        • Callen JL
        • et al.
        Failure to follow-up test results for ambulatory patients: a systematic review.
        J Gen Intern Med. 2011; 27: 1334-1348
        • Zimolzak AJ
        • et al.
        Why test results are still getting “lost” to follow-up: a qualitative study of implementation gaps.
        J Gen Intern Med. 2022; 37: 137-144
        • Giardina TD
        • et al.
        Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes.
        J Am Med Inform Assoc. 2022 May 11; 29: 1091-1100
      15. Congress.gov. H.R. 34—21st Century Cures Act. 2015–2016. Accessed Aug 15, 2022. https://www.congress.gov/bill/116th-congress/house-bill/1309.