Introduction: Harm from prescriptions written for patients with contraindications to their use is common and preventable; nearly one-third of all antibiotic and nonsteroidal anti-inflammatory drug (NSAID) prescriptions in outpatient care settings are potentially inappropriate and contribute to adverse drug events. Audit and feedback is a promising and effective intervention to change clinician behavior. However, adoption can be challenging to scale and failure to place the user (i.e., clinician) at the center of the design contributes to underuse, workarounds, and unintended consequences. We focused on user needs to determine workflow integration and visualization needs to develop a tool for clinicians to quickly and easily view and better understand their prescribing practice to improve adherence to guideline concordant care. 
 Methods: In this study, we employed a user-centered design framework, a four-phase, iterative approach to prototype development that starts with understanding user needs (Phase 1) and requirements (Phase 2), which then progresses to formative (Phase 3) and summative (Phase 4) evaluation through an iterative process of increasingly higher fidelity prototypes. We developed and refined a clinician prescribing feedback system, entitled “CRAFT” (Care Review, Assessment, and Feedback Tool), for acute outpatient clinicians in the Veteran’s Health Administration in the Tennessee Valley Healthcare System in Middle Tennessee. 
We developed a low-fidelity prototype (PowerPoint slide) using input from the research team and ED clinical champions. This initial prototype used hypothetical data that aggregated overall clinician performance such as appropriateness of prescribing and “notable events” including return visits and adverse drug events. In addition to aggregated data, we provided patient-level detail for each prescription. 
To better understand the user needs and requirements for CRAFT, we conducted 25 interviews with physicians, advanced practice providers, and pharmacists, and asked about their prescribing practices and response to the low-fidelity prototype. From these data, we further iterated the prototype, addressing usability, data visualization, and information prioritization concerns. We then conducted four 60-minute design sessions with clinicians, developers, and biostatisticians to create an interactive prototype that contained derived but realistic data verified by clinicians. Prototypes were updated between sessions. Once the prototype was deemed complete by the study team and ready to be evaluated by potential users, we progressed to individual usability evaluation sessions. 
Six ED physicians completed individual 60-minute virtual usability evaluation sessions where the clinicians interacted with the prototype using one of two counterbalanced scenarios based on performance by drug class: Scenario One showed the clinician good antibiotic and poor NSAID performance and Scenario Two showed poor antibiotic and good NSAID performance. During the sessions, clinicians were asked to “talk aloud” about what they were seeing and expected to see. At the conclusion of these sessions, key issues were brought back to the team along with recommendations for changes. The team then iterated design recommendations and a refined prototype was created. This prototype was then evaluated by five physicians using the same procedure as discussed above.  
Results: Overall, there were six themes represented from the interviews including: usability, clarity and unintended consequences of language, actionability, comparators, and features. Participants expressed that they were busy and inundated with information, and there was a need for communicating key information efficiently. Respondent input identified that overall organization seemed to be disconnected and that there was too much information to digest on the welcome page.  
This initial round of usability testing also saw participants have issues and recommendations around wording and labeling, peer comparison, and navigation. Participants questioned the use of the labels “notable events” and “eligible prescriptions” as they seemed vague even after reading the detail available in the information icon. They also expressed concern over the negative connotation of the word “inappropriate,” and the need to clearly define the guidelines and appropriateness assessments. Physicians questioned the use of peer comparison and how it might influence behavior, especially if it was below the clinician reported performance. There were usability issues associated with simplifying ways to navigate from the aggregated clinician data to detailed patient information in the most efficient way possible.  
With these results, we refined the prototype, also added a visualization of all eligible prescriptions that was used in a second round of usability testing. Continued usability testing identified that respondents did not understand why some prescriptions were inconsistent with recommendations and how to act accordingly. They also noted that rather than a peer or historical comparison with their own performance, they preferred an alternative form of comparison. Further, the visualization of “nonevent prescriptions” did not contribute much and their primary interest involved the prescriptions with either an unexpected return visit or an adverse event. This resulted in the final prototype that was used to build the user interface in a pilot implementation of CRAFT. 
Discussion and Conclusions: Overall, participants saw value in the tool’s concept and indicated that they would likely use it, but had important optimizations, clarifications, and changes they noted during the user-centered design process that underscored the importance of this type of visualization and workflow integration evaluation.  
Importantly, the usability evaluation results revealed that substantial changes were needed to address navigation issues, to clarify language and connection to guidelines, and finally to provide a simplified message that clinicians could easily understand and act upon. We hypothesize that this will improve usability and acceptance by clinicians, and ongoing usability testing in the summative evaluation (Phase 4) that will be conducted following the pilot implementation of CRAFT and prior to a randomized trial in the VA health system.