Implementing a Course Review Process for a Continuous Quality Improvement Model for a Medical School Curriculum
In 1901, Abraham Flexner, a research scholar at the Carnegie Foundation for the Advancement of Teaching, visited 155 medical schools in the United States and Canada to assess medical education. Flexner's recommendations became the foundation for the Liaison Committee on Medical Education accreditation, a voluntary, peer-reviewed quality assurance process to determine whether a medical education program meets established standards.4 The Meharry Medical College School of Medicine, a historically Black college/university (HBCU) established the Office of Curriculum Evaluation and Effectiveness in 2013 to ensure the consistent monitoring of the medical education program's compliance with accreditation standards. The motto and logo, LCME 24/7, highlight the school's emphasis on meeting accreditation standards. The school uses the 1994 Plan-Do-Study-Act Cycle for Learning and Improvement for continuous review of course content, outcomes, and evaluations. This process identifies strengths, challenges, and opportunities for innovative steps for continuous quality improvements to the curriculum.
Course review, curriculum, Liaison Committee on Medical Education, curriculum evaluation, continuous quality improvement (CQI)
In 1901, Abraham Flexner, a research scholar at the Carnegie Foundation for the Advancement of Teaching, assessed medical education in North America by visiting the 155 medical schools in operation in the United States and Canada.1 Flexner made several recommendations that included expansion of the laboratory component of the first two years and augmenting the requirement of hospital experiences (clinical/clerkship) during the third and fourth years.2 Based on his observations, Flexner concluded creating a model of theory to practice would mean that "students no longer merely watch, listen, and memorize, but be actively involved in the learning of medicine. An [End Page 63] education in medicine involves both learning and learning how; the student cannot effectively know, unless he knows how."2[p.53] Using this model for evaluation, Flexner recommended that 31 medical schools should remain in operation. Of that number, two were Black medical schools, Meharry Medical College in Nashville, Tennessee and Howard University College of Medicine in Washington, D.C.3 Flexner's critique of medical education precipitated an evolutionary change that still exists in the 21st century.1
Founded in 1942, the Liaison Committee on Medical Education (LCME)4 started the accreditation process using a foundational, basic, and clinical sciences model recommended by Flexner. Accreditation by the LCME is a voluntary, peer-reviewed process of quality assurance that determines whether a medical education program meets established standards.5 Full accreditation of a medical school is normally assessed at eight-year intervals.5 Between reviews, the LCME recommends that the performance of certain accreditation elements be continuously reviewed by medical schools.6 In 2014, the LCME began requiring medical schools to develop and implement an internal continuous quality improvement (CQI) process,6 to ensure effective monitoring of the medical education program's compliance with accreditation standards.5 The language addressing this requirement was added to Standard 1: Element 1:1: Strategic Planning and Continuous Quality Improvement, which states:
A medical school engages in ongoing planning and continuous quality improvement processes that establish short and long-term programmatic goals, result in the achievement of measurable outcomes that are used to improve programmatic quality, and ensure effective monitoring of the medical education program's compliance with accreditation standards.5[p.1]
Facilitation of Course Review
In preparation of fulfilling this requirement, during the fall of 2013, the Meharry Medical College School of Medicine established the Office of Curriculum Evaluation and Effectiveness (OCEE). The office created a motto and logo, LCME 24/7. The logo and motto represent a constant monitoring of accreditation standards for continuous quality improvement throughout the medical education program. This approach is consistent with what is considered a functional continuous quality improvement (CQI) process focused directly on accreditation standards, to result in the improvement of educational quality and outcomes.7 Figure 1 is an image of the LCME 24/7 logo.
One of the major responsibilities of the OCEE is the facilitation of course reviews. The effectiveness component of the office monitors the quality of teaching and learning in the medical school. In the context of accreditation, CQI is defined as a process both to monitor compliance with accreditation standards in the interval between full accreditation reviews and to act on the results.6 The OCEE worked with the Curriculum Evaluation Committee, a subcommittee of the Curriculum Committee, to develop recommendations for an individual course review process to the Curriculum Committee that set the tone for determining quality of all courses and clerkships. The course review process is designed to become the critical component of continuous [End Page 64] improvement. The course review process consists of four steps: 1) taking an in-depth, course-by-course review of the curriculum; 2) helping the curriculum committee determine required changes/modifications and/or enhancements; 3) determining appropriate modifications, if needed; and 4) ultimately determining the effectiveness of the course in preparation for the review of the entire curriculum, and the overall medical education program based on measurable outcomes. This four-step process, which was approved by the Curriculum Committee, is utilized in conjunction with the Individual Course Analysis (ICA) metrics.
Course review metrics
The Individual Course Analysis (ICA) metrics are used to evaluate and monitor courses for both the preclinical (MS1/MS2) and the clinical (MS3/MS4) years. The metrics align with LCME Standard 8: Element 8.4: A medical school collects and uses a variety of outcome data, including national norms of accomplishment, to demonstrate the extent to which medical students are achieving medical education program objectives and to enhance medical education program quality.5[p.12] A sample of the course review metrics for the Individual Course Analysis (ICA) Report is included in Box 1. The metrics include data provided through electronic surveys, evaluations, and internal and national databases. These metrics provide faculty reviewers with all data relative to each course, which include the syllabus, methods of instruction, curriculum map, and course evaluations. Although there are standard metrics used throughout the curriculum (preclinical phase and clinical phase), there are different types of course metrics to determine effectiveness, such as first-time pass rates on the United States Medical Licensing Examinations Step 1, Step 2 Clinical Knowledge (CK), and Step 2 Clinical Skills (CS); National Residency Match Program (NRMP) Match statistics; graduation rate, and licensure rates.
Quality Improvement Model
A continuous quality improvement model was developed using the Plan-Do-Study-Act (PDSA) Cycle, a systematic series of steps for gaining valuable learning and knowledge for continual improvement, also known as the Deming Wheel, or Deming Cycle.8 The PDSA Cycle consists of four steps, 1) Plan, 2) Do, 3) Study, and 4) Act, and parallels the steps utilized for the course review process. The model is included in Figure 2.8
The Plan step involves identifying a goal or purpose, formulating a theory, [End Page 65]
defining success metrics, collecting data, and putting a plan into action.8 This step was operationalized by the OCEE and the Curriculum Committee after the course review process was modified and a plan for conducting the reviews was established to facilitate an in-depth view of the curriculum, course by course. To review courses effectively, a comprehensive course evaluation instrument was redesigned. This instrument was designed to monitor the effectiveness of the course based on identified metrics. The course review evaluation instrument was developed in E*Value and is also completed [End Page 66] in E*Value.9 E*Value is a web-based evaluation system designed to help manage, collate, and analyze the large volume of information associated with health sciences programs.9 Although the evaluation instrument is completed in E*Value, the course content, due to its size, is viewed in Blackboard, the learning management system.10 The operational plan entails the following: informing the course/clerkship director of an upcoming course review; the assignment of course reviewers (preclinical and clinical faculty), based on the discipline/specialty of the course designated for review; clarifying the purpose and objectives of the course review to reviewers; and gathering and analyzing the data relative to the metrics identified for course reviews. A timeframe for the completion of the course review is also established during this phase. It is important to note that faculty serving as course reviewers are not members of the department whose course is being reviewed.
The Do step involves the implementation of the components of the plan.8 The course review process is implemented by the OCEE. This phase of the process contributes to the curriculum committee's determinations regarding recommended changes/modifications and/or enhancements. The OCEE gathers all data specific to the course and the data are linked in the course review instrument in E*Value. A meeting is held with the course/clerkship director to review the data in one setting, with an opportunity for comments and/or corrections. A training session is held for the course reviewers regarding the course review process, E*Value, and Blackboard access. Upon completion of the training, the course review instrument is made available to the reviewers for an in-depth review of the course based on outcome measures that identify strengths/successes, challenges/barriers, gaps and redundancies, and areas in need of improvement. At the conclusion of the review, the course reviewers report their findings to the curriculum committee for further actions. In addition, during the course review process, the course reviewers may also decide to meet with the course/clerkship director to obtain additional information and if the course/clerkship is in session, the reviewers may also visit selected sessions of the course/clerkship.
The Study step is where outcomes are monitored to test the validity of the plan for signs of progress and success, or problems and areas for improvement.8 The [End Page 67] Study step allows the Curriculum Committee and its three subcommittees—preclinical, clinical, and curriculum evaluation—to review the outcomes of the course review findings; to determine if modifications are needed; and to determine how the course/clerkship can be improved.
The Act step closes the cycle, integrating the learning generated by the entire process, which can be used to adjust the goal, change methods or even reformulate a new theory altogether.8 Upon completion of the course review, the course reviewers present the findings to the Curriculum Committee. The preclinical/clinical subcommittee recommends the next steps of action to the Curriculum Committee. When the outcome yields positive results, the course remains in its existing structure, but continues to be monitored via the individual condensed course reports. The condensed course report is prepared by the OCEE at the completion of each course and includes a condensed version of performance metrics. This report is provided to the course/clerkship director and the Curriculum Committee. When the outcome reveals problems or areas in need of improvements, modifications are made accordingly. In addition to the condensed course report, the specific areas of concern are monitored and tracked. Prior to starting the cycle again, all recommended actions are carried out. Like the PDSA Cycle, the progressive outcomes are preserved and steps are taken to maintain the improvements and accomplishments.8 This step ultimately determines the effectiveness of the course in preparation for the comprehensive review of the curriculum, and the overall medical education program based on measurable outcomes.
The lessons learned from the course review process are included below in the form of strengths, challenges, and a look into the future with innovative next steps.
The OCEE provides a basis for implementing new, innovative improvements to the curriculum. The system established for the course review process allows the course/clerkship director and the curriculum committee to view all metrics regarding the course in one setting. Since the curriculum is integrated, other disciplines are reviewed as well, providing insight into the status of the curriculum throughout the process. Students, who serve as members of the Curriculum Committee, are aware of the process, findings, and changes instituted in real time. In addition, students share their experiences regarding the course to the Curriculum Committee.
The course review process is an intense survey; therefore, in some instances, conflicts arise with faculty schedules based on teaching, clinic, and/or research obligations, which can result in a delay in the scheduled timeframe for completing a course review.
Innovative Next Steps
The next step for the OCEE is to participate in the institutional initiative to build a data ecosystem. The components of the data ecosystem include a variety of big data, which involve data-generating activities that produce various types/formats of data resources.11 Within the data ecosystem, an infrastructure is established for selecting the right tools for storing, processing and analyzing data.12 The data ecosystem is a valuable resource to allow the "silos" of data to become actionable insights on educational effectiveness.13 Data science is the key to unlocking data from its silos, data elements self-contained in its native environment, such as databases, and allowing the data to tell its story. The act of 'unlocking data from silos' is a data science technique entitled associations, one of the most powerful and utilized algorithms in reviewing [End Page 68] large swaths of data.14 Associations are used to find frequent patterns of data hidden relationships and relationships of interests among the various data sets.14 The era of big data has introduced unstructured data, societal behavior data sets, such as, social media, which is now included with traditional structured data sets can now provide data associations never thought of before, looking to not just associations between individual student educational data and curricular data, but also the effect of behavior, environmental elements, and societal influences.15 Having access to these types of metrics will reveal other factors that contribute to the success of student performance as well as target specific instructional, curricular, and other resources to support the achievement of specific student learning objectives12 for continuous quality improvement. By allowing the data to tell its story, associations from within the different data sets collected by the OCEE and other areas throughout the college will provide the basis for new prospective and innovative approaches that improve curriculum design, implementation, and evaluation.
CASSANDRA S. WARD is Associate Dean of Curriculum Evaluation and Effectiveness and Assistant Professor in the Department of Professional and Medical Education at the Meharry Medical College School of Medicine. AMY ANDRADE is Senior Advisor to the President for Technology and Innovation and Assistant Vice President of Research at Meharry Medical College. LENA WALKER-WINFREE is Project Manager for the Data Science Center at Meharry Medical College.
The authors would like to acknowledge Ms. Katrina Otuonye and Dr. Juanita Buford of Meharry Medical College for reviewing and providing feedback on this manuscript.