practitioner research literacy skills in undergraduate medical education: thinking globally, acting...

Medical Science Educator © IAMSE 2012 Volume 22(3S) 162

MEDICAL SCIENCE EDUCATOR The Journal of the International Association of Medical Science Educators Med Sci Educ 2012; 22(3S): 162-184

MONOGRAPH

Practitioner Research Literacy Skills in Undergraduate Medical Education: Thinking Globally, Acting Locally

des Anges Cruser, Sarah K. Brown, Jessica R. Ingram, Frank Papa,

Alan L. Podawiltz, David Lee & Vesna Knox University of North Texas Health Science Center, Texas College of Osteopathic Medicine, Fort

Worth, TX, USA

Introduction In this era of global communications, web-based search engines enable us to access and enjoy an ever increasing number of internationally authored papers, free-access journals, reports and scholarly works on any topic and within minutes of their publication. As this torrent of information washes upon us we often find that to more fully understand a concept of interest or answer a question, we must continually dig deeper. We must utilize our intellectual skills to process and integrate multiple bits and pieces of information in order to construct a knowledge base representing a deeper level of comprehension related to the topic at hand. We suggest that this process of deepening comprehension is similar to weaving a visually appealing, yet upon closer inspection, complex tapestry from multi-textured conceptual threads. Indeed, the central task of our intellectual armamentarium in this regard, is to construct a tapestry of rich conceptual coherence from threads of information gathered from an international landscape. It has become increasingly apparent to those involved in the transformation of medical students into future health care practitioners, that medical education is poised on the cusp of a new paradigm. At the heart of this emerging paradigm lie the core competencies. In this manuscript we have gathered together a number of contextual threads from the international literature associated with these core

competencies, and woven them into a coherent, multi-textured conceptual tapestry which we refer to as "Practitioner Research Literacy Skills" (PRLS). The primary aim of this monograph is to construct a plausibly universal approach to equipping medical students with a foundation in PRLS that is grounded in emerging global perspectives on core competencies, and will prepare them to fully engage in those core competencies as a full measure of the roles of the 21st century physician. Our proposed definition of PRLS includes but is greater than the sum of the tools of statistics and epidemiology. This definition includes those skills that characterize the physician who can successfully navigate the world of research and its associated data in medicine and healthcare, locally and globally. These are physicians who epitomize the roles of "scholar," "teacher," "scientist," and "collaborator," as reflected in the international compendia of core competencies. Although contemporary Evidence-Based Medicine (EBM) experts are mindful that research literacy skills are essential to the appropriate use of its principles and practices, medical students and educators alike remain inclined to view PRLS as peripheral and irrelevant in undergraduate medical education (UME). While there is currently no globally embraced definition of practitioner research literacy skills, we suggest that these skills prominently exist as discrete elements that are inextricably interwoven into the fabric of the core competencies. As such we further suggest that these PRLS are readily suited for integration into a competency-based UME curriculum, and are likely to add value to the overall training outcomes.

Corresponding author: dès Anges Cruser, PhD, MPA, Texas College of Osteopathic Medicine, UNTHSC Fort Worth TX 76107, USA. Tel: +1 817-735-2009, Email: [email protected]


With the emerging globalization of competency-based standards, there is a simultaneous movement reframing the traditional linear approach to medical education, as one that uses an upward, spiral approach, with educative materials both vertically and hierarchically integrated throughout the curriculum. Thus, to formulate a plausible universal approach to teaching practitioner research literacy skills (hereinafter referred to as PRLS) to medical students, we sought answers to the following three questions: 1. What are the global trends in physician core

competency standards and competency-based medical education?

2. What role does a theoretical construct of practitioner research literacy skills play within the core competencies?

3. Can thinking globally facilitate the integration of practitioner research literacy skills into a competency-based medical education curriculum?

This monograph is aimed at synthesizing globally accessible published ideas on how to equip medical students with fundamental PRLS. To accomplish this, we collected 65 peer reviewed articles and scholarly papers, and 15 authenticated reports and published accreditation standards in the English language, through PubMed, Google Scholar, and medical education related organizations' web sites. Our primary search terms were: physician core competencies, research literacy, medical education and biostatistics/epidemiology/research methods/ methodology, or statistics, and we also combined those terms with key international classifications and organizations such as European, or the World Health Organization. We traced referenced literature for seminal articles and papers addressing models for teaching to the core competencies including competency-based medical education; and we searched for papers on paradigms related to lifelong learning and other skills thought to characterize the complete 21st Century healthcare professional. From this information and from our own experiences developing and teaching PRLS, and interacting with an international network of medical educators, we have woven a metaphorical tapestry illustrative of the relationship between PRLS and the emerging global core competencies for physicians, and the suitability of PRLS for integration into the undergraduate medical education (UME) curriculum. This monograph is organized around the three primary questions, followed by a section on application, and finally a conclusions section,

synthesizing our findings. Thusly we provide the reader with the following index to the five sections in this paper. Section 1. What are the global trends in physician core competency standards and competency-based medical education?

1.1. The major authorities in competency standards 1.2 Introducing core competencies in medical education

Section 2. What role does a theoretical construct of practitioner research literacy skills play within the core competencies?

2.1. The model 2.2. Conceptual paradigms linking core competencies with practitioner research literacy skills 2.3. Summary

Section 3. Can thinking globally facilitate the integration of practitioner research literacy skills into a competency-based medical education curriculum?

3.1. Emerging global perspectives 3.2. Trends 3.3. Five Models 3.4. Summary

Section 4. Applying the theories and models.

4.1. Defining PRLS throughout the core competencies 4.2. Curriculum integration 4.3. Rubrics and Learning Tools 4.4. Measuring outcomes

Section 5. Conclusions


1. What are the global trends in physician core competency

standards and competency-based medical education? Four short years ago, Albanese et al. published a study of global core competencies, concluding at that time the worldwide diversity of definitions for physicians' core competencies limited any meaningful cross-systems comparison. Today however, in 2012, while each authority may continue to use slightly different terminology, refinements in the definitions of those physicians’ core competencies have facilitated a cross-systems comparison as we have shown in Table 1. In this table, we have used the six core competencies

published by United States (US) medical education authorities, to anchor a summary grid of all core competencies published by CanMEDS, the General Medical Council (GMC), the Australian Medical Council, and MEDINE, and the Global Minimum Essential Requirements (GMER) developed by the International Institute for Medical Education (IIME).12

Throughout this monograph, we use the term “authorities” to refer to countries, nations, organizations and other geo-political entities that have either legitimate or referent authority for policy and standards in medical education.

1.1. The major authorities in competency standards In the following six subsections, we briefly review the approach of each major authority listed in Table 1, to defining the core competencies. Thus, we review the United States of America, Canada, the United Kingdom, Australia and New Zealand, the

European Communities, and other world countries per the table legend. Last, we describe the efforts of the Foundation for the Advancement of International Medical Education and Research in international medical education affairs, and summarize the trends in emerging global core competencies particularly as related to research literacy skills.


The United States of America In the United States of America (US), the Accreditation Council for Graduate Medical Education (ACGME) began requiring all residents in allopathic graduate medical education (GME) programs to achieve six core competencies in 1999.3 The American Osteopathic Association (AOA) followed suit in 2003 for osteopathic (DO) physicians.13 Shortly thereafter, the US Institute of Medicine (IOM) addressed UME, recommending that medical students develop proficiencies in five core areas: patient care, interdisciplinary teamwork, EBM, quality improvement, and information technology.14 With this recommendation, the IOM had effectively shifted the national focus from teaching to the core competencies in the GME arena, to orchestrating them into the UME curriculum. The Liaison Committee on Medical Education (LCME), accrediting allopathic (MD) medical schools in the US and Canada, states that the “accreditation process requires a medical education program to provide assurances that its graduates exhibit general professional competencies that are appropriate for entry to the next stage of their training and that serve as the foundation for lifelong learning and proficient medical care.”7 Lifelong learning is a crucial element in practitioner research literacy skills. In 2006 the Association of American Medical Colleges (AAMC) raised the competency bar with a bold statement that every future physician should receive a thorough education in the basic principles of research, and that this academic component should become an accreditation standard for undergraduate and for graduate medical education.15 The National Board of Osteopathic Medical Examiners (NBOME) 2011 competency guidelines now include specific elements and outcomes for research related skills throughout their competency domains. The research related skills in the NBOME guidelines specify in considerable detail, a variety of statistics and EBM concepts that students must acquire.4 In the MD medical education realm the LCME also specifies that all of the six ACGME competencies must be addressed in the curriculum, and recommends that students participate in research. LCME standards require introduction to basic scientific and ethical principles of clinical and translational research and their application to patient care.7 Understanding scientific and ethical principles of research is an essential skill in the PRLS model.

Canada The Royal College of Physicians and Surgeons of Canada published The CanMEDS 2005 Physician Competency Framework, defining the competencies needed for medical training and practice.5 That document states specifically: “CanMEDS is not only meant to guide curriculum, it is also meant to be the basis for bedside teaching.” CanMEDS characterizes a "medical expert," as having multiple roles. The role of scholar appears to be the most closely aligned with Practice Based Learning & Improvement in the US-based core competencies list, characterizing one who is skilled in critical appraisal, research literacy, ethical issues in research, asking effective learning questions, EBM skills, and scientific inquiry. These skills are essential elements of PRLS. The United Kingdom The General Medical Council (GMC), responsible for the health and safety of the public, and for medical education in the four countries of the United Kingdom (England, Wales, Northern Ireland, & Scotland), defines explicit outcomes for all students graduating five years of medical school. In Tomorrow’s Doctors, the GMC refers to three roles: the scientist and scholar, the practitioner and the professional.8 The GMC includes several research literacy constructs in the role of scientist and scholar. The GMC uses slightly different terminology than other authorities we reviewed, but the overarching ideas and goals are similar. As a scholar and scientist, the physician is expected to be able to explain principles of investigative techniques; explain, discuss, and apply theoretical frameworks of population health; apply the scientific method; evaluate and critically appraise data; and formulate and apply findings from the literature to answer clinical questions, all essential PRLS elements. Australia and New Zealand The Australian Medical Council (AMC) is the authority for accreditation of medical schools in Australia and New Zealand.9 These unique countries have Indigenous ethnic populations to which medical education has a special responsibility. There are 40 overarching goals in acquiring knowledge, skills, and attitudes that the AMC declares would characterize a junior doctor. Among them, using information technology as an essential resource in patient care is associated with patient care skills, and the role of teacher. Patient safety, communications, cost-effectiveness, and interprofessional teamwork, as well as mindfulness are also specified. The scientific method, courses in population health sciences, communications and


interprofessional education, and research methods are each separately defined standards for AMC accredited medical schools. This authority emphasizes elements of PRLS throughout its competencies. European Communities In 2007, a group of international organizations under the leadership of the Thematic Network of Medical Education in Europe (MEDINE), published the World Federation for Medical Education (WFME) Global Standards for Quality Improvement in Medical Education: European Specifications.7 That document defines standards for basic, postgraduate, and continuing medical education as tools for reform and improvement in the European nations affiliated with the WFME and MEDINE.5 That document states that the medical school must teach the principles of scientific method, evidence-based medicine, and analytical and critical thinking throughout the curriculum. It further recommends specific elements for training students in research methods. MEDINE also recommends that physicians become scholars in their chosen specialty and acquire lifelong learning skills. In this arena, PRLS are emphasized as being important elements throughout the curriculum. Other Countries of the World In 2008, Hans Karle, the WFME President at that time, opened a public debate on international recognition of basic medical education institutions and programs in a report that benchmarked progress of the accreditation process worldwide.16 That paper provides a comprehensive overview of the history and pathways taken by multiple organizations to produce standards for medical education for countries without such standards. Then, in 2010, Manisha Nair from India, and Premila Webster from the UK, together provided an international review of the literature on education for health professionals in emerging market economies (EMEs).17 That team noted that according to the literature they reviewed, outcome-based education, another name for competency-based medical education (CBME) according to their reference, has been implemented in Switzerland, the Netherlands, Scotland, Canada, the US, the UK, India and China. This may lead one to believe that all of those countries' medical schools use a CBME model, yet other papers indicate this is not necessarily accurate.18-20

Both the Karle and the Nair papers refer to the WFME and the WHO as leading organizations in the movement toward adopting “global minimum essential requirements” (GMER) to standardize

quality of medical education worldwide. The GMER include seven competency domains that closely parallel those adopted in the US, the UK, Canada, Australia, and the European communities in MEDINE, and recommends these competencies for all other countries. The GMER are categorized in Table 1 as we have conceptually associated their terminology and definitions with the US-based competencies in the first column of that table. Another perspective on other counties and EMEs can be found in the The Foundation for the Advancement of International Medical Education and Research (FAIMER). FAIMER, located in the US, maintains an International Medical Education Directory (IMED). FAIMER’s mission is to promote excellence in international health professions' education through programmatic and research activities.21 As of the beginning of 2012, there were approximately 2,218 medical schools in about 177 of the registered countries, all of which reported that they expected their graduates to demonstrate in some measurable way that they acquired the knowledge, simply put, to "treat the sick and injured with competence and compassion and without prejudice," in accordance with two highly regarded and insightful published discussions of the purpose of medicine.22,23

In 2006, Cueto et al. reported that nine developing nations in Latin America, Africa, India, Southeast Asia, and the Middle East, had accreditation processes in place under governmental auspices. That report however, provided no information we could utilize to populate Table 1.22 More recently, the 2010 review by Nair et al. examining EMEs in Asian, Latin American, Russian, and Eastern European countries reported that the rather sparse literature available at that time provided few details about progress in accreditation standards, leading to the conclusion that medical education reform should be a major priority for most of these EMEs.17

Limited information is readily available about physician competencies and medical education standards for EMEs and developing countries. For example, it appears that many Asian and Latin American countries have registered with FAIMER, but not all have adopted the WFME guidelines. The process of accrediting medical schools, using WFME recommended competencies as standards, differs around the world. The initiatives of the WFME are beginning however, to close the gap between accreditation and competency standards. For those interested in the accreditation process, van Zanten et al. have provided a succinct and thorough


worldwide review of that process in their 2008 paper.24

Summary trends in emerging global core competencies for physicians Today there are six basic core competencies defined by US authorities. Medical Knowledge, and Patient Care are the sine-qua-non foundation of medicine. Those competencies are also referred to, as reflected in Table 1, by non-US authorities as "medical expert," "practitioner," and "scientific foundation of medicine;" and by "health advocate," "patient advocate," and "clinical skills" respectively. Communications Skills, and Professionalism, with some overlapping and complementary elements and specifications are found consistently across all authorities we reviewed. The GMER also specifically include "professional values, attitudes and ethics" in their list of similar competency domains. For this paper, we have used the term Communications Skills without the word "interprofessional," eliminating some redundancy, as all communications are predominately interpersonal, and in healthcare they are interprofessional. Systems-Based Practice (SBP), and Practice-Based Learning and Improvement (PBLI), also have some shared elements, that we discuss later in this paper. SBP focuses on the physician interacting with the larger health care system, interprofessional and interdisciplinary teamwork, and the physician as manager and leader. PBLI and its related global companion competencies contain most, but not all of the PRLS related elements in our theoretical construct, and are therefore noted with double asterisks in Table 1. We discuss later in the paper, how PRLS are also interwoven throughout the other competencies and facilitate their interrelationships.

1.2. Introducing core competencies

in medical education Historically, graduate medical education (GME) or residency programs were the venue where physicians were expected to acquire their required core competencies.25 Current trends suggest however, that by 2020, medical school applicants around the world will be expected to have already acquired foundational skills in several core competencies in their pre-baccalaureate programs.26 But for today, the undergraduate medical education (UME) venue is the focal point for medical education authorities, who expect to see core competencies clearly articulated in learning objectives, instructional methods, and outcome measures throughout the curriculum. This

expectation immediately raises two questions, each of which we have separately addressed in the following sections. These are:

1) When and how can we introduce core competencies?

2) How can we measure them? When and how can we introduce core competencies? While it is beyond the scope of UME to ensure full proficiency in any competency, there are levels of proficiency that may be defined within each competency as progressively sophisticated levels of performance achieved across the spectrum of medical education and practice. Medical education experts recommend competency-based medical education (CBME) to facilitate the when and the how of introducing core competencies in the UME process. Competency-based education has its roots in pre-baccalaureate public education systems of the 1970s, and was introduced into medical schools in the 1990s.27,28

CBME utilizes the concept of life-roles and their associated activities to direct curricular strategies, with a focus on outcomes and performance. The nature of these life-roles and associated activities, and thus the outcomes, are defined by the core competencies. While half of the authorities we reviewed (Table 1), define competencies using life-roles using terminology such as "scholar," and "manager," others use different terminologies. More thorough definitional schema can be found in the individual authorities' guidelines, elucidating their similarities. Thus all are suitable for a CBME model. In 2010, an international group of medical education experts, known as the International CBME Collaborators, published a paper reviewing the CBME literature, and proposing several principles on which this model rests.29 Two central CBME principles are most closely related to our metaphorical PRLS tapestry.

First, CBME is learner-centered and uses hierarchical, integrated instructional design and methods, atypical in most medical schools today. Second, CBME focuses on outcomes, and places crucial emphasis on demonstrable abilities, and perhaps more importantly, the developmental progression of those abilities across all years of the UME experience.


CBME could dramatically change the way we teach, as methods such as self-directed learning, small group processes, and experiential activities would foster reflection, and promote deeper thinking and mastery of skills, thereby outperforming the linear, stepwise model.3

When we consider that for nearly a century, much of medical education has been more linear than hierarchical, with little educative materials being fully integrated either vertically or horizontally, a fully implemented CBME has the potential to revolutionize UME teaching. How to measure performance In 2009, Lurie et al. published a systematic review of the US literature on how GME programs have measured core competency performance.17 In their review, Lurie et al. noted that “teaching to the core competencies” seemed to improve the process of learning and facilitate program maturation, but that it was still unclear to what extent residents had mastered the competencies. We found no parallel international papers in this regard. Historically there has been some imprecision in attempts to define and measure competency outcomes. Overall, the global rating forms, 360-degree evaluations, and direct observation instruments reviewed by Lurie et al., tended to produce only one or two factorial constructs from any one instrument, even though the instrument had attempted to measure multiple discrete competencies. The literature provides several criteria for defining the characteristics of educational competencies. Seven were recommended by Harden et al, in 1999; and in 2008 Mark Albanese and colleagues added five additional criteria to Harden’s earlier list.4,30 Albanese's entire discussion provides considerable practical and conceptual material to use in faculty discussions about measuring outcomes and performance in a substantially integrated competency-based curriculum. Here we have provided a greatly condensed version of these 12 criteria, with recommendation that the reader enjoy the complete Albanese paper to acquire all of the criteria. Extrapolating from that article we find that educational competencies should:

• Be defined in clear, unambiguous, and specific terms, but at an appropriate level of generality;

• Have a manageable number of outcomes, the relationships among which are clearly described;

• Focus on the end-product and define expectations as an application of learning objectives in the immediate instructional program;

• Be expressed as measureable behavior, addressing performance over recollection of information;

• Articulate measureable criteria that are criterion-related;

• Communicate clear outcome expectations. Inasmuch as measuring performance outcomes continues to beleaguer medical educators around the globe, under a CBME model these criteria can be useful in faculty development programs and establishing institutional expectations for course syllabi.

2. What role does a theoretical

construct of practitioner research literacy skills play within the core

competencies? Emerging global core competencies and the competency-based medical education model together, have an almost unprecedented potential to elucidate and crystallize not only what constitutes practitioner research literacy skills, but also how and when students will be expected to acquire them. Both the scholarly literature and all of the international core competency documents we reviewed tend to define PRLS as discrete and unconnected tools and concepts in for example, biostatistics, epidemiology, and critical appraisal, rather than as a coherent construct. However, it is of considerable interest that Kenneth Calman, in his 2007 book "Medical Education: Past, Present and Future,” defined the doctor's role as an agent of change; that is specifically one who could produce evidence and present it effectively to those who matter.23 Therein lies a quintessential example of how a single theoretical construct of PRLS, with the power to conceptually coalesce the currently fragmented list of research-related skills scattered throughout the existing core competencies, could aid in defining demonstrable role-related abilities. It is our premise therefore, that PRLS are interwoven throughout all of the core competencies, and we have endeavored to illustrate this premise in the following discussion.

2.1. The Model The Venn diagram in Figure 1 represents a metaphorical tapestry of PRLS. We have designed this diagram to illustrate how the core competencies and skill sets included in our model are interrelated, and how PRLS serves as the weft fibers stretching

https://www.researchgate.net/publication/285661374_AMEE_Guide_No_14_Outcome-based_education_Part_5_-_From_competency_to_meta-competency_A_model_for_the_specification_of_learning_outcomes?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==


throughout the warp yarn of the competencies and skill sets. The Venn of PRLS is thus a theoretical representation of the interrelationships among the core competencies, other discrete skill sets, and PRLS. As we discuss this model, we anticipate questions will arise as to how each of the competencies relate to the others. For example, one could argue that Communications and Professionalism should be contiguous with Systems-Based Practice as much as with any other

competency domain. Nonetheless, each competency has its own definition. Thus, if one imagines this rendering as a multi-dimensional tapestry, the threads of which cross the permeable boundaries of each other, all components are interrelated. For the purposes of this monograph, we have positioned each domain and skill set proximate to those we hypothesized would be most closely associated with it.

In constructing this model, we have used the US core competency domain nomenclature shown as the anchor column for Table 1. Inasmuch as this nomenclature shares elements of equivalent competencies for other international authorities shown in the other five columns of Table 1, this model embraces all of the emerging global core competencies. The three large circles in Figure 1, represent four of the US-based competencies (Medical Knowledge, Patient Care, Systems-Based Practice (SBP), and Practice-Based Learning and Improvement (PBLI)). The other two competency domains (Communications & Professionalism), reside in the area of overlap between Medical

Knowledge & Patient Care, and PBLI. We have highlighted the roles of the physician as scholar, scientist, teacher, manager, and collaborator, as reflected by their classifications in Table 1, in the overlap between PBLI and SBP labeled Scientific Inquiry & Scholarship. Informatics has been included as a discrete skill set in the overlap between SPB and Medical Knowledge & Patient Care. Medical Knowledge & Patient Care together represent the technical depth that characterizes a well-trained medical practitioner. Today’s physicians however, are also expected to acquire a


breadth of skills to enrich their professional and personal portfolios, enabling them to function effectively in an interprofessional environment, and effectively apply critical thinking and problem solving skills in a highly complex healthcare system. EBM skills are shown as intricately woven into the fabric of these competency domains. Communications & Professionalism play a vital role in developing the professional dimension of “breadth.” In our theoretical model, these two competency domains are manifestations of the interaction between PBLI and EBM. Physicians convey their knowledge and abilities through their communications and behaviors. These two competencies are closely aligned with PRLS as the vehicles for discovering, verifying, and sharing information (knowledge) with peers, subordinates, superiors and patients. Systems-Based Practice (SBP) includes skills and abilities in interprofessional practice and working as part of a team, as well as being connected with the larger system of care. The Global Minimum Essential Requirements (GMER), developed by the Institute for International Medical Education specifically use the terms "information management," and "critical thinking and research," in ways that we associate primarily with PBLI (see Table 1). The US definitions of PBLI and SBP however, both contain elements of critical thinking, research-associated concepts, and information management (but not informatics). Thus we believe these GMER items are most appropriately included within Scientific Inquiry & Scholarship, where PBLI and SBP overlap. In addition to these broad competency relationships we have also endeavored to illustrate where the fields of Medical Statistics and Biostatistics & Epidemiology best fit in this theoretical model. These are tools that support and facilitate the interactions between competency domains, and thus are represented as two small circles at their intersections. We have placed Medical Statistics, including such skills as those that describe the reliability and validity of medical tests and values, as a body of knowledge overlapping Informatics, Medical Knowledge, Patient Care and EBM, linking them to PRLS. Informatics, with its own set of unique skills, is a growing and vital area of expertise for most healthcare professions, and is concerned with the optimal use of information, aided in part by technology, to improve health and healthcare.31 Informatics and medical statistics are interdependent in that informatics deals with searching and acquiring information, and medical

statistics deals with the understanding and interpretation of that information. Further, EBM is concerned primarily with combining medical knowledge with information, and applying the results to patient care. Informatics also plays an important role in SBP. For physicians as managers, leaders and collaborators to be involved in the larger system of care, they need tools to stay abreast of how the healthcare system works. Slawson and Shaughnessy underscored the value information management as a skill set necessary for the busy clinical practitioner who must focus on the usefulness of information on-the-spot with the patient. They defined usefulness as "relevance times validity divided by work."32 In our metaphorical PRLS tapestry, information management is included in "informatics." Medical statistics, informatics, information management, and EBM are necessarily related to, but do not, in our model, encompass PRLS. Likewise foundational PRLS cannot be proxies for them. Biostatistics & Epidemiology, as distinct disciplines, are bodies of knowledge that can facilitate meaningful interactions between PBLI, Communications & Professionalism, and Scientific Inquiry & Scholarship. Biostatistics and epidemiology are often referred to in the literature as proxies for research literacy rather than as essential tools in the PRLS repertoire. While neither of those bodies of knowledge alone is sufficient for 21st Century physicians to conduct scientific inquiry at any level of sophistication or to fully engage in scholarship, particularly in a global healthcare community, together they represent one of the many characteristics of physicians who have mastered PBLI and can effectively navigate EBM territory. Furthermore, although biostatistics and epidemiology are critical components of practitioner research literacy skills, they do not embody them. Rather, they facilitate the acquisition of the PBLI competency and enhance the roles of scholar, scientist and teacher. Without a conversancy in biostatistics and epidemiology, lifelong learning becomes quite difficult. In this theoretical model, PRLS are interwoven among all of the competency domains, and include skills associated with Informatics, Scientific Inquiry & Scholarship, and Communications & Professionalism. We have positioned Informatics in the overlap between SBP and Medical Knowledge & Patient Care. In keeping with observed trends in the emerging global competencies, we defined the area of overlap between PBLI and SBP as representative of the roles of scholar, scientist, teacher, manager,


and collaborator. PBLI is more associated with the lifelong learning aspect of those roles, whereas SBP is more associated with the roles of manager, and collaborator. The relationship between PRLS and Communications & Professionalism is explained by the elements of critical thinking, creativity, curiosity, and ethics. These as well as values and attitudes characterize the "complete 21st Century healthcare practitioner."

2.2. Conceptual paradigms linking core competencies with practitioner research literacy skills. There are three conceptual paradigms associated in the international literature with the core competencies that also serve to link them with PRLS. These paradigms are lifelong learning, generic skills, and the T-shaped professional. Together they enrich the texture of our metaphorical PRLS tapestry, and help to elucidate the relationship between PRLS and the core competencies, and the unique suitability of PRLS for integration in UME. In the following sections we discuss each paradigm as a thread running throughout the literature on both core competencies and CBME. The Lifelong Learning Paradigm is a commonly accepted paradigm often encountered in the literature and in all of the core competency standards. Multiple tools have been developed in an attempt to measure lifelong learning, asserting that this observed phenomenon has a defined set of characteristics.33 The items on most instruments that purport to measure lifelong learning as a construct, actually define attitudes and self-reported behaviors, and do not test demonstrable skills that have become associated with lifelong learners. From the items in those instruments however, one can glean several composite skills that characterize a lifelong learner, such as effective acquisition of new knowledge, efficient information search and synthesis skills, and demonstrable understanding of scientific literature. The Generic Skills Paradigm is a construct described by Murdoch-Eaton and Whittle as a collection of skills associated with lifelong learning and critical thinking.34 This construct dominates the efforts in secondary schools and colleges in England, Wales and Northern Ireland, to define and measure "functional skills" expected of graduates. The developing "functional skills" will be the next generation of the current "basic skills" that include number-based skills, communication, information and communication technology skills, and the

ability to improve one’s own learning and performance, and to work with others. The literature on the generic skills paradigm could inspire important discussions among medical educators when considering how to define and measure competency-based learning outcomes and to link performance in the core competencies to the application of research literacy skills. The T-Shaped Professional Paradigm has its origins in the computer sciences industry where individuals were needed who had both the depth of knowledge in their technical field and broader skills such as effective interpersonal communications and problem-solving abilities. The vertical stroke of the "T" represents the depth, and the horizontal stroke represents the breadth. In a call for research-driven medical education, Donofrio et al. suggest that medical education has focused on the depth, but because of the “too much to know and too little time to teach it” scenario, it has historically under-attended to the breadth.35 He advocates a “research-driven” medical curriculum in which there is greater attention to horizontal integration between disciplines, and vertical integration between basic and clinical sciences, with a balance between training generalists and specialists. The idea of a T-shaped professional helps to elucidate the connection between the core competencies and PRLS in a CBME model: depth in medical knowledge and breadth in critical thinking, problem solving, and other skills in the genetic composition of PRLS.

2.3. Summary These three paradigms support the function of a CBME in developing the T-shaped professional. If we think of lifelong learning and generic skills as the horizontal stroke of the T (the “breadth”), and the fundamentals of medicine as the vertical stroke (the “depth”), we can see that a competency-based education model would be more effective than the linear model has been at inspiring such a professional. Recommendations from a joint study by the AAMC and the Howard Hughes Medical Institute in 2009, serendipitously describe a “T-shaped professional,” in describing one who is likely to have the qualities of a lifelong learner. That report, Scientific Foundations for Future Physicians, calls for all students entering medical school to already have demonstrated competencies in scientific inquiry, and the process of discovering and validating new knowledge.26 The list of expected skills within those competency areas include basic, foundational research literacy skills such as


developing testable questions, designing research studies, acquiring and analyzing data and presenting the findings. But the list does not stop there. It also includes informatics, critical appraisal, ethics and teamwork, all at a basic, foundational level of proficiency. Clearly if pursued, the recommendations of that report will eventually have a broad and profound impact not only on pre-medical school curricula, but also on national qualifying examinations and the medical school curriculum in the US. So important to our PRLS tapestry are four of the conclusions in the joint report by the AAMC and the Howard Hughes Medical Institute, that we have provided them below, with certain terms and phrases underlined for emphasis.

• The practice of medicine requires grounding in scientific principles and knowledge, as well as understanding how current medical knowledge is scientifically justified and how that knowledge evolves.

• It is essential not only to read the medical and scientific literature of one’s discipline, but to examine it critically to achieve lifelong learning. These activities require knowledge and skills in critical analysis, statistical inference, and experimental design.

• Curiosity, skepticism, objectivity, and the use of scientific reasoning are fundamental to the practice of medicine. These attributes should permeate the entire medical education continuum.

• Effective clinical problem solving and the ability to evaluate competing claims in the medical literature and by those in medical industries depend on the acquisition, understanding, and application of scientific knowledge and scientific reasoning based on evidence.

Given these nascent expectations for students entering medical school, it is reasonable to forecast that by 2020, we will be teaching men and women who will not only be prepared for, but will also expect a level of sophistication in their medical training experiences that many medical schools cannot currently provide. There is a groundswell of imperatives to move well beyond "teaching" to integrating research-related competencies into the UME curriculum. This portrayal of a conceptual tapestry of interrelated competencies and skills sets is integral to the idea that there might be a plausible universal

model for integrating research literacy skills into UME. It also helps to explain the idea of PRLS as an empirical construct. At the center, these competencies and skills are linked by the components of PRLS, thus establishing PRLS as uniquely suited for integration into a competency-based medical education curriculum.

3. Can thinking globally facilitate the

integration of practitioner research literacy skills into a competency-

based medical education curriculum?

3.1. Emerging global perspectives Within the last two years, two unrelated reports on the role of basic sciences in medical education have shed some light on the value of practitioner research literacy skills in the context of core competencies. The first of these reports was published in 2010, summarizing the findings of a study group convened by the International Association of Medical Science Educators (IAMSE Study Group).3 This IAMSE Study Group endorsed an extensive list of sciences they referred to as “foundational sciences,” including the traditionally defined “basic sciences,” and adding others including behavioral sciences, public health sciences, and biostatistics. They affirmed that such foundational sciences are essential for clinical applications and for developing thinking skills. After the IAMSE Study Group deliberated about when and how these foundational sciences should be incorporated into the curriculum, they concluded that an upward, spiral approach should be used. They recognized this approach as dramatically different from the linear approach historically used in UME. Finally, the study group agreed that knowing what to teach is insufficient; whereas knowing what, and how, and when to teach material is essential. The IAMSE study group provides a meaningful foundation for discussions among medical educators about curricular reform addressing the effective horizontal integration of basic and clinical sciences. The second report was published in 2011 in Perspectives in Biology and Medicine as a summary of a remarkable study by the AAMC and the Howard Hughes Medical Institute on competencies in premedical and medical education.36 Two of the conclusions from that report most relevant to our tapestry, were that competencies rather than course requirements should shape the curricula, and that the foundational sciences (similarly defined as in Finnerty et al.) should be better integrated with clinical courses.


PRLS enhance and support all the emerging global core competencies, as they are broader and more foundational than previously thought, and lay the foundation for lifelong learning and generic skills essential to the complete professional of the future. These skills are developed over time, and cannot be acquired in a single class in biostatistics or epidemiology, or in a week of “informatics,” or a weekend of EBM training. If we embrace a CBME model, grounded in learner-centered frameworks in which there is horizontal integration between disciplines, and vertical integration between basic and clinical sciences, then it follows that PRLS as generic skills closely associated with lifelong learning and the T-shaped professional, can be integrated in a hierarchical fashion across all years of UME.

3.2. Trends As we mentioned earlier, biostatistics and epidemiology have historically served as proxies for PRLS. One of the earliest published papers on the value of teaching statistics and epidemiologic principles to medical students reported the results of a symposium at the American Statistical Association's 1967 annual meeting.37 That paper provides several enduring learning objectives and principles for effectively teaching medical students to understand the scholarly literature. Some of the learning objectives identified by the symposium participants include data management, sampling methodology, formulating testable questions, as well as a short list of concepts such as variability in biologic data, tests of significance, confidence intervals, statistical distribution, probability theory, assumptions for statistical tests, Type I and Type II errors, and regression analysis. Some of the principles endorsed by the participants are: use of practical, contemporary, preferably local research examples in which to apply the statistical concepts, avoidance of complex computations, integration of statistics and epidemiology in the clinical pedagogy across all years of the UME. Although the symposium reported reference to understanding experimental design in the list of learning objectives, the recommended topics did not include orientation to the various types of research, research design and methods, or ethics in research. The importance of that paper lies in the vision of the study group, that statisticians were predicting a need for PRLS in the UME curriculum. Further, the list of concepts identified by that group has not changed and has been iterated and expanded throughout the emerging global core competencies.

The caveat however, is that perhaps all medical students do not need to attain an advanced level of understanding of biostatistics and epidemiology. All medical students do however, need to achieve a level of conversancy with research-related concepts to enable them to fully engage in the required core competencies. Fast-forwarding to 1987, we get a peek at a controlled trial testing the benefits of teaching critical appraisal skills to medical students at McMaster University in Canada, birthplace of EBM. That well-crafted study included pre-intervention training of clerkship preceptors as critical appraisal coaches, multi-method approaches to teaching, and previously tested critical appraisal "packages" with course objectives and guidelines, clinical scenarios, and relevant literature. Exposed third year clinical clerks demonstrated sufficient improvement in measurably defined critical appraisal skills compared to the non-exposed group. These differences convinced the curriculum committee to add the learning activities and materials to all of the clinical rotations for McMaster University medical students. One caveat is of utmost importance regarding that study. The curriculum at McMaster University is quite likely unique among medical education systems. Their medical students are exposed to research-related concepts early in the curriculum, and thus may be better prepared to benefit from brief critical appraisal skill training in their third year. Although the richness of the academic experience enjoyed by McMaster University medical students may not be conducive to efficient replication in all medical schools worldwide, their strategies can be applied universally by using mixed methods of teaching, incorporating practice of knowledge using practical analytic tools, and measuring outcomes that secured the endorsement of policy makers for curricular change. As we have mentioned earlier, a single coherent theoretical construct of PRLS is broader in scope than biostatistics, epidemiology or EBM alone. Thus our goal for this monograph is to suggest a plausible universal approach to integrating PRLS into the UME curriculum using clearly defined and measurable learning objectives and outcomes. To this end we include in the following sections, a brief synopsis of five previously published curricular models for teaching biostatistics and epidemiology to medical students.


3.3. Five Models A multi-disciplinary team teaching model incorporating clinically relevant activities for teaching statistics to medical students in the UK, was described by Freeman et al.39 Their model used mixed methods, combining lectures with video vignettes followed with small group facilitated discussions. While their novel approach was more resource intensive from beginning to end than the previous course, students reported significant improvement in knowledge and attitudes. Although their course was short, and there has been no published longitudinal outcome study of its benefits, they used several innovative methods including team teaching, learning objectives that defined the end-point in terms of roles (consumer of statistics versus a researcher), and combined objective and subjective outcome measures. That model is readily replicable. A case-discussion model was described as beneficial for teaching epidemiological and biostatistical concepts to first year US medical students.40 The process required a three week series of lectures and six weeks of 2-hour small group seminars. Marantz et al. measured students' attitudes and opinions, and learned that some students wanted more technical rigor, and some were confused about the clinical content of the cases. Additionally this course relied heavily on pre-curriculum faculty development and large numbers of faculty facilitators for the small group sessions. The use of clinical case scenarios may be too complex for first year students, immersed as they are in the foundational sciences. This strategy however may be suited to second year students as they work with case scenarios in their clinical science courses. PRLS can be integrated into existing clinical case discussions, and models are available to accomplish that integration.41

An enhanced linear research education curriculum, published by a German-Swiss collaborative team, improved students' attitudes toward research, and discovered that students who felt more secure in their understanding of research methods and statistics were more likely to perform MEDLINE searches and use original journal articles in their research. Furthermore, students in the traditional curriculum were more likely to take an optional 5th year EBM course than students in the enhanced curriculum.42 The most relevant aspect of that study for our theoretical PRLS model is the important differences found between students interested in pursuing research as part of their

career goals compared to those who prefer to focus more exclusively on clinical practice. In a 20-hour course in research methodology, medical students at Zagreb University in Croatia reported improved attitudes toward research and improved knowledge scores after, compared to before the course.43 Hren et al. came closest to approximating our theoretical construct of PRLS in their model by using a broader approach to familiarizing students early in their education with the language of research and its role in science and medicine. Finally, a two part course in EBM, with a short 22-hour lecture-based course followed by integrating EBM assignments into the clerkship/clinical rotations significantly improved US students' self-rated knowledge and attitudes toward EBM.44 In that study, because of the focus on EBM skills specifically, the authors used an EBM-specific outcomes assessment tool not suitable for measuring first or second year students' understanding of research-related concepts.

3.4. Summary The preponderance of published models that might be suitable as a plausible universal approach, aiming for the preparation of all physicians in the roles defined by the emerging global core competencies, address only some of the components of PRLS. Furthermore most curricular models for teaching research-related skills have been stand-alone models rather than vertically or hierarchically integrated. Because most medical schools have used a linear model for UME, it has been difficult to teach PRLS from a competency-based perspective. Nonetheless we can benefit from the innovative teaching and learning strategies they have tested.

4. Applying the Theories and Models

4.1. Defining PRLS throughout the

core competencies Perhaps the most sweeping statement about globalization of standards for medical education and competency expectations can be found in a 2010 paper by an international team discussing imperatives for how we train future healthcare professionals. That paper placed great importance on creating scholars with critical thinking skills of a breadth sufficient to effectively solve health problems in an interprofessional environment in this century and beyond.45 To ensure that all future physicians possess the qualities of scholar and scientist, we have an obligation to begin early,


balance the perspectives to which they are exposed, and provide meaningful context to their learning experiences in relevant skills. Our definition of PRLS incorporates the elements in each of the emerging global core competencies that refer to research-related skills, the principles underlying the CBME model, and the criteria for defining and measuring performance. The elements of the core competencies referring to research-related skills can be gathered as threads to weave a coherent construct of PRLS, the details of which play out in discrete teaching moments.

Figure 2 depicts the hierarchical, developmental progression of PRLS suitable for use in a competency-based curricular model. Each of these steps in the PRLS hierarchy calls for and deserves different learning objectives, teaching strategies, learning activities, and outcome measures. The Basic, Intermediate and Apprentice levels approximate years one through four of medical education. For five year programs, any of those steps may span several temporal divisions. In this model, students would progress along a continuum, from simple to more complex cognitive tasks, achieving mastery in critical thinking, communications, problem-solving, and other meta-cognitive skills.

4.2. Curriculum Integration The consensus in the international literature we reviewed on the optimum methods and timing for teaching research-related skills in UME were that it should be 1) designed with the stage of the learner in mind, 2) introduced early, and 3) provided hierarchically and iteratively across all years. Our proposed PRLS training model could be hierarchically integrated across the UME program. The process begins in the first year and continues incrementally and iteratively across all years.

Kahn and Coomaraasamy, a team from the UK, found, from their literature review in 2006, that the use of interactive and clinically integrated teaching and learning activities for post-graduate and continuing medical education in EBM is most effective.46 Our premise is that the same method can also be used to teach PRLS in UME. The greatest challenge to integrating new materials into the UME curriculum is the time and space available in the curriculum and in the students' academic life. To address this barrier to teaching PRLS in the UME curriculum, we propose multiple strategies


including the most effective evidence-based teaching and learning methods. Basic PRLS can be defined and organized to be achievable by a student with limited-to-no exposure to clinical research. Many new medical students are acquainted with basic science wet lab research tasks such as pipetting, preparing slides, recording from laboratory experiments, and running simple lab tests. Few however, have any experience with human subjects' or other clinical or biomedical research. The primary learning objective for the Basic level of PRLS is to be able to competently and confidently read and understand a simple to minimally complex biomedical/clinical research article. The delivery vehicle becomes a set of learning objectives that can be achieved using self-directed, interactive, on-line learning, such as a course we have developed for use at our medical school. The international literature also supports e-learning as equally effective as lectures or other face-to-face learning activities, and easy to use for computer-literate learners.47-50 Current technology enables the capture of grades and provides security such as

single-user access codes, and tracking of on-line activity. According to the published best practices, selection of learning materials should be guided by the developmental stage of the learner. However, we found no guidance in the literature addressing exactly what articles should be selected to be suitable to the developmental stage of the learner. Thus to increase the likelihood of success for students at a Basic level of learning PRLS, we recommend using a broad set of criteria to help match the complexity of the article with the readiness of the learner. Generally we recommend evaluating articles using criteria within four categories: the clinical topic, the research design, the study methods and the level of statistical complexity. Thus a suitable article for introduction to Basic PRLS would likely be one with a simple and transparent study design, preferably non-experimental, with methods well described by the authors, and containing mostly descriptive statistics. Furthermore, the clinical topic should be aligned with the students’ stage of medical training.

A second level of complexity would build on the previous scenario. At that next level the article chosen might use a simple experimental design, with equally straightforward inferential statistics such as t-tests. According to our model, the most complex article for a student at the Basic PRLS level

would be a cohort study using for example, data from a nationally representative survey. Such an article would include a more complicated sampling strategy, and use slightly more sophisticated statistics such as simple regression. As before, the


clinical topic should be in keeping with the students’ stage in medical training. Broadly defined learning modules, all based on the overall learning objectives, and maintaining the criteria of "minimally complex," would equip the students with information that would allow them for example, to perform the following knowledge tasks.

Define different areas and types of biomedical and clinical research

Recognize different types of research design

Define the relationship between the study question and its design and methods

Identify strengths and limitations of simple research designs

Recognize concepts related to study validity

Define descriptive and inferential statistics

Recognize the purpose of the most commonly used statistics

List the fundamental concepts of epidemiology

Describe the most important contemporary guidelines for reporting research

Recognize and define the use of basic steps and concepts in EBM

Answer simple questions about ethics in human subjects' research

The Basic level provides a platform for students to take the step to the next level that is Intermediate PRLS, requiring application of the foundational concepts – taking the rote knowledge to the next level – by discussing a research study on a relevant clinical topic in a classroom based interactive exercise or small group discussion. The on-line modules would remain a resource for them to revisit concepts as needed to understand the article and participate in facilitated learning activities. Ideally the faculty facilitators would include both a researcher or a clinical biostatistician (not a statistician), and a clinical expert in the topic. Faculty preparation is crucial for a successful learning experience in the Intermediate PRLS developmental stage. Faculty preparation requires several conditions be present, including at least: 1) demonstrated expertise in the topic and learning materials, 2) clear communication between the members of the teaching team, 3) well scripted roles and materials suited for each team member, and 4) a commitment to the learning objectives of the course.

4.3. Rubrics and Learning Tools Rubrics can help educators to craft measurable learning outcomes and develop relevant outcome measures. Gagné et al. illustrate the use of rubrics that rely on adult learning theories, that we suggest can be applied to multiple learning objectives within the family of PRLS concepts.51 A rubric is basically a blueprint or map that guides concept mastery starting at the Basic level and proceeding through the Intermediate, Apprentice and Advanced levels, thus needs only to be created once. A well-crafted rubric organizes learning objectives in the form of skills or proficiencies as goals for acquisition in incremental steps across time. For example, PRLS development begins with the most basic ability to recognize, recall, and discriminate factual knowledge of fundamental concepts, and progresses along a continuum through application, to synthesis and problem solving in later stages of learning. Figure 4 provides an example of a PRLS Acquisition Rubric for the concept of research questions and testable hypotheses. This rubric is organized to present a progressively developmental approach to teaching these skills. Pre-clinical years are most suited for the learning objectives with the lightest shading. These years would focus on fundamental knowledge of the definitions and rules for question and hypothesis formulation. Then as the student progresses through the levels, they will be able to formulate testable questions, develop variations of those questions, and ultimately develop testable hypotheses for applying the scientific method or solving a problem. This type of rubric can be created for a range of research-related concepts such as learning about different types of research design, or interpreting inferential statistics, but also might deal with very narrow topics such as learning about institutional review board protocol development, or presenting scientific findings orally or in print. The rubric permits students to self-pace through the stages of learning, moving forwards and backwards as needed for acquiring mastery. Furthermore, the level of integration into existing curriculum will vary necessarily, as some elements (such as knowledge discrimination) might require a more didactic or straightforward setting, but others such as application of problem solving may best occur in a journal club or research team setting.


Practitioner Research Literacy Skills- Article Review Tool While the idea of reviewing research literature is not new, early career medical students without PRLS usually take one of two approaches. The first is that they report reading only the abstract and conclusions section, generally skipping over the methods and findings sections. The second is that rather than reviewing the articles for research concepts and content and how the material might apply to their clinical practice, they instead provide a critique of the article. However, such approaches will lose many of the nuances related to specific research concepts and the overall clinical implications of the material. Through our own teaching experiences, we quickly realized that when presenting UME students with research literature to read, we needed to provide a template for them to use in guided article review. We created a tool called the Practitioner Research Literacy Skills - Article Review Tool (PRLS-ART)

that provides a systematized list of questions that walks the reader through an article. This tool differs from those that are related to EBM specific exercises in that instead of reviewing literature to determine how it fits into patient care, it emphasizes the recognition and identification of research concepts as the reader moves through the article. This unique approach facilitates practice understanding fundamental research literacy skills. Question items follow the general arrangement of contemporary research literature. Each question asks the reader to identify key concepts in each section of the article in context such as hypothesis, research design, outcome variables and how they were measured, and statistical findings. For figure 5, see appendix. The PRLS-ART has been used effectively with students and practitioners in workshops we have conducted, as well as in residents' journal club activities. For students at the Basic PRLS stage, it can be used for practice in recognizing and correctly


identifying or abstracting information from a minimally complex article, and can be used across all the steps of PRLS development. For students in the intermediate process of skill development, this tool can serve as a guide for classroom or small group discussions about the relationship between a clinical case scenario and contemporary research literature.

4.4. Measuring outcomes In 2006, Shanneyfelt et al. reported identifying 104 unique instruments to measure skills in "evidence-based practice" (EBP).52 That extensive article categorized learning outcome measurements into three types: individual formative or summative EBP evaluation, programmatic curriculum evaluation, and behavioral evaluation. Most of the instruments were used in studies evaluating practitioners' demonstrated skills in formulating answerable clinical questions, conducting literature searches, making clinical decisions using EBP tools, and interpreting evidence using case scenarios. Only five published studies in that review included medical students, and all of them were medical students in clinical training, thus had completed one to two years of UME. The authors concluded that we need more evaluation methods balancing psychometric robustness with feasibility, and that evaluation of behaviors has lagged behind the science of evaluating knowledge and skills. Donna Windish reported in a 2007 study that among 367 medicine residents in 11 GME programs, most lacked the knowledge in biostatistics considered necessary to interpret clinical research findings.53 She provided her complete knowledge test instrument. In our article on learning outcomes from a biomedical research course, we included with permission, seven of her 20 items in our 20-item knowledge test. We used less complex questions because we were approaching first and second year students at an earlier stage of their medical education than post-graduate physicians. We also provided a "no response" option to discourage guessing. Following a 24 contact-hour course using traditional lecture, reading assignments, and tests, students significantly improved their test scores, but still performed on average at only 40%.54 Another interesting dimension of our research is that the best predictor of test performance was the extent to which students expressed confidence in their ability to interpret certain research-related concepts. Nevertheless, students correctly answered on average, less than half of the questions they attempted.

Carraccio et al. discuss plausible approaches to accomplish the shift in paradigms from structure and process-based medical education models to a fully implemented and evaluated CBME.28 They reinforce the idea that benchmarks for defining competencies and thresholds for demonstrating proficiency in them, must be clearly delineated to maximize the benefits of a CBME model. With the heavy reliance on tests of knowledge, it is difficult to shift our thinking to process and attitude, and performance or behavioral manifestations of competencies. There are however several emerging methodologies for evaluating outcomes in the context of a CBME model. All of these will require a paradigm shift in thinking, and improvements in institutional infrastructures. These include for example, the portfolio approach described by Dannefer et al., and the empowerment approach described by Fetterman et al.56, 57

The portfolio approach to evaluation is utilized more by behavioral and public health professions than by the more technically oriented healthcare professions such as medicine. It requires a different orientation to learning. This is a process that requires students to identify their own abilities and limitations, and to develop strategies to convert limitations to strengths. It requires reflection on learning, essays, a significant amount of data on activities and knowledge acquired, and formative assessment sessions with mentors. The portfolio approach to evaluation is not an objective collection of materials, but an intimate and continuous interaction with self and others that is documented and reviewed with the goal of growth and maturity. The empowerment evaluation process described by Fetterman requires five conditions to be present: a culture of evidence, a critical friend, a cycle of reflection and action, a community of learners, and reflective practitioners. Each of these conditions is accompanied by a program of preparation and activities to bring them to life. Collecting information and conducting meetings to review and discuss the data needs to occur regularly and continuously to ensure this process functions as it is intended. It requires time, openness and forthrightness on the part of all who participate. Undertaking this type of evaluation process requires the commitment of leadership to support the time required for its success.


5. Conclusions The primary aim of this monograph was to synthesize the international trends in developing core competencies specifically as related to practitioner research literacy skills (PRLS), in such a way as to formulate a plausibly universal approach to integrating PRLS into a CBME curriculum. A universal approach to teaching PRLS in the UME would equip future physicians with a measure of PRLS during their undergraduate training years that would position them to effectively engage in the core competencies in residency and beyond. We propose that such a strategy is supported by the emerging global value placed on those skills that support the multiple roles of the 21st century physician. In essence, PRLS can be defined as those skills that characterize a "scholar," a "scientist," "teacher," "manager," and "collaborator." As our society and healthcare systems become increasingly dependent on and driven by data, physicians are increasingly expected to be able to be discerning and judicious users of those data in their various roles. After reviewing the literature and published reports on emerging trends in physicians' core competencies, and the methods and measurements for teaching research-related skills in medical schools, we have suggested that a coherent conceptual construct of PRLS is interwoven throughout all of the core competencies, and that PRLS can be effectively integrated into a CBME curriculum. Except for those programs that are research intensive, and provide optional career paths in academic and research arenas, most medical schools focus on training physicians for almost exclusively the patient care setting, thus given less attention to research literacy. And while some schools teach EBM principles, the overt instructional and performance assessment components are often elided. It appears nonetheless, from the reports by the Institute of Medicine, the AAMC and the Howard Hughes Medical Institute, and international study groups such as IAMSE, that there is an emerging imperative to ensure that all future physicians are capable of effectively navigating the world of research in medicine and healthcare. These skills include being able to examine and apply research information in their practice and their interactions with others, invest themselves in lifelong learning, and participate informatively in the larger healthcare arena, locally and globally. As we move toward fully implemented CBME models in undergraduate medical education, PRLS are

genuinely suited for integration both vertically and hierarchically throughout the curriculum, thus responding effectively to this imperative. Throughout the scholarly international literature on competencies, medical education and the foundational sciences, study groups have urged that educators excite students about research; that they share with them a spirit of curiosity, skepticism, creativity, and beauty about science and the scientific method. Specific skills iterated by these learned groups include critical analysis, statistical inference, scientific principles, and how knowledge evolves. Medical educators and students should consider embracing PRLS as an integral part of their medical education; skills that are no longer peripheral, or irrelevant to, or isolated from clinical training. In keeping with global perspectives, PRLS need to be clearly defined and linked throughout the core competencies utilizing appropriate, incremental levels of cognitive challenge and higher order thinking paradigms to achieve measurable outcomes as behaviors reflecting the desired characteristics of the expected roles of the physician. As educators of future physicians, we should continue to challenge ourselves to create interesting and exciting, as well as utilitarian methods of integrating PRLS into the fabric of all of the core competencies and thus into a competency-based medical education curriculum.


Notes on Contributors DES ANGES CRUSER, PhD, MPA is Associate Professor at University of North Texas Health Science Center, Texas College of Osteopathic Medicine Fort Worth, Texas, USA. SARAH K. BROWN, DrPH is Assistant Professor at University of North Texas Health Science Center, Texas College of Osteopathic Medicine Fort Worth, Texas, USA. JESSICA R. INGRAM, MPH is Instructor at University of North Texas Health Science Center, Texas College of Osteopathic Medicine Fort Worth, Texas, USA. FRANK PAPA, DO, PhD is Professor at University of North Texas Health Science Center, Texas College of Osteopathic Medicine Fort Worth, Texas, USA. ALAN L. PODAWILTZ, DO, MS is Associate Professor at University of North Texas Health Science Center, Texas College of Osteopathic Medicine Fort Worth, Texas, USA. DAVID LEE, is Research Assistant at University of North Texas Health Science Center, Texas College of Osteopathic Medicine Fort Worth, Texas, USA. VESNA KNOX, CSM is Technical Advisor at University of North Texas Health Science Center, Texas College of Osteopathic Medicine Fort Worth, Texas, USA.

Keywords Research Literacy Skills, Undergraduate Medical Education

References 1. Jenicek M. The hard art of soft science:

Evidence-Based Medicine, Reasoned Medicine or both? J Eval Clin Pract 2006 Aug;12(4):410-419.

2. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ 1996 Jan 13;312(7023):71-72.

3. Finnerty EP, Chauvin S, Bonaminio G, Andrews M, Carroll RG, Pangaro LN. Flexner revisited: the role and value of the basic sciences in medical education. Acad Med 2010 Feb;85(2):349-355.

4. Albanese MA, Mejicano G, Mullan P, Kokotailo P, Gruppen L. Defining characteristics of educational competencies. Med Educ 2008 Mar;42(3):248-255.

5. Stewart MG. Core Competencies [Internet]. 2001 [cited 2012 Mar 5]. Available from: http://www.acgme.org/acWebsite/RRC_280/280_coreComp.asp.

6. National Board of Osteopathic Medical Examiners. Fundamental osteopathic medical competency domains: guidelines for osteopathic medical licensure and the practice of osteopathic medicine. Chicago (IL): National Board of Osteopathic Medical Examiners (US); 2011 Jun.

7. Liaison Committee on Medical Education. Functions and structure of a medical school: standards for accreditation of medical education programs leading to the M.D. degree. Chicago (IL): Liaison Committee on Medical Education (US); 2011 May.

8. Frank, JR. The CanMEDS 2005 physician competency framework: better standards, better physicians, better care. Ottawa (CAN): The Royal College of Physicians and Surgeons of Canada (CAN); 2005 Mar.

9. General Medical Council. Tomorrow’s doctors: education outcomes and standards for undergraduate medical education. London (UK): General Medical Council (UK); 2009 Sep.

10. Australian Medical Council. Standards for assessment and accreditation of medical schools by the Australian Medical Council 2010. Kingston (AUS): Australian Medical Council Limited (AUS); 2010 Dec.

11. WFME/AMSE International Task Force. WFME global standards for quality improvement in medical education: European specifications. Denmark (UK): World Federation for Medical Education (UK); 2007.

https://www.researchgate.net/publication/6879208_The_hard_art_of_soft_science_Evidence-Based_Medicine_Reasoned_Medicine_or_both?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/14644450_Evidence-based_Medicine_What_It_Is_and_What_It_Isn't?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/5577484_Defining_characteristics_of_educational_competencies?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/41166371_Flexner_Revisited_The_Role_and_Value_of_the_Basic_Sciences_in_Medical_Education?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==









12. Core Committee Institute for International Medical Education. Global minimum essential requirements in medical education. Medical Teacher 2002;24(2):130–135.

13. American Osteopathic Association. AOA basic documents for postdoctoral training. Chicago (IL): American Osteopathic Association (US); 2011 Jul.

14. Greiner AC, Knebel E. Health Professions Education: A Bridge to Quality. Washington, DC: The National Academies Press; 2003.

15. AAMC’s Task Force II on Clinical Research. Promoting translational and clinical science: the critical role of medical schools and teaching hospitals. Washington (DC): Association of American Medical Colleges (US); 2006.

16. Karle H, Executive Council, World Federation for Medical Education. International recognition of basic medical education programmes. Med Educ 2008 Jan;42(1):12-17.

17. Nair M, Webster P. Education for health professionals in the emerging market economies: a literature review. Med Educ 2010 Sep;44(9):856-863.

18. Pulido PAM, Cravioto A, Pereda A, Rondon R, Pereira G. Changes, trends and challenges of medical education in Latin America. MedTeach 2006;28(1):24–29.

19. Schwarz MR, Wojtczak A, Zhou T. Medical education in China's leading medical schools. Med Teach 2004 May;26(3):215-222.

20. Cueto J,Jr, Burch VC, Adnan NA, Afolabi BB, Ismail Z, Jafri W, et al. Accreditation of undergraduate medical training programs: practices in nine developing countries as compared with the United States. Educ Health (Abingdon) 2006 Jul;19(2):207-222.

21. Foundation for Advancement of International Medical Education and Research: Strategic Plan [Internet]. 2012 [updated 2012 Feb 22; cited 2012 Mar 5]. Available from: http://www.faimer.org/about-strategic-plan.html

22. American Medical Association. Professing Medicine: Strengthening the Ethics and Professionalism of Tomorrow's Physicians. Chicago, IL: American Medical Association; 2001.

23. Calman K. Medical Education Past, Present and Furture: Handing on Learning. 1st ed. Philadelphia, PA: Churchill Livingstone Elsevier; 2007.

24. van Zanten M, Norcini JJ, Boulet JR, Simon F. Overview of accreditation of undergraduate medical education programmes worldwide. Med Educ 2008 Sep;42(9):930-937.

25. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Acad Med 2009 Mar;84(3):301-309.

26. AAMC-HHMI Committee. Scientific foundations for future physicians. Washington (DC): Association of American Medical Colleges (US); 2009.

27. Spady WG. Competency based education: a bandwagon in search of a definition. Educational Researcher 1977 Jan;6(1):9-14.

28. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med 2002 May;77(5):361-367.

29. Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, et al. Competency-based medical education: theory to practice. Med Teach 2010;32(8):638-645.

30. Harden RM, Crosby JR, Davis MH, Friedman M. AMEE Guide No. 14. Outcome-based education, part 5. From competency to meta-competency: a model for the specification of learning outcomes. Med Teach 1999;21(6):546–52.

31. Hersh W. A stimulus to define informatics and health information technology. BMC Med Inform Decis Mak 2009 May 15;9:24.

32. Slawson DC, Shaughnessy AF. Teaching evidence-based medicine: should we be teaching information management instead? Acad Med 2005 Jul;80(7):685-689.

33. Wetzel AP, Mazmanian PE, Hojat M, Kreutzer KO, Carrico RJ, Carr C, et al. Measuring medical students' orientation toward lifelong learning: a psychometric evaluation. Acad Med 2010 Oct;85(10 Suppl):S41-4.

34. Murdoch-Eaton D, Whittle S. Generic skills in medical education: developing the tools for successful lifelong learning. Med Educ 2012 Jan;46(1):120-128.

35. Donofrio N, Sophrer J, Zadeh HS. Research-driven medical education and practice: a case for t-shaped professionals. MJA Viewpoint [Internet]. 2010 [Cited 2012 Mar 5]. Available from: http://www.ceri.msu.edu/wp-content/uploads/2010/06/A-Case-for-T-Shaped-Professionals-20090907-Hossein.pdf.

36. Alpern RJ, Belitsky R, Long S. Competencies in premedical and medical education: the AAMC-HHMI report. Perspect Biol Med 2011 Winter;54(1):30-35.

37. Bahn AK. Teaching biostatistics to medical students. J Med Educ 1969 Jul;44(7):622-626.

38. Bennett KJ, Sackett DL, Haynes RB, Neufeld VR, Tugwell P, Roberts R. A controlled trial of

https://www.researchgate.net/publication/5668241_International_recognition_of_basic_medical_education_programmes?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==







https://www.researchgate.net/publication/45694124_Education_for_health_professionals_in_the_emerging_market_economies_A_literature_review?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/7153838_Changes_trends_and_challenges_of_medical_education_in_Latin_America?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/8503436_Medical_education_in_China's_leading_medical_schools?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/250182157_Competency_Based_Education_A_Bandwagon_in_Search_of_a_Definition1?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/11360237_Shifting_paradigms_From_flexner_to_competencies?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/24430636_A_Stimulus_to_Define_Informatics_and_Health_Information_Technology?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/7763887_Teaching_Evidence-Based_Medicine_Should_We_Be_Teaching_Information_Management_Instead?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/51445899_Measuring_Medical_Students'_Orientation_Toward_Lifelong_Learning_A_Psychometric_Evaluation?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==





https://www.researchgate.net/publication/51860566_Generic_skills_in_medical_education_Developing_the_tools_for_successful_lifelong_learning?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/50377017_Competencies_in_premedical_and_medical_education_The_AAMC-HHMI_report?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/17392393_Teaching_biostatistics_to_medical_students?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==

https://www.researchgate.net/publication/17392393_Teaching_biostatistics_to_medical_students?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==

https://www.researchgate.net/publication/24037981_Lurie_SJ_Mooney_CJ_Lyness_JM_Measurement_of_the_general_competencies_of_the_Accreditation_Council_for_Graduate_Medical_Education_A_systematic_review?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==








https://www.researchgate.net/publication/23163066_Overview_of_accreditation_of_undergraduate_medical_education_programmes?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==











teaching critical appraisal of the clinical literature to medical students. JAMA 1987 May 8;257(18):2451-2454.

39. Freeman JV, Collier S, Staniforth D, Smith KJ. Innovations in curriculum design: a multi-disciplinary approach to teaching statistics to undergraduate medical students. BMC Med Educ 2008 May 1;8:28.

40. Marantz PR, Burton W, Steiner-Grossman P. Using the case-discussion method to teach epidemiology and biostatistics. Acad Med 2003 Apr;78(4):365-371.

41. Lasater K, Salanti S, Fleishman S, Coletto J, Jin H, Lore R, et al. Learning activities to enhance research literacy in a CAM college curriculum. Altern Ther Health Med 2009 Jul-Aug;15(4):46-54.

42. Pruskil S, Burgwinkel P, Georg W, Keil T, Kiessling C. Medical students' attitudes towards science and involvement in research activities: a comparative study with students from a reformed and a traditional curriculum. Med Teach 2009 Jun;31(6):e254-9.

43. Hren D, Lukic IK, Marusic A, Vodopivec I, Vujaklija A, Hrabak M, et al. Teaching research methodology in medical schools: students' attitudes towards and knowledge about science. Med Educ 2004 Jan;38(1):81-86.

44. West CP, McDonald FS. Evaluation of a longitudinal medical school evidence-based medicine curriculum: a pilot study. J Gen Intern Med 2008 Jul;23(7):1057-1059.

45. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 2010 Dec 4;376(9756):1923-1958.

46. Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ 2006 Dec 15;6:59.

47. Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, Coomarasamy A. Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomised controlled trial. BMC Med Educ 2007 Jul 20;7:23.

48. Gagnon MP, Legare F, Labrecque M, Fremont P, Cauchon M, Desmartis M. Perceived barriers to completing an e-learning program on evidence-based medicine. Inform Prim Care 2007;15(2):83-91.

49. Hadley J, Kulier R, Zamora J, Coppus SF, Weinbrenner S, Meyerrose B, et al. Effectiveness of an e-learning course in evidence-based medicine for foundation (internship) training. J R Soc Med 2010 Jul;103(7):288-294.

50. Peska DN, Lewis KO. Uniform instruction using web-based, asynchronous technology in a geographically distributed clinical clerkship: analysis of osteopathic medical student participation and satisfaction. J Am Osteopath Assoc 2010 Mar;110(3):135-142.

51. Gagne RM, Wager WW, Golas KC, Keller JM. Principles of Instructional Design. 5th ed. Belmont, CA: Thomson Wadsworth; 2005.

52. Shaneyfelt T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, et al. Instruments for evaluating education in evidence-based practice: a systematic review. JAMA 2006 Sep 6;296(9):1116-1127.

53. Windish DM, Huot SJ, Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. JAMA 2007 Sep 5;298(9):1010-1022.

54. Cruser A, Brown SK, Ingram JR, Podawiltz AL, Dubin BD, Colston JS, et al. Learning outcomes from a biomedical research course for second year osteopathic medical students. Osteopath Med Prim Care 2010 Jul 8;4:4.

55. Cruser A, Dubin B, Brown SK, Bakken LL, Licciardone JC, Podawiltz AL, et al. Biomedical research competencies for osteopathic medical students. Osteopath Med Prim Care 2009 Oct 13;3:10.

56. Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med 2007 May;82(5):493-502.

57. Fetterman DM, Deitz J, Gesundheit N. Empowerment evaluation: a collaborative approach to evaluating and transforming a medical school curriculum. Acad Med 2010 May;85(5):813-820.

https://www.researchgate.net/publication/26687364_Learning_activities_to_enhance_research_literacy_in_a_CAM_college_curriculum?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==





https://www.researchgate.net/publication/26878060_Medical_students'_attitudes_towards_science_and_involvement_in_research_activities_A_comparative_study_with_students_from_a_reformed_and_a_traditional_curriculum?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==






https://www.researchgate.net/publication/227727719_Teaching_research_methodology_in_medical_schools_Students'_attitudes_towards_and_knowledge_about_science?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==





https://www.researchgate.net/publication/5229252_Evaluation_of_a_Longitudinal_Medical_School_Evidence-Based_Medicine_Curriculum_A_Pilot_Study?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/6625803_A_hierarchy_of_effective_teaching_and_learning_to_acquire_competence_in_evidence-based_medicine?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/6180101_Computer-based_teaching_is_as_good_as_face_to_face_lecture-based_teaching_of_evidence_based_medicine_A_randomised_controlled_trial?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==






https://www.researchgate.net/publication/43131203_Uniform_Instruction_Using_Web-Based_Asynchronous_Technology_in_a_Geographically_Distributed_Clinical_Clerkship_Analysis_of_Osteopathic_Medical_Student_Participation_and_Satisfaction?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==






https://www.researchgate.net/publication/6836602_Instruments_For_Evaluating_Education_In_Evidence-Based_Practice_A_Systematic_review?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==








https://www.researchgate.net/publication/6054180_Windish_DM_Huot_SJ_Green_MLMedicine_residents'_understanding_of_the_biostatistics_and_results_in_the_medical_literature_JAMA_2981010-1022?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/45717554_Learning_outcomes_from_a_biomedical_research_course_for_second_year_osteopathic_medical_students?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==





https://www.researchgate.net/publication/26890791_Biomedical_research_competencies_for_osteopathic_medical_students?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==





https://www.researchgate.net/publication/6372297_The_Portfolio_Approach_to_Competency-Based_Assessment_at_the_Cleveland_Clinic_Lerner_College_of_Medicine?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/44646570_Empowerment_Evaluation_A_Collaborative_Approach_to_Evaluating_and_Transforming_a_Medical_School_Curriculum?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




https://www.researchgate.net/publication/10807705_Using_the_Case-discussion_Method_to_Teach_Epidemiology_and_Biostatistics?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/5330463_Innovations_in_curriculum_design_A_multi-disciplinary_approach_to_teaching_statistics_to_undergraduate_medical_students?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==





https://www.researchgate.net/publication/272288463_A_Controlled_Trial_of_Teaching_Critical_Appraisal_of_the_Clinical_Literature_to_Medical_Students?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==



https://www.researchgate.net/publication/5964938_Perceived_barriers_to_completing_an_e-learning_program_on_evidence-based_medicine?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==





https://www.researchgate.net/publication/229525294_Principles_of_Instructional_Design?el=1_x_8&enrichId=rgreq-15a1e7ee9389423c0b1db6f187546f41-XXX&enrichSource=Y292ZXJQYWdlOzI2NDg5MDI3NDtBUzoxNDA1NDQ0MzcxMzMzMTJAMTQxMDUxOTgxMzU5Mg==




APPENDIX

practitioner research literacy skills in undergraduate medical education: thinking globally, acting...

Documents