the aim of the journal of asynchronous learning networks ... filejournal of asynchronous learning...

Journal of Asynchronous Learning N

etworks

Volume 17 Issue 2 - July 2013

Journal of

AsynchronousLearning Networks

Volume 17, Issue 4 - February 2014

ISSN 1939-5256 (print)ISSN 1092-8235 (online)

The aim of the Journal of Asynchronous Learning Networks is to describe original work in asynchronous learning networks (ALN), including experimental results. Our mission is to provide practitioners in online education with knowledge about the very best research in online learning. Papers emphasizing results, backed by data are the norm. Occasionally, papers reviewing broad areas are published, including critical reviews of thematic areas. Entire issues are published from time-to-time around single topic or disciplinary areas. The Journal adheres to traditional standards of review and authors are encouraged to provide quantitative data. The original objective of the Journal was to establish ALN as a �eld by publishing articles from authoritative and reliable sources. The Journal is now a major resource for knowledge about online learning.

Guest Editors:Laurie P. Dringus

Anthony G. PiccianoKaren Vignare

Introduction to the Special Issue on Online Learning: A Universe of Possibilities!

Journal of Asynchronous Learning Networks, Volume 17: Issue 4

INTRODUCTION TO THE SPECIAL ISSUE ON ONLINE LEARNING: A UNIVERSE OF POSSIBILITIES! Anthony G. Picciano Associate Editor: JALN In November 2013, the Sloan Consortium held its 19th Annual Conference on Online Learning, which attracted almost 3,000 attendees from all over the world. The theme of the conference, Online Learning: A Universe of Opportunities, focused on the growing popularity of online and blended learning as the instructional modalities of choice among students, faculty, and administrators. Historically, the major attraction of this conference has been the depth and breadth of its program. For 2013, over five hundred scholars, researchers, and practitioners presented papers, gave presentations, conducted workshops and poster sessions on the myriad of issues associated with online and blended learning. Three of the eight articles in this special edition reflect the best papers delivered at this conference as selected by the program committee. The other five articles were solicited because they exemplify the extent of issues and possibilities that online and blended learning are having on instruction.

The first article, Three Institutions, Three Approaches, One Goal: Addressing Quality Assurance in Online Learning, by Britto, Ford and Wise, examines approaches from three different types of institutions in addressing quality assurance in online education on their respective campuses. Specifically, this paper presents three case studies and describes each institution’s:

1) background and overview; 2) quality definition; 3) approach to quality assurance; 4) models and approaches; 5) goals; 6) successes; 7) challenges; and 8) lessons learned.

A comparison reveals that despite differences in scope, size, location, mission and extent of online development, there is consistency in the institutions’ strategies to addressing quality assurance in online learning.

The next article, Building a Foundation for Success through Student Services for Online Learners, by Newberry and DeLuca, examines student support services focused on the retention of students in distance learning programs at Duquesne University. The authors’ conclusion is that retaining online students goes beyond offering “equivalent” services to online learners; it requires a holistic and networked approach that leverages existing technologies and an institutional commitment to its online learners.



Matthews-DeNatale in, Are We Who We Think We Are? ePortfolios as a Tool for Curriculum Redesign, focuses on the use of ePortfolios to inform redesign and development of academic degree programs. She reports on Northeastern University’s decision in 2001, to institute an ePortfolio requirement for students enrolled in the University's Master of Education program. Systematic review of student ePortfolios informed planning for a 2012 master’s program redesign that began implementation in fall 2013. This redesign included the articulation of a program vision, program- and concentration-level outcomes, course map with signature assignments designed to evidence outcomes, ePortfolio curricular integration, and shared foundational courses across four concentrations. Bledsoe and Simmerok in, A Multimedia-Rich Platform to Enhance Student Engagement and Learning in an Online Environment, explore the implementation of a photo-rich comprehensive counseling center environment through which students of a semester-long online graduate psychology class learned about important research methodology concepts. Student feedback is provided as part of the findings along with implications for student learning in future online course endeavors.

Schwartz’s article provides a provocative examination entitled, Khan Academy: The Illusion of Understanding, and uses Khan Academy as an example of what some individuals are claiming to be the future of education. He offers five guiding observations that provide a structure for understanding the learning process and applies them to Khan Academy as a means of revealing what he calls the illusion of understanding, and he replaces that view with a more authentic understanding of the learning process and the means to achieve that understanding.

The next article, Conceptual Framework of Blended Professional Development for Mathematics Teachers,is a study by Lee that presents a conceptual framework of a blended professional development program for teacher success in learning a subject matter and enhancing instructional practice. Twenty-nine middle school teachers (22 first-year and 7 continuing teachers) participated in a year-long, blended professional development (PD) program. Effects of the blended PD program were measured by analyzing the content of participants’ virtual interactions; i.e., 1,149 online threaded discussion messages. Results showed participants’ teaching practices changed toward more student-centered lessons, encouraging verbal engagement, using various questioning strategies, and using collaborative group work while employing a wider variety of resources. Meyer in, An Analysis of the Research on Faculty Development for Online Teaching and identification of New Directions, presents the results of an extensive review of the published literature on faculty development for online teaching. This review included 68 articles and five books, which were reviewed to identify elements of the training and the findings (e.g., theoretical bases, training aims and organization, content of training, nature of sample (number and type of participants), outcome measures used). The review produced seven insights, from the importance of basing faculty development on theory, the frequency of publications that present models of faculty development for online teaching, the need to disentangle treatments, the need for rigorous evaluations, and the importance of considering individual faculty differences, designing and evaluating faculty development with specific outcome measures, and considering cost (either cost-efficiency or cost-effectiveness) when evaluating faculty development programs.



The last article in this special edition, A Conceptual Framework for Integrating Industry/Client-sponsored Projects into Online Capstone Courses, by Khan, addresses the growing interest in incorporating experiential learning into online degree programs. Khan proposes a design that could be applicable to any program at any institution. The research and data used to develop the framework was gathered from a literature review and a survey of University of Maryland University College (UMUC) graduate programs. The proposed framework was tested in two capstone courses using industry/client sponsored projects in the author's disciplines as the first stage of testing of the model. In closing, we would like to acknowledge the efforts of a number of individuals who have helped make this special edition possible. First, thanks go to the 2013 International Sloan-C Conference Planning Committee especially its chairman, Eric Frederickson. Second, the Conference Program Committee, especially Karen Vignare, the track chairs, and the reviewers of presentation proposals who assisted in selecting the best papers for this edition. Third, the operations staff of Sloan-C under the leadership of Kathleen Ives, who work tirelessly to make every Sloan-C event something special.

Lastly, we dedicate this special edition to Bruce Chaloux, former Executive Director of the Sloan Consortium, who passed away on September 28, 2013.

The editors of this special edition of JALN hope our readers enjoy this issue and welcome any comments. Laurie Dringus [email protected] Heather Fidalgo [email protected] Anthony G. Picciano [email protected]; Karen Vignare [email protected]

mailto:[email protected]




Three Institutions, Three Approaches, One Goal: Addressing Quality Assurance in Online Learning

Journal of Asynchronous Learning Networks, Volume 17: Issue 4 11

THREE INSTITUTIONS, THREE APPROACHES, ONE GOAL: ADDRESSING QUALITY ASSURANCE IN ONLINE LEARNING Marwin Britto University of Saskatchewan—University Library Cristi Ford University of the District of Columbia—Research Academy for Integrated Learning Jean-Marc Wise Florida State University—Office of Distance Learning ABSTRACT The rapid growth of online academic programs in higher education has prompted institutions to develop processes and implement strategies to ensure the quality of their online offerings. Although there is no one-size-fits-all approach, there are quality standards that institutions can effectively implement regardless of context. This paper examines the approaches of three different types of institutions in addressing quality assurance in online education on their respective campuses. Specifically, this paper presents three case studies and describes each institution’s (1) background and overview, (2) definition of quality, (3) general approach to quality assurance, (4) models and strategies, (5) goals, (6) successes, (7) challenges, and (8) lessons learned. A comparison reveals that despite differences in scope, size, location, mission, and extent of online development, there is consistency in the institutions’ strategies to addressing quality assurance in online learning. KEYWORDS Quality assurance, online learning

I. INTRODUCTION Despite the continuous growth of online learning programs in all academic sectors, there is still some uncertainty about the quality of online educational programs at higher education institutions [1, 2]. With the proliferation of online learning initiatives in higher education, such as massive open online courses (MOOCs), it is critical that the issue of quality be effectively addressed [3]. Institutional oversight of quality in online learning programs is a key factor to programmatic success [4]. Over the past decade, a number of benchmarks and rubrics have emerged through organizations such as the Sloan Consortium (see www.sloanconsortium.org) and Quality Matters (see www.qualitymatters.org), which offer guidelines and processes for assessing and delivering quality in online learning programs [5]. However, further research is needed to identify and document the evidence of the process management of online quality assurance at the institutional level. Accordingly, this article examines the quality assurance processes and metrics at three diverse higher education institutions that vary in size, student demographics, location, mission, and degree of development of online learning programs. As practitioners continue to seek insight regarding the process of quality assurance of online learning programs at multiple levels, this article describes the context, challenges, success, and metrics that define quality.

A. Context Quality assurance evaluations of online programs can be achieved using several methods. However, determining the quality of the actual process can be achieved in one of two ways (or both): benchmarking



and specification of standards [2]. Benchmarking compares an institution’s process and performance in an operational environment against a similar setting. Benchmarking takes into account the use of best practices across multiple institutions. Assessing quality by a specification of standards provides a baseline that can be used as a referenced set of criteria during the evaluation of performance. In online education today, the Sloan Consortium and Quality Matters are two organizations that provide specifications of standards that are often referenced as the primary guides for evaluating quality assurance. Sloan Consortium’s central mission is to help organizations improve the quality of online education, but more specifically they provide the framework that enables institutions to focus on the “five pillars of quality” in higher education that can support quality online learning environments, namely (1) learning effectiveness, (2) scale, (3) access, (4) faculty satisfaction, and (5) student satisfaction [6]. In addition, Sloan offers oversight of quality through the Sloan Quality Scorecard, which allows administrators to engage in an evaluation of their online programs using seventy quality indicators. Quality Matters [7] is a peer-review approach to continuous improvement in online education and student learning that allows programs to review with thoughtful intent the course design process using a set of eight general standards developed from research on student engagement and systematic instructional design. The standards include (1) course overview and introduction, (2) learning objectives, (3) assessment and measurement, (4) instructional materials, (5) learner interaction and engagement, (6) course technology, (7) learner support, and (8) accessibility. Online course quality has traditionally been reviewed for relevance to the learner by assessing course design, learning resources, and the course delivery process [2]. Moreover, institutions have been exploring metrics to compare the academic rigor and the integrity of online courses to their face-to-face counterparts [3]. The quality evaluation process takes into account the process at different levels ranging in scope from macro, such as the institutional oversight, down the micro level of student experience, such as offering support in orienting the student to the online learning environment through the types of engagement being offered in the course delivery [6]. This article highlights three case studies that document and compare the processes and quality assurance metrics at three higher education institutions, and describes the similarities and differences in their strategies, challenges, and opportunities for success.

II. CASE STUDIES A. Lone Star College System 1. Background and Overview The Lone Star College System (LSCS) is a multicampus community college system located in the north Houston area. LSCS serves students across 1,400 square miles spanning two counties in Texas. The system is comprised of six colleges, five campus centers, and two university centers and supports eleven K–12 school districts. LSCS is the largest community college system in Houston, and in 2012 was named the fastest-growing community college in the United States [8], with the greatest growth experienced in online student enrollment. Of the 90,000 students enrolled at Lone Star during the spring semester of 2013, more than 32,700 students were enrolled in 1,748 online courses taught by almost 900 online faculty. As of 2013, 35 certificates or degree programs are offered completely online. Lone Star College-Online (LSC-Online) is a system-wide service and support department responsible for providing ongoing support and resources to all online students and faculty. Although the department does not have academic authority or academic purview, it does serve several important functions. These include (1) offering guidance and training in designing and delivering highly effective and engaging online courses; (2) collaborating with and engaging academic administrators and faculty through the Online Learning Council (consisting of Vice Presidents of Instruction and Faculty Senate Presidents from each of the six college campuses) in matters including quality assurance in online learning; and (3) making recommendations for the development and modification of academic policies and procedures designed to enhance the teaching and learning experience for online students and faculty alike.



2. Defining Quality Most stakeholders in higher education would agree that precisely defining the term “quality” and reaching any sort of consensus on its description are quite challenging. Consequently, many institutions generally use best practices and accepted benchmarks to approximate a standard of quality. Lone Star is no exception. The responsibility for and oversight of defining academic quality at Lone Star rest on the shoulders of the system-wide and campus-based academic administration who have been provided the authority, knowledge, and resources to design and shape these quality efforts on their respective campuses. Accordingly, each campus implements a variety of quality assurance measures for online courses, such as online course reviews and academic approvals, and the use of Quality Matters for course design.

3. Quality Assurance Student success and completion are the highest priority at Lone Star College, and quality assurance is an integral component of achieving these goals. The Office of Research and Institutional Effectiveness has launched a system-wide initiative of Continuous Quality Improvement to assist campuses and departments in designing and implementing plans to optimize academic quality in academic programs and support departments. In 2012, LSC-Online designed a Continuous Quality Improvement model, which the Executive Director of LSC-Online entitled Quality by Design, or QbD.

4. Models and Approaches LSC-Online’s QbD model, as shown below in the Lone Star design in Figure 1, provides a framework for continuous quality improvement. The structure of this design, based on Sloan’s five pillars of quality in online education [9], signifies a level of interdependence between each of the five components—or star arms—that are integral parts of a greater whole. In the QbD framework, each of the five components is organized around four categories: (1) a goal, (2) a process or practice, (3) a metric, and (4) progress indices. The model is designed to evolve over time as data are collected to establish what works well and what does not.

Figure 1. LSC-Online’s Quality by Design—A Continuous Quality Improvement Model

The model’s name, Quality by Design, reflects that the alignment of these five components to quality is intentional, deliberate, and planned. While some institutions may choose to commit resources to just one or a few of these components, LSC-Online’s philosophy is that all components are essential for a successful online learning experience. Consequently, LSC-Online’s commitment is to achieve a high level of quality in all five areas. In order to do this, a systematic and systemic approach is imperative. The model is systematic in the sense that each of these components will address quality standards and metrics,



and it is systemic in LSC-Online’s understanding that each of these components is interrelated, interdependent, and necessarily interactive with one another, and LSC-Online must plan its component strategies accordingly. The first step in creating Lone Star’s QbD model was to identify the quality standards for each of the five pillars or star arms in Figure 1, and then to identify or create the metrics through which they would be assessed. In some areas represented by these five components, quality standards and metrics already exist that may be appropriate to LSC-Online’s context. Quality Matters, for example, used by several hundred higher education institutions in the United States, is a set of general and specific standards used to evaluate the design of online and blended courses. LSC-Online’s instructional designers already use these standards as a guide for their instructional design process when partnering with faculty members to develop or improve online courses. The Quality Matters program falls under the “effectiveness” component of QbD, and is one of several quality standards within this component.

5. Goals In 2013, quality standards and benchmarks, along with their associated metrics, were identified, with each component aligned to one of Lone Star’s Institutional Strategic Goals. Although great progress has been made in implementing this model, more work is needed to fully realize its value. The data points for each component highlight LSC-Online’s strengths and weaknesses and help define a roadmap for improvement and innovation. The QbD model is predicated on the judicious use of these data for continuous quality improvement. Currently, LSC-Online’s data largely exist in silos. The LSC-Online department has data from student support requests, faculty requests, student advising, etc. But it is not known how these interact and influence one another, nor is the data consistently used for improvement of support services. For example, does more and better quality training for online faculty, and/or do more extensive student support services online translate to better student performance, greater student satisfaction, and increased student completion, success, and persistence? The implementation of LSC-Online’s Quality by Design framework seeks to answer these questions through the collection and attention to specific and aggregate data, and application of appropriate statistical analyses. The analyses of these data, known as analytics, will allow LSC-Online to identify areas for improvement, and adjust and re-allocate resources accordingly. A new Director of Client Relations and Quality Assurance position was created and filled in fall 2012. This individual coordinates, facilitates, and leads these efforts and will be instrumental in the further implementation and ongoing activities required by the Quality by Design model.

6. Early Success Existing national benchmarks were useful for comparative purposes. For example, at Lone Star College, faculty who teach online must go through a self-paced Faculty Orientation: Online Teaching Certification program designed to prepare them to teach online. Anecdotal data suggest that LSC-Online’s program takes faculty an average of two hours to complete. Nationally, in 2010 higher education institutions reported that they require their faculty, on average, to complete 22 hours of training (with a range of 17–60 hours) before they can teach online courses [10]. Although LSC-Online has not determined how successful its faculty are in teaching online classes after completing Lone Star’s required Faculty Orientation: Online Teaching Certification program, the significantly higher national average in faculty online training hours suggested that LSC-Online needed to take a closer look at the quantity and quality of its training and revise and improve its faculty certification program accordingly. Accordingly, in 2012 LSC-Online provided summer funding for 12 experienced online faculty to redesign the program based on an extensive review of the literature and soliciting best and exemplary practices in online teaching and learning from peers around the world. By spring of 2013, a newly designed model emerged, which used a cohort-model and mentor-facilitated 24-hour Online Teaching Certification and Faculty Mentoring Program. This program quickly gained support from faculty and campus academic administrators. Subsequently, it was approved by Lone Star’s Executive Council for piloting in fall 2013 and full adoption in January 2014.



7. Challenges As with any large, multicampus institution, getting faculty buy-in and support and building consensus with the appropriate stakeholders is a challenging and slow process. LSC-Online took almost eight months to garner the necessary support and approval to move forward the QbD initiative. Consequently, it became clear that it is crucial to build sufficient time into the overall process.

8. Lessons Learned For the successful adoption of any initiative that may have a direct or indirect impact on faculty, it is imperative to include them as partners and advocates in the process from conception to implementation. It is equally important to identify and develop strategic relationships with those faculty and administrators who are opinion leaders and/or have influence and funding access. Even concepts that may be considered effective, efficient, and productive in higher education require the right mix of exposure, support, and approval to be successfully implemented, adopted, and sustained.

B. University of the District of Columbia 1. Background and Overview The University of the District of Columbia (UDC) is an urban land-grant institution, a historically black college and university, and the only public institution of higher education in the District of Columbia in the United States. UDC offers associate, baccalaureate, and graduate degrees through its six colleges and schools. The total student enrollment at UDC is approaching 6,000 with nearly 80 percent from underrepresented minorities. Historically, the university has offered a small set (20–40) of online courses in programs across the campus. UDC began using the Blackboard Learning Management System in 2004 due to faculty interest. While this was an early path for creating a strategic opportunity for faculty to utilize a learning management system, there was no concerted effort to formalize an online learning initiative at the university. In 2006, the Learning Resources Division and the Office of the Provost and Vice President of Academic Affairs commissioned an advanced Blackboard Users Workshop called Blackboard Scholars, through which 12 UDC faculty members were selected to create 10 high demand courses. By 2007, a pilot program entitled Faculty Course in Online Training (FCOT) was created. FCOT was a comprehensive in-house online training and certification program that was conducted from January to December 2007. The FCOT program trained and certified 17 UDC faculty members from multiple disciplines (e.g., Business, History, English, Education, Sociology, and Criminology). By 2009, the Center for Academic Technology was formed to offer strategic and centralized guidance to UDC faculty members with regard to the online opportunities at the university. During this time, there was a formalized focus on policy and process flow in a concerted effort to improve the quality and oversight of the development in new online courses. As the demand for high quality online teaching and learning continues to grow at UDC, there is an increased need for a more engaged, experiential and “deep learning” approach to training our faculty [11]. In response to these requests and in an effort to assist faculty in establishing teaching methods unique to the needs of our learning community, the Research Academy for Integrated Learning (RAIL) was formed. Specifically, RAIL has been commissioned with the oversight and guidance of online education, but also with effectiveness in teaching and learning with respect to our face-to-face classrooms. In 2012, a city council mandated a right-sizing effort requiring the institution to submit a plan to the DC Council that preserved the mission of the institution but also renewed the vision, enrollment, and relevance of programming for students.

2. Defining Quality As the goal of the right-sizing plan was being actualized, the 20/20 strategic vision plan focused its emphasis on increasing online course offerings as a key element in providing efficient, cost-effective,



high-quality online programs and courses that would draw new students to the campus. At UDC, online education serves the mission, vision, and values of the university by providing a flexible and convenient pathway to higher education for the people of Washington, DC. RAIL, as a part of the University’s overall strategic plan and mission, is committed to improving the quality of all learning experiences at the institution, with the online learning environment being no exception. 3. Quality Assurance RAIL fills a vital role as it pertains to the quality of online learning courses. While online courses are subject to the same scrutiny that face-to-face courses receive when being evaluated, RAIL offers additional structured guidance and support toward the quality of online, student-focused courses. End-of-term evaluations are administered by RAIL with a specific focus on online education.

4. Models and Approaches The metrics and standards developed to assess quality in all areas of online education are based on trends and national best-practice standards. As for many institutions, at the course level, the benchmark of excellence in course design is the use of the Quality Matters (QM) rubric [12]. The QM standards are used to develop a comprehensive process for the way new online and hybrid courses are developed and offered. All new courses are reviewed by internal peer reviewer teams to determine if the course is ready to be offered. Courses that do not pass the internal peer review process are pulled from the registrar’s list and are not offered as online courses to students until feedback from the reviews is incorporated that reflects student engagement and a student-centered focus as defined by the Quality Matters rubric. To support the infrastructure and quality of the policy process for online education at UDC, the Committee for Online Learning (C4OL) was established. The mission of C4OL is to develop and disseminate guidelines, policies, and procedures that facilitate online teaching and learning at UDC. As a committee, C4OL has established guidelines consistent with the standards set forth by the Middle States Commission on Higher Education and National Best Practices. In addition, the Committee for Online Learning will lead UDC's assessment practices of online course proposals and online courses, as well as guide the University's Institutional Strategic Planning Process for Online Learning. The body consists of faculty members representing academic colleges across the university’s main flagship and community college campus.

5. Goals The importance and value of the goals of quality assurance are in line with Vision 2020: the University of the District of Columbia’s Strategic Planning Process. The Academic Affairs portion of the strategic plan offers several specific goals and strategic outcomes that focus on online education. Senior leadership is committed to increasing UDC’s online footprint, but not at the expense of providing quality programs and courses. Additional strategic goals for quality assurance have been outlined in the RAIL department’s strategic goals and initiatives. As of fall 2013, there is a strategic plan and development for offering fully online certificates, programs, and concentrations throughout the UDC system.

6. Success The increased focus on quality has resulted in better teaching and learning experiences for both online faculty and students. As a result of defining quality metrics that considered the online student perspective, a robust online orientation was created that online students can access through a self-enroll option. Moreover, as a result of creating an environment that focuses on quality, online faculty have an opportunity to pursue three additional certification tracks. These new tracks are designed to enhance and broaden the faculty's online teaching skills while maintaining a consistent level of quality. The five certification tracks enable online faculty to actively participate in the quality process, which in turn contributes to an improved overall online learning teaching and learning experience at the university.

7. Challenges



One of the most pressing challenges faced by RAIL, which serves as the central department for supporting the university in all aspects of online learning, has been the limited human resources to serve both the flagship campus and the branch campus of the community college. The two sites are in multiple locations, and continuing to offer strategic oversight of new online growth has been a challenge. In addition, in past years quality control and buy-in to the formalized quality assurance process has taken some time for all constituents to value and appreciate, and to understand its importance. However, increased buy-in from faculty and deans has just begun to take shape and is now gaining momentum. Reorganizations of the current online learning division have offered opportunities for great success, but sustaining their momentum has also proved challenging. The current iteration of the department, RAIL, offers more strategic connections to university schools and colleges by working closely with academic deans and senior leadership to offer insights concerning continued growth.

8. Lessons Learned For many faculty at UDC, the response to a need for quality oversight of new online programs has been interpreted as a long and difficult process. UDC knows now that continuing to build capacity and engaging faculty in conversations about quality in courses are central to increasing buy-in and adoption. With a greater emphasis from UDC’s accrediting body on online education, the dialogue has shifted dramatically to allow faculty to understand and see the value of adhering to the stated polices or to offer alternative solutions that will meet the same objectives.

C. Florida State University 1. Background and Overview Florida State University (FSU) is a state-funded research institution in Tallahassee, Florida, with more than 40,000 undergraduate and graduate students enrolled annually. Online teaching and learning at FSU started well over a decade ago and has been growing at a steadily increasing rate. Today, FSU offers more than 400 online courses and 30 online programs, with annual enrollments topping 30,000. One in every 10 students is taking at least one online course per semester, and one in every 20 students is enrolled in a fully online program. Students have a wide range of online degree and certificate programs to choose from, including business, communication, computer science, criminology, education, nursing, public safety and security, and social work, and new courses and programs are added every year. As a tier-one research institution, FSU holds all programs and courses to very high standards, regardless of mode of delivery. Academic programs and individual courses pass through rigorous reviews at various administrative levels including program, department, college, and university, before they are approved for delivery. Faculty members who are assigned to teach online courses are held to the same standards as those teaching in the traditional classroom but receive special training and support in order to maximize their effectiveness online. Students are admitted based on the same criteria whether they enroll in online or campus-based programs, and they are all held to the same academic standards.

2. Defining Quality The university distinguishes various aspects of quality, including course design, delivery, mentoring, administration, support services, and technologies. Together, these dimensions affect the overall quality of an online student’s learning experience. While quality can be assessed using objective criteria and applicable standards, each student’s experience is subjective and unique, and it may be biased because of key events or components of the course. For example, some students may perceive the quality of a particular course to be very high because it uses lots of interesting videos, while other students might rate that same course as low in quality because their Internet connection was not fast enough to show the videos in high definition. While a well-designed course, in theory, stands a greater chance of succeeding, it is the students’ impression of the quality that ultimately drives the success of a course, rather than the score it receives on an objective, criterion-based course review. Quality, therefore, is a variable that must be assessed from different perspectives, and its definition consists of multiple dimensions. Consequently,



the university does not use a single definition of the term but rather defines criteria and sets standards for critical factors that affect quality overall.

3. Quality Assurance The university delegates much of the responsibility for quality assurance of teaching and learning to the academic departments, which in turn adhere to field-specific accreditation requirements and applicable professional standards. Program directors and department chairs typically coordinate the goals and objectives of programs, and individual faculty members take responsibility for the quality of their own courses. Within this spirit of academic autonomy, faculty and university administration agree that high quality of courses and instruction is paramount, and the university has adopted policies and procedures that allow for specific checks and balances, including the use of university-wide, standardized course evaluations, quality enhancement reviews, and standardized forms and procedures for the approval of course syllabi and program specifications, to name but a few. The Office of Distance Learning (ODL) fills an important role in the university-wide effort to ensure the quality of online courses and programs. Guided by a distance learning policy approved by the faculty senate along with a strategic plan, ODL uses a variety of methods and instruments to manage the development, implementation, delivery, support, and evaluation of all distance learning at FSU. However, given the relative autonomy of the faculty, activities of the office are typically based on voluntary participation encouraged by funding for course development, awards for excellence, and individual support.

4. Models and Approaches The criteria established for the assessment of quality in each dimension are based on relevant theories and best practices. For example, course design is evaluated according to the principles of systematic instructional design. These principles form the foundation of the QM rubric [12], which is used in two ways to assess the quality of online courses. A first review focusing on the most critical standards of the rubric is conducted at the end of the course development phase. Formative feedback is provided in terms of comments from three reviewers in addition to the ratings of each standard. After a course has been taught at least once, a full review is conducted using the complete QM rubric. Suggestions for improvement of the course design are shared with the instructor and the course is logged as having been approved unless substantial changes are required.



Figure 2. Quality Assurance Model, Office of Distance Learning, FSU

Quality assurance is guided by a comprehensive model (see Figure 2) that aligns instructional design and delivery activities with the strategic plan and includes specific assessments and evaluations for each step in the process. Course delivery is assessed by means of two instruments: (1) official, campus-wide course evaluations completed by students, and (2) additional end-of-term surveys focused on online courses and completed by faculty, course mentors, and students. The instruments consist of standard quantitative items as well as open-ended comments, which are analyzed for content and summarized in reports that are used to improve online course design and delivery. Feedback from these surveys is also used as input for improving instructor and mentor training and related information provided online. Additionally, the end-of-term surveys include items assessing the quality of administrative, support, and technology services and products, and feedback is used to document effectiveness and make improvements as needed. Furthermore, new technologies in support of online course development and delivery are researched and evaluated on an ongoing basis.

5. Goals The goals for quality assurance of online teaching and learning at FSU are in line with the university’s overall strategic goals, which hold the quality of research and education at their core and underscore the importance of dedicating appropriate resources to ensure high-quality programs and courses [13]. The Office of Distance Learning Strategic Plan lists the determination of quality standards and best practices in instructional design and development, technology, student support, and assessment and evaluation as its first strategic goal [14]. Additional strategic goals further specify promoting the consistent use of standards and best practices in online teaching and learning as well as the use of innovative distance learning strategies and technologies. This ongoing effort ensures that online courses and programs at FSU are the result of research based instructional and pedagogical design and that they use innovative technology to offer online students an effective and stimulating learning environment.

6. Successes The efforts of ODL have resulted in a series of online resources that systematically guide novice online instructors, course developers, course mentors, and students. Materials include guidelines, instructions, tutorials, and online courses that supplement individual consultations and support. Online instructors can



download worksheets and templates that facilitate course design and development and ensure high quality results that are consistent with university policies and include up-to-date information. FSU online courses are generally very well received, as evidenced by outstanding ratings and feedback in course evaluations and end-of-term surveys, as well as growing enrollments. Furthermore, instructors and mentors express their satisfaction with the training and support they receive and appreciate the resources that are available to them to ensure that the quality of their courses is consistently high. Every year, departments compete for ODL funding and support to help bring more courses and programs online. Similarly, the processes and policies developed to guide the administration of online courses and programs have proved effective. Streamlining the budgeting, finances, course listings, and registration within a single office for all academic departments that offer online courses and programs is essential to ensure consistency and avoid costly mistakes that could affect accountability, audits, and accreditation.

7. Challenges Perhaps the greatest challenge in the case of FSU is to strike an adequate balance between preserving departmental and individual autonomy while at the same time ensuring that all online courses and programs offered at the university are of consistently high quality. The structure and governance of the university imposes limits on a centralized approach to quality assurance. Since the academic departments and colleges—and indeed the individual members of the faculty—enjoy a certain degree of autonomy concerning the choice of course content and delivery, the establishment and campus-wide adoption of shared standards is a sensitive topic and difficult to implement. Furthermore, the development of online courses is affected by this challenge in that ODL has limited authority to respond to the potential lack of cooperation and performance of subject matter experts and academic faculty assigned to development projects. Compensation for full-time academic faculty is governed by a collective bargaining agreement that does not allow compensation to be directly tied to their performance on a single project.

8. Lessons Learned In response to the challenges described above, ODL is focusing on increasing individual support, customizing training, and highlighting and showcasing the benefits of following standards. In an analogy to Hersey and Blanchard’s situational leadership model [15], academic faculty approach online course development from varying levels of readiness, and a personalized approach can be developed to provide the appropriate level of support and guidance during the development project. Professors experience the benefit of following the quality standards and guidelines by seeing their courses receive high scores on quality design reviews and receiving positive feedback and ratings in end-of-term evaluations. Furthermore, excellence is encouraged and celebrated by a series of awards that provide opportunities for additional funding, recognition, and monetary rewards. Finally, the key to success is systematic design and project management. By setting precise goals, objectives, and standards along with specific deadlines for deliverables, projects are completed in a timely manner with high-quality products.

III. CONCLUSION Quality assurance in online learning continues to be of interest to the research community. The focus of recent research on quality ranges from the use of individual technologies or instructional approaches to research on courses, programs, institutions, districts and regions, or entire countries [16, 17, 18, 19, 20]. A comparison of the three cases presented in this study showed that although the three institutions differ in size, focus, and culture, their approaches to assuring quality in online education are, in fact, surprisingly similar. First, the goals for quality assurance are aligned with the institutions’ strategic plans, an approach that is well supported in the literature [18, 21]. This critical aspect ensures that the efforts of the faculty and staff enjoy the support of the institution’s leadership, sending a message that the quality of all educational programs is equally important regardless of delivery method. In addition, it demonstrates that the institution is prepared to invest adequate resources to ensure that online education meets appropriate standards. Second, all three institutions have developed initiatives to control the quality of online courses and programs across the system, and base their approaches on established models, including Quality



Matters [12] and Sloan’s five pillars [9]. Their approaches are also similar in terms of offering support programs that lead to training and certification of online instructors, which is not only a best practice for promoting high-quality instruction in innovative contexts, but is becoming a shared standard of excellence in the context of institutional accreditation. The importance of the training and professional development of online instructors is also emphasized in the recent literature [22, 23]. Third, the three institutions are similar in that they feature centralized support units whose mission includes the development of appropriate quality standards as well as system-wide rollout, training, and evaluation to ensure that the standards are met. Centralizing the support function for distance learning is not uncommon [24]. Fourth, the three case studies also reveal that despite their differences, the institutions are facing similar challenges in their effort to promote high quality online education. Faculty buy-in was listed by all as a key challenge, which reflects a commonly observed tension in academia between the faculty and the administration [25]. Academic freedom is highly valued, especially in times when budgets are cut, affecting the quality of programs and courses and the ability of faculty to pursue their academic mission on independent paths. Centrally imposed systems that require faculty to reach consensus and conform to shared standards are regarded with suspicion and rarely welcomed. Competition is fierce in academia and members of the faculty are faced with competing priorities and a lack of adequate rewards for collaboration, leading them to develop an increased focus on individual achievements. In order for quality enhancement projects to succeed, faculty members have to be involved in the process and given an opportunity to help shape it. This conclusion is shared as an important lesson learned in all three cases in this study and is confirmed in other studies [26]. Furthermore, it is important to acknowledge that individual faculty members have different learning styles, needs, knowledge, abilities, and attitudes with respect to developing online courses and teaching online. In order to be effective, administrative units must provide a spectrum of support and professional development opportunities for individual faculty members. Quality assurance at institutions of higher education is often a double-edged sword. On the one hand, it is evident that consistently high quality education requires shared standards, appropriate training, and adequate resources. In addition, accountability is critical across the institution, especially at institutions with relatively large numbers of faculty who have different levels of comfort and experience, and who may have competing priorities that prevent them from investing the time to learn and adopt the new technologies and pedagogies needed to succeed in online teaching. On the other hand, a diverse faculty forms the basis for a dynamic institution of higher education, where new knowledge is created and students are exposed to a variety of perspectives. Imposing artificial structures on faculty may have a stifling effect if they feel restricted or unnecessarily criticized regarding their ability to teach effectively. Effective management of these conflicting dynamics must focus on the benefits for faculty and their students rather than suggest that the individual’s freedom must be sacrificed for the greater good. In order to achieve this result, faculty must experience direct, meaningful benefits from participating and be given an opportunity to embrace the suggested improvements as their own. Involvement in the design and implementation of shared standards is a prerequisite for this. Accordingly, support units should design high quality learning environments for online faculty and students, which foster collaboration, active learning and student engagement, and ultimately offer a common vision of excellence and opportunities for shared success.

IV. ABOUT THE AUTHORS Marwin Britto served as the Executive Director of Lone Star College-Online at the Lone Star College System until September 2013. He currently serves as the Associate Dean of the University Library at the University of Saskatchewan in Saskatoon, Canada. In addition, he has served in a number of administrative positions in higher education, and has had 10 years of university experience as a full-time faculty member. He has published and presented widely in academic journals and conferences in the areas of distance education/online learning, teaching and learning, teacher education, instructional technology,



library science and change management. He holds four graduate degrees, including a PhD in Instructional Technology from the University of Georgia. Cristi D. Ford is Director of the Research Academy for Integrated Learning (RAIL) at the University of the District of Columbia. This unit oversees all online learning programs for the university, as well as broader teaching and learning initiatives. She has expertise and experience in building new online learning programs and has an interest in research related to teaching and learning in the online environment. She has held several leadership positions in the online education realm and has served as a full-time and adjunct faculty member. She holds a PhD in Educational Leadership from the University of Missouri-Columbia. Jean-Marc Wise is a faculty member at the Florida State University Office of Distance Learning and specializes in strategic planning, institutional effectiveness, online course design, assessment, and evaluation. He coordinates quality planning, design, and evaluation of distance learning courses and programs in support of academic departments. He has extensive experience in a wide range of educational technology and information systems and has been teaching in academic, nonprofit, and private sector contexts over the past two decades. He holds a PhD in Instructional Systems Design from Florida State University and is actively engaged in international collaboration and research projects.

V. REFERENCES 1. Lee, J. and Dziuban, C. Using Quality Assurance Strategies for Online Programs. Educational

Technology Review, 10(2): 69–78 (2002). 2. Oliver, R. Quality Assurance and E-learning: Blue Skies and Pragmatism. Alt-J, 13(3): 173–187

(2005). 3. Chua, A. and Lam, W. Quality Assurance in Online Education: The Universitas 21 Global

Approach. British Journal of Educational Technology, 38(1): 133–152 (2007). 4. Rovai, A.P. A Practical Framework for Evaluating Online Distance Education Programs. The

Internet and Higher Education, 6(2): 109–124 (2003). 5. Mcloughlin, C. and Luca, J. Quality in Online Delivery: What Does It Mean for Assessment in E-

learning Environments? In G. Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds.), Meeting at the Crossroads. Proceedings of Australasian Society for Computers in Learning in Tertiary Education (ASCILITE): 417–426. Melbourne, Australia: Biomedical Multimedia Unit, The University of Melbourne (2001).

6. Moore, J.C. A Synthesis of Sloan-C Effective Practices. Journal of Asynchronous Learning Networks, 16(1): 91–113 (2011).

7. Quality Matters. About Quality Matters. (2013). https://www.qualitymatters.org/about 8. Pulsinelli, O. Lone Star College Breaks Record Enrollment. Houston Business Journal, September

18, 2012. http://www.bizjournals.com/houston/morning_call/2012/09/lone-star-college-system-breaks.html

9. Lorenzo, G. and Moore J.C. Report to the Nation: Five Pillars of Quality Online Education. The Sloan Consortium (2002). http://sloanconsortium.org

10. WCET, and Campus Computing Project. Managing Online Education (2010). http://wcet.wiche.edu/advance/managing-online-education-survey

11. Bain, K. What The Best College Teachers Do. Harvard University Press: Cambridge, MA, 2004. 12. Quality Matters. Quality Matters Rubric (2013). https://www.qualitymatters.org/rubric 13. Florida State University. Florida State University Strategic Plan (2013).

http://president.fsu.edu/content/download/83755/886586/ 14. Office of Distance Learning. Office of Distance Learning Strategic Plan (2013).

http://distance.fsu.edu/docs/admin_docs/ODLStrategicPlan.pdf 15. Hersey, P. and Blanchard, K.H. Leadership Style: Attitudes and Behaviors. Training and

Development Journal, 36(5): 50–52 (1982).

https://www.qualitymatters.org/about

http://www.bizjournals.com/houston/morning_call/2012/09/lone-star-college-system-breaks.html

http://www.bizjournals.com/houston/morning_call/2012/09/lone-star-college-system-breaks.html

http://sloanconsortium.org/

http://wcet.wiche.edu/advance/managing-online-education-survey

https://www.qualitymatters.org/rubric

http://president.fsu.edu/content/download/83755/886586/

http://distance.fsu.edu/docs/admin_docs/ODLStrategicPlan.pdf



16. Nandi, D., Hamilton, M., and Harland, J. Evaluating the Quality of Interaction in Asynchronous Discussion Forums in Fully Online Courses. Distance Education, 33(1): 5–30 (2012).

17. Mashaw, B. A Model for Measuring Effectiveness of an Online Course. Decision Sciences Journal of Innovative Education, 10(2): 189–221 (2012).

18. Miltenoff, P., Keengwe, J., and Schnellert, G. Technological Strategic Planning and Globalization in Higher Education. International Journal of Information and Communication Technology Education, 7(3): 51–61 (2011).

19. Cooper, S.B. A Closer Look at Distance Learning in the Kansas City, Missouri School District. Distance Learning, 9(2): 26–34 (2012)

20. Looi, C.-K., So, H.-J., Toh, Y., and Chen, W. The Singapore Experience: Synergy of National Policy, Classroom Practice and Design Research. International Journal of Computer-Supported Collaborative Learning, 6(1): 9–37 (2011).

21. Orr, R., Williams, M.R., and Pennington, K. Institutional Efforts to Support Faculty in Online Teaching. Innovative Higher Education, 34(4): 257–268 (2009).

22. Baran, E., Correia, A.-P., and Thompson, A. Transforming Online Teaching Practice: Critical Analysis of the Literature on the Roles and Competencies of Online Teachers. Distance Education, 32(3): 421–439 (2011).

23. Keengwe, J. and Georgina, D. The Digital Course Training Workshop for Online Learning and Teaching. Education and Information Technologies, 17(4): 365–379 (2011).

24. Kruger-Ross, M.J. and Waters, R.D. Predicting Online Learning Success: Applying the Situational Theory of Publics to the Virtual Classroom. Computers & Education, 61: 176–184 (2013).

25. Baran, E., Correia, A.-P. and Thompson, A. Transforming Online Teaching Practice: Critical Analysis of the Literature on the Roles and Competencies of Online Teachers. Distance Education, 32(3): 421–439 (2011).

26. Herman, J.H. Faculty Incentives for Online Course Design, Delivery, and Professional Development. Innovative Higher Education, 38(5): 397–410 (2012).

Building a Foundation for Success through Student Services for Online Learners


BUILDING A FOUNDATION FOR SUCCESS THROUGH STUDENT SERVICES FOR ONLINE LEARNERS

Dr. Ruth Newberry Duquesne University Catherine DeLuca Duquesne University ABSTRACT Retaining students in distance education programs requires institutions to field a strategically aligned, cross-unit team able to provide streamlined and effective services to online learners throughout the student’s academic life (from applicant to alumni). If any team member drops the ball during interaction with the online student, the student becomes frustrated or distracted, and student retention is jeopardized. In today’s highly competitive marketplace, students have a “universe of opportunities” for obtaining an education and advancing their careers. Attracting and retaining students is crucial to traditional institutions like Duquesne University in order to remain competitive in the online education marketplace. The university's recent commitment to formalizing graduate courses online has brought strategic planning, oversight, resources, and support to students. The Online Campus director initiated an examination of the services necessary for online students to be successful learners. As Duquesne found out, retaining online students goes beyond offering resident-equivalent services to online learners; it requires a holistic and networked approach that leverages existing technologies and an institutional commitment to its online learners, or what the authors have designed as the COMFORT model. KEYWORDS Online student, student services, support services, student-centric, academic services, holistic, team-based, online support models

I. INTRODUCTION In Allen and Seaman’s (2013) report on distance education in the United States, they estimated that 32% of all students enrolled in post-secondary education were taking at least one online course. While the growth in online enrollment has slowed from previous years, it has continued to grow at a higher rate than the general student rate in higher education [1]. According to Online College Students 2013: Comprehensive Data on Demands and Trends (2013), students who completed online courses or programs “considered it a good investment of their time and money” [2, p. 4]. In response to the overall growth of students in online programs, many traditional institutions are exploring online options as a new venture, and those institutions that have been offering online education are looking to expand their offerings to remain attractive to prospective students. The factors that continue to draw students, particularly graduate students, to online education are “scheduling flexibility” and “the freedom to manage other responsibilities” [2, p. 5]. In today’s competitive and economically charged marketplace, colleges and universities cannot afford to be reactive to the needs of online students; rather, they need to


26 Journal of Asynchronous Learning Networks, Volume 17: Issue 4

proactively develop a strategic vision and plan to ensure online students have the best experience possible [3]. An institution’s reputation, along with price, is critical in the decision process for prospective online students. Our proposed approach to online student services seeks to address the quality of interactions between the institution and its students. In Reworking the Student Departure Puzzle (2000), the authors suggested that an institution's engagement with its students is a significant factor in shaping online learners' experiences and outcomes [4]. Boston, Ice, and Gibson (2011) proposed that faculty, administrative staff and offices, and other students affect the kind of engagement students have with the institution, and the quality of these engagements is important to online students and their success [5]. Consequently, those institutions, such as for-profit institutions, that focus on the needs of online students continue to benefit from increased market share in online education [1, 3]. With institutions cutting budgets and reducing staff and faculty during tight economic conditions, and the high cost of an education compounded by downward enrollment trends, institutions of all kinds must compete for a smaller pool of students [6]. Located in the urban setting of Pittsburgh, Pennsylvania, Duquesne University is a private liberal arts and doctoral research institution that entered the online education market in 1997. Like other traditional institutions, Duquesne has faced similar concerns with enrollment trends, budget tightening, and reduction in employees from retirement options. With 10,045 full-time equivalent students (5,917 undergraduates, 4,128 graduates), Duquesne offers its students educational opportunities within 10 schools of study. Founded in 1878 by Catholic missionaries from the Spiritan Order, Duquesne University is a mission-centered and mission-driven institution seeking to continue its strong heritage of social justice, commitment and service to the poor, ecumenicalism, and academic excellence through the experiences of its students, faculty, staff, and alumni. While a recent survey of undergraduate alumni captured the positive interactions students had with Duquesne [7], the responses of current online students reflected a more disjointed connection to the institution. The approach the authors describe in this article attempts to address this sense of disjointedness online students feel with Duquesne University as the institution becomes more strategic in how it competes in online education. According to LaPadula (2003) and Crawley (2012), a positive student experience depends primarily on the quality of the interactions students have with an institution [8, 9]. On-campus students interact with various departments, faculty, services, and peers across campus during their time at the university with the option to communicate in person with key institutional players as the students navigate through the system toward graduation. Online students, who must also interact with many of these same departments, faculty, and services, do not have the opportunity for in-person encounters. Instead, these students must rely on the contact methods the institution provides them, which all too often remain tied to campus operating hours and in-person procedures and any available web services. In her study on retention in higher education online programs, Heyman (2010) found that student satisfaction with the institution's support services was among the top contributing factors to retention. Students who feel supported by their institution are more likely to remain enrolled and eventually graduate, resulting in a positive or perceived positive experience with the institution [10]. Through improvements to the quality of university services, online students can be retained and, more importantly, form a lasting relationship with the institution. As online students achieve their goal of graduation and become alumni, the institution’s alumni pool grows. When these alumni perceive a positive relationship with the institution, the institution’s reputation and brand recognition is furthered through the positive support and outreach these alumni conduct in their communities. This brand recognition is, as Aslanian and Clinefelter found, one of the top three factors for students selecting an online education, along with accreditation and prospective students’ acquaintance with other attendees [2, p. 5]. With a universe of options for online degrees available to prospective students, institutional brand recognition and reputation become critical considerations. An effective student support services model for online students must not only work to retain students as Heyman (2010) notes, but it must also be strategic in nature, collaborative in effort, and originate from an analysis of what the institution is doing well and where gaps exist in its services to online students [10].



Smith (2005) suggests three key objectives to building effective and efficient online student support services: “identify the needs of its online and face-to-face learners; make services available when the learner wants them, rather than when the school is comfortable providing them; and provide online services that are as good or better than face-to-face equivalents” [11, p. 29]. Duquesne University is developing an effective student services plan grounded in Smith’s (2005) key objectives for effective student online support services and the findings of LaPadula (2003) and Heyman (2010) regarding the relationships institutions construct with students through interactions [11, 8, 10]. Our approach to this challenge is what we are calling the COMFORT model, a holistic strategy of providing services from which both the institution and online students benefit. The COMFORT model includes the following strategies outline in figure 1:

Figure 1. COMFORT Model

C = a well-developed communication plan for students, faculty, and support staff O = orientations (plural) for students and faculty M = the creation and maintenance of student-centered policies for success F = comprehensive feedback to improve existing support and academic services and to create new

ones O = various outreach initiatives to students R = easily accessible, learner-focused academic and support resources T = a team-based approach and commitment to online students’ success during their academic

lifespan The COMFORT model is a compilation of best practices and evidence based research that we as practitioners believe will improve the experience of Duquesne University’s online students. Neither the authors nor the institution suggest compromising academic standards to fit the needs of online students. Instead, we propose that an effective student services model is a partnership between the institution and the online learner in the student’s journey from applicant to alumnus that is grounded in a clear understanding of each other’s expectations and responsibilities in this journey.

II. BACKGROUND Duquesne University is not new to online education. Of the 10,045 students at Duquesne University, 724 are completing their degrees in online programs. These students, primarily graduate and working adult students, are enrolled in online programs offered through the following schools: Nursing, Leadership, Music, and Education. Duquesne’s School of Nursing was one of the nation’s first to offer an online doctorate in Nursing in 1997. Today, the School of Nursing includes online master's programs, a post-master's certificate, a doctorate of philosophy in Nursing, a doctorate of nursing practice, and starting in fall 2014, a RN to BSN bridge program. The School of Leadership and Professional Advancement began its online master’s program in Leadership in 2000, having already offered courses in its award-winning, accelerated bachelor of professional sciences available in online and face-to-face formats. The Mary Pappert School of Music and the School of Education moved selected programs and courses online starting in the late 1990s and the early 2000s. A significant challenge for both online students and the Online Campus, which formed in November 2012, is the decentralization of services and support for online education at Duquesne University. Paper-based and in-person processes slowly transition to online-friendly processes, and early-adopter faculty individually venture into online education, causing Duquesne University's online student services to be scattered and uneven. While an entrepreneurial approach to online education moved Duquesne University early into the online education space, decisions as to which courses and programs should move online



were not necessarily tied to enrollment trends and market needs. Today, the earlier entrepreneurial spirit must be tempered somewhat. As online education grows in popularity and viability for adult students, particularly as the disruptive presence of massive open online courses raises awareness of online education in the public consciousness, skepticism about the quality of instruction and learning dissipates. Alongside concerns about student debt and the value of higher education, mid-sized, brick and mortar liberal arts and research institutions like Duquesne University must determine the role and degree to which online education will have a presence in their strategic plans, their educational philosophy, and their identity. If they choose, as Duquesne has, to compete within the online education marketplace, then they must find ways to strategically allocate resources to build a sustainable infrastructure to support and retain students and faculty and to protect, enhance, and grow their brand within the constraints of market competition, institutional economic realities, and institutional identity and branding. Even as Duquesne’s decentralized structure enabled individual faculty and schools to develop school-specific online courses and programs, these paths have not been sustainable within ever-tightening budgets. And as our initial assessment of Duquesne’s online learners revealed, they are experiencing student services unfriendly to online students and inconsistently applied across the institution. For instance, technical assistance remains tied to campus class schedule hours and the eastern time zone, while other forms of assistance are available only during university business hours by phone or e-mail. In 2013, the process to change a student’s grade still required signatures on paper forms. However, as the institution addresses the requests of on-campus students for 24/7 access to class registration, course descriptions, drop/add procedures, course evaluations, and extended assistance for library research, more services are gradually becoming online-friendly for all Duquesne students. With the November 2012 formation of the Online Campus, Duquesne University is signaling to the campus community that the university is strategically formalizing its efforts in online education by providing leadership, guidance, structure, processes, and advocacy for the institution’s online students. Additionally, the recent restructuring of the Enrollment Management Group further aligns resources toward applying market trends to student recruitment and emphasizes student retention at the undergraduate and graduate levels. This restructuring has sparked many conversations across campus regarding the role of online education at Duquesne. Two themes are predominant in these campus conversations: faculty concern regarding professional development and assistance for teaching online and the preparedness of the campus to provide effective support services to online students. As the university tackles these broad challenges simultaneously, this article focuses specifically on improving student services for online learners. The assumptions underwriting this initiative to improve student services are

x the importance of Duquesne’s mission and Spiritan tradition of “serving God by serving students” to enhance the experience of all Duquesne students [12];

x the student-centric approach Duquesne takes toward the educational experience of its students and faculty, acknowledging this approach as a critical differentiator and value-add element for online education at Duquesne;

x the institution’s desire to grow graduate enrollments across the 10 disciplines through online opportunities;

x the necessity of bringing consistency to communications and processes across the schools for graduate services since Duquesne lacks a graduate school to centralize these efforts;

x the lessons learned from the Schools of Nursing, Leadership, Music, and Education’s delivery of online programs and the small, dedicated workgroup they provided; and

x the importance of applying Duquesne’s student-centric approach to course design and engaging faculty in student retention efforts.

In other words, to highlight the value added by an online Duquesne educational experience as expressed by faculty and students with both online and campus experiences, student support services needed to promote in the online realm the same sense of “connectedness” to faculty and services (or the representatives of the institution) as on-campus students said they experienced. Guiding our approach to



this challenge of providing quality interactions between Duquesne and its online learners through necessary, easy to use, and timely support services that can be holistically and programmatically applied across all 10 disciplines in a cost-efficient manner, we adopted Smith’s (2005) key objectives, which suggested this approach can lead to a wholly satisfactory online experience for online learners [11].

III. EVALUATION To improve student services for online learners, we approached the problem from a practitioner perspective, which led us to examine current processes and services delivered to current online students. The evaluation, initiated by the Online Campus director, involved identifying the student support structure within the schools offering online programs and the university at large: their policies, procedures, and services for online students. Additionally, the authors undertook a review of best practices and strategies for supporting online students; school committees designed and deployed surveys to current online students and faculty teaching online. The Online Campus director gathered key stakeholders from across campus to investigate specific areas of concern and to identify and recommend areas for improvements. The analysis of the support services structure for the schools offering online programs made obvious the strengths, weaknesses, and inconsistencies across the schools offering online education. For instance, the schools might have some but not all of the components Pipitone and Poirer (2011) describe as part of the student “life-cycle” or “the continuum of service from the first point of contact throughout the student’s College experience” [13, p. 1]. While schools relied heavily on institutional functional areas, such as financial aid, health services, and career services, to help with servicing online students, the schools rarely had internal staff members for the high-touch points of the student’s academic life, such as recruiters, admission counselors, advisors for enrolled student, and alumni officers to engage alumni. Lacking a graduate school to help with centralizing these services, our online student support model not only needed to design services around the academic life of the Duquesne online student, but also had to account for this lack of consistent structure within the schools offering online programs.

A. Policies and Procedures Review Duquesne University currently offers 14 online degrees through four schools. From the review of policies, student handbooks, and procedures for online learners, the only thing that was consistent across the online programs was inconsistency. Although we had been providing online education for almost 20 years, school websites and course sites were difficult to navigate, and many common student tasks were not yet online student friendly. For example, academic integrity was presented differently among the four schools, with most relying on the instructor’s course syllabus to state the school’s policy and procedures. To begin correcting these issues, the School of Nursing is piloting an academic integrity module that requires online students to view a video explaining what academic integrity means to them as students and nursing professionals, particularly the expectations and responsibilities they have as learners, and to submit their acknowledgement in each course that they will comply with the policy before they can gain access to the content of the course. Additionally, the university’s Academic Integrity Committee is nearing its work on standardizing school procedures with university policy.

B. Task Forces During the spring 2013 semester, the Online Campus director initiated three ad hoc committees to assess the current state of online education and services at Duquesne University: the Online Orientation Committee, the Online Student Services Committee, and the Academic-Student Affairs Committee. The goals for these three committees aligned with the aims of the Online Campus School Committees that the director began in January 2013. These school committees were established in partnership with each school’s dean and were intended to prompt explorations regarding the role of online education in the school and to identify faculty attitudes, program possibilities, faculty and student support needs, and to define required policies and desired procedures that would benefit online students and faculty teaching



online. The three committees further broadened the conversations among key stakeholders regarding an appropriate and sustainable university infrastructure for online education, while keeping each committee of manageable size and thus schedule meetings. The Online Campus director was the constant throughout, providing information and possibilities for collaborating with other schools.

1. Online Orientation Committee The Online Orientation Committee was charged with examining the current welcome information sent to students and orientation practices in order to develop a university-wide online orientation to bring consistency and accuracy to the information students received. The committee is currently designing a streamlined, technology-enhanced orientation process for new campus and online graduate students. Participants in this part of the study included graduate student representatives and representatives from International Programs, Graduate Admissions, and graduate advisors.

2. Online Student Services Committee The Online Student Services Committee was tasked with identifying and reviewing the existing online student services for adequacy and ease of use in addition to assessing options for holistic improvements aligned with the university’s motto of “education for the mind, heart, and spirit.” Participants in this part of the study were graduate advisors in online programs and representatives from Financial Aid, Retention and Advising, Graduate Admissions, International Programs, and Veterans' Education.

3. Academic–Student Affairs Committee For this committee, the Online Campus director worked with those units providing upcoming online education initiatives so they could (1) plan resources and staffing to service online students and (2) examine how their current processes might need adjustment to service online students if the units had not already done so. This committee was comprised of representatives from Academic Affairs, such as the University Writing Center, Gumberg Library, Retention Office, and Learning Skills Center, as well as units from Student Affairs, such as Career Services, Student Services (special needs), and Programs and Activities (Student Affairs). Additionally, a representative from Central IT’s Help Desk also attended.

C. Survey Outside of surveys from Alumni Affairs to recent university graduates and semester course evaluations, no attempts had been made to gather information specifically from online students regarding their satisfaction with student services and their experience as online students. Neither had there been a satisfaction assessment given to faculty to learn about their experience as online instructors. To better understand current student satisfaction with support services, a survey was developed by three of the school committees with current online programs (Nursing, Leadership, and Education) to collect this baseline information. The online student survey was created to gather input from enrolled students in online programs regarding their use, perceptions of, and satisfaction with those services provided by the university and those services specific to the school. This survey consisted of four sections:

• Part 1: respondent demographics and background information • Part 2: student use and satisfaction with services offered by the school • Part 3: student use and satisfaction with services offered by the university • Part 4: suggestions for additional services and improvements to current services

IV. RESULTS A. COMFORT Model Building upon the findings of Heyman (2010) and Smith (2005) and the spirit of Duquesne’s mission statement, our desire was to design a student-centric support service in which the institution and the online student are invested partners in the success of the online student learner [10, 11]. The COMFORT



model, as we call it, results from this philosophy and is based on trust, responsibility, accountability, and shared goals. The goal is to achieve what LaPadula calls “quality of interaction” [8]. The COMFORT model does not imply the institution compromises in any way its educational standards for the student’s happiness or comfort; instead, the relationship is mutually beneficial without either party having to lessen expectations of the other. The model ensures online students will receive prompt, timely, and efficient support services so they can focus on their courses and learning instead of taking precious time from their studies to find services or resolve issues. Through implementation of effective student support services, online students continue to make progress on their degree, are retained, and reach graduation to become alumni. They become “satisfied” students in the terminology of the Sloan-C Five Pillars for Online Program Success [14]. As we describe each part of the COMFORT model within our institutional context with key implementation points for other institutions, refer to figure 1.

1. Communication Plan in the COMFORT Model A well-developed communication plan for students, faculty, and support staff is a critical component of the model because our survey respondents indicated communication of services was often disjointed, confusing, and contradictory. An effective communication plan sets clear expectations for all involved. For example, institutions are responsible for timely delivery of clear, concise, and relevant information necessary to the online student’s degree progress, and the students is responsible for reading and acting upon the provided information in a timely manner. Developing a partnership with online students also requires improving relationships with faculty and staff across the university who are involved in supporting online students. In her analysis of post-secondary institutions and student service directors, Dunn (2005) suggests that commonsense practices are often challenged by decentralized institutional structures that on-campus students can navigate more effectively than online students [15]. To construct an effective communication plan at Duquesne, we had to bring together the key constituents necessary to provide student services to online students, which we did through the formation of the task-force committees described earlier. More importantly, we needed to provide a safe and open collaborative space early on for these stakeholders to discuss and learn from each other about communicating necessary information to students. These stakeholders also needed to discuss how each school was delivering information in order to identify and recommend improvements to reduce duplication of materials, streamline messages, and select appropriate information delivery methods. According to Dunn (2005), communication should not be limited to e-mail, phone, or in-person meetings; an institution should employ any medium available, such as announcements through Blackboard or websites, text messages, social media (Facebook, Twitter), chat (Skype, Collaborate, Oovoo), and even video [15]. Ideally, online students should be able to select their preferred message delivery system. At Duquesne, however, we do not yet have a communication system that enables such an option for students. Therefore, we agreed our first priority was to ensure messages were concise, relevant, and timely and delivered in ways that best meet the needs of online students. The committee defined the following as the keys for a successful communication plan:

x Understand the roles of key support players within the institution and the high-touch points for online students during their academic lifespan.

x Construct a communication plan that key players have ownership of that can be consistently implemented and maintained.

x Understand the needs of the target audience to create clear and concise messages to them. x Adopt various tools to push communications to online student.

2. Orientations (plural) in the C.O.M.F.O.R.T Model Most of the literature on student services recommends having an orientation to acclimate students to the institution and to their program. Each school at Duquesne offering online education employs some form of orientation for online students. For instance, the School of Nursing requires a one-week, on-campus orientation, while the School of Leadership provides an optional virtual orientation. Whether students attended an on-site or virtual orientation, our survey respondents spoke positively about the overall



orientation experience. However, they were dissatisfied with the amount of information given and the late release of some information. Our model adopts the term "orientations" to recommend the need for a series of targeted orientations for both online students and the faculty/staff who work with them.

a. Student Orientations To reduce attrition, both Crawley (2012) and Harrell (2008) suggested a comprehensive orientation to help online students learn about the institution, be introduced to their program and peers, and to prepare them for course work [9, 16]. Although Harrell’s research is based on community colleges, his findings relate to online students as well [16]. While orientation to Blackboard and other technologies are important, they should neither be the focus nor omitted from orientation [9]. The task-force committees believed that in addition to the technology, online students needed to learn about the structure of the institution and its philosophy; become acquainted with the many student and academic support services available and how to engage with them as online students; the operational structure of the school, its policies, and educational philosophy; the curricular structure of the program and their educational path; and the institutions' expectations and the students' responsibilities. Because Nursing’s one-week, on-campus orientation adopts this comprehensive model, Nursing’s online students responded most favorably for awareness and use of school support services. As the number of students entering online programs grows, so does the need to have a frank discussion about the commitment an online program requires of the student to be successful—conversations that need to happen as early as the recruitment and carry through the program. While online programs and degrees have been thought of as less challenging than traditional face-to-face learning, Allen and Seaman (2013) reported that 77% of their respondents saw online education as equal to or more demanding than face-to-face learning [1, 2]. Our respondents reported similarly, with 76.3% of the students indicting student learning in online courses was more academically challenging when compared to traditional face-to-face courses. Allen and Seaman (2013) also reported that online students find online courses as rigorous and demanding as traditional classroom courses [1, 2]. Our respondents ranked self-discipline, self-motivation, organization, and time availability as the top requirements for success. To build a successful partnership between online students and the institution, prospective online students should understand up front the expectations and demands of the program so they can plan accordingly so as to not be distracted, overwhelmed, and derailed in the middle of coursework when issues or events occur outside the classroom. By providing students with a comprehensive orientation (either on-campus or virtually), we provide online students with the tools to be successful in their educational experience with us.

b. Faculty/Staff Orientations Effective student services is an institutional effort requiring the commitment of all key players within the institution to ensure that online students transition successfully through the stages of their academic lives so that both the students and the institution benefit [13, 17, 18]. Because the characteristics and needs of the online student continue to evolve, it is essential to periodically refresh faculty and staff understanding of who the typical online student is, the needs of the student, the difference from the traditional student, and the student's expectations of the educational experience. By being better informed about the adult online student, faculty and staff can better understand their roles in retention and support services [17]. According to Clinefelter (2012), faculty play a key role in the success of online learners, particularly through the quality, not frequency, of their interactions with students in courses [19]. Not only is an institution’s faculty one of the top factors during the selection process for prospective students [2], but as Crawley and Fetzner have noted, “Faculty members are typically the first to know when students are in trouble, and they often are called on to connect these students with the necessary support services” [17, p. 7]. Thus, faculty involvement and understanding of their roles in student retention is important to the success of online students, including support service referral. Finally, just as faculty who teach online receive training and development, so should advisors and recruiters who will work with online students.



The recruiter is the initial face of the institution, while the advisor is a continual presence throughout the student’s academic progress.

From our initial development of online student orientations, we have identified the following keys for successful orientations:

x Plan and deliver engaging, student-centric, and learning focused orientations. x Prioritize sequence of orientations and embed them appropriately into student, faculty, and staff

sessions. x Adopt an appropriate delivery model for the orientation that allows participants to obtain the

information they need in the most convenient and flexible manner while creating a feeling connectedness and support by the institution.

x Leverage the “success network” concept existing within the Starfish Retention Systems software Duquesne has adopted to enhance the values of “connectedness” and “partnership” that align with Duquesne’s mission and the COMFORT model during the orientation process [20].

x Engage in continuous/periodic evaluation of the orientations for improvement to address the needs of online students, faculty who teach online, and the institutional and school personnel who provide support services to online students.

3. Maintenance in the COMFORT Model While many models and best practices exist for designing effective online education programs in the institution, such as WCET’s Best Practices for Electronically Offered Degree and Certificate Programs (2001)[18], the Sloan Consortium’s Sloan-C’s Quality Framework: The 5 Pillars (2002)[14], and Middle States’ Distance Education Programs: Interregional Guidelines for the Evaluation of Distance Education (Online Learning) (rev. 2011) [21], Duquesne’s entrepreneurial, early-adopter, and school-specific entrance into online education led schools to focus primarily on curriculum development, technology, content preparation for online delivery, and pedagogical strategies for online teaching with student support services evolving as needs arose. As Hardy and Meyer-Griffith (2012) suggested, it is easy for programs to forget that policies, processes, and procedures also have to be adjusted to align with the needs of online students [22]. Our survey respondents noted that too often, course and institutional tasks seemed tied to faculty preferences, institutional business hours, and staff work schedules than to their needs as working professionals. Mariasingam and Hanna (2006) remind us of the following:

We would be foolish to ignore the fact that ultimately, because of the very nature of e-learning, learners (or customers) are able to choose from a multitude of educational offerings regardless of what we or anyone else believes about the quality or legitimacy of such products. We would also be naïve not to recognize that education has become a commodity. It can be bought, sold, and transferred just like any other commodity. And educational providers, whether not-for-profit, public, private, or proprietary, are all ultimately bottom-line driven. If an institution's products do not have market value the institution will not have a market. [23]

Going forward, the task force committees recognized the level of commitment necessary and the importance of this commitment to create, maintain, and sustain support services for online students, and the quality of our interactions with them to retain and graduate them. Some of the keys for successful maintenance of policies, processes, and procedures are as follows:

x Periodically review policies, processes, and procedures to ensure feasibility, clarity, and relevance for online students.

x Centralize and streamline processes for efficiency and effectiveness for both online students and institutional and school stakeholders.

x Leverage existing technology—such as Starfish Retention Solutions, Blackboard, and the Campus portal—to create a one-stop shop or shared resources area to ease navigation issues, reduce time, and eliminate confusion for online students.



x Create and maintain a feeling of connectedness for online students with the institution even during off-hours, weekends, and holidays so they can complete their work, plan ahead, and problem solve.

4. Feedback in the COMFORT Model Remaining attentive to the needs of online student learners is critical to their success, the program’s success, and the institution’s success in offering online education [13, 21, 22]. Instituting an assessment cycle of continuous review and improvement is equally important to the success of any program and service [8, 13, 22]. Such assessment helps an institution to determine how well it meets the needs of students and where to make improvements. A comprehensive evaluation plan would incorporate the perspectives of the key stakeholders—students, faculty, and support service staff—and employ a variety of methods, including formal evaluations, such as focus groups or surveys and informal evaluations, such as debriefing meetings with faculty and staff after an event or a discussion board for new ideas [13, 15]. Additionally, feedback would be collected at various points during the student’s academic life regarding satisfaction and use of support services. The surveys used for our initial assessment are a small piece of a much larger upcoming assessment plan. The current surveys are a starting point for benchmarking this model. They capture current online students’ perception and satisfaction with the student services now offered by the university and the schools. As we continue to implement the COMFORT model across current and new online programs, we will focus on what the collected data says about how well the model retains students, helps them continue their progress toward their degree, and as alumni, the level of satisfaction they have for their overall experience as online students at Duquesne University. Early planning for future assessment includes these key factors for successful feedback:

x Create shared governance among stakeholders that includes student and faculty representation to identify the goals, design the assessments, and prioritize the improvements.

x Use various approaches and technologies (Starfish, social media, Blackboard, ticketing systems, etc.) to collect analytics and data on services and interactions.

x Collect and analyze the feedback throughout the year, not just annually. x Manage changes by implementing them in small but significant ways so as not to be

overwhelmed by proposed changes. x Report assessment findings to current online students and faculty as well as service units and

administrators. 5. Outreach in the COMFORT Model Online students are susceptible to feeling isolated because they do not have the same campus experience as traditional students [16, 2]. Because this sense of isolation is common for online students, it is important for the institution not only to build an online community in which students have the opportunity to interact and socialize with other online students [9], but it is also important for the institution to build relationships and rapport with online students through student services. Oftentimes, as student services practitioners, we are not aware that a student is struggling until the individual hits a breaking point and seeks out assistance. While the faculty may be the first to sense an online student’s difficulty in the context of course performance, the faculty may not see it as his or her responsibility to reach out to the student in the same way a student services practitioner might [16, 17]. By developing a relationship with other online students, these students might be more likely to communicate concerns they have, such as feeling overwhelmed, before these concerns affect their performance in the program [17]. Being proactive with online students demonstrates to them that they are connected to the institution [9]. This outreach begins early in the recruitment phase and should carry through the student’s academic life via interactions with support services and an effective communication plan. Each student service representative and faculty is accountable for helping online students be successful. Thus, a collaborative effort is critical to developing a well-planned, well-implemented outreach plan and preventing online students from



becoming overwhelmed and off track. Research, in combination with our own experiences, led to these keys for successful outreach initiatives:

x The outreach plan should result from a collaborative effort among all institutional stakeholders to limit duplicated communications.

x Outreach should involve an early warning system (such as Starfish Retention Solutions) to identify, connect, and track the outcomes of interventions with at-risk students.

x Outreach should identify potential options for students to consider and make decisions. 6. Resources in the COMFORT Model As LaPadula (2003) and Crawley (2013) explained, student-centric support services are critical to student success [8, 9]. To construct an effective student service model, Smith (2005) reminds us to keep our target audience in mind so that support services are “available when the learner wants them, rather than when the school is comfortable providing them” [11, p. 29]. Institutions should develop a support environment in which students can use 24/7 self-service options to find answers to their questions; complete common tasks, such as registering for a class, adding or dropping a class, or checking on their progress with financial aid; and access links to handbooks, procedures and forms, academic calendars, and FAQs. Even as 86% of students surveyed felt that the online student services offered by the university contributed to their success in their program, and 84.7% of students felt that the online student services contributed to their success in their programs, students gave low satisfaction ratings to other services, such as technology assistance, writing, tutoring and mentoring, and peer and career networking. In the School of Nursing, online students ranked online registration; the Graduate Advisor’s Blackboard site containing information on programs, registration, and general program outcomes; and the school’s online student portal where polices, handbooks, forms and clinical information can be found as the top three student services offered by the School of Nursing. However, students noted they had to go to three sites to find information; they wanted one place. Institutional stakeholders must determine what services are critical to online learner and student success before investing in infrastructure, technology, and training. And the institution needs to commit to these support services so resources can be allocated, both financial and human, to building and sustaining the services; otherwise, the institution jeopardizes the partnership they have with online students. The task force committees identified these key areas for establishing successful learner-focused academic and support resources:

x Commit both financial and human resources from the institution to create, maintain, and grow online support services to students.

x Create student services for online students, not just equivalents services, to foster support and a forge a connection between the school and the online learner.

x Create holistic online support services that focus on the student academically, socially, and individually.

7. Team-Based Approach in the COMFORT Model Retaining students in online education programs requires a cross-unit team that serves as the foundation for the institution’s overall focus on online students [15]. Although the team-based approach is the last piece of the COMFORT model, identifying the cross-unit team is the necessary first step to making the other pieces work. The cross-unit team is tasked with implementing the online education vision of the institution and aligning student success strategies with the mission and organizational culture of the institution to build, provide, and sustain streamlined and effective services. The cross-unit team is involved in each of the above parts of the COMFORT model, from the identification and execution of strategies and services to implementing improvements and sharing information. The team includes representatives from each of the high-touch points online students encounter from recruitment to applicant, student, and finally to alumni. The purpose of the cross-unit team is to gather key stakeholders from around the university to create champions within the represented units, reduce potential barriers by addressing them at the planning stage, and limit potential political battles over initiatives by having the



cross-unit team set the agenda rather than any one unit, department, or service [24]. Commitment to the retention and success of the online student should be a shared goal to ensure the best possible outcome for both students and the institution. The task force committees suggested these key elements for building a team-based approach for student success:

x Align online student support strategies with institutional mission and culture. x Select key stakeholders invested in holistic and on-going support of student learners. x View the role of student services as a retention tool for online students. x Identify best practices from internal institution and external models. x Implement a well-developed communication plan. x Involve students and faculty as well as support services staff.

V. IMPLEMENTATION AND NEXT STEPS As explained previously, Duquesne University is in the process of implementing the COMFORT model to address student service needs for online students with the aim of improving retention, enhancing brand reputation, and ensuring success for all students. While the first stages of the COMFORT model are underway, several steps remain both in the short term and long term. Short term goals include the following:

x Enhance current collaboration among recruitment, advisement, faculty, support services, and alumni to ensure online students are being supported holistically across the academic life.

x Implement a unified welcome communication to be sent to all new graduate students upon acceptance to streamline and better coordinate the substance and flow of information new graduate students receive.

x Develop an academic preparedness online orientation for new online students that includes technology preparation, introduction to the library, information on writing resources, overview of institutional structure, and introductions to the support services available.

x Design and construct a New Graduate Student portal able to address rolling admissions for new online graduate students to help them prepare for the start of their programs and complete a checklist of necessary tasks before the start of classes.

x Recommend that orientations for online students be mandatory whether the delivery mode is online or in-person.

x Further the development and use of available technologies, such as Starfish Retention Solutions, across all online programs.

Although the short-term goals continue the work that has begun, the longer-term goals require additional planning, resources and commitment from the university before they are able to be realized. These goals include the following:

x Continual support and commitment is necessary from the deans, academic and business support entities, and the campus community at large in order to construct a sustainable and flexible student support infrastructure for launching new online programs and growing current programs.

x Create professional advisor positions not currently available in schools offering online programs. We attribute part of the School of Nursing and the School of Leadership’s success over the years to the role of the professional advisor. Professional advisors often serve as the lifeline between the student and the institution.

x Incorporate additional university services to support online students, such as campus ministry, career services, student organizations, and counseling. While Duquesne has made significant progress over the years in creating online student services, the incorporation of these services would allow the university to further build the critical connectedness between the student and the institution.



x Better leverage existing technology across all student services. It is critical that Duquesne use the technology and tools available to better service students and also meet university needs.

x Fully implement the COMFORT model across all online graduate programs. As the university focuses efforts on expanding online programs at the graduate level, the COMFORT model ensures streamlined processes, consistent and accurate information, and available effective and efficient online services and resources are in place to support new online programs and make growth sustainable.

VI. CONCLUSION The authors recognize that the COMFORT model may not fit the structure and culture of other institutions or online programs; however, it can at minimum be a starting point for creating a model that will. Successful student support services reflect an institution’s ability to identify the very basics of what online students require to move successfully from applicant to alumni. Our COMFORT model is a partnership between the institution and the online student. This partnership lives across the student’s academic lifespan; both the student and the institution benefit from the investment in the relationship. Based on trust, responsibility, accountability, and shared goals, this mutually beneficial relationship requires the institution to establish first a commitment to building the necessary team to identify the services, goals, and philosophy to underwrite the online student services. Without such a commitment, the institution jeopardizes the partnership with online students and their success. Next in the process is improving relationships and communication with colleagues across the university involved in supporting online students. From the team’s commitment and shared goals (matched by the institution in resources), the design and implementation of the model can begin, followed by continuous improvements. For online students to commit to the partnership, they must accept and understand up front the expectations and demands of the program, engaging with the provided resources and services when life "happens." By providing consistent, easy to use services and resources targeted toward their individual situations, the institution demonstrates its commitment to the online student’s success. The COMFORT model is ambitious and will involve commitment from the institution, allocation of resources, both financial and human, and may require some culture and organizational change to successfully create quality interactions with students. In this case, effective and efficient online student services improve retention and build lasting relationships between the student and institution. Long gone are the days when offering a course or program online gives an institution a competitive advantage in the higher education marketplace. Choice abounds for prospective students and to attract new online students, current online students must have an overall positive experience greatly influenced by the student services they receive while enrolled. We have to do more than be academically rigorous. Our online students must be successful learners, stay focused on their learning, and move toward degree completion. Their success is our success as an institution in the online education market.

VII. ACKNOWLEDGEMENTS The authors thank the Duquesne University community, including the administration, deans, the Online Orientation Committee, the Online Student Services Committee, the Academic-Student Affairs Committee, and academic and business support entities for their work in the investigation, evaluation, and beginning steps of the implementation of our COMFORT model. We also thank the online students for their participation in the surveys that have allowed us to take these first steps.

VIII. ABOUT THE AUTHORS Dr. Ruth Newberry is the current director of Duquesne University’s Online Campus, where she provides leadership, advocacy, and support to the university’s online initiative by working closely with Academic Affairs, deans, faculty, advisors, enrollment management, and many others across campus. Previously, she was the director of educational technology at Duquesne from 2003 to 2012 and was responsible along



with her staff for supporting classroom technology, video services, and the Blackboard Learn and Communities systems. During this time, she instituted a sustainability plan for classroom technology, moved the campus from 20% to 90% technology-enhanced classrooms, grew Blackboard adoption from 10% to 85% for course use, and designed and implemented a required training program for faculty, graduate students, and staff using Blackboard. Drawing upon her teacher preparation background and 18 years of college teaching experience in the traditional and online environments, Ruth has been involved in faculty development, student support, and the effective application of technology to achieve teaching aims and student learning outcomes. She has co-chaired the university’s Educational Technology Committee, served on the university’s Student Retention and Academic Integrity Committees, and is president of Pittsburgh’s Blackboard Users Group (PghBug). In 2005 and 2007, she was conference chair for regional conferences on online learning. Ruth has presented at Sloan, Educause, Mid-Atlantic Educause, Syllabus, Blackboard World, College Composition & Communication Conference, National Peer Tutoring Conference, and others and is a peer reviewer for Educause Online Review. Her doctorate is in Twentieth-Century American Literature. Catherine DeLuca joined the Duquesne University School of Nursing in 2004 and has served in various capacities since then, including her current position as the graduate academic advisor. She not only serves as the graduate academic advisor to students, but also as grant manager for some of the school’s federal grants in addition to her work at the university level with the Online Campus, Enrollment Management, and Financial Aid departments. She has also been successful in the development and implementation of several new initiatives to enhance the student experience for the online graduate learner. She is currently completing her doctorate in Higher Education Management at the University of Pittsburgh.

IX. REFERENCES 1. Allen, I., and Seaman, J. Changing Course: Ten Years of Tracking Online Education in the United

States. Wellesley, MA: Babson Survey Research Group, 2013. 2. Aslanian, C.B., and Clinefelter, D.L. Online College Students 2013: Comprehensive Data on

Demands and Preferences. Louisville, KY: The Learning House, Inc., 2013. 3. Wherry, M., and Windes, D.L. Square Pegs, Round Holes: Distance Students and Campus Priorities

in the Academic Community. Continuing Higher Education Review, 74: 110-117 (2010). 4. Braxton, J.M. (Ed.). Reworking the Student Departure Puzzle. Nashville: Vanderbilt University

Press, 2000. 5. Boston, W.E., Ice, P., and Gibson, A.M. Comprehensive Assessment of Student Retention in Online

Learning Environments. Online Journal of Distance Learning Administration, 4(1) (Spring 2011). http://www.westga.edu/~distance/ojdla/spring141/boston_ice_gibson141.pdf.

6. Slavin, R. Moody’s Report Says Grad Enrollment Decline Hits Higher Education Credit. Diverse Issues in Higher Education (October 8, 2012). www.diverseeducation.com/article/48572#.

7. Duquesne University. Alumni survey results, internal document (2013). 8. LaPadula, M. A Comprehensive Look at Online Student Support Services for Distance Learners.

American Journal of Distance Education, 17(2): 119-128 (2003). 9. Crawley, A. Supporting Online Student: A Guide to Planning, Implementing and Evaluating

Services. San Francisco: Jossey-Bass, 2012. 10. Heyman, E. Overcoming Student Retention Issues in Higher Education Online Programs. Online

Journal of Distance Learning Administration, 13(4) (2010). 11. Smith, B. Online Student Support Services. Community College Journal, 76(2): 26-29 (2005). 12. Duquesne University. Duquesne University’s Strategic Plan for 2010-

2015. http://www.duq.edu/about/administration/strategic-plan. 13. Pipitone, B., and Poirer, W. Student Services Revitalization/Transformation. College Quarterly,

14(1) (2011). http://www.collegequarterly.ca/2011-vol14-num01-winter/pipitone-poirer.html. 14. The Sloan Consortium. Sloan-C Quality Framework: The Five Pillars,

2002. http://sloanconsortium.org/5pillars.

http://www.westga.edu/~distance/ojdla/spring141/boston_ice_gibson141.pdf

http://www.diverseeducation.com/article/48572

http://www.duq.edu/about/administration/strategic-plan

http://www.collegequarterly.ca/2011-vol14-num01-winter/pipitone-poirer.html

http://sloanconsortium.org/5pillars



15. Dunn, S. M. A Place of Transition: Directors' Experiences of Providing Counseling and Advising to Distance Students. Journal of Distance Education, 20(2): 40-57 (2005).

16. Harrell, I.L. Increasing the Success of Online Students. Inquiry, 13(1): 36-44 (2008). http://www.vccaedu.org/inquiry/inquiry-spring-2008/1-13-Harrell.html.

17. Crawley, A., and Fetzner, M. Providing Service Innovations to Students Inside and Outside the Online Classroom: Focusing on Student Success. Journal of Asynchronous Learning Networks, 17(1): 7-12 (January 2013).

18. WCET Best Practices to Promote Western Cooperative for Educational Telecommunications (WCET). Best Practices for Electronically Offered Degree and Certificate Programs, 2002. http://wcet.wiche.edu/wcet/docs/cigs/studentauthentication/Accrediting_BestPractices.pdf.

19. Clinefelter, D. Best Practices in Online Faculty Development. Learning House White paper, 2012. http://www.learninghouse.com/best-practices-in-online-faculty-development.

20. Starfish Retention Solutions. Starfish Retention Solutions Philosophy, 2013. http://www.starfishsolutions.com/sf/philosophy.php.

21. Middle States Commission on Higher Education. Distance Education Programs: Interregional Guidelines for the Evaluation of Distance Education (Online Learning), 2011, revised. http://www.msche.org/publications/Guidelines-for-the-Evaluation-of-Distance-Education-Programs.pdf.

22. Hardy, K., and Meyer-Griffith, K. Meeting Accreditation Requirements: Are You Serving Distance Learners? Journal of Asynchronous Learning Networks, 16(5): 7-18 (2012).

23. Mariasingam, M.A., and Hanna, D.E. Benchmarking Quality in Online Degree Programs: Status and Prospects. Online Journal of Distance Learning Administration, 9(3) (Fall 2006).

24. Shea, P. Serving Students Online: Enhancing Their Learning Experience. New Directions for Student Services, 2005(112): 15-24 (2005).

http://www.vccaedu.org/inquiry/inquiry-spring-2008/1-13-Harrell.html

http://wcet.wiche.edu/wcet/docs/cigs/studentauthentication/Accrediting_BestPractices.pdf

http://www.learninghouse.com/best-practices-in-online-faculty-development

http://www.starfishsolutions.com/sf/philosophy.php

http://www.msche.org/publications/Guidelines-for-the-Evaluation-of-Distance-Education-Programs.pdf

http://www.msche.org/publications/Guidelines-for-the-Evaluation-of-Distance-Education-Programs.pdf

Are We Who We Think We Are? ePortfolios as a Tool for Curriculum Redesign


ARE WE WHO WE THINK WE ARE? EPORTFOLIOS AS A TOOL FOR CURRICULUM REDESIGN Gail Matthews-DeNatale Northeastern University

ABSTRACT This paper focuses on the use of ePortfolios to inform the redesign and development of academic degree programs. The practice aligns with the Connect to Learning (C2L) design principles of inquiry, reflection, and integration (IRI), which are nested within C2L’s “Catalyst for Learning” model for successful ePortfolio implementation [1]. In 2011 Northeastern instituted an ePortfolio requirement for students enrolled in the university's Master of Education program. Systematic review of student ePortfolios as a collection informed planning for a 2012 master’s program redesign that began implementation in fall 2013. This redesign includes the articulation of a program vision, program- and concentration-level outcomes, course map with signature assignments designed to evidence outcomes, ePortfolio curricular integration, and shared foundational courses across four concentrations. The program redesign is intended to improve the integration of theory with professional practice, enhance the connection between academic and workplace-based learning, increase value for students in the portfolio development process, create greater outcomes transparency for faculty and students, and support ongoing improvement of the program. KEYWORDS Program design, curriculum design, learning design, ePortfolios, formative assessment, signature assignments, learning outcomes

I. BACKGROUND: THE EMERGING EPORTFOLIO LANDSCAPE The term “ePortfolio” can be difficult to define because its meaning takes many forms. From a logistical standpoint, ePortfolios are online tools designed to support students in gathering and documenting work samples (also known as artifacts or evidence). At first glance, an ePortfolio looks like a website or blog. Yet the ePortfolio also references an approach to learning that aligns with constructivist, experiential, and—more recently—connectivist pedagogy [2, 3]. People who are unfamiliar with the literature on ePortfolios typically perceive of them to be only a showcase, perhaps because the term “portfolio” is associated with an artistic collection of exemplary work. ePortfolios, and the portfolio process, can serve a host of goals, and categorization is important to the conceptualization of ePortfolios, but there is no definitive typology [4]. Three prominent “genres” or purposes include the following:

x Documentation or directed portfolios are often organized around professional standards or program outcomes. The templates for these portfolios are often highly structured, with areas in which students upload files to demonstrate achievement in relationship to a standard or criterion. These ePortfolios often grow out of an intentional process of outcomes mapping and assessment, as evidenced by the ePortfolio implementation framework described in a recent publication by Chen, Penny Light, and Ittelson [5].



x Integrated learning or developmental portfolios engage students in iterative cycles of creating work, reflecting upon work, revising work in response to metacognition, and integrating revised work within a larger context of professional and/or learning identity formation [6, 7]. These portfolios often involve a process described as “social pedagogy” that usually includes peer feedback, discussion, and group work. Randy Bass and Heidi Elmendorf observe that social pedagogy challenges students to develop “their knowledge in contexts that centrally ask them to think of their audience as someone other than the professor” [8].

x Showcase portfolios are most often associated with the culmination of an educational experience, for example within a capstone project or course. Career services and marketing units also often express interest in these types of portfolios because they can help pending graduates prepare for interviews and promote program visibility to prospective students [9].

Considering the varied forms that portfolios can take, one might ask if ePortfolios are a technology, a pedagogical method for connected and integrated learning, or a strategy for evidence-based formative and summative assessment of learning outcomes. The answer is yes. Many portfolio programs are designed to advance several of these goals simultaneously. However, Helen Barrett notes that in hybrid-use scenarios there is an inherent tension between the individualization and creative rein that fosters student engagement, and the standardization that facilitates institutional goals for maintaining accreditation and conducting program-level outcomes assessment [10]. Given the complexity of ePortfolios and their manifold uses, there is a pressing need for better research on effective design principles and pedagogical impact [11]. In 2005 Kathleen Blake Yancey, Barbara Cambridge, and Darren Cambridge founded the Inter/National Coalition on Electronic Portfolio Research (NCEPR) consortium. In 2009 Trent Batson and others founded the nonprofit Association for Authentic, Evidence-Based and Experiential Learning (AAEEBL), which hosts a series of annual and regional conferences dedicated to ePortfolio pedagogy and research [12]. And in 2011 Edward Watson of Virginia Tech convened a review board to found the International Journal of ePortfolio. These initiatives represent a community of practice dedicated to discerning ePortfolio learning principles, design practices, and impact. They also indicate that the field has reached a tipping point in the recognition and support for ePortfolios within higher education. Within this context, ePortfolios extend the boundaries of the learning landscape beyond the confines of courses, programs, and schools; they are an essential component for supporting learning in a connected, integrated, and digital world. One project is notably missing in the preceding landscape survey of the ePortfolio field. I set aside discussion of the Connect to Learning Project (C2L), whose framework has most influenced the work of Northeastern’s Master of Education program, in order to describe the initiative in depth. C2L launched in 2011 as a three-year project sponsored by the Making Connections National Resource Center at LaGuardia Community College, in partnership with AAEEBL, and with support from the Fund for Innovation in Post Secondary Education (FIPSE). Project leadership included Bret Eynon and Judit Török of LaGuardia Community College, Laura Gambino of Guttman Community College, and Trent Batson of AAEEBL. Randy Bass of Georgetown University and Helen Chen of Stanford University served as the project’s research scholars. C2L is a network of twenty-four institutions dedicated to shared inquiry and research, with the goal of increased understanding and sophistication in ePortfolio practice. It was thoughtfully designed as a learning community in which members serve as online collaborators in negotiating the challenges, opportunities, and promising practices afforded by ePortfolios. C2L required its institutional affiliates to use portfolio methodology in the documentation, analysis, and interpretation of project work. For the duration of the project, each institution kept an ePortfolio of its work. Every two to three months, the membership engaged in a “Jam,” a two-week collaborative learning experience similar to an asynchronous online course. During the first week of a Jam, consortium members were given a set of background readings and guiding questions related to an ePortfolio topic (e.g., assessment, faculty development, social pedagogy, scaling up, technology). One or more members from each institution



authored the first draft of a “promising practice” (case study) related to the Jam topic and posted it in its ePortfolio. During the second week of the Jam, C2L members toured their colleagues’ portfolios, commented on each other’s work, participated in threaded discussions within Moodle, and then returned to their institutional portfolios to revise their promising practices. Each institution emerged from the project with an ePortfolio representing its learning and accomplishments. The Northeastern case study described in this article originated as a “polished practice” authored during a C2L Jam. In addition to the Jams, institutional members agreed to administer Core Surveys twice a year to students and faculty actively involved with portfolio-based learning, with the aim of gathering quantitative data to discern patterns in effective practice. The Jam write-ups also served as a form of qualitative data, furthering the project’s quest for evidence of impact and learning design principles. Some of these stories told of impressive gains, such as the 19% increase in retention for second-year students at San Francisco University who were actively involved with ePortfolios. Tunxis Community College observed an 18.7% increase in persistence among students who took three active ePortfolio courses as compared to students whose courses did not include an ePortfolio component. Other institutions’ stories described the identification of challenges, such as LaGuardia Community College’s use of ePortfolios to examine thousands of student work samples in relationship to the AAC&U VALUE Rubrics [13, 14]. This practice allowed LaGuardia to zero in on the need for improvement in students’ oral and written communication, which prompted the development of an integrated, cross-disciplinary approach to communication education. These and other case studies will become available on the Catalyst for Learning website, a collection of “field-tested practices and detailed stories that trace the developmental trajectories of successful ePortfolio initiatives.” The website is scheduled for release in January 2014 [1]. It is important to keep in mind that ePortfolios are not a magic bullet; an ePortfolio initiative might not achieve the desired results if a critical component is missing. C2L seeks to understand the ecology in which successful ePortfolio implementations take place. Drawing on the contributions of network membership, the project has articulated a multifaceted description of the coordinated system of support critical to reaping the benefits of ePortfolios on a large scale (see Figure 1). This framework involves professional learning and systems changes in almost every sector of the institution, including faculty professional development, changes in pedagogy, evidence-driven attention to outcomes, strategies for scaling up, as well as a technology infrastructure of policies, tools, and support. Looking at ePortfolios from this ecological vantage point, it is no wonder that Randy Bass and Brent Eynon describe ePortfolio practice as playing a pivotal role in “changing the landscape of higher education” during an emerging “age of learning.” For example, they describe ePortfolios as “living out the tension” between data-driven strategies for verification and accountability, and personalized learning that is greater than the sum of its parts and therefore difficult to measure. God (or the Devil) is in the

Figure 1. The Catalyst for Learning Framework



details, and the details constitute learning design. If student ePortfolios do not provide evidence of what we purport to be achieving, that unquiet truth then tells us we need to go back to the drawing board and redesign our courses, programs, and even our institutions. Within this model, faculty are positioned as the “designers” of learning [15]. What is learning design? At its outermost ring, C2L’s Catalyst for Learning framework describes growth as an iterative process of inquiry, reflection, and integration (IRI) (see Figure 1). In the cycle depicted in the Catalyst framework, inquiry, reflection, and integration are described as “core design principles” [1]. What is a design principle? Educators borrowed this term from the field of visual arts because it encompasses elements of design that are otherwise difficult to describe. A principle is a fundamental idea about good practice. According to the C2L website, a design principle is a concept that informs the organization and structure of a learning process [1]. When applied with finesse, these principles improve the quality and outcome of learning. In the case of IRI, each principle is distinct yet synergistic when combined with the others. They can be described as follows:

• Inquiry: investigating, asking questions, grappling with ambiguity, cultivating habits of mind • Reflection: analyzing experience, identifying patterns, creating meaning, deepening learning • Integration: making connections, linking learning, applying/transferring knowledge across

domains The science and theory of learning is ripe with cycles, such as Kolb’s spiral of experiential learning, and many educators are therefore accustomed to this iterative approach in the design of student learning [16]. What is new is the concept of using ePortfolio IRI for curricular improvement, program growth, and institutional development, elevating authentic, evidence-driven, inquiry-based learning into organizational practice. Student ePortfolios provide a window into student self-perception and metacognition; they are an act of self-representation. The IRI process presents an opportunity to view student ePortfolios in aggregate as information, as a reflection of our impact that can be analyzed and interpreted for the purpose of program improvement. The following case study provides an example of IRI program redesign in action.

II. CASE STUDY OF IRI IN ACTION A. The Context: Background and First Steps Northeastern University is a private research university located in Boston, Massachusetts. With an enrollment of over 26,000 students, the school is best known for its undergraduate co-op program that integrates academic study with workplace-based experiential learning. It is also home to the College of Professional Study (CPS), in which the Graduate School of Education is located, whose primary focus is adult learners who are already in the workplace. In addition, many of the CPS programs are fully online, including the Master of Education program. CPS is committed to offering its virtual graduate students the same quality of workplace-based experiential learning that is enjoyed by undergraduates. A key strategy for attaining this goal within the college is the use of ePortfolios because they provide an online learning space that spans courses, can include authentic evidence of learning in a range of formats, and provide opportunities for perceiving macro-level connections between students’ academic and professional experience. They allow students to “zoom in” and “zoom out,” metaphorically speaking, on their learning experience, placing the specifics of one course within a space that has the capacity to encompass all endeavors within in their program and profession. This is particularly helpful for adult learners juggling the competing demands of work, family, and school, because the ePortfolios provide a space to refocus despite multiple interruptions. As a microcosm of the college, the Graduate School of Education also has a commitment to engage its students in a theory-with-practice experience that prepares them to help shape the future of learning within the field of education. Students’ ePortfolios offer a place in which students can articulate their aspirations and plans for action.



The Graduate School of Education began its work with ePortfolios in 2010, focusing first on its master’s-level programs. These programs include a Master of Teaching (MAT) for pre-service licensure in K–12 teaching and a Master of Education program that includes four concentrations: Learning and Instruction (focus on K–12 learning), Higher Education Administration, Special Education, and as of fall 2013 a new concentration in eLearning and Instructional Design (with a focus on adult eLearning). The MAT program was the first to become involved with ePortfolios, requiring its students to create ePortfolios with a template structured according to licensing standards, with the goal of making it easy for evaluators to assess student competencies. The MAT students have a common focus and a defined need, and therefore a directed portfolio is most appropriate for the task. MAT students only have twelve weeks to create their ePortfolios, simultaneous with their student teaching responsibilities. Even though this initiative has been logistically successful, faculty members observe that students are not going beyond the basic requirements in their ePortfolio use, indicating limited engagement. In January 2011 the master’s program instituted an ePortfolio requirement for all Master of Education students. Unlike MAT students, Master of Education program students work within many sectors: eLearning, higher education administration, educational nonprofits, organizational learning, and K–12. The Master of Education ePortfolio initiative needed to be flexible enough to serve the needs of this more heterogeneous group. In addition, some faculty expressed a desire for ePortfolios to help students position their program-based learning within the larger context of their developing professional identities, offering a narrative of growth. It was decided that students would create one ePortfolio and use it throughout the program. The faculty also crafted a mission statement to communicate their rationale and aspirations for the initiative to students and part-time faculty:

By pursuing a Master's degree, you are in a process of—metaphorically speaking—writing the next chapter in the story of your life. Your courses are one part of that process. You construct meaning on your journey, considering how your program informs and shapes personal and professional goals. Your ePortfolio is designed to help you document this narrative of growth to see how parts relate to the whole. We believe it will help you improve the quality of your learning experience, both during and after your time in the program.

The template that students used in creating their ePortfolios was simple (see figure 2). It included the following sections:

x About Me x Goals x My Program

The template included open-ended writing prompts, with the goal of providing some structure and focusing students’ attention on professional learning while also allowing ample space for self-direction. For example, prompts in the “About Me” and “Goals” sections included:

x Why did you enroll in the program, and what do you hope to get out of it? Based on your experiences so far, how have you grown as a person and as a professional (e.g., what have you learned, what are you working on, what are your aspirations for the future)?

x How have your professional and life goals changed over time? For example, did you always want to be in the profession that you are in now? Is your current job something that you'd like to do for many years into the future?

Figure 2. Master of Education ePortfolio Template



x What experiences have shaped the path that you have taken thus far in your life? Based on where you are now in your development, where would you like to be three, five, and ten years from now?

Within the “My Program” area students were instructed to create a separate page for each course they took, to upload at least one piece of work from each class, and to write a reflection about their learning. This section of the template also included embedded writing prompts, such as the following:

x Tell the reader what you learned from the course. How did your thinking change? What new skills or capabilities did you gain? If the course influenced and changed your goals, or raised new questions for you, incorporate that into your reflection as well.

In addition to the template and embedded prompts, the program developed a support website for faculty and students that integrated technical tutorials with an orientation to the use of ePortfolios for integrated, reflective learning. Our strategy was to encourage autonomy and intrinsic motivation, within a context of parameters and guidance, and examine the ePortfolios that students created under this system during a summer retreat.

B. Redesign Process, Findings, and Outcomes 1. Inquiry: Who are our students? What is their self-conceptualization?

Institutions, programs, and class cohorts are cultural entities. They are groups of people who, over time, develop a mythos about their identity and accomplishments, a shared narrative that is often unspoken. This narrative includes ingrained beliefs about students: their backgrounds, hopes for the future, and understanding of what they have learned, and how they intend to use that learning in the future. As organizations, how can we make these tacit assumptions visible so that we can reconcile belief and reality, with the goal of growing and improving as programs?

ePortfolios provide a window into student self-perception, making their learning visible in ways that are otherwise difficult to achieve. Because the ePortfolio gathers together work that spans courses and even years of a student’s learning, it is greater than the sum of its parts. Viewed as a whole, including the student’s self-description and explanation of his or her learning, the ePortfolio is a student-centered construct of identity [17]. When the ePortfolios from an entire program are viewed in aggregate, they depict a community of learners, making it possible for us to spot cultural patterns that are not readily apparent in enrollment statistics. In summer 2012 the master’s faculty held an all-day planning retreat, starting the retreat off with a review and discussion of student ePortfolios. We anticipated that process would help us base program revisions on patterns of need in student academic performance. We had no idea that the process would challenge our fundamental beliefs about who we were as a program. The Master’s program ePortfolio requirement had been in effect for about eight months prior to the retreat and had yielded approximately 150 ePortfolios. Prior to the retreat, the ePortfolio Coordinator, who is also a faculty member, reviewed all of the ePortfolios. The other full-time faculty members were each given a subset of 6–8 “active” ePortfolios to review, with the caveat that activity does not always equal quality. ePortfolios were considered to be active if they had been edited or added to within the previous three weeks. Faculty were provided with the following prompts for guided inquiry:

1. Looking at each ePortfolio, what can you tell about the person's background, motivation for pursuing further education, goals, and aspirations for the future? How does this compare with your understanding of the profiles of students who enroll in our programs? 2. Looking at the person's work samples, what appear to be her/his areas of strength and need? For example, what is the person's current capacity for reflection, intellectual curiosity, intrinsic motivation, self-awareness, problem solving, leadership, and ability to create work that is well written and/or well presented? If the person is not well positioned



to succeed, how do you think that should be addressed? 3. Given the strengths and challenges that you observe in the ePortfolio, what does the person need to thrive, both in our programs and after graduation? What are your thoughts about helping the person grow as a reflective practitioner and as an agent of change? Comparing that with your understanding of our programs, what do you think we need to emphasize, keep the same, decrease, or add in the curriculum? What other suggestions do you have for program improvement?

2. Reflection: We are not who we think we are. ePortfolios can serve as a catalyst for discussion and gap analysis. When faculty members examine entire ePortfolios as artifacts of student professional self-identity, they have an opportunity to compare that view with their unarticulated assumptions about program purposes. The day before the retreat, a faculty member from the Higher Education Administration (HEA) concentration expressed concern that she had not been given a representative sample of ePortfolios. Her ePortfolio collection depicted students who were barely out of college and represented less socioeconomic, racial, and ethnic diversity than she would have expected. This was both surprising and problematic. Northeastern University is an urban school with a long history of commitment to "nontraditional" learners, and the HEA concentration is specifically designed for professionals who have been working in the field of education at least three years. The ePortfolio Coordinator assured the faculty member that the sample was representative of the ePortfolio set. The day of the retreat, faculty members reported additional surprises during the ePortfolio discussion, observations that could not be gleaned from the demographic data typically gathered about students. As expected, most ePortfolios depicted people who were employed, but they often weren’t employed in educational capacities—many were preparing for a transition into education from another profession. When writing about goals in their ePortfolios, some students voiced a desire to "break into the business" of higher education, describing unrealistic pathways up the ladder of success that revealed serious misunderstandings about the institution of higher education—for example, envisioning a move from Enrollment to Academic Dean. In retrospect, this profile should have been expected, because people who are earlier in their careers tend to change jobs more often, particularly those with higher levels of education [18]. The curricular and experiential learning needs of these students differed radically from the needs of more mature education professionals, whose motivation for an advanced degree often includes developing and advancing in leadership within the field. In addition, a high percentage of the students were first-generation college graduates, people who perhaps needed extra guidance in navigating their development as education professionals, in addition to an orientation to education as a field of study. Other faculty members observed that the ePortfolios examined were "way too personal," not reflecting a professional sense of self or indicating substantive reflection on how they were developing as professionals in the program. The goals that students expressed were sometimes not even connected to a professional future, such as one student whose goal was to own a house with a garden. Why were most students not representing themselves and their learning within the context of a developing professional identity, in spite of the considerable thought we put into the mission statement and guiding questions? In a report for the Inter/National Coalition for Electronic Portfolio Research, Laurie Poklop and Chris Gallagher observe that genres of writing “are understood to be dynamic and emergent—evolving over time through social interaction.” They theorize that ePortfolios entail evolving genres of writing that challenge students to address new audiences [19]. This conclusion resonates with Bass and Elmendorf’s observations about the intersection between ePortfolios and social pedagogy. Was an ambiguous or heterogeneous sense of audience a factor in how students represented themselves in their ePortfolios? For whom where they writing, and to whom were they presenting their work? To what extent might that audience change over time as they progressed through the program?



Or was the level of metacognition we were expecting too much of a stretch for students’ current capacity? We were asking students to create a page for each course that included work samples and reflections, but perhaps that wasn’t enough. Did the experience need to be more scaffolded? Could the results we were seeing be a consequence of “Facebook culture”? Perhaps students did not know how to create reflective, sophisticated professional self-portraits online using words and images. We speculated that all these factors were at play and that each had significant implications for program redesign. The ePortfolio review sparked considerable debate. To what extent is it our responsibility to help students reflect upon, develop, and articulate a professional identity? Should the style or “genre” of writing vary according to the type of ePortfolio (e.g., documentation, integrative learning, or showcase)? Were we accountable for what students do (or do not do) with their learning? How could the program be redesigned to meet the needs of established and transitioning education professionals? Consensus emerged that the program should prepare students to represent themselves as professionals, speaking to their strengths and aspirations for the future, and that graduates should be able to perceive and articulate connections within courses, the program, and their professional lives. But each decision opened up a new Pandora’s box of questions. Did we need to reconsider program admittance and/or realign the curriculum to the needs of pre-professional students? Were we missing the mark in achieving integrated program learning outcomes? Was there a problem with the ePortfolio requirement implementation? The consensus was that the problem was a combination of all three issues, and this informed subsequent reformulation of the program. We rolled up our sleeves and got to work. The first order of business was to co-author a statement about the program’s mission and distinctiveness, and the competencies that the program envisions for its graduates. That statement follows:

Mission The Master of Education program at Northeastern University develops educators with the skills and intellectual acumen to be effective, to question systemic norms, and to anticipate and shape a more equitable, globally connected society.

Distinctiveness The Master of Education Program x Effectively melds applied professional practice and scholarship x Examines the societal impact of technology upon education worldwide x Explores the learner in context throughout the lifespan x Instills an understanding of education from a worldwide perspective x Guides the development of a degree-wide integrative ePortfolio that is retained by the

graduate for continuing professional advancement.

Competencies A graduate of the Masters Program is a Systems Thinker x Perceives self as an educator participating within a larger system of education x Demonstrates strategic awareness x Looks for patterns and makes connections x Sees how parts relate to the whole, including the implications of systems and

organizational change



Communicator x Proficient as a writer and as a presenter x Capacity to work within groups to listen well, speak well, and co-author x Able to collaborate and communicate in a range of modalities (face-to-face and

online) x Facile with technology, keeps current with emerging technologies and social media Creative Problem Solver x Situational awareness and leadership in identifying and defining challenges x Thinks creatively to generate ideas and be open to alternatives x Develops and implements plans for addressing problems and effecting change x Is resilient in less-than-optimal circumstances Culturally Responsive x Perceives the professional self as functioning within a global context of education x Aware of the dynamics of race, class, gender, and the other cultural factors within

community dynamics and intercultural communication x Interest in, and capacity to perceive, multiple perspectives x Self-aware of cultural perspective and privilege x Capacity to serve as an agent for social justice

These vision statements served as the guiding star and driver for the master’s program redesign.

3. Integration: Re-envisioning the curriculum Following the summer retreat, faculty continued to expand upon the set of vision statements, articulating program-level outcomes and a course map designed to support students in their development of those skills and abilities. The redesigned curriculum weaves ePortfolios throughout the program, within the context of a newly formed required course sequence. The goal is to make competencies explicit and transparent, for both teachers and students, and to scaffold student learning through ongoing engagement with ePortfolio development. Students' journeys begin with the creation of a “Learning ePortfolio” in the foundation course, mature through iterative addition and revision throughout the program, and culminate in a capstone experience during which they transform their learning portfolios into “Showcase Portfolios” that depict both the quality of their work and the caliber of their professionalism. Bass and Eynon observe that ePortfolios have the capacity to record and intensify a “recentered” curriculum in which students and faculty can gain perspective to see the value of learning [15]. In this same vein, the master’s redesign transformed the program from a collection of courses into an intentionally designed learning experience. In just under a year the master’s was transformed from a “degree with a portfolio requirement” into a portfolio program whose students graduate with a collection of signature work that evidences their capabilities. Features of the new program include co-designed cognitive apprenticeship, orientation to the professional context, integrated opportunities for connected experiential learning, variation nested within continuity, and looking back to look ahead. a. Co-Designed Cognitive Apprenticeship The first three to four courses in each concentration have been co-designed by faculty as an integrated suite that takes students through a “cognitive apprenticeship” in the skills, understandings, and capabilities of professionals within the field [20]. They are designed to foster connected learning, in which each course builds upon and complements the next, and the faculty have a clear understanding of how “their” courses intersect with and reinforce other courses in the program.



Each course in the program has a designated “signature assignment” that marries theory and practice, often simulating workplace situations or examining concerns of the profession. In addition to meeting course objectives, the signature assignments align with program competencies and concentration outcomes. Students add signature assignments to their portfolios, often with a written metacognitive reflection on strengths and weaknesses in the work, lessons learned, connections with other professional and program learning, and a self-assessment of progress toward objectives, outcomes, and competencies. For example, Figure 3 depicts the first two courses in the eLearning and Instructional Design concentration, illustrating how the course objectives and signature work function as a set, supporting the development of concentration-level outcomes and program-level competencies. Because the signature assignments have been identified in advance and mapped to the competencies, students have an opportunity to build and refine their work over time. For example, interviews that students conduct with practitioners in the field during an introductory course can be used as the foundation for case studies, opinion pieces, or co-op projects in subsequent courses. Even though the ePortfolio template offers significant opportunities for individualization, it also includes embedded program competencies and concentration outcomes to serve as a lens through which students and faculty can examine samples of signature work. Professors have the ability to view signature coursework within the context of their students' entire learning portfolio. The new ePortfolio template adds a "professional profile" area that provides students with space and prompts for use in describing themselves as professionals, which can be revisited and revised over time to reflect students' developing professional identities.

Figure 3. Example of program constructed as a through line.

Assignments provide evidence of learning and progress toward objectives, outcomes, and competencies.

b. Orientation to the Professional Context A new gateway course entitled “Education as an Advanced Field of Study” carries five credits, instead of four, and includes an orientation for students to ePortfolio development and its anticipated role in their learning. Students are more likely to make connections between their coursework, developing capability,

Master of Education Program Competencies & Outcomes

- Systems Thinker - Communicator - Creative Problem Solver - Culturally Responsive

eLearning & Instructional Design Concentration Competencies & Outcomes - Professional Expertise - Contextual Cognizance - Leadership and Vision

Course 1: How People Learn Objectives (e.g., examine and be able to apply)

- Learning Science, Theories, and Principles - Factors that Influence Learning (e.g. learner perspective, experience, setting) - Critical Analysis and Evaluation Methods Signature Assignments

- Professional Learning Plan (aligned w. competencies); Case Study (analysis of learning principles & evaluation of learner experience in a course or resource)

Course 2: Models for Learning Design Objectives (e.g., examine and be able to apply)

- Models and Methods for Learning Design - Contextual Factors that Influence Decision-Making and Apprach - Selecting an Appropriate Model for the Context and Learning Scenario Signature Assignment

-Prototype for an Online Learning Module, Mobile Learning App, or Open Learning Resource



and emerging professional identity if they perceive themselves as joining a “field” instead of just joining a “program.” The course engenders the perception of education as a complex system, as a community of practice. It orients students to “education” as a profession whose members share common concerns even if they specialize in different pieces of the puzzle. Students investigate the face-to-face and virtual venues where education professionals come together (organizations), processes through which burning issues are identified and controversies are debated, and the print and online publications in which knowledge is shared. The “text” for the ePortfolio component of this course is a Master of Education ePortfolio website that integrates information about the program’s missions, competencies, and curriculum with a multifaceted overview of ePortfolio as tool, process, and product [21]. Students work within their ePortfolios every other week during the course. These assignments integrate technology instruction, an orientation to portfolio pedagogy, and exercises designed to support professional learning. For example, in the second week they draft a “positionality statement” about their perspective, values, and motivations for becoming an education professional and put the document in their ePortfolios. They revisit and revise this statement at critical junctures in the course and in the program (e.g., to compile a list of “burning issues” or controversies in the field of education that resonate with their positionality, to create the “professional profile” section of their ePortfolios). This guided work orients them to the “genres” of ePortfolio writing, mentors them in the connection of theory with practice, and demonstrates the value of regularly updating their ePortfolios.

c. Integrated Opportunities for Connected Experiential Learning In addition to the process of redesigning concentrations, the program elected to develop a new concentration in eLearning and Instructional Design. At the 2012 Educause Learning Initiative conference, organization president Diana Oblinger proclaimed in her opening remarks that “the next step in next-generation learning is connected learning.” This statement resonated with the insights and lessons learned during the Master’s program’s ePortfolio review and curriculum redesign. Oblinger’s words served as a call to action that prompted faculty to ask, “If connected learning is the future, what skills and abilities will next-generation learning designers need in order to help shape and create that future?” In fall 2013 the Master’s program launched the new concentration, designed from the outset to be an integrated learning experience that marries the art of learning design with the evidence-driven inquiry of learning science [22]. The curriculum includes an experiential component that integrates program-based and workplace-based learning. At the midpoint of their studies, students take a course entitled “Connecting Theory with Practice.” The students use their ePortfolios to examine the work they’ve done so far in the program to

x review their progress in relationship to program and concentration competencies, x identify their strengths and opportunities for growth in relationship to the program and

concentration competencies, x document interviews with prospective co-op employers (or conduct an environmental scan within

their own places of work), and x develop a proposal for a piece of significant professional work that addresses a real topic or issue

related to the future of eLearning design, placing the practice of eLearning design within its field of study. The employer is also involved in approving the plan.

This guided but flexible structure makes it possible for both transitioning and established professionals in the field to benefit from workplace-based experiential learning. Because this component is entirely online, it is referred to as a “Virtual Co-op” or “Online Experiential Learning.” During the second half of the program, students carry out their plan independently, but they also participate in portfolio-based Jams (hearkening back to the structure of Connect to Learning), documenting their experience within their ePortfolios in response to a shared thematic prompt (e.g.,



connected learning, design as a collaborative profession), viewing peer portfolios, providing feedback, and refining their Jam write-ups into professional thought pieces. This process uses the ePortfolio as a venue for engaging in a dialectic between theory and practice (see Figure 4), an approach to experiential learning that enacts Bass and Eynon’s concept of a “recentered” curriculum.

Figure 4. Integrative ePortfolio supports dialectic between theory and practice

During the capstone, students draw on this documentation to create a retrospective of their experiential work and develop a professional portfolio. As a final project, they present their portfolio in a live online setting open to all students in the concentration and the faculty.

d. Variation Nested Within Continuity Each master’s concentration prepares its graduates for a different professional trajectory, and so it makes sense that there would be variation in ePortfolio implementation across the concentrations. To what extent are the students keeping documentation portfolios, integrated learning portfolios, or a mixture of the two? At what points in a concentration’s course of study do students reflect upon and improve upon their work? These differences in professional perspectives have been perhaps the greatest challenge, often discussed in heated debate. Some concentrations prefer to have students use their ePortfolios for documentation only during the majority of their studies, waiting until the last capstone course to revise work and develop showcase portfolios. Other concentrations want to have their students keep integrated learning portfolios during the program (documentation plus reflection), revising this work into a showcase portfolio during the capstone. Regardless of differences, the shared goal is to develop a rigorous and coherent master’s-level program that also maintains self-determination within concentrations and among faculty. The primary strategy, as with other aspects of the program redesign, is to make the shared and varying components as explicit as possible among faculty and students. For example, each concentration has its own area within the Master of Education ePortfolio resource site, with total discretion over how the use of ePortfolios within that concentration is described.

e. Looking Back to Look Ahead Even though faculty members weren’t always aware of it at the time, the integrated redesign approach, which touched upon many aspects of the system, is perhaps the most prominent feature of the revised



curriculum. It is reminiscent of Connect to Learning’s ecological catalyst model for institutional transformation. Before the redesign, tacit assumptions about the program goals and aspirations for program graduates were unexamined and unarticulated. Before the redesign, courses were perceived and treated as discrete units, so it is no wonder that students had difficulty perceiving the program as an integrated pathway for professional development and learning. Before the redesign, there were no competencies and outcomes to guide conceptual understanding of the program’s purpose. Before the redesign, the ePortfolio and experiential learning components were add-ons, not woven into the fabric of the program. Before the redesign, faculty developed courses individually and did not benefit from the synergy of co-authorship that comes with learning design.

III. NEXT STEPS AND FURTHER INVESTIGATION The ePortfolios that students created before the redesign are still useful forms of data. They provide a snapshot of students' presentation of self prior to the re-envisioning of the program. The intent is to use ePortfolios generated under the new system as one source of evidence in an iterative process of outcomes assessment and program improvement, which is scheduled to begin nine months after the redesigned program’s launch. The program will use a sampling methodology, similar to the approach taken during the summer retreat, this time using the competencies and outcomes as a guide for review. Portfolios viewed as a whole can serve as qualitative or ethnographic data; the examination of signature work samples in relationship to competency-generated rubrics can provide quantitative data. It is anticipated that, in comparing pre-redesign ePortfolios with those generated under the new system, there will be an increase in professional sophistication, articulated connections between academic and workplace practice, and cognitive connections across the course of study. ePortfolios will continue to serve as an outward and visible sign of the alignment between student learning and program aspirations, keeping us honest about the impact of our program and shining a flashlight on areas for improvement. The program will continue its waltz of improvement through inquiry, reflection, and integration.

IV. ACKNOWLEDGEMENTS The author wishes to express appreciation for the Sloan-C Best-in-Track award that led to this publication. She also thanks Bret Eynon. Judit Török, Laura Gambino, and countless others in the Connect to Learning network who have provided feedback and support during the past two years. Thanks go out to Trent Batston, without whose recommendation she would not have become a part of the C2L community. Thank you to Lydia Young and Doug DeNatale, who provided feedback on earlier drafts of this article. Finally, it is with great appreciation that she thanks the Fund for Innovation in Post-Secondary Education for the grant that made this all possible.

V. ABOUT THE AUTHOR Dr. Gail Matthews-DeNatale serves on the faculty at Northeastern University's Graduate School of Education, where she leads the M.Ed. concentration in eLearning and Instructional Design and specializes in the use of portfolios for online experiential learning. She is a Co-Principal Investigator for the national Connect to Learning: Catalyst for Learning Initiative, funded by the Fund for Innovation in Post-Secondary Education. Prior to Northeastern, she held positions at Simmons College, George Mason University, and the University of South Carolina. She is on the boards of the Association for Authentic, Experiential, and Evidence-Based Learning (AAEEBL), Higher Education Teaching and Learning Review, former member of the NERCOMP's Board of Trustees, and has served on the Educause Learning Initiative's Blended and Online Learning focus advisory group. Her successful 2009 grant award from the Alfred P. Sloan Foundation launched the Simmons College Blended Learning Initiative, and her 2013 Sloan-C Conference presentation on ePortfolios as a tool for curriculum redesign received the Best-in-



Track award. Matthews-DeNatale's research interests include digital storytelling, fluency with emerging technology, and authentic formative assessment.

VI. REFERENCES 1. Eynon, B. et al. Catalyst for Learning, Making Connections National Resource Center,

2014. http://c2l.mcnrc.org 2. Tosh, D., Werdmuller, B., Chen, H., Penny Light, T., and Haywood J. The Learning Landscape:

A Conceptual Framework for ePortfolios. In: Jafari, A., Kauffman, C. (Eds.), Handbook of Research on ePortfolios, IDEA group: Hershey, PA, 2006.

3. Siemens, G. Connectivism: A Learning Theory for the Digital Age (2004). http://www.elearnspace.org/Articles/connectivism.htm

4. Higher education ePortfolio support websites present a host of typologies, indicating that categorization is important to the conceptualization of ePortfolios, but there is no definitive typology. See, for example, the following: - Stonybrook University: http://it.stonybrook.edu/help/kb/types-of-eportfolios - Cornell University: http://eportfoliohelp.cit.cornell.edu/types-of-eportfolios - Regis University: http://academic.regis.edu/LAAP/eportfolio/basics_types.htm - eP California: http://eportfolioca.org/index.php?option=com_content&view=article&id=58

5. Chen, H., Penny Light, T., and Ittelson, J. Documenting Learning with ePortfolios: A Guide for College Instructors, Jossey-Bass: Indianapolis, IN, 2011.

6. Peet, M., Lonn, S., Gurin, P., Boyer, K.P., Mattney, M., Marra, T., Taylor, S., and Daley, A. Fostering Integrative Knowledge through ePortfolios. International Journal of ePortfolio 1(1): 11–31 (2011). http://www.theijep.com/pdf/IJEP39.pdf

7. Peet M. The Integrative Knowledge Portfolio Process: A Program Guide for Educating Reflective Practitioners and Lifelong Learners. MedEdPORTAL, 2010. https://www.mededportal.org/publication/7892

8. Bass, R. and Elmendorf, H. Designing for Difficulty: Social Pedagogies as a Framework for Course Design in Undergraduate Education. Teagle Foundation White Paper, 2012. https://blogs.commons.georgetown.edu/bassr/social-pedagogies

9. Wick, J. Using ePortfolio: A Career Instruction Manual. Connecticut Distance Learning Consortium. Connecticut Distance Learning Consortium: Newington, CT, 2004. http://www.ctdlc.org/evaluation/fipsedocs/careermanual2.pdf

10. Barrett, H. Balancing the Two Faces of ePortfolios. In: Hirtz, S., Kelly, K. (Eds.), Education for a Digital World 2.0: Innovations in Education, Volume 2. Open School BC: British Columbia, CA, 291–311, 2011. http://openschool.bc.ca/info/edu/7540006133_2.pdf

11. Yancey, K.B. Electronic Portfolios a Decade into the Twenty-first Century: What We Know, What We Need to Know. Peer Review 11(1): 28–32 (2009).

12. The Association for Authentic, Experiential and Evidence-Based Learning (AAEEBL). About. (n.d.) http://www.aaeebl.org/about

13. AAC&U. VALUE: Valid Assessment of Learning In Undergraduate Education. (n.d.) http://www.aacu.org/value/rubrics

14. Arcario, P., Eynon, B., and Klages, M. ePortfolio-Based Outcomes Assessment. Video Report for the Bellwether Foundation, 2012. http://faculty.laguardia.edu/ctl/mc/bellwether

15. Bass, R. and Eynon. B. ePortfolios and the Changing Landscape of Higher Education. AAEEBL Keynote Presentation Conference Presentation. Boston, MA, 2013.

16. Kolb, D. Experiential Learning: Experience as the Source of Learning and Development. Prentice-Hall: Englewood Cliffs, NJ, 1984.

17. Bass, R. et al. The Visible Knowledge Project. (n.d.) https://cndls.georgetown.edu/about/grants/vkp 18. Bureau of Labor Statistics. Number of Jobs, Labor Market Experience, and Earnings Growth:

Results From a Longitudinal Survey (2012). http://www.bls.gov/news.release/nlsoy.toc.htm

http://c2l.mcnrc.org/

http://www.elearnspace.org/Articles/connectivism.htm

http://it.stonybrook.edu/help/kb/types-of-eportfolios

http://eportfoliohelp.cit.cornell.edu/types-of-eportfolios

http://academic.regis.edu/LAAP/eportfolio/basics_types.htm

http://eportfolioca.org/index.php?option=com_content&view=article&id=58

http://www.theijep.com/pdf/IJEP39.pdf

https://www.mededportal.org/publication/7892

https://blogs.commons.georgetown.edu/bassr/social-pedagogies

http://www.ctdlc.org/evaluation/fipsedocs/careermanual2.pdf

http://openschool.bc.ca/info/edu/7540006133_2.pdf

http://www.aaeebl.org/about

http://www.aacu.org/value/rubrics

http://faculty.laguardia.edu/ctl/mc/bellwether

https://cndls.georgetown.edu/about/grants/vkp

http://www.bls.gov/news.release/nlsoy.toc.htm



19. Gallagher, C. and Poklop, L. The Effects of ePortfolios on Audience Considerations in First-Year Writing Courses. A research report prepared for Cohort VI of the Inter/National Coalition of Electronic Portfolio Research, 11–12, 2012.

20. Collins, A., Brown, J.S., and Holum, A. Cognitive Apprenticeship: Making Thinking Visible. American Educator 15(3): 6–46 (1991).

21. Matthews-DeNatale, G. et al. Master of Education Program ePortfolio Resources (2013). https://northeastern.digication.com/2013_master_of_education_eportfolio_resources

22. Laurillard, D. Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology. Routledge: New York, 2012.

https://northeastern.digication.com/2013_master_of_education_eportfolio_resources

A Multimedia-Rich Platform to Enhance Student Engagement and Learning in an Online Environment


A MULTIMEDIA-RICH PLATFORM TO ENHANCE STUDENT ENGAGEMENT AND LEARNING IN AN ONLINE ENVIRONMENT T. Scott Bledsoe Azusa Pacific University Bruce D. Simmerok Azusa Pacific University ABSTRACT Utilizing multimedia tools, such as videos and audio clips, can be an effective way to promote student learning and engagement in online settings. This study explores the implementation of a photo-rich comprehensive counseling center environment through which students of a semester-long online graduate psychology class learned about important research methodology concepts. Student feedback is provided along with implications for student learning in future online course endeavors. This multimedia course design was the recipient of the 2013 Sloan-C Effective Practices Award.

KEY WORDS Multimedia, online classroom, student learning, mock counseling center, research methodology, web design.

I. INTRODUCTION Educational institutions often champion online learning as a viable supplement to face-to-face instruction for their students. In promoting web-based programs, many institutions cite studies that have consistently shown the effectiveness in teaching students across a range of disciplines online [15]. Though some students tend to prefer in-class schooling over its online counterpart, the advent of new multimedia features, such as virtual rooms, videos, photos, and audio clips, may ultimately lead to increased student acceptance of this instructional format. Citing its environment of virtual learning with a broad range of modalities from which to choose, McFarlane [12] noted that “the [media-filled] playgrounds of today’s generations and this fact makes them ideal as platforms for educating today’s and tomorrow’s youth” (p. 5). A. The Online Learning Paradigm Online education, also called distance learning and Internet learning, can be defined as the process of providing systematic training to students in a given area of study within a web-based format. The vehicle for delivery of this platform, referred to as “educational cyberspace” [12] includes all testing procedures, lesson plans, administration of grades, and communication with instructors. A program that provides the framework through which students complete their coursework (e.g., Blackboard, Sakai) operates within a virtual learning environment [4]. Online learning systems have been scrutinized for their educational value in academic settings. In a meta-analysis of 50 studies conducted between 1994 and 2006 that compared online to face-to-face classroom formats, it was found that “students in online conditions performed modestly better, on average, than those learning the same material through traditional face-to-face instruction” [15]. Factors that contributed to greater effect sizes included types of knowledge tested, strength of the study design, and instructional approach. McFarlane [12] noted that a determining factor for implementing successful

A Multimedia-Rich Platform to Engage Student Engagement and Learning in an Online Environment


online programs was organization structure and the adherence to consistent rules in addition to a chain of command and teamwork among instructors, students, and technical support officials that was conducive to student learning. Some online settings support computer-simulated learning in which participants are immersed in scalable 3D settings and navigate through alternate worlds [14], a concept popularly known as virtual reality. One example can be found in the computer program, Second Life, which invites constructivist-based, student-centered learning [3] and allows participants to design and manage avatars that interact with each other in real time [5, 7, 9]. The virtual world program, Second Life, was created in 2003 and by the end of the decade, many educational institutions bought and designed their own virtual campuses that were often located on elaborate 3D islands within the program [5, 7]. Using their avatars, students visited the campuses, which provided resources such as informational brochures and lectures in the form of downloadable PowerPoint presentations. Yet since its inception, the program has been challenged by design flaws that make avatar control and interaction difficult to manage [5]. Sizable hardware requirements for Second Life have created additional problems for many students, resulting in frequent program freezes and computer reboots. Subscription fees and costs of creating and maintaining virtual campuses are sometimes prohibitive, and their academic benefits have yet to be proven, making the program questionable for efficient virtual online learning. A practical alternative to Second Life exists in the form of augmented reality (AR), a variation of virtual reality in which users utilize “virtual objects superimposed upon or composited with the real world” [1]. For instance, in an AR setting a photo of a classroom on a course website could be enhanced with a virtual book on which online students could click to access instructions. As such, “AR supplements reality instead of completely replacing it” (p. 356). Because this definition is more compatible with the current study, all references to online multimedia environments will subsequently refer to augmented as opposed to virtual reality.

B. Multimedia Tools to Supplement Online Learning Students utilize technology when they communicate through social networking or share videos and information with others via their smartphones. Many creators of online teaching programs also seek to incorporate popular technology to promote learning-based agendas. Kumar [10] claimed that student engagement in online environments was heightened when there was evidence of “effective and engaging multimedia resources to scaffold and anchor purposeful and active learning” (p. 6). Course instructors may thus seek to integrate socially sanctioned platforms into online settings in a way that promotes effective learning for their students. Doolittle, as cited in Mandernach [11], defined multimedia as a “presentation of instruction that involves more than one delivery media, presentation mode, and/or sensory modality” (p. 2). Compiling the right tools for a successful multimedia environment may thus be an important step in promoting student learning. Miller [13] suggested using an even blend of media components that were tailored to the course and to the profession of study. He recommended supplementing the required text with video segments, audio clips, and interaction with social media (e.g., Facebook) to promote student collaboration and feedback. When utilizing multimedia in web-based course delivery, Mandernach [11] advocated using instructor-personalized multimedia components that promote collaboration in the often “faceless environment of online learning” (p. 5). In a study involving online students of a general psychology course taught across sequential terms, the author utilized weekly videos and audio-based PowerPoint presentations and designated specific times for e-mail and chat with students. She concluded that tailoring an interactive course protocol with a consistent multimedia presence created a high level of engagement and a strong atmosphere for learning, as evidenced by positive qualitative and quantitative student feedback at the end of the course. Additional successful multimedia-based online programs focused on language instruction [2, 6] and the teaching of data structures [8].



Learning strategies that incorporate multimedia in significant ways may become more prominent in future online course designs. Yet more data is needed in order for academic institutions to actively promote this idea. The current study attempts to achieve this goal by promoting dynamic learning of course content through an environment that is both stimulating and practical. The research questions for this study are as follows: In what ways can multimedia environments enhance student engagement? How can multimedia environments promote student learning?

II. METHOD A website with a strong multimedia component was created to enhance student learning of graduate psychology in an existing online research methodology class. In developing the site, the creators envisioned moving beyond a traditional online format to provide students with a hands-on feel for research as it applied to their field of study. The designers utilized the Sakai web platform to showcase a mock community counseling center environment replete with a virtual office, therapy rooms, and a classroom, which featured assignments and instructional videos. During the spring of 2013, 59 students in four separate sections were enrolled in the new photo-rich course. At periodic points in the semester, they provided feedback about their experiences of navigating the online site as they learned research-based concepts. In developing the multimedia aspect of the course, the site creators, one a web designer and the other an instructor of the class, took photos of various rooms inside a local community counseling center. Ten of the photos were eventually assembled as templates for the online environment, and a web design program, Dreamweaver, was used to create a framework for easy navigation of the course. To link the rooms together, the main office photo was modified to represent a visual hub for the center, which featured a directory of each room, a course calendar, and links to the course syllabus (figure 1). The designers incorporated a photo map for easy access to each of the available rooms on the website and FAQs about the course and the instructor’s bio and hours of availability were also provided.

Figure 1. Main Office of the Mock Community Counseling Center



The classroom photo offered students a visual prompt for all course assignments. By clicking on a whiteboard on which each session was listed, students ascertained their weekly tasks. During the first week of the semester, for example, students clicked “Session 1” and were linked to a supplementary classroom that contained the assignments for that week (figure 2). The classroom photo also featured a wall-mounted television set that linked directly to a series of instructional videos based on course assignments. Each session-based classroom also contained a television set which, when clicked, offered 10-minute videos of research lessons and assignments provided by the course instructor.

Figure 2. Classroom for Session 1

Additional rooms provided visual cues and instructions for completing tasks, such as creating therapy treatment plans, reviewing research articles, and participating in discussion threads. A waiting room offered a coffee table that linked directly to helpful resources for students, and a group room featured instructions for a semester-long group assignment. To promote student self-care, a relaxation room provided soothing sounds and prompts. Each room also offered audio recordings of former students discussing ideas for successfully completing course assignments. Student photos accompanied the recordings and by directly clicking on the photos, site visitors accessed helpful hints of previous students from the course. The most elaborate visual and auditory display appeared in the Group Room in which 12 student photos were assembled to provide auditory instructions on group project assignments throughout the semester (figure 3). Because students worked in groups to complete the semester-long project, the room also served to build a sense of community among current and former members of the class.



Figure 3. Group Room

The course designers utilized resource tabs from the Sakai web format to promote learning concepts in the counseling center environment. The Assignments tab accessed weekly assignments and quizzes, research article reports, and treatment plan activities. The Forums tab provided student access to topical discussions about research. Specific rooms (e.g., the Forums Room) were also created to help students understand the purpose of each assignment. For the Resources tab, a Doc Sharing folder was created in which important and supplemental course documents were stored. Students sent comments and concerns to the instructor by utilizing the Messages tab and a chat tool. All assignments were listed in the course syllabus and explained in weekly instructional videos accessed by clicking on the television in the classroom (figure 2). Quizzes and reports were graded by the instructor and posted to the Sakai Gradebook. The instructor invited student feedback throughout the course, and during the sixth week, a SurveyMonkey questionnaire was distributed in order to evaluate student experiences of the online class. Six items specifically addressed their level of satisfaction with the mock community counseling center environment (table 1). Later in the semester, students were asked to provide input on ways that the virtual counseling center environment could be improved upon to augment their learning of research concepts.

III. RESULTS As part of the SurveyMonkey questionnaire, students were asked to rate their experiences of the online research methodology course on a scale from one to five, with five representing a high degree of satisfaction. For items that addressed the mock counseling center environment specifically, 76% of the students agreed or strongly agreed that the online counseling center helped them learn in an online setting, and 71% endorsed the helpfulness of the environment in their learning of research concepts (table 1). Regarding specific virtual rooms of the course, the office and classroom generated the most positive responses (84% and 91% respectively), while opinions about the Group Room were less enthusiastic (69%). When asked about their preference for online platforms, only 19% expressed either agreement or strong agreement that they preferred a more traditional online format for the course.



Variable %~Agree M SD ______________________________________________________________________ Student impressions of the Mock Counseling Center experience (Scale = 1-5) The mock counseling center environment helps me 76 3.95 0.86 learn in an online setting. The mock counseling center environment is helpful 71 3.80 0.90 in my learning of research concepts. The Office helps me to navigate the online course. 84 4.21 0.91 The Group Room photos and audio clips help me to understand the group projects more clearly. 69 3.88 0.89 The Classroom setting is nicely organized for aiding 91 4.28 0.63 in my completion of weekly tasks. I would prefer a more traditional online course format. 19 2.50 1.21 Table 1. SurveyMonkey Questionnaire (N=43) In the online survey evaluations, students also provided qualitative feedback about their experience with the mock community counseling center. Positive comments included the following: “Compared to other online courses, I am surprised at how much I like the interactive setup of this course,” “There are lots of resources at my fingertips,” and “All the information and assignments that I have navigated through was well thought out and developed,” and “I love the office and the amount of effort that [the instructor] has put into this online course to make it a good experience.” Areas of concern were expressed as follows: “The pitfall of the website is that it is overwhelming with all of the different rooms and buttons. It's almost too complicated” and “I did find the Sakai classroom idea confusing at first. I am not sure how it could be clearer but I would like that.” When asked to identify areas of growth for the website, several students responded with comments such as, “I think the site is outstanding, no ideas here . . .” and “I'm not a big fan of online classes but this course is optimal for the material we are learning. I am enjoying it.” Quantitative data from the online class was compared over a four-year period with the class taught by the same instructor who utilized a similar format of course delivery (e.g., research-based article reviews and treatment plans, threaded discussions, weekly assignments, and a semester-long group project). The most recent year of data reflected the implementation of the mock counseling center setting (table 2). Noteworthy findings included a significant increase in student endorsement of their enjoyment of the course (79% as opposed to 65% the previous year) and reduced difficulty in understanding the online format of the course (12% vs. 42% the previous year).



Variable %~Agree ______________________________________________________________________ 2013 2012 2011 2010 I am enjoying the Research Methodology course so far. 79 65 69 70 I am learning important research concepts. 86 86 84 83 I am having difficulty with the online format of the course. 12 42 10 17 Table 2. Yearly Survey Comparisons

Student knowledge of research concepts was tested consistently throughout the online course. Through weekly quizzes and periodic reports, participants demonstrated their understanding of salient statistical information. For example, a multiple-choice quiz was presented in the final instructional video, and students were instructed to watch the film and submit their answers through the appropriate Sakai Assignment tab. Although the test was not standardized and results cannot be generalized beyond the current study, 96% of students scored 90% or higher, indicating at least a general understanding of research concepts presented in the course. During the ninth week of the online course, students were asked to discuss ways that the mock community counseling center environment could be modified to improve their learning. The following ideas were suggested: “Perhaps add a checklist page or room, where the students could check off every step of what is due that week,” and “Sometimes I wish that everything I needed for the week was in one place,” and “Add a link to the office that will take us directly to the documents in the ‘Doc Sharing’ file under Resources. I've found these documents very helpful but sometimes I don't check this folder as frequently as I should.” Students also expressed enthusiasm for the course format, as evidenced by the following comments: “I have taken several online classes between undergrad and grad school. I must say, this is by far the most interactive and positive experience that I have had with an online class. The ‘mock clinic’ has been a great visual for me as I navigate Sakai,” and “It almost feels like an in-person class in some ways!”

IV. CONCLUSION Online multimedia environments provide students with opportunities to learn and interact in a variety of ways [11, 12]. Participants can learn important concepts by following course procedures enhanced by text, photos, videos, and a variety of computer programs; interacting with fellow students through e-mail, chat, and social network outlets; and completing such programs in the comfort of their own homes. Research studies have demonstrated the effectiveness of utilizing such online classroom formats [2, 6, 8, 11] in educational settings. In the current study, online students of a graduate psychology research methodology class completed a semester-long course in which they learned important statistical concepts, collaborated in groups on research-based projects, and demonstrated their knowledge through tests and quizzes. Students utilized photo-based rooms in a mock community counseling center environment to enhance their understanding and reinforce important research concepts. Throughout the course, students provided positive feedback about the multimedia setting, as evidenced by the following statement: “I love the office and the amount



of effort that [the instructor] has put into this online course to make it a good experience.” When compared to participants of the same course from earlier years (without the photo-rich environment), students graded their enjoyment of the course much higher (79% vs. 65% from the previous year), and their difficulty navigating the online environment as lower (12% vs. 42% from the previous year). Additionally, scores on quizzes and other course assignments indicate that students appeared to have learned effectively in the multimedia-focused environment. In conclusion, online environments in which students interact with a range of multimedia tools can offer endless ways to learn and process new information while enjoying the benefits of technology. By creating a realistic setting, such as a mock community counseling center setting and linking rooms in ways that approximate reality and stimulate learning, students may demonstrate improvements in learning and increased enjoyment in the online course environment.

V. ABOUT THE AUTHORS T. Scott Bledsoe, PsyD. is an Assistant Professor in the Department of Graduate Psychology at Azusa Pacific University (APU). He graduated from APU and received his doctoral degree in clinical psychology with an emphasis on family therapy. He received his undergraduate degree in business administration from Birmingham-Southern College in Alabama and taught business and marketing as a Peace Corps volunteer in Kenya, Africa. Before embarking on a career in clinical psychology, he worked as an elementary school teacher and technology coordinator in South Central Los Angeles, California.

Bruce D. Simmerok, Ph.D. is the Director of Instructional Technology for the Center for Innovative Teaching and Technology. He received his BA from, Taylor University, Upland, Indiana, 1968 in Psychology, MS from Indiana State University, Terre Haute, Indiana, 1969 in Psychology and Ph.D. from University of Southern California, Los Angeles, California, 1977 in Special Education. He was previously Executive Director of the Center for eLearning and Teaching at Azusa Pacific University since 1998. Previously he was the Director of the Office of Faculty Development and Program Director of Pupil Personnel Services and Special Education.

VI. REFERENCES 1. Azuma, R.T. A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments,

6(4): 355-385 (August 1997). http://www.cs.unc.edu/~azuma/ARpresence.pdf. 2. Banados, E. A Blended-learning Pedagogical Model for Teaching and Learning EFL Successfully

through an Online Interactive Multimedia Environment. CALICO Journal, 23(3): 533-550 (2006). https://www.calico.org/html/article_105.pdf.

3. Bledsoe, T.S., and Harmeyer, D. Constructivist-based Teaching in Second Life from a Student’s Perspective: A Model Proposal. Journal of Educational Technology, 8(2): 34-41 (July 2011).

4. Dillenbourg, P. Virtual Learning Environments. Workshop presented at EUN Conference 2000, Geneva, Switzerland (2000).

5. Gorman, P. Has Second Life Lived Up to Expectations? Ariadne, 68 (March 2012). http://www.ariadne.ac.uk/issue68/gorman.

6. Huang, S.X., Dedegikas, C., and Walls, J. Using Multimedia Technology to Teach Modern Greek Language Online in China: Development, Implementation, and Evaluation. European Journal of Online and Distance Learning (September 2011). http://www.eurodl.org/?p=current&article=417.

7. Inman, C., Wright, V.H., and Hartman, J.A. Use of Second Life in K-12 and Higher Education: A Review of Research. Journal of Interactive Online Learning, 9(1) (Spring 2010). http://www.ncolr.org/jiol/issues/pdf/9.1.3.pdf.

8. Junaidu, S. Effectiveness of Multimedia in Learning and Teaching Data Structures Online. Turkish Online Journal of Distance Education, 9(4): 97-107 (October 2008). http://files.eric.ed.gov/fulltext/EJ816483.pdf.

http://www.cs.unc.edu/~azuma/ARpresence.pdf

https://www.calico.org/html/article_105.pdf

http://www.ariadne.ac.uk/issue68/gorman

http://www.eurodl.org/?p=current&article=417

http://www.ncolr.org/jiol/issues/pdf/9.1.3.pdf

http://files.eric.ed.gov/fulltext/EJ816483.pdf



9. Kirriemuir, J. The Second Life of UK Academics. Ariadne, 53 (October 2007). http://www.ariadne.ac.uk/issue53/kirriemuir.

10. Kumar, M. A Critical Discourse in Multimedia Design: A Pedagogical Perspective to Creating Engaging Online Courseware. e-Journal of Instructional Science and Technology, 7(2) (2004). http://www.ascilite.org.au/ajet/e-jist/docs/Vol7_no2/FullPapers/Muthukumar_140105_2.pdf.

11. Mandernach, B.J. Effect of Instructor-personalized Multimedia in the Online Classroom. International Review of Research in Open and Distance Learning, 10(3): 1-19 (June 2009). http://www.irrodl.org/index.php/irrodl/article/view/606/1263.

12. McFarlane, D.A. A Comparison of Organizational Structure and Pedagogical Approach: Online Versus Face-to-face. The Journal of Educators Online, 8(1): 1-43 (January 2011). http://files.eric.ed.gov/fulltext/EJ917871.pdf.

13. Miller, M.V. Integrating Online Multimedia into College Course and Classroom: With Application to the Social Sciences. MERLOT Journal of Online Learning and Teaching, 5(2): 395-423 (June 2009). http://files.eric.ed.gov/fulltext/ED506000.pdf.

14. The New Media Consortium & Educause Learning Initiative. The Horizon Report (2007). http://www.nmc.org/pdf/2007_Horizon_Report.pdf.

15. U.S. Department of Education. Evaluation of Evidence-based Practices in Online Learning: A Meta-analysis and Review of Online Learning Studies (September 2010). http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.

.

http://www.ariadne.ac.uk/issue53/kirriemuir

http://www.ascilite.org.au/ajet/e-jist/docs/Vol7_no2/FullPapers/Muthukumar_140105_2.pdf

http://www.irrodl.org/index.php/irrodl/article/view/606/1263

http://files.eric.ed.gov/fulltext/EJ917871.pdf

http://files.eric.ed.gov/fulltext/ED506000.pdf

http://www.nmc.org/pdf/2007_Horizon_Report.pdf

http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf

Khan Academy: The Illusion of Understanding


KHAN ACADEMY: THE ILLUSION OF UNDERSTANDING Marc Schwartz University of Texas at Arlington ABSTRACT This paper examines the ongoing challenge of defining what learning means from the perspective of the cognitive and learning sciences, especially as it unfolds in online environments. To better define learning as well as offer guiding principles, this paper uses Khan Academy as an example of what some high-profile individuals, such as Bill Gates, are claiming to be the future of education. I offer five guiding observations that provide a structure for understanding the learning process and apply them to Khan Academy as a means of revealing what I call the illusion of understanding, and I replace that view with a more authentic understanding of the learning process and the means to achieve that understanding. KEYWORDS Khan Academy, cognitive development, hierarchical development, understanding, practice, feedback, context sensitive, online education, MOOC, e-learning, pedagogy, instructional design

I. INTRODUCTION: THE ILLUSION OF UNDERSTANDING A serious challenge for educators and students is avoiding what I call the illusion of understanding. Most often, the illusion arises when educators and students fall into the following relationship: “I’ll pretend to teach as long as you pretend to understand.” The interaction is neither malicious nor necessarily conscious. In fact, the relationship emerges out of a persistent and pervasive misunderstanding of the learning process, one repeated in numerous contexts throughout the history of education, apparent in the introductions of textbooks over the last 150 years (which justify the newest and latest textbook), and now repeated at Khan Academy. Recognizing the relationship in educational contexts is challenging because educators are embedded in the process and, if they are reading this article, are most likely a product of the process. But the effort is necessary if we are to understand and address our misconceptions about teaching and learning and, more importantly, to avoid succumbing to the illusion that real teaching and learning is occurring.

A. What does the illusion of understanding look like? An illustration from my own work of the illusion of understanding occurred years ago when my colleagues and I were training science educators in a graduate course at the Harvard Graduate School of Education [1, 2]. A significant part of the course involved students addressing a number of problems involving basic principles of science. The problem students found the most intriguing is the following:

Imagine you come upon a canoe in a swimming pool and you remove the large anvil you find in the canoe and submerge the anvil in the pool. If you note the level of water before commencing the operation and again after the anvil is completely submerged, does the water level of the pool change?

This question, describing an unlikely situation, might have created some dissonance, but our students were well embedded in the process of answering similar questions and accepted the challenge. You might want to consider this problem as well before reading further. This scenario was created to challenge students’ understanding of Archimedes’ principle, a concept found in many school curricula around the



world. At Khan Academy, this principle is directly addressed in two of twelve sessions on “Fluids” (parts five and six): http://www.khanacademy.org/science/physics/fluids/v/fluids--part-5. What you should note about the canoe scenario is that it can be solved entirely without mathematics. The problem is purely conceptual and has just three possible solutions: the water level will go up, remain unchanged, or go down. For five years my colleagues and I posed this and similar challenges to students, and, surprisingly, for five years the distribution of their answers appeared to be no better than as if by chance [2].

B. What does the illusion of understanding feel like? Students were also surprised, although unpleasantly, at their inability to arrive at a definitive answer to the canoe problem. They thought they understood Archimedes’ principle until they faced this or similar conceptual problems to which they had to apply the principle. The level of distress these problems created for students surprised us. Some expressed a feeling of panic at the thought that if they didn’t understand this principle, perhaps they didn’t understand anything they had learned in school. This was perhaps the most surprising insight for us: we are all subject to the illusion of understanding until we are somehow forced to face it. But the most insidious aspect of the illusion of understanding is that it masks what I will call authentic understanding.

C. What is authentic understanding? Over time, my colleagues and I learned to appreciate a small number of important observations about the nature of authentic understanding that distinguishes it from the illusion of understanding. The observations are straightforward, but the nuances of each make them easy to dismiss because they are hard to integrate into what many educators, including Khan, believe understanding to mean. To be fair, those educators’ views are not often explicitly stated but are revealed through their choice of interventions and assessments. Meeting this challenge is not trivial. Understanding is a complex phenomenon. While difficult to define or measure precisely [3], it is nonetheless possible to identify key characteristics and processes that support its development. The insights and recommendations that follow have emerged not only from my own observations but also from the work of numerous cognitive scientists, neuroscientists, and educators over the past century [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. Here, I describe five critical insights concerning the nature and development of authentic understanding. The insights help confront the illusion of understanding, which continues to survive and thrive in a variety of educational contexts. Khan Academy is just the latest reminder of our collective struggle to profit from the learning sciences; however, more challenging in the context of the virtual classroom is that the teacher-student relationship cannot profit from even the most basic form of communication—the student’s confused look. As everyone stares at the magic Smartboard, the illusion that Khan is teaching and that the observer is learning is that much easier to perform. To be fair, the problem inherent at Khan Academy is one that appears in many educational contexts, and the academy’s potential strengths are noteworthy, but not because they necessarily support authentic understanding. As Khan highlights in interviews, the central issues he addresses are student attention span, the availability of instruction, and student control of the pace of instruction. While these features allow students to more easily consume videos intended to be instructional, they are nonetheless peripheral to the goal of developing students’ authentic understanding.

II. FIVE KEY OBSERVATIONS ABOUT AUTHENTIC UNDERSTANDING The following five observations offer a working framework of authentic understanding:

• Authentic understanding depends on hierarchically organized knowledge. • Authentic understanding is grounded in direct experience. • Authentic understanding is stabilized by practice (generally at every level within the hierarchy). • Authentic understanding requires formative feedback.



• Authentic understanding is context-sensitive. The generality of these criteria is supported by the fact that our students were extremely bright, motivated, and often just as surprised as we were by their inability to leverage the formulas they had learned by heart and used for years in science classrooms to solve problems. Dismissing our students’ struggle with the conceptual problems as being peculiar to them was not easy, as they were considered to be among the best in the nation. Thus, the first important realization was that intelligence had little to do with the challenges the students were facing. In any situation in which these five criteria are ignored, understanding becomes fragile and unstable, and our ability to recall or apply what we remember is compromised.

A. Observation one: authentic understanding depends on hierarchically organized knowledge. Understanding, the nature of which has been unpacked by over a century of research, is hierarchical in structure [14, 15, 16, 17, 18, 19]. Each new achievement within the hierarchy becomes the foundation for the next more complex, more integrated coordination of earlier achievements. The work of Piaget and scores of neo-Piagetian scholars have documented this process, in which sensorimotor experiences (i.e., what we learn through our senses) become the foundation for representations [20] of the content of experiences (e.g., words, pictures, graphs, tables, etc.), and later those representations are coordinated into abstractions that transcend the concrete nature of our experiences and the representations and relationships that emerged earlier from our sensory experiences. Thus, abstractions, such as democracy, justice, or Archimedes’ principle, emerge as a new way of understanding the rich coordination of representations accumulated through various contexts and practiced on multiple occasions. However, this achievement, which is the outcome of a powerful synthesis of concepts, is often belied by simple-looking words and phrases like “democracy” or “Archimedes’ principle,” which at face value do not appear to be remarkably different from other words students use, such as “gavel” and “anvil.” All of the three tiers (sensorimotor, representation, and abstraction) are qualitatively very different ways of understanding, and each depends on the earlier tier to achieve the more complex understanding of later tiers. Progress for students is, however, more nuanced than what is captured by these three major developmental steps. Within each tier researchers have noted finer degrees of achievement, described as levels [15, 16, 19, 21]. For example, the first level of the representational tier signifies the ability to name objects, such as the object in the canoe (i.e., the anvil). The next level (in the representational tier) involves coordinating exemplars from the first level into a new understanding, such as knowing that putting the canoe (or any object) into the pool will raise the water level. There are still two more levels of complexity in each tier that require additional exemplars of earlier levels as well as further coordination before moving to the next tier. The transition to the abstract tier requires the consolidation of representational skills from all four levels into a qualitatively new way of thinking about the world, which is summarized in words such as “justice,” “democracy,” or “Archimedes’ principle.” Details regarding the nature of levels and tiers, their structure and relationship to each other, the criteria for measuring the complexity of each level, and movement from one level to the next are well documented and illustrated by numerous researchers [16, 19]. For the purposes of this article, there are two important notions about any skill demonstrated at any level in any tier. First, the skill is observable, which is of particular importance to educators and students. Second, movement up the hierarchy involves the coordination of less complex skills into skills of greater complexity, like a juggler adding more objects into a juggling routine. As skills become more complex, the coordination becomes increasingly more complex. Time and practice will allow for some consolidation of earlier skills into more stable skills of greater complexity. However, as every juggler knows, practice is necessary to maintain the coordination of plates, balls and knives; otherwise ideas, like objects, crash to the ground.



1. Do Khan Academy lessons support the development of hierarchical knowledge? Looking at Khan’s series of videos related to Archimedes’ principle as a representative example, the layout of the lessons is not compatible with the observation that humans build knowledge hierarchically. For the most part, he introduces ideas at the most complex levels of the representational tier or the next higher tier (i.e., abstraction). While it is true that the video is sensitive to sequencing concepts, the work is done from the perspective of an expert. This is an important distinction. The expert’s perspective offers a global view of relevant concepts, the heuristics to understand relationships between concepts, and the algorithms that reveal more precise understandings of those concepts. From this vantage point, the expert sees the educational challenge as unpacking complex skills into less complex skills. Thus, it would make sense from an expert’s point of view to introduce density before discussing Archimedes’ principle, which depends on density. This approach generates a sequence of abstractions relevant to the expert, but not a sequence that the learner can necessarily construct or see as relevant. Learners typically confront a different challenge—one of building more complex understandings, as encountered in higher levels and tiers. Every graph, formula, drawing, and arrow on Khan’s Smartboard is a relevant representation (otherwise it wouldn’t be in the video); however, he unconsciously ignores the sequences of experiences that allow students to coordinate or juggle these (and other necessary) representations into concepts like density and, eventually, Archimedes’ principle. One important exception to this general observation is the “Knowledge Map” in the area of math. While the map is very important in understanding the development of a discipline, the map is not a picture of the development of understanding in individual learners. It is important to note that while the map is a hierarchical framework, the hierarchy emerges from the perspectives of experts, not learners. Similar work can be seen in the Atlas of Science Literacy [22]. Here the authors claim something more explicit: “[The] Atlas of Science Literacy is a two-volume collection of conceptual strand maps … that show how students’ understanding of the ideas and skills that lead to literacy in science, mathematics, and technology might develop from kindergarten through 12th grade” [22, p.1]. This claim is seductive. The actual path to understanding might be broadly marked out by these maps (as indicated by the authors), but this work was accomplished by experts retrospectively reflecting on how they learned science, so we must emphasize that they do not necessarily characterize how students actually build that knowledge. In similar fashion, the math Knowledge Map connects a number of relevant topics, beginning with addition and subtraction and ending with calculus, and within each topic there are exercises that test student understanding. I will underscore here and beyond that the knowledge map as well as the Atlas of Science Literacy offer an important perspective on the development of a discipline, but whether the maps lead students to authentic knowledge is not obvious; thus, we must be circumspect with how we use these maps to create interventions and assessments in the belief that students are re-creating the same understanding as experts.

B. Observation two: experience is the foundation of authentic understanding. Educators recognize the importance of creating experiences, but it is important to note that not all experiences are equal. Central to the first observation is that understanding develops first through the use and integration of our senses. Formulas for concepts such as density or pressure (d = m/v and p = mgh) belong to the representational tier and are far removed from the senses that students need to immediately coordinate and employ when confronting a new and complex juggling routine. While many elementary or middle school curricula do incorporate activities such as submerging small blocks of iron or wood to see how much water is displaced, such experiences are too remote and inaccessible to high school students trying to understand how the laws and algorithms their teachers are using encapsulate their earlier experiences. The power in laws and principles is in the fact that they summarize numerous experiences, but those insights cannot be transferred directly from one person to another—the abstraction, divorced from the numerous experiences that gave rise to it in the first place, is lost upon the student who doesn’t



have the foundation of experiences that give rise to personally constructed representations, which in turn will support the abstractions valued by educators. The task of teaching begins in carefully choosing experiences that challenge the student’s intuition about the world, and allow for numerous interactions between the student, their experiences, and the challenges they are facing. This dynamic process, which requires time and multiple opportunities to engage with the content, is necessary to develop more complex representations [21, 23]. Eventually, this kind of work can set the stage for watching and appreciating the complexity in Khan’s juggling of representations such as pressure, volume, gravity, etc., embedded in parts five and six of “Fluids.”

1. Do Khan Academy lessons ground complex ideas in sensorimotor experience? Like many accomplished science educators, Khan demonstrates his ability to coordinate numerous representations, which is analogous to observing a master juggler. Watching Khan carry out a complex performance of understanding in 15 minutes or less is also much like watching a professional musician play a piece of similar duration. In this case, audience members do not generally believe themselves capable of reproducing the same performance afterward. However, what is curious in educational contexts is that after listening to a lecture, teachers and students frequently believe that the student should be capable of performing at the same level as the teacher and with the same level of authentic understanding. But that isn’t the case, even after students spend time on questions, algorithms, charts, and graphs found on worksheets or at the end of the chapter. Just practicing within the representational tier doesn’t appear to support authentic understanding. One promising aspect of Khan Academy is the effort to develop authentic understanding through intuitive practice, again in the area of math. The technology embedded in the problems allows students to experiment with the impact of different variables that give rise to important concepts, such as standard deviation or average; however, changing the distribution of data on a two dimensional graph, for example, still requires focusing on juggling representations and not yet how the representations connect to tangible experiences. Authentic understanding requires a wide platform of experiences, which in turn provides a foundation for the representations that are the basis of student work in any school environment, virtual or real. Without the benefit of this foundation, representations practiced in school are reduced to borrowed ideas that are limited in scope and decay rapidly [1].

C. Observation three: authentic understanding is stabilized by scaffolded practice. Students who can recall the formulas they practiced using in school often struggle to employ those formulas outside the rigid contexts in which they were learned [24, 25, 26, 27, 28]. Such was the case with my students. Their knowledge of the formulas left them with an illusion of understanding, which unraveled when they confronted conceptual problems based upon the related algorithms. Formulas are important tools that mathematically represent relationships observed in nature. They often show up in demonstrations as a way of expressing complicated relationships, but they do not necessarily reveal the conceptual basis for the relationships between the variables or the numerous experiences underlying the formula, no matter where they appear—classroom whiteboards or Khan’s Smartboard. Khan’s presentations look no different from what many practiced physics teachers create for their students even though the duration may be much shorter in Khan’s case (see reference section for other, longer, examples). Demonstrations that illustrate the teacher’s understanding also reinforce the illusion of understanding [29]. The viewer watches the instructor demonstrate the outcome of years of their own practice, creating for students the afterglow of an experience in which meaning was created, but not by the student. Students who report having trouble following demonstrations also report that they quickly lose interest; thus, it is not surprising that the short attention span that Khan highlights as a universal educational problem becomes a defining feature of his presentations. Educators must address the disconnect students experience between their current understanding and the demonstration they are observing, which includes



the less obvious knowledge embedded in the instructor’s choice of objects and tools, as well as the order in which both are used and manipulated during the demonstration. One approach we used to address the disconnect mentioned above was through the kinds of problems we offered students. We found that without instruction or encouragement students would take control of problems and practice with them at home, using whatever tools and supplies were available to them. They surprised us by returning to class explaining what they had learned. While we were impressed by individual performances of understanding, we unfortunately remained vulnerable to the illusion of understanding. Initially, we succumbed to the easy belief that the students listening to these stories of success had also achieved the same level of juggling demonstrated by their peers, but they quickly punctured that illusion. A few promptly announced that they didn’t understand what their peers had said, and did not understand until they attempted the demonstration. Then, in their own words they tried to explain what they understood. In effect, authentic understanding emerged as a process of students juggling the ideas and tools on their own. The ongoing challenge educators must face is identifying demonstrations that challenge their students’ intuition and inviting them to assume control and practice juggling the relevant elements of the demonstration. Our students learned that they had to practice re-presenting their explanations so that their complex ideas could develop stability over time. In one dramatic moment, a student correctly described what would happen to the water level in the pool, but then paused, looked at the class, and publically admitted he didn’t understand what he had just said. For several minutes he had created and sustained an understanding that allowed him to see the causal connection between his experiences and representations and what would happen to the level of water in the pool. But he also realized that the understanding was temporal. Understanding does not sit in our minds like books on shelves [30]. We have to re-create understanding so that it can achieve some level of permanence. But as jugglers know, if they don’t practice there is little guarantee they can pick up the blocks (or concepts) and begin juggling again. It is useful to practice with the same problem a number of times, like a musician practicing the same piece until she is comfortable with every aspect of the score. An important dimension of practicing is that the learner can easily observe changes in performance over time. The notes become easier to play; there are fewer mistakes and greater fluidity, etc. As with the canoe problem, practice allows the student to recognize the importance of volume or how the impact of volume on the water level is different when evaluating the role of the canoe and the anvil (before it is removed from the canoe). Similarly, the mass of the anvil plays a different role (on the outcome of the water level) depending on whether it is in the canoe or in the water. Coordinating these ideas into a more complex representation is a necessary foundation for achieving new abstractions (such as Archimedes’ principle).

1. Does Khan Academy scaffold practice? There are currently a limited number of subjects taught at the Academy that offer opportunities to practice. The sole area in which practice is offered, which is particularly well developed, is mathematics; however, as pointed out earlier, practice is limited to the representational tier. While scaffolding is provided within this tier to help students unpack, for example, the meaning of median or mode, this work is executed through the use of other representations. There are no explicit connections to relevant experiences that students can use to construct for themselves authentic representations at any level. The risk, as noted earlier, is that students are left with an illusion of understanding that is fragile and highly context specific (a problem explored in observation five) and disconnected from the real world. Outside the area of math, as in the case of “Fluids” and Archimedes’ principle (parts 5 and 6), practice is not yet an option. As the Academy creates practice conditions for students, there are two important considerations. First, practice must focus on the careful choice of similar problems (like musical compositions that feature the same technical challenges, such as rhythm) that reinforce the skills used and afford greater comfort with concepts, such as volume and mass (or notes, rhythm, and intonation, as the case may be). Problems similar to each other in scope and complexity are instrumental in allowing students and teachers to consider the important role context plays in teaching and learning. Second,



practice must lead to meaningful feedback. Feedback not only helps adjust performance as a skill is stabilized, but it should also provide the platform for reaching the next more complex level of understanding.

D. Observation four: meaningful feedback is relevant and timely. All life depends on relevant feedback. This is true across a range of complexity beginning with single celled organisms looking for nutrients, to multicellular organs achieving homeostasis, to multicellular organisms attempting to survive in their niche. Relevant feedback for students and educators is generated through actions they believe will allow them to achieve a particular goal. Students need to recognize the target they are attempting to reach and be able to identify likely responses they believe will lead them to success. More importantly, they need the freedom to try out those actions. Only in the context of attempting to achieve a goal do our actions make sense, and only then can we meaningfully interpret the outcome of our actions [31, 32, 33]. While this might seem obvious, executing this in educational contexts is challenging. This task of identifying goals that students understand and that also match their current ability to juggle is not easy but necessary. A number of researchers have pointed out that lessons need to be strongly guided by goals that focus a student’s attention [7, 10, 21, 31, 33]. An effective goal provides students with the opportunity to identify promising strategies to reach the goal, which in turn creates a meaningful foundation for creating and coordinating more complex representations and abstractions. The problem with school lessons whose “goals,” for example, require students to submerge blocks of iron or copper or wood in a column of water is that for many students there is no obvious or explicit reason for the activity. Students should be asking, what is the problem for which this lesson (i.e., submerging blocks) is the solution? But students don’t ask this question because they are more accustomed to asking how they should participate. Thus, the likelihood of recognizing the need to ask why Khan is making the choices he makes during his video presentation is even more remote. It may not be clear to a student what goals are guiding Khan’s actions. At times Khan’s indecision is obvious in that he will start a drawing and then change his mind and start something different. Why? What problem was he facing that required a change in teaching strategy? This style of teaching is not unique to Khan, as the teaching environment is dynamic. Teachers often change their minds and approaches as they become involved in the teaching moment. The energy and creativity that often characterize these moments easily support the illusion of understanding, as the learner experiences the theater of the teacher’s goals successfully directing the teacher’s actions. Unfortunately, the learner is often too passive during the whole experience. More dramatic is the online experience, which creates a new challenge in that the “teaching moment” is not really shared with the audience or influenced by it. As the dynamic nature of teaching becomes codified, the video risks becoming a permanent reminder of the distance between the instructor and student, further reinforcing the need to passively observe. As passive observers, students eventually let go of asking why any demonstration (of the teacher’s expertise) is unfolding as it is, because students know they will not be answered. To confront this challenge, instructors need to consider how to set up problems that have clear actionable goals, and that action on the part of students generates feedback that is meaningful to them, thus requiring less feedback from the instructor. In regard to the canoe problem, one should note that nature, rather than the instructor, provides the feedback. Students don’t need to ask the instructor what the answer is, although they often will. In these cases the instructor can direct students to find the answer on their own in the lab or at home. Finding problems like the canoe problem also provides students with the opportunity to consider the importance of their own work in creating new knowledge. These kinds of problems can provide high intrinsic motivation similar to the state of “flow” in which the sense of time compresses, focus and concentration increases, and distractions disappear [34]. The experience is also similar to what individuals experience when playing certain video games [35]. In contrast, extrinsic motivation is what educators use when students don’t respond in ways intended by



the lesson. This situation can result from a number of problems: students don’t understand the goal, don’t recognize the appropriate action, don’t have the ability to execute the required action, or can’t interpret feedback generated by their action (or given by the instructor). Extrinsic feedback often depends on rewards (such as money, grades, extra time at recess, less homework, etc.) that are unrelated to the goal of making students want or need to create more complex representations or abstractions. Extrinsic feedback is less powerful at generating more complex understandings, as observed by drivers who get information about how to drive from the person in the back seat. The risk of a system based upon extrinsic rewards is that it easily contributes to the illusion of understanding.

1. What does feedback look like at Khan Academy? Like many school environments, Khan Academy depends on extrinsic feedback, in which students receive “badges” and “energy points” after completing lessons. If they wish, students can share with selected audiences their progress through the lessons. Second, students can also ask questions in a blog format, to which other members of the Academy can respond. It’s not clear that answers from the community are any more helpful than Khan’s videos or my students’ explanations of their understanding to their peers. Both forms of feedback fall short of what counts as meaningful and timely feedback. Badges do not contribute to developing an understanding of fluid dynamics. I, for example, earned three badges while watching “Fluids”; however, those badges are not predictors of how successful I would be with the canoe problem or variations of the canoe problem or any similar problem in a new context (e.g., applying Archimedes’ principle with gasses instead of fluids). The third form of feedback, developed as part of the math Knowledge Map at Khan Academy, is embedded in the available practice problems. The software evaluates the student’s answer, and if the answer is incorrect the software offers one or more layers of suggestions that guide students to the right answer. Alternatively, the student can choose to unpack the layers without committing to an answer, and teachers can follow student progress. This form of feedback resembles coaching, in which students are provided with specific support in response to a problem in executing a skill. While coaching can help students achieve greater proficiency with skills or practice more effectively, the more basic challenge of developing authentic knowledge remains. If the short-term goal is for students to get better at solving quadratic equations, then they may never discover the problem for which the quadratic equation was the solution in the first place. Developing a richer palette of sensorimotor experiences that are tightly linked to representations still needs to occur before teachers become overly focused on training students to solve quadratic equations. Otherwise the students’ work becomes no more important or useful than a parlor trick restricted to very specific contexts. This challenge is already one that teachers find difficult to meet in classrooms and, expectedly, much more challenging in an online environment. One possible solution is letting the video set up problems that students can execute on their own and in turn allow them to judge the impact of their actions. In our case, the problems we posed students provided this opportunity. In turn, course instructors began to play a more marginal role in providing answers because nature could offer immediate feedback. Given this kind of experience, our students were better able to adapt to situations where nature could not conveniently respond to their questions. In such cases, students were better prepared to consider the answers of experts, and to compare them to their own. In situations where nature cannot provide an answer, current technology can collect student responses and allow them to compare and contrast answers through graphs or tables that are continually updated. A distribution of student answers has the potential to encourage students to take a deeper look at the original challenge and the answers.

E. Observation five: authentic understanding is context-sensitive. While practice provides opportunities to challenge the stability of a complex idea, practicing in a variety of contexts challenges the robustness of one’s grasp of the principle underlying any particular problem.



Thus, initially the juggler may be comfortable juggling balls in front of friends, but the same act on stage in front of strangers can create a different experience and result. In this case, changing contexts where competence is demonstrated will challenge how easily ideas—or balls—are coordinated. While there is value in establishing this kind of stability with a problem, the greater challenge is establishing stability with the principle behind related but novel problems [24, 28, 36, 37]. The canoe problem offered students the chance to practice and demonstrate their understanding through different approaches. Inviting students to demonstrate their understanding through writing, oral presentations and, in turn, using their understanding with their students created stability with this one problem, but not necessarily with Archimedes’ principle. Developing comfort with Archimedes’ principle requires changing the problem in slight ways to provide students the opportunity to challenge their understanding. For example, changing the context slightly by asking what would happen to the level of water if the anvil were replaced with a large piece of foam or a piece of balsa wood or a smaller version of the anvil allows students to focus on particular elements of a problem growing in familiarity. Working with the original problem provides a framework that remains constant over time so students can control elements within the problem to judge the importance and role the variables (such as volume, mass, floating, sinking, etc.) play. After a trial period during which students explore the impact of the variables in the problem, the curriculum can invite students to consider new and unfamiliar contexts where the principle is still operating. Consider for example a planet like Mars where the atmosphere is predominantly carbon dioxide. If an astronaut fills her birthday balloon with oxygen and releases it, will the balloon fall, float in place, or climb? Although this problem requires an understanding of chemistry, the example illustrates the kind of opportunity that educators need to create in which students can consider how Archimedes’ principle is operating. Problems like this already exist in numerous textbooks [38, 39]; however, to make effective use of such problems, careful and purposeful attention to earlier experiences and observations is still necessary. The experience students had with the canoe problem allowed them to begin considering more modest changes in context. Asking students what variables they would change, and to consider the impact of those changes, invited students to take ownership of the problem. The shift in responsibility for the problem as well as the answer offers a unique opportunity to explore the role of feedback, in that students are now looking for answers to their own questions.

1. Is the curriculum at Khan Academy context sensitive? All academies of learning, including Khan Academy, face the challenge of how to vary the goals in meaningful ways to allow students to evaluate how robust their understanding is. The virtual environment provides the unique opportunity to layer numerous dynamic contexts to create a foundation of understanding that leads to multiple applications of the principles being studied. Much of the material necessary is already available on the web as products of teacher inventiveness to help students understand the plethora of concepts now required for state testing. Building on this work while maintaining attention to all five observations is a challenge not just for Kahn but for all educators. This issue is especially relevant as universities try to reach larger audiences through Massive Open Online Courses (MOOCs). The courses organized through initiatives such as edX or Coursera seek to distribute “understanding” to hundreds of thousands of students. But new educational environments such as these are not immune to the observations noted as necessary for authentic learning. For example, the common use of multiple-choice exams and peer evaluations falls far short of creating effective feedback. While the outcome of the MOOC experiment is still hard to gauge, without a meaningful framework of learning, they are unlikely to succeed, a situation already noted by the press [40, 41, 42]. Currently, completion rates hover around 10 percent for most MOOCS [43], which may be a signal that students are aware of the illusion of understanding and are looking elsewhere for a meaningful learning experience. The tools of technology do offer educators and students a future that is both promising and complex. The revolution in education that Bill Gates describes when referring to Khan Academy is based upon the



power of magnification that technology offers; but a technological platform that claims to “provide a free world-class education for anyone anywhere,” as Khan Academy promises, still requires understanding how students learn as well as how to effectively evaluate the impact of instruction at the individual level [44]. Gates’ experience in technology is noteworthy, but the thousands of hours of practice and feedback that unfolded as he pursued his goal of understanding computers is unique [45]; thus, educators must be careful with how individual experiences and perspectives such as his are leveraged when making educational decisions, especially decisions that will impact the millions Gates wants to help. While he does recognize that caution is necessary and that placing a video in front of a captive audience is not a universal solution for educators and students [46], Gates’ background does not reasonably support the level of analysis required, nor the ability to identify the most promising solutions, without his also having dedicated thousands of hours to studying teaching and learning. Thus, the temptation is to quickly overreach with the use of technology without seriously considering the implications for students and teachers. For example, this year Carlos Slim, the Mexican telecom entrepreneur, agreed to provide the resources to translate Khan academy into Spanish, but whether Khan’s representations and personal experiences will cross the cultural divide will be a real test of how context sensitive the videos are. I’m reminded of a lecture I gave years ago in Quebec, Canada, where I introduced the canoe problem. Several individuals asked me what an anvil was, and I referred to the Looney Tunes episode in which Wile E. Coyote tries to drop an anvil on the Road Runner [47]. The audience feedback was immediate. Blank stares informed me I was either outdated or culturally out of touch, and I needed to try again.

III. CONCLUSION—THE ILLUSION OF UNDERSTANDING Unfortunately, there is no way that this paper, which has taken much more than 15 minutes to write and read, will convince anyone that they now understand what we had to observe repeatedly, practice constantly, and evaluate through feedback from our students over years. Educators must apply these observations every day, just as musicians and artists practice every day to become masters of their crafts. The observations noted in this paper will still be easily discounted by strong intuitions about teaching and learning developed over years as a result of surviving an educational system that is and has been mostly didactic. Factors such as the lack of time or resources will easily undermine the need to carefully scaffold experiences that allow students to build more complex understandings. Those factors undermine the importance of developing student experiences as a foundation for new representations or a new coordination of familiar representations. They weaken our resolve to encourage students to practice or seek new contexts in which student understandings can be challenged. And perhaps the most pernicious outcome of a pedagogy based on the lack of time, resources, or student feedback is that students become dependent on their instructor’s feedback to judge their success, and instructors miss the opportunity to recognize and develop lesson goals in which student action becomes a source of feedback that students use to judge their success in learning. Our students had mastered numerous tests of knowledge over many years and demonstrated their skill in using formulas learned in numerous carefully constructed contexts, such as books, problem sets, and tests. This work did not provide them with the conceptual understanding underlying a basic principle in the sciences. Perpetuating the illusion of understanding with students is an easy trap for instructors to fall into whenever they assume that they are responsible for the entire feedback loop. Instructors need to focus on carefully chosen goals that challenge a student’s intuition, and, in turn, allows that student to complete the feedback loop largely on their own. This operation will of course challenge the instructor’s intuition about what teaching means. Khan, like many educators, falls victim to this illusion in his teaching, partly because he doesn’t get the kind of feedback that forces educators to challenge their assumptions about learning. Bill Gates and Sal Khan are not alone in using their experiences as the lens through which they understand and respond to educational problems. Using experiences for decision-making is natural and to be expected when creating understandings (a point made repeatedly throughout this article). However, despite the “naturalness” of



such actions, the risk we face collectively is of making educational decisions that ignore the last hundred years of cognitive science and emerge predominately from our intuition, a strategy that is usually ineffective [48]. Shortening demonstrations to match an ever-shrinking attention span or creating a plethora of on-demand videos completely misses the shift in perspective necessary to expose the illusion and allow teachers and students to make the adjustments necessary to support and develop authentic understanding.

IV. ACKNOWLEDGEMENTS I wish to recognize my colleagues who read and commented on this piece as well as encouraged me to write it: Mike Connell, Eugenia Garduno, Irwin Shapiro, Bruce Gregory, Jeanne Gerlach, and Ken Williford. I also want to recognize my students who read drafts of this paper and interacted with me over time on the observations noted in the article. The input of all these individuals led to a stronger, more focused framework for the challenging work that all educators face. I am also thankful to Gary Miller, who saw a purpose and need to make explicit a framework for challenging the way we think about education and the way it unfolds in an online environment.

V. ABOUT THE AUTHORS Marc Schwartz is Professor of Mind, Brain and Education at the University of Texas at Arlington. He is also director of the Southwest Center for Mind, Brain and Education at UTA, which seeks to identify and support promising research agendas at the intersection of neuroscience and cognitive science to inform (and be informed by) educational practice and leadership. Dr. Schwartz is a charter member of the International Mind, Brain and Education Society (IMBES), past vice president, and current president. The mission of IMBES is to facilitate cross-cultural collaboration in biology, education and the cognitive and developmental sciences. Dr. Schwartz is also an Associate Researcher in the Science Education Department at the Harvard-Smithsonian Center for Astrophysics (CfA).

VI. REFERENCES 1. Schwartz, M.S. and Fischer, K.W. Building vs. Borrowing: The Challenge of Actively

Constructing Ideas. Liberal Education (Summer): 22–29 (2003). 2. Schwartz, M.S, Shapiro, I., & Gregory, B. Confronting the need for conceptual change in pre-

service science education. Higher Education Studies. 3(5), 11-28 (2013). 3. Elgin, C. From Knowledge to Understanding. In: Hetherington, S. (Ed.), Epistemology Futures,

Oxford: Clarendon, 2006. 4. Bransford, J.D., Brown, A.L., and Cocking, R.R., eds. How People Learn: Brain, Mind,

Experience and School. Washington, DC: National Academy Press, 2000. 5. Elman, J., Bates, E.A. Johnson, M.H., Karmiloff-Smith, A., Parisi, D., and Plunkett, K.

Rethinking Innateness: A Connectionist Perspective on Development. Cambridge, MA: MIT Press, 1996.

6. Klahr, D. Beyond Piaget: A Perspective From Studies of Children’s Problem Solving Abilities. In: Slater, A. and Quinn, P. (Eds.), Refreshing Developmental Psychology: Beyond the Classic Studies, London: Sage Publishers, in press.

7. Perkins, D. Smart Schools: From Training Memories to Educating Minds. New York: The Free Press, 1992.

8. Piaget, J. The Equilibration of Cognitive Structures: The Central Problem of Intellectual Development. Trans. by T.T. Brown and K.J. Thampy. Chicago, IL: The University of Chicago Press, 1985.

9. Posner, M. and Rothbart, M. Educating the Human Brain, Human Brain Development Series. Washington, DC: American Psychological Association, 2006.



10. Powers, W. Making Sense of Behavior: The Meaning of Control. New Canaan, CT: Benchmark Publications Inc., 1998.

11. Siegler, R.S. Emerging Minds: The Process of Change in Children’s Thinking. New York: Oxford University Press, 1996.

12. Schank, R. Teaching Minds: How Cognitive Science Can Save Our Schools. New York: Teachers College Press, 2011.

13. Schwartz, M.S. Cognitive Development and Learning: Analyzing the Building of Skills in Classrooms. Mind, Brain and Education 3(4): 198–208 (2009).

14. Case, R. The Structure and Process of Intellectual Development. International Journal of Psychology 22: 571–607 (1987).

15. Commons, M.L., Trudeau, E.J., Stein, S.A., Richards, S.A., and Krause, S.R. Hierarchical Complexity of Tasks Shows the Existence of Developmental Stages. Developmental Review 18: 237–278 (1998).

16. Fischer, K.W. and Bidell, T.R. Dynamic Development of Psychological Structures in Action and Thought. In: Damon, W. and Lerner, R.M. (Eds.), Handbook of Child Psychology: Theoretical Models of Human Development. New York: John Wiley & Sons, 1998.

17. Piaget, J. Piaget's Theory. In: Mussen, P.M. (Ed.), Handbook of Child Psychology. New York: Wiley, 1983.

18. van Geert, P. A Dynamic Systems Model of Basic Developmental Mechanisms: Piaget, Vygotsky, and Beyond. Child Development 105: 634–677 (1998).

19. Dawson-Tunik, T.L., Commons, M., Wilson, M., and Fischer, K.W. The Shape of Development. European Journal of Developmental Psychology 2(2): 163–195 (2005).

20. The word “representation,” as used here, focuses on the act of “re-presenting” as in re-enacting or re-creating understanding, and also intentionally underscores the ability to reproduce the same understanding at a later time.

21. Schwartz, M.S. and Sadler, P. Empowerment in Science Curriculum Development: A Microdevelopmental Approach. International Journal of Science Education 29(18): 987–1,017 (2007).

22. AAAS. Atlas of Science Literacy: Project 2061. Washington, DC: American Association for the Advancement of Science, 2001.

23. Doucerain, M. and Schwartz, M.S. Analyzing Learning About Conservation of Matter in Students While Adapting to the Needs of a School. Mind, Brain and Education 4(3): 112–124 (2010).

24. Fischer, K.W., Bullock, D., Rotenberg, E.J., and Raya, P. The Dynamics of Competence: How Context Contributes Directly to Skill. In: Wozniak, R. and Fischer, K.W. (Eds.), Development in Context: Acting and Thinking in Specific Environments. Hillsdale, NJ: Erlbaum, 1993.

25. Lave, J. Word Problems: A Microcosm of Theories of Learning. In: Light, P. and Butterworth, G. (Eds.), Context and Cognition: Ways of Learning and Knowing. Hillsdale, NJ: Lawrence Erlbaum Associates, 1993.

26. Nardi, B. Studying Context: A Comparison of Activity Theory, Situated Action Models, and Distributed Cognition. In: Nardi, B. (Ed.), Context and Consciousness: Activity Theory and Human-Computer Interaction. Cambridge, MA: MIT Press, 1996.

27. Wertsch, J.V., Minick, N., and Arms, F.J. The Creation of Context in Joint Problem Solving. In: Rogoff, B. and Lave, J. (Eds.), Everyday Cognition: Its Development in Social Context. Cambridge, MA: Harvard University Press, 1984.

28. De Corte, E. Constructive, Self-Regulated, Situated, and Collaborative Learning: An Approach for the Acquisition of Adaptive Competence. Journal of Education 192(2): 33–47 (2012).

29. Tai, R.H. and Sadler, P.S. High School Chemistry Instructional Practices and Their Association With College Chemistry Grades. Journal of Chemical Education 84(6): 1,040–1,046 (2007).

30. Fuster, J.M. Cortex and Memory: Emergence of a New Paradigm. Journal of Cognitive Neuroscience 21(11): 2,047–2,072 (2009).

31. Langer, E.J. The Power of Mindful Learning. Reading, MA: Addison-Wesley, 1997.



32. Powers, W. Perceptual Control Theory. Hawthorne, NY: Aldine DeGruyter, 1973. 33. Schank, R. and Cleary, C. Engines for Education. Hillsdale, NJ: Lawrence Erlbaum Associates,

1995. 34. Csikszentmihalyi, M. Flow: The Psychology of Optimal Experience. New York: Harper & Collins,

1990. 35. Schwartz, M.S. and Sadler, P.S. Goals and Technology Education: The Example of Design

Challenges. Paper read at Proceedings of the Second AAAS Research in Technology Education Conference, in Washington, DC (2001).

36. Granott, N., Fischer, K.W., and Parziale, J. Bridging to the Unknown: A Transition Mechanism in Learning and Problem-Solving. In: Granott, N. and Parziale, J. (Eds.), Microdevelopment: Transition Processes in Development and Learning. Cambridge, UK: Cambridge University Press, 2002.

37. Griffin, M.M. You Can't Get There From Here: Situated Learning, Transfer, and Map Skills. Contemporary Educational Psychology 20(1): 65–87 (1995).

38. Epstein, L. and Hewitt, P.G. Thinking Physics: Practical Lessons in Critical Thinking. San Francisco: Insight Press, 1981.

39. Hewitt, P.G. Conceptual Physics. Boston, MA: Addison-Wesley Publishing, 2010. 40. Rees, J. The MOOC Racket. Slate (2013). 41. Wilson, E. The Huge Growth of MOOCs Threatens America's Great University System. Alternet

(2013). http://www.alternet.org/education/who-profits-moocs?akid=10609.314356.jndowS&rd=1&src=newsletter858873&t=7.

42. Koh, A. Weekend Reading: The MOOC Catchup Edition. The Chronicle of Higher Education (2013). http://chronicle.com/blogs/profhacker/weekend-reading-the-mooc-catchup-edition/50111.

43. Jordan, K. Synthesising MOOC Completion Rates. MoocMoocher (blog) (2013). http://moocmoocher.wordpress.com/2013/02/13/synthesising-mooc-completion-rates/

44. Khan Academy. About. (2013). https://www.khanacademy.org/about 45. Gladwell, M. Outliers: The Story of Success. New York: Little, Brown and Company, 2008. 46. Gates, B. and Microsoft. Innovation & Opportunity: The Contribution of Computing to Improving

Our World. In Virtual Faculty Summit. Redmond: Microsoft, 2013. 47. Wikipedia. Wile E. Coyote and the Road Runner. Wikipedia Foundation, Inc. (2013).

http://en.wikipedia.org/wiki/Wile_E._Coyote_and_The_Road_Runner. 48. Kahneman, D. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.

Conceptual Framework of Blended Professional Development for Mathematics Teachers


CONCEPTUAL FRAMEWORK OF BLENDED PROFESSIONAL DEVELOPMENT FOR MATHEMATICS TEACHERS

Hea-Jin Lee Ohio State University-Lima ABSTRACT The study presents a conceptual framework of a blended professional development program for teacher success in learning a subject matter and enhancing instructional practice. Twenty-nine middle school teachers (22 first-year and 7 continuing teachers) participated in a year-long, blended professional development (PD) program. Effects of the blended PD program were measured by analyzing the content of participants’ virtual interactions; i.e., 1,149 online threaded discussion messages. The continuing teachers were more actively engaged in the online community of practice than the first-year teachers. The types of structured assignments and learning time influenced the level of participation but did not make a significant impact on the interaction content. Most virtual interactions involved sharing instructional ideas based on personal experiences. Results showed participants’ teaching practices changed toward more student-centered lessons, encouraging verbal engagement, using various questioning strategies, and using collaborative group work while employing a wider variety of resources. KEYWORDS Blended professional development, mathematics teacher education, content of virtual interactions

I. INTRODUCTION There are an increasing number of online and blended professional development programs [1]. However, considering the growing number, the effectiveness of the professional development (PD) program, including systemic aids, PD tasks, and evaluation strategies, remain vague. This study aims to understand how to best support teachers learning through an online system in a blended PD program. This study is based on a year-long professional development program for middle school mathematics teachers. The purpose of the program was to maximize teacher learning in mathematics, improve instructional practice, and foster productive professional interactions among mathematics teachers. This paper was guided by the research question: how do blended professional learning opportunities impact the professional development of mathematics teachers?

II. CONCEPTUAL FRAMEWORK The features of an effective blended professional development program can be designed to facilitate professional development for teachers. The PD program investigated provided a blended learning option that has the potential to increase sense of community [2] and professional learning.



Figure 1 depicts a conceptual framework for a professional development program involving contextualized practice and active construction of knowledge.

Since Dewey, the social nature of all human learning and the role of communication skills and abilities in the human development process are the most often discussed concepts in higher education [3, 4]. Learning in communities of practice relies on successful communication between individuals, and learning occurs as interaction among practitioners takes place [4, 5]. As far as communities of practice are concerned, “there is no one on the stage who is the knowledge source, but all individuals have an equal right to share their experience, and their stories are valuable contributions to the community” [4, para. 3]. It was recognized that a sense of community was critical for sustaining teachers’ interests and that face-to-face meetings were vital in expanding and supporting professional learning communities. Participants in the program came from various backgrounds and diverse experiences, which made bonding a challenge. The assumption was that if teachers could form a community in supporting and learning from each other, then they would be more likely to succeed with their learning. Salmon [6] presents a five-stage model for online courses with roles for participants and instructor:

1. Access and Motivation 2. Socialization 3. Information Exchange 4. Knowledge Construction 5. Development

Salmon’s model served as a basis for designing a blended PD program for this study, for selecting the PD topics, and for organizing the sequence of the PD topics. While Salmon’s model was designed for 100% online courses and general populations, the conceptual framework of this study is for a blended PD program for practicing teachers, and the five-stage model was modified:

1. Motivation and Socialization 2. Information Exchange 3. Knowledge Construction 4. Development 5. Contextualized Practice

In a blended PD program, the Motivation and Socialization stage can be approached either face-to-face or virtually. If face-to-face contact is utilized, the first stage can be accomplished in a shorter amount of time but is as equally effective as virtual contact. During this stage, participants should gain information about the PD program/courses, get familiar with the PD system, learn the technology to be used for the online

Figure 1. Conceptual Framework of the Professional Development Program



sessions, begin to establish a network with other participants and instructors, and become motivated to participate in the PD. Deep understanding or implementation of learning is not expected in the Motivation and Socialization stage. Instead, the PD instructors present course guidelines in the Information Exchange stage and personalize course activities while encouraging participants to exchange. In the Knowledge Construction and Development stages, participants collaborate to establish common understanding of mathematical concepts and instructional strategies taught in the PD and reflect on their learning. Rovai [7] found a “positive significant relationship between a sense of community and cognitive learning” (p. 328). Mere interaction among participants, however, does not guarantee meaningful cognitive engagement or facilitate meaningful learning and understanding. According to Walker and Chappell [8], when we discuss student learning, our question should not be whether students can or cannot achieve mathematical skills but whether students will realize maximum success in mathematics. This view suggests an instructional approach that encourages students to communicate, problem solve, and engage in learning mathematics conceptually [9, 10] and should guide PD program activities. The participating teachers in this study were expected to “build knowledge through inquiry-based collaborative interaction with other teachers and become co-learners and act as facilitators” [11, p.4]. Professional development programs should provide opportunities for participating teachers to “learn mathematics around specific content and teaching situations that may arise in practice” [9, p. 56], as well as provide opportunities for teachers to implement/practice their learning in their own classrooms, which is called the Contextualized Practice stage in the study. The key concerns, therefore, are ensuring peer exchange of ideas and information; the creation and assimilation of knowledge [12]; and providing a climate of accelerated change in which participants access up-to-date knowledge and apply new skills in changing circumstances [13]. The community of practice should not simply be a forum for the exchange of information but should progress in response to the diverse needs of learners and the communities in which they work [14].

III. PROFESSIONAL DEVELOPMENT PROGRAM OF THE STUDY The professional development program of this study, Teaching Algebra in Context, Community, and Connections (TACCC), was designed to improve student mathematics knowledge and attitude by (1) increasing teacher content knowledge and (2) improving instructional skills to teach mathematics concepts in an environment rich in context, community, and connections. Figure 2 depicts the blended PD program of the study.

Figure 2. PD Model of the Study

The major activities of the project consisted of three components: face-to-face workshop courses, web-based learning sessions through virtual interactions, and classroom implementation. As Blank, de las Alas, and Smith [15] also reported, the PD developers also believe that high quality, in-service PD should include content focus, active learning by teachers, collective participation, coherence with curriculum/standards, and sufficient time. The assumption was that if teachers could form a community in supporting and learning from each other, then they would be more likely to succeed with their professional learning.



A. Face-to-face Workshops The PD program of the study was a year-long program with three graduate-level workshop courses. Face-to-face meetings were held once a month over nine months. There were groups of 2–4 teachers from the same school district with a total of 29 participants. This number of participants provided teachers with opportunities to develop a support system and work collaboratively, even while their different backgrounds and diverse experiences made bonding a challenge. In order to facilitate the active learning of these teachers, face-to-face workshops were conducted through discussions, collaborative group work, hands-on activities, problem-solving opportunities, and presentations by participants. Project assignments included discussions of readings, reflection on workshop activities, solving mathematical problems using invented methods, and developing mathematical tasks and analyzing their solutions.

B. Web-based Learning through Virtual Interactions Virtual interactions occurred between monthly face-to-face meetings using Desire2Learn. Asynchronous communication tools (e-mail and threaded discussion boards) were used not only to allow participants to contribute at their convenience, but also to give them time to implement and reflect on their learning and practice. The objective of online activities in the TACCC blending learning system was not to replace face-to-face instruction but to enhance the face-to-face learning experience. That is, the online activities/assignments aligned with the face-to-face workshop agenda and were scheduled to align with overall TACCC PD objectives. Participants had the opportunity to experience the first two stages of the modified Salmon model [6] (Motivation and Socialization as well as Information Exchange) before participating in online activities during the first face-to-face meeting and communicating via e-mail. Regarding the online activities, assignment 1 consisted of reading reflection and discussion and assignment 2 focused on discussion of videotaped best practices. Assignments 1 and 2 focused on the Knowledge Construction stage; assignments 3, 4, and 5 (implementing a discovery-based lesson) targeted the Development and Contextualized Practice stages. The five structured online assignments were given during the academic months. Teachers were required to post an initial message responding to the prompt for each online assignment. In addition, they were expected to participate in threaded discussions, providing comments on other teachers’ messages—at least five per assignment. During the summer, they participated in face-to-face meetings for one week, and no online activities were given at that time.

C. Classroom Implementation To make a difference in classrooms, PD programs must build in time for teachers to implement new practices between PD meetings either face-to-face or virtual. Effective practice is developed best through ongoing support of teachers in real classroom settings. Between monthly face-to-face workshop meetings in the study, participants were expected to implement ideas learned from the TACCC PD in their own classrooms. They were also encouraged to share ideas from the program with other teachers in their buildings and districts. In addition, a project instructor visited each participant’s classroom and observed their teaching or co-taught one or two lessons with the participant. Each face-to-face meeting started by discussing and sharing the outcomes of the newly implemented instructional methods and student responses/learning. Participating teachers were also required to share their lesson plans and reflections with the rest of the PD participants during the online portion of the program.

IV. METHODOLOGY Measuring the effectiveness of professional development for mathematics teachers has been undertaken with various foci. As discussed earlier in the conceptual framework section, teacher success can be achieved through continuous effort in constructing knowledge and implementing practice under the support of communities of practice. For the purpose of this study, the focus was on the teachers’



construction of knowledge and learning processes, as observed in virtual interactions in the online community of practice.

A. Data Collection Twenty-nine middle school teachers participated in TACCC. Twenty-two teachers were new TACCC participants, and seven teachers had participated in an earlier TACCC PD program. The TACCC PD courses were offered in three quarters. Since the first quarter was devoted to refreshing teachers’ understanding of algebra concepts and to exploring the use of algebraic concepts across the grade levels in middle school, the study organizers assumed that the seven returning teachers received this instruction from their previous PD sessions. Therefore, the seven returning teachers joined the PD program in the summer quarter, after the first year teachers completed the first course. The second and third courses focused on both understanding of algebra concepts and teaching algebra in context, community, and connections. Between face-to-face workshop meetings, teachers posted messages in response to a given discussion prompt and communicated with each other using asynchronous online discussion posting. Assigned prompts/topics and total numbers of posted messages are summarized in table 1.

Assignment Topic Purpose Total no. of messages

1 Reading reflection and discussion Knowledge Construction 237 2 Discussion on videotaped best practice

lessons Knowledge Construction 204

3 Implementing a discovery-based lesson Development & Contextualized Practice

166


260


282

Table 1. Virtual Interaction Data

B. Data Analysis The discussion threads in each topic were used as the unit of data collection and analysis. All of the postings (1,149 messages) among the 29 participants were analyzed and coded using the criteria described in figure 3. Three coders were trained and checked for inter-rater reliability at the end of training by using a sample set. Assignment 1 postings (237 messages) were coded by all three coders and checked for inter-rater reliability. For assignments 2, 3, 4, and 5, each coder was assigned a series of threads to code. The coders and researcher had regular meetings to discuss issues and questions. Figure 3 depicts the subcategories of each variable and the data analysis structure.



Figure 3. Data Analysis Structure

The content analysis codes were founded on Gareis and Nussbaum-Beach [16], Interstate New Teacher Assessment and Support Consortium [17], and the Joint Committee on Standards for Educational Evaluation [18]. The codes were finalized after two initial analyses. Gareis and Nussbaum-Beach [16] used six categories: Acknowledgement, Guided Advice, Sharing Experience, Specific Questions/Seeking Specific Information, Explicit Issue/Problem, and Reflection/Professional Growth. In addition to these six categories, TACCC added three content analysis codes: simple agreement, instructional ideas, and students. In order to examine the influence of the PD course materials, the instructional ideas code was added to differentiate from the Sharing Experiences category. The students code was added because TACCC participants were classroom teachers, and issues/problems related to students also received attention from project participants. Table 2 provides sample messages for each code. Codes Sample messages Reflection “I should be more forth coming making sure they hold the spaghetti towards the end

of the spaghetti and maybe they would have achieved better results.” Sharing Experience

“I did this activity with my Algebra class. I found this lesson through the Math Portal site. The lesson took two class periods, which are about 42 minutes in length.”

Issue/Problem “…but I always have a few kids who just write one or two words…” Question/ Suggestion

“With all of the variables that prevented a linear graph, was it worth it or would you use a different route?”

Instructional Ideas

“Geometry Made Simple (High School Edition), Frank Schaffer Publications—This is where I found the Logic 101 questions. www.printable-puzzles.com—This is where I found the logic puzzle. This is a great website with a lot of different types of puzzles (crosswords, sudoku, cryptograms, word searches, etc.).”

Students “They always want to help so and so add to their answer—no one ever tells another student, 'No, you're wrong.'”

Guided Advice “I covered my box with metallic Contact paper, which by the way is dry erase. I then used a piece of sparkle craft foam for the screen and white craft foam for the dial. Lastly, I used a pair of old school (large) headphones for the top. Easy and durable!”

Simple Agreement “I agree that the number line is a great way to get the year started.” Acknowledgement “I am interested in seeing this in action. It does sound like a good activity.”

Table 2. Sample Messages

http://www.printable-puzzles.com/



V. RESULTS This study investigated the learning process of middle school mathematics teachers by measuring their knowledge construction and learning implementation within their own classrooms and by analyzing project participants’ virtual interactions. The findings were summarized in three sections: the level of participation in relation to the other variables, the content of interactions, and the relationship between discussion content and associated assignment topics.

A. Level of Participation Figure 4 compares the level of participation in each assignment by calculating the average number of messages posted per assignment. The first-year teachers were more engaged in the Knowledge Construction stage assignments (assignments 1 and 2) than the Development and Contextualized Practice stage assignments (assignments 3, 4, and 5). Assignments 3, 4, and 5 required creation and implementation of discovery-based lessons as well as critique of peer teachers’ work. The study could not directly compare first-year teachers' engagement to the engagement of continuing teachers for the first three assignments because the continuing teachers only joined the program in the summer after the first set of online assignments had been completed.

For assignments 4 and 5, the level of engagement was much higher for the continuing teachers; they interacted with an average of 12.4 messages per assignment, while the first-year teachers posted an average of 8.9 messages per assignment. Every continuing teacher posted more than 8 messages for each assignment, while approximately 10 of the 22 first-year teachers posted more than 8 messages per assignment. Both groups exceeded the minimum requirement of five messages per assignment. The TACCC project team expected more frequent and active participation in the Development and Practice stage for assignments 3, 4, and 5 because teachers should have built a certain level of an online community of practice by then. In addition, since assignments 3, 4, and 5 were learner controlled activities, the project team anticipated internally motivated participants with more ideas to share in the Development and Practice assignments than in the Knowledge Construction assignments. However, the first-year teachers participated less in the Development and Practice assignments than they did in the first two assignments. However, a positive finding was that the first-year teachers increased their level of engagement specifically in the Development and Practice assignments from 7.5 to 8.1 messages per teacher in assignments 3 and 4 respectively before reaching 8.6 messages per teacher in assignments 5. The continuing teachers’ participation was much higher than the first-year teachers in the Development and Practice assignments, possibly due to their comfort level with the web-based learning support system and their greater appreciation of virtual communication with other professionals.

B. Content of Interactions Several common concerns and topics appeared in participants’ virtual interactions, regardless of the

Figure 4. Average No. of Messages per Assignment



intended purpose of the assignments or the level of discussion, both in initial and response postings. Figure 5 summarizes the frequency of topics that occurred in participants’ interactions.

Figure 5. Content of Interactions More than 50% of participants’ initial messages were devoted to sharing their experiences, and 35% of messages maintained the topic of the initial postings. Sharing experiences remained the most popular response topic (50%) followed by acknowledgement and reflection, and questions/suggestions. Teachers’ virtual interactions (online postings) were mostly focused on how to work with certain math concepts, sharing their personal experiences, and asking questions concerning instructional delivery. Specifically, most discussions touched on how to relate math concepts to the real world; how well a specific instructional strategy works; games and websites that could help their students with learning concepts; creative ways to demonstrate concepts to their students; the most effective way to use manipulatives in the classroom; how to allow for more student-centered time in the classroom; the concept of mistakes in math and how they may be turned into mini-lessons; connections between different concepts covered and standards that needed to be addressed; the use of vocabulary as a way to help with math communication; and how to overcome time constraints to allow participants to teach the “why” of math as much as they would like to. Most of these topics were face-to-face workshop agenda items and project goals. Teachers also shared struggles they personally faced with math and how they have used those experiences to help their students. Participants discussed personal connections from their own learning in school and also things they still want to learn in order to ensure students have better experiences than they did. They also discussed the common fear of not knowing how to explain the “why” of math concepts to students, which was the most important disposition advocated by the TACCC program. Participating in a professional development program seemed to help these teachers better understand students’ feelings and dispositions. There was also a great deal of discussion about how important it is for a teacher to be able to continue to learn and make changes in the classroom in small increments to make it both easier and more feasible. Participants started to value a community of practice even more after participating in virtual interactions with other teachers. They expressed how important it is to find time to talk to and work with colleagues.

C. Relationship between the Content of Interactions and Assignment Topics per Discussion Level



Figures 6a and 6b show the relationship between the content of interactions and the assignment topics within the level of discussions for both initial and response postings. Reading Reflection (assignment 1) and Video Critique (assignment 2) were tasks to support teacher construction of knowledge. The Discovery-Based Lesson Plan (LP) (assignments 3, 4, and 5) aimed to enhance teachers’ classroom practice.

An average of 25% of teacher interactions consisted of Reflection and 51% consisted of Sharing Experience. Other content areas each made up less than 10% of the total interactions: 9% Acknowledgement, 7% Questions/Suggestions, 3% Simple Agreement, 2% Student Related Issues, and 1% of Guided Advice. The Reading Reflection assignment, assignment 1 did not trigger different discussion foci between initial and response interactions. Sixty-eight percent of interactions were about sharing personal experiences even though teachers were supposed to discuss the reading material and reflect on their practice as compared to the recommended practice. The project team anticipated more interactions would be classified as Reflection and Instruction than other discussion content categories The discussion in response to the Reading Reflection assignment included issues of time constraints; the need to re-teach concepts; knowing how to explain the “why” of concepts; use of math vocabulary; the problem of teaching differently than the way the teacher originally learned concepts; ways to assist colleagues with engaging students in lessons; teaching specific concepts, such as graphing, equations, and the most effective way to use manipulatives; the importance of relating math concepts to real-world experiences; collaborating with other educators; using games, websites, and hands-on activities to teach concepts; and recognizing that all teachers across subject areas can and do struggle with similar issues in the classroom. Assignment 2 involved watching either a 7th or 8th grade algebra class and critiquing the lesson—focusing on the teacher’s practice and students’ interactions/engagement. In the online environment, the two main areas of focus in initial postings in this assignment were Reflection (67%) and Sharing Experience (21%); participants continued to interact primarily in the areas of Sharing Experience (30%) and Reflection (26%) in follow-up responses in the discussion thread. Even through participants were supposed to criticize and provide feedback about another teacher’s instruction, their main focus was not instruction but sharing their own experience and reflecting on their own instruction. These findings do not imply a negative outcome for the project or that the professional learning was not met but shows that certain PD assignments do not serve the intended goal/objective.

a. Initial Postings b. Response Postings

Figure 6. Content Comparison within the Level of Discussions by Project Focus



There was a difference in the content of initial messages among assignments 3, 4, and 5. Figures 7a and 7b depict the assignments separately. Assignments 3, 4, and 5 involved the development of a discovery-based lesson plan, implementation of the lesson, reflection on the lesson, and commenting on peer teachers’ lesson plans. Figures 6a and 6b show the combined data of assignments 3, 4, and 5, three discovery-based lesson implementations, the purpose being development and contextualized practice. The majority of both initial and response discussion postings were Sharing Experience (60% of initial posts and 48% of responses). If we only look at initial interactions, approximately 30% of messages were classified as Reflection. Sixty-seven percent of discussion posts were classified as Reflection after assignment 3, while teachers spent more time in sharing experiences (over 80%) after teaching their second and third discovery-based lessons (figure 7a). On the other hand, 24% of response postings were Reflection and 24% of response postings were Sharing Experiences following the first discovery-based lesson, while 60% of messages were Sharing Experiences for both the second and third lessons (figure 7b). It is not clear why messages related to the second and third discovery-based lessons were different from the messages for the first one. Possibly, the teachers followed a standardized development process for the first lesson, which ends with reflection. However, for the second and third discovery-based assignments, project participants did not reflect on their lesson as much but shared their ideas and experiences instead. Teachers shared ideas about how to include more student-led lessons and ways to more easily teach mathematics with the use of relevant connections. One of the teachers mentioned in a discussion post that “it is okay to jump around and cover things in your own way as long as your students are learning. You don’t have to follow a specific set of guidelines in one set order to accomplish what is needed.” This message reflects that instructors modeled the face-to-face workshop sessions and that the program motivated more discussions about resources and ideas to make math lessons more interactive and interesting.

VI. CONCLUSIONS This study investigated the effectiveness of a blended professional development program by examining the nature of participants’ learning processes in knowledge construction and classroom practice through virtual interactions. Regarding the effectiveness of the virtual interactions, there were some trivial

a. Initial Postings b. Response Postings

Figure 7. Content Comparison for Discovery-based Lesson Plan Assignments



findings and unexpected outcomes that can provide insight for mathematics educators. Most face-to-face discussions centered on the activity or mathematics problems that were presented in class, whereas online discussions focused on reflection and shared experiences from the participants' classrooms. The study suggests that the use of online learning combined with face-to-face meetings is very beneficial. Teachers were able to share actual classroom experience and get feedback from other professionals or share new ideas and available resource possibilities. The teachers in this study had very productive conversations via online discussion postings, in which they were talking about specific classroom events and asking questions to get more information about mathematics and teaching mathematics. These queries were in turn given well-developed responses. The content of online discussions did not vary significantly between the different assignment types and the project focus. Even though TACCC assignments were designed to address Knowledge Construction, Development, and Contextualized Practice, the main focus of virtual interactions was, in all cases, the sharing of personal experiences rather than addressing material learned from the readings and classroom observations. It is important to investigate how different online discussion topics contribute to the teachers’ active participation and the content of interactions. However, the content of interactions does not measure the depth of learning. That is, analyzing the content of interactions (the actual learning process) can lead to the assessment of objective achievement in online assignments (expected learning) but cannot measure the depth of learning: surface learning versus deep learning. Ways to connect the purpose and content of interactions with the depth of learning are needed as a follow-up to the content analysis study. Another interesting finding was the difference in the level of participation. First-year teachers contributed more to Knowledge Construction assignments, in which they were asked to reflect on book readings or videotaped best practices, than for the Development and Practice assignments, which required developing and implementing a discovery-based lesson. What teacher educators can learn from this finding is the sequential order of the online professional development program as described in the conceptual framework of the study: gaining knowledge, deepening understanding, and implementing learning. Regarding blended PD programs, this study determined that practitioners need plenty of time to familiarize themselves with the technology. That is, additional structured activities need to be provided for in-service teachers to fully utilize Salmon’s [6] first two stages: (1) Access and Motivation and (2) Socialization. Teacher educators should also consider the importance of investigating how different online discussion topics and assignments contribute to their active learning and participation. Professional development programs for mathematics teachers deal with different topics than regular college level (mathematics) courses. Therefore, assignments for knowledge construction and development should aim at linking theory and practice and reflect participants’ teaching contexts, their level of subject familiarity, and the needs of the individual teacher/school. Teacher educators should also investigate the different social media tools that other teachers use in the classroom. The level of participation, learning process, and outcomes may be influenced by the types of tasks, project focus, the mode of communication (verbal vs. written), and the type of systemic tools.

VII. ABOUT THE AUTHORS Hea-Jin Lee is an Associate Professor of Mathematics Education at the Ohio State University-Lima. Dr. Lee’s research addresses: improving reflective thinking and practice; assessing professional growth; designing teacher-needs based professional development programs; analyzing the effectiveness of technology-enabled teacher professional development; and teaching mathematics to children at various levels.



VIII. REFERENCES 1. Borba, M.D.C., and Llinares, S. Online Mathematics Teacher Education: Overview of an

Emergent Field of Research. ZDM Mathematics Education, 44, 697–704 (2012). DOI 10.1007/s11858-012-0457-3.

2. Rovai, A.P., and Jordan, H.M. Blended Learning and Sense of Community: A Comparative Analysis with Traditional and Fully Online Graduate Courses. International Review of Research in Open and Distance Learning (August 2004).

3. Feldman, D.H. Forward. In: John-Steiner, V., Creative Collaboration Oxford University Press: New York, ix-xiii, 2000.

4. Heinze, A., and Procter, C. Communication—a Challenge and an Enabler for Facilitating Blended Learning Community (2005). http://orgs.man.ac.uk/projects/include/experiment/heinze_proctor.pdf.

5. Kahan, S. Etienne Wenger on Communities of Practice: Engagement, Identity & Innovation. The Journal of Association Leadership (March 2004).

6. Salmon, G. E-Moderating: The Key to Teaching and Learning Online. Kogan Page: London, 2000. 7. Rovai, A.P. Sense of Community, Perceived Cognitive Learning, and Persistence in Asynchronous

Learning Networks. Internet & Higher Education, 5(4): 319–332 (2002). 8. Walker, P.C., and Chappell, M.F. Reshaping Perspectives on Teaching Mathematics in Diverse

Urban Schools. In Trentacosta, J. (Ed.), Multicultural and Gender Equity in the Mathematics Classroom: The Gift of Diversity, National Council of Teachers of Mathematics: Reston, VA, 201–208, 1997 Yearbook.

9. Brown N., and Benkenp, B. So When Do We Teach Mathematics? Vital Elements of Professional Development for High School Mathematics Teachers in an Urban Context. Teacher Education Quarterly, 55–73 (Summer 2009).

10. National Council of Teachers of Mathematics. Principles and Standards for School Mathematics. Author: Reston, VA, 2000.

11. U.S. Department of Education (USDE), Office of Planning, Evaluation, and Policy Development. Evaluation of Evidence-based Practices in Online Learning: A Meta-analysis and Review of Online Learning Studies, USDE: Washington, D.C., 2010.

12. Thomas, G. Knowledge as an Asset—a View from Industry, Journal of Computer Assisted Learning, 8: 131–152 (1992).

13. Hargreaves, A. Changing Teachers, Changing Times. Cassell: London, 1994. 14. Chapman, C., and Ramondt, L. (Eds.). Online Communities—Final Report: Developments in the

Communities of Practice and the Programme Communities. Report to the National College of School Leadership, September 2002–October 2003.

15. Blank, R.K., de las Alas, N., and Smith, C. Analysis of the Quality of Professional Development Programs for Mathematics and Science Teachers: Findings from a Cross-State Study, Council of Chief State School Officers: Washington, DC, 2008. http://programs.ccsso.org/content/pdfs/Year_2_IMPDE_Fall_06_Rpt_with_errata-041708.pdf.

16. Gareis, C.R., and Nussbaum-Beach, S. Electronically Mentoring to Develop Accomplished Professional Teachers. Journal of Personal Evaluation in the Education Website, 20: 227–246 (2007).

17. Interstate New Teacher Assessment and Support Consortium (InTASC). Model Standards for Beginning Teacher Licensing and Development: A Resource for State Dialogue, InTASC: Washington, DC, 1992.

18. Joint Committee on Standards for Educational Evaluation. The Personnel Evaluation Standards, Corwin: Newbury Park, CA, 1988.

http://orgs.man.ac.uk/projects/include/experiment/heinze_proctor.pdf

http://programs.ccsso.org/content/pdfs/Year_2_IMPDE_Fall_06_Rpt_with_errata-041708.pdf

An Analysis of the Research on Faculty Development for Online Teaching and Identification of New Directions


AN ANALYSIS OF THE RESEARCH ON FACULTY DEVELOPMENT FOR ONLINE TEACHING AND IDENTIFICATION OF NEW DIRECTIONS Katrina A. Meyer University of Memphis

ABSTRACT This article presents the results of an extensive review of the published literature on faculty development for online teaching. This review included 68 articles and five books, which were reviewed to identify elements of the training and the findings (e.g., theoretical bases, training aims and organization, content of training, nature of sample (number and type of participants), outcome measures used. The emphasis was not on the recommendations of the authors, but the methods of arriving at the findings. The review produced seven insights, from the importance of basing faculty development on theory, the frequency of publications that present models of faculty development for online teaching, the need to disentangle treatments, the need for rigorous evaluations, and the importance of considering individual faculty differences, designing and evaluating faculty development with specific outcome measures, and considering cost (either cost-efficiency or cost-effectiveness) when evaluating faculty development programs.

KEYWORDS Faculty development, elements of training, theory, evaluation, outcome measures, cost

I. INTRODUCTION What can we learn from the literature on faculty development that can improve our efforts to help faculty learn how to teach online? This question guided a review and analysis of the research and theoretical literature that will move from research on pre-Internet faculty development efforts to faculty development for online teaching, evaluate what we know against what we need to know, and finally, identify ways to increase what we know about how best to help faculty make the transition to online teaching. This is a lot of ground to cover, but understanding the issues involved in both faculty development and the theories and models currently in use is essential to arriving at some clear, understandable, and testable directions for the future. Faculty development plays a role in the Sloan Consortium’s [1] five pillars of quality. Faculty satisfaction is one of those pillars, which stresses the importance of faculty satisfaction with the online teaching experience and the faculty’s commitment to improving what they do in their online courses. Sloan-C characterizes faculty satisfaction as resulting from institutional support (also rewards and involvement in governance), which in turn is defined as the opportunity for “training in online instructional skills” (1, ¶5). Thus, faculty development in online teaching is a critical foundation for quality online education and justified the creation by Sloan-C of an advisory panel of practitioners and researchers focused on faculty satisfaction, development, and support (http://sloanconsortium.org/jaln_advisory_panel_fs). As a first step in the work of the advisory panel, it was critical to identify the current state of knowledge for faculty development for online teaching, which is the aim and purpose of the research that follows.

II. REVIEW OF LITERATURE

A. Faculty Development

http://sloanconsortium.org/jaln_advisory_panel_fs



1. Before Online Teaching Since the formation of higher education institutions, faculty development may have always been part of a faculty person’s duties, albeit characterized as staying abreast of the content matter taught by the instructor. In time, faculty development efforts became more organized but differed over time in focus. Sorcinelli et al. [2] proposed five distinct historical periods for faculty development efforts. The 1950s and 1960s were the age of the scholar and focused on research skills and productivity. The 1960s and 1970s were the age of the teacher and focused on improving teaching skills of the faculty. The 1980s were the age of the developer, which recognized the expertise of faculty developers whose efforts formalized and extended faculty development programs. The 1990s were the age of the learner, which recognized the shift in focus from teaching to student learning in higher education. The current era is the age of the network, which focuses on collaboration across faculty to encourage interdisciplinarity and more recently focuses on the electronic network – the Internet – and assisting faculty to learn how to best use technology for face-to-face courses and ultimately fully online courses. Several current guides to faculty development help define the enterprise and provide guidance for organizing and implementing faculty development efforts. Sorcinelli et al. [2] focus on future directions and priorities for faculty development – where online teaching is mentioned – and Gillespie and Robertson [3] deal more with faculty development as an important service within an institution. Cook and Kaplan [4] stress the ways a teaching center can not only provide essential faculty development but also create a culture of teaching. Schroeder [5] calls on faculty developers to revision their role as one of organizational development. While [3, 4, 5] include a single chapter in each guide on use of technology, these chapters may provide insufficient guidance to a unit specifically charged with helping faculty teach online. Although all of these guides are excellent primers on the faculty development enterprise in higher education, none are specifically directed to the specific problems of faculty development for online teaching. For this type of information, research articles found in various journals may be more helpful. Currently, faculty development primarily focuses on teaching, including use of problem-based learning [6], use of interdisciplinary instruction for diversity courses [7], a plan for achieving diversity [8], teaching science in higher education [9], teaching marketing education [10], teaching clinicians, medical doctors, or medical educators [11, 12], and addressing demands that may be different for faculty at different career stages [13]. It is likely there are additional uses. As may be clear from this list, the faculty development effort may be focused on one preferred pedagogy being promoted (e.g., problem-based learning, use of case studies) or one discipline (science, medical education) and may not address faculty needs for more pedagogies that work in many disciplines or general education level courses. In two national studies of community college faculty [14, 15], faculty development programs rarely addressed the variety of pedagogical challenges that these faculty faced on a daily basis. Faculty development has also been used to help faculty with other duties, such as understanding and having confidence in their ability to conduct assessment [16], although it is difficult to unravel which segment or portion of the four-part workshop series described by the authors actually contributed to changes in the faculty. This last is a common criticism of many faculty development programs, which provide a number of activities but cannot detail which activity was the most effective or whether change was dependent on all activities working in concert. In fact, as this review makes clear, the literature seems to include many faculty development articles that are sometimes hortative (“we must teach better”), sometimes descriptive (“here’s what we did”), and many with modest evaluations of the faculty development effort that are unable to unravel which specific activities are proven to work. A relevant issue is how the faculty development initiative is organized and delivered; does it matter if the effort is centralized or decentralized, provided one-to-one, in workshops, or in classes? Certainly, there are arguments for all options. Watson and Grossman [17] argue for the effectiveness of consortia, cooperatives, and distributed models since each organization may reflect particular institutional values or needs or ways of conducting its business. Many faculty development models have been proposed as well: the Master Teacher Initiative [10], New England Center for Inclusive Teaching [18], and Resource-



Enriched Learning Model [19]. Later in this literature review, we can find some authors (such as [20, 21] arguing for the use of online means to teach better ways of teaching in traditional classrooms, although the focus is not yet on learning to teach online. That development will be taken up in the next section. What is evident in this abbreviated review of faculty development is the lack of clarity on organizational effectiveness, a comparison of models, a breaking apart of the models and workshops to understand which activities actually affect faculty behavior, and a better grasp of which outcomes measures might be worth capturing.

2. For Online Teaching With the growth in online education, institutions of higher education have been “faced with the challenge of developing faculty who are ready, willing and able to teach in the online world” [22, p. 27]. Faculty developers have responded to this need with a wide ranging set of services and activities, so it is not surprising that faculty development for online teaching has been represented in the literature with the same diversity as for faculty development prior to online teaching. Publications propound on the different types of faculty development – from workshops to short courses, local and regional programs [23] – that originally had very narrow foci – from use of a specific course management system to learning how to do podcasts [24]. Only later did it become important to help faculty move away from face-to-face teaching methods toward more appropriate methods for online courses [25]. As a first stage, faculty were encouraged to add technology to existing face-to-face courses, and then to move to new technologies and pedagogies [25] useful for offering entire courses online. Articles have been published on special efforts to help faculty learn how to teach with technologies in the sciences [26, 27], clinical education [23], science education [27], marketing education [28], teacher education [24], community colleges [29], and to implement cloud-based technologies across the disciplines [30]. An area of continuing interest is in helping faculty learn how to create community in an online course [31] given its tie to improving student retention. Readers interested in thorough studies of the faculty online teaching experience from the point-of-view of the faculty person are encouraged to review Conceição [32], Major [33], and Meyer [34], among others. Conceição [32] found that faculty teaching online came to find the experience very rewarding in new ways. Major [33] found that faculty experienced a myriad of changes, from time demands to relationships with students, and needed support to help navigate these changes to their roles. Meyer [34] found that experienced online tenured or tenure-track faculty had found ways to improve the learning productivity (the amount or speed of learning) of students as well as their own professional productivity (inclusive of teaching, research, and service). Learning as much as we can about how faculty persons actually learn to teach online and experience online teaching can provide additional perspectives that can inform how faculty development programs should adjust to achieve their ends. When the different types or content of faculty development offerings are tracked over time, it becomes evident that the focus has shifted from learning about a new tool or approach to focusing on pedagogy as well as instructional design options [19]. Instructional design helps faculty understand the various pedagogies that can be used to help students learn, and different ways of using different technologies to implement these pedagogies. Evans [28] is a good example of an early view of what faculty needed to teach with technology: Just tell them “how the available technology works” and “how to apply the technology” [28, p. 6] and that was all they needed to teach well with technology. However, faculty recognize that the challenge of online learning demands more and have expressed interest in learning more about pedagogical techniques for online learning [35, 36, 37].

B. Faculty Development Models for Online Teaching The literature contains many articles about specific faculty development programs at specific universities. For example, the University of Central Florida requires all faculty who teach online to participate in a 70-hour faculty development course; Central Michigan University also implemented professional



development for online faculty that went beyond the one-time workshop to include weekly tips, online mentoring, and online teaching resources. The University of Houston system created the CampusNet Online Workshop program that includes faculty networking, hands-on practice, and a comfortable environment for asking questions of all kinds [38]. The University of Colorado created a Web Camp, offered over the summer and winter months, where faculty participate in a week-long intensive workshop that also includes hands-on training and design [36]. Michigan State University used master’s students enrolled in an instructional design course to help faculty design an online course [22]. The Open University tackled development of its faculty for mobile learning by providing events, communities, exploratory spaces, and resources [39]. PBS Teacherline [40] has extensive faculty development opportunities based on problem-based approaches [41]. Capella University [42] uses a META model (for Mentoring, Engagement, Technology, and Assessment) for its faculty development for online teaching. The University of Cincinnati funded grants that were proposed by faculty or departments [43]. Colorado State University used active mastery learning, using Bloom’s taxonomy and systems theory to create faculty development for online courses [44]. Fetters and Duby [45] described a faculty development program at Babson College which tied Rogers’ [46] theory of innovation diffusion to blended learning. Florida Atlantic University [47] developed a detailed plan for a new central elearning unit. A community of practice approach [48] pulled nursing faculty together across multiple campuses for faculty development. A three-tiered approach was used for online faculty development, from orientation, to mentoring, and ongoing support [49]. Finally, a three-week training session at University of Wisconsin-La Crosse was described in [50]. The point of reviewing these different programs is to make clear that many institutions, perhaps nearly all of them, have implemented a variety of faculty development programs aimed at helping faculty design and teach online courses in addition to using technology wisely in a traditional classroom. However, it is hard to compare the programs or assess what parts of these programs may be working well. What is disconcerting is the lack of stringent evaluations of some (although not all) of these programs. Evaluations can help developers at other institutions decide which interventions work best based on particular outcome measures that support the conclusions. And while many of these programs may have been based on assumed theoretical frameworks, these are not clear in the articles. Perhaps these faculty developers were basing their choice of activities on adult learning theory but did not make this clear in the published material. In all fairness to these institutions and their hard-working faculty development staff, collecting detailed evaluations may not have been of immediate concern since they were likely experiencing pressures to get something underway and respond quickly to a felt need. However, to build expertise and understanding of what specific activities work and why, the field of faculty development may need to contemplate a number of changes to what they are currently doing.

III. METHODOLOGY

A. Scholarship of Integration Eugene Rice [52] began the reexamination of the types of research conducted by faculty and proposed new terms to describe these types of scholarship. Subsequently, Ernest Boyer, then President of the Carnegie Foundation for the Advancement of Teaching, published Scholarship Reconsidered [52] and popularized Rice’s terms for four different kinds of scholarship. Up until this time, the definition of scholarship was primarily limited to what Rice and Boyer called the “scholarship of discovery,” or the unique discovery of primarily new knowledge (but which can also include confirmatory knowledge) from the lab or human subjects. To help broaden higher education’s view of scholarship in its research and service activities, Rice and Boyer proposed three additional types of scholarship: integration, application, and teaching. Higher education responded by accepting Boyer’s expanded definitions, and many institutions revised tenure and promotion guidelines to expand the types of scholarship accepted by and acceptable to faculty tenure and promotion committees.



The second scholarship, integration, focuses on making connections across disciplines and across research methodologies and interpreting this literature for possible future research studies. Integration draws insights from a broader set of publications and research studies, and depends upon a thorough and wide-ranging review of the research literature, including the theoretical literature, and especially multiple theories that may be relevant to the field. It “pulls disparate views and information together in creative ways” and requires scholars who can “synthesize . . . look for new relationships between the parts and the whole . . . relate the past and future to the present” [52, p. 13]. While a simple review of literature might be narrowly construed to prepare the groundwork for a single study and/or a set of hypotheses, this type of integration is more broad and can well provide the foundation for a new line of research. The effort described in this article is a review of the literature in the tradition of the scholarship of integration.

B. Sample To locate the articles included in this review, a number of database searches were conducted in WilsonWeb, ExpandedAcademic, and JSTOR using such search terms as “faculty development” and “faculty development online.” To look for faculty development articles specifically for online teaching, scholar.google.com was also searched, which tends to identify more articles in online journals that do not yet appear in mainstream databases. Articles were found in a large number of journals (see Appendix A), which demonstrates the difficulty of finding appropriate research articles since they seem to appear in a range of journals on higher education, faculty, online education, adult education, and separate disciplines or professions; limiting the search to a few journals would have provided a limited view of the range and diversity of these types of articles. Articles based on formal evaluations or research methodologies were preferred and opinion articles eliminated. No distinction was made in favor of evaluations or research studies; both were included in the review and will be treated as nearly equivalent for our purposes of understanding and critiquing the current faculty development literature. As articles were reviewed, references to prior research or theoretical literature were located, copied, and included in the analysis. The final set of articles included a total of 68 research articles and five books. Only those manuscripts which are referred to in this article are included in the list of references.

C. Analysis The analytical process followed in this review of literature followed the over-100 guidelines from a popular guide to preparing literature reviews [53]. To summarize these guidelines, all articles were read carefully and salient elements of each study were identified (e.g., theoretical bases, training aims, content of training, nature of sample (number and type of participants), outcome measures). The emphasis was not on analysis of the findings of the studies or recommendations of the authors, but on the methods of arriving at the findings. Then, each article was reread and notes taken to identify each element and then how each element was differentiated, such as different theories used, different aims of training, different content of training, different sample types, or different outcome measures. It was also important to note where an article seemed to have contributed something unique to the literature or had achieved a level of research worth commending. Of course, it was also important to evaluate this material in light of guides to quantitative and qualitative research design [54, 55], including specification of (as appropriate to the methodology) research questions, variable definition, development of the evaluation instrument and validation, data collection, and analysis. This analysis produced both similarities and dissimilarities across the studies as well as significant holes or missing elements. The results of this analysis are presented as findings across studies from various disciplines and in light of standards of research design [54, 55].



IV. FINDINGS

A. Importance of Theory In Creswell’s Research Design [54], theory has a critical role in both quantitative and qualitative methodologies, albeit very different roles. In quantitative research, theory is used deductively, as a way to conceptualize a study, derive testable hypotheses, and confirm (or disconfirm) the validity of the theory in the current case under study. In qualitative research, theory is used more inductively, after data are collected, or it can be developed to explain the data or more simply to explain the data that have been uncovered. In both cases, theory can be used to connect “new knowledge . . . to the vast body of knowledge to which it is relevant . . . without theory, we cannot have conceptual direction” [56, ¶1] or conceptual clarity where theory clarifies relationships and impacts. Theory, in other words, is a lens to view what may be disconnected bits of experience or data so that a relatively more coherent view of phenomena can be ascertained. Theory is an abstraction of more concrete experience, allowing individuals to see what might not be evident when immersed in daily experience. In quantitative research, a theory is “an interrelated set of constructs (or variables) formed into propositions or hypotheses that specify the relationship among variables” [54, p. 82] and thus it dictates the variables (their number, specificity, and relationships among them) to be collected and tested in the study. Theories come in many different forms, as micro-level, meso-level, and macro-level [57]; as grand theories that explain large categories of phenomena, middle-range theories (which fall between minor hypotheses and grand theories), and substantive theories that are restricted to a particular setting, group, time, population, or problem [58]; and as if-then statements or hypotheses, or visual models (especially helpful in clarifying the relationships among many variables and critical to testing causal or directional models). Given the deductive nature of quantitative research, a theory precedes all specifications of hypotheses, data collection approaches, and analysis. In qualitative research, a new theory may be proposed to explain relationships found in the data as in grounded theory research or an existing theory can be used to explain what has been found. In this latter case, patterns may be uncovered among the data collected that may emerge as a function of data analysis; patterns may interconnect in whole or only in some parts and may provide a basis for support of a pre-existing theory or lay the groundwork for a new or modified theory. However, a perfect fit between prior existing theories and results derived from qualitative research may not occur; this is not considered a flaw or a test as it would be in quantitative research, but an opportunity to see the theory differently or modify its propositions to fit the data. Given the inductive process of qualitative research, a theory is more likely to appear near the end of a qualitative study as interpretation or as a final, cohesive theory (as in grounded theory). Research into faculty development has benefited from the use of several theories, from adult learning theory to transformative learning theory, but other theories may also be useful to future faculty developers.

1. Adult Learning Theory Adult learning theories were first developed by Knowles [59], who proposed the use of the term “andragogy” (rather than pedagogy) for the ways that adults learn; later, Knowles recognized that both children and adults can and do learn using teacher-directed or learner-directed approaches and redefined andragogy to be determined by the learning situation rather than the age of the learner [60]. Andragogy emphasizes the importance to adults of pursuing learning that is important to them (such as professional development or learning to satisfy a personal interest), something they are motivated to learn, based on their prior experiences and immediate usefulness, and is self-directed. For example, a faculty bootcamp was designed and based on andragogy principles [61]; adult learning theories (including self-directed learning) were the basis of [62]; and adult learning principles were critical for the redesign of a faculty development program [63]. In a review of the research literature on



faculty development, McQuiggan [64] found evidence that studies of faculty development had been undertaken within an “adult education” or adult learning framework. Self-directed learning has been investigated for online learning, applied to civil servants in Taiwan [65] and casino workers [66], as well as other populations, including students, faculty, and librarians [67]. Self-directed learning became especially interesting to researchers with the advent of the Internet and its perceived usefulness for satisfying an individual’s need for learning about many topics [68, 66]. In fact, as an individual’s online skills improve, he or she is more likely to engage in self-directed learning [67]. For librarians, a self-directed technology training program resulted in individuals more motivated to continue their learning and more likely to incorporate new technology into their home or work duties [67]. Online instructors at community colleges were asked about their attitudes toward self-directed learning to predict their use of Web 2.0 technologies [69]; these attitudes are important in light of Web 2.0 tools that can liberate the individual to create new knowledge and take control of their learning process [70]. Self-directed learning has also been proposed as a foundation of transformative learning (see next section) by Cranton [71], so the practice (self-directed learning) and the theory (transformative learning) appear to be connected or may overlap, working to encourage change in participants. Merriam [60] posits that transformation learning is a second goal of self-directed learning, further connecting the two approaches. For example, Piling-Cormick [72] specifically proposed that transformative learning can and does occur in self-directed learning. This implies that these two theories can and do work together to create learning in adults.

2. Transformative Learning Theory Transformative learning [73] is the process that changes an adult’s perspective. Mezirow [74, p. 5] proposed that “Adults have acquired a coherent body of experience – associations, concepts, values, feelings, conditioned responses – frames of reference that define their life worlds . . . They selectively shape and delimit expectations, perceptions, cognition, and feelings . . . we have a strong tendency to reject ideas that fail to fit our preconceptions.” Adults naturally seek evidence as they learn that their views are correct, or they can establish new views, or transform those view(s) to something else. Perspective transformation is accomplished through the disorienting dilemma – an experience that forces the individual to question prior beliefs – and through autonomous thinking which involves serious reflection on one’s beliefs and assumptions, discussion of new information, and empathy toward other perspectives. Reflection is not a simple process, but includes three levels of content, process, and premise. Content reflection examines the content or description of the problem; process reflection examines the problem-solving strategies used; and premise reflection is the key to a transformation of meaning [75]. Reflection can further be applied to one’s emotions, the efficacy of one’s perceptions, thoughts, and actions, and one’s judgments, concepts, and theories that result from poor or limited information [75]. All of these reflection types create a critical reflection model of learning, which might be especially helpful to faculty developers trying to change a faculty member’s perspectives about teaching online, student learning, or their role as instructor. It is not surprising that, given this definition of transformative learning, several evaluations and research studies conducted on faculty development programs are based on this theory. McQuiggan [76] is an excellent example of an action research study based on reflection journals and interviews with participants after an extensive faculty development experience including planning, face-to-face sessions, and web conferencing. By including activities specifically intended to encourage reflection on core teaching beliefs, participants examined their reliance on lecturing, revised assignments to emphasize student construction of knowledge, and changed their teaching practices in face-to-face classes. Critical reflection was found to be important in the perspective transformation of 62% of the faculty involved in faculty development [77], where participants developed more open-minded attitudes towards others, became more appreciative of multiple perspectives, and developed stronger reflective practices applied to their work. In a study of teacher educators undergoing training in educational technologies,



participants changed their views of the teaching profession as well as what constitutes good educational practice [78], a finding that is consistent with transformational learning theory. Hubball, Collins, and Pratt [79] incorporated critical reflective practice (based on Schön’s [80] work of the same name) into an 8-month certificate program for faculty at the University of British Columbia; using the Teaching Perspectives Inventory, scores for those participating in the program increased significantly on all of the measures. McQuiggan [64], in an earlier review of the faculty development literature, found evidence that faculty development efforts were helping faculty transform their views of teaching. Early experiences while teaching online were bewildering, overwhelming, and disempowering for faculty, but reflection and assessment of their prior beliefs about what constitutes good teaching led participants to change their views of their role in the online course. McQuiggan [64] asserts, however, that faculty development that focuses primarily on teaching technology skills will not necessarily lead to challenging prior attitudes; she argues that faculty development needs to do both. For institutions and faculty developers deeply committed to helping faculty transform their views of what good teaching is, basing faculty development for online teaching on transformational learning theory may be an excellent choice.

3. Other Theories While the majority of faculty development articles that include a reference to theory use adult learning or transformative learning theory, other theories have been suggested although not yet included in studies. Trotter [81] has recommended “age and stage” theories such as the theory of ego development where adults move from conformity to independence and then reconciliation with conflicts [82]; Kohlberg’s [83] theory of moral development which includes the individual’s orientation to authority and others across the life cycle; and Kegan’s [84] theory of individual development from self-centered to other-centered to a more balanced view of self and other. Perry [85] proposed a cognitive development theory tracking development from dualism to commitment to relativism, and Gibb [86] developed a functional theory of adult learning that stresses the importance of learning that is problem or experience based. Perhaps multiple intelligences theory [87] can contribute to research on faculty development (as urged by [88]) or application of learning styles (e.g., auditory, visual, kinesthetic). These are necessarily incomplete descriptions of these theories, and the interested reader is encouraged to investigate the authors further to better understand how these theories might affect the delivery and success of faculty development activities. It may be useful to apply some learning theories and concepts researched in regards to student learning to the faculty development process. We know that experiential learning [89] is a powerful approach for learners to discover new meaning and that this is consistent with adult learning theory given its emphasis on building upon adults’ experiences [90]. Experiential learning also has the advantage of providing superior retention of learning rates (as high as 75%, compared to lecture which is 5%) [90, Figure 1]. Connectivism [91] is another theory not yet applied to faculty learning, but which integrates ideas from chaos, network, complexity, and self-organization theories and relies on seeking multiple information sources (including the Internet) and developing the ability to see connections between fields, ideas, and concepts. Phelps [92] investigated the use of complexity theory which is non-linear, stresses meaning in context, and uncertainty in causality; this is seen as a more representative learning theory since real life is not ordered or logically structured. The social constructivist theory of learning was applied to understanding the professional development of traditional lecturers adjusting to the online mode of learning [93]. Although never applied (so far) to faculty development conducted in an online setting, perhaps the Community of Inquiry model [94] may provide intriguing insights into how faculty persons learn in an online class by understanding cognitive, teaching, and social presence in the online course they are learning in and the online course they are teaching. In any case a wealth of learning theories – in addition to the valuable theories of adult learning and transformational learning – may be usefully applied to understanding how faculty learn to teach online. Another construct that might be especially worthwhile for faculty development for online teaching is TPCK: Technological Pedagogical Content Knowledge [95]. Teaching online is characterized as needing



all three – content, pedagogy, and technical knowledge – but also as a complex web of relationships between and among the single topics. For instance, pedagogical content knowledge is the knowledge of pedagogy that is applicable to teaching specific content, technological content knowledge is the understanding of how technology and content relate to each other, and technological pedagogical knowledge emphasizes how different technologies can be used in teaching [95, p. 743]. TPCK brings together these separate and relating concepts to produce online learning that is founded on the best understanding of how technology, pedagogy, and content combine to create learning. While this construct may not be a theory in the sense of the learning theories above, it may provide a structure that helps faculty see what they do online in multiple frames and to recognize the importance of trying to design activities that make the best use of technology and the best choice of pedagogy for the learning objective or content of the unit or course. In this sense, TPCK is an integrative theory of online coursework that recognizes the complexity of good online instruction.

4. Use of Theory in Faculty Development Research It is important to stress that the articles reviewed which were based on or referred to a theory were few in number, totaling only 15% of the articles reviewed. This should not be construed as meaning that faculty developers were unaware of theories; perhaps their choices were based unconsciously on theories of learning. The main point to be made is that a theoretical underpinning was not made clear in the article. If faculty development providers and evaluators wish to establish their practice and changes to those practices on more solid footing, more research that is based on theory (or seeks to establish new theory as is possible in qualitative research) is required. This is a serious flaw in faculty development articles, and could easily be remedied with use of existing theories such as transformative learning, which seems particularly apt for faculty developers wishing to change how faculty members understand their teaching role, or the development of new or revised theory that is appropriate for this population and the online teaching situation.

B. Models for Practitioners The emphasis on the need for theory above should not be construed to mean that the many articles describing different institutional models in an earlier section have no value. It is sufficient to say that faculty developers have been creative in their designs and models and often wish to share these with others in the field. Other developers, either new to the field or needing new ideas to stimulate their own practice, can find good ideas among such articles and – more importantly – further improve upon their own or other’s ideas in ways that ought to be shared with the professional community of faculty developers. However, to make progress in understanding what works and why, theory is likely needed. And to lift the model above its one-institution norm, models – or pieces of models – need to be replicated across institutions and careful analyses conducted. Another suggestion for improving the discussion of models is to arrive at some consistency when describing pieces of models. A seminar in one model is a workshop in another and workshops can be distinguished by their length, but are there important differences in a one-hour workshop versus a three-hour workshop? While the variety of models can be seen as a strength within the faculty development community, the Babel of terms and approaches works against producing insights or research results that can be easily understood and transferred from one development enterprise to another. Perhaps arriving at common definitions or descriptions can help move the discussion and research on faculty development models into more rigorous and consistent findings.

C. Disentangling Treatments As referred to in an earlier section, it is intriguing how many faculty development efforts are not one thing at all: they are several activities, structured in many ways, taking an afternoon or a week or a summer. How can evaluations unravel all of these treatments so that faculty developers can know which treatment is best for this faculty member or for that cost? This is especially critical as institutional and



program budgets are cut or constrained, since faculty developers may increasingly be asked to produce more or better results with the same or lower budget, and thus faculty development professionals must ask which activities to keep and which may need to be dropped. To make this critical decision, faculty developers will need to ask how each element of the faculty development effort contributes to the changes it wants; for example, it will need to break down the effectiveness of specific parts of the workshop – the introductions, the collaborations, the one-on-one consulting, the group discussions, the pedagogies used to teach new pedagogies, the technologies used to teach new technologies, and so forth – so that specific information is available. King [76] attempted to disentangle the influences on faculty by asking them questions about what activities influenced their perspective transformation: 86.1% of participants mentioned learning activities, which is further broken down into discussion (69.4% of participants), journals (52.8%), reflection (47.2%) and readings (47.2%). A total of 72% of participants also mentioned the influence of other persons, including a professor (33%), classmate (28%) or other student (28%). This is a useful attempt to begin disentangling the activities and influences going on in faculty development efforts. This is enormously difficult to do and will require a different approach to evaluation, perhaps exploring just-in-time evaluation (an evaluation screen that pops up at the completion of an activity), or reflective evaluations (asking participants to identify what activity helped them to learn or understand a concept), or authentic assessments (asking participants – or program completers – to produce an example of the learning intended). This may go well beyond what is usually conducted under the name of formative and summative evaluations, and may include multiple formative evaluations (to capture learning that is the result of a specific activity) as well as longer-term evaluations (to capture the learning that may take time to sink in). The goal of such efforts of disentangling the effects of a variety of treatments is to understand – precisely – what is working and for whom and conversely, what is not working and for whom.

D. Rigorous Evaluations Kucsera and Svinicki [96] conducted a literature review of nine journals that published faculty development evaluations between 1992 and 2007. Unfortunately, they concluded that only a few studies “met best practice standards” [96, p. 5] for program evaluations or for precision in those evaluations. To put this insight into perspective, only 47% of the articles included in this review could be construed as “research” (defined broadly as both quantitative and qualitative and including any outcome measures at all). The lack of good program evaluations is because faculty development programs are complex (comprising many parts and activities as noted above), take place over an extended period of time, and comprise small samples of faculty who are evaluated immediately at the end of the training, rather than being followed over time. While randomization and other qualities of good evaluation may never be possible given the constrained budgets of faculty development programs, the authors conclude that perhaps qualitative research methods – such as ethnographies, anthropological methods, and case studies [96, p. 8] – would be more likely to lead to useful insights into the training provided to faculty. Evaluations of faculty development programs or trainings have increasingly depended on qualitative research methods (as recommended by [96] above) and eschewed the identification of outcome measures a priori. For example, Lackey [97] interviewed six participants in faculty development for teaching online and found that one-on-one assistance as well as both technical and pedagogical training were most beneficial to preparing them to teach online. While an example of good qualitative research, more studies like this one and McQuiggan’s [76] are needed as well as larger studies. Both in addition to evaluating a particular faculty development activity, the literature so far lacks rigorous research comparing the effects of different faculty development models, programs, or activities, or comparing these across different institutions. It is understandable why this has not been done, since it would be costly to conduct and gather such cross-institutional data and no current source of funding for such research is available. However, perhaps individual faculty developers can pool resources across institutions to undertake such an endeavor in the future.



E. Faculty Differences One of the very curious absences in this review of the literature was the lack of attention to individual learner differences in the evaluations of faculty development programs. Why might this be so? Do developers and those who design and carry out evaluations believe that faculty members learn in a homogenous fashion? Do faculty think, act, and believe as a single entity? Currently, much of faculty development is packaged as a “one-size-fits-all” endeavor; yes, there are courses, and workshops, and one-on-one assistance. But perhaps all of these forms only reveal the same assumptions about how faculty persons think, feel, and learn. Faculty do need to learn new skills, but perhaps the ways they learn are more diverse than our current faculty development models account for. Certainly, individual differences are explored by some authors, especially Grant [24] who looked at intrinsic and extrinsic factors that influenced faculty satisfaction with training on a course management system. Edwards [9] found that faculty persons who saw their role as guides to learning were more likely to complete all of the faculty development modules than faculty who saw their role as providers of content. Findings such as these help us better understand whether faculty will benefit from a development experience and whether developers ought to provide multiple training types for different types of faculty learners. We need studies that delve into the faculty’s different perceptions of their role, student learning, and appropriate pedagogy for their discipline. We also need to better understand the faculty person’s learning styles and preferences, as well as differences by other kinds of personal and professional variables.

F. Outcome Measures What kinds of outcome measures did these studies use to base their assessment of the faculty development effort? This question has a disquieting answer: not many are named, and the few that are mentioned are not particularly clear or robust. Here is a partial list: number of new educational programs added [12], opinions about effectiveness of the training [98], adoption of case studies in instruction [20], improved teaching [10], professional growth of faculty [8], usefulness [7], satisfaction or relevance to participant [11], use of portfolios [19], more cooperation across disciplines [43], and confidence with and attitudes about assessment [16]. [11] compiled a variety of outcome measures used in faculty development effort for medical educators, from a positive change in attitudes to increased knowledge of and change in teaching behavior or student learning. [40] included such outcome measures of the faculty development provided to PBS online teachers as a score on the course rubric, learner course grades, and turnover of faculty. Edwards [9] also identified possible moderating variables: faculty who think of themselves as facilitators of learning (rather than disseminator of information) or had a higher sense of personal efficacy were more successful in completing all of the faculty development modules; also, knowledge of pedagogy and innovative course design were also important for successful change. Koepke [50] found that training changed faculty’s conceptions or “myths” of online learning, away from more critical or negative points of view as well as changed several teaching behaviors: from adding video and audio files to providing more, and more prompt, feedback. Orozco et al. [47] found that faculty development yielded such outcomes as increased comfort with using technology but also 27 detailed evaluations of the training provided (from “objective clarity” to “ease of interaction” to “discussion effectiveness”). What is clear from this list is that some do a better job of delineating outcomes of interest and either developing an evaluation instrument to measure those or delving more deeply into the factors that lead to faculty change [9, 16], and others are too amorphous or poorly defined to be identified with any confidence. Outcome measures for faculty development for online teaching include usefulness (as assessed by the participant), willingness to recommend the training to another faculty, self-reported knowledge or ratings of self-efficacy, changes in behavior, beliefs, and attitudes [23]. Schrum et al. [29] included many of these measures, but also self-reports of participants’ altering their pedagogy, redesigning their courses, or experiencing community online. In an attempt to understand the reasons for satisfaction with faculty



development, Grant [24] investigated an individual’s intrinsic factors (convenience, comfort, interests) and extrinsic factor (external pressure to teach online). In a more detailed evaluation of Purdue University at Calumet’s online courses for faculty, 47 participants underwent a year-long development program and then were followed over four years; the program evaluation included 72 evaluation items, from “I am satisfied” to “My on-campus teaching has improved” [99]. Potter and Meisels [27] included such authentic measurements as giving an example of how the faculty development impacted the individual’s “ability to think critically and use information to solve problems and answer questions,” “understanding of science in the news,” and applications of “problem-solving approaches learned” in the training (p. 194). These examples are then enhanced with further reflective questions that focus faculty persons’ attention on their teaching beliefs and application of concepts to other courses taught. What is clear from this information is that outcome measures are often poorly defined or poorly measured, depending on the honesty and self-understanding of those undergoing the training. While many faculty possess these qualities, evaluations of faculty development should not depend solely on such measures and should attempt to develop new ways to identify authentic outcomes of the training. Although authentic assessments are often more cumbersome and more costly than simple Likert-scale items, perhaps they are a way to give flesh to the bones of our current set of outcome measures.

G. Cost-Effectiveness of Faculty Development Another intriguing hole in the literature is the lack of consideration of cost-effectiveness of faculty development offerings. Certainly, this would be a difficult undertaking, but as institutional budgets are becoming more constrained by a decline in state funding, increased competition among internal parts of an institution for institutional resources, and increased demands on institutions for services as well as accountability, assessing cost-effectiveness of faculty development becomes more essential. How can faculty developers justify their activities if they cannot provide a good estimate of a) the cost of each effort, b) the efficiency of that effort (defined by the number of outcomes produced for the cost or cost-efficiency), and c) the quality of those outcomes for the cost (defined as cost-effectiveness)? Of course, undertaking such a calculation involves several of the above issues: a better understanding of the outcomes produced by faculty development efforts as well as a way to evaluate all aspects of faculty development efforts. With better attention to these elements, faculty developers may one day be able to say that while a workshop costs less to provide, one-on-one consulting provides more pertinent changes to online courses. Or that efforts that emphasize transformative learning and take time to conduct may change a participant’s philosophy of teaching and therefore be very costly, but may have secondary and tertiary outcomes in courses taught by the faculty person compared to a workshop on preparing a podcast, wiki, or blog. While the latter may still be essential, perhaps less costly ways of providing these skills can be explored. In any case, the final result would be faculty developers empowered with the data and the insights into their enterprise to make the best choices for the outcomes desired and the budget available to them.

V. RECOMMENDATIONS The analysis of the literature so far has found seven problematic qualities that lead to seven recommendations for consideration by faculty developers as well as those who conduct research in this area. First, the lack of a theory base for design of the training is problematic and discourages progress in our understanding of why or how faculty development can be improved. This should not be construed to mean that the designers of the training did not have a learning theory in mind when designing the training, but such theory is not clearly indicated in some of the articles based on the training. This is a potential hole in our understanding of faculty development and is an uncertain base that may preclude professionalizing faculty development practice. Without theory or theories, faculty development will likely stay an auxiliary function in higher education when it could be an essential foundation to helping



faculty learn the skills they need to be vital and productive members of the academy as well as helping institutions chart a course through the changes that are happening to higher education now and in the future. Second, articles will likely continue to be published that profile a certain institution’s faculty development program simply because such articles are a means for faculty developers to share what they are doing and any new developments or activities (or combinations of activities) with others in the field. There is a valuable role for this function in the literature, but let us recognize that these publications do not necessarily advance the field and our understanding of what works and why. Third, research needs to be undertaken that disentangles the various treatments included in faculty development programs. Rather than evaluate the program as a whole, we must attempt to assess the usefulness of separate activities, each (hopefully) with a theoretical basis for inclusion in the program. Because many current articles tend to evaluate a set of treatments or a model, it is impossible to evaluate the worth of these treatments from these articles or know which individual action or activity might work best for the individual faculty experiencing the activity. By disentangling a myriad of treatments and understanding the role and impact of each one separately, faculty developers will have made progress toward tailoring training for faculty that is matched to that faculty person’s learning needs. Fourth, research is needed that is rigorous, possessing the aspects of quality research or high-quality evaluation. This would require familiarity with such evaluation guides as [100] and the will and resources to do evaluation well. It would also require the willing participation of faculty, whose time is at a premium. Also, to move evaluations to a more rigorous level will mean that evaluations based on one institution or one model of training must be made more precise (see next recommendation) or include more comparisons to other faculty development activities, programs, or institutions. Research on only one program may produce idiosyncratic results (that is, results peculiar to that institution) that may or may not be generalizable to other institutions. This last point is integrally tied to the fifth recommendation, to develop outcome measures that are more than “satisfaction” with the training or an assessment of the training’s usefulness in the participant’s opinion. Fortunately, several articles above have proposed outcome measures that focus on the impact of the training on faculty and on their face-to-face courses, on student learning, and on the institution as a whole. This is an open area which requires additional work to be done, not only to incorporate the outcomes proposed by various theories, but also to keep our eyes out for unintended consequences of our efforts. One unintended consequence that seems possible is the replacement of “one best” teaching method with another “best” method, which reinforces an individual’s expectation that one solution will solve a variety of teaching problems, rather than learning how to design coursework that is flexible and fitting for many types of students. The final two recommendations which follow address two sizable “holes” in the research literature identified above. Sixth, articles published in the journals rarely include variables or constructs that capture individual faculty learning differences, or account for why such differences might affect the outcomes (in this way, the literature on faculty development seems to be behind the literature on student learning). Seventh, only a very few articles discuss the cost of the training and none assess the training in terms of a cost-benefit analysis. As higher education continues to feel the constraints of declining state funding and rising costs, assessing the cost of faculty development training programs is essential. Cost analyses are a demanding endeavor – requiring time, money, and wisdom to do well -- but they are necessary to understand the cost-effectiveness or -efficiency of this training in terms of the outcomes the training is intended to achieve. These findings tend to support several new directions for research on faculty development of online teaching that combine or cross these seven recommendations. First, we need to coordinate our efforts, which may be a useful function of the Sloan Consortium and other bodies of active faculty developers. We need studies that include several institutions or different institutional types, disciplines, faculty types, and outcomes. This can be done when several faculty developers at different institutions choose to work



together to test similar approaches, develop and share outcomes measures, and share materials developed at one institution but shared with many. One can see a time when multiple institutions, working together, develop multiple tests, evaluation tools, or online modules that can then be shared among the institutions, thereby saving the development cost of doing all of these functions separately. Second, we need to adopt one or several theoretical constructs for these studies, from adult learning theory, to transformational learning, to TCPK mentioned earlier; these are not an exhaustive set of useful learning theories that may apply to faculty. We need comparisons of theory-based approaches to determine if some theories seem more viable for this function than others or if different theories work for different learning outcomes. Third, we need to draw upon the literature on individual differences in adult learning so that these can be incorporated into research on faculty development, from learning styles to Myers-Briggs Types to other approaches that illuminate how an individual might respond to online teaching or various faculty development approaches in a unique manner. Fourth, we may need to work with educational researchers to design ways to test small, discrete events in the training so that these can be assessed for their value. Fifth, researchers in faculty development may wish to discuss and adopt some common definitions of training approaches and outcome measures so that results can more easily be compared across studies. All of these suggestions require some concerted action and such coordination may lift the faculty development field from one of active practice and exploration to solid, research-based foundations that have been tested and proven across many campuses. In other words, based on this analysis of the research and theoretical literature applied to faculty development for online teaching, I wish to argue that the research on faculty development for online teaching needs to be improved along the lines indicated above so that its findings can more reliably and helpfully inform future practice among institutions wishing to help faculty improve their online teaching. The future of higher education depends upon vibrant, active, and knowledgeable faculty members who are continuously improving their online teaching skills and helping our students learn well in our online courses and programs. Faculty developers are critical to encouraging and supporting these changes among the faculty who teach online.

VI. ACKNOWLEDGEMENTS This project could not have been completed without the assistance of two graduate assistants – Dr. Kristin Schaefer-Mumiuka at the University of Memphis and Jessica Lanham at Texas Tech University – who located many of the articles included in this review. I am also grateful for the editorial expertise of Dr. Stephanie Jones, Associate Professor of Higher Education at Texas Tech University, who reviewed an early draft of this article and provided critical suggestions. While each of these provided necessary help to the author, any errors are my own.

VII. ABOUT THE AUTHOR Dr. Katrina Meyer is currently professor of higher and adult education at The University of Memphis specializing in online learning and higher education. She is the author of Lessons Learned from Virtual Universities, a 2009 publication in the New Directions in Higher Education series, and Cost-Efficiencies of Online Learning, a 2006 publication of the ASHE Higher Education Report Series. For over three years, she was Director of Distance Learning and Technology for the University and Community College System of Nevada. Prior to this, she served over 8 years as Associate Director of Academic Affairs for the Higher Education Coordinating Board in the state of Washington and was responsible for technology planning and policy related to online learning.



VIII. REFERENCES 1. Sloan Consortium. Quality framework narrative, the 5

pillars. http://sloanconsortium.org/Quality_Framework_Narrative_5_pillars. 2. Sorcinelli, M., Austin, A., Eddy, P., & Beach, A. Creating the future of faculty development:

Learning from the past, understanding the present. Bolton, MA: Anker Publishing, 2006. 3. Gillespie, K.J., & Robertson, D.L. A Guide to Faculty Development (2nd edition). San Francisco,

CA: Jossey-Bass, 2010. 4. Cook, C., Kaplan, M., & Monts, L.P. Advancing the Culture of Teaching on Campus. Sterling,

VA: Stylus, 2011. 5. Schroeder, C. Coming in from the margins. Sterling, VA: Stylus, 2010. 6. Major, C.H., & Palmer, B. Reshaping teaching and learning: The transformation of faculty

pedagogical content knowledge. Higher Education, 51(4): 619-647, 2006. 7. Lindman, J.M., & Tahamont, M. Transforming selves, transforming courses: Faculty and staff

development and the construction of interdisciplinary diversity courses. Innovative Higher Education, 39(4): 289-304, 2006.

8. DiLorenzo, T.M., & Heppner, P.P. The role of an academic department in promoting faculty development: Recognizing diversity and leading to excellence. Journal of Counseling & Development, 72(5): 485-491, 1994.

9. Edwards, L., Freeman, L.M., Hodges, J., Johnston, R.A., Odell, M., & Sunal, C.S. Teaching science in higher education. School Science and Mathematics, 101(5): 246-257, 2001.

10. Atheny, S., & Hoffman, K.D. The Master Teacher Initiative: A framework for faculty development. Marketing Education Review, 17(3): 1-9, 2007.

11. Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., & Prideaux, D. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education. Medical Teacher, 28(6): 497-526, 2006.

12. Gruppen, L.D., Frohna, A.Z., Anderson, R.M., & Lowe, K.D. Faculty development for educational leadership and scholarship. Academic Medicine, 78(2): 137-141, 2003.

13. Bland, C.J., & Risbey, K.R. Faculty development programs. Effective Practices for Academic Leaders, 1(7): 1-16, 2006.

14. Murray, J. Faculty development in a national sample of community colleges. Community College Review, 27(3): 47-64, 1999.

15. Murray, J. Faculty development in publicly supported two-year colleges. Community College Journal of Research and Practice, 25(7): 487-502, 2001.

16. Haviland, D., Shin, S., & Turley, S. Now I’m ready: The impact of a professional development initiative on faculty concerns with program assessment. Innovative Higher Education, 35(4): 261-275, 2010.

17. Watson, G., & Grossman, L.H. Pursuing a comprehensive faculty development program: Making fragmentation work. Journal of Counseling & Development, 72(5): 465-473, 1994.

18. Daly, C.J., & Dee, J.R. Innovative models for organizing faculty development programs: Pedagogical reflexivity, student learning empathy, and faculty agency. Human Architecture: Journal of the Sociology of Self-Knowledge, 7(1): 1-22, 2009.

19. Lavoie, D., & Rosman, A.J. Using active student-centered learning-based instructional design to develop faculty and improve course design, delivery, and evaluation. Issues in Accounting Education, 22(1): 105-118, 2007.

20. Anurag, S., & Brajesh, K. An approach for faculty development. Advances in Management, 2(10): 54-58, 2009.

21. Shea, T. P., Sherer, P. D., & Kristensen, E. W. Harnessing the potential of online faculty development: Challenges and opportunities. To Improve The Academy, 20: 162-179, 2002.

http://sloanconsortium.org/Quality_Framework_Narrative_5_pillars



22. Koehler, M.J., Mishra, P., Hershey, K., & Peruski, L. With a little help from your students: A new model for faculty development and online course design. Journal of Technology and Teacher Education, 12(1): 25-55, 2004.

23. Skeff, K.M., Stratos, G.A., Mygdal, W., DeWitt, T.A., Manfred, L., Quirk, M., Roberts, K., Greenberg, L., & Bland, C.J. Faculty development. Journal of General Internal Medicine, 12: 56-63, 1997.

24. Grant, M.M. Learning to teach with the web: Factors influencing teacher education faculty. Internet and Higher Education, 7:329-341, 2004.

25. Howell, S.L., Saba, F., Lindsay, N.K., & Williams, P.B. Seven strategies for enabling faculty success in distance education. Internet and Higher Education, 7:33-49, 2004.

26. Kane, S.A. Interdisciplinary faculty development seminar: A model for learning emerging technologies while developing interdisciplinary partnerships. Journal of Science Education and Technology, 12(4): 421-430, 2003.

27. Potter, R., & Meisels, G. Enhancing teacher preparation and improving faculty teaching skills: Lessons learned from implementing “Science That Matters” a standards based interdisciplinary science course sequence. Journal of Science Education and Technology, 14(2): 191-204, 2005.

28. Evans, J.R. The emerging role of the Internet in marketing education from traditional teaching to technology-based education. Marketing Education Review, 11(3): 1014, 2001.

29. Schrum, L., Burbank, M.D., Engle, J., Chambers, J.A., & Glassett, K.F. Post-secondary educators’ professional development: Investigation of an online approach to enhancing teaching and learning. Internet and Higher Education, 8: 279-289, 2005.

30. Diaz, V. Cloud-based technologies: Faculty development, support, and implementation. Journal of Asynchronous Learning Networks, 15(1): 95-102, 2011.

31. Palloff, R.M., & Pratt, K. Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass, 1999.

32. Conceição, S.C.O. Faculty lived experiences in the online environment. Adult Education Quarterly, 57(1): 26-45, 2006.

33. Major, C.H. Do virtual professors dream of electric students? University faculty experiences with online distance education. Teachers College Record, 112(8), 2154-2208, 2010.

34. Meyer, K.A. The influence of online teaching on faculty productivity. Innovative Higher Education, 37(1), 37-52, 2012.

35. Hinson, J. M., & LaPrairie, K. N. Learning to teach online: Promoting success through professional development. Community College Journal of Research and Practice, 29, 483-493, 2005.

36. Lowenthal, P.R., & Thomas, D. Digital campfires: Innovations in helping faculty explore the online learning wilderness. MERLOT Journal of Online Learning and Teaching, 6(3), 2010. http://jolt.merlot.org/vol6no3/lowenthal_0910.htm

37. Pankowski, P. Faculty Training. T.H.E. Journal. September 2004. 38. Kidney, G.W., & Frieden, S. When the cows come home: A proven path of professional

development for faculty pursuing e-learning. The Journal, 2004. http://thejournal.com/Articles/2004/06/01/When-the-Cows-Come-Home-A-Proven-Path-of Professional-Development-for-Faculty-Pursuing-ELearning.aspx

39. Kukulska-Hulme, A. How should the higher education workforce adapt to advancements in technology for teaching and learning? Internet and Higher Education, 15(4), 247-254, 2012.

40. Storandt, B.C., Lacher, A.P., & Dossin, L.C. Toward an understanding of what works in professional development for online instructors: The case of PBS Teacherline. Journal of Asynchronous Learning Networks, 12(1): 121-162, 2012.

41. Southern Regional Education Board (SREB). Guidelines for professional development for online teachers. Southern Regional Education Board: Atlanta, GA, 2009. http://publications.sreb.org/2009/09T01_Guide_profdev_online_teach.pdf

http://jolt.merlot.org/vol6no3/lowenthal_0910.htm

http://thejournal.com/Articles/2004/06/01/When-the-Cows-Come-Home-A-Proven-Path-of%20Professional-Development-for-Faculty-Pursuing-ELearning.aspx

http://thejournal.com/Articles/2004/06/01/When-the-Cows-Come-Home-A-Proven-Path-of%20Professional-Development-for-Faculty-Pursuing-ELearning.aspx

http://publications.sreb.org/2009/09T01_Guide_profdev_online_teach.pdf



42. Dittmar, E., & McCracken, H. Promoting continuous quality improvement in online education: The META model. Journal of Asynchronous Learning Networks, 12(1): 163-176, 2012.

43. Camblin, L.D., & Steger, J.A. Rethinking faculty development. Higher Education, 39(1):1-18, 2000.

44. Puzziferro., M., & Shelton, K. A model for developing high-quality online courses: Integrating a systems approach with learning theory. Journal of Asynchronous Learning Networks, 12(3-4): 119-136, 2008.

45. Fetters, M.L., & Duby, T.G. Faculty development: A stage model matched to blended learning maturation. Journal of Asynchronous Learning Networks,15(1): 77-85, 2011.

46. Rogers, E.M. Diffusion of Innovation. New York: Free Press, 2003. 47. Orozcco, M., Fowkles, J.K., Jerzak, P., & Musgrove, A. Zero to sixty plus in 108 days: Launching

a central elearning unit and its first faculty development program. Journal of Asynchronous Learning Networks, 12(1): 177-192, 2012.

48. Reilly, J., Vandenhouten, C., & Gallagher-Lepak, S. Faculty development for e-learning: A multi-campus community of practice (COP) approach. Journal of Asynchronous Learning Networks, 12(1): 99-110, 2012.

49. Vaill, A.L., & Testori, P.A. Orientation, mentoring and ongoing support: A three-tiered approach to online faculty development. Journal of Asynchronous Learning Networks, 12(1): 111-120, 2012.

50. Koepke, K., & O’Brien, A. Advancing pedagogy: Evidence for the role of online instructor training in improved pedagogical practices. Journal of Asynchronous Learning Networks, 12(1): 73-84, 2012.

51. Rice, R.E. The new American scholar. Metropolitan Universities, 1(4): 7-18, 1991. 52. Boyer, E. L. Scholarship reconsidered: Priorities of the professoriate. San Francisco: Jossey-Bass,

1997. 53. Galvan, J.L. Writing Literature Reviews (4th edition). Glendale, CA: Pyrczak Publishing, 2009. 54. Creswell, J.W. Research Design: Quantitative and Qualitative Approaches. Thousand Oaks, CA:

Sage Publications, 2004. 55. Maxwell, J.A. Qualitative Research Design: An Interactive Approach. Thousand Oaks, CA: Sage

Publications, 1996. 56. DePoy, E., & Gitlin, L.N. Introduction to Research: Multiple Strategies for Health and Human

Services (2nd edition). St. Louis, MO: Mosby, 1998. Retrieved from http://www.umdnj.edu/idsweb/shared/why_is_theory_important_chpt6_depoy.htm

57. Neuman, W.L. Social research methods: Qualitative and quantitative approaches. Boston, MA: Allyn & Bacon, 1991.

58. Merriam, S.B. Case study research in education: A qualitative approach. San Francisco: Jossey-Bass, 1988.

59. Knowles, M.S. Self-directed learning. New York: Association Press, 1975. 60. Merriam, S.B. Andragogy and self-directed learning: Pillars of adult learning theory. New

Directions for Adult and Continuing Education, 89: 3-13, 2001. 61. Johnson, T., Wisniewski, M.A., Kuhlemeyer, G., Isaacs, G., & Krzykowski, J. Technology

adoption in higher education: Overcoming anxiety through faculty bootcamp. Journal of Asynchronous Learning Networks, 12(1): 63-72, 2012.

62. Henning, T.B. Writing professor as adult learner: An autoethnography of online professional development. Journal of Asynchronous Learning Networks, 12(1): 9-26, 2012.

63. DeNoyelles, A., Cobb, C., & Lowe, D. Influence of reduced seat time on satisfaction and perception of course development goals: A case study in faculty development. Journal of Asynchronous Learning Networks, 12(1): 85-98, 2012.

64. McQuiggan, C.A. The role of faculty development in online teaching’s potential to question teaching beliefs and assumptions. Online Journal of Distance Learning Administration, 10(3), 2007. http://www.westga.edu/~distance/ojdla/fall103/mcquiggan103.pdf

http://www.umdnj.edu/idsweb/shared/why_is_theory_important_chpt6_depoy.htm

http://www.westga.edu/~distance/ojdla/fall103/mcquiggan103.pdf



65. Lai, H. The influence of adult learners’ self-directed learning readiness and network literacy on online learning effectiveness: A study of civil servants in Taiwan. Educational Technology & Society, 14(2): 98-106, 2011.

66. Lema, J., & Agrusa, J. Relationship of WWW usage and employee learning in the casino industry. International Journal of Hospitality Management, 28(1): 18-25, 2009.

67. Quinney, K.L., Smith, S.D., & Galbraith, Q. Bridging the gap: Self-directed staff technology training. Information Technology and Libraries, 29(4): 2-5-213, 2010.

68. Hiemstra, R. Is the internet changing self-directed learning? International Journal of Self-directed Learning, 3(2); 45-60, 2006.

69. Ulrich, J., & Karvonen, M. Faculty instructional attitudes, interest, and intention: Predictors of Web 2.0 use in online courses. Internet and Higher Education, 14: 207-216, 2011.

70. Kop, R. Web 2.0 technologies: Disruptive or liberating for adult education? Adult Education Research Conference, 2008. http://www.adulterc.org/Proceedings/2008/Proceedings/Kop.pdf

71. Cranton, P. Professional Development as Transformative Learning. San Francisco: Jossey-Bass, 1996.

72. Piling-Cormick, J. Transformative and self-directed learning in practice. New Directions for Adult and Continuing Education, 74: 69-77, 1997.

73. Mezirow, J. Transformative Dimension of Adult Learning. San Francisco: Jossey-Bass, 1991. 74. Mezirow, J. Transformative learning: Theory to practice. New Directions for Adult and Continuing

Education, 74: 5-12, 1997. 75. Wang, V.C.X & King, K.P. Understanding Mezirow’s theory of reflectivity from Confucian

perspectives: A model and perspective. Radical Pedagogy, 8(1), 2006. http://radicalpedagogy.icaap.org/content/issue8_1/wang.html

76. McQuiggan, C.A. Faculty development for online teaching as a catalyst for change. Journal of Asynchronous Learning Networks, 12(1): 27-62, 2012.

77. King, K.P. Both sides now: Examining transformative learning and professional development of educators. Innovative Higher Education, 29(2): 155-174, 2004.

78. King, K.P. Educational technology professional development as transformative learning opportunities. Computers & Education, 39: 283-297, 2002.

79. Hubball, H., Collins, J., & Pratt, D. Enhancing reflective teaching practices: Implications for faculty development programs. The Canadian Journal of Higher Education, 35(3): 57-81, 2005.

80. Schön, D.A. Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco: Jossey-Bass, Inc., 1996.

81. Trotter, Y.D. Adult learning theories: Impacting professional development programs. Delta Kappa Gamma Bulletin, 72(2): 8-13, 2006.

82. Loevinger, J. Ego Development: Conceptions and Theories. San Francisco: Jossey-Bass, 1976. 83. Kohlberg, L. Stage and sequence: The cognitive development approach to socialization.” In D.A.

Goslin (Ed.), Handbook of Socialization Theory and Research (p. 347-380). Chicago: Rand-McNally, 1969.

84. Kegan, R. The Evolving Self: Problem and Process in Human Development. Cambridge, MA: Harvard University Press, 1982

85. Perry, W.G. Forms of Intellectual and Ethical Development in the College Years. New York: Holt, Rinehart and Winston, 1970.

86. Gibb, J.R. Learning theory in adult education. In M.S. Knowles (Ed.), Handbook of Adult Education in the United States, (p. 54-64). Washington D.C.: Adult Education Association of the USA, 1960.

87. Gardner, H. Multiple intelligences: New horizons. New York: Basic Books, 2006. 88. Riha, M., & Robles-Piña, R.A. The influence of multiple intelligence theory on web-based learning.

MERLOT Journal of Online Learning and Teaching, 5(1): 97-103. 89. Kolb, D.A. Experiential Learning: Experience as the Source of Learning and Development.

Englewood Cliffs, NH: Prentice-Hall, 1984.

http://www.adulterc.org/Proceedings/2008/Proceedings/Kop.pdf

http://radicalpedagogy.icaap.org/content/issue8_1/wang.html



90. Lobel, M., Neubauer, M., & Swedburg, R. Elements of group interaction in a real-time synchronous online learning-by-doing classroom without F2F participation. USDLA Journal, 16, 2002. http://www.usdla.or/html/journal/APR02_issue/article01.html

91. Siemens, G. Connectivism: A learning theory for the digital age. http://www.ingedewaard.net/papers/connectivism/2005_siemens_ALearningTheoryForTheDigitalAge.pdf

92. Phelps, R. Developing online from simplicity toward complexity: Going with the flow of non-linear learning. http://naweb.unb.ca/proceedings/2003/PaperPhelps.html

93. Maor, D. Using reflective diagrams in professional development with university lecturers: A development tool in online teaching. Internet and Higher Education, 9: 133-145, 2006.

94. Garrison, D.R., Anderson, T., & Archer, W. Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1): 7-23, 2001.

95. Koehler, M.J., Mishra, P., & Yahya, K. Tracing the development of teacher knowledge in a design seminar: Integrating content, pedagogy and technology. Computers & Education, 49: 740-762, 2007.

96. Kucsera, J.V., & Svinicki, M. Rigorous evaluations of faculty development programs. Journal of Faculty Development, 24(2): 5-18, 2010.

97. Lackey, K. Faculty development: An analysis of current and effective training strategies for preparing faculty to teach online. Online Journal of Distance Learning Administration, 14(5), 2011.

98. http://www.westga.edu/~distance/ojdle/winter144/lackey144.html 99. Maxwell, W., & Kazlauskas, E. Which faculty development methods really work in community

colleges? Community College Journal of Research and Practice, 16(4): 351-360, 1992. 100. Hixon, E., Barczyk, C., Buckenmeyer, J., & Feldman, L. Mentoring university faculty to

become high quality online educators: A program evaluation. Online Journal of Distance Learning Administration, 14(5), 2011. http://www.westga.edu/~distance/ojdla/winter144/hixon_Barczyk_Buckenmeyer_feldman144.html

101. Boulemetis, J. & Dutwin, P. The ABCs of Evaluation. San Francisco: Jossey-Bass, 2005.

IX. APPENDIX A List of Peer-Reviewed Journals That Produced Faculty Development Articles

Journal Name

Number of Articles in Each Journal

Journal of Asynchronous Learning Networks 13 Internet and Higher Education 6 Innovative Higher Education 4 Online Journal of Distance Learning Administration New Directions for Adult and Continuing Education

3

Community College Journal of Research and Practice Computers & Education Higher Education Journal of Counseling & Development Journal of Science Education and Technology Marketing Education Review MERLOT Journal of Online Learning and Teaching

2

Academic Medicine Adult Education Quarterly

1

http://www.usdla.or/html/journal/APR02_issue/article01.html

http://www.ingedewaard.net/papers/connectivism/2005_siemens_ALearningTheoryForTheDigitalAge.pdf

http://www.ingedewaard.net/papers/connectivism/2005_siemens_ALearningTheoryForTheDigitalAge.pdf

http://naweb.unb.ca/proceedings/2003/PaperPhelps.html

http://www.westga.edu/~distance/ojdle/winter144/lackey144.html

http://www.westga.edu/~distance/ojdla/winter144/hixon_Barczyk_Buckenmeyer_feldman144.html

http://www.westga.edu/~distance/ojdla/winter144/hixon_Barczyk_Buckenmeyer_feldman144.html



Journal Name

Number of Articles in Each Journal

Advances in Management American Journal of Distance Education Community College Review Delta Kappa Gamma Bulletin Educational Technology & Society Effective Practices for Academic Leaders Human Architecture: Journal of the Sociology of Self-Knowledge Information Technology and Libraries International Journal of Hospitality Management International Journal of Self-directed Learning Issues in Accounting Education Journal of Faculty Development Journal of General Internal Medicine Journal of Technology and Teacher Education Medical Teacher Metropolitan Universities Radical Pedagogy School Science and Mathematics Teachers College Record The Canadian Journal of Higher Education The Journal To Improve The Academy USDLA Journal Total Articles 68

A Conceptual Framework for Integrating Industry/Client-Sponsored Projects into Online Capstone Courses


A CONCEPTUAL FRAMEWORK FOR INTEGRATING INDUSTRY/CLIENT-SPONSORED PROJECTS INTO ONLINE CAPSTONE COURSES Rana Khan University of Maryland University College Julie Hill University of Maryland University College ABSTRACT To address the growing need for incorporating experiential learning into online degree programs, this paper proposes a design framework that would integrate industry-sponsored projects into online capstone courses. The design could be applicable to any program at any institution. The research and data used to develop the framework was gathered from literature review and a survey of University of Maryland University College (UMUC) graduate programs. The proposed framework was tested in two capstone courses using industry/client sponsored projects in the authors’ disciplines as the first stage of testing of the model. KEYWORDS Capstone, design, framework, graduate, online, project, client/sponsor, industry partner

I. INTRODUCTION

The authors used research and data from a literature review and a survey of UMUC graduate programs to develop a framework that incorporates industry-sponsored projects into online capstone courses. Most master’s programs at UMUC end with a capstone course. Several of the capstone courses use industry/client sponsored projects, including the ones in the authors’ respective disciplines, biotechnology and public relations. The proposed framework was tested with in the biotechnology and public relations programs as the first stage of testing of the model.

Integrating hands-on experience and/or experiential learning into the curricula of degree programs is gaining ground in academic programs in part because academicians and employers alike see value in providing students with practical opportunities to demonstrate proficiency in their fields of study. To meet this need, capstone courses that incorporate some form of experiential learning opportunity have emerged as a mainstay of many college and university programs, primarily at the senior undergraduate level. Capstone courses provide practical experience in a variety of ways, including internships, cooperative education opportunities, simulations, or projects. In some cases, academic programs utilize real-world industry/client-sponsored projects as the foci of the students’ final work products. Institutions form partnerships with employers, the sponsors receive low- or no-cost project work, and students leave with a portfolio of work and professional connections that could lead to employment opportunities.

Regardless of the specific design of the practical capstone courses, a common denominator is to provide a culminating experience that allows students to demonstrate mastery of knowledge and skills gained through prior classwork during the degree program. Kelly [1] emphasizes the need to pay special attention to the development and design of capstone courses such that they provide the right culminating

A Conceptual Framework for Integrating Industry/Client-Sponsored Projects into Online Capstone

Courses


experience with reflection, integration, and summation of the knowledge and skills gained through the program.

As one of the first institutions in the world to offer distance education opportunities, UMUC has established itself as a leader in the field. A public university offering primarily online programs at both graduate and undergraduate levels, UMUC reported a student headcount of 93,193 in FY 2013 with 63,103 undergraduate and 33,694 graduate students. Most of the 18 graduate programs at UMUC include a set of required core courses followed by a set of specialization-specific courses. UMUC’s focus is on providing students with a strong and workforce-relevant curriculum. While face-to-face teaching environments allow for fewer obstacles in providing experiential learning opportunities for students, UMUC has long recognized the value of integrating hands-on experiential learning activities and has adapted traditional face-to-face teaching and learning methods to the online environment to accomplish this goal.

II. LITERATURE REVIEW A number of researchers have explored the value of incorporating client-based projects into capstone courses. Most of the research has focused on undergraduate degree programs and capstone courses offered in a traditional face-to-face learning environment.

Applied science and engineering programs in particular make extensive use of capstone projects that often incorporate work with real industry/client-sponsored projects [2, 3]. Bruhn and Camp [4] highlighted the impact on all stakeholders of an engineering capstone course for college seniors developed in consultation with the industry. The faculty worked with students to assign members and projects to each team and provided guidelines on how to produce a deliverable that would be useful to the sponsor. The course employed a three-tiered monitoring system that included student self-monitoring, a corporate mentor, and faculty monitoring. The process allowed every team member to pull equal weight and perform a fair share of the tasks. At the conclusion of the course, all students felt better equipped to move on to a job, and there was a consensus that this course was invaluable to the curriculum.

Gorka, Miller, and Howe [5] found that for capstone projects with industry partners to be successful, industry participation is essential, confidentiality must be maintained, and there should be an industry representative or liaison. Berheide [6] explored the potential of capstone courses for assessing student learning and found considerable support for such assessments if the course is properly designed and executed, making capstone courses ideal summative assessment tools [7].

A consortium of faculty at the Western Michigan University’s College of Engineering and Applied Sciences was tasked with designing a capstone with a uniform experience across the college [3]. They shared design expectations and evaluation materials and then identified common needs and developed common activities. The outcome was a multidisciplinary capstone course designed to include topics such as written and oral communication, teamwork, patent issues, and intellectual property that were applicable across the college. In addition, common evaluation materials for proposal, presentation, report, and literature review were developed.

Gorman [8] wrote about the best practices he identified in designing a capstone course for an undergraduate program in operations management. This course is offered in a six-credit, two-semester format. In the first semester, students complete a one-credit requirement of proposal development based on client needs. In the second semester, each student team earns five credits by implementing the proposal. Conclusions drawn from this capstone course over a period of eight years suggest that a consulting-style project provides a unique and rewarding experience for students, and the project is equally valuable to the clients. The key determinants of success include a well-defined scope of the project, deep commitment of faculty and the client, simplicity of approach used to address the problem, reliance on the facts provided, and excellent written and oral presentation skills of the students.



At the graduate level, there is limited research on the design of capstone courses that, similar to undergraduate capstones, exist in a variety of forms. Although fewer graduate than undergraduate programs incorporate industry/client-sponsored projects, there is growing recognition and evidence that such projects have significant benefits for students, the academic programs and institutions, and the industry partners/clients [4, 5, 9].

Within this body of research, some educators outline how their institutions’ capstone courses are designed and what seems to work for them. Hagan [13] provided an example of a client project model in an online graduate program. The paper focused on the design of a project model that the author used to successfully integrate hands-on experience in a business course. A client project secured by the instructor was integrated into a marketing course on strategic communication and public relations in the MBA program at a private non-profit business college. Each student team applied their knowledge by developing a public relations campaign for a non-profit organization. The teams developed a work plan and provided a progress report, final report, and presentation to the client. Team members evaluated each other and the client offered feedback to the teams at the end of the project.

Other educators suggested that they follow some best practices, including clearly identifying project expectations and deliverables upfront, ensuring industry and faculty leads are committed to the project and the course is well-structured. [3]. Still others proposed a set of best practices to follow when real client projects are used in a capstone course [14, 15, 16]. For example, Estell and Hurtig [14] described the capstone course in the Electrical and Computer Engineering and Computer Science department at Ohio Northern University. The department “adopted both an industry-based project management standard and a corresponding corporate project management documentation practice as an operational framework” [14]. The department also employed a project review board made up of faculty with specific expertise related to the projects and formative and summative assessments based on common rubrics.

Research to date has found only a few professional associations and accrediting boards that recommend/require the inclusion of a capstone course as part of a higher education curriculum. Of those, a handful specify the inclusion of some type of integrative, experiential capstone: the Accreditation Board for Engineering and Technology, the Commission on Accreditation for Health Informatics and Information Management Education, the Commission on Accreditation Healthcare Management Education, the Society for Human Resource Management, and the Commission on Public Relations Education.

Despite growing interest in and use of industry partners in capstone courses, what has yet to emerge is a single, unified framework that integrates the various best practices and success factors researchers have identified. We believe that developing a framework that considers the range of variables available in creating an industry-partnered capstone course would assist academic institutions with the design or revision of these experiential and competency-based courses.

Considering that more and more schools are moving to online delivery, and increasing numbers of students are taking classes online [17], it is important to consider the options for providing a real-life learning experience to students in these classes.

III. CAPSTONE COURSES AT UMUC UMUC offers 18 different master’s degree programs in five departments as shown in figure 1. Within each department are multiple master's degrees and various specializations; some programs are structured around six-credit courses and others have mostly three-credit courses. Total degree credits range from 36–42, depending on the program, and most feature a combination of core and specialization courses.


Courses


The Graduate School

Business and Executive Programs

Management Accounting and

Finance

Information and Technology

Systems Cybersecurity Education

Figure 1. UMUC Graduate School Departments

The capstone course may be part of the core or specialization programs. In either case, the purpose is to integrate and apply the learning that has taken place up to that point. However, the approach used to provide the culminating experience varies.

A. Survey of UMUC Graduate Programs A fall 2012 survey of graduate program directors included all 18 graduate programs at UMUC and identified (1) whether the programs were offering a capstone experience and (2) how the capstone experience was offered.

Survey results found that 89% (16) of the programs offer a capstone course. The types of capstone courses at UMUC are listed in table 1. Courses in the "Other" category involve research papers, short assignments, or a combination of several activities.

Projects with Industry 28%/5 Projects (Hypothetical) 17%/3 Case Study 17%/3 Simulation 11%/2 Seminar 5.6%/1 Other 22%/4

Table 1. Types of Capstone Projects within UMUC Graduate School Programs

Across all the programs that offer a capstone course project, there was an almost even split between offering it as a group project, individual project, or a combination of the two. Approximately 35% of programs use a project as a course centerpiece. Only 5 of the 18 programs/specializations offer a capstone project with the industry. Of these, two are part of the core courses and the remaining three are included in the specialization courses. The authors’ research and experience found that several factors contribute to the successful integration of an industry-sponsored project into the capstone course. These factors include a very structured course, a robust and sustainable approach to soliciting projects from the industry, course design that would ensure accountability and timeliness, clearly outlined expectations for the companies and the students, a fair and comprehensive evaluation of student learning and a product that fulfills the goals of the project.



IV. FRAMEWORK FOR DESIGN OF CAPSTONES WITH INDUSTRY-SPONSORED PROJECTS

Integrating industry/client-sponsored projects into capstone courses is not without challenges, especially for large programs and those operating in asynchronous environments. Issues of scalability, varying degrees of deliverable quality, students’ abilities to work with teams asynchronously and in different time zones, and the time/effort required to identify potential partners/clients with appropriate projects are all issues that must be addressed. A model or framework that incorporates best practices and is flexible enough to adapt to the needs of a program can be useful to institutions considering the development of a capstone course or modifying an existing one.

Based on the UMUC capstone course survey results and literature review, we developed a framework as illustrated in figure 2. This framework has four components that we found to be essential in the successful design and implementation of an industry-sponsored capstone project. Each component encompasses elements that provide options and issues to consider. Taken together, these elements and issues determine the ultimate nature of the project and the student experience.

Figure 2. Proposed Capstone Framework

The four components of the framework are interdependent, as shown by the arrows in figure 2. The project selection process, including the nature of the project, determines the roles and responsibilities of the various stakeholders (students, faculty, and clients/sponsors) and the course design. Two components depicted in figure 2, Roles and Responsibilities and Course, affect the design of the evaluations. Evaluations lead to revision and changes in all components of the framework.

Table 2 provides a detailed tabular view of the framework, showing various capstone course options. Each component of the course is broken into several elements. The items specified under the Capstone Course Design Options section correspond to each element and are presented in columns to facilitate the selection of a design option. In cases where an element spans the entire category (e.g., Project goals align with course outcomes), the element is considered essential for all projects.


Courses


Component Elements Capstone Course Design Options Projects Solicitation

Students generate clients/projects following established criteria

Faculty member solicit/secure projects

Nature or Type Project goals align with course outcomes No financial contributions from client/sponsor

Sponsor/client provides financial support to institution, funds research, and/or pays fee

Problem based: students develop proposals with or without implementation

Results based: implementation with outcomes specified by client/sponsor

Confidentiality/privacy agreement(s) not required

Confidentiality/privacy agreement(s) required

Client is like a customer Sponsor is like mentor or coach Structure The project scope has well defined goals and objectives that are realistic for the length

of the semester(s) Individual work Team-developed work Competitive: similar to a request for proposal (RFP) process, students compete to develop winning proposal for same client

Collaborative: Students work on team or independent projects, but share ideas, assist each other, and do not compete to submit a winning proposal

More than one student or group works on the same project

One person/team per project

Course Structure One term/semester Two or more terms/semesters One course Two or more courses Majority of time focused on the project Project time augmented with lectures,

discussions, and additional readings Deliverables Various types of assignments (research

paper, report, proposal, presentation, quiz, exam, visual material, etc.)

Single assignment, typically a research paper, proposal, or report

Components due and graded throughout the semester(s)/term(s)

Majority of work due at the conclusion of the semester(s)/term(s)

Students incorporate feedback into final deliverable

Informal feedback provided throughout the semester without any graded components

Interaction Asynchronous environment Synchronous or face-to-face environment Faculty member mediates interaction between student and sponsor/client

Students have direct interaction with sponsor/client

Roles and Responsibilities

Faculty Facilitator Mentor Reviews work before final submission to the client

Provides ongoing guidance to student/teams

Establishes and enforces deadlines and client expectations as outlined in the client’s project summary

Establishes and enforces deadlines but client manages own expectations for the project goals

Client/ Sponsor Arms-length from students or interacts minimally with students directly

Works directly/closely with students, including regular meetings and feedback

Students Each individual student is responsible for all elements of the project

Students work in teams with defined roles (assigned or member negotiated)

Evaluations Students’ Work Grading rubrics Faculty member reviews work before students’ proposals are submitted to the client/sponsor

Client/sponsor reviews students’ work before it is submitted for grading

Faculty grading Client/sponsor evaluations Student self-evaluation Team member evaluations Formative Summative

Course/ Project Student evaluations using questionnaires

Client/sponsor evaluations using questionnaires

Faculty Student evaluations using questionnaires Table 2. Detailed Framework View



As illustrated in table 2, the proposed framework for designing capstone courses with industry-sponsored projects has four main components: Projects, Course, Roles and Responsibilities, and Evaluations. Additional details for each component follow.

A. Projects The main component of the capstone course is a project with an industry partner or client.

1. Solicitations The first and critical part of the course is to secure projects from industry organizations. a) Student identified: Asking students to identify and bring in projects that meet established criteria is a

good option for a couple of reasons. First, it is more likely to be a project in which the student has a vested interest. Second, it lessens the demands on the faculty or program manager. However, this option could reduce the time to work on the project because the process only starts once the semester begins. Some form of advanced project approval before the start of a semester or term could provide students with more time to work on the project once the semester/term begins.

b) Faculty or program manager identified: Having the instructor or program manager secure the projects for students to select on the first day of class enables students to start right away. Repeat clients/sponsors, industry contacts, adjunct faculty who work in the industry, alumni, and contacts through professional organizations are all possible resources for projects. The substantial time commitment on the part of the instructor or program manager to secure projects before the start of the semester is a major challenge with this option.

2. Nature or Type The type of project selected for an experiential capstone course must be appropriate to the goals of the program and must involve the following variables: a) Project goals: Project goals should follow the outcomes established for the course. The outcomes in

turn align to the program outcomes, which are mapped to the institution’s outcomes. b) Funded or unfunded projects: A particular project may receive financial support from the client and

may or may not be supported by the client or sponsor in the form of a fee or financial support to the institution. In general, there are more cases of non-funded projects than funded projects.

c) Problem or results-based projects: Problem-based projects allow students to develop proposals with or without implementation. Results-based projects allow students to present and implement results based on outcomes specified by the client/sponsor.

d) Confidentiality/privacy: Depending on the nature or type of client and the work outlined in the project, it may be necessary and desirable for students to sign confidentiality agreements.

e) Client vs. sponsor role for industry partner: Defining the nature of the relationship with the industry partner is key to determining other critical elements of the course. As a client, the industry partner is generally only minimally involved in the students’ work and does little more than respond to questions about the client organization or the project. As a sponsor, the industry partner takes an active role in working with students.

3. Structure The capstone course involving industry partners could be structured or designed several different ways, depending on the goals of the program and the nature or types of projects selected. a) Scope of work: It is essential to work with the industry partner to ensure the scope of work is

appropriate for the students’ level of performance and to ensure that the projects can be realistically accomplished in the prescribed length of time for the course.

b) Individual or team projects: The course can be structured such that each student works independently or as part of a team.

c) Competitive or collaborative: The course can be designed to provide a competitive environment to have students compete to submit the “winning” proposal determined by the client. In these cases, students work to develop proposals for the same client. The course can also be designed to have


Courses


individuals or teams of students develop and/or implement projects on behalf of different clients to promote a much more collaborative environment.

d) Number of students per project: Depending on the project structure, it is possible to have an entire class work on one project or have students work individually or in teams on distinct and separate projects.

B. Course The overall course should have the following key elements that are essential in providing a successful and meaningful experience to the stakeholders.

1. Structure There are several options for structuring the capstone course involving industry partners.

a) Length: The capstone course may be one or two semesters/terms in length. b) Number of courses: If the course is two semesters in length, it may be split into two separate courses.

For example, the first course could be focused on developing a proposal and the second course could be focused on implementing the proposal.

c) Percentage of project-focused time: The course may be centered on the project, which then carries more than 50% of the course grade, or the focus on the project may be a smaller part as one of several deliverables/activities.

2. Deliverables The nature and type of deliverables students produce should be consistent with course goals and client/sponsor expectations. a) Types of assignments: Depending on the nature of the project, instructors can incorporate a variety of

graded assignments (e.g., exams, quizzes, reports, proposals, etc.) or have students focus on a single major deliverable (such as a paper, proposal, or report).

b) Assignment submittals: The number and nature of deliverables that students (individuals or teams) are asked to provide should ensure that the project is moving in the right direction at the right pace and with each individual doing a fair share of the work. A single graded deliverable at the end of the course or graded deliverables throughout the term/semester are options for assignments. For a team project, the most common use of assignments is a mix of individual and group contributions.

c) Timing and use of feedback: If the course includes multiple deliverables, students may have the opportunity to incorporate feedback into improvements on the final deliverable. In the case of a final major deliverable, students may only receive informal feedback throughout the project.

3. Interaction Whether the class is asynchronous or face-to-face/synchronous has implications for how the course is designed and the roles and responsibilities established. a) The interactions may be synchronous (conference calls, video meetings) between the students and the

sponsor/client or asynchronous (e-mails, posting in a virtual classroom). b) Depending on the structure of the course and project, the faculty member may mediate interaction

between students and the clients/sponsors or students may interact directly with their clients/sponsors. C. Roles and Responsibilities

The roles and responsibilities of the instructors, client/sponsor, and students vary depending on the goals, course structure, and expectations for the project.

1. Faculty The faculty member’s role is dependent on the nature of the project, design of the course, and needs of the students. a) Roles: The faculty member may take on a role of facilitator or mentor.



b) Review process for students’ work: The faculty member may review and/or grade students’ deliverable(s) before sending them to the client, or the faculty member may rely on the client to provide the primary feedback regarding project quality and student performance.

c) Responsibility for establishing and enforcing deadlines and expectations: The faculty member may choose to enforce deadlines and ensure that client expectations are met or establish and enforce deadlines but leave it to the client/sponsor to manage the expectations for the project.

2. Client/Sponsor How much or how little involvement the client/sponsor has in the students’ work depends on the structure of the projects and course goals. a) Amount of involvement: The role of the client/sponsor could range from minimal, arms-length

involvement to working closely with students on the completion of a project. When the client/sponsor is closely involved in the progress and direction of the project, the client interacts with the student/team on a regular basis and reviews each deliverable to ensure that the objectives are being addressed appropriately before students submit projects for grading.

3. Students Students can be organized to accomplish the work in teams, individually, or some combination of team and individual assignments.

a) Responsibility for work product: The project may be individual, with each student completing the entirety of the work; team based, with students organized into teams to accomplish the project work; or some combination of individual and team-based work may be required. Students working individually have found it most useful to have tasks broken down into smaller parts that are due periodically rather than having an entire work product due at one time. Whether the instructor assigns roles to each member of a team or the team negotiates roles, it is most helpful to the overall progress of the project if the roles include a project manager, sponsor liaison (if allowed within the structure of the course), researcher(s), writer(s)/drafter(s), and an editor.

D. Evaluations Measuring results to assess learning and the ability to apply skills and knowledge is always an important element in education. 1. Evaluations of Students’ Work

While the faculty member is ultimately responsible for assigning grades, the faculty member may have varying degrees of input into evaluating the students’ deliverables for the project.

a) Rubrics: In all cases, a rubric that outlines the performance expectations and criteria for successful project completion is important for ensuring clarity and consistency in the grading process.

b) Grading/evaluation: Determining whether the faculty member grades work before it is submitted to the client/sponsor or whether students first submit their work to the client/sponsor for feedback before grading depends on the faculty and client/sponsor definitions.

c) Student/peer evaluations: The overall grading process could include student self-evaluations and/or student peer evaluations. A relatively small percentage of the final project grade (such as 5 to 10%) may be assigned to individual and/or peer assessments.

d) Formative vs. summative assessments: Most of the grading and feedback occurs at the end of the semester/term in summative assessments. Grading and feedback throughout the semester/term forms the basis for formative assessments of the students’ work.

2. Course/Project A standard feature in most educational settings is end-of-semester evaluations of a course.

a) Student opinions gathered through questionnaires are the primary source of course evaluations. b) Clients/sponsors may also be asked to provide feedback. 3. Faculty

A standard feature in most educational settings is the end-of-semester evaluation of the instructional faculty.


Courses


a) Student opinions gathered through questionnaires are the primary source of course evaluations.

V. TESTING THE FRAMEWORK: TWO UMUC CAPSTONE COURSES

The authors of this paper offer capstone courses in their own disciplines, biotechnology (BIOT) and public relations (PRPA), with industry/client sponsored projects. The proposed framework was tested with their courses as the first stage of testing of the framework (see tables 3 and 4). The design options selected from the framework by the BIOT and PRPA capstone course are depicted by the gray-shaded cells.



Faculty member solicits/secures projects


Sponsor/client provides financial support to institution, funds research, and/or pays fee





Client is like a customer Sponsor is like mentor or coach Structure The project scope has well defined goals and objectives that are realistic for the length of

the semester(s) Individual work Team-developed work Competitive: similar to a request for proposal (RFP) process, students compete to develop winning proposal for same client

Collaborative: students work on the same or independent projects, but they share ideas, assist each other, and do not compete to submit a winning proposal



Table 3. BIOT Project Framework Selections



Faculty member solicits/secures projects


Sponsor/client provides financial support to institution, funds research and/or pays fee





Client is like a customer Sponsor is like a mentor or coach Structure The project scope has well defined goals and objectives that are realistic for the length of

the semester(s) Individual work products Team-developed work products Competitive: similar to a request for proposal (RFP) process, students compete to develop winning proposal for same client

Collaborative: students work on the same or independent projects, but they share ideas, assist each other, and do not compete to submit a winning proposal



Table 4. PRPA Project Framework Selections



The design of two sample graduate-level capstone courses was mapped based on the elements of the first component of the framework (table 3) showing how design options from column 1 or 2, or a combination of the two can be part of the same course. One example of both options being part of the same course is confidentiality agreements (Projects---Nature/Type) in the BIOT and PRPA programs, which may or not be required depending on the needs of the client/sponsor.

Component Elements Capstone Course Design Options Course Structure One term/semester Two or more terms/semesters

One course Two or more courses Majority of time focused on the project Project time augmented with lectures,




Components due and graded throughout semester(s)

Majority of work due at the conclusion of semester(s)/term(s)

Students incorporate grading feedback into final deliverable

Students receive informal feedback and guidance without any graded components


Students have direct interaction with the sponsor/client

Table 5. BIOT Course Framework Selections

Component Elements Capstone Course Design Options Course Structure One term/semester Two or more terms/semesters

One course Two or more courses Majority of time focused on the project Project time augmented with lectures,




Components due and graded throughout semester(s)

Majority of work due at the conclusion of semester(s)/term(s)

Students incorporate grading feedback into final deliverable

Students receive informal feedback and guidance without any graded components


Students have direct interaction with the sponsor/client

Table 6. PRPA Course Framework Selections

The BIOT and PRPA capstone courses use similar design options for the Course component. The difference is in the types of assignments required as deliverables and the nature of interaction between the three stakeholders. In BIOT, students/teams have direct interaction with the sponsor, while in PRPA, the faculty member mediates the interaction between student/teams and the client.

The differences in design between the BIOT and PRPA courses in terms of roles and responsibilities of the students, faculty, and client/sponsor is significant, as illustrated in tables 7 and 8. The evaluation selections are also provided in these tables.


Courses


Component Elements Capstone Course Design Options Roles and Responsibilities


Provides general guidance to student/teams

Establishes and enforces class assignment deadlines and client expectations

Establishes and enforces deadlines but client manages own expectations

Client/ Sponsor

Arms-length from students or interacts minimally with students directly



Students work in teams with defined team member roles (assigned or team member negotiated)

Evaluations Students’ Work

Grading rubrics

Faculty member reviews work before students’ proposals are submitted to the client/sponsor


Faculty grading Client/sponsor evaluations Students submit self-evaluation Team member evaluations Formative Summative

Course Project

Student evaluations using questionnaires Client/sponsor evaluations using questionnaires

Faculty Student evaluations using questionnaires Client/sponsor evaluations using questionnaires

Table 7. BIOT Roles and Responsibilities and Evaluation Framework Selections

Component Elements Capstone Course Design Options Roles and Responsibilities


Provides general guidance to student/teams

Establishes and enforces class assignment deadlines and client expectations

Establishes and enforces deadlines but client manages own expectations

Client/ Sponsor

Arms-length from students or interacts minimally with students directly



Students work in teams with defined team member roles (assigned or team member-negotiated)

Evaluations Students’ Work

Grading rubrics

Faculty member reviews work before students’ proposals are submitted to the client/sponsor


Faculty grading Client/sponsor evaluations Students submit self-evaluation Team member evaluations Formative Summative

Course Project

Student evaluations using questionnaires Client/sponsor evaluations using questionnaires

Faculty Student evaluations using questionnaires Client/sponsor evaluations using questionnaires

Table 8. PRPA Roles and Responsibilities and Evaluation Framework Selections

VI. DISCUSSION The research is generally consistent regarding the reasons for offering capstone courses that include real-life projects offered by industry partners for students [2, 3, 4, 5, 8, 12, 13, 14]. The real question then, is what drives the decisions in the construction of the capstone course?

The first consideration is how to ensure a course involving industry partners offers experiential learning opportunities that are appropriately aligned with the mission and goals of the institution and the specific



academic program. Since the intent of this type of capstone course is to provide students with meaningful, portfolio-worthy demonstrations of the knowledge and skills they have gained throughout their courses of study, the curriculum preceding the capstone must provide the necessary foundation for successful performance in the capstone course.

The second consideration is the availability of resources, both in terms of faculty/institutional support and a pool of potential clients. Capstone courses, particularly those that incorporate industry partners, are especially demanding on faculty because of the often intense engagement the faculty member must have throughout the process. Available and interested faculty who are committed to the course demands are essential. Depending on the size of the program and numbers of students participating in a given capstone course, more than one instructor may be needed.

Securing projects can be a time consuming process unless a program has a pool of potential clients/sponsors or consistent participation from existing clients/sponsors. Assessing the ongoing availability of industry partners is critical to developing a sustainable capstone program that uses these types of projects. For programs with large numbers of students, decisions about course structure and whether multiple students can work on a single project determines the number of industry projects needed for a capstone course. For example, the biotechnology master’s degree program at UMUC has approximately 475 students. The course is structured to assign one team of students (4–5 members) to one project. This means that the program requires several projects each semester. In the spring 2013 semester, this program offered three sections of the capstone course with a total of 68 students. Based on the model used, the program manager secured 17 projects to accommodate all of the sections and students.

The public relations program is smaller (approximately 250 students) and is structured differently. Depending on students’ interests, they can either identify and develop a project for their own client or work individually to develop competitive proposals in response to a pre-selected client’s problem statement. This process is designed to mirror the request for proposal (RFP) process that is common in the industry. With this course design flexibility, fewer industry partners are required for each semester. For example, in the spring 2013 semester, there was one section of the capstone course with 21 students enrolled. Several students elected to work on one client project while other students identified their own client projects.

The framework presented here is intended to allow educators to see the variety of options available in designing a capstone course that uses industry-sponsored projects. The framework identifies key variables and allows educators to pick and choose elements that make the most sense for the program, the students, and the goals established for the course.

VII. CONCLUSION Increasingly, capstone courses are becoming an integral and necessary part of graduate education, particularly in online academic programs. Significant educational and professional benefits are created from offering experiential learning opportunities that include industry-sponsored capstone projects in culminating courses.

The framework proposed here should address the needs of a range of capstone course designs. Framework testing in two diverse capstone courses offered the first validation of this model. However, additional research is needed to test and refine the framework to ensure it is inclusive of the range programs that can include a capstone experience.

VIII. ABOUT THE AUTHORS Rana Khan is a professor and Chair of the Information and Technology Systems department at the University of Maryland University College. She is also the director of Professional Science Master’s program in Biotechnology. She has published articles on integrating industry-sponsored projects in online capstone courses and designing and implementing an E-mentoring program for online graduate students.


Courses


Prior to her work in online education, she was a research scientist. She holds a Ph.D. in Cell and Molecular Biology from the University of Maryland at College Park.

Julie Hill is a professor and director of the Public Relations Program in the Graduate School of the University of Maryland University College. Prior to her appointment there, she spent most of her career in local government and law enforcement public affairs. She is co-author of the book, Strategic Communication Practices: A Toolkit for Law Enforcement Executives, published by the DOJ/COPS office in 2012. Julie earned an undergraduate degree from Emerson College in Boston, Massachusetts, and her M.S. and Ph.D. from Bowling Green State University in Ohio.

IX. REFERENCES 1. Kelly, R. Capstone Courses Prepare Students for Transition to Working World. Academic Leader,

22(2): 8 (2006). 2. Farr, J., Lee, M., Metro, R., and Sutton, J. Using a Systematic Engineering Design Process to

Conduct Undergraduate Engineering Management Capstone Projects. Journal of Engineering Education: 193–197 (2001).

3. Aller, B.M., Kline, A.A., and Tsang, E. Work in progress: Improving the Senior Capstone Design Experience through Shared Perspectives and Best Practices. 34th ASEE/IEEE Frontiers in Education Conference, Savannah, GA (October 20–23, 2004).

4. Bruhn, R.E., and Camp, J. Capstone Course Creates Useful Business Products and Corporate-ready Students. Inroads—The SIGCSE Bulletin, 8, 36(2): 89–92 (2004).

5. Gorka, S., Miller, J.R., and Howe, B.J. Developing Realistic Capstone Projects in Conjunction with Industry. SIGITE: 27–31 (2007).

6. Berheide, C.W. Doing Less Work, Collecting Better Data: Using Capstone Courses to Assess Learning. Peer review 9 (2) (2007).

7. Clark, E., Engel, D., Napolitano, M., Richardson, J., Rodriquez, M., Sterling-Deer, C., and Kasprzak, C. Integration Reflection, Closure & Transition Advancing Capstone Learning at LaGuardia. Report of the Faculty Research Team on Capstone Education (November 15, 2008).

8. Gorman, M.F. The University of Dayton Operations Management Capstone Course: Undergraduate Student Field Consulting Applies Theory to Practice. Interfaces 40(6): 432–443 (2010).

9. Bojanova, I., and Khan, R. Designing Online Capstone Courses in Partnership with Industry. Paper presented at the 26th Annual Conference on Distance Teaching & Learning (August 4–6, 2010).

10. Goold, A. Providing Process for Projects in Capstone Courses. Paper presented at the ITiCSE Conference, Thessaloniki, Greece (2003).

11. Adams, L., Daniels, M., Goold, A., Hazzan, O., Lynch, K., and Newman, I. Challenges in Teaching Capstone Courses. Paper presented at the ITiCSE Conference, Thessaloniki, Greece (2003).

12. Goldberg, J., Cariapa, V., Corliss, G., and Kaiser, K. The Benefits of Industry Involvement in the Multidisciplinary Capstone Design Course at Marquette University. Paper presented at the Capstone Design Conference 2012, Aurora, CO (2012).

13. Hagan, L.M. Fostering Experiential Learning and Service through Client Projects in Graduate Business Courses Offered Online. American Journal of Business Education, 5(5): 623–632 (2012).

14. Estell, J.K., and Hurtig, J.K. Adopting Best Corporate Practices for Capstone Courses. Paper presented at the Capstone Design Conference 2012, Aurora, CO (2012).

15. Nassersharif, B., and Riley, L.A. Some Best Practices in Industry-sponsored Capstone Design Projects. Paper presented at the Capstone Design Conference 2012, Aurora, CO (2012).

16. Conroy, R., and Khan, R. Integrating Virtual Internships into Online Classrooms. Journal of Commercial Biotechnology, 15: 97–11 (2009).

17. Allen, I.E., and Seaman, J. Going the Distance: Online Education in the United States. Wellsley, MA: Babson Survey Research Group and Quahog Research Group, LLC (2011).

the aim of the journal of asynchronous learning networks ... filejournal of asynchronous learning...

Documents