university of southern indiana€¦ · web viewthis means that the data for both the initial set...

University of Southern IndianaRomain College of BusinessSixth Year Review Report

January 15, 2014

Romain College of Business AACSB Sixth Year ReportUniversity of Southern Indiana Business

Table of ContentsI: Introduction..............................................................................................................................................2

Issue Identified........................................................................................................................................2

Expectations for resolution......................................................................................................................2

II: Responses to Issues Identified................................................................................................................3

Concern #1: (a) The Apparent Lack of Maturity of the AOL system – BS/BA program............................3

Program Learning Goals and Learning Objectives...................................................................................5

Learning Goal #1:.................................................................................................................................5

Learning Goal #2:.................................................................................................................................6

Learning Goal #3:.................................................................................................................................6

Learning Goal #4:.................................................................................................................................6

Learning Goal #5:.................................................................................................................................6

Learning Goal #6:.................................................................................................................................7

Learning Goal #7:.................................................................................................................................7

Learning Goal #8:.................................................................................................................................7

Performance Benchmarks for Program Learning Goals...........................................................................7

Outcomes Evidence for Program Learning Goals.....................................................................................8

Program-Level Changes - Curricular and Assessment - Based on the AOL Process..................................9

Explanation of Data Aggregation Using Rubrics.....................................................................................10

Assessment Evidence: MBA Program Learning Goals (#2 -#9 and since Spring 2012 #2 - #5)...............17

Concern #2:...........................................................................................................................................20

Concern #3:...........................................................................................................................................21

III: Appendix..............................................................................................................................................24

1


University of Southern IndianaRomain College of Business

Sixth Year Review Report

I: Introduction

As noted in the letter from the Maintenance of Accreditation Committee, dated April 25, 2013, the Sixth Year Review period provides additional time to address a number of educational quality issues. This section highlights the issued identified and expectations for resolution based on the relevant accreditation standards identified in the April 25, 2013 letter.

Issue Identified 1. Concerns regarding the Assurance of Learning system across the BS/BA and MBA programs

a. It appeared that the AOL system was not fully maturedb. It appeared that the focus was on the assessment of majors instead of programs

Expectations for resolution a. Provide documentation that clearly demonstrates that an AOL system is in place with learning

goals and objectives across each program.b. Be able to show that these learning goals have been measured using rubrics designed for

assessing the activity c. Be able to show that these learning goals are comprehensively evaluated

Where outcomes reveal deficiencies, curricular changes should be made

Issue Identified2. Concern that the criteria for establishing and maintaining PQ status to ensure that professional

experience and continual development activities are consistent with the teaching assignment.a. It appeared that multiple PQ status faculty may not be meeting the requirements for maintenance of PQ status

Expectations for resolutiona. Review the criteria for establishing and maintaining PQ status to ensure that (a) professional

experience and (b) continual development activities are consistent with the teaching assignment.

2


II: Responses to Issues Identified

Concern #1: (a) The Apparent Lack of Maturity of the AOL system – BS/BA program

A number of steps were taken during the past ten months to address this concern. The first involved determination of the criteria for evaluating the maturity of the College’s AOL system. Based on discussions, a review of the relevant literature, and knowledge sharing among faculty members, a five step framework based on a presentation at the 2013 International Conference and Annual Meeting1, was utilized at a faculty retreat on April 16, 2013 to evaluate the College’s AOL system. These activities helped to develop a common understanding and following working definition of a mature AOL system.A mature AOL system is one whose components are well established and function effectively to assure sustainability of the system. In particular, at the program level, there are clearly defined learning goals, with processes and activities that: (a) establish the extent of student learning; (b) serve as catalysts for curricula changes and other responses to learning outcomes evidence; and (c) sustain continuous improvements in curricula management and learning outcomes. In addition, these interdependent components serve to maintain stability of the AOL system while adapting to change resulting from internal and external influences.

Subsequent steps involved activities geared towards clarifying and modifying the College’s AOL system, as well as implementation of the College’s AOL plan. Issues identified during the Peer Review Team visit such as the lack of clarity about data aggregation of results pertaining to skill learning objectives and performance benchmarks for program-level data were also addressed in the process of clarifying the College’s AOL system.

The major outcomes from the steps taken during the past ten months are:

A more precise statement of the number of program learning goals. There are eight program learning goals and previously, there were instances when seven program learning goals appeared to be identified. Sometimes the separate program goals, effective oral communication and effective written communication, were referred to as the effective communication program learning goal and in some instances the distinction between the skills-related learning goals and the knowledge-related program learning goal was the source of this tendency.

Outcomes for the program learning goals and the related performance benchmarks are now more clearly articulated. There is also clearer explanation and greater understanding of the process of aggregating assessment information for learning objectives to measure the learning outcomes for the program leaning goals.

1 Recasting Assurance of Learning by Kathryn Martell and Tracy Taylor at the April 2013 International Conference and Annual Meeting, Chicago. It was noted at the presentation that the framework was based on the 2007 (and subsequently updated in 2013) AACSB paper on Assurance of Learning Standards: An Interpretation.

3


There is now a clearer delineation of responsibilities between the Teaching Learning and Assessment Committee (TLAC) and the Dean’s office with regard to the collection, storage, analysis, recommendations, and dissemination of assessment information as well as the coordination of curriculum changes based on assessment information. Collection, compilation, and storage of assessment information for program learning goals are the responsibility of the Dean’s office. The TLAC is responsible for analysis, dissemination, recommendations, and coordination of curriculum changes.

The assessment plan has been modified so that assessment of all program learning goals is no longer conducted every semester. The modified assessment plan requires four program goals to be assessed every year. In 2013, the four program learning goals assessed were the knowledge-related program learning goal, and the teamwork, ethical decision-making, and analytical problem solving program learning goals. We are awaiting results for the major field test in business from the ETS conducted in Fall 2013 and the outcomes for the other three program learning outcomes are presented in the following section of the report. Assessment information for the prior years can be accessed at: http://business.usi.edu/aacsb/

The rubric associated with the technology-related program learning goal has been modified as a result of our AOL review during the past several months. (Appendix)

Awareness of a gap with regard to comprehensive assessment of the technology program goal at the end of the program has led to the decision to conduct the assessment of the technology program learning goal in the BS/BA program’s capstone course. This will be implemented in conformity with the assessment plan during 2014.

Increased focus on reviewing the current program learning goals in the context of the themes of innovation, engagement, and impact with regard to continuous improvement contained in the 2013 business accreditation standards.

As indicated in Figure 1, processes and activities associated with each of the identified components have been in place since 2007 and the connections among these components have been enhanced since the Peer Review Team’s visit. The maturity of the AOL system is reflected in the systematic and ongoing manner in which assessment information and insights provided about student learning have impacted curriculum management. In addition, the capacity of the AOL system to implement changes prompted by the Peer Review Team’s visit and most recently, begin the process for incorporating the 2013 accreditation standards is reflective of its stability and effectiveness. Steps taken and a status report on the progress made toward a review of the business core curriculum are available at: http://business.u si.edu/aacsb/

4

http://business.usi.edu/aacsb/



Program Learning Goals and Learning Objectives

We have eight learning goals for the undergraduate BS/BA program. Each learning goal has associated learning objectives, which allow for the assessment of learning goals. What will students learn?

Learning Goal #1: - Students will demonstrate a general understanding of the fundamental areas of business.

How will we know students demonstrate general understanding of the fundamental areas of business? Students will demonstrate a general understanding based on their performance in the Major Field Test in Business developed by the Educational Testing Service (ETS). Assessment information with regard to this program learning goal is obtained through administration of this test to undergraduate students in the Policy Formulation & Implementation course every Fall semester. The results of these tests are compared to the performance of other students across the country in the following areas:i. Accountingii. Economicsiii. Financeiv. Information systemsv. Management

5


vi. Marketingvii. Legal/Socialviii. Quantitativeix. International

Learning Goal #2: -Effective critical thinking skills.How will we know students demonstrate effective critical thinking skills? Students will demonstrate competency in the following critical thinking learning objectives:i. Use relevant information/evidenceii. Consider relevant viewpointsiii. Identify relevant assumptionsiv. Consider implications of alternative courses of actionv. Develop clear logical conclusions from prior analysis

Learning Goal #3:-Effective ethical decision making skills.

How will we know students demonstrate effective ethical decision-making skills?Students will demonstrate competency in the following ethical decision making learning objectives:i. Identify ethical issue(s)/problem(s)ii. Incorporate evaluation of relevant stakeholdersiii. Evaluate implications of alternative courses of actioniv. Develop logical conclusions from prior analysis

Learning Goal #4:-Effective analytical problem solving skills.How will we know students demonstrate effective analytical problem solving skills?Students will demonstrate competency in the following analytical problem solving learning objectives:i. Incorporate all relevant concepts and accurately use themii. Incorporate all relevant informationiii. Correctly utilize analytical operationsiv. Interpret logically from the prior analysis

Learning Goal #5:-Effective oral communication skills.How will we know students demonstrate effective oral communication skills?Students will demonstrate competency in the following oral communication learning objectives:i. Use an effective opening statementii. Follow a very clear organizational plan and reinforce it periodicallyiii. Include content clearly relevant to topic/purposeiv. Consistently use appropriate voice, pace, eye contactv. Show enthusiasm/vitalityvi. Use standard English and appropriate vocabularyvii. Use clear and effective summary, conclusion, and/or recommendation and closing

statement

6


7


Learning Goal #6:-Effective written communication skills.How will we know students demonstrate effective written communication skills?Students will demonstrate competency in the following written communication learning objectives:i. Use clear statement of purposeii. Use clear organization and logiciii. Use clear, accurate, and relevant information and conceptsiv. Use standard English and appropriate vocabularyv. Use clear and effective summary, conclusion, and/or recommendation and closing statement

Learning Goal #7:-Ability to work effectively in teams.How will we know students can work effectively in teams?Students will demonstrate competency in the following team learning objectives:i. Take initiative in helping group set goals and get organizedii. Thoroughly complete assigned tasks on/ahead of scheduleiii. Clearly communicate ideas and consistently listens to others’ ideasiv. Actively participate in helping group work cohesively (give productive feedback and

willingly accepts feedback)

Learning Goal #8:-Working knowledge of current technologies. How will we know students have a working knowledge of current technologies?Students will demonstrate competency in the following technology learning objectives:i. Show basic competency in spreadsheet application skillsii. Show basic competency in word processing application skillsiii. Show basic competency in presentation software application skillsiv. Show basic competency in database software application skills

Performance Benchmarks for Program Learning Goals Subsequent to the Peer Review Team’s visit we have clarified and documented program-level performance benchmarks, reduced the number of program goals that are assessed from eight every year to four every year, as well as the corresponding assessments for each of our learning goals. In addition, this section details how learning objective assessment is integrated with program-level learning goal assessment.

Students in all majors will demonstrate:

Learning Goal #1: General understanding of the fundamental areas of business. Benchmark for Learning Goal #1: Students will meet or exceed the national average for the overall ETS Major Field Test score (Table 1). A failure to meet or exceed the national average for the overall ETS Major Field Test score will trigger a more specific review of discipline area scores.Rubrics for each of the program learning goals (#2 - #8) have been used since 2007 to determine student competency in associated learning objectives.

8


Table 1: Romain College of Business - Undergraduate Major Field Test Scores – Percentiles and Averages - 2008 – 2012Year/Semester Acct. Econ. Mngt. Quant. Fin. Mktg. Legal/

SocialInter-

nationalInfo.

Systems

USI Romain College of Business Average

National Average

2008 (Fall) 52 52 60 48 64 53 49 58 58 156.0 151.62009 (Fall) 53 52 62 49 65 57 51 61 61 157.0 151.62010 (Fall) 52 52 61 48 62 56 52 59 59 156.0 151.32011 (Fall) 47 48 59 44 50 60 57 56 56 154.0 150.32012 (Fall) 48 48 58 42 50 62 61 58 50 155.0 150.3

Learning Goal #2: Effective critical thinking skillsLearning Goal #3: Effective ethical decision making skillsLearning Goal #4: Effective analytical problem solving skillsLearning Goal #5: Effective oral communication skillsLearning Goal #6: Effective written communication skillsLearning Goal #7: Ability to work effectively in teamsLearning Goal #8: Working knowledge of current technologies

Benchmark for Learning Goals #2 - #8: Seventy-five percent of students will consistently meet or exceed the expectations for the overall performance benchmark associated with each skill learning goal (Table B). A five-year rolling average will be used to determine the overall performance benchmark. A failure to meet or exceed the overall benchmark will trigger a more specific review of yearly performance scores and associated learning objective trait performance.

Outcomes Evidence for Program Learning Goals As Table 1 shows, student performance for graduating seniors in the Romain College of Business in terms of average scores exceeded the national average score between 2008 and 2012. While student performance across topic areas and over time is relatively consistent the areas with relatively lower percentiles such as quantitative, accounting, and economics have resulted in steps to conduct a detailed item analysis of the major field test in business to determine the concepts, theories, and principles in which students are underperforming.

Outcomes evidence for program goals #2 to #8 are provided in Table 2. In addition to evidence of learning outcomes for the 2008-2012 period, results for 2013 reflect the implementation of a modified assessment plan established subsequent to the Peer Review Team’s visit. The modified assessment plan represents a shift from conducting assessment activities for all program learning goals every semester to a select number of program learning goals each year. The modified schedule for assessment activities is shown in the Appendix. As Table 2 shows, between 2008 and 2012 students met or exceeded the overall performance benchmark (averaged across the five-year assessment time period for each program goal) for all program goals except analytical problem solving. At the same the results for 2013 are suggestive of the positive impacts that curriculum interventions are having on student performance related to the analytical problem solving learning goal.

9


Table 2: Outcomes – BS/BA Program Learning Goals (2008 – 2013)

Year

Knowledge of Current

Technologies

Analytical Problem

Solving Skills

Critical Thinking

Skills

Ethical Decision Making

Oral Comm. Skills

Written Comm. Skills

Work Effectively in Teams

2008 77 69 76 91 72 82 98

2009 77 69 85 90 92 78 97

2010 74 62 76 85 74 82 89

2011 75 66 77 77 64 94 72

2012 72 72 71 71 85 84 86

Average 75 68 77 83 77 84 88

2013 93 86 86

Program-Level Changes - Curricular and Assessment - Based on the AOL Process Within the College’s AOL framework the aggregate results for each of the learning goals serve as the impetus for program-level curriculum changes. Program-level assessment evidence on the learning objectives for each learning goal also provides the basis for curriculum changes. The aggregate results shown in Table 2, in combination with the 75 percent performance benchmark account for the emphasis placed on curriculum changes related to the analytical problem solving learning goal. In response to the less than desired learning outcome in 2008, attention was initially placed on identifying variations in student learning outcomes for each of the learning objectives associated the analytical problem solving learning goal. In order to do so, variations in the learning objectives were monitored over a two-year period after the results were observed at the end of 2008.

Based on the observed patterns in 2009 and 2010, it became evident that students were consistently underperforming in the following learning objectives: interpretation of results and use of the appropriate analytical operations. This assessment information was disseminated and faculty members providing instruction in courses that introduced and reinforced the analytical problem solving learning goal engaged in discussions about appropriate interventions. The next phase involved an increased emphasis on course activities and assignments focused on recommendations and interpretation of results. In addition, a review of the curriculum map with regard to the introduction and reinforcement of the analytical problem solving skill at the program level, led to a focus on redesigning the Business Statistic course, which all business majors are required to take. All faculty members providing instruction in the Business Statistics course were engaged in that activity and a principal outcome was the adoption of a business process oriented approach as the guiding framework for the curriculum redesign of the Business Statistics course.

10


The redesigned course now has a reallocated mix of homework and in-class assignments and utilizes a case study particularly oriented to using data to make decisions. We have also adopted the use of a software platform which makes it easier to identify patterns in data for ease in interpretability.

The new approach was piloted in Fall 2012. In addition to this course redesign, as part of the process of faculty engagement and immersion in assessment activities, curriculum adjustments in response to shared assessment information about the analytical problem solving learning goal and associated learning objectives were also implemented in several other required courses. As Table 2 shows, the results of the Fall 2013 assessment of the analytical problem solving learning goal indicate a marked improvement in 2013.

In addition, based on our review, the rubric used to assess the learning objectives for the use of technology program learning goal did not provide adequate diagnostic value. As a result, a designated group of faculty members developed a modified technology use rubric that is more specific in terms of traits for the identification of strengths and weaknesses for learning objectives assessment (See Appendix).

Explanation of Data Aggregation Using Rubrics The rubrics for ethical decision-making, teamwork, and analytical problem solving are utilized to assess skills of groups of graduating seniors. For the ethical decision making and analytical problem solving learning goals, student responses pertain to questions on customized assignments developed by the Teaching, Learning Assessment, and Curriculum Committee. The graduating seniors sampled are enrolled in different sections of one of the last courses that graduating seniors take - Career Planning and Professional Development course (BCOM 401). Students must have completed 90 hours in order to enroll in BCOM 401 (a required program course), which provides the best opportunity to assess skill and content development at or near the “exit point” of the program. The cases were not part of the course nor were the scores from the cases included in the calculation of the course grade. Further, the assessment was completed after the conclusion of the course, so that it was not viewed as a course component.

As an illustration, the following Tables (3-5) show the results of assessments related to teamwork, ethical decision-making, and analytical problem-solving cases from samples of graduating seniors enrolled in 2013. These results provide examples to illustrate the yearly process that is used to determine the percentage of students meeting or exceeding expectations for learning objectives associated with program learning goals. The outcomes for program learning goals indicated in Table 2 are averages (equally weighted across learning objectives) calculated and then summed for the Meets Expectations and Exceeds Expectations categories. For these assessments, scoring was performed independently by two trained graduate students.

Table 3: Teamwork skills – Spring 2013Teamwork Skills(n = 92)

Does not MeetExpectations

MeetsExpectations

Exceeds Expectations

Organization and Goal Setting 18% 68% 13%Task Completion 0% 58% 42%Cooperation 23% 58% 20%Development and Roles 14% 61% 25%

11


Table 4: Ethical Decision Making – Fall 2013Ethical Decision Making(n = 24)

Does not Meet Expectations

MeetsExpectations

Exceeds Expectations

Identification of an ethical issue 4% 58% 38%Stakeholder Analysis 13% 46% 41%Implication / Consequences 25% 67% 8%Conclusion / Solution 13% 63% 24%

Table 5: Analytical Problem Solving – Fall 2013Analytical Problem Solving(n = 26)

Does not MeetExpectations

MeetsExpectations

ExceedsExpectations

Concepts 4% 58% 38%Information 0% 58% 42%Analytical Operations 12% 73% 15%Interpretation 12% 69% 19%

Assessment reports of this process are disseminated each semester, typically during the designated University Assessment Day, and discussed with faculty and are also used as the basis for making program-level modifications (see, Appendix for hyperlinks to assessment evidence for other program learning goals during the 2008 – 2012 period).

Our self-review also related to concerns raised by the most recent Peer Review Team that noted some additional weaknesses in our AOL processes. Some program-level assessments are conducted in sophomore-level classes (some communication and technology use learning goal assessment); therefore, results might not be the best indicator of student performance close to graduation. As a result, decisions were made to move these assessments to students at the senior level. Further, it was recognized that better communication of the program-level assessment process is required so that faculty have a better shared understanding of the process. To this end, we have addressed AOL issues in additional faculty meetings beyond the designated assessment day meetings where faculty have been made aware of deficiencies and where faculty input has helped address deficiencies. Specifically, topics related to learning goals and objectives, performance benchmarks, assessment data collection and scheduling, evaluation processes, and “closing the loop” topics have been addressed in spring and fall faculty meetings.

An important issue discussed in the Spring 2013 semester was the need to clarify roles and responsibilities in the AOL system. Defining goals and objectives, aligning curriculum, identifying measures, and acting upon AOL outcomes and overall AOL communication will be coordinated by the Teaching, Learning, and Assessment Committee (TLAC) while the yearly collecting, analyzing, organizing, and disseminating results will be coordinated by the Dean’s office. Since some membership changes yearly on the TLAC (based on faculty elections) for the sake of continuity, all assessment data will be housed in the Dean’s office.

12


In summary, our response is based on concerns noted in the Maintenance of Accreditation Committee letter as well as observations made by the Peer Review Team. In this section, we have focused on concerns related to the AOL system as it relates to the undergraduate BS/BA program. Specific concerns addressed relate to the lack of clarity of the presentation of the program-level process. The presentation of this information now conforms to standards with a specific focus on articulating learning goals and objectives clearly; there is also greater understanding among faculty members about how program-level student performance is recorded and analyzed as well as how it initiates curriculum and AOL changes.

Concern #1: (b) The apparent lack of maturity of the AOL system – MBA Program Learning Goals and Learning Objectives

Between 2006 and 2010 there were nine learning goals for the MBA program. Based on insights provided by the MBA AOL system, the number of program learning goals was changed during 2011, and since Spring 2012, the number learning goals for the MBA program has been five. Important contextual information for the MBA program’s AOL system is the structure of the program. The MBA program is comprised of ten courses that all MBA students are required to complete. Based on the size of the MBA program, only one section is typically scheduled when each course is offered. Learning Goal #1: Students will demonstrate a general understanding of the fundamental areas of business.

Similar to the undergraduate program, the MBA program uses a standardized test for content area assessment. Prior to the last accreditation review there were no formal assessment techniques being employed to evaluate knowledge assessment for the graduate program other than in-class exams, assignments, and presentations. However, the Graduate Council, in conjunction with the Romain College of Business graduate faculty, have since adopted the major field test (MFT) developed by the Educational Testing Service (ETS) as a benchmarking tool for assessing business knowledge at the graduate level. The MFT is used as a measure of students’ content knowledge in the following five areas:

1. Marketing2. Management3. Finance4. Managerial Accounting5. Strategic Integration

The MFT is administered to all MBA students during the capstone course of the graduate program. The MFT is administered during the capstone course because of the sequencing of graduate courses in the MBA program. Since the capstone course of the MBA program is taken at the end of the program, it was deemed to be the best candidate for assessing our MBA students’ knowledge development. If the MFT was going to be a valid measure of how well the MBA program at USI imparted business content knowledge it was important that students take the exam at the end of their program. Placing the MFT at the end of the program accomplished two goals:

13


1. ensured that only students that were making successful progress in the program were being tested, since students failing to meet minimum standards of content proficiency would not have progressed to the cornerstone course, and

2. to ensure that students taking the MFT have been exposed to content areas that were being measured by the MFT. In addition, the MFT is administered in the latter part of the semester of the capstone course to maximize the students’ exposure to the business-related content being evaluated by the exam.

The MFT measures the student’s mastery of the essential business disciplines through an on-line three (3) hour exam. Results are reported on five (5) essential functional areas of business. Due to the small enrollment size of the capstone course it was determined by the Romain College of Business to administer the MFT every fall and spring semester. By administering the MFT every fall and spring semester we were able to increase the number of students being assessed, which will give us a larger and more reliable data set. In addition, it was determined that the MFT should count as a part of the students’ grade in the capstone course (5% percent of the students’ course grade). Making the MFT score a part of the students’ course grade was instituted to help incentivize the students to perform to the best of their abilities. Results of the MFT for the MBA program over the past five years are included in Table 6. The results of the MFT over the past five years indicate that USI MBA students consistently perform at high levels in comparison to a national sample of MBA students across the various business content areas.

As a part of the Graduate Council’s regular charges, the MFT scores are reviewed every semester (every fall and spring semester) and disseminated to the graduate faculty. A threshold or minimum score requirement was adopted during 2012 for initiating corrective action. The minimum standard score for each of the five content areas is 50%. Any score that fails to meet the 50 th percentile automatically initiates a review by the Graduate Council. As a part of the standard procedure, the Graduate Council, in consultation with discipline-specific faculty work together on investigating possible causes and explanations for the lower performance and then develop a plan of action to rectify the situation. Since this new threshold policy has been instituted it has not initiated any automatic reviews.

14

Table 6: Graduate Major Field Test Scores 2008 – 2012 – Percentiles and AveragesYear/Semester Marketing Management Finance Managerial

AccountingStrategic Integration

USI College of Business Average

National Average

Compared to National Average

2008 (Fall)* 68 64 56 60 62 261.0 249.2 4.74%

2009 (Spring) 64 67 51 64 63 260.0 249.2 4.33%

2009 (Fall) 59 62 45 57 56 256.0 249.2 2.73%2010 (Spring) 68 66 48 57 59 258.0 250.2 3.12%

2010 (Fall) 67 72 52 62 64 263.0 250.2 5.12%2011 (Spring) 73 72 54 62 63 265.2 248.9 6.55%

2011 (Fall) 64 72 48 62 62 260.0 248.9 4.46%2012 (Spring) 70 73 55 64 65 265.0 248.5 6.64%

2012 (Fall) 70 73 58 66 67 267.0 248.5 7.44%* it should be noted that the MFT for MBA students adopted a new format in 2008. Since then, the scores were recalibrated after the fall 2008 semester and therefore, the scores prior to and including the fall 2008 are not comparable with subsequent MFT scores.


Learning goals (#2 – #9)

In addition to administering the MFT to measure the students’ knowledge of discipline specific content, rubrics have been employed in assessing business-related skills development. In order to assess skill development, the Romain College of Business, under the direction and guidance of the Graduate Council and graduate faculty, identified a total of eight (8) learning goals that were deemed to be essential for a quality MBA graduate. It is the goal of the MBA program to develop effective:

2. Data acquisition skills3. Analytical thinking skills4. Synthesizing information skills5. Strategic thinking skills6. Ethical analysis skills7. Written communication skills8. Oral communication skills9. Leadership skills

During the ensuing years following the initial development of rubrics and measurement of the eight skills, the Graduate Council undertook a review of the assessment process. The major outcome of this assessment process review was a proposal to decrease the number of program learning goals from a total of nine to five. The proposal to reduce the number of assessment rubrics from nine to five was approved by the Graduate Council and graduate faculty of the Romain College of Business in spring 2012. Three considerations served as the primary motivators in making this change.

1) Length of assessment cycle. The excessively long timeframe for completing the cycle of assessment due to the large number of assessment rubrics was deemed to be unnecessarily lengthy. The time it took to complete the assessment cycle under the old schedule was a little over three years, which included the time to administer the individual rubrics and then have them evaluated. In order to expedite the assessment cycle and provide more timely assessment data, a critical review of the eight assessment rubrics was undertaken. The revised assessment cycle, which includes five program learning goals, provides more current and useful data in a timely manner. The newly devised assessment plan makes it possible to complete the assessment cycle in less than two years. The quicker turnaround time allows the Graduate Council to make more expeditious corrections/improvements/modifications to the MBA program.

2) Redundancy across assessment rubrics. After conducting a thorough review of the skill-based rubrics and the assessment process utilized throughout a complete cycle, the Graduate Council determined that there was a large amount of redundancy across the initial eight rubrics. In order to correct and remove this redundancy the initial eight skills-focused rubrics were consolidated into four rubrics, so that each rubric measures a separate and distinct learning goal. It is also argued that the final four skills-based rubrics represent a more concise and direct measure of the goals and objectives of the graduate program.

3) Need for a more focused approach. The initial nine learning goals was a first attempt by the Graduate Council to identify the primary skills that graduates of the MBA program of the Romain College of Business should possess. After a lengthy developmental process, in which the graduate faculty and Graduate Council worked through an iterative process, the previous eight skills-based

15


learning goals were identified. As is common when developing a new process for assessment, the result was an overly ambitious agenda. This required the graduate faculty and Graduate Council to reevaluate the desired outcomes of the MBA program in the area of skills development. After revisiting the goals and objectives of the MBA program it was determined that the four revised skill-based learning goals (listed below) were more appropriate for the Romain College of Business.

Based on the above outlined analysis it was determined that number of rubrics utilized for assessing skill development should be reduced from eight to four. The revised and current version of the skills development rubrics is focused on developing effective:

1. Ethical analysis skills2. Critical/strategic thinking skills3. Written communication skills4. Analytical thinking skills

The current schedule for assessing these learning goals in the graduate program is contained in Table 7 and reflects both, the previous and current, learning assessment process timeline. The newly developed assessment cycle is reflected in Table 7 starting with the spring 2012 semester, which will serve as the pattern for the future. As can be seen from Table 7, in addition to the administration of the MFT during every fall and spring semester, two skills-development rubrics will be assessed in each semester.

Table 7: Assessment Schedule For Graduate Program Learning GoalsProgram Learning Goals

2006-2007

2007-2008

2008-2009

2009-2010

Spring 2012*

Fall 2012

Spring 2013

Fall 2013

Revised Rubrics

Discipline Specific Content (MFT Exam) X X X X X X X X Discipline Specific

Content (MFT Exam)Data Acquisition X XAnalytical Thinking X X X X Analytical ThinkingSynthesizing of Information X X

Strategic Thinking X X X X Critical/Strategic Thinking

Ethical Analysis X X X Ethical AnalysisWritten Communication X X X Written

CommunicationOral Communications XLeadership Skills X*No rubrics were administered while the Graduate Committee revised the learning outcomes assessment rubric structure.

A standing charge of the Romain College of Business’ Graduate Council is to review the results of the assessment process and make suggestions or adjustments, in consultation with discipline-specific faculty, to improve the development of business-related skills. Results obtained from the administration of the assessment rubrics are disseminated among the graduate faculty of the Romain College of Business. Input from the graduate faculty is solicited in order to make any necessary improvements. After the results have been disseminated, the graduate faculty is encouraged to engage in a process of self-reflection and make suggestions to enhance student learning and/or integrate student learning outcome improvements at the department and individual faculty member level.

16


The assessment evidence provided in Tables 8-15 was derived from using a combination of embedded and stand-alone assignments in graduate courses. The process utilized two different processes. Some results were obtained through embedded course assignments that were integrated into the existing courses. Therefore, the assignments were used for assessing program outcomes and also were required by the professor as a component of the course. Other assessment rubrics were evaluated using extracurricular assignments that were obtained in a classroom setting, but were not evaluated by the professor. Such stand-alone assignments, while administered within the confines of a graduate course, were not used in determining the course grade. When non-embedded assignments were used to assess skills development they were evaluated by two trained graduate students.

Table 8: Percent of MBA Students Meeting Learning Goals (2007 – 2013)Ethical

AnalysisCritical/StrategicThinking

WrittenComm.

AnalyticalThinking

Leadership OralComm.

DataAcquisition

StrategicThinking

Synthesisof

InformationFall 2007

81 91

Spring 2008

95 98

Fall 2009

93 85

Spring 2010

91 93

Spring 2012

87 89

Fall 2012

84 79

Spring2013

96 90

Average 88 90 88 86 95 98 85 91 93

Tables 8-15 contain the results from administering the skills-development rubrics over the past five and one-half years. The data are presented in chronological order, based on when the individual skill was being assessed. Therefore, all of the data that has been collected since the time of our last accreditation visit is included. This means that the data for both the initial set of assessment rubrics (2006-2010), as well as the revised set of rubrics (2012-2013), has been included. It should be noted that the revised rubrics replaced the initial set of rubrics starting in the spring 2012 semester. Overall, the results suggest that the majority of the ratings for each of the skills assessed place the performance of the sample of USI MBA students at or above expectations as defined by the four revised rubrics.

Table 9 represents the results from administering the ethical analysis and written communication rubrics. The results were based on a case study that was administered in the graduate marketing course. The cases were evaluated using two trained graduate students. One area that was identified for review was the ability to apply ethical frameworks. For this particular skill 44% of the students did not meet minimum expectations. Students also performed poorly on being able to understand the implications and consequences of ethical decisions. A total of 32% of the students assessed did not meet expectations (see Table 13).

17


In light of these less than acceptable results, a teaching module was incorporated into the marketing course that placed more emphasis on ethical decision-making frameworks (for example the Virtue Matrix developed by Dean of the Rotman School of Management at the University of Toronto).

Table 9: Assessment Rubrics Administered Fall 2007

Ethical Analysis Does not meet Expectations % of total Meets

Expectations % of total Exceeds Expectations % of total

Issue Identification of Sensitivity 0 0% 18 72% 7 28%Relevant Sources of Ethical Standard 6 24% 10 40% 9 36%Identification of Stakeholders 0 0% 13 52% 12 48%Alternative Approaches 6 24% 11 44% 8 32%Applies Ethical Framework 11 44% 14 56% 0 0%

Written Communication Rubric Does not meet Expectations % of total Meets


Purpose/Opening Statement 4 16% 12 48% 9 36%Organization and Logic 0 0% 16 64% 9 36%Content Quality 1 4% 18 72% 6 24%Standard English and Vocabulary 1 4% 10 40% 14 56%Closing Statement 5 20% 19 76% 1 4%

In addition, another class session was devoted to understanding the implications and consequences of ethical decisions in the marketing course and an out-of-class ethics case was added to help students apply the ethics rubric. When the ethical skill development rubric was next administered in the spring 2013 semester drastic improvements were noted (see Table 15). Only 5% or one student did not meet expectations after the corrective actions were taken to address the issues observed in prior assessment cycles. Written communication skills were evaluated using an embedded assignment in the leadership course. Results suggest that students demonstrate adequate written communication skills at a level that is appropriate for graduate-level coursework (Table 14).

Assessment Evidence: MBA Program Learning Goals (#2 -#9 and since Spring 2012 #2 - #5)

The leadership and oral communication skills were assessed within the confines of the leadership course in the MBA program. By using embedded course assignments in the leadership skills course leadership and oral communication skills were measured utilizing a two-pronged approach. First, a self-assessment assignment of leadership skills and traits and a variety of articles drawn from the leadership literature were used to help students identify and develop an awareness of their own leadership style, as well as an the ability to recognize different leadership styles and situations. Second, oral communication skills were reinforced using in-class presentations. The professor teaching the course was responsible for evaluating the assessment rubrics. Results indicate that an overwhelming majority of the students meet or exceeded expectations (Table 10).

18


Table 10: Assessment Rubrics Administered Spring 2008

Leadership Skill Rubric Does not meet Expectations % of total Meets


Recognition of Differing Styles 2 11% 8 42% 9 47%Situational Implications of Styles 2 11% 13 68% 4 21%Awareness of Personal Style 0 0% 19 100% 0 0%

Oral Communication Skill Rubric Does not meet Expectations % of total Meets


Opening Statement and Set Up 3 16% 9 47% 7 37%Organization of Presentation 0 0% 15 79% 4 21%Quality of Content 0 0% 12 63% 7 37%Voice, Pace, and Eye Contact 0 0% 10 53% 9 47%Standard English and Vocabulary 0 0% 19 100% 0 0%Closing Statement 0 0% 17 89% 2 11%

Table 11 shows the results of the data acquisition and analytical thinking assessment rubrics. The data was collected in the graduate decision sciences course using an embedded course assignment that was evaluated by the professor of the course. Results suggest that students demonstrate the ability to apply analytical thinking skills within the context of decision-making cases. Embedded course assignments included cases analyses and written reports. The written reports reflected an appropriate understanding and usage of data sources by the students. However, when the analytical thinking skills rubric was administered again in the fall 2012 semester results were drastically worse.

Table 14 shows that a total of 63% or 12 of the 19 students assessed failed to meet expectations on the information component of the analytical thinking rubric. However, it should be noted that rubric was assessed using a team project approach. This means that each of the students was not independently assessed for this skill. It is believed that such an approach is not appropriate with the goals and objectives of the MBA program at USI and will not be repeated again in the future. Analytical thinking skills are being reassessed during the fall 2013 semester using individual assignments. It is believed that results from the individual assignment approach will provide more accurate data. Once the results have been tabulated the Graduate Council will undertake a review of the data and take the appropriate actions.


Data Acquisition Does not meet Expectations % of total Meets


Data Sources 2 3% 43 70% 16 26%Relevance of Data 0 0% 39 63% 23 37%Quality of Data 9 15% 45 73% 8 13%

Analytical Thinking Does not meet Expectations % of total Meets


Issue Identification 0 0% 43 68% 20 32%Implications 2 3% 39 63% 21 34%Motives 3 5% 48 77% 11 18%Quantitative Framework 2 3% 54 87% 6 10%

The synthesis of information and strategic thinking rubrics (Table 12) were administered in the graduate marketing course. A case analysis was used in collecting the data and the results were compiled and assess using trained graduate assistants.

19


The results indicate that the vast majority of the students demonstrated the ability to think strategically about a problem and devise a solution that synthesizes the relevant information available. However, when the revised rubric of critical/strategic thinking was administered in the spring 2012 semester 32% of the students did not meet the minimum expectations in being able to adequately understand the implications and consequences of their proposed recommendations. This reflects an overall inability to think critically about the ramifications of business decisions. In order to address this problem both the marketing course and strategic management courses had devoted additional class time to addressing the issue of critical thinking. In addition, cases and projects have been added to help students develop their skills in understanding the interrelationships among constituencies, the impact that business decision can have the various stakeholders, and stakeholder theory. An additional module on critical thinking was also added to the management information course to help rectify this deficiency.


Synthesis of Information Does not meet Expectations % of total Meets


Relevance of Information 0 0% 43 70% 19 31%Quality of Information 2 3% 40 65% 20 32%Conclusions 3 5% 43 69% 16 26%

Strategic Thinking Does not meet Expectations % of total Meets


Strategic Objectives of the Agent 0 0% 41 66% 21 34%Strategic Environment of the Agent 0 0% 31 50% 31 50%Interdependence of Actions 9 15% 38 61% 15 24%Prescribed Strategy(ies) 0 0% 40 65% 22 35%

The most recent results for the revised rubrics are included in Tables 13-15. The results of each of these developmental skills and the corresponding data were previously discussed within the context of this document. Therefore, we will not revisit these results again at this time.




Ethical Standards/Concepts 4 21% 9 47% 6 32%Stakeholder Analysis 0 0% 11 58% 8 42%Implications and/or Consequences 6 32% 9 47% 4 21%Conclusions/Solution 3 16% 13 68% 3 16%

Critical/Strategic Thinking Does not meet Expectations % of total Meets


Information/Evidence 3 16% 10 53% 6 32%Concepts 3 16% 13 68% 3 16%Assumptions 0 0% 7 37% 12 63%Implications/Consequences 6 32% 7 37% 6 32%Interpretation and Recommendation 2 11% 12 63% 5 26%

Written communication and analytical thinking skills were assessed via embedded course assignments in the leadership and financial management courses, respectively. Both assignments were part of the course requirements and were evaluated by the professor of the course.

20



Written Communication Does not meet Expectations % of total Meets


Purpose/Opening Statement 4 12% 10 30% 19 58%Organization and Logic 4 12% 2 6% 27 82%Content Quality 2 6% 4 12% 27 82%Standard English and Vocabulary 4 12% 8 24% 21 64%Closing Statement 6 18% 8 24% 19 58%

Analytical Thinking Does not meet Expectations % of total Meets


Concepts 4 21% 9 47% 6 32%Information 12 63% 5 26% 2 11%Interpretation 0 0% 12 63% 7 37%

Table 15 shows that adjustments and modifications made to improve students’ ethical reasoning skills have begun to bear fruit and are having an impact on their ability to apply ethical frameworks and understand ethical implications. Similarly, the ability to think critically is beginning to show signs of improvement, although there is still work to be done in this area. More class time has been devoted to discussing and practicing critical thinking skills in the marketing and strategic management courses. Additional assignments have been incorporated into the marketing class to assist students in further developing their critical thinking skills, while more time in the strategic management course is being devoted to understanding stakeholder theory and identifying key assumptions and interdependencies that are embedded in business decision making.




Ethical Standards/Concepts 1 5% 15 79% 3 16%Stakeholder Analysis 1 5% 13 68% 5 26%Implications and/or Consequences 1 5% 10 53% 8 42%Conclusions/Solution 1 5% 9 47% 9 47%

Critical/Strategic Thinking Does not meet Expectations % of total Meets


Information/Evidence 0 0% 10 53% 9 47%Concepts 2 11% 14 74% 3 16%Assumptions 5 26% 11 58% 3 16%Implications/Consequences 5 26% 12 63% 2 11%Interpretation and Recommendation 1 5% 13 68% 5 26%

Concern #2: -The apparent focus was on the assessment of majors instead of programsTo address this concern, during the past ten months there have been systematic and ongoing efforts to clarify the layers of assessment practice and evidence among members of the College. As a result, there is greater clarity among faculty members about the focus of the College’s AOL system with regard to fulfilling the assessment expectations of the University’s accrediting body – the Higher Learning Commission – and the assurance of learning standards of the AACSB International.

21


Since the mid-2000s, the College has adopted a layered approach to assurance of learning. Each faculty member engages in course-level assessment as part of the approach to assuring a sustainable culture of assessment. Each of the majors also has specific learning goals and assessment activities aimed at meeting the University’s program review requirements.

Finally, the AOL system is designed to address the program-level learning goals that have been described earlier. Given the multiplicity of objectives that the assurance of learning system seeks to address, there is considerable room for confusion to emerge particularly, for faculty members conducting assessment at the course level but not directly involved in program level assessment.

In response to the gaps identified in the College’s assessment information with regard to majors offered by the College during the last Higher Learning Commission visit, there has been a concerted focus since 2012 on developing capacity within the Romain College of Business to provide information for the next Higher Learning Commission visit in 2016. The immersive nature of these efforts with regard to the assessment of majors clearly created some challenges in articulating the AOL system during the Peer Review Team’s visit.

This focus, however, was not at the expense of the implementation of the College’s AOL plan. Nonetheless, to the extent that elements of the AOL system were being utilized to fulfill the generation of assessment information at the level of the departmental majors, and because every faculty member was not involved in delivering the program level curriculum, there tended to be greater familiarity with the details about assessment at the major level compared to the program level. In addition, since every faculty member has been engaged in assessment activities at the course level every semester since 2008 as part of the College’s approach to enhancing a culture of assessment and building a sustainable AOL system, it is not surprising that there were varying levels of clarity among faculty members when describing the College’s AOL system.

Steps taken to address this concern are documented in the agenda items of two faculty retreats in 2013 that focused on clarifying the layers of assessment activities at the faculty, department, and program levels and how these are integrated to assure continuous improvement in student learning outcomes. Elaboration of the meaning of program assessment from the perspective of the AACSB International and the Higher Learning Commission was one of the focus areas. Systematic efforts continue and will continue to emphasize the distinctions between program level and major level assessment activities and dedicated assessment day activities each semester will continue to focus on this distinction.

Concern #3: -The apparent inconsistency between the professional experience and continual development activities of some PQ faculty and their teaching assignments.

In response to this concern a number of steps were taken during the past ten months. First, faculty members reviewed and approved (May 1, 2013 Faculty Meeting) a change in the criteria effective from Fall 2013, related to the designation of professionally qualified faculty members. This change has resulted in one PQ designated faculty member providing more meaningful service in a professional association within the past ten months and the establishment of a faculty internship in her field during Summer 2014.

22


A second step was to review and make adjustments beginning in Fall 2013 to the teaching assignments of PQ faculty members who were identified by the Peer Review Team as teaching the type of courses that were not in alignment with their professional experience and continual development activities. In particular, certain upper-level courses assigned to two PQ faculty members in previous semesters are no longer assigned since the Fall 2013 semester.

In addition, we have also introduced internal controls (e.g., an additional review procedure, conducted by the Assistant Dean for the Romain College of Business, prior to finalizing class) in our scheduling process to ascertain that PQ faculty members are providing instruction in their areas of expertise and/or specialization.

At the time of our 2008-2012 maintenance report, in order to be considered Professionally Qualified a faculty member had to meet the following standards:

To be PQ a faculty member must, at the time the faculty member is hired: Possess a Master’s degree or higher in a relevant field; and

(a) Have a record of significant relevant professional experience in three of the last five years, or(b) Possess one of the following:o Professional certification in fieldo Two scholarly products over the last five years (see above examples)o Record of professional engagement. Examples of professional engagement include:

Consulting Faculty internships Continuing professional education (including completion of graduate courses in a relevant

discipline) Meaningful service in a professional association Participation in teaching improvement activities

To maintain PQ status, the faculty member must possess one of the following:o Current professional certification in fieldo Two scholarly products over the last five years (see above examples)o Recent record of professional engagement. Examples of professional engagement include:



23


After the review of the PQ standards, the following change was made with regard to PQ faculty members maintaining their PQ status. Effective from the 2013-2014 academic year: To maintain PQ status, a faculty member must possess two of the following:o Current professional certification in fieldo Two scholarly products over the last five years (see above examples)o Recent record of professional engagement. Examples of professional engagement include:



24


III: Appendix

Documents related to the following items are available at http://business.usi.edu/aacsb/

A. Schedule of BA/BS assessmentB. RubricsC. Learning Goals alignment with curriculumD. Assessment AssignmentsE. Assessment results: F. PQ Faculty members: http://business.usi.edu/aacsb/PQ-AQ.aspG. List of Faculty meetings and AgendasH. Program level re-design of curriculum that reinforces Analytical Problem Solving skillI. Conceptual framework for designing a new Business CoreJ. Fifth Year Maintenance Report

25

http://business.usi.edu/aacsb/PQ-AQ.asp
