standard four: assessment made incredibly easy (not for dummies)

Assessment made easy but definitely

NOT for Dummies Pam Braden

West Virginia University at Parkersburg

#ACBSPAccredited @ACBSP2015

and accredita@on

Expected outcomes. From this session, you will be able to: •  Explain the vocabulary and process ACBSP uses including

•  6 Standards and 4-‐step “approach” •  Iden@fy the “big deal” issues about assessment and accredita@on • Differen@ate between assessment for standards 3 & 4 (and others) •  Evaluate the sources for data •  Select from recommenda@ons for compara@ve data • Q/A


ACBSP accreditaFon vocabulary. . .

• 6 Standards • Sub-‐criteria • Portal helps you stay on message. • Argument = narra@ve / Self Study • Evidence = documenta@on.


ACBSP vocabulary & terminology

Con$nuous Improvement Process and Organiza$onal Learning • APPROACH – Do you have a plan for each standard? • DEPLOYMENT/IMPLEMENTATION – Have you implemented the plan and how is it going so far? What data have you collected? • RESULTS/LEARNING – What did you learn from the data collec@on? Analyze the data. •  IMPROVEMENTS – What did you FIX, CHANGE, or DO as a result of analyzing the data? How did you USE the data?


More Vocabulary: Types/labels for data •  Forma@ve versus Summa@ve •  Forma@ve = Along the way – course-‐level test banks, end of freshman year, BUS 101 course. Not grades! •  Summa@ve – Near the end of the road – MFT

•  External vs. Internal •  External = Test created by someone outside your ins@tu@on

•  ETS, ACT, EBI, . . .usually something you pay for) •  Simula@ons – Glo-‐Bus, BSG, CapSim

•  Internal – “home grown” assessment projects or exams •  Porgolio project •  Integra@ve capstone course/project – case analysis

• Compara@ve, Benchmarking


Assessment vocabulary 101 – ACBSP style! • Objec@ve = what you are working toward in the future (beginning) • Outcome = the results at the end (ending) •  Focus Assessment info on the OUTCOMES! What actually happened

•  Lots of synonyms = evaluate, review, assess, measure


Put the SELF in self study. . .

•  The CampusLabs portal seems sterile and uninspiring. Try to put your ins@tu@on’s “personality” in the documents.


SELF = Include faculty at every step of the

process.

Assessment for every standard Quick overview of the 6 standards and how assessment should be included on every standard.


OrganizaFonal Profile • Organiza@onal Profile & Overview = your school’s context and student profile •  Programs offered •  Courses taught outside business unit •  Campuses/Centers where business programs are taught

• Organiza@onal charts • Regional accredita@on • Mission statements


•  Provide info to the public (catalog & web site)

•  Organiza@onal Descrip@on •  Organiza@onal Challenges

Standard 1 Leadership •  Standard 1 = Leadership – ins$tu$on and B-‐School •  1.1 –Dean/Chair leads the B School •  Faculty involvement •  Evalua@on (=measurement/assessment) of faculty and administrators •  Have a policy. •  Follow the policy.

•  Social Responsibility (use the table Figure 1.1) •  Ethical prac@ces followed •  Regulatory compliance (federal compliance)


Standard 2 Strategic Planning 2.1 Planning process:

•  Policy – HOW you plan, faculty involved, iden@fy measures

•  HR Plan -‐ # faculty needed, disciplines, expected turnover, re@rements

•  Strategic Plan – Aligns with the ins@tu@onal Strategic Plan

2.2 Deployment •  Demonstrate that you follow the planning process on the books.

•  Measures used for each short-‐ and long-‐term goal. As the plan progresses, include the measurements to show progress.


Standard 3 – Student and Stakeholder Focus 3.1 Who are your target markets?

Iden@fy the groups Students: tradi@onal, non-‐trad, commuters, online) External stakeholders: Advisory boards, local community, employers

3.2 How do you LISTEN to them and LEARN from them?

How do you collect data from them? When? 3.3 How do you review (evaluate/assess) how well the listening approaches are working? 3.4 Listening for all processes (planning new programs, marke@ng, process improvement, etc.)


Standard 3 Student and Stakeholder Focus

• 3.5 Recruitment and reten@on focus • 3.6 Student complaints. • 3.7 System /process to determine sa@sfac@on and dissa@sfac@on. • 3.8 Tables (Just do it; it’s in the book).


Process check. . . Isn’t this easy? See? I told you so! J


Standard 4 Measurement and Analysis of Student Learning and Performance

1.  Have an assessment plan / program for every academic program.

2.  Report results from assessment – at least 3 itera@ons of successive data (same data set 3-‐5 @mes).

3.  Compare your school’s results from year to year and against peer ins@tu@ons.

4.  Use the results to improve student learning.


Standard 5 – Faculty and Staff Focus •  How do you ensure quality in teaching? TABLES •  HR Plan (from St. 2) (Helps assess the effecGveness of deployment in St. 5)

•  Employment Prac@ces – full-‐@me and adjuncts – Composi@on matches the mission and program objec@ves

•  HR process for hiring, firing, evaluaGng, training faculty

•  Qualifica@ons Figure 5.1 (see Handbook for defini@ons)

•  Credit hour produc@on by faculty Figure 5.2

•  Coverage by creden@al Figure 5.3


•  Deployment by mode, loca@on, level (associate, bachelors, masters) Figure 5.4

•  Faculty load Figure 5.5 •  Faculty EvaluaGon – have policy, follow it •  Faculty Handbook •  Faculty Development •  Scholarship & Professional Ac@vi@es Figure 5.6

Standard 6 EducaFonal and Business Process Management •  Educa@onal design and delivery

•  Modes (online, blended, various sites/campuses – all have same outcomes)

• Common Professional Component (CPC) – show for each level (associate, bachelor’s, masters) • Course Syllabus for every required business course (evidence) • Curriculum beyond the CPC for each program = gen ed & “major” requirements • Design & Delivery Evalua@on (measurement & assessment) •  Educa@onal support services (assessment of each)


And now for some useful hints. . . You can thank me later. . .


Hint: Read, READ, READ!


Sample self studies, best prac@ces, etc. ACBSP web site

Standard 3 v. Standard 4 – Types of data needed

•  Standard 3 – Student and Stakeholder Focus •  “Warm Fuzzy” data = sa@sfac@on data •  Examples:

•  Student evalua@on of faculty/ course (internal)

•  Current students using a na@onally normed assessment such as NSEE, CCSSE, Noel Levitz (External)

•  Employer survey data •  Graduate survey data •  Faculty and staff – Don’t forget!

•  Standard 4 – Student Achievement on Learning Outcomes – “HARD” data •  “Just the facts, Ma'am” •  Student performance on standardized tests


Separate the data -‐ Standard 3 vs. Standard 4 Standard 3 – Students and Stakeholders = “Warm fuzzy” data Standard 4 = no sa$sfac$on data

Keep each sec$on separate and focused on the “ques$on”

• Be careful to keep each sub-‐sec@on separate and try not to repeat info. •  Standards 3 and 6 have some similar data.


3 4

Standard 3 Advisory Boards •  Populated with local business leaders that

are representa@ve of poten@al employers •  Hold semiannual mee@ngs •  Great source for feedback •  Minimal costs for hospitality •  Include minutes of mee@ngs in the self study

as evidence. •  Invite them to a breakfast with the site visit

team during the visit.


Standard 3 Sources of saFsfacFon data Sa@sfac@on surveys run by MKTG Research Course Project (Free, great project!) •  Likely somewhat low response rates •  Personal connec@ons will help

(FaceBook) Survey – every 2-‐3 years? (Plan!) •  Employers •  Students – CCSSE, NSSE, Noel Levitz, etc. •  Graduates •  Faculty and staff


Standard 4 •  External comparaGve data is the gold standard for assessment of student learning and performance. •  Examples include: •  Major Fields Test •  Glo-‐Bus Simula@on •  Porgolio (LiveText, EBI, etc.) •  Tons of vendors out in the hall at the conference.

• Does not need to cost a lot to be effec@ve!


Standard 4 Vocabulary: Types/labels for data (again!) •  Forma@ve versus Summa@ve Assessment

•  Forma@ve = along the way – end of freshman year, BUS 101 course •  Summa@ve – near the end of the road – MFT Associate, MFT Business, MFT MBA

•  External vs. Internal Assessment •  External = test created by someone outside your ins@tu@on

•  ETS, ACT, EBI, . . .usually something you pay for) •  Simula@ons – Glo-‐Bus, BSG

•  Internal – “home grown” assessment projects or exams •  Porgolio project •  Capstone course/project – case analysis

•  An effecGve Assessment Plan has elements of all these!


Standard 4 – Glo-‐Bus SimulaFon


•  My favorite simula@on – Glo-‐Bus (marketed by McGraw Hill) •  Used by >300 ins@tu@ons in 30 countries •  Inexpensive: bundle with book, simula@on, and cases.

GLO-‐BUS simulaFon – 2 data sets provided Performance Benchmark Report

•  Overall score – compare your students’ scores against all other ins@tu@ons around the world using the simula@on. •  ROI •  Stock Price •  EPS •  Credit Ra@ng •  Image Ra@ng

Learning Assurances Report •  Leadership Skills (based on peer evalua@ons) •  Collabora@on and Teamwork Skills (peer eval) •  Financial Analysis •  Financial Management •  Opera@ons Management •  Marke@ng Management •  Human Resources Management •  Strategic Analysis and Planning •  Corporate Social Responsibility


Glo-‐Bus “Learning Assurances” Data


Includes “Learning Assurances report” on sov and analysis skills: 1.  Leadership Skills 2.  Collabora@on & Teamwork 3.  Analy@cal Skills 4.  Opera@ons Mgmt 5.  Financial Mgmt 6. Marke@ng Mgmt 7.  HR Mgmt 8.  Strategic Planning & Analysis 9.  Corporate Social

Responsibility

External = Compara@ve Benchmarking against 300 other schools; Internal – Compare semester to semester over @me

Glo-‐Bus Performance Reports

• Overall Score (algorithm) •  Earning Per Share • Return on Equity •  Stock Price • Credit Ra@ng •  Image Ra@ng


ETS Major Fields Test – results


MFT Sub-‐scores correlate with CPC CPC Bachelor’s requirements: •  Func@onal areas:

•  Marke@ng •  Finance •  Accoun@ng •  Mgmt (including Prod/Op, OB, HR)

•  Business Environment: •  Legal Environment of Business •  Economics •  Business Ethics •  Global Dimensions of Business

•  Technical Skills •  Informa@on Systems •  Quan@ta@ve Techniques/Stats

•  Integra@ve Areas: •  Business Policy or comprehensive integra@ng

experience

9 MFT Sub-‐scores (Bachelor’s test) $25/test = add a fee?

Subscrip@on for sub-‐scores = $700/yr A1 -‐ Accoun@ng A2 -‐ Economics A3 – Management A4 -‐ Quan@ta@ve Business Analysis A5 – Finance A6 – Marke@ng A7 -‐ Legal and Social Environment A8 – Informa@on Systems A9 – Interna@onal Issues


Sub-‐scores data drills down to concepts you can FIX! (Useable data to improve student learning!)


0

10

20

30

40

50

60

70

Percen

t Correct

WVUP Performance on Quan$ta$ve Business Analysis Sub-‐Scores Fall 2013 -‐ Fall 2014

WVUP % Correct

Na@onal % Correct

RecommendaFons – “Just do it” • Own the process. If your assessment program is in development, acknowledge that you are working on implemen@ng the new plan. Trying to “fake it” doesn’t work!

•  Focus on PROCESS and Con@nuous Improvement in every part of the ins@tu@on and your business program opera@ons.


Complete the “recommended” tables and figures in the handbook for EVERY standard.


Key words to think about all the Fme: •  Faculty involvement • Con@nuous improvement • High quality • Documenta@on / evidence •  Strategic Plan and PLANNING • HR Plan • Advisory Boards • Have a policy; follow it!


[email protected] Good luck! If you have ques@ons or need help, please email me! J


standard four: assessment made incredibly easy (not for dummies)

Education