accrditation tool for quality assurance

Accreditation and

Quality Assurance Surendra Prasad Chairman, NBA

Some Questions

Do you (really) need Accreditation? It is now almost Universal!

A tool for quality assurance of engineering education

Q. Who Needs the Assurance?

All Stakeholders: Prospective Students Their Parents Faculty Management

& Government

What is quality and How do we measure it?

Quality may be taken for granted, when there is a brand name: IITs, IISc.

Reputation earned over the years becomes an assurance of some kind.

This is rather subjective. Accreditation: A system which attempts to

assess quality a little more objectively for Assurance.

What are the possible benefits? Let us see

Students

Four Years of Valuable time, @18 yrs. Will I become a good engineering

professional? Survive the competition? Admission to an Accredited Institution is a

safe bet!

Parents

Will our son/daughter have a good life? Is the investment in the education OK?

Faculty

Working at an Accredited Institution leads to a better career!

So, Better Faculty Induction!

Management

Institution has better recognition. Greater Contribution to Society.

Government

Better Educated Engineers community leads to better economic growth.

What can be more important to the government?

Accreditation for Quality Assurance

Q. Is the Accreditation Process Universal? Ans. Very Unlikely. Every System of Education Own

Angularities. Washington Accord Many Systems

Jointly Acceptable India now a permanent signatory.

Accreditation What accreditation is NOT?

1. Not a ranking procedure (Gold, Silver, 1,2,3,..) 2. Not an Investigation (There has been no

complaint) 3. (Should not be): A Regulatory Process,

Accreditation is not a ranking system. NBA (or ABET) do not rank programs in any way.

What is it then?

It is a Process by which:

The Institution being looked at is given a SEAL of approval by stakeholders in its activity - as meeting their expectations.

(Stakeholders: Students, Govt., Industry, Faculty,)

Beyond Certification

However, its aim is not mere certification.

It is a process which encourages continuous look at how you teach, what is its impact, and how to achieve the set objectives better.

It should lead to a Continuous Improvement System.

Relevance to Indian Scenario

Indian Engineering Education Scene: Large, Multi-layered, and Complex.

Common criticism: Sub-standard education delivery.

Quality: New Mantra. : If we wish to do justice to aspirations of the young people.

NBA

NBA hopes: To create a movement towards quality through accreditation with credibility.

Accreditation only a tool.

Real Goal: Improved teaching and learning at our Engineering Institutions.

Real Benefit of Washington Accord!

Assessment & Evaluation +

Closing the Loop : Continuous Improvement of

Program Outcomes

Definitions

Program Educational Objectives: Broad statements that describe what graduates are expected to attain within a few years after graduation.

Program Outcomes/Student Outcomes/Graduate Attributes: What students are expected to know and able to do by the time of graduation.

: Relate to the knowledge, skills, attitudes and behaviors that students acquire as they progress through the program

Definitions

Outputs AND Outcomes Outputs: Graduation rates, Employment Statistics etc. Outcomes: Student Learning, Skills, Behavior.

Focus on Program Outcomes

Criterion 9 is concerned with continuous improvement in Student or Program Outcomes.

Note: Assessment of PEO not easy.

However PEOs important and relevant to vision and mission of the program and the institution.

Not to be discussed in this discourse.

Program Outcomes: Central to Short-Term and Long-Term Improvements in Curriculum and Pedagogy.

Curriculum

Curriculum Design exercise in many Indian institutions: Examine the practices in some good institutions (IITs), and adopt suitably.

Workable, but not necessarily the best approach for your students and your objectives.

Backward Design Approach

A Desirable Order of Goals Improved Student learning: Four Dimensions

Knowledge Skills Attitudes Behaviour

Improved learning Environment. Faculty Efficiency and Efficacy. Accountability to Stake-holders. Accreditation.

Goals as Seen by Many Institutions

Goals: same, But in a Different Order Accreditation. Faculty Efficiency and Efficacy. Improved Student learning

Knowledge Skills Attitudes Behaviour

Improved learning Environment. Accountability to Stake-holders.

Faculty Engagement in Continuous Improvement

Successful achievement of Goals requires faculty participation and engagement.

Is their focus right?

Is the Question:

What do we have to do to get NBA accreditation? OR

How can we effect improved student learning?

Definition

Assessment:

One or more processes that identify, collect, and prepare data to evaluate attainment of student outcomes.

Transformational Assessment

Goals of Assessment: Information based decision-making.

Transformational Assessment: When assessment used to help enhance student learning. Change Pedagogy Modify courses or facilities Redesign curriculum

Information basis

Smart Data Collection is the Key

Collect data that would be useful.

Sufficient to provide a reasonable basis for action.

Data Information.

Sufficient Granularity.

Criterion 9: Continuous Improvement

Statement of NBA criterion not so crisp, although essence well captured in Criterion 2 and 9 seen together.

More meaningful to look at the ABET statement.

The program must regularly use appropriate, documented processes for assessing and evaluating the extent to which the student outcomes are being attained. The results of these evaluations must be systematically utilized as input for the continuous improvement of the program. Other available information may also be used to assist in the continuous improvement of the program.

Closing the Loop

This concept, although the essence of quality assurance, is also seen to be the weakest link in most institutions seeking accreditation.

Most institutions fail to spend enough effort in closing the loop.

Coming from an IIT, I would say, this is true even for the IITs to a large extent.

Why this weakness?

After all, we all swear by continuous improvement! Then why this weakness? Several Reasons:

Lack of Leadership/Ownership. Sometimes inadequate appreciation of this aspect.

Non-Compliant Faculty: Too Busy.

Ineffective Tools. Discomfort with Assessment. Inconclusive Results.

Other Reasons

Trying to do too much. Unsustainable

Inadequate Resources.

Many more.

Where Do We Start?

If We Wish To.

Management Plan

Processes

Collecting/Consolidating Data.

Analyzing and Reporting.

Evaluation

Results

Data Information Knowledge Decisions Curriculum/

Pedagogy Improvements

Improvement Actions

Curriculum changes.

Pedagogy Others.

Mapping of Program Outcomes with Courses

Course No./Outcome

EE 101 HU 203 EE306 EE 347 (Design Project)

EE 450 (Capstone)

Engineering Knowledge

I,P P,E

Problem Analysis

P P,E

Design/Development

P,E

Investigation of Complex Problems

P,E

Engineer & Society

I

Communication

I P

Benefits of Mapping

Shows what desired outcomes are already taught, and at what level.

Reveals gaps (Outcome xx).

Potential for cross-course collaboration (Faculty buy-in).

Makes course, program level outcomes clearer to students.

Shows where interventions may be required.

Collecting Evidence

Evidence is the key to making decisions about improvements and closing the loop.

Direct Evidence usually primary for program outcomes.

Indirect Evidence: Acceptable and valuable as secondary evidence. Should not be all you use.

Both qualitative/quantitative form of (both direct/indirect) evidence useful.

Sampling (Needed for Larger Programs)

Using part of the whole to represent all students or all student work.

Sampling acceptable and even desirable for assessment of larger programs.

Samples must be representative of student population.

Assessment is the Key

Program Outcomes broadly defined by NBA.

Up to Institutions to assess them as per their defined objectives: They retain the flexibility to assess them any way which is appropriate to measure the attainment of desired outcomes.

Many innovations possible here.

Main Players: Faculty.

The Assessment Challenge

Faculty: Subject Experts.

May not be Assessment Experts.

Can and do bring their own intuition and insight into: What constitutes Acceptable Performance in their classes?

Challenge: To involve them more deeply into the Assessment Conversation to measure effectiveness of what they do individually, or collectively

: additional effort should seem worthwhile to faculty.

Definitions

Evaluation: One or more processes for interpreting the data and evidence accumulated through assessment processes.

Evaluation determines the extent to which student outcomes are being attained. Evaluation leads to decisions and actions regarding program improvement.

Performance Criteria: Specific, measurable statements identifying the performance required to meet the outcome confirmable through evidence

Definitions

Rubric: a method for explicitly stating the expectations for student performance

Generally has three components: Performance Indicators (Dimensions of Learning) Level of Performance (Scales) Descriptors

Level of Performance: Discrete vs. Continuous

Rubrics

A method for explicitly stating the expectations for student performance.

Describe characteristics for each level of performance.

Provide clear information how well the students performed

Sample Assessment Rubric for NBA Outcome

Example Outcome: Ability to Apply knowledge of mathematics, science and engineering.

Possible Rubrics: Standard: The student shows clear evidence of being able to

apply knowledge, techniques, skills and modern tools learnt in the discipline and use knowledge of these disciplines to identify, analyze and solve technical problems.

Exceeding the Standard: The student shows superior evidence of being able to apply knowledge, techniques, skills..


Approaching the Standard: The student shows marginal evidence of being able to apply knowledge, techniques, skills...

Not Meeting the Standard: The student shows no evidence of being able to apply knowledge, techniques, skills

Assessment Types

Outcome Assessment Process may use multiple methods of assessment (and allowing for triangulation): direct and indirect, qualitative and quantitative:

Direct: Provide direct examination or observation of student performance. Students assignments. Exams, Quizzes (Course Embedded assessment) Student Portfolios. Locally developed exams. National exams (University Exams, GATE etc.)

Indirect Assessments

Ascertain opinion or self-report: extent of attainment of outcomes.

Terminal (at graduation) Surveys (Employer Surveys?) Focus Groups (Alumni). Advisory Board Recommendations.

Exit Interviews.

Course Embedded Assessment (Direct Assessment)

One direct form of Assessment: through the Course itself.

Easy for engineering/computer science /applied science/technology faculty to do.

Robust.

Students motivation (or lack of it) for assessment not a factor.

Requires careful design of assignments, tests, quizzes etc., to ascertain different performance criteria.

Assessment Process

Must include thresholds (or targets) for acceptable performance.

Must be measurable.

Exam & Grading Assessment.

However, both may be done together via suitable embedding strategies.

Setting Targets or Thresholds

Critical part of closing the loop.

Not advisable to set targets once for all be very flexible here.

No hard and fast rules: pilot through and alter based on experience.

Be realistic about the programs context in setting targets.

Sample Outcomes Assessment Processes

Assessment Methods

Description Target Frequency

Direct Assessment of Course Material

Specific Course Material used to attain POs

70% must meet expectations (standard): Average Rubric Score: 7/10

Yearly (see Schedule)

Exit Survey of Final Year Students

Specific Questions mapped to POs

75% : 7.5/10 Yearly

End of the year survey (Student Feedback?)

Supplemental Questions mapped to POs

80% agree or strongly agree

Yearly

Co-opt employer survey

Specific questions mapped to POs

70% agree or strongly agree.

Bi-annually

Triangulation

Ultimate Target: Assess the truth for the attainment of various desired outcomes through multiple assessments.

Truth

Student work

Focus Group

Survey

Triangulation

More realistic

Student work

Focus Group

Surveys

Truth

Closing the Loop

When targets not met and improvement needed:

Action required to analyse and improve. curriculum (and pedagogy?).

Mapping against outcomes may help identify the changes needed.

Program Level Outcome Assessed At Course level

Course: CE xxx ETHICS MATRIX

Direct Assessment of Ethics Matrix

State Course Outcome Objective

% exceeding standard >80

% meeting standard

70-80

% approaching standard 60-70

% not meeting standard

How Often Do We measure?

Every semester? Every Year?

Regularly?

NBA Year?

Closing the Loop

: Regularly Scheduled Process: Not every semester or every year for all outcomes: Lighter on faculty.

Student Outcomes 14-15 15-16 16-17 17-18 18-19

Understanding of professional and ethical responsibilities

Understanding the relevance of Science, Mathematics and Engineering on Contemporary Issues

Ability to work in teams

Understanding of diverse cultural traditions in the country.

Ability to communicate effectively, orally, in written and visual forms.

Summary

Establish Program and Course Outcomes

Select where you want to assess (and in which course(s)).

Select components of outcome.

Select performance level.

Select criteria of success.

Use information to Close the loop.

Questions for Closing the Loop

What do the findings tell us?

Which improvements are critical for greater effectiveness: In curriculum, pedagogy, infrastructure, delivery systems?

Did the changes made improve learning?

How good is our assessment process? How can it be improved further?

Express learning as a year-wise chart or Table, for a better appreciation.

Need for Change of Attitudes

From Grading Scoring right/wrong

answers. Comparison of

Students

Secretive, Exclusive.

Add On. Challenging Faculty.

To Assessment Considering the whole

reasoning process. Comparison to

measurable outcomes.

Public, Open, participative.

Embedded. Helping Students.

Thanks for Your Attention

Program Assessment Report Cover Sheet

Name of the Program : B.Tech (xxx Engg). Report Submitted by: Faculty name Date:

Intended Outcome and Associated College Goals Outcome No. 1: Students will be able to apply current

knowledge, techniques and modern tools learned in the discipline and by adapting emerging applications of mathematics, science and engineering to identify, analyze and solve technical problems. (NBA Criterion: yy)

Program Assessment Report Cover Sheet

College Goals Supported: Goals Nos. aa, bb, etc. aa: To foster teaching and learning in a supportive

environment. bb: To involve students in solving problems of interest to

industry, Government and Community organizations.

Outcome No. 2: Students will be able to conduct, analyze and interpret experiments and apply experimental results to improve processes related to the field.

Indicate connection with College Level goals (or PEOs) etc.

Program Level Assessment Report Outcome wise

Outcome No. 1: Students will be able to apply current knowledge, techniques and modern tools learned in the discipline and by adapting emerging applications of mathematics, science and engineering to identify, analyze and solve technical problems.

1. Means of Assessment for Outcome No.1: Course level Assessment of a List of Courses Relevant to Outcome No.1: Give List: CExxx, Ceyyy, CE zzz etc. These assessments compiled and evaluated by a group of assessor, to analyse the attainment of Outcome No1.


Assessment Rubrics: As discussed before in terms of students exceeding the standard, meeting the standard, approaching the standard and not meeting the standard.

Criteria for success: Say: 70% of the students meet or exceed the standard.

2. Description of Population to be Sampled: All students in these courses required to participate.


3. Methods Used to Choose the Sample (Minimum sample size 20 % for a population of 30 or above; otherwise whole class). No sampling used in this case.

4. Summary of Major Findings:

72% students exceeded or met the standard: Outcome successfully achieved.

5. Action needed to address assessment results: No action required in this case.

Performance Indicators

Key to measuring the outcomes.

Articulate key characteristics of an outcome.


Two essential characteristics:

1. Should be based on the focus of instruction. (Steps of problem solving, ethical analysis, organising

presentations, etc.).

2. Should contain appropriate action verbs (with clear message for both faculty and student: measurement of a specific outcome for faculty, and a specific performance for the student.

Performance Indicators: Example 1

Outcome: Students will demonstrate an ability to identify, formulate and solve problems.

Students are able to: identify key issues and variables. Recognize the need for multiple solutions. Analyse alternative solutions to a problem. Justify a solution to the problem.

Performance Indicators: Example 2:

Students will have the ability to perform experiments and interpret data.

Students are able to: Observe good safety practices. Design an experimental plan. Acquire data on appropriate variables. Compare experimental data and results to appropriate

theoretical models. Explain observed differences between model and experiment.

How do we assess these traits?

Students will Write experiment pre-

reports which include the experiment plan.

Perform the experiment. Submit data sheets from

the lab experiment.

Write a final lab report that includes the analysis and comparison.

Faculty will Assess quality of

experimental plan.

Observe safety practices.

Check if proper data for needed variables are measured.

Check whether comparisons have been made to theory with understanding.

Rubrics

A method for explicitly stating the expectations for student performance.

Describe characteristics for each level of performance

Provide clear information how well the students performed


Example Outcome: Ability to Apply knowledge of mathematics, science and engineering.

Possible Rubrics: Standard: The student shows clear evidence of being able to

apply knowledge, techniques, skills and modern tools learnt in the discipline and use knowledge of these disciplines to identify, analyze and solve technical problems.

Exceeding the Standard: The student shows superior evidence of being able to apply knowledge, techniques, skills..


Approaching the Standard: The student shows marginal evidence of being able to apply knowledge, techniques, skills...

Not Meeting the Standard: The student shows no evidence of being able to apply knowledge, techniques, skills

Analysis and Decision Making

Rubric Data: Useful information for making decisions about improvements.

For the outcome on Experimental Ability, a collective look at samples of data for all courses requiring experimentation would be useful.

Will bring out areas of strength and weakness for suitable actions.

Actions: should be collective, to make an impact. Check for Results on Actions in succeeding periods. No NBA penalty if some actions or strategies do not

work!

What Do We Measure?

NBA Criteria do not tell us how to measure outcomes.

What should we look for in student performance to determine the level to which an outcome is being attained?

We measure performance indicators.

Degree of attainment based on students level of performance on these indicators.

Performance Indicators: Skills or Behaviours That Can be Observed

Performance Indicator 1




Outcome


Program Educational Objectives

Graduates will work successfully as a member of a team to solve real world problems

Student Outcomes

Students will be able to demonstrate Ability to identify

and solve Real World Complex Problems.

Ability to conduct experiments and analyze data


Program Educational Objectives

Graduates will work successfully as a member of a team to solve real world problems

Student Outcomes

Ability to conduct experiments and analyze data


Observes safe lab practices.

Prepares a plan for the Experiment.

Gathers data on suitable variables.

Compares results to a model or a theory.

Developing Performance Indicators

Articulate key characteristics of an outcome.

Faculty need to learn to know it when they see it: Experience counts.

3 - 5 such characteristics for an outcome.

Should be observable and measurable, or demonstrated in student work.

Grading Assessment; But you can do both together.

When Do We Measure?

For most outcomes need multiple points (embedded perhaps in multiple courses, relevant to an outcome)

At what level (Lower level courses? Upper level courses?).

Terminal Courses (Statics-Strength of Materials Structures I Structures II etc.)

Capstone Course or Project.

accrditation tool for quality assurance

Documents