Undergradute Curriculum Assessment/Evaluation Report

Ted BakerCS Faculty Workshop

18 August 2009

Overview

• Training for instructors• Review of student exit survey responses• Accreditation status summary• Things we need to doNot the most logical order, but the most critical items first.

Training for Instructors• Go to www.cs.fsu.edu/ugcc for

– approved course objectives; include them.– SMALC outcomes that you must assess (if any)

• Rubric you must use• Action plan from prior year

– end-of term reports from prior instructors• Go to DOF website for most recent

– Attendance statement– ADA statement– Honor code statement

• Send syllabus to DUGS at start of semester– Preferably, in time for review before start of classes

• We will soon (hopefully) have a web-based system for uploading the syllabi.

Other Beginning-of-Term Matters

• Confer with instructor from prior term, if possible. Get advice, materials.

• Find out who is teaching the course for distance/PC campus, and communicate with them.

• If you want to change textbooks, propose the change to the UGCC (but try for stability).

• Beware of university and departmental requirement for final examination in all UG courses, given during final exam week.

• If you have a TA, be sure to supervise, observe, mentor, and evaluate.

Issues from Exit Survey

• See http://www.cs.fsu.edu/ugcc/smalcs09/surveys.html• Today, see spreadsheet.

Issues raised by student advisory panel

• See http://www.cs.fsu.edu/ugcc/minutes/ugsac_02_10_09.html

• Discrete math sequence is not working– Some math instructors are not teaching the courses well– Students are not motivated, because they don’t see the applications– We are not using the material enough in our courses soon enough, so

student forget what they did learn– Significant repetition between and within Discrete Math courses,

COP4530, and COP4531– Many students seem to be discourage dfrom major by the DM sequence

• Have CS faculty teach discrete math• Integrate DM II into CS courses where it is used

More• COP4531 tries to rush through too many algorithms, without adequately

developing the underlying design techniques• Courses that are logically connected, as prerequisites, need to be offered

in contiguous semesters, so that students don’t forget what is needed from the preceding course

• Overlapping course pairs not well coordinated and not consistent: sometimes duplication, and sometimes gaps. Need more instructor coordination, more consistency.• COP3014-COP3330• COP3330-COP4530• COP4530-COP4531

More• Problems with instructors who are new to a course. Suggest less

turnover of instructors in a course.• Excessive use of Powerpoint makes lectures move to fast, too

hard to follow. Suggest more hand-writing to engage students.• Need more exposure to event-driven programming• Electives are a problem. Department should poll students to see

what electives they want.• Need more room for electives in the curriculum (less required

courses). • Department seems to put more priority on graduate education

than undergraduate.• Too much overlap between CDA3101 and CDA3100.

More• COP4710 requires students to use Java. This ends up being

duplicative of the Java course.• Intro to Unix and Unix Tools courses sometimes overlap a lot, and

vary a lot in content from term to term.• We are not making good use of recitation meetings. They should

not just be another lecture, perhaps by the TA. They should be used to interact with students, with a focused activity.

• COT4420 notes need updating.• More use of IDEs (along with command line) in courses beyond

the introductory level.• COP3014 might attract more students if there were more

interesting (e.g., GUI) assignments in it.

Accreditation Status

• Interim report submitted in June, now being reviewed by ABET/CAC representative. – You can read the report if you go to

http://www.cs.fsu.edu/ugcc/interim.pdf.– Username/password = smalcs/smalcs

• If this report is approved (in July 2010), we will be approved for another 4 years.

• Otherwise, we would probably be asked to do another interim report in June 2011.

Issues in the Report

• Coverage of social and ethical issues related to computing.

• Adequacy of elective offerings.• Sufficiency of university resources.• Outcomes and assessment process.

Ethical & Social Issues• Added material to DB and SE courses• Created new course, CIS4250• This addresses the ABET complaint, but

– DB and SE course treatment is now duplicative– It can’t be phased out until grandfathered students graduate,– and we find a way to handle distance students.– Distance students can’t do oral debates.– Oral debates can’t be made optional for CIS4250, since it is an “oral

competency” course for SACS accreditation purposes.• One idea:

– Create another ethics course, related to computing (maybe only 2 hrs, maybe only DL, maybe only taught by adjuncts)

– Students who take that course must also take SPC2600 to cover oral competency requirement.

Elective Offerings• This is partly a student perception problem.• Report shows that we actually are offering more electives than the

institution of the chair of the team that made the criticism.• Remaining issues:

– Many are not practically available to most students, due to tough prerequisites (dual-listed graduate courses, other electives as prerequisite, senior-level courses as pre-requisite)

– Distance students have fewer options• What we are doing:

– Offer popular ones, without daunting pre-requisites, more often.• Things we might do:

– Allow students to take some electives out of CS?– Work with other Florida universities to share distance course offerings?

Adequacy of Resources

• ABET reviewers probably thought they were helping the program, by putting pressure on the FSU administration to replace lost faculty lines.

• Our interim report explains effects of current economic distress on the University, and that

• we are lucky to have suffered no layoffs, and to have been allowed to replace Ken Baldauf and Chrissy.

• I have explained this also, in person, to the ABET reviewer.• We can hope that ABET will appreciate putting further

pressure on FSU will not help.

Assessment

• We have expanded our assessment process.• Tried to combine ABET requirements with SACS/SMALC

requirements.• Doubled number of SMALC outcomes

– to cover min. set of ABET/CAC-required outcomes• Added end-of-term instructor course reports

– to provide more useful information for program improvements• Added focus-group meeting with undergraduate students• Added more formal review and reporting process

– including this report, to the workshop

Remaining Problems

• Faculty education.– Everyone needs to understand the process and

the goal.– Therefore, this talk and more to come.

• Increased workload (“death by assessment”)– Therefore, proposals to come, to reduce load.

• Fundamental fallacy in ABET model– We can still make a good faith effort.

“Death by Assessment”• Gathering the assessment data is a lot of work.• This is distracting us from other essential duties, including the evaluation

and planning that are supposed to be driven by the assessment process.• Faculty members still do not all understand what is expected.• The DUGS is overwhelmed by the workload.

– The DUGS has already spent more than a man-month at this, dunning for data, meeting with people to explain what is needed, extracting files from e-mails and Web pages, editing, renaming to a standard system, copying into directories, reformatting into .pdf, renaming and reformatting them, creating web links to the files, etc. Moreover, thhis manual work is very error-prone.

– Much of the supporting data files still need to be populated on our web site, to which the interim report refers.

– The DUGS has been meeting with the webmaster since late fall 2008 to develop an on-line tool that would automate most of this, but the process seems to have stayed on the back burner.

Assessement Processes

• State Mandated Academic Learning Compacts– We specify some specific minimal achievements

(“student educational outcomes”, and “program outcome”) for our graduating students, which we must assess and report every year.

• ABET– Starts similarly to SMALC, but goes further.

We must demonstrate that we use the data from assessments as the basis for program improvements.

ABET Model

• Engineering method should be applied to the process of education.

• We define a set of desired outcomes for our educational program.

• We gather data on how well those outcomes are being achieved.

• We review and evaluate the data, which drives our plans to improve our program.– Not just the instructional process, but also the meta-

processes (outcome statements, assessment methods, evaluation process, etc.)

Fallacy in the Model

• Assumes we have a repeatable process.– So that we can vary one parameter, and assess how this has

improved/hurt the program.– This seems to be called “regression-discontinuity design” by

experimental design theorists.– We have different people teaching courses, sometimes in the

same term.– Students coming into courses also have had different

experiences in preceding courses.– There are so many differences in courses and students that

we can’t isolate the effects of any specific change with reliability.

What can we do?

• Try to make the process more repeatable?– Not rotate instructors so frequently.– Communicate more between successive instructors,

reusing text, slides, etc.• Take a more holistic view of the program

assessment, recognizing that regression-discontinuity experimental method is not feasible in this environment.

• Turn in syllabus before the term, and have it reviewed for consistency.

How to reduce workload

• Increase priority of on-line data collection.• Collect SMALC data only once a year.• Reduce set of courses in which SMALC data is collected, to a

few capstone courses.• Collect instructor reports in every course, and rely more on

them for directing program improvements.• Extend survey to gather more feedback from alumni.• Continue • Do thorough review of assessment data less frequently,

rotating through courses and outcomes on a multi-year cycle.– Which ones to start with? The math/theory sequence?

Remaining Instructor Responsibilities

• SMALC data collection, in some courses.• End-of-term report, in every course.• Reviewing SMALC and other data, and making

specific recommendations to DUGS and/or UGCC.

Other UG Curriculum Issues

• New BA degree requirements• Relaxing prerequisites, to make courses more

accessible

New BA Requirements

• Tougher than BS; requires 12 more courses than BS, in humanities.

• 2 students graduated with BA in 2007, none in 2008 & 2009.• ABET considers this a separate degree; charges double fees,

sends extra accreditation visitor; wants examples of student transcripts to prove program is vital.

• We can’t provide example transcripts.• We decide to drop ABET accreditation of the BA degree.• This opens up possibilities

– To attract more students to CS– By eliminating for the BA some requirements imposed solely to

satisfy ABET/CAC

Relaxing Prerequisites

• CS curriculum discourages cross-disciplinary studies, due to very deep prerequisite structure.

• This also causes scheduling problems for students, and limits their choices of electives.

• It is not clear that we actually depend on all these prerequisites.

• It is time to review all prerequisites, to see if they are essential.

Some changes, already

• Java for non-majors– dropped C++ prerequisite

• C++ for non-majors– Reduced math pre-requisite

• COP4610– Reduced CDA3101 to co-requisite or instructor permission

• Security courses– COP4530 replaced by CDA3100

• Can we go further?