department of mathematics & statistics assessment...

Department of Mathematics & Statistics Assessment Report based on Graduate Learning Outcomes

February 2018 The following assessment report primarily covers the period from the fall semester 2015 through fall semester 2017, but in order to look at overall trends, assessment data obtained over this period is contrasted with data from fall 2011 through spring 2015. This comparison is made somewhat difficult by the fact that the department of Mathematics & Statistics at Missouri S&T used its own set of Graduate Learning Outcomes (GLOs) that aligned with the campus-wide learning outcomes in use at that time, but the campus-wide learning outcomes for graduate students were redrawn, so as to be uniform across departments, in fall 2015. While there is a common theme shared by both the old and the new GLOs, a direct one-to-one comparison of the results measured by the two metrics is not possible. Some overall comparisons, however, will be made. Results for the Period from Fall 2011 through Spring 2015 The learning outcomes on 14 doctoral students (7 with mathematics emphasis and 7 with statistics emphasis) who graduated during the period from fall 2011 through spring 2015 were measured using the old GLOs. Only a few masters’ students graduated under the thesis option during this period and no routine assessment on the learning outcomes on these masters’ students was conducted. One posthumous doctoral degree was also awarded during this period. The summary of the assessments made using the old GLOS are given in Tables 1 and 2, together with the rubric used for these assessments. Table 1a gives the summary statistics for doctoral students who selected the mathematics emphasis and Table 1b provides the rubric. Table 2a gives the summary statistics for doctoral students who selected the statistics emphasis and Table 2b provides the rubric. The definition of each numbered learning outcomes are given in the Appendix. In brief, they are: 1. Knowledge, 2. Critical Thinking, 3. Communication, and 4. Professional Development. Note that no assessments were made on master’s students with the non-thesis option.

Table 1a. Statistics on the Performance of Doctoral Students with Mathematics Emphasis Using the Old GLOs from Fall 2011 through Spring 2015

Department Learning Outcome

Performance Level 1

Unsatisfactory Performance

2 Average

Performance

3 Above Average

Performance

4 Outstanding Performance

Average Score

1Kno

wle

dge

a

1

6

3.71

b

1

6

3.71

2 C

ritic

al T

hink

ing

a

1

1

5

3.57

b

3

4

3.57

c

1

2

4

3.42

d

5

2

3.28

e

1

4

2

3.14

3 C

omm

unic

atio

n

a

2

2

3

3.14

b

1

3

3

3.28

4 Pr

ofes

sion

al

Dev

elop

men

t

1

2

4

3.42

Table 1 b. Evaluation Rubric for Mathematics Emphasis Doctoral Students – Old GLOs Department Learning Outcome

Performance Level 1


2 Average Performance

3 Above Average

Performance


1K

now

ledg

e

a

Below a B-average for courses listed in the program and/or C grade received in 10 or more hours; no in-depth knowledge in a specialty area

A GPA between 3.0 and 3.5 in core courses including courses that provide in-depth knowledge of a specialty area

A GPA above 3.5 and below 3.75 in core courses including courses that provide in-depth knowledge of a specialty area


b

Below 3.0 GPA performance in courses that require real-world problem solving

Has taken one or more courses that require real world problem solving and obtained a GPA between 3.0 and 3.5 in these courses

Has taken one or more courses that require real world problem solving and obtained a GPA above 3.5 and below 3.75 in these courses

Has taken one or more courses that require real world problem solving and obtained a GPA between 3.75 and below 4.0 in these courses

2

Crit

ical

Thi

nkin

g

a

Is not able to come up with conjectures and hypotheses, even with external guidance

Is able to come up with conjectures or hypotheses with minimal external help

Is able to come up with conjectures or hypotheses with no external help

Have shown a consistent ability to come up with conjectures or hypotheses with no external help

b

Is not able to prove theorems or carry out routine derivations without external help

Is able to prove theorems and carry out routine derivations independently

Is able to prove difficult theorems and carry out complex derivations independently

Have demonstrated an exceptional ability to prove difficult theorems and carry out highly complex derivations independently

c

Has difficulty reading and understanding published papers and verifying results without external guidance

Is able to independently read and understand published papers and verify results

Is able to independently read and understand published papers and verify results; have the ability to identify possible extensions/generalizations

In addition to performance level 3, have an ability to identify possible extensions and generalizations and prove them

d

Research is not original. At best only minimal contribution to area

Original research with a valuable contribution to area

Original research with a significant contribution to area

Original research with a highly significant contribution to area

e

Unable to explain the relevance of research in context of existing literature

Is able to explain the relevance of research in context of some of the existing literature

Is able to explain the relevance of research in context of all existing literature related to the research topic

Is able to explain the relevance of research in context of all existing literature related to the research topic and map out its impact on future research

3

Com

mun

icat

ion

a

Thesis is poorly written. Clarity of presentation and logical reasoning are lacking

Thesis has minor flaws. The clarity of presentation is satisfactory. Logical reasoning is present

Thesis has no flaws. Clarity of presentation is very good. Arguments are presented in a consistent and logical manner

Extremely well written thesis with no flaws. Presentation is exceptionally clear with rigorous arguments presented in a very logical and consistent manner

b

Presentation lacks coherence or a logical structure. Poor use of visuals and presentation aids

Satisfactory presentation with room for improvement. Minimal use of visuals or other aids

Very good presentation with main ideas explained well. Good use of visuals and other aids

Excellent presentation. Key points easily understood by audience. Highly effective use of visuals and other aids

4

Prof

essi

onal

D

evel

opm

ent Lack of solid background in

subject area. Minimal self-directed research reading or participation in seminars or colloquia

Good subject matter background. Satisfactory level of self-directed research reading and participation in seminars and colloquia

Above average subject matter background. Good level of self-directed research reading and participation in seminars and colloquia

Strong subject matter background. A high level of self-directed research reading and participation in seminars and colloquia

Table2a. Statistics on the Performance of Doctoral Students with Statistics Emphasis using the Old GLOs – Fall 2011 through Spring 2015


Performance Level 1


2 Average

Performance

3 Above Average

Performance


Average Score

1 K

now

ledg

e a

7

4.0

b

7

4.0

C

2

5

3.71

2 C

ritic

al T

hink

ing

a

6

1

3.14

b

6

1

3.14

c

5

2

3.28

d

5

2

3.28

e

1

5

1

3.0

f

3

4

3.57

3 C

omm

unic

atio

n

a

5

1

3.16*

b

2

5

3.71

4 Pr

ofes

sion

al

Dev

elop

men

t

4

3

3.42

*Only 6 candidates were scored on this learning outcome.

Table 2b. Evaluation Rubric for Statistics Emphasis Doctoral Students- Old GLOs


Performance Level 1


2 Average

Performance

3 Above Average

Performance


1

a

Below a B-average for courses listed in the program and/or C grade received in 10 or more hours; no in-depth knowledge in a specialty area


A GPA above 3.5 and below 3.75 in core courses including courses that provide in-depth knowledge of a specialty area


b

Below 3.0 GPA performance in courses that require real-world problem solving

Has taken one or more courses that require real world problem solving and obtained a GPA between 3.0 and 3.5 in these courses

Has taken one or more courses that require real world problem solving and obtained a GPA above 3.5 and below 3.75 in these courses

Has taken one or more courses that require real world problem solving and obtained a GPA between 3.75 and below 4.0 in these courses

c

Have not successfully completed a course or project involving programming competency and computational skills

Have successfully completed a course or project requiring programming competency and computational skills

Have successfully completed a course or project requiring above average programming competency and computational skills

Have successfully completed a very complex project requiring a high degree of programming competency and computational skills

2

a

Is not able to come up with conjectures and hypotheses, even with external guidance

Is able to come up with conjectures or hypotheses with minimal external help

Is able to come up with conjectures or hypotheses with no external help

Have shown a consistent ability to come up with conjectures or hypotheses with no external help

b

Has difficulty reading and understanding published papers and verifying results without external guidance

Is able to independently read and understand published papers and verify results

Is able to independently read and understand published papers and verify results; have the ability to identify possible extensions/generalizations

In addition to performance level 3, have a ability to identify possible extensions and generalizations and prove them

c

Research is not original. At best only minimal contribution to area

Original research with a valuable contribution to area

Original research with a significant contribution to area

Original research with a highly significant contribution to area

d

Unable to explain the relevance of research in context of existing literature

Is able to explain the relevance of research in context of some of the existing literature

Is able to explain the relevance of research in context of all existing literature related to the research topic

Is able to explain the relevance of research in context of all existing literature related to the research topic and map out its impact on future research

e

Unable to determine appropriate statistical procedures for solving real-world problems or develop new methodologies and models

Is able to determine appropriate statistical techniques solving standard real-world problems and able to develop new methodologies/models to solve non-standard problems

Is able to determine appropriate statistical techniques for complex real-world problems and able to develop new methodologies/models to solve complex non-standard problems

Highly creative in finding solution to very complex real-world problems and capable of developing entirely new and generalizable methodologies and models

f

Unable to correctly analyze data and make valid interpretations

Is able to analyze data from straightforward experiments and surveys and make valid interpretations of results

Is able to analyze data from complex but standard experiments and surveys and make valid interpretations

Is able to analyze data from complex and non- standard experiments and surveys and make valid interpretations

Table 2b Continued…


Performance Level

1 Unsatisfactory Performance

2 Average

Performance

3 Above Average

Performance


3

a

Thesis is poorly written. Clarity of presentation and logical reasoning are lacking

Thesis has minor flaws. The clarity of presentation is satisfactory. Logical reasoning is present

Thesis has no flaws. Clarity of presentation is very good. Arguments are presented in a consistent and logical manner

Extremely well written thesis with no flaws. Presentation is exceptionally clear with rigorous arguments presented in a very logical and consistent manner

b

Presentation lacks coherence or a logical structure. Poor use of visuals and presentation aids

Satisfactory presentation with room for improvement. Minimal use of visuals or other aids

Very good presentation with main ideas explained well. Good use of visuals and other aids

Excellent presentation. Key points easily understood by audience. Highly effective use of visuals and other aids

4

Lack of solid background in subject area. Minimal self-directed research reading or participation in seminars or colloquia

Good subject matter background. Satisfactory level of self-directed research reading and participation in seminars and colloquia

Above average subject matter background. Good level of self-directed research reading and participation in seminars and colloquia

Strong subject matter background. A high level of self-directed research reading and participation in seminars and colloquia

Observations made and Actions taken based on 2011-2015 Data

For this assessment, scores above a 2 were deemed as satisfactory. On average, the students performed well and satisfactorily met all the learning outcomes with a few exceptions. One student with mathematics emphasis achieved average performance (Score=2) in many areas and two students in the same emphasis area were marked average in thesis writing. Even those who scored well on thesis writing, several had difficulty in with the initial draft of the thesis, according to anecdotal evidence gleaned from their advisors. Actions taken: (1) the department encouraged all students to use the thesis writing support provided by the Office of Graduate Studies and (2) the department re-evaluated the English proficiency criteria used in admitting international students and/or expect those with writing deficiencies to take remedial action (such as enrolling in writing courses). While the TOEFL/IELTS minimums for admission eligibility were not raised, the practice of admitting fully-funded students without an IELTS score of 6.5 or above was heavily curtailed. Another area caught our attention is critical thinking, where one student in mathematics and one student in statistics score at the average level, but no action was taken because it was felt that this was an anomaly specific to one individual student.

Results for the Period from Fall 2015 through Spring 2017 As mentioned earlier, the primary focus of this report is to provide assessments on student performance for the period fall 2015 through fall 2017. The GLO data were collected not only for master’s students and doctoral students at their thesis/dissertation defenses, but also during the comprehensive exams of Ph.D. students. Feedback from thesis committee members obtained after written and oral exams were also utilized. As before, no data were gathered on master’s student graduating with a non-thesis option. The newly revised Graduate Learning Outcomes used since fall 2015 are given in Table 3. Table 4 presents the summary results for master’s students with the thesis option. Summary results for Ph.D. comprehensive exam are reported in Table 5 while Table 6 provides results recorded at the dissertation defenses of Ph.D. students. Table 3. Graduate Learning Outcomes Adopted in Fall 2015 Learning Outcome Description of Ability Knowledge An ability to apply knowledge of subject matter within

their field of study Communication An ability to communicate effectively within their field

of study Critical Thinking An ability to apply knowledge of subject matter within

their field of study Professional Development [not recommended for non-research students]

An ability to develop professionally within their field of study

Table 4. Summary Results for Master’s Students with the Thesis Option – New GLOs Learning Outcome Scores of MS Students with Thesis Option

Semester Student Knowledge Communication Critical Thinking

Professional Development

Average Percentile

SP2016 1 5 5 5 5 5.0 100.0% 2 5 5 5 5 5.0 100.0%

SP2017 3 5 5 5 5 5.0 100.0% SP2017 4 4 4 4 5 4.3 85.0%

SP2017 5 5 5 5 5 5 100.0%

FS17 6 3 4 4 5 4.0 80.0% Average 4.5 4.67 4.67 5 4.7 94.0%

Rating Scale: 1- Unsubstantiated, 2- Developing, 3- Acceptable, 4- Proficient, 5- Exceptional

Table 5. Summary Results for Ph.D. Comprehensive Exam – New GLOs Learning Outcome Scores of Students taking the Ph.D. Comprehensive Exam

Semester Number Knowledge Communication Critical Thinking


Average Percentile

SP2016 1 4.8 4.6 4.8 4.6 4.7 94.0%

SP2016 2 4 4 4 4 4.0 80.0% 3 3.2 3.6 3 3.2 3.3 65.0%

4 4 4.2 4.2 4.4 4.2 84.0% FS2016 5 4.2 4.2 4.4 4.2 4.3 85.0%

FS2016 6 4.2 3.8 3.8 4 3.95 79.0% SP2017 7 3.6 3.8 3.4 3.6 3.6 72.0%

SP2017 8 4 4 4 5 4.25 85.0%

SP2017 9 4.5 4 4.5 4.5 4.375 87.5%

SP2018 10 3 4 4 3 3.5 70.0% Average 3.95 4.02 4.01 4.05 4.00 80.0% Rating Scale: 1- Unsubstantiated, 2- Developing, 3- Acceptable, 4- Proficient, 5- Exceptional The results in Table 5 show that at the master’s level, all students are performing at a proficient or exceptional level with respect to all learning outcomes. At the doctoral comprehensive exam (Table 6 above), students performed at the proficient level with a performance at only the acceptable level seen in the knowledge category for three out of the ten students tested. The department would like to see an 80% rate of proficiency with the rest at an acceptable level in the future. At the doctoral dissertation defense level, one student scored a 2 for communication, another scored a 2.5 while a third scored a 3. In addition, one student scored a 2 in critical thinking while two others had acceptable, but not proficient scores. These low scores in communication and critical thinking are obtained by the same set of three students (out of 17) and this is reflected in their low average scores of 2.8, 2.9 and 3.25. Note that the first two low scores were obtained prior to spring semester 2016. Only one student performed below proficient below the acceptable level in any single category since spring 2016. While it seems, on the surface, that the problems we observed with the communication abilities of a few students during the 2011-2015 period persist in spite of some of the remedial action cation in 2015, this problem was mostly resolved by spring 2016. Action Items: Already taken – (1) the department streamlined and strengthened the quality control on the Ph.D. Qualifying Exam by mandating that each subject test be prepared and graded by a team of two faculty members, with the intent of maintaining high standards, and (2) the graduate policy committee has recommended that doctoral students should start their research earlier than has been the practice in the department, and strongly recommended that students taking the Ph.D. comprehensive exam provide written examples of research work such as papers written for publication. It is anticipated that the first action will ensure that only well-prepared students who are doctoral material will pass the qualifying exam and the second action will enable students and their advisors to work towards enhancing critical thinking and scientific writing skills of doctoral students buy identifying weaknesses

earlier in a student’s career and provide time for students to develop their critical thinking, research, and writing skills.

Table 6. Summary Results for Doctoral Students Dissertation Defense – New GLOs Learning Outcomes Scores of Doctoral Students Dissertation Defense

Semester Student Knowledge Communication Critical Thinking


Average Percentile

FS2015 1 5 4 5 5 4.8 95.0%

2 4 2.5 2 3 2.9 57.5% 3 5 5 4 5 4.8 95.0%

4 3 2 3 3 2.8 55.0%

SP 2016 5 4 5 4 4 4.25 85.0% 6 3.8 4.2 3.8 4 3.95 79.0%

7 4 4 4 4 4 80.0%

SS2016 8 4.4 4.2 4.6 4.6 4.45 89.0% SP2017 9 4.2 4.4 4.6 4.4 4.4 88.0%

SP17 Int 10 4.2 4.4 4.4 4.6 4.4 88.0%

SP2017 11 4 3 4 4 3.75 75.0% SS2017 12 3.75 2.75 3.5 3 3.25 65.0%

SP17 Int 13 4.6 4 4.6 4.6 4.45 89.0% SP17 Int 14 5 5 5 5 5 100.0%

SP17 Int 15 3.8 3.4 3 3 3.3 66.0%

SP17 Int 16 4.4 4.2 4 4.4 4.25 85.0%

FS17 17 4 4 4 5 4.25 85.0% Average 4.19 3.89 3.97 4.15 4.05 81.0% Rating Scale: 1- Unsubstantiated, 2- Developing, 3- Acceptable, 4- Proficient, 5- Exceptional Note: Int – Defended during intersession Current Department Policies and Protocol in addressing issues raised by the Assessment Process Issues identified, through the assessment of GLOs, as needing remedial action are brought up for discussion at the Department Graduate Policy Committee. Proposed remedial action plans are then voted on by the department faculty is such actions effect department policies, and directly implemented by the Policy Committee if the remedies are of a non-policy nature. Examples of such remedial actions were given previously.

Appendix

Department of Mathematics and Statistics GRADUATE LEARNING OUTCOMES – Fall 2011 through Spring 2015

Missouri S&T and Department Learning Outcomes

The Learning Outcomes defined Missouri S&T for all of its graduate programs are as follows:

1. Knowledge 2. Critical Thinking 3. Communication 4. Professional Development

The Department of Mathematics and Statistics adopted the following learning outcomes for its Applied Mathematics master’s program:

Learning Outcomes for Students in the Master’s Program with Mathematics Emphasis

1. Students will demonstrate a broad-based knowledge of mathematics and related areas through the successful completion of a course curriculum that contains courses in (a) fundamental core areas of mathematics and (b) areas of mathematics that can be applied to solving real-world problems or in subject areas that employ mathematics as a fundamental component.

2. Students will demonstrate the ability to (a) make conjectures and form hypotheses, (b) reason mathematically by constructing mathematical proofs/derivations, and (c) understand proofs and derivations found in published literature and verify their validity.

3. Students will demonstrate their ability to communicate mathematical ideas effectively by (a) producing a clear, concise, and well-written master’s thesis and (b) making an oral presentation of their research that highlights the key ideas and results in a clear, logical, and accessible manner, using visual and other aids if necessary.

4. Students will demonstrate that they have the necessary skills to be life-long learners and possess the ability to develop professionally after their graduation by achieving Learning Outcomes 1 and 2 above and through self directed reading of mathematical literature and/or attendance of graduate seminars and colloquia in their own department as well as in other departments.

Learning Outcomes for Students in the Master’s Program with Statistics Emphasis

1. Students will demonstrate a broad-based knowledge of statistics and related areas through the successful completion of a course curriculum that contains courses in fundamental core areas of (a) theoretical and (b) applied statistics courses that emphasize solving of real-world problems.

2. Students will demonstrate the ability to (a) reason mathematically by constructing proofs or derivations, (b) understand proofs, derivations, and statistical methodologies found in published literature and verify the validity of statistical claims made in such literature, (c) determine the appropriate statistical methodologies or models required for finding answers to real-world problems, and (d) correctly analyze data and make valid interpretation of results from such analyses.

3. Students will demonstrate their ability to communicate statistical ideas effectively by (a) producing a clear, concise, and well-written master’s thesis and (b) making an oral presentation of their research that highlights the key ideas and results in a clear, logical, and accessible manner, using visual and other aids if necessary.

4. Students will demonstrate that they have the necessary skills to be life-long learners and possess the ability to develop professionally after their graduation by achieving Learning Outcomes 1 and 2 above and through self-directed reading of statistical literature and/or attendance of graduate seminars and colloquia in their own department as well as in other departments.

Learning Outcomes for Students in the Doctoral Program with Mathematics Emphasis

1. Students will demonstrate a broad-based knowledge of mathematics and related areas as well as an in-depth knowledge of a specialty area in mathematics through the successful completion of a course curriculum that contains courses in (a) fundamental core areas of mathematics and (b) areas of mathematics that can be applied to solving real-world problems or in subject areas that employ mathematics as a fundamental component.

2. Students will demonstrate the ability to (a) make conjectures and form hypotheses, (b) reason mathematically by constructing mathematical proofs/derivations, (c) understand proofs and derivations found in published literature and verify their validity, (d) create new knowledge through original research, and (e) explain the relevance of their research in the context of existing research literature .

3. Students will demonstrate their ability to communicate mathematical ideas effectively and understand mathematical literature written in other languages by (a) producing a clear, concise, and well-written Ph.D. dissertation and (b) making an oral presentation of their research that highlights the key ideas and results in a clear, logical, and accessible manner, using visual and other aids if necessary.

4. Students will demonstrate that they have the necessary skills to be life-long learners and possess the ability to develop professionally after their graduation by achieving Learning Outcomes 1 and 2 above and by self-directed reading of mathematical literature and

attendance of graduate seminars and colloquia in their own department as well as in other departments.

Learning Outcomes for Students in the Doctoral Program with Statistics Emphasis

1. Students will demonstrate: a broad-based knowledge of statistics and related areas, an in-depth knowledge of a specialty area, and computational skills, through (a) successful completion courses in fundamental core areas of theoretical statistics, (b) successful completion of applied statistics courses which emphasize solving of real-world problems, and (c) successful completion of a course or a project which shows evidence of computationally intensive programming competency and computational skills.

2. Students will demonstrate the ability to (a) reason mathematically by constructing proofs and derivations, (b) understand proofs, derivations and statistical claims found in published literature and verify their validity, (c) create new knowledge through original research, (d) explain the relevance of their research in the context of existing research literature, (e) determine the appropriate statistical methodologies or models required for finding answers to real-world problems as well as develop new methodologies and models, and (f) correctly analyze data and make valid interpretation of results from such analysis.

3. Students will demonstrate their ability to communicate statistical ideas effectively by (a) producing a clear, concise, and well-written Ph.D. Dissertation and (b) making an oral presentation of their research that highlights the key ideas and results in a clear, logical, and accessible manner, using visual and other aids if necessary.

4. Students will demonstrate that they have the necessary skills to be life-long learners and possess the ability to develop professionally after their graduation by achieving Learning Outcomes 1 and 2 above and through self-directed reading of statistical literature and attendance of graduate seminars and colloquia in their own department as well as in other departments.

department of mathematics & statistics assessment...

Documents