ii. methods and instrumentsbioquest.org/teagle/beloit-philosophy-dept-teagle-report...teagle grant...
TRANSCRIPT
Teagle Grant Project:
The Philosophy Program at Beloit College
Phil Shields, Matt Tedesco, Heath Massey
June, 2008
I. Task
Shortly before undergoing an external review in fall 2006, the members of the Department of Philosophy
and Religious Studies agreed to participate in “Assessing the Value Added to Liberal Education by
Academic Majors.” For the departmental self-study required for this review, we were charged with
considering how the philosophy and religious studies majors support the general education goals expressed
in Beloit College’s recently-revised mission statement. So, it appeared that we would be in an excellent
position to contribute to the Teagle Foundation project by examining “the complementary relationship
between general education and the major[s].” This examination, we learned, would focus on three
particular emphases of a liberal arts education: critical thinking, civic engagement, and quantitative
reasoning. Our task would be to determine how our coursework and our advising contributes to the
realization of these general education goals and then to devise ways to pursue these goals more effectively.
Our self-study in the fall of 2006 gave us the chance to begin discussing the role of our department with
respect to the mission and goals of the college. Then, in the spring of 2007, we focused on what kind of
information we needed and what instruments could provide it. We decided to use examples of student
writing from different stages of their coursework as a means of direct assessment, as well as exit surveys
and transcript breakdowns for indirect assessment. By the end of the academic year 2006-7, we possessed
writing portfolios and questionnaires provided by graduating seniors, as well as transcript breakdowns for
philosophy majors and minors dating back to 2000. Over the summer of 2007, we began to process this
information by reading all the materials and discussing possible strategies for addressing what we learn
from them. This work has continued through academic year 2007-8, which has yielded a new batch of
survey and portfolio data, and during which we began to implement some of the changes that followed
from our early work on this project, as well as our self-study.
II. Methods and Instruments
In order to assess the value of our department’s contributions to liberal education at Beloit, it was necessary
to identify methods for determining how our major benefits students with respect to the three areas of
emphasis specified in the grant project. Next, it was necessary to produce instruments for gathering the
information needed for this assessment. In the humanities, this is somewhat challenging, for much of what
we teach—critical thinking, as a prime example—does not readily allow for quantitative measurement. We
were somewhat skeptical about the possibility of measuring outcomes with respect to our educational goals.
So, the challenge was to determine how to find data that would help us accomplish the task of this project
without substituting a reductive, piecemeal form of analysis for the holistic approach to student learning
that is central to philosophy and religious studies.
As our department is comprised of two distinct programs, we approached our task in different ways,
resulting in the employment of different instruments. On the philosophy side, we chose to use three
primary instruments:
1. Questionnaire for Graduating Majors
In 2005, we developed a brief but relatively detailed exit questionnaire for our graduating philosophy
majors so that we could learn from their experiences in the program. The questionnaire, as initially
developed, was comprised of seven open-ended questions, such as: “If you were given free reign to
redesign the department (including classes offered, requirements for the major or minor, etc.), what
changes would you make, and why?” Student responses to these questions therefore focus on their
particular concerns. In spring 2007, in response to the charge of this assessment project, we added six
“circle-the-number” questions which ask students to rate their agreement or disagreement with a statement.
In spring 2008, we further modified the questionnaire, expanding the “circle-the-number” questions to ten
in order to get better, more precise feedback from our graduating students. We designed these questions to
address the specific issues of critical thinking, civic engagement, and quantitative reasoning, or the more
general topic of the relation of coursework and advising in philosophy to all-college learning objectives.
For example, students are asked whether they agree or disagree with this: “Advising sessions with
philosophy professors helped me to make good choices about courses to take in other fields.” The purpose
of these questions is to help us to gauge our students’ perception of how well the philosophy program
supports the general education goals in question.
2. Transcript Analysis
Critical thinking is a skill that we aim to develop in most if not all of the courses we offer. We are less
focused, however, on promoting quantitative reasoning and civic engagement, except to the extent that
courses in ethics will enable students to be more reflective when they do engage in political or community-
based activities. Consequently, we need to know whether students who major or minor in philosophy are
cultivating these competencies in their other coursework. In order to help us determine this, we requested
from the registrar a breakdown of course distribution by division for all majors over the last decade, with
interdisciplinary studies and study abroad included separately. Our intention was to use this data to track
general trends in the kinds of classes our students are (and are not) taking over the course of their time at
Beloit. Since this data does not include which particular courses were taken by each student, it cannot
provide a detailed picture of whether and how students pursue general education goals in their coursework
outside philosophy. To address this concern, we supplemented what we have gleaned from the registrar’s
data by reviewing actual transcripts of recent philosophy majors with attention to particular kinds of classes
that stress quantitative reasoning and to study abroad and internships which demonstrate a concern with
civic engagement.
We began our transcript analysis by asking the Registrar’s Office for copies of the transcripts of all
philosophy majors from the graduating classes of 2000 to 2008. Then we examined these transcripts for
courses and certain co-curricular experiences related to quantitative reasoning and civic engagement. In
addition, we looked at courses in foreign languages and courses related to creative expression, which are
two other areas we believe are central to a good liberal education. We created a spreadsheet including each
student’s year of graduation, major (in addition to philosophy), minor, and coursework related to
quantitative reasoning, civic engagement, and languages and arts. Under each category, we listed courses
that the student took which promote these general education goals. Since Beloit has no college-wide
requirement addressing these goals specifically, there are no general standards for which courses promote
them. Consequently, we had to use our judgment in determining which courses to include in our analysis.
Under quantitative reasoning, we included most courses in mathematics, computer science, physics, and
economics, as well as courses in biology, psychology, sociology, and anthropology that focus on statistics.
Under “civic engagement,” we included courses in ethics, political science, and a variety of other
disciplines, as well as internships, study abroad, and domestic, off-campus study. In order to determine
whether to include a course in this category, we considered whether it appears to prepare students to act
thoughtfully and ethically. Finally, we created our own category “language and arts,” under which we
included art, dance, music, theater, and creative writing. After creating the spreadsheet displaying each
student’s cousework in these areas, we counted the total number and the percentage of students who had
taken courses in these areas. The results of our transcript analysis are presented in Appendix B.
3. Writing Portfolio
By the time they are seniors, all philosophy majors will have taken Introduction to Philosophy, one or more
ethics courses (including Ethical Theory, Environmental Ethics, Biomedical Ethics, and other topics), and
Colloquium in Philosophy, a capstone course in the major. These common experiences provide an
opportunity for the direct assessment of student learning. In philosophy, as in other areas in the humanities,
writing assignments provide not only a basis for student evaluation, but also a means for their reflective
engagement with the subject matter. Writing carefully and thinking critically go hand in hand. Also, most
major assignments in our courses are papers or essay exams. Through our discussion of how to directly
assess the contributions of coursework in philosophy to critical thinking, civic engagement, and
quantitative reasoning, we decided that it would be beneficial to review student writing at various stages.
In order to make this possible, we asked all senior philosophy majors to submit a writing portfolio
comprised of a paper submitted in their first philosophy class, a paper from an ethics class, and the research
paper they produced in the capstone colloquium. Over the summer of 2007, we reviewed the portfolios
from that year’s majors with the aim of discovering how the quality of critical thinking they exhibit
improves (or does not). Rather than beginning with rubrics for assessment of the paper assignments, which
is one approach taken by our colleagues in religious studies, we first read the papers, focusing on what
signs of good thinking they do (or do not) display. We compared each student’s early, middle, and later
work to determine what kind of development is apparent. At the end of the process, we discussed our
findings and attempted to articulate what, in our collective view, characterizes sound critical thinking and
to identify ways that our courses appear to be more and less successful in cultivating it. More recently, we
have collected those same portfolios from our 2008 graduating majors, with the goal of repeating the
review undertaken last summer. Two sample portfolios are included in Appendix C.
III. Self-assessment
Participating departments in this grant project have been charged with assessing their majors with respect to
three particular learning goals: critical thinking, quantitative reasoning, and civic engagement. We will
address each of these individually.
1. CRITICAL THINKING
Critical thinking is rightly thought to be central to a philosophical education, though what it means to think
critically takes many different forms depending on the course and the instructor. We try to teach students
to recognize arguments, to analyze them, and to respond critically in a variety of ways. We are all
committed to the idea that thinking critically involves being able to recapitulate a line of reasoning and
engage in dialogue, raising objections and concerns, to see if that reasoning approaches the truth. At one
extreme we teach deductive reasoning, where students learn to evaluate the validity or invalidity of
inferring one statement from another or other statements. A valid deductive argument is one whose
premises provide conclusive grounds for the truth of the conclusion. Many arguments and inferences made
in philosophy, as in life, do not lend themselves to such definitive analysis, and most of our courses involve
inductive reasoning as well, where inferences are not simply valid or invalid, but gray and probabilistic.
Inductive arguments can only be evaluated as strong or weak, depending on the degree of support the
premises lend to the conclusion. Critical thinking can also involve unpacking the assumptions behind a
statement or argument, uncovering unstated premises and making these underlying assumptions more
explicit. While this in itself need not involve evaluation of the statement or argument, it may at least help
us recognize the implicit complexity and ambiguity, thereby making our understanding and assessment of it
more adequate and responsible. While the deductive reasoning is explicitly addressed in the formal logic
course, it remains implicit in our other courses as a precondition for other types of critical thinking. The
second and third types of critical thinking, inductive reasoning and presupposition analysis, are explicitly
addressed in all our other courses from the introductory level to upper-level seminars.
The senior exit questionnaire provides some indirect means of assessment of critical thinking in general.
On a scale of 5 from “Strongly agree” to “Strongly disagree,” 8 out of the 9 senior exit questionnaires in
2007 were marked 1 for “Strongly agree” to the statement, “The courses I took in philosophy promoted my
critical thinking abilities.” The response to this question was even more effusive in 2008, with all ten
respondents offering strong agreement. And this sentiment was echoed in the other questions focused on
critical thinking that were added to the survey in 2008 (see Appendix A for full details). Taken together,
these responses signal strong agreement among our majors and minors that their study of philosophy
developed their capacity for critical thinking.
Prior to the quantitative questions were some open-ended qualitative questions. In 2007, over half of these
bring up critical thinking as a strength of the department or a capacity they have gained from their
experience in the department. 2007 graduates write, for example: “The department trains its students to
think in a very critical way, which is extremely valuable to me, because I now apply that method in many
other areas of my life;” “I found the critical thinking skills learned in my philosophy classes crossed over to
my science classes (especially in organic chemistry lab);” and “I feel as though the subject, while it did not
prepare me directly for any particular job, I still think about things critically every day, and I feel as though
it has really helped me become a more balanced and calm person.” All of the other four questionnaires
identify closely related skills like “clear analysis” and “effective argumentation” in answer to the open-
ended question as to what they had gained from their experience in the department. Clearly the students
perceive that they are acquiring critical thinking skills.
The open-ended questions in the 2008 surveys reflected these sentiments as well, where the vast majority of
them once again explicitly pick out critical thinking as a clear benefit of their philosophical studies. These
graduates write, for example: “I’ve also become a more analytic thinker and I’m able to identify and
articulate assumptions in arguments and flaws in reasoning. This has helped me in other disciplines as well
as philosophy;” I have gained a love of philosophy and the search for truth, a set of tools for critically
evaluating my own and other’s arguments and beliefs, and an expanded view of the nature of reality;” and
“I’ve gained the ability to think critically, examine arguments, and see situations from many perspectives.
The thought-provoking discussions that took place in my classes in the department, even in Logic,
expanded the way I think and analyze arguments.” Comments like these can be found in virtually every
questionnaire.
Our direct method of assessment involved portfolio analysis. This is both more meaningful and more
difficult to implement. We are looking to track progress in critical thinking skills by examining three
papers from three different stages in the students’ study, ranging from their introductory course, a 200-level
course and a 300-level seminar. We have identified several criteria for tracking progress. First,
intelligibility. To what degree does the paper maintain a coherent focus? Second, critical distance. Is the
writer too attached to initial intuitions and does he or she fail to seriously consider obvious or important
objections? Third, sophistication. Does the paper make inappropriately sweeping generalizations and fail
to qualify statements carefully? And fourth, judiciousness. Does the writer fail to perceive and prioritize
the relative significance of criticisms and objections, or if doing research, the relative significance of the
sources?
2007 was the first year in which we have collected portfolios from graduating seniors in philosophy, and
unfortunately our results are hampered by the fact that we only received four complete portfolios from
graduates. For a variety of reasons, we have struggled with gathering these portfolios in 2008 as well.
Given the limited sample from 2007, we still discerned clear progress in the four areas listed above, though
there were also some disappointments where students seem to have been ill-prepared for the challenges of
writing an independent research paper for colloquium and regressed to earlier levels according to some of
our four criteria. [EXPAND WITH REFERENCE TO APPENDIX C, INCLUDING REFERENCES TO
SPECIFIC PORTFOLIO EXAMPLES] We have implemented changes in our colloquium in attempt to
address these weaknesses. We also plan to be more proactive in collecting portfolios in the future so this
important direct measure of progress in critical thinking will be more significant.
2. QUANTITATIVE REASONING
Of the three learning goals that comprise the emphasis of this Teagle grant project, quantitative reasoning is
the one goal that the philosophy department least engages. That is, while we believe that we are intensely
engaged in advancing our student majors with respect to both critical thinking and civic engagement, very
little of what we do within our departmental courses qualifies as quantitative reasoning. What little we do
with respect to quantitative reasoning can be found in the logic course taught each semester, to the extent
that logic is a kind of precondition for mathematics, and the kinds of symbolic proofs emphasized in formal
propositional and predicate logic model the procedures for derivation underlying mathematical proofs. The
connection here, however, is tenuous at best.
Given the dearth of quantitative reasoning offered in our departmental courses, we have employed two
different strategies for learning whether or not our majors seriously engage this learning goal during their
time at Beloit College. First, we have investigated the transcripts of recent graduates to find evidence of
quantitative reasoning in their choice of courses outside of the philosophy major. There is, however, a
serious limitation to this kind of analysis that should be highlighted here. Some courses, by name and
catalog description alone, provide sufficient reason to believe that the student is engaged in quantitative
reasoning in the course. The clearest examples here are most mathematics courses, as well as discipline-
specific statistics course that are offered in many departments in the natural sciences and social sciences.
But importantly, not finding these kinds of classes does not demonstrate that the student is not engaged in
quantitative reasoning. This is because many courses, depending on the particular instructor and particular
syllabus, may prominently feature quantitative reasoning even if the course title or description fail to make
this clear.
Using our best judgment in what to include under coursework promoting quantitative reasoning, we
discovered that a majority of recent philosophy majors have taken at least one course that we would expect
to demand significant quantitative reasoning, but few of them have taken more than a handful of such
courses. According to our transcript analysis (see Appendix B), Only 22% of philosophy majors
graduating between 2000 and 2008 did not take any courses related to quantitative reasoning. In
comparison, 78% of all students took one or more of such courses, 44% took two or more, and 24% took
three or more. Thus, it appears that four out of five recent philosophy majors have had at least some
training in quantitative reasoning at Beloit, but fewer than half have taken multiple courses in which this
competency is taught, and only about a quarter of them have pursued math, physics, computer science,
economics, or statistics in depth. If we suppose that students who take fewer than two courses in these
areas are likely not to develop good quantitative reasoning, then it is incumbent on us to use advising
sessions to encourage students to make coursework in these areas a higher priority.
Besides transcript analysis, we have also added objective questions that specifically address advising to our
recently-developed senior exit questionnaire. These questions emerge from the idea that, because our
major does not substantially address quantitative reasoning (among other valuable learning goals), the onus
is on us as advisors to see that, as best we can, these goals are being met in the classes that our advisees
choose outside of the major. In discussions about advising prior to the return of the questionnaires, we had
agreed that, generally speaking, we had paid insufficient attention to these kinds of considerations in
advising. The questionnaires confirmed this: in comparison with other objective questions about our
students’ experiences in the major, the scores on questions concerning advising outside the major generally,
and concerning quantitative reasoning specifically, were uniformly lower. The two advising questions in
the questionnaire received the lowest scores (that is, the most disagreement) from our majors and minors,
and the question concerning our advising with respect to quantitative reasoning in particular clearly stood
out from the rest (see Appendix A). Tellingly, with 16 respondents over two years, only two strongly
agreed that their advisor encouraged them to take courses outside of the department that focus on
quantitative reasoning. While the sample size here is admittedly small—at present, just two graduating
classes—the results were clear enough to suggest that we need to pay more attention to our advising in this
respect. These preliminary results raise further, deeper questions about the nature and purpose of
advising—questions also briefly gestured towards in Appendix A—but those questions are largely beyond
the scope of this project.
At this point, it is also worth mentioning that a tension exists in advising between classes that emphasize
quantitative reasoning and classes that emphasize other valuable learning goals that our department does
not significantly address. While this particular grant project concerns three specific learning goals, it is
worth noting that Beloit College’s mission statement explicitly mentions six essential skills that students
are expected to develop, of which quantitative reasoning is merely one. And this list is not exhaustive: it
does not include, for example, creative expression, which seems on the face of it to be no less important a
learning goal than quantitative reasoning. This creates some puzzles about advising that do not have a clear
resolution: advising certainly should not be reduced to a checklist of all imaginable learning goals, and
given that different classes presumably engage these learning goals differently, it is not clear that a student
should be steered away from one kind of class with one kind of learning goal towards a different kind of
class with a different learning goal. And this puzzle, of course, sits against the backdrop of the problem
raised above: namely, that it is not at all obvious which courses meet learning goals such as quantitative
reasoning, given that a course’s name and catalog description very inadequately reveal the actual content of
the class.
To better understand and measure our majors’ progress in meeting these other valuable learning goals
beyond those proscribed in this grant project, we did not limit our transcript analysis to classes with an
emphasis on quantitative reasoning. Instead, we also gathered data from transcripts on the arts and
languages courses taken by our students. It is interesting to compare the two sets of data. For instance,
while only 22% of students took no quantitative courses, an alarming 52% took no courses in a foreign
language. On the other hand, 37% took two or more language courses, compared to 44% in quantitative
reasoning, and 24% took three or more, the same as in quantitative reasoning (see Appendix B). Thus, the
portion of philosophy majors who study foreign languages in some depth—about one quarter—is
comparable to the portion who study mathematics or math-based fields in depth. However a smaller
portion of students take at least one or two language courses, as compared with those who take one or two
quantitative courses. Since there is no general education requirement to study a foreign language, and
Beloit’s “breadth requirement” provides many alternatives for students who wish to take languages, this is
another area that we need to address in our advising. It is particularly disturbing how few philosophy
majors study languages that would enable them to read philosophy in other languages in which it has been
written, especially Greek, Latin, German, and French. In addition to noting courses in the foreign
languages, we tracked courses that encourage students to express themselves creatively. In this area, 32%
of recent philosophy majors did not take a single course in art, dance, music, theater, or creative writing,
while 43% took two or more such courses, and 29% took three or more. So, while a slightly higher
percentage of students pursue the arts in more depth than those who do so in courses related to quantitative
reasoning or foreign languages, almost a third of all philosophy majors have completely missed out on this
part of a liberal education.
3. CIVIC ENGAGEMENT
Of the three learning goals emphasized by this Teagle grant project, civic engagement is in some respects
the most complex. With respect to the other two goals—quantitative reasoning and critical thinking—there
is a sense in which practice is sufficient for learning. That is to say, for students to develop their critical
thinking or quantitative capacity, it is enough that they engage in the practice: we want students simply to
think critically and work with numbers, whatever the context. These skills are like muscles that need to be
exercised, where presumably any number of exercises would be sufficient for this purpose. Civic
engagement, on the other hand, is quite different: we don’t want students to simply act, but we went their
actions to be well-aimed. Presumably, in other words, we want students to direct their civic engagement
well. Civic engagement is valuable only when it is informed throughout by thoughtful and critical
reflection; when civic engagement fails to be thoughtful in this way, it reduced to mere action. This
complexity can be seen most clearly when we imagine contexts where a carefully engaged student would in
fact refrain from taking action. Unlike the other two learning goals emphasized in this project, then, critical
thinking has a qualitative component that the others lack. We don’t merely want our students to act; we
want their actions to be careful and considered, and we want them to know when to act and when not to act.
It is in this respect that civic engagement, appropriately understood, is closely related to critical thinking:
civic engagement requires a certain kind of critical thought on the part of the engaged student.
Given this complexity, different sorts of learning experiences will develop the different features of civic
engagement. One kind of learning experience, certainly, that involves civic engagement is the learning
experience that is practical and requires action on the part of the student. Examples include certain study
abroad experiences and internships. In those two respects, while we explicitly encourage both (particularly
the former), transcript analysis reveals that students have not engaged in either opportunity in recent years
as often as we would like. Our data show that only 22% of recent philosophy majors participated in an
internship or field term, and only 30% participated in a study abroad or domestic, off-campus study
program. Our department, though, is particularly well-suited to cultivating the reflective component of
civic engagement, for the way that many of our courses deal specifically with moral reasoning. Beyond the
fact that ethics has traditionally been considered one of the core areas of philosophy, our department has
particular strengths in ethics. There is an ethical component to many classes that we teach in the
department, and our curriculum is weighted towards courses that are primarily ethical—all of our
introductory classes emphasize ethics, and other examples include Ethical Theory, Environmental Ethics,
Biomedical Ethics, Philosophy of Law, as well as a range of seminars. What these classes all have in
common—and what makes them ideally suited to cultivating the reflective component of civic
engagement—is the way that they challenge students to critically examine the grounds of their values and
their views on a wide range of moral issues.
In our transcript analysis, we tracked the courses taken by our students in both philosophy and other
disciplines that should prepare students for thoughtful, ethical action, and we found that 100% of
philosophy majors who graduated betweeen 2000 and 2008 took at least one such course, and 97% took
two or more (see Appendix B). While this by no means demonstrates that we have been successful at
promoting civic engagement, it does at least suggest that philosophy majors are taking the kind of courses
that will help prepare them to be socially responsible actors in their communities. In our transcript
analysis, we also took note of activities or programs that may serve as indications of philosophy majors’
studies going beyond their regular classroom experiences, such as leadership programs, special projects,
symposium presentations, and even teaching assistantships; however, we have not attempted to draw any
conclusions from their participation in such activities. In connection with our transcript analysis, we also
began collecting anecdotal data about philosophy majors’ participation in student government, community
service, and other extra-curricular activities related to civic engagement. In the future, we will use advising
sessions to gather more information of this kind in order to supplement what we glean from our transcript
analysis.
As a part of this grant project, we have attempted to measure our majors’ developed capacity for well-
aimed civic engagement both indirectly and directly. Indirectly, we have added objective questions to our
senior exit questionnaire that specifically ask whether or not their experiences in ethics courses in the
department developed their capacity for civic engagement. All majors are required to take Ethical Theory,
so every senior will have had at least that one ethics-oriented class; practically, given the number of classes
we offer each semester and the number of classes they need to take to satisfy the major requirements, many
student have had one of more ethics-oriented classes besides Ethical Theory. Again, while the sample size
is small, results here were on the whole much more positive than the results of the objective questions
added to address advising and quantitative reasoning: we were given generally high marks that our ethics
courses did meet this goal. To the statement, “The ethics course(s) I took helped to enhance my capacity for
social responsibility,” our mean score from 18 respondents over two years was 1.67, where marking 1
signals strong agreement (see Appendix A).
While there was a consensus in this respect, a review of the data in Appendix A shows that the agreement
here was not quite as strong as with other statements in the questionnaire. One possible explanation for this
is the deep way in which the philosophical study of ethics is connected with civic engagement, such that it
might not be immediately obvious to very recent graduates. This hypothesis could conceivably be explored
by collecting survey data, delivered via departmental newsletter, from graduates several years after
graduation. We propose such a survey in our action plan (below). Currently, however, in the absence of
this comparative data, a more direct measure of the developed capacity among our majors for well-aimed
civic engagement is needed.
In this spirit, more directly, we have utilized the senior writing portfolios to qualitatively measure our
students’ developed capacity for well-directed civic engagement. These portfolios asked for three papers
from our graduating majors: a paper from their first philosophy class (presumably, though not necessarily,
Introduction to Philosophy), a paper from a 200-level ethics class, and a senior paper from a 300-level
colloquium or seminar. While these portfolios are also being used to measure critical thinking, it is not
surprising that the portfolios are useful here as well, given the close correlation between general critical
thinking skills and the particular critical skill of well-aimed action. Certainly here, the emphasis is on the
200-level ethics paper, where we were looking for evidence of a developed ability (as measured against
their first paper) to carefully articulate a position, to recognize important distinctions, and to anticipate and
respond well to concerns and objections, all fundamentally connected with the notion of acting rightly. As
mentioned above in our discussion of critical thinking, our portfolio analyses are hindered by the fact that
the sample size is so very small, with just four submitted to us in 2007; a further hindrance worth noting
here is that one portfolio of the four submitted in 2007 failed to include a 200-level ethics paper (though
worth noting is the fact that two of the portfolios featured multiple ethics papers from the various levels).
We are still in the process of gathering portfolios from our 2008 graduates, and so have not begun to
analyze them yet in any detail. Our findings from our analysis of the 2007 portfolios map very closely onto
the discussion of our portfolio analysis found in the section of this report focused on critical thinking—
again, this is not surprising, given their similarities. [PROVIDE SPECIFIC TRANSCRIPT EXAMPLE(S),
REFERENCE THOSE INCLUDED IN APPENDIX C] We therefore refer you to the section on critical
thinking in this report for further details.
IV. Summary of Self-assessment
With respect to the three learning goals emphasized in this grant project, then, the philosophy department
sees itself as intensely engaged in advancing its majors with respect to both critical thinking and the moral
reasoning that is essential to well-aimed civic engagement. These two goals are closely related, insofar as
critical thinking lies at the core of moral reasoning, and as such, the single tool of the senior writing
portfolio serves as a direct measure for both of these learning goals. While our initial sample size of
portfolios was quite limited, they have pointed towards the common experience of our senior colloquium as
one area where we might enhance our majors’ development in meeting these two learning goals.
Indirectly, our developing exit questionnaires indicate that we have not, in our advising, adequately
emphasized our attention to learning goals that are largely or entirely not addressed within the philosophy
major, including (though certainly not limited to) quantitative reasoning, the third and last goal emphasized
in this project. These surveys have been encouraging, however, for the ways that our students themselves
highlight the special training we have offered them in developing their critical thinking skills. Certainly,
with both the portfolios and with our survey data, we would benefit from more and better information
going forward.
There is an important tension in our use of writing portfolios that needs clarification at this point. We want
to say that portfolios do, or will when we get a better return, provide meaningful evidence of improved
critical thinking in our students. We discussed several criteria that we may use to evaluate the quality of
the critical thinking reflected in these samples of student writings. But earlier when introducing the
instrument of senior writing portfolios in Section II, Part 3, we refused to develop rubrics in advance to
quantify our evaluation of the writing portfolios. This raises the question: What is the difference between
the criteria we discussed and the rubrics we rejected? Could we not simply use our criteria as rubrics of
assessment? The difference is that the criteria we listed are ad hoc and cannot replace our holistic
judgment as experts. We are merely trying to use the criteria to gesture toward the kinds of things our
discernment picks up, but if there is a discrepancy, we would criticize and modify our criteria to fit our
intuitions/judgment before we would criticize or modify our judgment to fit these criteria. In other words,
the criteria depend on and follow from our holistic assessment instead of the assessment following from a
mechanical summing of pre-established rubrics. That is not to say
that our intuitions/judgments cannot be criticized and that we are not accountable, but only that we are
accountable to the intuition/judgment of other experts in our field (this is the point of "peer-review").
These experts may in turn try to articulate criteria to support or explain their judgment. In fact, we expect
an attempt at articulation and it has an important role to play, but it is still the judgment that carries the
weight. So we do not want to imply that critical thinking eludes adequate assessment, but only that it may
elude assessment by formalized criteria. We believe that there are no mechanical shortcuts which could
bypass expertise and be meaningfully employed by non-experts. Therefore we believe it is inappropriate to
produce rubrics for critical thinking to serve as an algorithm for assessment, preferring instead a qualitative
assessment of critical thinking by experts in the field.
Finally, to return for a moment to the guiding question of this grant project, what exactly is the value added
to liberal education by a philosophy major? At Beloit, we stand committed to the pursuits of “knowledge
through free inquiry” and “personal, social, and intellectual development through multiple paths.” We
share a conviction in “the educational benefits of engaging diverse perspectives” and the vital importance
of “active, responsible citizenship.” We strive to cultivate “a passion for learning within and beyond the
classroom” and to impart the skills of “effective written and oral communication, logical thinking,
…problem-solving, [and] judgment.” While such values and learning goals are represented across the
college in a variety of ways, we believe that a major in philosophy emphasizes them particularly well.
Thinking carefully and responsibly is, of course, one of the primary focuses of the study of philosophy.
Without our having to thematize “critical thinking” as what we teach students to do, there appears to be a
consensus among philosophy majors that our courses train them well in this area. A significant part of our
mission is to provide students with opportunities and topics for sustained self-reflection, and to guide them
in thinking, writing, and speaking about their answers to some of the enduring questions of human
existence. Not only do we train our students in critical reflection on their own views, but we also guide
them in the study of intellectual history, providing them with the vocabulary, concepts, and logical
framework necessary to understand it. In other words, we both encourage our students to engage in free
inquiry and educate them in many of its traditional forms. Yet we recognize that the unity of “our
intellectual tradition” is a problematic one, and we aim to foster students’ awareness of the many points of
view included in and excluded from it. Finally, by maintaining a variety of offerings in ethics, we ensure
that students are exposed not only to interesting theoretical issues, but practical ones as well. Even at the
introductory level, we challenge students to analyze and evaluate different solutions to moral problems, that
is, to exercise their sense of moral responsibility and their capacity for moral reasoning. So, we aim to
ensure that they adopt a critical stance not just to their own beliefs, and not just to the theories of “great
philosophers,” but to their own values and the values embodied by their society and its history.
To summarize, it is primarily by means of these core competencies—critical thinking, knowledge of
intellectual history, and moral reasoning—that the philosophy major promotes the objectives that are
articulated in Beloit’s mission statement and learning goals, and thus adds value to general education at
Beloit.
V. Action
As a result of our self-assessment, we have begun to develop a plan of action for improving student
learning and strengthening the relationship between the philosophy major and Beloit’s learning goals. Our
plan is still in its early stages and subject to revision. At this point, our strategy involves meeting each
semester after advising week, developing a newsletter to foster contacts with alumni, restructuring the
capstone course in philosophy, and employing a variety of activities to further strengthen the philosophical
community at Beloit College.
1. Post-advising debriefings
Our exit questionnaires and transcript breakdowns, while not providing the most detailed information about
student needs, have given us at least an indication of several ways that advising in the major could be
improved. For example, it appears that philosophy majors participate in study abroad less than the average
Beloit student. While the college has set the goal of a 50% rate of participation in study abroad, only about
30% of recent philosophy majors have participated (see Appendix B). Also, many of our recent graduates
did not agree that their major advisors encouraged them to pursue courses devoted to quantitative reasoning
or helped them to make good choices in general about coursework outside of philosophy.
As advisors, the three of us already engage in a lot of informal, “in-the-hallway” discussions about student
needs. One further course of action we plan to pursue is to schedule at the end of each semester’s advising
period a meeting devoted to dialogue about our advising sessions with students. This will help us to
establish and maintain an ongoing conversation about our questions and concerns, surprises and
disappointments, and successes and failures in student advising. During these post-advising debriefings,
one thing we will focus on is how the learning goals of the college as a whole factored into our most recent
meetings with our advisees. We hope that setting aside a time each semester to take stock and exchange
ideas will help us to ensure that we are encouraging students to stretch themselves intellectually and to
make wise choices about their schoolwork outside of the philosophy major.
2. Departmental newsletter
Surveying recent graduates of Beloit about their perceptions of how the philosophy major contributed to
their education as a whole provided us with some helpful information, but students’ perspectives on their
college education will obviously change and hopefully sharpen over time. Especially since there are
typically ten or fewer philosophy majors graduating each year, we concluded that we need to tap into our
alumni in order to obtain more information that will be useful for the ongoing process of self-assessment.
Sending questionnaires to students, say, five years after their graduation from Beloit will serve the dual
purpose of providing another, possibly more mature perspective on the job we are doing and increasing the
size of our sample, thus the amount of available information.
However, the philosophy program has never had any organized way of remaining in contact with our
majors after graduation. To address this lack, we plan to put together a brief annual newsletter that will
allow us to stay in touch and keep our alumni informed of things happening with us and our students. First,
we will turn to the Office of External Affairs to help us organize contact information. Then, the three of us
will design the newsletter, drawing on the experience of other departments who already produce an annual
publication. Finally, we will use our students and alumni with whom we are already in touch to contribute
ideas and material. With any luck, the newsletter will open a channel of information flowing not just from
us to alumni, but in the opposite direction as well, providing us with more feedback on the strengths and
weaknesses of the major.
3. Colloquium renovation
Our first batch of writing portfolios and our questionnaires indicate that while some students clearly
flourish and grow as thinkers over the course of pursuing a degree in philosophy, others appear to need
more guidance. In the writing portfolios submitted by graduates in the class of 2007, the research papers
that they produced for the capstone course, Colloquium in Philosophy, were a mixed bag. A number of
these papers, as well as others we have received in recent semesters, do not demonstrate the level of
intelligibility, critical distance, sophistication, and judiciousness that we expect from a senior philosophy
major. We have identified several ways to assist students in making a final leap to careful research on
philosophical issues and the quality of critical thinking that attends it.
Our renovation of the colloquium, begun in the fall of 2007, changes the capstone experience from a ¼-
credit course that majors are required to take twice, to a ½-credit course that they take once, in the fall of
their senior year. The amount of work that we expect them to undertake in this course alone justifies this
increase. Another change is that we now encourage students to meet with the advisor of their research
project earlier in the semester and more often than students have been in the habit of doing in the past.
Moreover, the three of us now begin the semester by devoting several class periods to discussing key
aspects of the research process with and modeling it for our students. For example, we will bring them to
the library for a preliminary research session using electronic resources like the Philosopher’s Index, we
will devote one session to turning a philosophical concern into a researchable question or thesis, and we
will demonstrate how to recognize and incorporate good sources into a research paper. For future
semesters, we plan to develop sessions on documenting sources, using footnotes, constructing a
bibliography, and other important issues, which we can include interchangeably based on student need as
we perceive it. As the renovation proceeds, we will solicit student input in order to maintain a “feedback
loop” for assessing the value of the changes made and the need for attention to other areas in the future.
We will also consider whether and how to integrate some of the material we prepare for colloquium into
lower-level courses when it would be appropriate. Academic year 2007-8 was the first to employ these
changes, and we were pleased to find that the papers as a group were generally stronger than those
submitted in recent years.
4. Strengthening our community
One comment frequently made in earlier incarnations of our graduate questionnaire concerned a desire for a
stronger, more active philosophical community at Beloit College. There has been a long-standing ice
cream social organized each semester, but beyond that event and classes, opportunities for students to
engage with both faculty and each other were minimal. The benefits of a stronger community are many,
but at least one benefit connects closely with the spirit of this project. We have here largely focused on
direct and indirect measures of assessing our program, but more informal means of feedback can
sometimes be more salient and effect more change. Finding more opportunities in new settings to interact
with our students can, among other things, help us to get a better anecdotal sense of ways to improve their
learning experiences in our department. It also remains the case that much of our advising happens
informally with students, rather than in planned advising appointments, and so increasing the opportunity
for informal engagement with students should increase the opportunities for this informal advising.
To this end, we have recently put in place several activities all aimed at creating a friendlier and more
intellectually stimulating philosophical community at Beloit. Beyond our aforementioned ice cream
socials, we have begun organizing an annual cookout for majors aimed at celebrating our graduating majors
and minors. We also have begun, in the fall of 2007, a “Philosophy and Film” series, where one several
evenings throughout the semester we view a film together with our students and then discuss its
philosophical implications afterwards. And, in conjunction with the change in our colloquium described
above, we began in the spring of 2008 a philosophy reading group for interested faculty and students.
While there have been some stumbles with the reading group as we grapple with the best ways to organize
it, we plan on continuing it in the spring of 2009 and beyond.
While none of these activities will yield the sort of data that assessment typically calls for, the knowledge
we hope to gain about ourselves and our program through these opening up new lines of communication
with our students is consistent with the goals at the heart of the project at hand.
Appendix A: graduate questionnaire and resulting data
A.1: Questionnaire (current version, spring 2008)
PHILOSOPHY DEPARTMENT GRADUATE QUESTIONNAIRE
Congratulations on graduating! You made the wise choice of a philosophy major or minor, and now we
want to benefit from your wisdom. Please take the time to answer the following questions, so that we can
best evaluate our curriculum in light of your experiences with it. Please provide as much detail as possible,
and also be as frank as possible. It will help us to know both the things you found most effective in your
experience in the department, as well as the things you found least effective. When you’ve completed this
questionnaire, please return it to Matt Tedesco at [email protected] if you fill it out electronically, or
else put in his office door at MI 210 if you print it and fill it out by hand. Thank you!
Name:
Year of graduation:
1) What classes did you take, and with whom, in the philosophy department?
2) What were the most effective, rewarding, or otherwise beneficial aspects of your experiences in the
department?
3) What were the least effective, rewarding, or otherwise beneficial aspects of your experiences in the
department?
4) What do you think you’ve gained from your experiences in the department?
5) If you were given free rein to redesign the department (including classes offered, requirements for the
major or minor, etc.), what changes would you make, and why?
6) If you were given free rein to redesign the department (including classes offered, requirements for the
major or minor, etc.), what would you be sure to retain, and why?
7) All things considered, given present facts about the department, would you recommend that someone
interested in majoring or minoring in philosophy at Beloit College do so? Why or why not?
Please circle the number that most accurately reflects your answer to these questions, or, if you’re filling it
out electronically, place an “x” next to the appropriate number:
Strongly Strongly
agree disagree
8) The courses I took in philosophy
promoted my critical thinking abilities 1 2 3 4 5
9) Studying philosophy has increased my ability
to analyze and evaluate arguments 1 2 3 4 5
10) Studying philosophy has contributed to my ability
to find and challenge assumptions 1 2 3 4 5
11) In my philosophy courses, I was taught
to think through opposing viewpoints 1 2 3 4 5
12) Studying philosophy has made me better
at building arguments to support my claims 1 2 3 4 5
13) The ethics course(s) I took helped to enhance
my capacity for social responsibility 1 2 3 4 5
14) My philosophy advisor encouraged me to take
courses outside the philosophy department
that focus on quantitative reasoning 1 2 3 4 5
15) Advising sessions with philosophy
professors helped me to make good choices
about courses to take in other fields 1 2 3 4 5
16) The philosophy curriculum successfully
advances the general education goals of
Beloit College (as I understand them) 1 2 3 4 5
17) My overall educational experience at Beloit
benefited from my study of philosophy 1 2 3 4 5
18) Any additional comments? Provide them here:
A.2: Questionnaire data: results
Before considering the data gathered so far from our questionnaires, a few clarifications need to be made.
First, because we have modified the survey further each year that we have used it, some of the objective
questions have two years of data, while others come from just a single year of feedback. Second, where
relevant, we have summarized some of the findings from other comments provided by our students in their
questionnaires. Third, the question numbers below correspond to the numbers as they appear on the above
version of the survey. And fourth, because of the scoring system used on the questionnaire, the lower the
score, the greater the disagreement, where perfect strong agreement would be a 1.0 and perfect strong
disagreement would be a 5.0.
Question #8: 1.05 mean (19 respondents)
2007: 8 (1), 1 (2)
2008: 10 (1)
Question #9: 1.00 mean (10 respondents)
2008: 10 (1)
Question #10: 1.11 mean (10 respondents)
2008: 9 (1), 1 (2)
Question #11: 1.38 mean (9 respondents)
2008: 8 (1), 1 (3), 1 (unanswered)
Question #12: 1.00 mean (10 respondents)
2008: 10 (1)
Question #13: 1.67 mean (18 respondents)
2007: 6 (1), 1 (2), 1 (3), 1 (unanswered)
2008: 5 (1), 1 (2), 4 (3)
Question #14: 2.81 mean (16 respondents)
2007: 2 (1), 4 (2), 1 (3), 1 (4), 1 (5)
2008: 2 (2), 2 (3), 2 (4), 1 (5), 3 (uanswered)
Question #15: 2.17 mean (18 respondents)
2007: 2 (1), 5 (2), 1 (4), 1 (5)
2008: 4 (1), 3 (2), 1 (3), 1 (5), 1 (uanswered)
Question #16: 1.21 mean (19 respondents)
2007: 7 (1), 1 (2), 1 (3)
2008: 9 (1), 1 (2)
Question #17: 1.11 mean (19 respondents)
2007: 8 (1), 1 (2)
2008: 9 (1), 1 (2)
A.3: Questionnaire data: preliminary findings
The findings, though preliminary given the relatively small sample size (particularly for questions only
answered by 2008 graduates), are revealing in a number of ways—regarding us as faculty, regarding the
students, and regarding the survey itself.
Regarding the survey specifically, it is an apparent weakness of it that no “not-applicable” option is
explicitly available for the “circle-the-number” questions. In some cases, students took it upon themselves
to avail themselves of such an option regardless; in other cases, however, the results may be skewed by
students signaling disagreement where applicability is dubious. For instance, with graduating minors in
particular, it is not clear that some or all of the advising questions are appropriate, yet their responses are
contained in the data here alongside the responses of majors. This is further complicated by the fact that
some minors had once planned or declared majors (and been advised accordingly), while others were
minors all along; the responses of the former, with respect to the advising questions, may be more relevant
than those of the latter.
Other students clearly struggled with how to respond to the questions about advising. It is not clear
whether this is simply a weakness of the survey tool, or rather raises much deeper questions about the
nature and purpose of advising. Some of the student comments in this respect are revealing. One 2007
graduate observed: “The previous questions regarding advising don’t quite match up with how I
approached advising. I always had my classes all figured out before I talked to Matt, so those questions are
fairly irrelevant. However, Matt provided EXCELLENT advising in areas not related to classes. He
advised on such topics as the future, how to get into grad schools, how to pick grad schools, making sure I
was finishing my requirements, how to approach teaching philosophy, etc. So while I did not find much
relevance for advising in class choices, Matt was a great advisor. That doesn’t seem to be reflected in the
questions, so I wanted to make it explicit.”
Several of the themes from this long comment are reflected in other responses from different students, and
with different departmental advisors. Another 2007 graduate commented, with respect to advising
connected with long-range life-planning, “I would have liked some kind of preparation for the future or
possible things one can do with a philosophy major—in general, more advising/life planning.” A 2008
graduate emphasized the fundamental importance of self-direction in planning a college career, observing,
“I always felt that when I had a question or needed support, it was available.” Additionally, several of the
2008 graduates felt that the questions about advising really weren’t relevant. One, who had a second major
and came to the philosophy major afterwards, noted, “I needed more advice about in-department course-
taking than out.” Another, who chose to fill in a “not-applicable” answer to question #14 regarding
quantitative reasoning, quickly explained, “I was doing that anyway.”
There is undoubtedly much more to consider with respect to the aims of advising, and what it means to be
an advisor at a liberal-arts college like Beloit. Yet, in the end, we suspected all along that our attention to
advising with respect to courses outside the department had generally been lacking, and these preliminary
results support that suspicion.
Appendix B: transcript data
Philosophy majors, 2000-2008
Total number of students Percentage of majors
Took no courses in an area related to quantitative literacy 14 22.22%
Took one or more courses in an area related to quantitative literacy 49 77.78%
Took two or more courses in an area related to quantitative literacy 28 44.44%
Took three or more courses in an area related to quantitative literacy 15 23.81%
Transferred one or more credits in an area related to quantitative literacy 19 30.16%
Took no courses in an area related to civic engagement 0 0.00%
Took one or more courses in an area related to civic engagement 63 100.00%
Took two or more courses in an area related to civic engagement 61 96.83%
Took three or more courses in an area related to civic engagement 53 84.13%
Participated in a field term or internship 14 22.22%
Participated in a study abroad or domestic off-campus study program 19 30.16%
Took no courses in a foreign language 33 52.38%
Took one or more courses in a foreign language 30 47.62%
Took two or more courses in a foreign language 23 36.51%
Took three or more courses in a foreign language 15 23.81%
Took no courses in an area related to creative expression 23 36.51%
Took one or more courses in an area related to creative expression 40 63.49%
Took two or more courses in an area related to creative 27 42.86%
expression
Took three or more courses in an area related to creative expression 18 28.57%
[INSERT EXCEL APPENDIX, PAGE 1]
[INSERT EXCEL APPENDIX B, PAGE 2]
[INSERT EXCEL APPENDIX B, PAGE 3]
[INSERT EXCEL APPENDIX B, PAGE 4]
[INSERT EXCEL APPENDIX B, PAGE 5]
[INSERT EXCEL APPENDIX B, PAGE 6]
[INSERT EXCEL APPENDIX B, PAGE 7]
Appendix C: sample portfolios
[FORTHCOMING]