assessment activity fund report: student attitudes on the ... · the fundamentals of engineering...

Assessment Activity Fund Report: Student Attitudes on the Fundamentals of Engineering Exam and the Impact on Program Assessment

Matthew Roberts*, Ph.D., Keith Thompson*, Ph.D., and Teresa Burns+, Ph.D * Department of Civil and Environmental Engineering + Humanities Department

1. Introduction The Fundamentals of Engineering (FE) Exam is an eight-hour exam typically taken by civil engineering students during their senior year. The exam covers a wide range of topics within engineering and passing the exam is required to become a licensed engineer. In order to pass the exam, students must correctly answer a minimum number of questions on the total exam. There is no requirement to show mastery of all exam topics or any specific topic.

Because the FE Exam covers so many aspects of engineering, it has been used to assess multiple student learning outcomes for the civil engineering program. Specifically, the following outcomes are assessed using the FE Exam:

• Applying math and science to solve engineering problems (FE results from chemistry; fluid mechanics; mathematics; strength of materials; statics)

• Solving engineering problems in five civil engineering emphasis areas (FE results from environmental engineering; construction management; soil mechanics; transportation; structural analysis)

• Understanding professional and ethical responsibility (FE results from ethics and business practices; legal and professional aspects)

Recent evaluation of the FE results as part of the CEE assessment process has shown student performance in structural analysis to be consistently below the performance criterion (see Figure 1). This report seeks to shed light on student attitudes about the FE exam and to investigate the possibility that test-taking strategies on the FE exam and student attitudes about structural analysis may be responsible for the consistent under-performance. Specifically, we wish to explore the hypothesis that because mastery of all topics is not required to pass the exam and students often perceive structural analysis to be a challenging subject, students are not giving their full effort on the structural analysis portion of the exam.

2. Methodology Two focus group discussions were held immediately after the completion of the FE exam—the first conducted after the Fall 2012 FE Exam and the second after the Spring 2013 FE Exam. The discussions were led by Dr. Teresa Burns. The protocol for the focus groups and the transcribed results are included in the Appendix. The protocol for the Spring 2013 focus group was slightly

1

modified from the Fall 2012 protocol. The primary change was to rearrange the order of some of the questions leaving the substance of both protocols essentially identical.

Figure 1 - Assessment results for Outcome 2 of the civil program in the CEE Department

3. Results The results from the focus groups seem to confirm our theory that not all students give their full effort on the structural analysis portion of the exam. In particular, both the Fall and Spring focus groups participants indicated a perception that to do well on the structural analysis portion of the FE exam requires completion of structural engineering technical electives such as Structural Steel Design or Wood Structures (see Table 1).

Table 1 - Focus group comments related to the need for structural engineering technical electives

“The structures area is the one where you need to know more from the upper-level courses (sounds of agreement). Geotech you could do with the basic class and most of the others, too.” (Fall 2012) “Structures is the one area where the classes really helped a lot.” (Fall 2012) “As far as the trans section and the construction section, it seemed pretty basic. We learned that stuff and it was something you’d learn in the basic class, but the structural stuff was something you’d only get in steel or some other class.” (Spring 2013) [Structural Steel Design is a technical elective taken by students emphasizing in structural engineering.] “There were questions [in the structural analysis section] that were right over our heads.” (Spring 2013)

In both focus groups there was general agreement that examinees should not spend too much time on subject matter that has not been mastered. Specifically talking about the structural analysis portion of the exam, one of the focus group participants commented

85%

90%

95%

100%

105%

110%

115%

120%

125%

130%

F07 S08 F08 S09 F09 S10

Outcome 2 - Technically Skilled in Civil Engineering

Environmental Engineering

Construction Management

Soil Mechanics

Transportation

Structural Analysis

Performance Criterion

2

It’s also not a high percentage on the exam, so if you’re not good at it, it doesn’t make sense to spend a lot of extra time. (Fall 2013)

In order to get more quantitative data about our thesis, we created a survey that was administered to the seven students in Spring 2013 before they began the focus group. The survey instrument and numeric results are available in the Appendix. The two quantitative questions were:

1. For each of the following topics on the F.E. exam, circle the response that best expresses how you feel about your performance. (Options: Very Confident, Confident, Unsure, Very Unsure)

2. For each of the following topics on the F.E. exam, circle the response that best expresses how often you guessed at the answers. (Options: Always, Most, Some, None)

The topics indicated on the survey were: Chemistry, Fluid Mechanics, Mathematics, Strength of Materials, Statics, Environmental engineering, Construction management, Soil mechanics, Transportation, and Structural analysis. Not surprisingly, there was a correlation between the confidence on a topic and the frequency of guessing, as can be seen in Figure 2.

Figure 2 - Relationship between confidence about a topic and frequency of guesses. Higher numeric values indicate higher confidence/lower frequency of guesses.

What was surprising, though, is that the only topic with more guessing than structural analysis was chemistry, and confidence on the structural analysis portion of the exam was equal to or higher than all other topics except for fluid mechanics and mathematics (see Figure 2). This could be due to the small sample size, or it could also reflect student perception on what it means to “guess” at an answer.

4. Conclusions Results of the focus groups indicated that students perceived that senior technical electives were required to perform well on the structural analysis portion of the FE exam. Survey results

2.50

2.70

2.90

3.10

3.30

3.50

3.70

3.90

1.50 2.00 2.50 3.00 3.50 4.00

Freq

uenc

y of

Gue

sses

Confidence about Topic

Structural Analysis Chemistry

Mathematics

Fluid Mechanics

Transportation

3

seemed to contradict this somewhat since the students indicated that structural analysis had one of the higher weighted averages for confidence as compared to the other topics on the FE exam. We feel, however, that the results of the focus group clearly indicate that many students in the civil engineering program do not put in their best effort on the structural analysis portion of the FE exam and this should not be the sole measure of aptitude for structural engineering in the program assessment plan.

4

APPENDIX

Student Attitudes on the Fundamentals of Engineering Exam and the Impact on Program Assessment

Protocol for Focus Group Discussions

Dr. Matthew Roberts and Dr. Keith Thompson, both Civil Engineering professors with expertise in structural analysis, will determine how to best randomly select students to participate in two focus group discussions led by Dr. Teresa Burns from Humanities and attended by a practicing professional engineer. The focus groups will explore the questions listed below. These ques-tions seek to shed light on student attitudes about the FE exam and to investigate the possibility that test-taking strategies on the FE exam and student attitudes about structural analysis may be responsible for the consistent under-performance in this area. Specifically, Drs. Roberts and Thompson wish to explore the hypothesis that because mastery of all topics is not required to pass the exam and because students often perceive structural analysis to be a challenging subject, students are not giving their full effort on the structural analysis portion of the exam.

Roberts and Thompson will randomly select five to six senior civil engineering students who have just taken the FE exam; if possible, the focus group will take place immediately after the exam. One focus group will be conducted after the Fall 2012 FE Exam; another focus group will be conducted after the Spring 2013 FE Exam. Roberts and Thompson will not attend the focus group, but they will also endeavor to locate a licensed professional engineer who has passed the P.E. exam, optimally a UW-Platteville graduate, to sit in on the sessions. This should allow stu-dents to speak more freely about how they prepared for the exam generally and the structural analysis portion specifically, since they will not have the two structures professors listing to their responses; however, they will have an engineer in the room in the event that their answers are either too technical for Burns to quickly record or so specific that they risk revealing an exam question. Burns is aware of the agreement that all licensure candidates must sign before they are permitted to take the exam. In particular, she and the licensed P.E. want to ensure that none of the students in the focus group unintentionally violates exam security. If a student starts to give an answer that seems to reveal a type of test question, she will redirect the question.

General framework

Students meet at a time and place determined by Roberts and Thompson. Some sort of food and drink will be provided (probably pizza and soft drinks.)

Burns will thank them for coming and introduce the P.E. who will also be attending. She will remind them of the agreement they signed before they took the exam, and ask them to not dis-close any exam content in their answers. She will also thank the P.E. who is observing, and let the students know that he/she is there, like they are, to help improve the program.

Appendix - Focus Group Protocol - Fall 2012

A-1

Questions

Each of these questions will be asked of all students present. If there seems some reason for a follow-up not listed below, Burns will ask a follow-up. If discussion continues and it concerns strategic planning for the exam, Burns will simply let it continue; if discussion ensues that seems to drift off-topic, she will redirect the discussion. All present will know that they can ask the practicing engineer any questions that seem relevant. They will be informed that the focus ses-sion will take no more than an hour (Burns should actually plan for 45 minutes).

1. What was your strategic approach in preparing for this exam? Please take a moment to think about this before answering, and don’t worry if what you say doesn’t “match” what you hear someone else say. (Pause.) (Start with whoever volunteers or pick someone.)

Follow-ups: Did you go to the review sessions? Did you pick certain sections not to worry about (and if so, which ones)? How was your preparation for the morning session different than that for the afternoon session?

2. How well do you think that strategy worked? (Again, start with whoever volun-teers or if no one does, pick someone. If the same person volunteers repeatedly, shift to asking those to start who have not spoken first.)

Follow-ups: Why? Was there a section you felt especially good about, and if so, why do you think you were so well prepared? [Note: this may simply ask students to state the obvious, but hopefully it will show to what degree this group is aware of which sections students here typical-ly do very well on.]

Is there a section you felt you did not do well on? Was that strategic (a section you decided not to prepare for) or a surprise?

If you had to do it over again, are there any courses you wish you had taken that might have al-lowed you to do better on the exam? If so, which course? Were you aware before the exam that you were weak in the area covered by this course, and if so, why did you decide not to take that course? [Note: again, this may simply ask students to state the obvious. On the other hand, if no student mentions a structural analysis course such as statics or dynamics, that will suggest asking about statics and dynamics as an alternate follow-up question.]


A-2

3. How would you advise a friend to strategically study for this test?

4. Tell me a little about where you are in the program here.

How far along are you? What internships have you had? What sort of other engineering activi-ties do you participate in? Is there some non-course related activity (such as an internship) that you felt also helped prepare you for this test?

5. From what you have learned in your courses or via your internships or via other networking, what are the specific technical areas in which a Civil Engineer should be skilled?

Follow-up: If no one mentions CEE learning outcomes in these areas, name those outcomes and ask again.

6. From what you have learned in your courses or via your internships or via other networking, what are specific non-technical aspects of engineering in which a Civ-il Engineer should be skilled?


7. Which of those areas mentioned in the last two questions does this exam do a good job of assessing? Which of those things does it not assess? Are there other measures of excellence you believe should be considered?

8. Did you specifically prepare for the structural analysis section part of the exam?

Follow-up: If so, how? Over the last two years, students from UW-Platteville have consistently underperformed on the structural analysis section of the FE exam: does this surprise you? Why or why not?

9. Is there anything else you’d like to share about this exam and how you prepared for it?


A-3

Focus Group Transcript Summary Fall 2012

Draft summary of focus group answers

Rachel Zimmer introduced herself. We explained the protocol.

Students agreed that their signing in was also giving permission to record.

Questions

1. What was your strategic approach in preparing for this exam? Please take a mo-ment to think about this before answering, and don’t worry if what you say doesn’t “match” what you hear someone else say. All students answered this.

I did three things. First, I got some review material from Kaplan, and the equations manual. Second, I went to the FE review sessions the university offered. Third, I went over my own notes and exam.

Next student: same as above – looked over problems more than re-solving them but went over materials section for the afternoon civil sections—the concrete mix and asphalt stuff wasn’t cov-ered in the Kaplan book but it was in the FE provided handbooks

Next student—same answer—sounds of agreement about doing practice exams

Q: How many other people did practice exams?

Three people did them and timed themselves; everyone looked at them.

Next student: Same, but structural section in Kaplan books had lots of typos especially in the structural section. I was better off going to my notes than looking at that book. (Sounds of agreement)

How many of you felt this way? (Four students and others nodding) Of these four, they men-tioned: the Kaplan book structural section didn’t correlate to the equations book, the transporta-tion section didn’t correlate and was given in all metric in Kaplan and in the other book was in US units; they had factors specifically meant for English units and didn’t say so. The geotech portion of Kaplan was also frustrating. The afternoon edition was frustrating to all who used it.

Rough count? Who agrees? (Now everyone in the room agrees with this.)

Next student: Focused more on afternoon session in studying, same three approaches to studying as student #1

Next: same

A-4


Q: Many of you have said you did more looking through the studying guide rather than working through it completely. Comments?

General agreement- complaints that the Kaplan review books were terrible and it was better to work through notes and look in FE review book—one student says the notation in the review book was not familiar to her so she had to relearn it the way they presented it -“especially in dy-namics- everything was in vector format.”

Next student: Borrowed reference material and heard by word-of-mouth you could rent out a book from the Civil office, so he got that and read through it, especially the ethics session, and read through problems. Went to review sessions.

Q: Was there anyone who didn’t go to the review sessions?

No. All went.

Next student: Inherited a review book—tried to figure out what he didn’t know and what he had time to learn, to see what he could know in the time he had.

Next student—again- review sessions, Kaplan books, notes, handbook. The variables in the Kaplan book looked different, so correlated it to notes and handbook.

Follow-ups:

Did you pick certain sections not to worry about (and if so, which ones)?

Answer in unison: circuits.

Q: Is there anyone who did worry about circuits?

No. Explanations: “We had the option of this and thermo, and circuits doesn’t click with Civils.” Sounds of agreement.

Q: Any others?

One student: I haven’t taken enviro yet, so I didn’t worry about it.

Another: I’ve taken it, but the equations were set up differently so I didn’t worry about it.

Another agreed.

A fourth: Enviro was the most different from all the subjects I’ve taken, like I hadn’t seen it be-fore?

Five agree. No one disagrees.

A-5


Other sections?

Dif-y-Q (almost everyone)

We don’t use it very often, if ever. (Agreement, from five or more.)

I read the sections:

Chemistry—anyone not prepare for this?

Yes—though not in depth

Fluid Mechanics—all prepared for it

Math—everyone

Strength and Materials—one student said there were strange questions (was advised to go no fur-ther here) and he prepared but hadn’t expected “such strange questions”

(Here and several places, on the CD you can hear students warning, jokingly, about “walking the line” of getting too close to talking about particular questions.)

Another student: there is a difference between the mechanics of materials and material science

Another: the material science he hadn’t prepared for – they had the strength of materials but not the properties. Another agrees.

Statics—all prepared

Construction—all prepared

Soil mechanics—prepared. Several “rocked” that section. Kaplan was not much help.

Transportation- all prepared. Several felt the material on it was mainly “traffic.” Warnings about “walking the line.”

Structural analysis—not a lot. And yes. So… how about if we divide this by areas?

Structures students felt the material was really familiar from their classes and the classes were the best review.

2. How well do you think that your strategy worked? (Again, start with whoever volunteers or if no one does, pick someone. If the same person volunteers re-peatedly, shift to asking those to start who have not spoken first.)

A-6


--Pretty well

--The first in the afternoon didn’t go well, but then the rest clicked.

--Next student agreed.

No one felt that their overall strategy was poor. Some studied together. Several in the room studied with each other.

How many were in a study group?

The FIVE who were in the study group, and they hadn’t heard of anyone else doing this. They met at the same time every week on Wednesday.

But most felt that everyone studied with another person. Two others were roommates.

If you had to do it over again, are there any courses you wish you had taken that might have al-lowed you to do better on the exam?

Circuits. (Laughter.)

If so, which course?

Circuits.

Enviro. (3)

Econ. (1)

Maybe thermo (1)

Were you aware before the exam that you were weak in the area covered by this course, and if so, why did you decide not to take that course?

--“In the grand scheme of my engineering career, circuits would not benefit me.”

--If you can pass the exam without taking the class, why would you?

--You can also get some of it from Physics 2. (3 agreed)

How would you advise a friend to strategically study for this test?

--Start early—do a section a night.

--Assess yourself to see where you need help first. Don’t waste time on something you’re al-ready good at. The AM section had two practice exams.

A-7


--On the other hand—I like reviewing everything. (2 others agree)

--Concentrate on your areas, but at least look at everything. By glancing over some things, I was able to solve problems I couldn’t have. (agreement)

--Go to review sessions. They help you realize what you’ve forgotten and need to work on.

--Compare your notes to someone else’s.

--Try to find different ways of asking the same question.

3. Tell me a little about where you are in the program here.

Senior—structure geotech construction

Senior trans and municipal

Trans “super senior”

Two seniors structures and geotech

Senior structure, trans and geotech

“Super senior,” trans

“Super senior” municipal and trans

Senior, trans and construction

What sorts of activities have prepared you for the exams?

--two internships the one especially with MnDOT helped him picture questions

--three internships surveying experience from internship helped on exam. The work ethic also helped her

--two internships the NorthShore gas helped with fluid mechanics problems because that was what they did; the other internship helped with surveying

--three internships with WiDOT and “it helped me with like one question” she did gain good experience but it wasn’t academically applicable for this test

A-8


--no internships, no comment

--several years construction experience with utility and pipeline which helped with survey and fluids, and an internship with the county that helped with hydrology, highway, and survey.

--nothing

--three internships with same company and construction experience in high school. Helped visu-alize things as others have said. If I got an answer that made no sense I’d recognize it. Also, having sat at a desk all day during the summer helped me concentrate for long periods of time. (Two others agree. Eight hours didn’t feel that long. “I was over-prepared for sitting this long.”)

Other activities?

--The labs we had helped—for surveying, fluids, geotech, several others.

--I could visualize things. What do they want? Oh, this?

Many other comments about labs. “Dr. Penn would be proud.” All agree that labs have been helpful

4. From what you have learned in your courses or via your internships or via other networking, what are the specific technical areas in which a Civil Engineer should be skilled?

Follow-up: If no one mentions CEE learning outcomes in these areas, name those outcomes and ask again

No one did—look at this section! Are these answers different from what one might expect?

--First student: discretion and humility. Know what you don’t know, and ask for help.

--Several others: “I like that answer” and sounds of agreement.

---Everything is full of uncertainty. You need to go to the professionals if you don’t understand.

How about if you think about particular learning outcomes?

A-9


--That we don’t know now?

--Are you looking for technical skills?

(Burns reread the question)

--In our courses, we’ve learned that we need to know what other people doing and how it all re-lates.

--Know who to ask.

--Know when something is out of your reach and you can’t answer.

--Your courses should give you good judgment, in all areas, to know if your answers just make no sense. If you get an answer that just looks crazy, you need to know that. Even if you aren’t in geotech, if a boring sample is crazy, you should know that.

Q: OK… so what are the technical areas that would allow you—rather than, say, someone like me—to know that a boring sample is crazy?

--the geotech course, or trans course, or …

--we have to take an intro in every emphasis

--in each class, we learn what range of values to expect, not just to take the value off of our cal-culators or we’ll get yelled at. I think we’ve all become very skilled at looking at an answer and seeing if it is in a range that makes sense.

--I used that on the FE all the time.

--Agreement, laughter.

Next, I read over the topics that are listed as outcomes, starting with:

“Successfully apply technical knowledge to solve engineering problems”

“Technical skills in math and science”

Q: are you assuming this so you don’t have to state it? I don’t want to put words in your mouths, but is this what you’re saying?

--unanimous agreement. Yes, they assume technical skill in math and science is too obvious to mention.

A-10


You can solve problems in four or more emphasis areas because you have had courses in those areas?

--Yes. unanimous agreement

And this allows you to be innovative? Or not?

--I think it enhances it

--you can say “what if we do it this way?” to someone who knows that area

What about the professional course of action? What about professionalism or ethnics?

--having internships helps with this.

--yes, this helps on the ethics portion

--a lot of that section is common sense. (agreement)

How about communication skills?

--Communication is a technical skill. We’re graded on that. Writing is technical.

--We get run through the wringer on structure and form and verbiage. (agreement)

--There are teachers who will stop grading if you have too many “fatal flaws.”

--If you can’t speak or write you can’t get through the Civil program.

--A lot of us will be consultants and we have to know how to communicate.

--In surveying, it really hit me that I had to write it so others could read it. It didn’t matter if I knew it if others can’t follow it. (Agreement.)

5. From what you have learned in your courses or via your internships or via other networking, what are specific non-technical aspects of engineering in which a Civ-il Engineer should be skilled?

Note: Ethnics and communication were covered above in #4 because they felt they were tech-nical skills or could be.

--People skills are different from just communicating what people need to know.

A-11


--Organization

--Confidence

--Knowing how to get away and think about your answer before you have to give it.

--Being able to work in groups—teamwork as opposed to communicating to those outside the group.

What about contemporary issues or global concepts? Do you think about this and if so, in what way?

--you can learn from the mistakes of others, like if a bridge failed

--Society for Civil Engineering has helped with this more than classes—conferences and guest speakers and weekly e-mails. (several agree—then everyone)

6. Which of those areas mentioned in the last two questions does this exam do a good job of assessing? Which of those things does it not assess? Are there other measures of excellence you believe should be considered?

--Didn’t talk about global aspects that much? Is it in the curriculum?

--There are a couple of classes that get into re-certification.

--It does a good job of getting into ethnics but not into communication.

(General agreement—added comment “it’s hard to assess this on a multiple choice test.”)

Other things that should be assessed:

--Resourcefulness?

Agreement and disagreement.

--it does assess resourcefulness? Or not?

Yes, it assesses this because to do well you need to know where to go to get the answers if you don’t know them immediately, and that’s true in terms of how you study for it as well.

(Agreement)

A-12


--To do well you have to learn a structure by which you can do well, but if you are resourceful you can figure it out.

7. Did you specifically prepare for the structural analysis section part of the exam? If so, how?

“Roberts” (General agreement and laughter.)

--If you had all of the classes, you didn’t really have to study for that section. “I second that”

--From “a trans student who had one structures course”—agreement

Is there anyone here who hasn’t had a structures course?

--Four people are there who aren’t in structures. Only one is only part-way through “RC.”

Over the last two years, students from UW-Platteville have consistently underperformed on the structural analysis section of the FE exam: does this surprise you?

Yes. (Five people are surprised. Not all are in structures.)

Why or why not?

--it isn’t a surprise to me because it is not necessarily second nature and difficult in general, a lot of ways to solve things that you forget or not know.

--non-structures people who it does surprise, could you raise your hand? All but one.

--For me personally, it was one of my hardest classes. I’m not that interested, I didn’t look over it as much, and I didn’t think they’d go that in depth.

Do those of you in structures want to respond to this?

--Obviously a structures emphasis person would have another area be weaker. For myself, I didn’t care that much about enviro—it was like my “structures.”

--Structures is one area out of six so if you look at that ratio, five out of six won’t know what to do with it. That’s no surprise. Do structures students do all right?

--I got the impression that the structures area is the one where you need to know more from the upper-level course. (Sounds of agreement.) Geotech you could do with the basic class and most of the others, too. So initially this surprised me, but now it doesn’t.

A-13


Those of you in structures, is that why you are saying the classes are preparing you better than the review sessions?

--Structures is the one area where the classes really helped a lot and you didn’t really have to look back.

--Also we had no review session for structures. I wish we did.

--The structural area—it’s a huge different mindset, like circuits. People have trouble guessing in circuits. If you don’t understand the mindset of structures, you can’t really guess, whereas in some other areas you can. It is not always common-sense.

--It’s also not a high percentage on the exam, so if you’re not good at it, it doesn’t make sense to spend a lot of extra time.


Get a good night’s sleep before. (Much agreement.)

Make a point of putting the books away and distress for a few hours the night before in addition to sleeping.

Know the handbook.

Pay attention to units. Even if you don’t know the problem, that might get you through.

Know the bigger picture.

Acknowledge there will be some stuff where you have no idea. Circle C and move on.

Is it a misconception that many people take this exam over so the first time is a practice run?

Yes!

--Don’t want to spend the money again or eight the eight hours (all agree)

--Don’t want to have others know you didn’t pass it.

A-14

General framework

Students meet at a time and place determined by Roberts and Thompson. Some sort of food and drink will be provided (probably pizza and soft drinks.)

Burns will thank them for coming and introduce the P.E. who will also be attending. She will remind them of the agreement they signed before they took the exam, and ask them to not dis-close any exam content in their answers. She will also thank the P.E. who is observing, and let the students know that he/she is there, like they are, to help improve the program.

Questions

Each of these questions will be asked of all students present. If there seems some reason for a follow-up not listed below, Burns will ask a follow-up. If discussion continues and it concerns strategic planning for the exam, Burns will simply let it continue; if discussion ensues that seems to drift off-topic, she will redirect the discussion. All present will know that they can ask the practicing engineer any questions that seem relevant. They will be informed that the focus ses-sion will take no more than an hour (Burns should actually plan for 45 minutes).

1. Tellmealittleaboutwhereyouareintheprogramhere.

(For instance: what is your area of emphasis, and how far along are you? What internships have you had? What sort of other engineering activities do you participate in? )

2. Whatwasyourstrategicapproachinpreparingforthisexam?Pleasetakeamo‐menttothinkaboutthisbeforeanswering,anddon’tworryifwhatyousaydoesn’t“match”whatyouhearsomeoneelsesay.(Pause.)(Startwithwhoevervolunteersorpicksomeone.)

Appendix - Focus Group Protocol - Spring 2013

A-15

Follow-ups: Did you go to the review sessions? We’d like to know whether you strategically studied for some sections more than others, or skipped others (and if so, which ones)? Please name the content areas of the test, and let tell whether you studied for that section. How was your preparation for the morning session different than that for the afternoon session?

3. Howwelldoyouthinkthatstrategyworked?(Again,startwithwhoevervolun‐teersorifnoonedoes,picksomeone.Ifthesamepersonvolunteersrepeatedly,shifttoaskingthosetostartwhohavenotspokenfirst.)

Follow-ups: Why? Was there a section you felt especially good about, and if so, why do you think you were so well prepared? [Note: this may simply ask students to state the obvious, but hopefully it will show to what degree this group is aware of which sections students here typical-ly do very well on.]

Is there a section you felt you did not do well on? Was that strategic (a section you decided not to prepare for) or a surprise?


A-16

If you had to do it over again, are there any courses you wish you had taken that might have al-lowed you to do better on the exam? If so, which course? Were you aware before the exam that you were weak in the area covered by this course, and if so, why did you decide not to take that course? [Note: again, this may simply ask students to state the obvious. On the other hand, if no student mentions a structural analysis course such as statics or dynamics, that will suggest asking about statics and dynamics as an alternate follow-up question.]

4. Howwouldyouadviseafriendtostrategicallystudyforthistest?

5.

6. Fromwhatyouhavelearnedinyourcoursesorviayourinternshipsorviaothernetworking,pleasetella)whatarethespecifictechnicalareasinwhichaCivilEngineershouldbeskilled,andb)doesthistestdoagoodjobofassessingthat?


7. Fromwhatyouhavelearnedinyourcoursesorviayourinternshipsorviaothernetworking,pleasetella)whatarethespecificnon‐technicalaspectsofengineer‐ingareasinwhichaCivilEngineershouldbeskilled,andb)doesthistestdoagoodjobofassessingthat?


A-17


8. Whichofthoseareasmentionedinthelasttwoquestionsdoesthisexamdoagoodjobofassessing?Whichofthosethingsdoesitnotassess?Arethereothermeasuresofexcellenceyoubelieveshouldbeconsidered?

9. Didyouspecificallyprepareforthestructuralanalysissectionpartoftheexam?

Follow-up: If so, how? Over the last two years, students from UW-Platteville have consistently underperformed on the structural analysis section of the FE exam: does this surprise you? Why or why not?

10. Isthereanythingelseyou’dliketoshareaboutthisexamandhowyoupreparedforit?


A-18

Focus Group Transcript Summary Spring 2013

Student Attitudes on Fundamentals of Engineering Exam

Focus Group 4-14-2013

Transcript of focus group session

Students were given surveys before the focus group; those have already been compiled.

The revised questions and the protocol are attached.

Major questions are listed as headings. Questions are asked by the moderator; follow-ups, when needed, are listed by indicating the moderator [“M”] asking them and then the responses.

______________________________________________________________________

All students have signed in and most have gotten some pizza and a beverage. As they have signed in, they’ve been told that the signature also constitutes permission to use the data and to record them.

M: I’m recording this, and as I’ve mentioned, by taking the survey, you’re also giving permission for me to record this, transcribe it, and give it to the two others doing this assessment. Is that all right?

[Multiple voices agreeing.]

M: There will be a little duplication in questions since I won’t be looking at the survey you just filled out.

Moderator thanks everyone for participating; introduces herself. Reminds students that they can’t talk about anything specific from exam nor disclose any exam content—as they know, they’ve signed an agreement stating they would not do that. Thanks the student present, Melissa, who was present at the last focus group as a participant, has agreed to help out and jump in if need be. She is there, like they are, to help improve the program.

M: I will ask these questions of all of you; sometime I will mechanically go around the room and sometimes I’ll just start with whoever wants to start first. This should take no more than an hour.

1. Tell me a little about where you are in the program here, where you are now, area of emphases, what internships you’ve had, or anything else you’d like us to know.

A-19


• I’ve had two internships, I’ll have a third in the summer, so I’m in the last phase, will graduate next semester, in structural construction. Have had a co-op, an internship last summer, and will have another next summer.

• I have one more semester, I’m structural, worked construction last summer in Green Bay, and next summer I’ll be in San Diego working for a general constructor on a job site… I’m a 5th year senior.

• Similar to these guys, I’m structural emphasis and will graduate this summer after five years. Have had some internships, one with the SOT and another last summer, will have another this summer.

• This is my fifth year, trans, I have not had an internship and but I was going to get one then I didn’t get it; I didn’t sign a sheet of paper so it fell through.

• Fifth year, trans emphasis, will graduate this summer, have not had an internship. • Will graduate next semester, construction/geotech. I’ll graduate in four and a half years,

I have had one internship, more along the lines of environmental safety than civil. • I’ll graduate this semester with an emphasis in construction; no internships

2. What was your strategic approach in preparing for this exam? Please take a moment

to think about this before answering, and don’t worry if what you say doesn’t “match” what you hear someone else say.

• I knew it was a big undertaking. I had a review packet; I studied five hours a day all

spring break, then took a practice test. Came back and kept going. For the last week I studied – with him, actually--about 14 hours, just doing practice problems. More along the lines of practice exam problems, not as much practice problems from the sections even though we did go over them. More “how is the test going to question you,” preparing for the actual exam.

• [Name] and me and another kid who couldn’t make it today went to nearly all the review sessions th campus provided. Those would be about an hour. Then ___ and ___ would go and study like 2-3 hours about that sections. That was months back. Then these past few weeks or so we started meeting 2-3 hours every day. And we had that big book. Sample [several voices break in to describe the book and agree and one says many people had the yellow book ] sample FE style questions, like chemistry, math, we went over that.

M: Let me just ask, how many of you went to the review sessions?

[All but two hold up hands. One person went to the first one; another said it is his second time taking it so he went to them last semester, then didn’t really study.]

A-20


M: How many of you went to all of the review sessions?

[Several say that the enviro one was never rescheduled due to the storm. Most went to at least two or three of the review sessions.]

M: How many of you used the .pdf or the Lindeberg (?) book? [Everyone. “Lindberg.”] [That one and the SE book; the reference manual; several people used practice manuals bought from NCS site, problems with sample tests. Discussion: most had “big 800 page one” another that was “500 pages.” Some did not have the problem book or reference book from on-line but it seems that all but one who didn’t buy it borrowed copies from friends. Return to general discussion]

• I borrowed the 800 page book and took some of the on-line tests but I don’t think those were too helpful, they didn’t give enough information.

• I used the big Lindberg book too and there was a Kaplan book, a civil book, that was less than helpful, some of the solutions in it were wrong….

• [Another student interjects that he heard it was terrible. Two others echo “terrible.” Melissa [student proctor] says that was the one so many students in her focus session had talked about. [Another student says it should be taken off the market and two others agree, saying it was “bad.”]

• The questions are terrible, the answers are wrong. Or they work the problem right and give you the wrong letter, or use the wrong equation. [More agreement.]

M: Did anyone use Kaplan and find it helpful?

• [Multiple negative responses.] • Everyone—our friends used it and hated it. • Thursday it was driving everyone crazy. • The Kaplan [review for the] morning section wasn’t as bad as the [review for the]

afternoon section. • The Civil portion of that one was a total waste of money. • But the benefits of the afternoon part, that Civil portion was the only review I had of

Civil I had so I just started paging through it to see what kinds of questions thy’d ask. [Agreement]

• When I got tired if Kaplan I started looking at old exams from civil classes, and that was more helpful.

A-21


M: Did anyone else look at old exams?

[Several “no”s]

• I had a different book for the afternoon session that had two different practice exams and it was pretty good, but I didn’t see anyone else who had that book.

• The key was understanding how to take the test especially the timing issue. Especially the morning. So taking a timed practice test really helped me to know how much time to give each problem. The curriculum isn’t that bad, but it’s the timing issue. Don’t go more than two minutes, and know when to guess. M: could you just as a group name the content areas of the test and let me know which ones you studied for? [Silence.] Okay, would anyone want to name off the morning sessions? And can you tell me if you studied for those particular sessions?

• Mathematics • Fluids [Another adds: “yeah, studied a lot for that.”] • Mechanics, materials, statics…. • Thermodynamics • Circuits [some laughter after this one] [“I didn’t study much for that.” • Computer spreadsheets, that was one of them • Heat transfer • Material Science • Heat transfer • Ethics • Chemistry

M: Which of these in the morning did you study for, which did you not study for, and which did you just study a little?

• All of them • Except ethics • Yeah, you just gotta know ethics. • Last FE I thought the ethics problems were relly difficult, because I thought they were all

right, but one was just “more right” than the others. I thought on this FE this time there would be a couple that were definitely wrong and one that was more right and it would jump out at you. [Agreement—“yeah, one would just kind of jump out at you.” “I thought it was kind of ‘eyeball’.]

• There’s an ethics code in the manual and you can read through the model but a lot is common sense.

A-22


M: What I’m hearing is that you didn’t study as programmatically for the ethics section, though students here do pretty well on that one. Is that right? [Agreement] Is there any other area? Let’s do it this way… how about chemistry? Did everyone study for that one? [Agreement]

• The only ones I didn’t study for were circuits and that computer one. • Yeah, stuff that we’ve never even taken classes on.. • … I was like, don’t even worry about it. • [Agreement] When it came up in a question… • …the key is, master what you know, and the other stuff let it go… • [Agreement] • Know what you know, know what you don’t know, and know when to guess. • I didn’t study for the math, because I realized that the calculator that you have can do all

the math, it does everything for you so you need to know nothing about math… • I studied a lot for the math • But that calculator that we had the TI 36, it does everything [agreement] • I wish I’d had that throughout college instead of my graphing calculator. • TI 36 Access Pro [?]

M: Let’s shift to the afternoon sessions. What were the afternoon sessions you took”

• Soil mechanics • Fluids • Foundations • Trans • Structural analysis, steel • Structural design • Reinforced concrete • Environmental engineering • Surveying • RC • Hydrology • Construction management • Material Strengths

M: So you all take the same sections, right? [Yes] So in this case, how did you study? (Pause.) Okay, think of the strategies you talked about before.

A-23


3. How well do you think that strategy worked? M: I’m going to start with you because of how you identified your area of emphasis. [Moderator begins with student who’d identified himself as a structures emphasis.]

• Yeah, I guess I studied less for structures because in that area I was more confident. Though I felt actually weaker when I got to that part of the test so I probably should have studied more. [laughter.] But I had just taken structures and reinforced concrete, those were some of my more recent classes, so I spent more time studying for environmental and geotech. And I did feel confident on those sections, but less confident on structures, which was a bigger part of the test.

• I concentrated more on the general stuff rather than civil, because I thought I’d have more time and thought I had a better understanding of the civil stuff. So on the general test I studied almost everything except I completely ignored thermo and chemistry because I thought those two would be a guess. I didn’t ever take thermo, so I figured I’d take my chances. I spent a lot of time on math. M: In the afternoon, how did others here feel you did on the structures section?

• Mediocre [note: not a structures student] • Not overly confident, but not terrible [also not a structures student. Construction] • Terrible. I hated it. I didn’t review it because I just don’t like it at all. I pretty much

focused on the general stuff and I felt more confident with the civil portion because that was two years ago and the more recent stuff was easier, like trans, that’s my emphasis.

• Well, I thought I knew hydrology [laughter] till I got to the test [more laughter]

M: I hear this laughter; was hydrology a section many of you were surprised by?

• I’ve never studied it • I was like, “what does this mean?” • I took hydrology last semester… • …so did I but… [more laughter] [over half the focus group surprised by this section.]

M: Let me ask the structures emphasis students here: how did you find that section?

• More than half I think I figured out, I could find an equation in the book and the variable were given in the problem but there were still a few I had no idea how to solve. That was

A-24


frustrating because I thought I should know them. I mean, maybe a guy like ___ doesn’t know it [laughter] because he’s in trans, but I should know it.

• As far as the trans section and the construction section, it seemed pretty basic. We learned that stuff and it was something you’d learn in the basic class, but the structural stuff was something you’d only get in steel or some other class.

• I did a silent cheer when I got to structures. The first half, geotech, soils, I literally skipped the first ten questions, Surveying was fine. But for structures I cheered. I did every single problem. There were only one or two I had problems with. I’m in steel design now and I was in RC and so every class the last year and a half has helped with structures.

• Same as him. The one section I had trouble with was surveying, and that was totally about the professor. He isn’t the best teacher. And the course is easy but the way he goes about it is terrible. But the structural was fine. Having previous knowledge from other structures courses really helped. If you didn’t have that you might get it from the book, but probably not. The rest… hydrology… that crap, I had one class, I got through it.

• No one has hydrology but municipal, or… it’s in your emphasis. Structural, we take basic, geotech, trans, enviro, that’s it. So there were questions that were right over our heads. M: If you had to do it over again, are there any courses you wish you had taken that might have allowed you to do better on the exam? If so, which course? For example, would you have taken hydrology? I notice a lot of fifth year seniors here… would it have been worth the extra time?

• I don’t think so. • It wouldn’t be worth the time and effort to take hydrology. • Not worth the time or money. You focus on what you know. • I took RC too early. I wish I’d taken it last year. • It would be best to take the test closer to your structures courses. If I had taken it right

after RC, I would have done much better on that section. • I didn‘t take thermo, but I don’t regret that. [laughter.] • [One student asks the others how they felt about the thermo section: “did everyone feel

okay about that?”] “I just used logic” “went back to charts” “constant pressure.

4. How would you advise a friend to strategically study for this test?

A-25


• I wouldn’t take advice from a 4.0. You ask people who have taken it, and you get the full gamut. Some people will say to study everything, go back over calculus, and others will say don’t study all the time.

• For the average person, start earlier. • Study a couple of hours, not ten hours a day like we [group of three] did. • Start earlier and do a subject a day, or just read the manual • If you know what the equations are… • The last couple of days I just paged through the equations and thought about what kinds

of problems would use that equations. You don’t need to know all that, but you need to know how to use it. [lots of agreement]

• Like he did, read over that over spring break and think how you’d use it. The supply manuals.

• I memorized some page numbers, so I’d think “I need to go to this page” or that page • Even in the afternoon, I was still flipping back to the morning stuff to the statics part… • … or even in the enviro there was chemistry stuff you had to flip back to, even the

periodic table… • … on the materials section I’d have to flip back, I thought “what section am I taking

here?”

5. From what you have learned in your courses or via your internships or via other networking, please tell a) what are the specific technical areas in which a Civil Engineer should be skilled, and b) does this test do a good job of assessing that?

• I don’t know if we’ll ever know how to do this stuff in our real jobs. • I’ll never need to know how to do a moment, I’ll just plug it into your computer. • It’s teaching you how to think like an engineer but you might not have to do these

problems • Troubleshooting, yes. Thinking under pressure.

M: So what do you mean by “think like an engineer?”

• Just like how to critically think, so when you have a problem, you can use what’s in it,

the units or such, to get to an answer, and I’d say if you haven’t done problems like that before, you’d be over your head, but if you’re in engineering you’d know how to get from here to there and piece things together.

A-26


M: It may be here that all of you are assuming that some things are too basic to say. So are you assuming, for instance, that a basic technical skill you need is math? [All agree} That is so obvious you’re not saying it? [Multiple nods.]

• You need to know how to use your resources. You need to know how to use that book. Because without it I’d only be able to asked like five out of 180 questions. M: SO you’re saying that a technical skill is knowing how to use those resources? [general agreement then:]

• That’s probably what we’ll use most. • We’re not going to be pulling these things out of our head, we’re always going to have

some source we can use, and someone checking your work or a computer checking your work.

• I think a big skill is time management because the curriculum is not that hard but you have to manage your time so well on every question so you’re not stuck and keep moving on and keep thinking.

• Also, know when to step away. If you spend a little time on one, make a note, and don’t skip any if you can make an educated guess. Also later three or four questions later you might see how to solve that problem and remember to go back.

M: Rather than identifying a particular bit of knowledge, I’m hearing you say that that it’s knowing how to use your resources and time management skills?

[Agreement.] [Follow-up: If no one mentions CEE learning outcomes in these areas, name those outcomes and ask again.] Agreement again… no one thinks of math as a technical skill because “it’s what we do.”

6. From what you have learned in your courses or via your internships or via other networking, please tell a) what are the specific non-technical aspects of engineering areas in which a Civil Engineer should be skilled, and b) does this test do a good job of assessing that?

• You have to have communication skills, but I’m not really sure how this test assesses you for that.

• I agree. The basic non-technical things I think are way more important on the job. You have to converse and work with people. The technical aspects are just kind of given. I

A-27


guess this test does assess those technical things, but I think the non-technical things like communication are more important but they can’t really be taught.

• You can’t really test that on something like this, its just something you pick up. • You learn that from either job experience or basically anything you do outside of a

classroom. M: But you do have a senior design where you’re writing and communicating verbally, don’t you?

• Yeah, I guess. • I’m not in that. • I suppose. I think I’m going to take back what I said,[that they can’t really be taught]. A

lot of courses do stress communication and group projects, not just senior design. Sooner or later you’re going to have to work in a group and communicate, maybe argue in a peaceful manner [laughter] M: All of you had comments on the ethics section. That’s a non-technical section. Does it do a good job of assessing that?

• Yes, but its basic. • Someone not in engineering could pass it. • Did anyone take Science Tech and Ethics? It’s a double counter. There were a bunch of

questions, Aristotle and that, it made that section easy. • You go over scenarios in your engineering classes, in construction classes. • Barely any class I had talked about ethnics until senior design and then there wasn’t

much. • When that guy came and talked to our senior design class, and talked about different

ways of conducting yourself around clients, that was good. • I think that the ease or difficulty of the ethics section also depends who you are as a

person. You say it was easy but I know some people who had no clue. I hate to say it but they were just that kind of person. [discussion: where did that sense of ethics or personal conduct come from or not? Some students getting pizza.]

7. We’ve already covered this a bit, but which of those areas mentioned in the last two questions does this exam do a good job of assessing? Which of those things does it not assess? Are there other measures of excellence you believe should be considered?

A-28


[“Seems like we covered it.”]

M: Does anyone have anything you want to add? [No.]

8. Did you specifically prepare for the structural analysis section part of the exam?

If so, how? Over the last two years, students from UW-Platteville have consistently underperformed on the structural analysis section of the FE exam: does this surprise you? Why or why not?

• It does surprise me. • Well, there were only a handful of structural emphasis people taking it. • I don’t want to put this emphasis on a pedestal, but it is not easy. So it is kind of

shocking to me that we wouldn’t do well. • I can see it, but I wouldn’t have thought of it. I mean, I’m sure these guys [gestures at

structures students] did well, but I didn’t take those courses and in my other courses you never revisit it. What I’ve learned.

• M: It seems that the four or three of you in structures studied less and you felt more confident. Is that correct?

• In all honestly it is hard to study for a structural analysis question [agreement] because there are so many different types of problems and you never know what you’re going to ask and you have to apply different equations and you never know which ones.

• It is the largest section and you never know what will be there. • There are questions on steel design, but if you’ve never been in steel design you’d never

know how to do that section.

M: So if those of you here in structures were to give advice to these other guys who are not in structures about how to study for that part of the test, what would it be? [Grinning.] [Laughter about who is capable about what]

• Know that section in the reference manual. • On the practice exam, I had to go to Dr. Roberts, who is one of the best people in the

department, and he didn’t even know. He had to go back to one of his books. It is a hard section to study for.

• Work problems and hope you get something like that. But you never know. • [questions about how particular emphasis areas do on parts of the test.]

M: The way aggregate scores come, you can’t know how one area of emphasis does on a particular part of the test. At least, as far as I know. Let me ask this: if you’re in

A-29


structures, so you think the FE exam is a good assessment of you knowledge of structures?

• There is so much they could test you on and so they could cover… you could take it [the test] twice and make one twice as long and still not cover everything.

• Also, the problems I went through, if you look back in your RC notes, probably your first three weeks of notes would help you the most with the majority of questions. Even though that reference book is crazy, they can still only ask you a four-minute question.

• Some of our homework takes a couple of hours, and they can’t ask a question like that. • So review basic concepts.

M: Would the three of you [not in structures] agree with that?

• Probably. • Yes, that would have helped. There was the same phrase in a lot of different places, and I

wished I’d have gone over it. If I’d gone over the first few pages of notes maybe I would have gotten it.

• I took the test last semester and didn’t do good on it. I probably should have gone over my structures and RC notes, but I didn’t, and it’s probably my fault, but I didn’t do it.

• I feel that’s a great suggestion [going over the first few pages of notes]. Stick with the basic core concepts in structures. Look over the theory stuff.

• They won’t have time to ask you a lot of in-depth questions so if you review the theory you can narrow it down better.

• [Discussion about the problem of not knowing how structures students do on that section as compared to non-structures students.]

• When you pass, that’s all it says is that you passed. • When I failed, I got a link I could click on and it said how you performed in every area

and how you compared to people who passed. So I looked at that. Another reason I didn’t study math was that I did way better than the average in the math section. I did poorly on everything else.


• My advice is that make sure you read through problems all the way before you start looking up equations or you might start to solve the wrong question.

• Yeah, keep it simple. • You turn it into a riddle, and then think, oh,it’s not that. • Take a timed practice test. You can do all the work you want but its stressful when you

sit down and see everyone. You have to put yourself through it once. • Know that manual because you’ll use it back and forth. • Know how to use the index.

A-30


• Some are definitions you can find in the index; go to that page, and BAM. • Know the graphics because some help you understand the equations.

Moderator thanks the group.

A-31

Name _____________________________________ CE Emphasis Area ___________________________

F.E. Exam Focus Group – Initial Survey 1. Briefly list the things you did to prepare for the F.E. Exam. Continue on the back if needed.

2. For each of the following topics on the F.E. exam, circle the response that best expresses how you feel about your performance:

Chemistry Very Confident Confident Unsure Very Unsure

Fluid Mechanics Very Confident Confident Unsure Very Unsure

Mathematics Very Confident Confident Unsure Very Unsure

Strength of Materials Very Confident Confident Unsure Very Unsure

Statics Very Confident Confident Unsure Very Unsure

Environmental engineering Very Confident Confident Unsure Very Unsure

Construction management Very Confident Confident Unsure Very Unsure

Soil mechanics Very Confident Confident Unsure Very Unsure

Transportation Very Confident Confident Unsure Very Unsure

Structural analysis Very Confident Confident Unsure Very Unsure

3. For each of the following topics on the F.E. exam, circle the response that best expresses how often

you guessed at the answers:

Chemistry Always Most Some None

Fluid Mechanics Always Most Some None

Mathematics Always Most Some None

Strength of Materials Always Most Some None

Statics Always Most Some None

Environmental engineering Always Most Some None

Construction management Always Most Some None

Soil mechanics Always Most Some None

Transportation Always Most Some None

Structural analysis Always Most Some None

A-32

Appendix - Focus Group Survey

Chem

istry

Fluid

Mechanics

Mathe

matics

Strength of

Materials

Statics

Environm

ental

Engine

ering

Constructio

n Managem

ent

Soil Mechanics

Transportatio

n

Structural

Analysis

ConfidenceVery Confident 0 1 4 0 1 1 1 1 2 2Confident 1 5 2 5 3 3 4 2 3 2Unsure 5 1 1 2 3 3 0 4 1 3Very Unsure 1 0 0 0 0 0 2 0 1 0Weighted Avg 2.00 3.00 3.43 2.71 2.71 2.71 2.57 2.57 2.86 2.86 Higher Score = More ConfidentGuessesNone 1 2 5 2 1 1 3 3 5 2Some 5 4 2 4 5 5 2 2 1 4Most 1 1 0 1 0 0 2 2 1 1All 0 0 0 0 0 0 0 0 0 0Weighted Avg 3.00 3.14 3.71 3.14 3.17 3.17 3.14 3.14 3.57 3.14 Higher Score = Less Guesses

Appendix - Numeric Survey Results

A-33

assessment activity fund report: student attitudes on the ... · the fundamentals of engineering...

Documents