Paper ID #16827

A Comparison of Paper vs. Electronic (Portfolio) Notebooks for EngineeringDesign Projects

Dr. Jessica A. Kuczenski, Santa Clara University

Dr. Jes Kuczenski joined the engineering faculty at Santa Clara University in 2014. She obtained her M.S.and Ph.D. from the University of Notre Dame and her B.S. from Iowa State University all in ChemicalEngineering. Dr. Kuczenski has been teaching since 2007 and focuses on courses which are commonlyfound in first years of an engineering education (e.g. introduction to engineering, engineering graphics,statics, and dynamics) or are heavily based in engineering design.

Ms. Erin Susan Araj, Santa Clara University

c©American Society for Engineering Education, 2016

A Comparison of Paper vs. Electronic (Portfolio) Notebooks for Engineering Design Projects

Abstract An engineer’s notebook has long been regarded as a critical feature in the engineering design process to capture the collection of information and design thinking not only for the author’s use, but also the use of others. As technology progresses, the use of a hardbound book for this recording of thoughts and reporting technical information has been extended to electronic formats. This paper details a comparison between engineer’s notebooks recorded on paper using bound notebooks and those done electronically using an electronic portfolio system. Both notebooks were completed as part of a 10­week community­based engineering design course in different quarters. An assessment method was developed to quantify the quality and frequency of particular types of artifacts including visuals, steps of the engineering design process, and reflective elements. Overall, the implementation of the electronic portfolio has largely been successful with clearly visible benefits. In this paper, we report on the results of the assessment process from both types of notebooks, the results from a survey on changes in student skills, and our conclusions. Introduction An engineering notebook is simply any notebook an engineer uses to record design thoughts and collected technical information during the course of a project. Engineers use project notebooks to record project work as it progresses as a method to reconstruct, interpret and evaluate the processes carried out. It is also important for the engineer to carefully record information so that it can be reviewed and considered in cases including patent applications to protect the design idea or for projects that will have controversial or economic ramifications.1 Historically, hardbound paper notebooks used by engineers are typically bound, containing 100­150 pages, and have a list of suggested guidelines including a table of contents and designated places at the bottom of pages for the author’s signature, a witness’s signature, and a date of completion.2 Recently, as technologies develop, the use of electronic engineering notebooks has been a focus of research (in both engineering and science)3­6, although they have never been fully implemented as standard engineering practice.7­10 A variety of electronic formats have been envisioned for documenting student design project progress examples of which include online spreadsheets,9 websites,11­12 e­posters,13 wikis and weblogs,14 electronic portfolios,15 and more. For this project, electronic portfolios (eFolios) are chosen as our electronic notebook technology, in part due to the similarities between electronic notebooks and electronic portfolios and also because there is a body of work in education that evaluates and discusses the impacts of eFolios. Overall, eFolios are “digital collections of artifacts that provide authentic, valid, and reliable evidence of a learning experience.”16 The act of creating of an eFolio is an important learning experience in itself as it increases student ability to organize, make connections, reflect on, and publish their coursework.17­21 The set of artifacts contained in an eFolio, together with reflections and annotations, tell a unique story about some aspect of the author’s learning by helping the

author make visible his/her knowledge, experience, and growth. We hope to utilize eFolios to assist the student learning process, to showcase student work and to assess the work completed. Ultimately, the electronic format chosen is likely not as important as how it is used. The importance of the implementation of an engineer’s notebook is in the process of maintaining design records in an authentic way.2 Our belief is that the engineer’s notebook completed by our students should involve formative (on­going) assessment as well as summative (end­of project) assessment, and should additionally help students reflect upon what they are learning and their methods for completing their design project. To this end, this paper describes an overview of the course where the notebooks were assigned followed by a comparison between paper and electronic engineering notebooks using a quantitative scoring method looking at five categories including organization, visuals, engineering design process, and reflection. Finally, student opinions regarding their abilities in these areas before and after this course were surveyed and presented below. Background The Community­Based Engineering Design course is a 10­week course where student teams are partnered with a local community business or organization and complete a design project from problem identification through final prototype. The purpose of the course focuses on “hands­on” experience in project management, building cross­disciplinary team skills, communication, and prototyping. In fact, all students are trained to use our Maker Lab on campus as part of the course and have the ability to use the laser cutters and 3D printers (e.g. MakerBots) in addition to general tools. The course is available to all students at all levels and is open to all majors; student teams are often purposely formulated with a mix of majors and ages. The course meets for 2 hrs and 45 min once per week. Class sizes are typically small with an average of 10­14 students per quarter. The beginning of the course is focused on community partner research including a presentation from the partner and ‘on­site’ interviews by student teams. Additionally, the students learn about project management, how to run and document effective meetings and how to use their own personalities (strengths and weaknesses) to develop team roles and structures, i.e., team contracts and Gantt charts. As the course progresses, teams are encouraged to work on their project in open lab sessions with weekly short project updates with the instructor. An oral report on the status of their project is presented to their community partners during week 7 with final project documentation including a final report and final oral presentation typically given during finals week (week 11). A major portion of the student grade (~33%) is to keep an engineer’s notebook to track the development of their project. The notebook contains an archive of the entire design experience from early brainstorming through design decisions and ultimately final results. Bound paper notebooks were used for this purpose from the course’s inception (circa 2009) through fall quarter 2014. Electronic notebooks began use in winter quarter 2015. The anticipated benefits of electronic notebooks included:

an increase accessibility to the notebooks between the student author and interested parties including their team members, community partners, and the instructor

enhanced navigability inclusion of clearer visuals and better multimedia capabilities

Digication was chosen as the electronic notebook platform. Digication is a university sponsored and maintained tool which interacts well with Google collaborative software already implemented on campus. Digication allows students to create their own personal eFolios by forming websites which are collections of webpages and has many ‘modules’ available for the addition of artifacts on each page including rich text, images, video, audio, among others. Students can access and edit their eFolio at any time from anywhere (with internet access) and allow for access throughout their college career including after graduation. Additionally, team members and even ‘external’ community partners can be set up to view the eFolios with permission. A Digication template was made for the course as an example scaffold for students. The template included a home section to serve as a welcome page, a biography section for their own biography as well as for team members, a ‘journey’ section for their individual project contributions, a team documentation section for all team related/contributed documentation, a reflections section for assigned reflective pieces, and a final project section to serve as a repository for their final project documentation. A screenshot of the template portfolio is given in Figure 1 where the different sections can be seen at the top menu ribbon. Template text and/or examples were added in all sections to introduce students to typical content in these areas. There were also a few introductory tutorial videos made available to students from our Academic Technologies department on campus, but no additional training was given as it was assumed that our students are ‘digital natives’, a term described by Prensky (2001) toward today’s college students who are fluent in the language of computers, video games and the Internet.22

Figure 1: Screenshot of the course template organization

Methods Our main research question is whether or not eFolios are an adequate substitute for paper engineering notebooks. To answer this question, we coded the notebooks for evidence of organization, visuals, engineering design process, teamwork, and reflection. A simple code of 0 ­“insufficient/no evidence”, 1 ­ “sufficient/provides evidence”, and 2­ “exemplary/extensive or multiple evidences” was used. Both authors coded the notebooks independently and results were compared and discussed until consensus was reached. The Google form used to code the notebooks is given in Appendix A. Each of the categories listed above were separated into particular areas of interest. Table B1 in Appendix B (due to length) describes each category with associated areas and a brief description. It should also be noted that when both paper and electronic engineering notebooks were used, the notebooks were collected weekly and feedback given within 24 hrs. Additionally, an IRB­approved survey is utilized in this class to assess student learning and opinions regarding the course content and information. For this study, a subset of survey questions relating to the engineering design process, teamwork, and communication was taken to further illuminate student learning. Importantly, we are looking to ensure that the quality of the course has not been negatively impacted by the use of the electronic notebooks to document the project progress. An ‘introduction’ survey is given at the beginning of the course for a baseline of students as they enter the course and results are compared to an ‘exit’ survey taken at the end of the course. The survey results were taken pairwise from each student and the number of students with ‘loss/no change’, ‘gain’, or ‘strong gains’ was noted and given as a percent of total students in the course. The IRB survey questions are given in Appendix C. Only student results where both the introduction and exit survey were taken were included. It should be noted that the survey was reviewed in fall quarter 2014 and the new survey approved and implemented winter quarter 2015; only the questions which were the same on both the older and newer versions were included. Results & Discussion The results from both the qualitative scoring of evidence presented in both the paper and electronic notebooks is presented below in the ‘Quantitative Scoring’ section, followed by the results of the IRB survey data for both notebook types in the ‘IRB Survey’ section. Quantitative Scoring ­ Paper Notebooks We first assessed the paper notebooks for the quality and quantity of artifacts as they relate to a design project. A random sample of 20 paper notebooks from spring quarter 2013 through fall quarter 2014 were collected. In sets of ten, the engineering notebooks were independently evaluated by each author through survey answers to 25 questions relevant to what an engineering notebook commonly includes, as previously discussed. The coded data collected the first set of 10 paper notebooks showed many similarities, but did contain some inconsistencies based on the evaluator. Out of 500 responses (10 journals, each evaluated twice with 25 of questions each), there was a 5­10% difference between the evaluations that were recorded. For any differences between evaluations, the differences were discussed including rationale for their ranking and the notebook in question was re­evaluated by

both authors together to ensure a consensus was reached. In particular, the first set of 10 evaluations helped to form a more cohesive understanding of how the notebooks should be evaluated. Therefore, the inter­rater reliability of the data increased as more notebooks were reviewed and ultimately decreased to an average of less than 2% for subsequent evaluation rounds. Overall observations from evaluation results for paper notebooks were that although engineer’s notebook documentation standards were typically followed (generally by the end of the quarter), organization was very poor. Only 2 paper notebooks utilized a table of contents for locating information, and only 60% made use of page titles or headings to help clarify information. With regard to visuals, it was found that sketching was present in the majority (~70%) of notebooks. A table summarizing the results from the qualitative coding process in each category is found in Table 1; full results given in Appendix D. Scores of 0 were deemed ‘insufficient’ and scores of 1 and 2 were ‘sufficient’. Totals from each question given in percent of total student notebooks are shown in Figure 2.

Table 1: Qualitative scoring from paper and electronic engineering notebooks

Organization Visuals Design Process Teamwork Reflection/Metacognition

Score O1 O2 O3 V1 V2 V3 D1 D2 D3 D4 D5 D6 D7 D8 D9 T1 T2 T3 R1 R2 R3 R4 R5 R6 R7

Paper Notebooks 1&2 14 3 8 11 10 9 9 11 10 9 7 13 10 6 2 3 13 0 6 14 8 7 3 6 10

0 6 17 12 9 10 11 11 9 10 11 13 7 10 14 18 17 7 20 14 6 12 13 17 14 10

Electronic Notebooks 1&2 0 20 20 10 9 14 15 12 13 15 14 18 18 8 19 13 17 19 13 11 12 14 20 14 20

0 20 0 0 10 11 6 5 8 7 5 6 2 2 12 1 7 3 1 7 9 8 6 0 6 0

Figure 2: Percent of paper and electronic notebooks with scores of ‘sufficient’ (score of 1 or 2 in the qualitative

scoring)

Qualitative Scoring ­ Electronic Notebooks Qualitative scoring for the electronic notebooks was completed from a random sample of 20 eFolios from winter, spring and fall 2015 quarters, following the same procedure discussed in previous sections. Table 1 also includes the qualitative scoring data from the electronic notebooks, summarized by ‘sufficient’ and ‘insufficient’ as previously presented for the paper notebooks; full results are given in Appendix D, Table D2. An illustration of the percent of electronic notebooks in each category overall is also presented as part of Figure 2. For clarity of comparison, data presented in Figure 2 are combined and summarized by question category in Figure 3. Our results indicate there are clear differences between the paper and electronic notebooks with clear gains in the electronic notebooks overall. This is represented by a greater than 50% ‘sufficient’ rating in all question categories whereas comparatively the paper notebooks only received 30­50% ‘sufficient’ ratings. The electronic notebooks contained more information relevant to all categories investigated, though particular gains were in documenting the engineering design process, teamwork, and reflections where over 70% over the electronic notebooks had a ‘sufficient’ rating compared to 50% or less in the paper notebooks. We hypothesize that gains in these categories were due to student awareness, as students were more aware of these items from the beginning (offered as part of the template) and had plans to include and organize them. Furthermore, students had ‘examples’ of other electronic student notebooks serving as examples to consider in their own implementation.

Figure 3: Percent of paper and eFolio notebooks with scores of ‘sufficient’ (score of 1 or 2 in the qualitative scoring) or

‘insufficient’ (score of 0 in the qualitative scoring) by question category.

Furthermore, the electronic notebooks were better organized due to their very nature as a collection of webpages. The eFolio webpages were comparatively more clear and searchable due to menu ribbons and page/subpage organization and due to general webpage search options (‘find’ functions) available. This organization was likely further enabled through the use of the eFolio template offered to students as we did not see a large deviation in notebook formats from the original template organization. Finally, although visuals were included in both forms of notebooks, we did notice a dramatic decrease in the amount of sketching in the electronic notebooks. We believe students were simply less likely to follow the multi­step process of sketching, digitizing images, then uploading their sketches. However, many more visuals were included overall in the form of website images, photographs of project progress and teamwork, and technical drawings. IRB Survey The results from the IRB surveys are shown below in Figure 4 which summarizes the results for 20 paper notebooks from 2011 to 2013 and 19 eFolio notebooks from 2015. For clarity, the responses are summarized from all questions and presented by percentage to normalize the data. Student results were paired using student IDs and gains or losses were reported by taking difference between the exit survey and introduction survey results. Results were classified into four categories, ‘strong loss’ for differences greater than ­1, ‘slight loss/no change’ for differences of ­1 or 0, ‘gain’ for differences of +1 or +2, and ‘strong gains’ for differences greater than +2. Results are given per question in Appendix E; see Table E for the data in tabular form Table E or Figures E1 and E2 for visual summary.

Figure 4: IRB survey results from all questions for paper and eFolio engineering notebooks.

As shown in Figure 4, it is clear that student opinions regarding a change in their abilities are generally more positive in the eFolio responses. The data illustrates that there were more ‘gains’ and ‘strong gains’ present in the eFolio documented classes (~ 60%) than in previous paper documented quarters where the majority of data was in the ‘loss’ or ‘no change’ classification (~60%). We can therefore surmise that requiring student project documentation in the electronic format has not impeded their progress in these areas. The positive changes between the introduction and exit surveys are perhaps more clearly revealed when looking looking at individual question results from the IRB survey, as shown in Figure 5 (the question text is overlaid for reference for each numbered question). As shown, for all questions positive changes between the introduction and exit surveys are illustrated for eFolios over paper notebooks. In particular, substantial gains were shown for questions 1­3 and 8­11 where a 30% or greater improvement was seen between eFolios and paper notebooks. The largest positive differential is seen for question 2 where 85% of students showed ‘gain’ or ‘strong gain’ using eFolios vs 20% using paper notebooks.

Figure 5: IRB survey results by question for paper and eFolio engineering notebooks. Only changes between the

introduction and exit surveys resulting in a ‘gain’ or ‘strong gain’ change are presented.

It is difficult to attribute the positive changes shown in IRB survey results to any one course characteristic. One exception might be for question 9, where the positive change was likely due to inclusion of Maker Lab training as part of the course. In general, we believe many of these gains were related to the more structured design communication expectations from the eFolio templates. Additionally, the course learning objectives have been more focused on project management, teamwork, and communication, with particular emphasis towards partner

communication with regard to design expectations. The application of these ‘soft’ skills in relation to their discipline skills has been well received. Conclusions When looking at student project progress using an engineer’s notebook, there seems to be a clear benefit to our eFolio­based electronic documentation over traditional paper notebook documentation for our community­based design course. There were clear gains in organization, visuals, the engineering design process, team documentation, and inclusion of reflective writing elements. It was interesting to note that although both notebooks made use of visuals, they were largely different types of visuals with the paper notebooks being largely sketch­based and the electronic notebooks including more technical drawings, web images, and project photographs. Finally, the results from an IRB survey taken by students prior to taking the course and at the end of the course indicates that student opinions of their abilities in areas concerned in this study showed strong gains using electronic notebooks. Works Cited [1] Dawn Kowalski.. (1994 ­ 2012). Project Notebooks. Writing@CSU. Colorado State University. Available at http://writing.colostate.edu/guides/guide.cfm?guideid=80. [2] Kelley, T. t. (2011). Engineer s Notebook ­­ A Design Assessment Tool. Technology & Engineering Teacher, 70(7), 30­35. [3] Bird, C., Willoughby, C., & Frey, J. (2013). “Laboratory notebooks in the digital era: the role of ELNs in record keeping for chemistry and other sciences.” Chemical Society Reviews, 42(20), 8157­8175. [4] Bruce, S. (2003). “A Look at the State of Electronic Lab Notebook Technology.” Scientific Computing & Instrumentation, 20(2), C­15. [5] Machina, H., & Wild, D. (n.d). “Electronic Laboratory Notebooks Progress and Challenges in Implementation.” Jala, 18(4), 264­268. [6] Butler, D. (2005). “Electronic notebooks: A new leaf.” Nature, 436(7047), 20­21. doi:10.1038/436020a [7] Gwizdka, J.; Fox, M.; Chignell, M. (1998). “Electronic engineering notebooks: A study in structuring design meeting notes.” Proceeding of Enabling Technologies: Infrastructure for Collaborative Enterprises. Standford, CA. [8] Puccinelli, J. P.; Nimunkar, A. J. (2014). “An experience with electronic laboratory notebooks in real­world, client­based BME design courses.” Paper presented at the 121st American Society for Engineering Education Annual Conference & Exposition, Indianapolis, IN. [9] Candenas, M. (2014). “An implementation of electronic laboratory notebooks (ELN) using a learning management system platform in an undergraduate experimental engineering course.” Paper presented at the 121st American Society for Engineering Education Annual Conference & Exposition, Indianapolis, IN. [10] Kudrle, W.; Iyer, R. (2012). “A semantic electronic lab notebook for education.” Computers in Education Journal, 22(3), 35­42.

[11] Kajfez, R. L.; Kecskemety, K. M.; Kross, M. (2015). “Electronic notebooks to document the engineering design process: From platform to impact.” Paper presented at the 122nd American Society for Engineering Education Annual Conference & Exposition, Seattle, WA. [12] Christy, A.; Lima, M. (1998). "The Use of Student Portfolios in Engineering Instruction." Journal of Engineering Education, April, 143­147. [13] Takayama, K.; Wilson, J. (2006). “Mapping student learning throughout the collaborative inquiry process: the progressive e­poster.” International Journal of Innovation in Science and Mathematics Education, 15(1), 24­32. [14] Chen, H. L.; Cannon, D.; Gabrio, J.; Leifer, G.; Toye, G.; Bailey, T. (2005). “Using wikis and weblogs to support reflective learning in an introductory engineering design course.” Paper presented at the 112th American Society for Engineering Education Annual Conference & Exposition, Portland, OR. [15] Johnston, J.; Kant, S.; Gysbers, V.; Hancock, D.; Denyer, G. (2013). “Using an eportfolio system as an electronic laboratory notebook in undergraduate biochemistry and molecular biology practical classes.” Biochemistry And Molecular Biology Education, 42(1), 50­57. [16] Carroll, N. L.; Calvo, R. A. (2005). “Certified assessment artifacts for ePortfolios.” Paper presented at the 3rd International Conference on Information Technology and Applications, Sydney, Australia. [17] Cambridge, B. (2001). "Electronic portfolios as knowledge builders." Electronic portfolios: Emerging practices in student, faculty, and institutional learning, Washington, D.C., American Association for Higher Education. [18] Hutchings, P. (1998). “The course portfolio: How faculty can examine their teaching to advance practice and improve student learning.” Washington, D.C.: American Association for Higher Education. [19] Shulman L. (1998). “Teacher Portfolios: A theoretical activity," With portfolio in hand: Validating the new teacher professionalism, N. Lyons, Ed., New York City, NY: Teachers College, Columbia University. [20] Porter, C.; Cleland, J.; (1995). “The portfolio as a learning strategy.” Portsmouth, NH: Boynton/Cook Publishers, Inc. [21] Belanoff, P.; Dickson, M. (1991). Eds., Portfolios: Process and product, Portsmouth, NH: Boynton/Cook Publishers, Inc. [22] Prensky, M. (2001). “Digital Natives, Digital Immigrants.” http://www.marcprensky.com/writing/Prensky%20­%20Digital%20Natives,%20Digital%20Immigrants%20­%20Part1.pdf

Appendices

Appendix A: Google form used for notebook scoring

Appendix B: Description of assessment tool for scoring engineering notebooks

Appendix C: Introduction and exit survey questions for all students

Appendix D: Engineering notebook qualitative scoring results

Appendix E: IRB survey results

Appendix A: Google form used for notebook scoring

Appendix B: Description of assessment tool for scoring engineering notebooks

Table B1: Evidence categorization for engineering notebooks with brief description

Category Questions Brief Description

Organization

O1

Standard rules and/or procedures of engineering design record keeping were followed

Paper notebooks have particular guidelines. Use of these guidelines was investigated for hardcopy notebooks. EFolios were assumed to be complying as Digication system is web historical and university backed­up

O2 Strategy allowed for simple query­based searching

Ability to search for particular item or evidence within notebook

O3 Organization format was clear and easy to use

Could locate artifacts within reason with relative ease

Visuals V1 Initial design ideas sketched

Noted initial thoughts/design ideas or other physical manifestations of early project work

V2 [Research on] existing/possible solution images

Images posted as research executed towards project deliverables

V3 Detailed descriptive images (refined design idea)

Final design image with dimensions/scale usually (not required) produced using CAD

Design Process D1 Clear problem definition Problem definition given in text

D2 Brainstorming/idea generation Several alternative ideas given with explanation

D3 Research Researched ideas and sources given

D4 Project specifications (constraints/criteria)

Objectives, goals, constraints, criteria specifically listed and/or explained

D5 Analysis of potential solutions Explanation of solutions with pros/cons

D6 Design decision made and rationalized

Logical method used such as an evaluation matrix or rationale of design decision made

D7 Model/prototype tested Parts or whole solution tested in preliminary way

D8 Community partner satisfaction verified/checked

Verified with partner that project met needs to their satisfaction

D9 Final project documentation present

Final report and/or presentation slides with progress towards documentation given

Teamwork T1 Team meeting documentation

Agendas and minutes given; explanation of meeting discussion, decisions and action items

T2 Customer input Notebook referenced customer input or clearly denoted which information came from partner

T3 Team project documentation Actual team documentation produced is present

Reflection/ Metacognition

R1 Rationale for design decisions Introspection and explanation given throughout project progress

R2 Description of expectations/ anticipated results

Can they predict what happens and or have intentions they plan to test?

R3 Interpretation of data/strategies/ results

After testing, can they interpret and iterate on available information

R4 Ethical implications of design process/ solution

Assigned reflective piece relating to civic implications, expectation of design solution impact discussion or design iteration

R5 Social/civic considerations of design process/solution

Assigned reflective piece relating to ethical implications, expectation of design solution impact discussion or design iteration

R6 Other expansive narratives relating to project work

Do students begin to reflect on any other areas of project? Use as journal of thoughts?

R7 Student learning from course Final assigned reflective piece asking students what skills or learning the course has provided to them specifically

Appendix C: Introduction and exit survey questions for all students Both introduction and exit survey questions were completed using a scale. For the 2014­2015 electronic notebook cohort the scale was a 7­point Likert scale from ‘Strongly Disagree’ to ‘Strongly Agree’, while the pre­2014 hardcopy notebook cohort had a 4­point scale from ‘Strongly Developed’ to ‘Not Developed’. The hardcopy data was scaled from the 4­point scale to the 7­pt scale for comparison. Please rate your skills and abilities in the following areas:

01. Solving engineering homework problems 02. Finding solutions to open­ended engineering problems 03. Developing an engineering plan that will lead to a solution to an engineering problem 04. Conceptualizing a unique design to solve an engineering problem 05. Working productively with a team on an engineering project 06. Lead a small team to an engineering solution staying within project constraints and

deadlines 07. Oral communication of engineering ideas to other engineers 08. Written communication of engineering ideas to other engineers 09. Producing a device or prototype that will have an impact on your field of engineering 10. Working with members in the community to define project requirements 11. Oral communication of engineering ideas to non­engineers 12. Written communication of engineering ideas to non­engineers

Appendix D: Engineering notebook qualitative scoring results

Table D1: Full results for paper notebook qualitative scoring Organization Visuals Design Process Teamwork Reflection/Metacognition

# O1 O2 O3 V1 V2 V3 D1 D2 D3 D4 D5 D6 D7 D8 D9 T1 T2 T3 R1 R2 R3 R4 R5 R6 R7

1 1 1 1 2 2 2 0 2 1 1 0 1 1 0 1 0 1 0 1 1 0 1 0 1 1

2 1 0 1 0 0 0 1 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 0 1

3 1 0 1 0 0 0 1 1 0 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 1

4 1 0 0 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 0 1 1 1 0 0 0

5 1 0 0 1 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 0

6 1 0 0 1 1 1 1 1 0 1 1 0 0 0 0 0 1 0 0 1 0 0 1 0 0

7 0 0 0 1 1 1 1 0 1 0 1 1 1 0 0 0 0 0 1 1 0 0 0 0 1

8 1 1 1 0 1 0 1 0 1 1 0 0 0 1 0 0 1 0 1 0 0 0 0 0 1

9 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0

11 1 0 0 0 0 1 0 1 0 1 1 1 0 1 0 0 0 0 0 1 1 0 0 0 1

12 1 0 0 1 1 1 0 1 1 0 0 1 1 0 0 0 1 0 0 1 1 0 0 1 1

13 1 0 1 1 2 2 1 1 1 1 0 1 0 0 0 0 1 0 1 1 0 1 1 1 1

14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

16 0 0 0 2 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

17 1 0 0 0 0 0 0 1 0 1 0 1 1 1 0 0 1 0 0 1 0 0 0 1 0

18 1 0 1 1 1 2 0 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0

19 1 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0

20 0 0 1 1 0 0 0 0 0 0 0 1 1 1 0 0 1 0 0 0 0 1 0 0 0

21 0 0 0 1 0 0 0 1 0 0 0 1 1 0 0 0 1 0 0 1 1 0 0 0 1

22 1 1 1 1 1 1 0 1 1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 1

Table D2: Full results for electronic notebook qualitative scoring

Organization Visuals Design Process Teamwork Reflection/Metacognition

# O1 O2 O3 V1 V2 V3 D1 D2 D3 D4 D5 D6 D7 D8 D9 T1 T2 T3 R1 R2 R3 R4 R5 R6 R7

E1 0 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1

E2 0 1 1 0 1 0 0 1 1 0 1 1 1 0 1 0 1 1 1 1 1 1 1 1 2

E3 0 2 2 2 2 2 1 1 2 1 1 1 1 1 2 1 1 1 1 1 1 1 1 2 2

E5 0 1 1 1 2 1 1 1 1 1 1 1 2 1 2 2 1 1 1 1 1 1 1 1 2

E6 0 1 1 1 0 1 1 1 0 1 0 1 1 0 1 1 1 1 1 1 0 1 1 1 2

E7 0 1 1 0 1 1 0 0 1 0 1 0 0 0 1 1 1 1 1 1 1 1 1 0 1

E8 0 1 1 0 0 0 1 1 1 1 1 1 1 0 2 1 1 1 1 0 0 1 1 1 1

E9 0 1 1 1 0 1 1 0 2 0 1 1 1 1 1 1 1 1 0 0 0 1 1 1 1

E10 0 1 1 1 1 2 1 1 1 1 1 1 2 1 2 1 1 1 1 1 1 1 1 1 1

E11 0 1 1 0 0 0 1 1 1 1 1 1 1 0 1 1 2 0 1 1 0 1 1 1 1

E12 0 2 1 1 1 1 1 1 0 0 1 1 1 1 2 1 0 1 0 0 0 0 1 0 1

E13 0 2 2 1 0 1 1 1 1 2 1 1 1 0 1 1 1 1 1 1 1 0 1 1 2

E14 0 1 1 0 0 0 1 0 0 1 0 0 0 0 0 0 1 1 0 0 0 1 1 0 1

E15 0 1 1 1 2 1 1 1 0 1 1 1 1 1 2 0 1 2 1 1 1 1 1 1 1

E16 0 1 1 0 0 2 1 0 0 1 0 1 2 0 1 1 1 1 0 0 1 0 1 0 1

E17 0 1 1 0 0 1 1 0 1 1 0 1 1 0 1 0 1 1 1 0 1 0 1 1 1

E18 0 1 1 0 0 0 0 0 0 0 0 1 1 1 1 0 0 1 0 0 0 1 1 0 1

E19 0 1 1 0 0 0 0 0 0 1 0 1 1 0 1 0 1 1 0 0 1 1 1 1 1

E20 0 1 2 1 0 1 0 0 1 1 1 1 1 0 1 0 0 1 1 0 1 0 1 0 1

E21 0 1 1 0 1 2 1 1 1 1 1 1 1 0 2 1 1 2 0 1 1 0 1 2 1

Appendix E: IRB survey results Hardcopy notebook data is presented for S2011, F2011, S2012, and W2013 quarters. Electronic notebook data is presented for W2015, S2015, and F2015. Table E1: IRB survey results. Numbers are reported in percent of students (‘#’ indicates total) who completed both the

introduction and exit surveys.

Quarter Sample Size Code

Question

1 2 3 4 5 6 7 8 9 10 11 12

S2011 8 strong loss 0% 0% 13% 0% 0% 25% 0% 25% 0% 0% 13% 0%

slight loss / no change 100% 75% 50% 25% 50% 50% 63% 38% 75% 25% 50% 63%

gain 0% 25% 25% 63% 50% 25% 38% 38% 13% 38% 38% 38%

strong gain 0% 0% 13% 13% 0% 0% 0% 0% 13% 38% 0% 0%

F2011 5 strong loss 0% 20% 20% 0% 20% 0% 20% 0% 20% 20% 20% 40%

slight loss / no change 40% 40% 20% 40% 60% 40% 0% 80% 60% 40% 60% 40%

gain 40% 20% 40% 40% 0% 40% 60% 0% 0% 20% 0% 0%

strong gain 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20%

S2012 4 strong loss 50% 0% 0% 0% 0% 0% 0% 25% 0% 0% 0% 25%

slight loss / no change 50% 100% 100% 75% 100% 50% 75% 75% 75% 75% 100% 75%

gain 0% 0% 0% 25% 0% 50% 25% 0% 25% 25% 0% 0%

strong gain 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

W2013 3 strong loss 33% 33% 0% 33% 0% 0% 0% 0% 0% 0% 0% 33%

slight loss / no change 33% 67% 100% 67% 100% 67% 33% 67% 33% 33% 67% 0%

gain 33% 0% 0% 0% 0% 33% 67% 33% 33% 67% 0% 33%

strong gain 0% 0% 0% 0% 0% 0% 0% 0% 33% 0% 33% 33%

W2015 7 strong loss 0% 0% 0% 0% 0% 14% 0% 0% 0% 0% 14% 0%

slight loss / no change 29% 14% 29% 29% 43% 29% 14% 0% 29% 0% 43% 71%

gain 71% 71% 14% 43% 43% 57% 86% 100% 14% 57% 43% 29%

strong gain 0% 14% 57% 29% 14% 0% 0% 0% 57% 43% 0% 0%

S2015 10 strong loss 0% 0% 0% 0% 0% 0% 10% 0% 0% 0% 0% 0%

slight loss / no change 30% 20% 30% 30% 70% 50% 40% 60% 20% 10% 20% 60%

gain 60% 60% 50% 50% 30% 40% 50% 40% 50% 60% 80% 40%

strong gain 10% 20% 20% 20% 0% 10% 0% 0% 30% 30% 0% 0%

F2015 2 strong loss 0% 0% 0% 0% 0% 50% 0% 0% 0% 0% 0% 0%

slight loss / no change 0% 0% 0% 0% 100% 50% 100% 100% 100% 100% 100% 100%

gain 100% 100% 100% 100% 0% 0% 0% 0% 0% 0% 0% 0%

strong gain 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

Figure E1: IRB survey results from paper engineering notebooks. The results are reported in chronological order for

each question S2011, F2011, S2012, and W2013 from left to right for each question.

Figure E2: IRB survey results from electronic engineering notebooks. Results are reported in chronological order

W2015 to F2015 from left to right for each question.