course analysis (2010/11 period 1) dd2380 artificial intelligence, 6hp

Course analysis (2010/11 period 1)DD2380 Artificial Intelligence, 6hp

Course responsible: Patric JensfeltLecturers: Patric Jensfelt (75%), Danica Kragic (25%)Course assistants: Javier Romero and Xavi GratalNumber of lectures: 12 lectures (24 hours)Registered students: 178 according to VIS (185 completed some part of it according to my lists)"Prestationsgrad": 94%"Examinationgrad": 91%

Course material:• Book: Artificial Intelligence: A Modern Approach (by Stuart J. Russell and

Peter Norvig, Prentice Hall• Lecture notes: Available for download from BILDA

Examination requirements:• LAB1 3hp

◦ 3 homework assignments consisting of three parts each▪ PartA: completed individually▪ PartB: can be completed in groups of two students. Requires an

implementation▪ PartC: review someone's part B (handed in a few days after PartB is

handed in)• PRO1 3hp

◦ Project completed in groups of 4 students◦ Solve a AI problem◦ Write a report◦ Present the work orally◦ Review/assess another group's report, work and presentation.

Summary of impressions:Overall I think that the course was well received and that students found it interesting (57.6% interesting and 33.1% very interesting). The new homework assignments caused some irritation which need to be addressed. The introduction of the milestone for the project seemed to have worked quite well.

Grading: Each of the three homework assignments and the project gave a grade A-F. The final grade is calculated as the mean grade on the results rounding to the closest grade down.

Relation to the previous years:

We were taken aback by the number of students attending the course. We have counted on 130-140 students but we got almost 200 students. There were 210 participants in BILDA. This meant that the lecture rooms where not big enough to host everyone. About a third of the lectures therefore had to be given twice so that everyone had a change to attend. Furthermore, one of the high priority changes from last year was to be able to provide prompt feedback on the homework assignments. This clashed heavily with the massive number of students. For more on this see below.

Prompt feedback: An important ingredient in efficient learning is prompt feedback. Previous years the delay was way too long. We wanted to keep the feedback sessions that were started last year but wanted to be able to be able to combine that with handing back corrected assignments. The idea with the feedback sessions were that shortly (typically a day or so) after the hand in of PartC for an assignment part of a lecture was used to provide feedback on the homework. To be able to correct the homework assignments that fast with this many students we needed to automate part of the correction work. We (Javier) built a system that would allow part of the assignments to be automatically corrected (in true AI spirit). This worked quite well although as expected with something that is used for the first time it was not perfect as pointed out by the

Lectures: A bit less material was presented during the lectures this year to give room for a bit more time for homework assignment feedback and discussions about the project. We let it take the time it took.

Project: We used Sokoban as the project problem again. The year before we noticed that people started very late. This year we introduced a milestone relatively early on in the course for the project to make people start thinking about the problem and flush out questions before it was too late as happened quite a lot last year.

Planned changes:• The most important change for next year is to make sure that the

homework assignments are made such that they are better adapted to fit an semi-autonomous correction process. Although the correction process was criticized we still believe that it the benefits of prompt feedback make it worth to see if we can adjust the assignments in a way that is better adapted to automatic correction while not being multiple-choice questions or just checking the answer of some long derivation. Some suggested using Kattis which is used for some of the programming courses which we will probably not do since it is not a programming course but we will ask for partial results so that one can assess the thought process a bit better.

• We will make the project count for more in the final grade.

Survey: Course Evaluation

Status: closed

Date: 2011-04-27 22:34:59

Group: Activated participants (DD2380 ArtificialIntelligence ai10)

Answered by: 119(211) (56%)

How difficult was this course?

number distribution answer choice

7 5,9% Easy

82 68,9% Medium

31 26,1% Difficult

119 has answered of 211 (56%)Maximum number of choices: 1

Was the aim of the course clear from the beginning?


3 2,5% Very unclear

20 16,8% Not so clear

85 71,4% Yes

12 10,1% Yes crystal clear


Survey results

Was the course interesting and meaningful?


3 2,5% Completely uninteresting

9 7,6% Not so interesting

68 57,6% Yes

39 33,1% Yes very interesting


Prerequisites for the course were courses in numerics, statistics and an introductory course in computer science. Doyou think that your prior knowledge on these topics was enough to follow the course?


7 5,9% No

22 18,5% Not sure

91 76,5% Yes


Comment:

- Involves a LOT of coding.- if i had done the matstatcourse it would have been easier so yes :)- However, how can you say that an "Introductory course" in Comp Science should be enough to pass the course?There are people having taken EL2310 and still can not manage the course... There should be two different projects forthe course. I propose something like this: The end result should be the same, i.e. Sokoban solver. If you want an A ora B should do it from scratch just as we did now. But then there should also be one option where C is maximum grade.If a person chooses this "track" they should be given a shell just as in Applied Estimation course with the EKF andParticle Filter lab.- I actually felt that programming skills would be enough to take this course.- I did not know nearly as much statistics as I ought to. That's why I got an A in every part of the course, except theMarkov-homework - where I got an E.

- Two of our group members did not know the first thing about programming despite fulfilling prereqs.- I thing data structures id also very important for getting good grade in this course.- As the project is purely programming the presequisites should have included an advanced programming course aswell.- My prior knowledge about programming was a little too basic to fulfill all assignments well, but I still got at least a Cfor each assignment.- I have never taken statistics, so maybe it would have helped me.- I think the project was the part where my prerequisites were on the low side since the other three in the group werequite skilled java programmers.- Some course in algorithms would had been very useful as a requirement (personaly i have taken several algocourses, and would have had a hard time without some of them)- The math for Homework 3 was difficult- i did the course without statistics, made HW3 hard but otherwise i maneged without it- If I hadn't taken the course Algorithm and Complexity I wouldn't have been able to follow along. And that course isn'tmarked in the prerequisites.- My prior knowledge was definitely enough to follow the course but it wasn't always enough to do the homework -especially the implementation parts.- The markov chains were a little hard to grasp at first but it was feasible.- Homework 3 was very Markov Theory dependant. I personally wasn't very good at that in Statistics so I didn't dovery well on it in this course either... felt a bit "unfair" perhaps.- For computer science basic topics, it was enough. But for probability and statistics parts, it was not enough for me.- more

Do you find course book by Russell and Norvig suitable?


15 12,6% Did not get the book

11 9,2% Got the book but did not read it

10 8,4% No

54 45,4% Yes

30 25,2% Yes it is great


Comment:

- The book seemed okay for reading about things, but when it came to looking up specific things, it was difficult tonavigate, even with a torrented, searchable .pdf version of it.- Its a very good book.- It's too much for a quarter of a year.- Did not need to.- I found it useful but not as much as I thought.- Found the book on google books, used it to find the answers to the homeworks.- But it is quite expensive!- I downloaded it off Pirate Bay. Very nice to have a searchable book for free if you feel like stealing the book..... :)- Good, though way too big too be read and re-read properly. Once I read a chapter, there was *no way* I could re-read it.- All homework questions were copied from the book with minor alterations, "smart" students realized that you can'tctrl+f in a book, but you can on google books.

- It's not always crystal clear but after have read it a few times you get a hang of it.- -- It nicely serves the purposes of the course.- It is a good book, however it is hard to keep up with the classes. I red only the pieces that were nececarry to answerthe homework questions.- I wish I had had time to read it though, because it seemed like it might have helped!- Not too useful, homeworks were based on the books- Interesting, even fun to read- Only read small parts for answering homework questions.- Somewhat hard to look stuff up in when there was such a difference between the two printings. But a pdf versionhelped alot when you wanted to find specific information fast.- I only used the book for the homeworks.- it's a great aid when you need a second opinion or a diffrent perspective on a topic.- Unfortunately I didn't have much time to read it..- Used it only for looking up the first questions at each homework, was good for that though- There was so much to read. I didn't have time to read all the pages recommended.- The chapter on statistics were a bit over my head and used strange notations otherwise it was excellent and i almostread all of it.- -- Actually very, very good.- No need to buy it. I had an e-book of it and only read relevant parts to answer homework questions.- It was really good actually. Better course material is tough to find.- Due to obligations in several really intense courses, i didn't really have time to read all the chapters of the book,which was a shame...- The fact that you HAVE to get the book to complete the homeworks is a really bad idea. Homeworks should beknowledge based, not based on your efforts following references in a single book.- The chapters on statistical methods were insufficient and the meat and potatoes of the practical elements of thecourse might as well have been handled by Cormen, Leiserson or a similar book on algorithms. Perhaps there is asmaller book that covers the AI specific parts of the course that could be combined with a general algorithm text?- Used it on the homeworks to nice effect, was useful and I'll probably spend some more time reading through thewhole of it.- I only used it a couple of times as a reference, e.g. for the Min-conflicts algorithm.

How many lectures did you attend?


28 23,5% Less than 30%

22 18,5% 30-60%

25 21% 60-80%

45 37,8% more than 80%


What do you think about the lectures? Were the topics clearly presented?


13 10,9% Did not attend

8 6,7% Bad

41 34,5% Ok

48 40,3% Good

10 8,4% Very good


Comments on the lectures

49 has answered of 211 (23%)

Comment:

- The lectures with the main lecturer were interesting and enjoyable. The lectures with the two secondary lecturers...not so much.- Good ones.- Sorry, but did not find them meaningful at all.- they were awesome, good mix of highlevel stuff and lowlevel maths etc- The should be more focused on the homeworks so that you actually can solve the homeworks based on the lectures.- Very good lectures!- N/A- "Question 8 is not answered"- I attended the first lectures with great interest, then my interest dropped off towards the end of the course.- I believe that this course is too ambitious; AI is too wide a topic to be covered in so little time.- Could really use more focus on the assignments. And I honestly didn't find the lecture room once, which makes it thefirst time in 4 years. The students from D really like to hang out around the D and E houses.- The lectures didn't go into much depth; reading the book was necessary in order to do the homeworks. That's not abad thing per se, but it should maybe be made clearer at the beginning.- Really like the lectures.- Great. Theory, some examples and implementations, even a few videos. Although the topic is veeery broad, you hadyour limitations: not to violate "domains" of other courses like neural networks, machine learning, image analysis, etc.- Good- -- Most were good. Some could have been better structured, however.- Interesting and fun.- Did not attend.- Did not go to most of the lectures because of the lack of seats, wanted to attend more lectures.- The lectures were good explained and the teachers listened all our questions.- wasn´t critical to go to the lecture to make the course. You didn't learn that much.- The lectures were good. Only the ones on HMMs were a little unclear.- The lectures were clear and understandable.- Quite a lot of examples were brought up that made the concept/algorithm quite understandable instead of tiringpseudocode of the same process. Great with inspiring movies and topics on current robotics and AI research, made meeven more enthusiastic of the subject.- The lectures about HMMs were really confusing and badly structured.- Liked your sense of humor. Overall the lectures gave a lot of interesting stuff.- I think that lectures can be improved by adding more details to the slides and give an simple example (withexplanation) for each important technique. Pro's and con's of each method on top of their applications in practice will

make the lecture more valuable.- As I spoke to both Patric and Javier at one time at albanova, if you don't use a good speaking technique it's all fornot as no one will hear you, or see the text on the board. Also remember to repeat questions and answers from theaudience. You can't count on the other people in the audience hearing what's being said just 'cause you do. And also,stop once in a while and let see if people are following along with a question of some sort, or something similar.Sometimes you don't have the time to ask a question 'cause the teacher are going so fast you need to concentrate onfollowing into the next step, so you don't have time to stop the lecturer and ask the question.- Generally very good and interesting.- Patrick's way of presenting is very nice! Danica's lectures didn't really fit into the cours' style. Javier's way was a littlebit confusing and his English was often hard to follow.- Sometimes the lecturer went trough the material too quickly, and sometimes too slowly. Topics like BFS and DFS areeasy and don't require a lot of time. However, stuff that is completely new like planning algorithms require some moretime.- The lectures were usually good but could have been a bit more hands-on to support the implementation of thehomeworks.- There were a fast overview to the book chapters. Therefore it is better to read the book after attending the lecture. Ifhad read the book before the lecture, I would have found the lecture useless.- Didnt attend many lectures since there were to many people in to small rooms, and the lectures never really hadenough depth, to much material per lecture.- 1) Obviously a lot was tried to be covered in a small span of time. The lessons seemed very fast and there weremany chapters that went unread because of this. 2) We had to obviously concentrate on those parts of the book whichcorresponded to the assignments. 3) I personally would have liked to have similar assignments for every chaptercovered as part of the course - this would make my foundations rock solid! 4) Solution - make the course longer :) andfor more credits.- Didn't give much the book couldn't give.- It felt unnecessary to have any lectures since you needed to read the book or google stuff anyways. The lectures feltlike a copy of the books own headings and pictures. So they weren't really helpful.- A bit dull and might be a bit fast. But doing 60-80 pages in the book over the course of two hours is hard.- I think I did not learn as much as I should during the lectures. Slides were ok but just following the slides fastly is notvery helpful. You can easily lost your motivation and fall asleep by watching the slides. The topics are skipped fastlysometimes. I prefer deeper covering of subjects during lectures, not just overall ideas. Moreover, for probabilisticreasoning lectures I found them very advance and I got nearly nothing during the lectures.- Patric was a really good lecturer. It was always really interesting and he made it easy to understand. Javier andDanica were okay too. Javier seems like a really cool guy.- Lectures was not clear enough and was just highlight of some terms. Lectures by Patric was okay but unclear;Danica's was great; and Javier's lecture n planning was almost waste of time.- erm, they were ok. However, when the examples are taken straight from the book, why bother show up? Make yourown examples, lazy!- They were mostly interesting, however I did not attend that many.- the lectures was good but I didn't feel they helped me much with the homeworks- Alot of repetition, wasting time. Going through the whole homework question for question is very tedious andunnecessary, as was pointed out in last year's course evaluation.- But some parts were really confuing - the HMM part.- Working on the problems by myself gave much more than any of the lectures.- Too small rooms in the beginning of the course.

Did you read the relevant book chapters before they were presented at the lectures?


59 49,6% Never

35 29,4% Rarely

24 20,2% Sometimes

2 1,7% Always


Did you think that the homeworks helped you learn the material in the course?


5 4,2% They were a complete waste of time

13 10,9% Not so much

60 50,4% Yes

41 34,5% Yes very much


Comment:

- I will elaborate below.- Common sense and a little google was enough to do the homeworks.- maybe telling what algorithms/techniques are supposed to be used or giving some hints for better results would benice. this way you implement and learn something that is good quality and useful instead of a bad solution..- the third hw could have been more like the others in terms of coding imo, that was fun- But they where very difficult sometimes.- Without it I would probably have had everything to read at the end.- Multiple choice questions is basically "look for keyword".- Homeworks are awesome.- I was directly looking for the answers in the book...- See earlier question. When you can ctrl+f an answer to a question, you generally won't learn anything. Suggestskipping questions/theory-homework for more coding assignments. A real winner would be integrating the course withkattis... why isn't it already?- Very good homeworks - Except that the assigments were constantly modified.- ...but even with the testparser the automatic grading system was hit-and-miss and that's not good enough at thislevel.- They where however confusing at times And "updating" the homework, the day before deadline is not good. If youwhere already done with it you would have to go through it again.- All tasks were too "isolated" from each other. Find algorithm in a book - implement it. I can barely remember whatthey were about! May be it could be better if part B was more "complex", i.e. consisted of several parts, was somehow"meaningful", not just another exercise.- Although there was way too little time to get decent grades on all of them (and the project too!), which wasdisappointing, as it was a very interesting course and I would had liked to do more.- the questions were bad formulated an most of the time was spent to understand the questions or to find were thecorrections to the questions were held. You didn't have to read the book too answer the questions, it was only tosearch for the key words since you only needed to pick an answer, not understand it. The other assignments werequite easy when you understood them and didn't cover much of the course material.- The homework "forced" me to read in the book.- The homeworks were big, but a good way to learn the material.- They helped to review the lecture topics while pointing out what I didn't understand yet.- I think all courses should be homework-based without exams since it forces us students to start studying from the

very first week and the knowledge gained is more long-lived.- The homeworks made the course; i learned alot from them.- unbalanced homeworks, hw1 very easy, hw2 normal, hw3 really hard- They were very large in the matter of time. Please do remember that the project AND the homework needs to bedone, and this course only have 1/3 or 1/4 of the total time of a week (40 laboring hours) as we have other courses toattend to.- They helped me learn mostly by giving an incentive to read the book, but also a bit on their own.- They forced me to actually learn and implement the material. Which is good.- In theory, the homeworks were fine. However, there were a lot of misunderstandings from badly phrased questions.Bad choice of words perhaps, and the automatic homework correcting system bugged out sometimes.- I think the theoretical parts were more helpful and implementation parts were not useful at all.- They were quite tough, and I hated some of the questions, but in the end you really learned a lot from it, and it wasdefinitely necessary considering the limited knowledge requirements for Sokoban.- at least some of it- I had a PDF version of the book. CTRL+F on any of the question would give me the answer without ever learning thematerial.- See above.- Very much, since they were given as part of the examination. Possibly some more time could have been given to dothe homeworks.- They were harder than I thought, part B took a lot of time.- Actually, they were the only thing (except from the project) that made me learn stuff in the course, since I only gotthe book as a PDF and therefore couldn't read the theory in a comfortable way.

How do you think is the balance between part A ( theoretical questions solvable without programming) and part B(implementation questions) in the homeworks?


2 1,7% There should be no partB

15 12,7% There should be more partA and less partB

78 66,1% It's fine as it is

19 16,1% There should be more partB and less partA

4 3,4% There should be no partA


Comment:

- I will elaborate below.- The checkbox questions of part A are really bad. They covered too large areas spread out in too large areas of thebook. This lead to a solving method where you could use Google books to search for the answers more often thantrying to understand the material.- Less coding and more of reading and working out.- If more B, put more emphasis on the report and being clear about the standard expected.- Part A was too easy. I guessed on the first question, since it only gave 14 points, the easy questions (2 and 3) gaveso much points that only doing them was enough for an OK grade. Part B was more fun and i learned more there.- Simply make it so that the implementation part encourages reading in the book, provide keywords to interestingtopics. If a student chooses to read the relevant chapter and look up the keywords it should be helpfull and relevant tothe implementation.- Homework took a lot of time if you wanted to do well. Part B varied a bit too much in difficulty between the different

assignments, I think.- I think Questions 2 and 3 in the A part was the part of each homework that helped me improve my understanding ofthe course topics the most. There was just too much to implement while studying three other courses at the sametime.- I allways tried to do part B but I never got anything usable in time.- If the course is about "AI-review" may be students should review more material/articles instead of implementingindividual examples.- The idea is good, but it should be reel questions without options if part A should give anything.- Part A is necessary for people who cannot program well.- theory and practical was in balance.- Part B was very time consuming for me.- Don't change this, it's great as is! Perfect with a part A that can be solved entirely without programming. For highergrades one have to solve B and C. Even if one does not hand-in a part B, it's great to take a look and analyze part Bcode from another student.- Great setup- I want to say that the ratio is fine. So my answer will be "There should be less of both A and B as they took too muchtime all together"- The problem with part b and the "calculation" question is that its really hard to figure out if your approach is evennear correct before the grading, and the fact questions didnt really do that much since they were quite ambigous andmainly trained your "looking-up-in-the-book/googling" skills. Rather than testing actual knownledge (but they still muchbetter than traditional sitting in a room for 5 hours exams).- -- Because the project became very code-oriented, I think it would be better to have a bit less programming in thehomeworks.- We didn't learn much about how to implement part B's. I prefer the whole HW to be written questions but no code,since the code do not teach you a lot but the questions did. We will do implementation in the project anyway.- In all honesty, for some of the HW3 questions it wasn't really feasible to answer them without programming (at leastnot without taking a lot of time). Not having to motivate the answer was pretty nice though, since I hate that ;).Otherwise, I found the system to be good.- Part A was easy, but the learning per time spent is really bad. Good if you want the students pass the course easilywithout learning anything. Also the questions' formulations are very poor and ambiguous. Other answers could be rightdepending on which theory is used (which the question doesn't mention).- While it's fine that part B was there, there was a marked difference in difficulty between A and B in some cases, thatweren't all related to the points gained. Part B on homework 3 also had too much theory/had too high theoreticaltreshold for an implementation part.

Did reviewing other peoples homework help you learn?


22 18,6% Not at all

60 50,8% A little

24 20,3% Yes

12 10,2% Yes it helped me very much


Comment:

- I will elaborate below.- I think this part is very useful for us, because we can check other ways to solve the problem and it helped us for next

time.- There are many sloppy students...- Did not review any other homework.- It sure gives some tips about coding, but generally it's not *that* interesting.- Reading someone elses' code is rarely helpfull. In the same way that "reading" math will never help you with math,reading someones code will not help you code... not by a lot anyway.- Not at all, but that's normal given my prior experience with implementation... it helped me correct a mistake I madein the last assignment, but that was a typo (that led to a bug hence our results differed).- It gave anther aspect on how you can solve the same assignment but with different eyes.- Even if it does not help a lot, it's still useful to learn to read other person's code.- It teached me a better coding style.- It gave me a new perspective.- Well, the reviews I had received were quite similar to my own work/code, so I found little to nothing new. May besome other guys had done a really great job, but I wasn't lucky to review them. Besides not all of the homeworkssuggested some "creativity".- The most important time when we learned was doing part B- Because I'm not that good at programming reviewing other peoples work was hard sometimes and I did not alwaysunderstand what they were doing. I think for people who are better at programming it would help.- The criterions for the reviewing part were vague.- We only reviewed one homework which wasn't very well done.- It would have been better if we had gone through the homeworks in class BEFORE we reviewed each other'shomeworks.- As I attend the Civilengineer and Teaching program I can tell you that peer-review is widly supported amongstpedagogs to be a really good method for learning and understanding. Please make sure you keep this part in thefuture, it is well needed and adds a lot of understanding.- It actually helped surprisingly much for the little time spend on it. But, in absolute terms not a lot. It's a good thing,definitely.- The requirement of review was not so clear to follow.- The payoff was not worth the effort though.- Sadly alot of the time the other guy had to inte exactly the same way as my self or didnt do the entire task. But itwas still good and informative, especially when you hadnt completeded the task yourself.- -- Some code was hard to understand, because of the lack of comments.- If you didn't do partB, maybe you can learn from others. But I think there should be correct answers announcedbefore we judge other's work.- You always get something from looking at a different solution, but I can't say I learned more about AI fromreviewing.- It was really weird reviewing other peoples homework when you weren't even sure you knew the right answers.- It helped me in some cases, where people had solved stuff. I suspect I'd have learned as much by getting an actualsolution that I know was correct however.- It's always interesting, though, to se how others have (potentially) solved the problem.- I had more hassle with trying to get the other people's code to run to learn much... Maybe some requirement thatthere should be a compileAndRun.sh script like for the project would help this.

What did you think about the feedback session when you were presented with the solution to the homeworks at thefirst lecture after the deadline for the homework?


16 14% Not good

12 10,5% Less than good

61 53,5% Ok

17 14,9% Good

8 7% Very good


Comment:

- To be blunt, it was completely uninteresting.- Don't know, did not attend, but can't check that.- Good when you needed them, a bit boring if you had the answers correct.- Did not attend- Ok, but not necessary! Online solution is enough!- sometimes the tempo was to fast hablas muy rapido ;)- It takes to much time from valuable lectures. Really irritating. Javier should be better prepared for these occasionsso that he can say just as much but in lesser time.- Did not attend.- Only attended the first feedback session- Too many students whining and trying to show everyone how smart they are....- Waste of time. Now, lots of people enjoy such sessions, I'm one of them. But they're not really helping with anything.- I didn't really care much, though.- boring.- -- I think it's necessary to have all things explained afterwards. Probably, it could have been done in shorter/separatetime or partly via e-mail.- It would be nice if the answers were provided earlier than the feedback session, so that you know what you didwrong in advance.- It is maybe better to spend less time on answering multiplechoice questions. Or maybe no time!- It was alright although hard for Javier to get people involved. Perhaps it better to simply present the solutions andurge people to protest and ask questions.- only attended the first one.- Didn't attend, missing choice!- Didn't give me any help- Did not attend.- did not attend- Solutions should be published before the feedback session so less time would be spent on trivial questions and moretime spent on more serious questions.- This part too is wildly supported amongst pedagogs to be good for learning. You as a teacher want to present theresponse to a students suggestion as fast as possible. Even the next session is consider "too long a time". If you wantto improve in this matter I suggest reading up on B.F. Skinners work.- It would be good to have the feedback session before handing in part C, because in the case you were not able tosolve a specific task, it is hard to do a good review of this task, without knowing the correct solution to it.- I don't think it's important enough to take time away from regular lectures. For instance, review of hw 1 took muchtime away from the planning lecutre, so Javier hurried through it, and I don't think anyone learned anything aboutplanning that day. Also, most people know most of the answers by this time, so for most of us it's just boring to listento the answers, and made worse when ONE person does not understand and drags the whole process out even furthur.- Only went to the first one.- Didn't attend.- -- There were some unclear questions, and multiple choice questions with more than one possible answers according tothe book. But they are also not accepted during feedback sessions. Part B's are not explained very well, which I thinkalso not thought during lectures deeply.- I think there should be more reasoning why the answers are what they are. Often Javier just went "So what do youthink the correct answer is? Oh, it's C. Next question."- only attended the first one, never returned for them- Waste of time. Knowing what score you got on each question is enough feedback. Especially for part A. The onlyreason one fails is due to the ambiguities in the questions, not because one didn'ät understand.- i didnt come to university yet at that time- The feedback sessions gave me absolutely nothing. I'd much rather move on and learn things than listen to peoplebitch and moan about trivialities such as grades.

Part of the homework was corrected automatically, what do you think about that?


22 18,5% Not good


34 28,6% Ok

22 18,5% Good

21 17,6% Very good


Comment:

- Too much ambiguity in the questions, that can be improved- I will elaborate below.- Errors were corrected by hand. Teacher was open for discussion and could be convinced of other solutions.- Sometimes it was difficault to understand how you were supposed to write the answer and how you should round offthe numbers so it would fit the auto-corrector. And deducting points for minor syntax errors(for your program) andstuff like that feels like its just lazyness from your side.- Worked surprisingly well- Success of automatic correction heavily depends on an exact problem formulation in the assignments - this wasunfortunately not granted for all homeworks- I would prefer a webform where you can enter the values and submit the homework, this was too tricky, and easy tocreate errors.- i don't know how the automatic correction works, it may be good...- to get minuspionts for not formatting right is pretty damn bad.. the points should reflect knowledge not paperwork!- The purpose of this is that the course administration is to little compared to the number of students I suppose. But ifthe number of students increase that means that you get more money from the government, and what do you do withthese money? You keep the course administration intact so that you can spend the extra money on other things thatteaching...That is really low from your part I must say! Also, the course feels very sloppy and unprofessional whenhomeworks are sent from a gmal address. What is wrong with splitting the course into, lets say, 10 groups, andpresent all 10 different homework types on Bilda in a professional PDF?- Since i got point deduction from it, i think it should be more precise how to answer for example the empty set, whichwas the exact same answer as if you dont have an answer.- A bit annoying when the correction was wrong and we had to contact the TA to correct it, happened to me twice Ithink.- To make larger questions (15-30p),that should be about deeper understanding, hit or miss seems... wrong. Getting abad input combined with less than crystal clear instructions could result in zero points, despite having a good grasp ofthe underlying theory. Either go KATTIS (or as with the sokoban server) and let people try repeatedly to get it right orhave a much larger human oversight in the correction.- As long as it's binary questions (true/false) it's ok.- Use kattis.- Problem was, when I thought the system had wronged me, actually, it was just less effort not to complain rather thancomplain, which may make for a suboptimal end result, but it was OK, a lot less messy than I had expected.- It saves the teachers a lot of time and if the auto-correcter get it wrong the student can simply complain to teacher.It's not optimal (since the students not always know if they deserve 9 or 10 or 11 for a half solved assignment so theymight just accept the result even thou it can contain slight errors with points)- It worked, but it has the obvious drawback of the answer being the important part, not the reasoning. Also, ifautomatically corrected, it would have been nice to have it as a web form (like this review) where one could just chosealternatives/fill in numbers. It's a bit awkward and error prone to manually create text files.- In classes this big it is probably the best way of doing it. It was fair, but a little messed up.- Automatic correction CAN be good. But there were too many errors.- Sometimes unclear questions and computer corrected homework doesent blend too well.

- Great! That's how it should be done considering the amount of students on the course! May be, it could be fullyautomatized. I mean, Part B submitted as a mini Soko-proj: compile & run scripts, several scenarios to testperformance and leave "code autopsy" and report for peer-reviewers. May be peer-reviewing should be done not for 1but for several works, so you could have several reviews on each work. Also it could be done in groups, e.g. projectgroups. Still, that depends on the part B: is it that interesting and challenging to be reviewed?- As long as there is room to get a second opinion if one disagrees with the score, it's ok.- It was totally unclear which policy was used rating the homework. I could never be shure about typos having a largeinfluence on the grade.- If there should have been clearer assignments and if there was a form too fill (too get response that you hadanswered and that you answered in a correct way directly) if would have been ok.- I can imagine that this works better for correcting with so many people taking the course.- It's fine, as long as the correct answers are used.- a lot of screw-ups and uncertainties revolving around the homeworks in general. i think we averaged 3 correctionand update mails per homework. if a automated process is chosen, the assignments should be crystal clear in everyaspect.- There were some errors in it but nothing catastrophic and if anyone thought there were errors they could raise theconcern and Javier listened. To have the HW as is there's no other way than automatic corrections.- It's important that the instructions are clear and easy to understand, there can be no room for missinterpetation ormissunderstanding.- I would say it was good if it had worked out better in practice. I heard this was the first year using the system so Iguess it'll be better next year.- Use Kattis!- The automated correction was a cause for very much frustration for me due to e.g. incorrect corrections.- There are so many interpretations of each question. Also, please consider naming the files different. Me and otherstudents managed to send in the previous homeworks files.- Its obviously worse than real grading with comments about wrong answers. Not a very much worse though.- Format for answering was unclear and problematic to use. The automatic correction also meant that the actualassingments couldn't be updated after the first version, requiring that you double check each question with thecorrected version with the wrong assignment order.- Good idea, please make it work better next time. It was somewhat embarrasing that we attended a course in AI andthe correction program couldn't figure anything out if you did the slightest variation. And make sure your questions arewell defined and not ambiguous as for example the elevator problem and node expansion problem. But the idea isreally good, please continue you work on it.- 1. There were lots of corrections, which caused further confusion. 2. The corrections gave no explanations what wentwrong, and gave no points for partially correct solutions. All in all, I have no idea why I did the errors I did and no ideahow to make it right. 3. If one takes into account all the problems, confusion and the learning curve (not to mentionhow we learned a lot less since we didn't get any comments) I doubt any time was saved using automated corrections.- The automatic correction of the homework creates in my opinion more work than it actually saves. It is a little bitunhandy and complicated to provide the answers machine readable in Python-Code.- The only problem I have with it is that it's painstaking to make sure everything is perfectly correct before submittingit.- Great job!- The idea is good but the execution was bad.- The problem is that it doesnt take into account how you solve problems, i know that some people solved theproblems in quite exotic ways and still got the correct answer, some form of explanation of how the result wasgenerated should be mandatory and taken into account when grading.- I can see the need to save time since we were many students on this course. Perhaps if there were more teachersassistants you could have combined automatic correction and assistant overview. Perhaps a person can understandreasoning behind answers that a automatic computer correction system can't.- Automatic correction is a sadistic way to set points since it punishes you severly for minor formatting errors thatreally have nothing to do with the assignment.- When it worked, it was good.- I swear that I got no points from a correct solution (pruning when there's a tie). Still got an A though, so I didn't bringit up.- These type of questions that are corrected by computer are not representative of knowledge in the part. E.g. Me andmy pair wrote a script to solve part A and generate python script, but our grade differs 15 points.- Error-prone, weird and unintuitive. Dependent on whitespace, wtf dudes!- The fact that it was corrected automatically was good, however it was to much effort to put into it. And the fact thatpoints were reducted for syntax errors is really bad, automatic correction is in order to decrease the workload for theteacher, not increase it for the students. To use python syntax and a home-made correction is not particullary goodeither, it would be a better idea to use one of the established methods, like for example kattis.- Making lists manually is just to ask for human errors. At least split up the variables into different sub answers toremove lists inside lists.- I dont like being penalized with 5 points if it doesnt compile, it's too much- parse.py was foul and had to be re-fixed for every homework in order to make it parse files in different directories.The implementation of the homework correction system seemed brittle to me, but the concept itself is sound.- There were many problems with that, due to the rigidity of automatical corrections. The program used to correct it

needs to be looked more carefully to avoid bugs and to make sure simple mistakes get a too harsh grading.- I guess manual correction is always the best from a student's perspective, but in this case with so many students Ican see that it might not be feasible for you.

Overall comments about the homeworks


Comment:

- Normally, a course can be measured using a one-dimensional spectrum from "bad" to "good". This course, however,was both good and terrible at the same time, forcing us to expand it into a two-dimensional spectrum. The main part ofthe course, that is, the lectures and, for the most part, the project, were really good. The homeworks, on the otherhand... oh dear lord, it burns, it burns. The entire structure of the homework is flawed beyond repair. First of all, auto-correction is inherently inferior to manually corrected homeworks. I realize that it would be an unreasonable task tocorrect that many homeworks, but auto-correction is NOT an acceptable replacement. Simply giving a true/falseanswer without being able to argue why it should be right (Whether or not it is right is irrelevant in this case.) iscompletely demotivating, not to mention annoying and soulless. With manually corrected homeworks, there's a degreeof flexibility and the possibility of motivating one's answer. Furthermore, multiple choice questions areBAAAAAAAAAAAAAAAAAAAD. (Capital letters and extra As used for emphasis.) Instead of a homework based around"Explain this concept", it became a homework of "Which of these almost identical options applies to this trivial andinsignificant aspect of this concept, the knowledge of which in no way reflects one's overall understanding of theconcept". Furthermore, the answers to some of the questions couldn't even be found in the book, despite considerablesearching and thinking. Even worse was when one knew the answer, but two of the choices were so similar that it wasimpossible, even with help from the book, to distinguish the exact meaning of the two. Also, the difficulty of thequestions varied considerably, which does NOT go well with the idea of randomly picking a set of questions for eachstudent. At one point, I got several near-impossible questions and, while reading through the complete set ofquestions, noticed that there were a lot of questions I easily knew the answer for, but of which I didn't get a single one.Yes, I realize that you want to minimize the possibility of cheating, but you essentially leave the score to chance, whichis hardly fair. As for the auto-corrected implementation portion, it too was a joke. As it's auto-corrected and only takesthe output into consideration, it has no regard for the method used to get the result. On math exams, one lists thesteps used to get the result. Even if one makes a slight mistake, the correcter can still follow the reasoning behind itand give one full score for understanding the concept. With your system, it only cares about the end result, whether ornot one makes it almost correct but accidentally mess up a tiny detail, or if one simply throws something together,hopes it works and it happens to output numbers within the acceptable range by chance. Again, I know you don't havethe resources to do something like this, but the current solution is NOT acceptable. Furthermore, once theimplementation has been made, there's not really any way to make sure it actually works. A considerably bettersystem to use would be something similar to KATTIS, allowing the students to check if their implementations work,instead of leaving everything to fate. And then there's the group-implementation portion. While the idea behind it isgood, there were still a major problem (which also affected the two previous portions), and that problem is ambiguity.A common principle in programming is "garbage in, garbage out": if you give bad input, you get bad output. The samegoes for the homework "specifications" (quotes added to show the faith I have in them), which could on average beinterpreted in about two million different ways (slight exaggeration). This is especially bad for the auto-correctedimplementation portion: Garbage in, garbage out, and seeing as the output is all it cares about, garbage input is waymore dangerous. Offering to "quickly" respond to questions via email is not an acceptable substitute for clarity,especially when one misinterprets an instruction without even realizing that it is ambiguous. Furthermore, having tocross-reference one's own copy of the homework instructions with the updated version online, where all the questions,answers and values are in a different order or completely different altogether, is awful, especially when one after aconsiderable amount of time working discover that one has accidentally used the problem parameters specified in theupdated version instead of the personal version, the problems of this system become devastatingly obvious. Evenworse is when the clarifications themselves add even more ambiguity and confusion, which did occur. I originally had aparagraph here regarding the student review portion, but it was a completely subjective rant on the low quality of thehomework we reviewed. Said paragraph didn't fit the semi-objective tone of the rest of this essay and was thereforeremoved. In closing: CHRIST, how can the course simultaneously be so good and such an abomination? Your approachto homework is fundamentally wrong. To be perfectly honest, I would strongly recommend AGAINST giving this courseagain UNTIL you've managed to make a proper homework system. Feel free to borrow (or even steal) ideas fromother courses and see how they solved various problems, but for the love of your deity of choice, don't use this systemever again. EVER.- It's good to have a step of automatic correction, but it's very flawed that we don't get checked on our answers. Someof us might've gotten the concept right, but a few calculations wrong. Having to explain your solution can be very goodfor us as well.- Very good learning mediums. The correction was sometimes surprising and unexpected.- Easy way to score a high grade on the course.- Really unprofessional as they are sent from a gmail address instead of handed out over Bilda. Also, someone shouldhelp Javier with expressing himself in English. The text is very vague in some places and it feels really annoying sinceI feel that a schools like KTH should be professional in all that it does.- Boring, and could be solved adequately by searching in the book for keywords and reading those sections.

- Only comment about the automatic homework correction- N/A- In theory they seem ok, this years practical results somewhat less so. Some short examples. (admittedly some arehearsay) Getting a file from the (your!) system identifying the submitters of a part B (in ones own part C) despite theywanting to be anonymous. Putting a letter in the answer that is not in any of the domains for the coloring problem is"not a typo". Then what is? Having the same implementation error as the first iteration of the automated correcterresults in almost zero points on that problem. The same small error by the PHD TA should also be a small error by thestudent. Getting told that you were kind with regards to the submissions of homework 1, but that you wouldn't be forhomework 2. Only problem with that is that the deadline for homework 2 passed way before any feedback from HW 1got back to anyone.- I'm very pleased with how they where.- For the love of God, stop making complicated sentences! (Part A) They can be interpreted in so many ways...- Make sure homework contains no errors....- Took a lot of time; if the project is meant to be more than a homework (in terms of time spent), then the homeworksshould be shorter, otherwise, well, there's just not enough time.- Relevant to the course, good difficulty level.- If you necessarily have to change the assignment texts, please mark the changes extremely clearly. Homework 2(hmm) became too hard because of the ambiguous starting points on the observations (Ozero, O1 etc.). Apart fromthat detail it was good.- -- They filled their purpose well. They forced us to read and understand all parts of the course.- Too many errors in the initial formulation of the problems. The de facto time to do homeworks is not one week whenyou have to send at least two emails/homework asking for clarifications and there being a list of final correctionspublished just two-three days before the deadline. For instance I lost 2 points on one homework by selecting the "mostobviously correct" answer, the second one also seeming correct and the third one was "none of the above". In anemail that came just 24h before the deadline and that I didn't see until the deadline had passed the "none of theabove" was changed to "both of the above". Also I think I lost 8 points on another part of a homework. The "correctanswer" that the automatic correction system returned was obviously mad. (I'm satisfied with my course grade and Ididn't have time to appeal, but I hope you spend time to get things right for next years participants)- They could have been better... Was the main idea to have us implementing several "out of context" algorithms fromthe book? I thought AI is about solving some non-standard problems using/combining/(re-)inventing those"algorithms". Amazon's task, for example, was more fun. Were the source codes for the best solutions published?- The homeworks where terrible in one way. They seemed to be unprepared and packed with errors. All the lastminute changes, forced me to start with the homeworks very late. Often the same day as the deadline, and that wasn'tenough time to solve everything.- Really suffered from short deadlines, else all was positive.- The homeworks were very useful to understand the course and the deadlines were good too.- The email-distribution was somewhat strage and unreliable. The rating was unclear. You could neer be shure if youactually submitted the right values, as it was all number-based.- There should be a response when you hand in a homework. The question should be clear.- It is good to have homeworks like these, however it would be good to have help session for people who are not thatgreat programmers.- The first homework had some very ambiguous questions, and one of the answers that I gave was due to amisunderstanding of the question. I thought this was supposed to be corrected but never was.- good content, poor execution- See above- Very good. Some more time would have been nice. The course overall is very dense.- Use Kattis as for all assignments.- Shouldn't be modified so many times, when you start doing it you have one version, and then during the followingdays suddenly you have 3 more versions. It's confusing- The homeworks took much time to do, and because of the automated correction system it was important to get atleast part A correct in terms of the answers. A misspelling or similar could result in lost points. The criterions for part Cwere vague. Also the points received for doing part C was few in comparison to the time needed to do it. Doing thehomework individually was a big disadvantage compared to doing it in pairs. Maybe you should say that it SHOULD bedone in pairs or have higher criterions for people working in pairs. Part B was very good. Also, please don't spam mymail box. It's very frustrating to first get one mail, and then another with a correction and then another one.- The automatically corrected part needs serious work. The actual software worked pretty good, but the idea is theproblem. You get one shot to answer an advanced question, if your answer is correct you get full score otherwiseNONE even if your reasoning was correct. And answers/question are always up to so much interpretation. There wherequestions where you could guess the answer with a 1/3 chance of guessing the correct answer, so the questions shouldbe weighted better. Consider more multiple step values, or independent answers. I'm not even sure this systemcomplies with KTH policy. Maybe you should consider "Kattis".- Really interersting part B:s!- Unclear assignments and some grammatical errors made the assignments hard to read. Too tight deadlines for partC. Corrections of the homeworks were not distributed timely. To enable students to learn from their mistakes, thecorrection of the previous homework should be returned some time before the first deadline on the next homework.- The third homework was much more difficult than the two others.

- Homework assignments were designed properly. However, the lack of clear explanation of some questions led me tosolve the questions / problems wrong. I think it won't hurt if the question is explained a bit more in clear English words.- To large when combined with the project. How ever, not having them at all would be a waste because they did easeup the workload form the project so it wasn't an all or nothing go for the project. How ever the balance between thetwo needs tinkering. I would say make the homeworks a little less time consuming.- Good coverage, TERRIBLE execution.- The first homework as a pure text email, was really annoying, but later the way of presenting the homeworkimproved (LaTeX-File). Still a need of better keeping track of the updates to a homework!- Some exercises were worthless. On hw 1 part A for example, the exercise with the elevators were worthless. Thealgorithm were given to us and we just had to follow it. It was completely mechanical. That particular exercise was afarce because it was so easy, but caused so much confusion because of ambiguities in the text. I think the idea ofhomeworks is good but the exercises have to be more meaningful in some cases.- Good way to learn the course.- Interesting! Help me a lot understand the concepts learned in lectures.- The homeworks was a pain to do. Not because of the content and the questions but because of the horrible way toreceive them and send them in. This really made it hard to enjoy the course and it's so much more difficult to learnwhen in a bad mood. Having the questions and being able to fill in the answers on the same place is a bare minimumof a functional system. A homepage with questions like this course evaluation form would be so much more suitablethan python scripting. The code should be corrected automatically by a system (preferably Kattis) that gives youprompt feedback before the deadline. I really wish for the sake of the mental health of future students of this coursethat the distribution and correction of the homeworks will be redesigned. Feel free to contact me if you would like todiscuss this in further detail. [email protected] -- 1) Assignments were a lot of fun and application oriented. 2) Many concepts became clear and easy to digest whendoing the assignment. 3) Assignments were vital, without which many of the lesson parts which were not wellunderstood, would have gone unnoticed. 4) Assignments were very time consuming - it took up all the time in theweek and rarely left us with time to look at any other courses' material with equal attention - without being excessivelystressed. eg. Machine Learning course was studied in detail just before exams- Wow, I haven't taken a single course with crappier homeworks. The actual ideas for the questions were perhapsgood, but getting 1-3 updates on each homework with new revisions and explanations made me feel uncertain doingthem, even when I was right.- In theory they were fine. Sometimes unclear of what was actually asked, it felt like one or two questions could beinterpreted badly and might have been very subjective to the person reading it.- Weirdest format on homeworks of any course I have ever read. A system where the points suddenly can be loweredbetween corrections is not appreciated.- Big variation between the homeworks. And if you had alot to do in other courses, you might not have had time to dopart B.- They helped me to learn more than the lectures did. But Part A is more useful than Part B, and I think we did notlearn enough to do Part B. There should maybe be no Part B at all. I lost lots of points from Part B and Part C, whichwas not fair compared to the time and effort I spent for them.- Good system that is great for learning.- Not so good. In my opinion homeworks that have a code structure and the task is completing some parts are moreinstructive. Also I think presenting homework is great! For me presenting homework and discussing with courseassistant is as informative as homework+lectures. If you can manage to make such a homework for next year itshould be a great news for students of next year.- very timecoonsuming- The fact that there were atleast a dussin revisions to the homeworks made it extremly frustrating. Much more workshould be put into them BEFORE they are distributed to the students, it was confusing to say the least.- fix the turn in system and formulate better questions- The difficulty ramp is very high. The first homework is just silly, making the students expect that level on the nexthomeworks as well. They should all be the same level (only assuming more knowledge of the course).- Varied. Matched the contents of the lectures well.- The fact that the homeworks got corrected automatically in part made me paranoid about it, it was not a fun feeling,specially when the homeworks had so many vague points where you don't know exactly what answer the programwants. Either make much more clear exercises, or make the program more forgiving.- I would have appreciated shorter delay between handing-in and correction distribution. That way it's easier to learnfrom your mistakes. Usually, I go through the correction and make sure I understand what faults I have made. In thiscase, I did only take note of what grade I got.

Did you think that the project helped you learn the material in the course?


3 2,5% Not at all

28 23,7% Very little

70 59,3% Yes

17 14,4% Yes it helped me learn a lot


What do you think about the milestone difficulty?


22 18,6% It was too easy; we solved it with little effort and thecode we submitted did not help us in the final submission

89 75,4% It was OK; the goals were achievable and it helped us inthe final submission

7 5,9% It was too difficult; we didn't have enough time to solve it


Comment:

- I think it was a little too easy to help in for the final submissions. But considering the time limit and as a milestoneproject I still think it was good.- Completing the milestone was done with minimal effort. Still, it's a good strategy to force us into starting early.- It wasnt that hard if you knew if you had an idea how to do it. But we didnt Achieved the goal intime because ofproblems with the group members. The penelty was abit harsh though.- should have been harder actually- I mean the entire project does not really help you to understand AI but more getting better in programming. That isnot stupid in itself since it is good to be a skilled programmer but the project should be easier.- milestone was fine- Though I had a bug, it was really easy.- -- May be it's a good idea to introduce another milestone before this one. Where each team is supposed to present"some project plan" for their work and UML diagram of what they are to implement, may be some data structures and

algorithms to be implemented. It wouldn't take much time to compile such document if you have started at leastdiscussing it. And then later completing the project students could reflect about changes they made and etc.- a little bit to easy maybe- I thought it was fine, we had no problem meeting it.- Reasonable level of the milestone and, again, force the groups to form and start working on the project.- The midterm milestone felt a bit like an assignment separate from the project. It didn't get us started earlier on thefinal implementation. We started on it later than we should have. Maybe moving the midterm mile stone to a laterpoint, closer to the deadline of the project would make it a more integral part of the work. Making it harder could alsoimprove that effect.- We didn't have time to sovle it, because 50% of my group was lazy and/or retarded.- Did the project last year.- The milestone only took a couple of hours to complete, but we used most of that code in the final solution.- forced us to get going, should definitely be there- It would have been better to have the milestone automatically corrected in a way that we can send in our code andhave it corrected within minutes so we can work iteratively. Kattis would be great for this.- Perhaps a bit to easy.- This was a vital step - without which the procrastinating students < like me :P> would have suffered!- Very good choice of difficulty because of the fact that when you feel secure about that you have passed the minimumdemands, any other improvement is just for fun/prestige and/or a good grade. So that was very nice, and it providedmotivation for making it better while at the same time having fun doing it.- It got you started with the very basics.- It didn't affect the grade now, and I don't think it should in the future either, unless it's a total catastrophe. Youshouldn't lose a grade because you forgot the situation where the man starts on a goal square. The milestone itself isgood though, and motivates us to begin early.- Why was the deadline mysteriously different from ALL the other assignments? All the other were to be handed in justbefore midnight, this was a couple of hours after midnight, which meant we missed it by a day. I know, we made amistake but please make it less easy to fuck up...- It was verry easy to achive, which I think were OK, it motivated us to get started. Perhaps it could be a little moredifficult.- C++ with the client was a problem.- It was a useful goal, and the knowledge that one had a grade guaranteed already after that was nice. Could maybehave been done in two steps, with a grade E and a grade D perhaps being guaranteed.- We were a little surprised that we did not pass the milestone, since we had got the impression that we had only got tosolve one of the first boards. Any way, the milestone helped us start with the project in time.

Did the project milestone help you organizing the team and getting the project done?


24 20,7% Not at all

36 31% A bit, but the benefits didn't compensate the effort weput on it

56 48,3% A lot; it activated us and helped us to create a code basisvaluable for the rest of the project


Comment:

- I will elaborate below.- There should have been somekind of milestone before this one. to make sure that the groups really worked. We

ended up with only two people left in the group and could not really do anything about it untill it was tolate.- we managed that independently of the milestone, however, I consider it a good thing to have the milestone to forcepeople to start early.- Some did not do anything at all...- Not the milestones fault.- Somewhere in between the middle and the last choice for this question.- The milestone meant we had invested time in a suboptimal solution that would influence the (incremental)development process later on, which proved, in the end, to be quite damaging overall.- -- Yes, that's exactly what it did. Although we failed it: we submitted a program literally "showing" (printing to console)how we were almost able to solve one of the first several boards. Still, I think, it helped a lot.- Didn't make the milestone deadline.- It made us start early.- Did the project last year.- We had other problems which were related to group dynamics rather then having problem doing the project.- Hm, I think there is not an alternative here that represents my experience. It was good to get the group started, butthe code was not used for the rest of the project much. But it was worth the effort I think.- We were quited organized anyway but it gave some more motivation to start early.- -- The point of the milestone was very unclear and mostly seemed like an excuse to produce some kind of result fromthe students without regard to how much effort is needed only to collect files and summarize the project forsubmission.- We had to redo everything, and still it was very unorganized at the end.- Without milestone, I am pretty sure that my group's result would be worse than now, because we will leave the workto the very last days as usual.- We didn't do much afterward besides discussing, but I'm sure that it was valuable to talk about it.- Because it was almost trivial contribution of group members was misleading.- It did help, but we ended up trashing 90% of the code written then because it got a bit hurried, since the milestonewas too close to the Homework turnins.

Do you think that sokoban was a good choice for an AI project?


2 1,7% Not good


24 20,5% Ok

45 38,5% Good

41 35% Very good


Comment:

- Too much effort (to get an A) to put into a problem that has already been studied by a lot of people, somehow it feltlike wasted effort. Do you want us to rediscover the wheel or to learn by copying?- I have no complaint at all regarding the choice of Sokoban, except that the problem is too popular, thus there's toomuch information about solving it online. Thus, one can't claim discoveries and realizations as one's own without beingaccused of simply reading about it online and pretend to discover it oneself.- Doesn't require a lot of AI knowledge, but depends a lot on one's skill as a programmer.

- didnt really have time to implement all that we talk about in the course- Good, but maybe it would be interesting to formulate a new problem for later courses.- It was fun and easy to understand- BUT you could have pointed us in the direction from rules to heuristics- However the project should be divided into two parts as I explained above. One part for A and B where you codefrom scratch and one part for maximum C where you code based on a shell similar to Applied Estimation EKF andParticle Filter labs...- I guess it was OK but I didn't personally learn anything new.- Interestingly difficult problem to solve.- Really fun choice!- AI is not just about search trees...- I didn't like the fact that most of the litterature and existing solvers concentrated on ad hoc (a.k.a. domain-specific)techniques, and that there didn't exist any good basic concepts to build upon. That said, I don't have a better idea.- I have no idea really but it was a choice that worked well.- -- Its a fun problem.- For my group it was more of a programming challenge :) Seriously speaking, it covers only a small part of thiscourse, I think. May be you could come up with several options for final project? Planning, decision making,communication, multi-agent stuff, etc.- It is hard to think of any better choice at this moment, there were just so many possibilities (but so little time...)- It was really good but I think too dificult regarding the number of credits for this subject.- It was definitely fun and challenging, but a project covering more of the topics presented in the lecture wouldmotivate more to go to the lectures.- It would have been nice to have a project that dealt with more topics that we saw in the course, but I don't knowwhat sort of project that could have been.- One should get more time to try different solutions.- It's so much fun!- I do not know the alternatives, but it was interesting.- It was though but not impossible. And as an extra plus it was easy to explain to your friends so they got allimpressed with your work. I liked it a lot.- The good thing was that it was hard. The bad thing was that it was mostly about search, the easiest and most wellunderstood (among students I think) topic in the course. Frankly I didn't learn a lot about planning, stochastic methods,etc. because they were not needed in the project. However, I knew most of the search techniques already.- Maybe put too much weights on programming. And why can't we use some languages like matlab?- A very good choice because it was not clear about which heuristics were good and bad. It was more of a trade offcase, which is probably important to learn for the future.- A big problem with alot of solutions.- I love Sokoban! It's probably too difficult, but I'd like to see something where there are two players, and each groupcould create an AI and have them compete against each other.- In my opinion projects which is near to real projects make more sense. Maybe a speech recognition, NLP or image orvideo processing task was better. These projects are too big to start from scratch but could be instructive once bedone on a good platform.- It's a very hard project since there are so many things that could be done if you had the time. Students don't havethe time.- Yes, I think it's a great example of tasks that are easily carried out by humans, but at the same time are hard forcomputers.

Do you feel like you had to do a lot more than the rest of the people in the project group to get the job done?


23 19,7% No, I did less than others

64 54,7% No, we split the work evenly

30 25,6% Yes


Comment:

- I will elaborate below.- But that was just between the two of us that were left.- I was sick and spent the week before the project deadline in the hospital, which is the reason I made less work thenthe others in my group. As far as I know, the other members split the work evenly.- More or less- Three of us did most of the work while one sometimes didn't show up on meetings. It wasn't a big deal for the threeof us but of course it is a bit unfair. Maybe it would be better to have smaller groups, or at least not restrict the numberof people to 4.- I was put in a random group, and as those things go, win some, lose some. This time I lost handily with severalmembers lacking the basic prerequisites for the course.- Some of the people in the group really really enjoyed the problem, thus causing them to stay up several nights andcoding away solving deadlock after deadlock. Thus causing 50 % of the group to not be as involved as they could be,including myself. But that was their own fault, having no self control...- Well, I was alone, but before that, well, if I hadn't done anything, there wouldn't have been anything, to start with.- Did 97 % of the programming work. That's life when there are imbalances in the programming skill levels betweenthe group members.- But it was quite voluntarily. It would have ended up all right even if I had not.- unfortunately I did a little less coding than the others in the group. But we did a lot of discussing before coding and Iwas very active. And in the start we developed in a group at the same computer. After that the most skilled andinteressted coders made some big improvements on the code. My role was planning, writing and preparing the reportand presentation.- 2/4 members contributed.- Sooner students start working, less problems there will be...- the others were far more experienced- I did not HAVE to, but I did do a lot more than the rest of the group. One of our team members slacked of a lot andcontributed very little. There's not much to do about this problem though.- I wouldn't say a lot more, although most of the code was written by me. We contributed evenly with ideas andplanning.- As those who did nothing left the course, this did not affect my grade. But a evaluation of this in some form would begood as a lot of the grade is given based on the project groups performance.- I did about 80% of the work in our group, I could easily have gotten the same grade on my own.- I did more work on the code than the others but slightly less work on the report. It was OK.- 1) We were Maria and the Machine Learners - one of us was really good at programming and he took most of theload when it came to that. I personally spent time on ideas, documentation and presentations. The other two membersalso pitched in with their full convictions to all parts of the project. However, it could be seen that the programmercould have felt like he had a LOT of load when it came to the actual implementation, just because he was already greatat programming and the others would take too much time to catch up to his level.- -- It depends on your background knowledge and abilities, plus the expectations from the course/grade. Thus, it wasdifficult to make everyone works the same. To be honest, we worked nearly the same but the final effects of what wedid, or quality of the job done was not the same.- The quality of what people produced differed a lot, but I can't say that everybody didn't try to do their best.- I wrote all code; prepare all slides; write opposition report; read other group's code; talk almost all the presentation.- 2 of us were doing more then the other 2 :)- Uneven course load was the reason for this, it'll balance out in the end.- While we never did a proper split, I ended up coding very little, which I found unfortunate, but I did alot of theReport, so it worked out.- Of course, the number of written source code lines was not exactly split evenly, but I think everyone contributed aswell as they could.

The project was carried out in groups. Do you think that your group worked well together?


12 10,3% Not good


20 17,1% Ok

46 39,3% Good

28 23,9% Very good


Comment:

- I will elaborate below.- The group where made by a theacher and noone followed up how it worked.- Students should not lie about their skills, as this brings more damage to those who work rather than themselves. Thisis a general problem for KTH, but very very obvious in this course. Course staff must be aware of the problem and beclear about prerequisites during first lecture!- I think that we formed a great group. All in the group was ok.- For us it ended up with some people running ahead and implementing new things for fun and things being generallyunorganized, but in the end everyone had found something to do to contribute.- A good bunch of guys and we had fun, but that was basically it. Group performance was very bad.- We had one in the project group who rushed things and somewhat left the rest of the group hanging rather then dothe tasks as a group. We got the project handsomely done but it would probably have been better if we had bettergroup cooperation.- Ambitious group though..- The chemistry was there but not the skill in all members. We got an exchange student that could barely write a line ofcode and barely spoke English.- We all knew each other beforehand, so the dynamic was already there to begin with.- We were not very clear about our programming knowledge from the beginning. It's hard to from groups when onedoesn't know anyone from before. The group-matching on BILDA probably help a few.- I worked a lot on the project but some other group members didn't work even 10 percent of what I did. I think it iscompletely unfair that we were forced to work on the project in groups of four. I'm pretty sure that if I was alone, Icould handle the project much more better.- This was probably a background thing, we were two person who had worked together before multiple times, and twocompletely new ones. I have worked with group dynamics both as a teacher and before that. It's a tricky thing tomake groups work nice and smooth.- We (me and my friend) were a bit unlucky with partners - they weren't exactly great to work with. It turned out wellin the end though.- As an exchange student I was together with 3 swedish students, which was quite nice, but sometimes very hard tofollow their disscussions in Swedish.- All members weren't involved equally in the code but we still worked well as a team.- Most of us knew each other well previously but we all have the same CS background and haven roughly taken thesame courses, so it was easy.- 1) There were obvious sparks that can come up with long hours together 2) The work could not be evenly divided asit was too little for a group of 4 people to be divided well and executed within the tiny time limit. 3) Ideally - thisproject would be suited for two people and for a longer execution time. 4) There were many times when members ofthe group would be left with no work or too much work. 5) We did make our best efforts to divide the work well andpaid real attention to it. Some of us have experience in managing projects as well - but we still failed.- -- Besides some minor communication problems, my group was very good at working together.- More communication early on would have helped a lot. People had a tendency to implement things that othermembers knew was useless/inefficient.

- Totally disappointing. Disaster was that they promise to deliver assigned task but they don't! It's bad that you forcedto do a 4people project in 1 month alone but it is a disaster when your group-mates disappear some days beforedeadline leaving you alone and you have to do everything in less than a week.- But it wasn't a fare investment of time from each of the team members- We all known each other from earlier and kept a good contact with each other mainly using IRC while not meetingaway from keyboard.

Comments about the project


Comment:

- I really enjoyed the project. It was interesting and stimulating. The problem, however, was the choice of four-persongroups. A friend and I always work together on everything, as we've found that we work well together and understandone another. Thus, working together goes smoothly. However, the demand to have four-person groups forced us toteam up with two people we just barely knew. They too always work together, but they are (I can't really say thiswithout sounding elitistic) way below our level. So, we have two groups who work well separately, but are oncompletely different levels. Gee, what could go wrong here? Besides the difference in skill, four-person groups areinherently more difficult to coordinate. Just to get the project started, my subgroup (i.e. my friend and I) began byconstructing a framework of basic classes to simplify the group work. In just a few hours, this evolved into a programcapable of meeting the milestone criteria. Unable to figure out how the heck the two subgroups should work togetherand distribute the work, we decided to simply work separately to create two algorithms, as a "contest", and then pickthe best or use both together for the final submission. The result was two algorithms with completely differentphilosophies, with the one from my subgroup being the objectively superior one, but with theirs being capable ofbeating a few boards we couldn't. Therefore, we decided to simply give both algorithms roughly 50% of the time each.(Merging the two into one program proved difficult though, as their implementation had radically altered the framework(which, again, was designed to simplify joining different algorithms together) for a reason that doesn't even makesense for anyone who actually understands how OOP works.) While the end result was pretty successful, the story I'vetold is meant to show that forcing people into large groups for such a small project can have annoying consequences,especially if there's a significant difference in skill between the students.- alittle to easy to get an pass.(The group that we were supposed to opposite to hadnt done anything right, didntcomment on anything and gave us about nothing to work with. It was like we could aswell have told them how to solvethe whole problem from scratch. That were about nothing right about it. It was impossible to be creative with that wegot. and there for we got just 1 of 3 point from opposition part.- Too large group. If not a D-student the risk of getting in a group with a lot of unknown and lazy people is too big andthe whole project is ruined.- a very good approach to allow students to get some practical experience.- the project got me so frustrated that I didn't enjoy taking the course. and that is bad because i think artificialintelligence is very interesting and topics in the course were also very interesting! 'mal sabor de boca'- Well defined project, alot of fun to work with- N/A- As I said earlier, win some, lose some when it comes to random groups. But one thing that disturbs me more thanothers is that 2 days, or so, before the milestone deadline we get an email from the TA asking for a _good_ group forsome golden student, that's just adding insult to injury. The others in my group actually thought it was I who bailed onthem. But anyway, for future reference, it would be nice if everyone got that same opportunity... And really, define thedomains and other attributes of the input beforehand, getting curve balls like "you might have to increase thehardcoded buffer in the PROVIDED code skeleton" is _not_ ok.- Fun fun fun!- I can completely understand having to write a project report. I can "sort of" understand having an oral presentation,even though there's no real point to one. I can't for my life understand the part about the project report review.- See above.- .- -- It was really fun. If anything, I think it would have been good with a bit of guidance on how to carry out an artificialintelligence project scientifically: Record results, reason before implementing, try to make "theoretically sound"heuristics etc. A lot of this of course go for computer science in general. Somehow all of us (in the project group) stillmanaged to forget most of this, however.- Trainwreck.- We didn't finish the project in time, and we tried to contact you about this a week before when we realized that wewouldn't be able to make the deadline, but we didn't get any answeres.- As with the homeworks, could had really used more time.- It was difficult and 4 per group is good but it's hard to share the work in 4 for an computer project.- We put much effort in it, but it was worth the work. Learned a lot about project management :-)- Hard to know how it would be evaluated in the end.- The division of points should be different, because for the milestone we got 3 points for solving 3 boards and finallywe solved +/- 50 boards, had a lot better ideas and a lot more work put into the project but still did not get any more

points. So all our extra effort did not get us anything.- It took a lot of time! Partly because I was in a ambitious group. But if one wasn't after the highest grades it wasdoable within reasonable time.- It was really fun! Would have liked to have another week :D- It was possible to do the project without any knowledge from of the course material. Maybe you should add somemore milestones so that the course material is actually used. E.g. you could require the group to use an A*-search forgrade C or so..- Interesting!!!!!- We started late with the project and we still had more ideas when the deadline approached. But overall I think weworked good as a team and had a lot of fun with it.- It was a fun project!- I just got a grade for the project which I didn't deserve. I think the evaluation method was not fair and the mainreason was the teaching assistant was alone and he had to deal with lots of assignments and project reports. Honestly,I feel so disappointed after devoting a lot of time, not sleeping two nights but a few hours to make the project bebetter.- It was fun and educational. Please keep the presentations within you own period next time. That might mean the youhave to cut it down a notch for people to make it in time. But the CL program will never be able to do presentations inthe next period as we always have this internship in the period after yours if we are 4th graders. And almost only 4thgraders are able to take you course with the prerequisites.- The project didn't reflect a whole lot of what we had learned, but I still think it was an excellent, challenging and funchoice.- It's a good alternative to an exam and it's a lot more fun to work on the project, though it's the same or even moretime you have to spend on it.- I think the number of people per group should be 2. Or, at least, as in the robotics course that you had, require eachmember to estimate every members contribution. In my group the work was very unevenly spead, and I imagine thatit was in many other groups too. However, when I think about it, I'm not sure I would have liked it to be more evenlyspread, because that would have made it more messy, the code more buggy etc. I think the best thing to do is tomake it two people per group.- I loved the project!- The project would have benefited of a more robust testing environment. The server went offline more than a fewtimes. A downloadable server would have been great. Using boost in the c++ client example is not nice. Why not baseit on the c-code without using boost?- Perhaps it could have had a bit more influence on the grade since a good project took a lot and i do mean a lot oftime.- 1) Structure and method for the project was brilliant! 2) Loved the whole approach and examination pattern. 3) Wishthere was also an added incentive of some kind of competition and prize scenario after the grading. 4) One finalmeeting with a summarizing of achievements of different teams was very very necessary for learning and giving alogical conclusion to the course.- Nice.- Ridiculously easy demands for a passing grade compared to the demands for any higher grade. Also really odd todemand such a amount of work for a higher grade on the implementation when it doesn't seem to influence the totalgrade that much.- Good have been nice with more information about grading, and just information about what was expected from youin the report, presentation and opposition.- Interesting but difficult.- Good project. A bit more information about how many boards needed to be solved for each grade would've beennice.- At least in my case, we put more time on developing a structure and make a working platform, finally we had aplatform that any heuristic or trick could be implemented easily but it was half a day before the deadline. I think thatthe project should design in a way that the focus be on AI part not just implementation.- We had a blast coding our sokoban agent...- It was fun, however it would have been more fun to use kattis (more competitive).- It was very unclear if the code was supposed to run on the school computers or not. Our group assumed that it was,which caused us to waste a lot of time. "boost" libraries are not available on the school computers. This MUST be fixedtill next year. Either make it clear that personal computers can be used (basically allowing any code, which makes thereview part harder), or force the students to run it on school computers or even Kattis. Not somewhere in between likeit was this year.- Fun to experiment with different approaches.- The project should have used Kattis, the kth programming judge. Perhaps with some extra data given to students,like the boards used and such, since Kattis itself is a good foundation that doesn't require connection to some serveretc. Also, exchanging the boards in the middle (and end) of the project was not appreciated. You should refrain fromdoing that, specially since at least our group used those boards for our statistical and run data. If you exchange boardsduring the project, that means that our assumptions as to where we are in terms of a specific grade get all warped.- It was very entertaining, especially since we managed to implement some nice designs.

How many other courses did you attend in parallel to the AI course?


3 2,6% None

18 15,4% One

53 45,3% Two

31 26,5% Three

12 10,3% Four or more


This is a 6hp course. Compared to other similar courses, is 6hp suitable?


0 0% Should be less than 6hp

49 41,9% 6hp is OK

68 58,1% Should be more than 6hp


Comment:

- The project was pretty big and I feel as I put down alot more work into this course than other courses that are only6hp- Surely worth 7.5 hp- but also there should be a longer course... there is a looooot to say about this and i think we didn't go deeply enoughin anything- I have never liked the idea of 6 credits. You need on average 7,5 credits per period to get a degree. If, in oneperiod, you do not like any 7,5-course you are forced to choose a 6-credit course. But then, to get those last 1,5credits you have study yet another 6- or 7,5-credit courses. And don't give me any crap about taking 1,5-course caus'let's face it, how would like to show that in your Diploma? "I've taken a 1,5-creadit course at the library with a selfappointed "Information Search Specialist" as lecturer"- This course felt like an OK introduction course for AI. I have also taken the course Machine Learning which containeda lot more complex and also took quite a lot of time and effort. I feel like this course had a good balance.- I would love to see an extended version of the course that cover more of the subject.

- 7.5 seems right.- If you're heading for an A, and also wanting a high score on the project - you'll put more time to ut than a 6 hpcourse I guess. It smells more like a 7,5 hp course to me.- Compared to other courses, it should definitely be more than 6hp.- 6 hp would be fine if the course ended at the project report deadline.- Definitely, I spent almost all my coursework time on this one, whereas the Machine Learning course, for example,had the same number of credits, but took me way less time. The fact that the course is based on project andassignment means it takes a lot of time for "good" students to produce good work, but it may also be easier for lessambitious students to pass, so it's hard to assess the right amount of points. If the course should remain at 6hp, then itshould be less demanding on the top objectives. That way, "good" students can just pick whatever exercise they likeand work more on that one, rather than having to work more on all of them.- Depending on the time you put into the project, it felt like more than 6hp. To just pass the project part, it wasprobably suitable with 6hp.- Felt more like 9hp. I had to virtually drop most of my other courses.- I think it could be a decent 7.5 course (mb you could even add something to it for the increased cost)!- It was extremely high paced, could be extended to run longer and give more hp (or just run longer at least).- This is my first study period here, but I had a do a lot more work for these 6hp than for the 6hp I got from MachineLearning and the topics were very similar. However I'm not sure if this class had a lot of work or that class had a littlework.- I think this course is a 7.5 points course. I currently attend Image Processing and Computer Vision and compared tothat one it would be reasonable to switch points between the courses.- I'd say 7.5- 7.5hp- See my earlier notes on time consumption.- compared with theory and methodology of science course.- It is way more than 6hp, if you aim for an A at least. I spent maybe 30h each week on the course..- The effort you need to put in depends so much on which grade you are aiming for.- assignements are consequent (more than in my other courses) Reading the book takes very long time. Many newthings/algorithm to learn- We spend alot of time on the project to get a good grade so perhaps 7.5 would be more suitable, but on the otherthis is KTH if i wanted to slack all day i would have chose economics rather than engineering.- Should be atleast 7.5 credit and ideally with more assignments easily expanded to be 9 or 12 credit courses. It wouldbe a great opportunity to attend a 12 credit course in AI and spread over 2 periods. I would definitely attend a moredetailed thorough course than a quick run through.- -- Maybe more, if anything.- I had 7,5hp courses which require less time and effort than this course. This is not fair.- It works, but I'd rather give it more than less.- the variance between courses is huge. These are some of the most expenxive points i've gotten, but it was worth theeffort.- If the project group wanted A, the project would take time comparable to 15 hp. The threshold for E was easy andfair for a 6 hp course. My personal filosophy is that high grade should be based on knowledge and NOT how much timeone spends. This is a bad setup.- Much more, the project is too large and would surely benefit from a longer run time as well, specially since you haveto start the project during the Homework turnins, which is a problem. We would gladly have seen the project runningfor another 2 weeks at least.- 6hp is well suited to three homework assignments, but the project itself needs 3hp! Therefore, for this course, a 9hpseems to fit better.

If you could make the choice again, would you have taken the course?


4 3,4% No way in hell

5 4,3% Probably not

13 11,1% Maybe

40 34,2% Yes probably

55 47% Yes definitely


Comment:

- Absolutely, if I could skip the god-awful homeworks.- The project ruined the course, the lecturer needs to be clear with all students what the prerequisites for the course is.If the student did not pass basic CS-courses this is NOT a suitable course to take.- Yes, because I have to. If not, then the answer would be "Probably not".- But, there are things to improve. Add a few more lectures or an exam to increase it to 7,5 credits- Ask me in 6 months... Given the same group, probably not. I was highly psyched going into this course and I stillthing the syllabus looks great, but the project sucked too much energy out of me without anything to show for muchthe work put I into it.- Course is currently broken.- That wasn't exactly what I had in mind when applied.- Despite the high pace one learns a lot, plus it's really interesting!- I have to because of my program. If it wasn't so I don't know. The course took quite a lot of time and you did learnvery little. I did get a good grade though.- If you let me do the project alone, I would DEFINITELY take the course once more time.- It's easy to point out flaws. But I am happy I took the course and I did like it. It gave me a lot of understandingabout AI programming that I didn't have before and that will be useful to me.- Depends on if Javier is still there.- I love this course.- I would have taken the course a year later to be better prepared for the markov chains.- I would, even though the course was mandatory, Before I probably wouldnt have chosen if it wasnt, so i was actuallysuprised how intreseting AI really is and i would 100% take it again.- -- Did not learn anything valuable from this course and was only worth the course points. If I were to take the courseagain it would only be to improve grades, not to reinforce any learning.- I always wanted to learn about AI, so I will take it anyway. Besides, it was interesting but I could have learnt more.- AI is awesome!- From the experience of this course I learned that in such a course that some task should be done in large group; Ishould arrange a meeting and ask group member about their background and expectation and drop the course oncetheir background are not related or their expectation were not the same as mine. Now, I prefer not taking the courseover working three times and still be 3/4 of a normal group.- duh, obligatory!!!- The administration of the course are the only reason I wouldnt attend it again, it was unacceptably bad.- as long as it is mandatory for me, otherwise probably no- I would never take it again. Sadly, this course scares people away from the field of machine learning. Being anintroducory course, it should be fun and broad instead of just algorithmic.- It's mandatory for my MSc, so...- AI is a very interesting and broad field, and I'm happy to learn as much as I can about it.- I would have to, since it's part of my master's program.

Things you liked/appreciated about the course:


Comment:

- The homeworks, to work in groups, and to learn about so many different things.- The main lecturer's lectures were rewarding and interesting. The choice of Sokoban was splendid and lead to aninteresting project.- The course content as such was very very interesting. Learnt a lot.- That we got to learned how to attack problem with a really huge decision tree.- Implementing the HW was interesting and very educational, especially compared to many other CS-courses.- I like so much look back now and see that we (the group) worked a lot, but in the end we have reached a greatproduct. I'm very proud of our program.- The project. It was fun to implement things and think about things.

- That the course examination was not an exam- N/A- Good topics, good lecturers, fast back and forth communication outside of the lectures. (i.e. PIM/email)- Sokoban. Homeworks. Automatic correction.- The teachers made an effort to make the slides pleasant (except Hidden Markov Model and logic... those were boringand *really* unclear; even I after understanding the topics could have done better) Following the course about MakingDecisions was a real pleasure.- Project was OK. Implementation part of the homework was ok.- The book offered a good in-depth discussion of the topics and it was necessary to read it to actually do most of theassignments, which is a good thing, because other "introductory" courses tend to be way too superficial IMHO.- The first and second edition of the book could be found online. It was an interesting course. The lectures wasn'talways super productive but overall fun. Keep bringing robots and such toys to class to show robotics and such, it'sfun. Keep using bilda, it's a great platform. Perhaps you should include even more things from the website to bilda.Just to have all information in the same place.- -- It gave me substantial philosophical insights. I consider the following most important (I may have forgotten some): -how a computer can reason (with and without uncertainty) - how the value of information can be estimated - how the"desires" of a system can be designed via reward and utility.- The lectures were interesting and fun.- Real life examples, good lectures Homeworks + project - good tandem- The sokoban game was a good project.- There was an insane amount of cool stuff to learn, and solving the homeworks/working on the project felt rewardingalmost always.- The overview was really clear and the aim too. Examples helped us to understand.- Projet related work, no exam, the lecturers.- Its an interesting topic.- Good organized course.- project / homework- I learned a lot about AI, and a lot of the algorithms seem very useful.- The subject is awesome. I think it should be given over two periods. Right now there is too much information and toolittle time.- No exam is a very good choice for this course.- The project was so much fun.- Giving an overall picture and understanding about the challenges in AI.- Most or maybe all of what we learned feels useful to know, and may definitely come in handy in the future. And theproject was fun.- Very interesting subject, homeworks challenging in a fun way.- The content and project was great.- Doing a project instead of an exam, I think I learned a lot more from it.- The book is a good reference in this regard. Homework assignments made me learn the theory better and befamiliarized how to apply AI techniques in practice.- The live examples of real world application for example the robot videos shown. That it was in project form and notan exam for theoretical knowledge. Much better with a project!- The subject is interesting and the lectures were good. The homeworks reflected the content of the lectures well. Theproject was inspiring, mostly because the problem is difficult.- That you tried to offer the same lecture twice a day because of rooms' capacity as often as possible and needable. Inmy home country no lecturer would have ever done so! Both thumbs up for this!!!- AI is fun and interesting.- The project, and the ptB/C of the homeworks!- The lectures and the broad content.- The project was the best part, very rarely you actually get to use all of your "CS-knowledge" at KTH most coursesgenerally have to easy or to specific programming task this was a challenging task in a good way and i really got touse alot of the stuff ive learnt during my 3 years at kth.- 1) Assignments 2) Teaching style of Patric 3) Teaching Assistant - Javier was doing the BEST under his control, wasvery available, approachable and efficient.- Fun project.- Javier and the whole project. It was overall good.- The attitude of the instructors, their close care for the course were good. The subjects were interesting. The projectwas very interesting.- Patric, the project, course material in general, fast mail responses from Javier.- Strict deadlines Lectures by Danica First introductory lecture and last lecture was great!- hands on use of all these algorithms we've learned over the years...- The project was fun and gave us the opportunitie to explore many interesting ideas.- I liked that - we could get feedback with the trainers if something was unclear (even in weekends!) thanks for this! -working in groups on the project, it was one of the first experiences and it was really interesting; -Patric is very clearin explanation and it's interesting to come to lectures- I liked the pragmatic and down to earth approach. Would have liked some discussion of machine learning applications

of similar techniques to the ones we used.- Choice of a practical and interesting Project choice. I liked Sokoban, it seemed much more interesting thanMinesweeper that I heard some older students did.- The project.

Things you would like to see changed next year:


Comment:

- The sokoban project- The homeworks. OH DEAR LORD, THE HOMEWORKS.- Homework can't get these last-second changes, it's not ok. Less Part-A checkbox questions and over better areas.Relate the homework the the project or Part B better.- Less emphasis on coding and more on human reasoning.- a better programming for the course, READING THAT AMOUNT OF BOOK WAS APPROXIMATELY IMPOSSIBLE. If weare not supposed to read that book, make it more clear to us, what should we read, what is more useful, I do notremember any of the questions of part A! Moreover, I did not learn enough from the lectures. Materials of the lecturesseemed to be so abstracted. Generally, I think that if a knowledge expands during years, the good way to teach it isnot to add the new material to the previous ones, and present the whole thing in the same amount of time. But toeliminate some useless parts, so that there will be enough time for understanding. What were we really supposed tolearn from this course?! It would be better to highlight them and focus on them, not to read 80 pages per lecture, Ihave to add that just by the slides was also hard to understand what is going on. So, I usually needed some othersource to use.- HW written in formatted text. Impossible to read and print as is, needed to paste into a word processor and manuallyformat it to get a decent readable text. All information about the course should be in one place, not split up between inlecture pdf:s and course homepage. Slow response time for HW1 part B. It was said that HW2B was to be judgedharder, but not knowing the standard from HW1B this statement meant nothing and just added confusion.- i would have needed help in organizing the project group and make it work. because of my almost null experience inteam work, i couldn't do this right and this messed up the course for me- Well, I think that the main problem is the time and you can't make anything with that.- It felt like the report and opposition took too much focus in the end. The project and presentation would be enough inmy opinion.- Perhaps abit more work on the homework before passing it out, it is really annoying to work on it and then see onthe web a new version.- N/A- The second half of part As needs a major overhaul with regards to the correcting. Possibly part B as well, but there,at least, you have a human look through it thoroughly. Faster turnarounds on the homework corrections so that onegets the chance to improve between homeworks. Prerequisite testing/checking in the door into the first lecture ;)- A more stable homework system. Please also use some clearer TeX:ed PDF's as in the last homework assignment.- Some Help sessions for the homework would be welcome. Really. This course is a bit too impersonal compared toother courses, the teachers are always changing and the number of students in the course reduced their availability.- Use kattis. Fix broken theory part in the homeworks. Cut time from oral presentations, should be replaced with somekind of review session with one of the teachers. They had to attend all oral presentations anyway, which in the enddidn't help them grade our work as much as a personal review session would have. Remove project report peer-review part.- See above about difficulty, grading, etc.- The homework assignments had sometimes vague explanations which led to confusion that sometimes led thestudents to calculate or program something similar but not exactly what the teacher had wanted. So make theexplanations crystal clear.- -- The part about Markov models could have been more clearly/pedagogically presented.- see "Overall comments about the homeworks"- Fully automated homework correction/peer-reviewing More interesting home assignments Mb, guest lecturer withdifferent point of view on AI or with interesting project? (just a guess)- Bigger lecturerooms. Homeworks with less errors. Split up the HP(högskole-poäng) e.g. 3.0 HP for homeworks & 3.0HP for the project.- Extended deadlines, rest was really good.- Little bit more details, even if we found all the information in the book, sometimes it's better to have it on the lecturesupport because we used it for the homework.- The project could cover more topics from the lecture. Just search is somewhat boring.- Add help sessions for the homework and someone that read the homework before it is handed out. Make the part Bson kattis and make a way for the students to make sure they handed in all the correct documents. Make the gradesmore clear for each assignment.- I would have liked to get more, and especially relevant, feedback from the coaches for the project. We did not getany useful feedback at all. Further I felt like the coaches did not read the report and relied on the review of the

opposition. The opposition had some comments that did not make any sense because they obviously did not know howa paper should look like.- Keep all information in one place. Grading, outlines for what should be in the project report and so on.- To use Kattis as homework system.- As I said, less mails, correct mails, less automated grading. Also I think that the grades for the homework should behanded out much earlier. Now we didn't know what grade was required on the project to get overall grade X.- You need to rework the idea behind automatically corrected homeworks.- A later and/or harder midterm milestone would likely be better at getting all groups to get to work on the final projectimplementation earlier. We just did the milestone quickly, did nothing for a couple of weeks and then started the "real"project too late.- Better communication with students. Updates on homework assignments provided in a single updated source.Corrections of homeworks provided before the next homework deadline. The multiple information channels (web site,Bilda, e-mail) was confusing at times. If possible, publish all information on both the web site and Bilda and reserve e-mail notifications for personal communication and urgent updates.- Reviewing other people's homework didn't work that well...- I think it is better if the probabilistic learning lectures stay in the book's framework. I also suggest that select someproblems in the book with their solution be available online for students and make it non-mandatory part for thosestudents would like to face more problems with solutions in this subject.- Make the automation in homework correction work better. Don't send us grades until we have actually done thethings being graded (like now when we couldn't make the presentation on time we got a mail saying no grade since thepresentation wasn't done. That is just annyoing, your system should have a function checking to see if a grade shouldbe mailed out or if special rules applies to a student). Less workload, but keep all parts as the ratio between them weregood.- The automated correction didn't work well at all. Use student correction aided by a solution sheet if you feel you don'thave the time to correct them by yourselves. I know I missed out on a lot of potential knowledge since it didn't saywhat I did wrong. The organization was TERRIBLE. Information all over the place (bilda, course page, email spam),meant we generally had to spend several minutes looking for even the simplest information. This lack of organizationcaused a lot of confusion, and this along with the automated correction really ruined the course for me (and I knowothers think the same). The subject is really interesting, but the organization gradually replaced the exhilaration I feltin the beginning with annoyance, to the point when I really just wanted the course to be over. I know Javier is toblame for a lot of this. I've also heard he did the exact same thing last year, got really bad reviews but didn't change athing. Thanks for ruining a potentially very good course, again...- About deadlines in specific an in general: I didn't like that you shifted the final project deadline from Friday to Sundayabout 3 or 4 days before the end. It was nice to have some more time (we actually really needed) but we should alsolearn to stick to deadlines and do a good planning of our work. Furthermore as it was a weekend, I already had madetravelling plans for that weekend, which I - of course - had to skip then. Some words in general: Why must deadlinesbe in the middle of a night - especially on Sundays? As everybody mostly will work until the end, you thereby cause alot of sleepless nights and travelling through Stockholm at impractical times.- I already mentioned most of what I would like to see change but: More meaningful exercises in the homeworks, 2people per project group. The project focused too much on search and not enough on planning, stochastic methods,knowledge representation, etc.- Better organization maybe?- How the homeworks are distributed and corrected! I would have liked to learn more about creating self aware agentsin contrast to searching in big solution trees.- treat less chapters- Perhaps the course could be given during 2 periods rather than 1. And though a bit harsh but; less people attendingthe course!- 1) Number of credits 2) Time length of the course 3) More assignments 4) More time for project - maybe with 2milestones, smaller group and a competition phase- Better and more thoroughly thought through questions.- There was to much focus on search algorithms. Perhaps try to explain how complex gaming AI works (like Starcraftor Command and Conquer (whatever)).- Condensed and clear course goals. Better maintained homeworks. A different grading system for the project oralternatively something making it worthwhile to get anything else than a E on the implementation.- Javier should make less drafting errors in mails and homeworks. More information about requirements on projectparts.- Lectures that do not heavily depend on only slides. More details about subjects during the lectures, not just overallviews and general ideas. Probabilistic reasoning parts should be tought more slowly and clearly. No Part B and optionalPart C in HWs. More fairly graded Part B and C. Less difficult project.- Look over the homework questions. Some were a bit unclear.- Change the style of homeworks to something like lab. Instruction+code structure+submission by strictdeadline(bilda)+in person presentation More course assistants in order to correct all parts of the homeworks by hand,and possibly in person submission. Smaller group. In a group of four; you are just 25% of the group and if two of thegroup members happen not to work you never ever can catch the project. Also under such a condition that 1-the onlyforce for doing the project is grade;2-some students satisfied with E;3-one can pass the course with a trivialmilestone;4-there is no exam; the strategy of "take the course, do nothing, hope someone do the task" works forsome students; and when the groups are large with a high probability someone will take the responsibility and that

strategy will work. In addition, most of the groups do the project in a period of 2-3 weeks. A group of four is too big forsuch a short time. Exam, exam, exam!- You might hook up with Stefan Nilsson and do some joint thingie for all the D students, since the AvAlg course andthe AI overlap a bit...- The administration was horrible, it needs to be fixed. Especially regarding the homeworks (to many revisions, made itconfusing). Bilda was also not especially easy to use, it had no practical way of navigation and were irritating to use.- C++: Make a code skeleton (for every language supported) for the project. Use kattis. State your standards clearly.Don't use boost. Allow C#. Remove part A of the homeworks and remove the review part. Homework 3 wassurprisingly hard compared to the other ones. The course is way too large for 6 hp. Narrow it down to some importantshapters.- I would like: -To have an option to increase your mark if by having some additional assignments. I couldn't come toKTH till 10th of September(visa issues) and didn't have enough time to solve part A and got an E for that, whichimpacted really bad my final score. -There were too much chapters to cover in a short time, making it really hard tomake things in time. -The last homework was much harder then previous ones, it's also because the explanations fromthe course were unclear, especially the practical part - how to really solve HMM problems. -to have a document withexplanation how marks are calculated (the project grade was computed not really straightforward), and have all thisinformation in one place so it would be easy to find it. We didn't have a clear explanation how will be the projectgraded. -not enough feedback for the homework parts -not enough feedback on the project (don't understand how igot the marks for the report/opposition report/presentation) i got some grades and that's it, i think if we spent so muchtime on the project a more serious feedback should be given so we could learn something from what we did wrong.- Standardize deadlines. Streamline the distribution of information, searching through bilda, email and the coursehomepage got tedious.- We would gladly have seen the project running for another 2 weeks at least. Also make sure the homeworkcorrection is fixed, as well as possible Kattis interaction of the project.- Proof-read homework assignments.

Do you think that you were treated differently in this course because of sex, sexuality, ethnic background or disability?


1 0,8% Yes

6 5,1% Not sure

110 93,2% No


Comment:

- wtf?- Don't think you should count my "No" here since I am a male Swede...- Quite frankly - Yes, due to my disability to understand the Markov shizzle ;)- I think being a caucasian heterosexual man with no disabilities made the fully automatic homework correction systemhate me. Maybe it wants cake.- Wow. Where does this question comes from?- lol- The project result is still a nightmare for me and I still cannot believe that whole my effort was simply ignored byteaching assistant...- I noticed no such thing. I always felt fairly treated and got fair answers when I question you judgment of any of myassignments.- It's the first time I see this kind of question in a course evaluation. Swedish way ?- -

- Not that I noticed.- =(- Exactly the same mistake as mine; not being compatible with Matlab r2009a but with r2009b; reduce the grade of myhomework by 50% but not changed the grade of a Swedish friend of mine.- Overly i really enjoyed this course, i learned a lot when doing homeworks, when working in groups on the project,but i feel unhappy with my results. (i wrote in the `Things you would like to see changed next year` section) Thanks!

course analysis (2010/11 period 1) dd2380 artificial intelligence, 6hp

Documents