Autonomous Band EECS 498: Autonomous Robotics Laboratory

University of Michigan Robert Bergen, Mark Isaacson, Sami Luber, and Zach Oligschalaeger

http://eecs498teammusic.webs.com/ Overview For our EECS 498 Autonomous Robotics Lab Final Project, we created an autonomous band, consisting of four robotic arms and two xylophones. An overhead camera is used to detect music notes, which are parsed into music and sent to the robotic arm controller. The arm controller uses motion planning to direct the arms to play notes while avoiding arm collisions. Autonomously reading and playing music is challenging because it requires building a robust music-reading system, careful synchronization between the music reader and the music player components, and motion planning between the arms to ensure notes are played smoothly and without arm collisions. Described in detail in the Music Reading section, our system continuously detects musical notes on the music board to allow users to see their desired music change in real-time on the GUI. However, to allow the user to set their desired song before playing it, the user controls when the new music is sent to the music player. At this point, the music player discards previously sent music and clears queued notes to accommodate the new music. Described in detail in the Music Playing section, based on the number of available robotic arms, the music player component schedules each robotic arm to play notes using motion planning to avoid arm collisions. Music Reading Representing Music Our system strives to resemble real sheet music as much as possible. Notes are represented by red circles, uniform in color and size. Users can set these notes as desired on the music board. Furthermore, our system supports reading and playing all G-clef notes (middle C - high G).

Camera Calibration

Our camera calibration calibrates by first detecting the four corners (marked by blue squares) on the music board (the system pauses if more or less than four corners are detected). Using the locations of the four corners, a homography is built to project notes from camera pixel space to physical music board space with respect to the center of the music board. Using knowledge of pre-defined line and stanza spacing to determine the note value and time of each music note based on its physical (x,y) position on the music board.

Detecting Notes When detecting music, each stanza is scanned separately for music notes. We detect music notes by using a Blob Detection algorithm. In the blob detection algorithm, there is a color threshold for notes in which colors are represented using HSV (more robust in environments with varying lighting). The blob detection algorithm is also blob size sensitive. More specifically, there is a threshold range on how many pixels a blob must be within to be considered a note. If a note blob is larger than expected, we assume there are overlapping notes on the music board that are being detected as one large note blob. Using the covariance matrix of the overlapping note blob and the expected size of a note, we determine the configuration of the overlapping notes and how many notes the blob should be broken into. The round shape of the notes is also important as it improves our variance calculation accuracy when building the covariance matrix.

Once a note is detected, the note blob is projected from camera pixel space to physical music board space. The note's value and time (within an eighth count from the beginning of the stanza) are determined based on a position threshold on the (x,y) physical location of the note on the music board. The time of the note is "snapped" (rounded) to the nearest eighth count, so that the music flows more smoothly. Finally, because our music reader actively reads music in real-time, our system is sensitive to any changes made to the music note configuration on the music board. The blob detection algorithm runs continuously and updates its parsed music based on any changes to the note configuration. Parsing Music Once all notes in each stanza are detected, we preform auto-spacing on the notes for "better sounding" music. More specifically, if there are three or four notes in a stanza, regardless of the "snapped" times for each note, note times will be adjusted such that the times are spread evenly across the stanza. The music speed is also automatically adjusted for the robot's physical limit. In other words, the music speed is slowed down enough to allow the robot arms ample time to play each note. Using the system's GUI, the user can also further slow down the speed of the music. GUI and Music Transmission As the music reader dynamically detects music, a GUI shows the user the currently detected music notes, a camera feed of the actual music board, and the music currently being played on the robot arms. This allows the user to adjust the notes on the music board without interrupting the currently playing music, and then, once the user is satisfied with the new music, send the new music to the robot arms to be played. Upon sending the new music to the music player component, a clear message is sent that invalidates previously sent music notes and clears the queue of current notes to be played (allowing these notes to be replaced with the updated music). An adjustable bar on the GUI also allows the user to change the speed of the music.

Music Playing Overview The music actuation component of the Rage with the Machine project featured four robotic arms autonomously playing xylophones in conjunction and communication with the vision component. Our system managed the planning, timing, and mechanical aspects of playing a desired piece of music and was written to be expandable to meet our resource budget. Planning Our planner served as a scheduler for our robotic arms. Each arm had its 'availability' to play a note mapped out in what we called a Planner Line, which was a sequence of actions to perform, each of which kept track of its start, 'hitting the key', and completion times, which we calibrated with trial information from the mechanical component of the project (see below). The planner would receive the sequence of notes it should play in an LCM (Lightweight Communications and Marshaling) message and proceed to distribute them to the Planner Lines via our arm allocation algorithm either by appending to the existing song or erasing and starting anew. As we continued to work, our algorithm grew more sophisticated to achieve our definition of optimality, which was maximizing the number of notes we could play in a given period of time. We implemented this in the following growth stages:

1. Query for the first free arm and use it to play the next note.

2. Query for all free arms and use the closest one of them to the next note to play as our choice.

3. To improve demoability by increasing the number of 'occupied' arms in a given period of time, we modified our algorithm to query for all free arms, then all of those which were tied for being closest to the next note, and finally chose the arm to use at random from that list.

4. Use method 3 as a quick and almost always successful greedy choice, but upon failure to add a note, achieve optimality by redistributing future notes via a branch and bound family algorithm. The planner would then maintain a thread that queried the Planner Lines to determine if they should play the note sitting at their individual playheads and signal the proper arm's state machine to do so.

Arm Interface Our arms were managed via an Arm class, which was responsible for knowing the configuration space locations of positions corresponding to hitting every key on our xylophones as well as positions directly above them and conducting smooth transitions between them. It also abstracted the mirroring of arms and their various positions from one side of a xylophone to the other. The process of making these movements smooth and reliably accurate took a number of design features to achieve. We had to establish a procedure that involved moving our arms one joint at a time in specific and varying orders to prevent collisions and unexpected behavior that arose by simply ordering the arm from place to place, a practice which unfortunately increased the delay between playing notes consecutively. For the purpose of achieving accuracy, we implemented extremely tight thresholds on what it meant for an arm joint to have 'arrived' at a location. The above system performed extremely well after tuning what it meant to be 'above' a key to decrease playing delays; however, our system encountered issues regarding servo overloads while attempting to sustain positions above keys at the edge of our arm's safe operating range which required a combination of tactics to solve. We determined that by lowering the maximum operating torque for a servo, we could allow it to last longer under stress, and wrote it into our code to achieve this when an arm was idle for a sufficiently long period of time. To achieve the best demoability and reliability for our system, we further determined that it would be safest to not only lower torque after being idle for long periods of time, but also to move the arm to a known safe position, above

the middle xylophone key. All of these tactics for sustainability were conducted blind to the planner, abstracted away and allowed for new instructions at a moment's notice. Mechanics The portion of the system we found we had the greatest difficulty refining was our mechanical interface with the xylophones. In the course of testing, we determined that in order to affect a sound off of a xylophone, it was necessary to not merely hit a note and release quickly with some artifact, but to pivot while doing so. After several design iterations and trials, we developed a rig which consisted of a 3" x ¾" hex bolt with one-degree-of-freedom for striking with a pivot to produce the appropriate ring from our xylophones.

Once the rig was constructed, we also had to ensure that we could actuate the servos in such a fashion that we could strike the key as intended. This required moving several joints in quick succession, and was a process that we changed several times over the course of development, and had the undesirable quality of often needing to be manually fine tuned for specific keys, rather than be based off of some mathematical model and adjustments. When we had settled on the motion of the arm, we ran a calibration program that ran through every possible motion of the arm between our pre-programmed positions and recorded the times taken over several trials for use in our planner. Success We view our system as a success. We were able to completely abstract the playing of music in a reliable planner that was able to achieve optimal playing capacity. We were able to play well known songs at a reasonable tempo with a relatively small number of robotic arms. The limitations of our system were either due to our resource budget or a consequence of being pressed for time. While our program could accommodate any number of robotic arms (and therefore play more complex and interesting music), we

were limited by the number available, the space required for each arm (4 sq. ft.), and the number of USB ports and number of AC power sockets (1 per arm). We could have further improved our system by reducing delays between notes, but were constrained by the amount of time it would've taken to find positions and movement patterns starting from closer above each note that still produced quality sound, and in the same way were limited from reducing servo overload issues by decreasing the operating radius of our arms. These known issues stated, we did manage to find solutions to both within the scope of programming, by making our planning algorithm optimal and our arms employ effective safety protocols, and as such, these issues were voided by demonstration day, and our system a success.