conductor heroweb.mit.edu/6.111/www/f2007/projects/ncheung... · • conductor - user interface -...
TRANSCRIPT
Conductor HeroNatalie Cheung, Yuta Kuboyama, Edgar Twigg
Introduction
Goal: Conduct an Orchestra based on Hand Movements
•Score transferred through a USB Port•User conducts the entire orchestra and smaller sections•Output orchestra music and graphical representation of the
orchestra on the screen
Overview
USBInput Score Data
Score Manager
Conductor
Mixer
enb enbpitch pitchpitch
dynamicsarticulationsection_sel
video_out
USB245M
note_data
beat_numbusynote_data(no beat)
video_input
enb
tempo
beat
stereo_left,stereo_right stereo_left, stereo_right
stereo_left, stereo_right
audio_out
reprogram
22
12
10
7 77
8 83
2x16 2x16 2x16
2x24Compressor
signal_sum
reset
reset
reset
ZBTnote_data
address address1919
34 34
2x16
clk2 clk2
clk2
clk2
clk2clk2
clk1
ZBT ZBT
write_enbread_enb
VideoDecoder
video_data
Instrument Manager
reset
USBInput Score Data
Score Manager
Conductor
Mixer
enb enbpitch pitchpitch
dynamicsarticulationsection_sel
video_out
USB245M
note_data
beat_numbusynote_data(no beat)
video_input
enb
tempo
beat
stereo_left,stereo_right stereo_left, stereo_right
stereo_left, stereo_right
audio_out
reprogram
22
12
10
7 77
8 83
2x16 2x16 2x16
2x24Compressor
signal_sum
reset
reset
reset
ZBTnote_data
address address1919
34 34
2x16
clk2 clk2
clk2
clk2
clk2clk2
clk1
ZBT ZBT
write_enbread_enb
VideoDecoder
video_data
Instrument Manager
reset
• Score Management- Music data handling- Takes numeric user signals and score data to determine what instruments play when
USBInput Score Data
Score Manager
Conductor
Mixer
enb enbpitch pitchpitch
dynamicsarticulationsection_sel
video_out
USB245M
note_data
beat_numbusynote_data(no beat)
video_input
enb
tempo
beat
stereo_left,stereo_right stereo_left, stereo_right
stereo_left, stereo_right
audio_out
reprogram
22
12
10
7 77
8 83
2x16 2x16 2x16
2x24Compressor
signal_sum
reset
reset
reset
ZBTnote_data
address address1919
34 34
2x16
clk2 clk2
clk2
clk2
clk2clk2
clk1
ZBT ZBT
write_enbread_enb
VideoDecoder
video_data
Instrument Manager
reset
• Conductor- User Interface- Generates numeric signals from user movement
• Score Management- Music data handling- Takes numeric user signals and score data to determine what instruments play when
USBInput Score Data
Score Manager
Conductor
Mixer
enb enbpitch pitchpitch
dynamicsarticulationsection_sel
video_out
USB245M
note_data
beat_numbusynote_data(no beat)
video_input
enb
tempo
beat
stereo_left,stereo_right stereo_left, stereo_right
stereo_left, stereo_right
audio_out
reprogram
22
12
10
7 77
8 83
2x16 2x16 2x16
2x24Compressor
signal_sum
reset
reset
reset
ZBTnote_data
address address1919
34 34
2x16
clk2 clk2
clk2
clk2
clk2clk2
clk1
ZBT ZBT
write_enbread_enb
VideoDecoder
video_data
Instrument Manager
reset
• Conductor- User Interface- Generates numeric signals from user movement
• Score Management- Music data handling- Takes numeric user signals and score data to determine what instruments play when
• Sound Synthesis- Given on/off and pitch- Generates actual sound
Time
24
ticks
beatbeatbeat
• 128 ticks per beat• At 50 bpm a tick lasts for 9.4ms
- humans can detect differences of ~20-30ms• Uneven note durations (e.g. triplets) won’t sound choppy
NoteData Format
Start Beat Start Tick Instrument Pitch Enable
0-11 (0-4095) 12-18 (0-127) 19-25 (0-127) 26-32 (0-127) 33 (0-1)
• Each NoteData takes up one 36 bit wide SRAM location• Two NoteData events per note (on and off) = 72 bits per note• 500 Measures in 4/4, average of 15 notes per beat = 2.06Mb
Score Management
note_data note_data
note_data
USBInput
ZBT Score Data
address
19
34 34
address
19
USB 245M
reprogram
22 12
Score Manager
reset
reset
reset
pitch 7 enable
beat
tempo
10
Tick Counter
pitch 7 enable pitch 7 enable
reset7
beat_num
current_tick
note_data note_data
note_data
USBInput
ZBT Score Data
address
19
34 34
address
19
USB 245M
reprogram
22 12
Score Manager
reset
reset
reset
pitch 7 enable
beat
tempo
10
Tick Counter
pitch 7 enable pitch 7 enable
reset7
beat_num
current_tick
puts USB data into
ZBT SRAM
note_data note_data
note_data
USBInput
ZBT Score Data
address
19
34 34
address
19
USB 245M
reprogram
22 12
Score Manager
reset
reset
reset
pitch 7 enable
beat
tempo
10
Tick Counter
pitch 7 enable pitch 7 enable
reset7
beat_num
current_tick
puts USB data into
ZBT SRAM
When beat_num > current beatbusy = highoutput new note_data
note_data note_data
note_data
USBInput
ZBT Score Data
address
19
34 34
address
19
USB 245M
reprogram
22 12
Score Manager
reset
reset
reset
pitch 7 enable
beat
tempo
10
Tick Counter
pitch 7 enable pitch 7 enable
reset7
beat_num
current_tick
puts USB data into
ZBT SRAM
When beat_num > current beatbusy = highoutput new note_data
If beat switches stateincrement beat_numwhile Score Data is busy
overwrite BRAM w/ new note_dataall data in BRAM now has same beat
If current_tick == tick in BRAMdo the note (turn on/off, set pitch, etc)
note_data note_data
note_data
USBInput
ZBT Score Data
address
19
34 34
address
19
USB 245M
reprogram
22 12
Score Manager
reset
reset
reset
pitch 7 enable
beat
tempo
10
Tick Counter
pitch 7 enable pitch 7 enable
reset7
beat_num
current_tick
puts USB data into
ZBT SRAM
When beat_num > current beatbusy = highoutput new note_data
If beat switches stateincrement beat_numwhile Score Data is busy
overwrite BRAM w/ new note_dataall data in BRAM now has same beat
If current_tick == tick in BRAMdo the note (turn on/off, set pitch, etc)
counts clock cyclescompares to #cyc/tickif a tick has passed
increment tick counter
USB Input (Data Reading Timings)
- excerpt from FTDI UM245R USB-Parallel
FIFO Development Module Datasheet
Tick Counter
Tick Counter
reset
7tempo
10
• Counts the # of clock cycles that have elapsed since the last tick• Calculates the # of clock cycles that should elapse before the next
tick based on the tempo• Increments the current tick when count = calculated
tempo4
beatsmin
_128 tickmin
_ 160minsec
= tempo_ 815ticksec
1tempo
_158sectick
_27,000,000 clksec
= 50,625,000tempo
clktick
Conductor
Conductor Module
Video Decoder• Detect LED using thresholds• Reject other colors• Store data into ZBT
Conductor Module
ZBT• Each frame
- ZBT1 collects information from video_data- Conductor module reads from ZBT2
• Every frame- ZBT1 <= ZBT2- ZBT2 <= ZBT1
Conductor Module
Conductor• Determine if there are 2 LEDs (conduct
the whole orchestra)• Determine position of LED• If position of LED remains the same for a
certain number of frames (based on a counter), the next time the LED moves to a different position, the beat enable is high
• Tempo is based on the velocity of the LED
Sound Synthesis
USBInput Score Data
Score Manager
Conductor
Mixer
enb enbpitch pitchpitch
dynamicsarticulationsection_sel
video_out
USB245M
note_data
beat_numbusynote_data(no beat)
video_input
enb
tempo
beat
stereo_left,stereo_right stereo_left, stereo_right
stereo_left, stereo_right
audio_out
reprogram
22
12
10
7 77
8 83
2x16 2x16 2x16
2x24Compressor
signal_sum
reset
reset
reset
ZBTnote_data
address address1919
34 34
2x16
clk2 clk2
clk2
clk2
clk2clk2
clk1
ZBT ZBT
write_enbread_enb
VideoDecoder
video_data
Instrument Manager
reset
•Represents a section of instruments (strings, brass,etc), and produces audio signals for the instruments at specified pitches
•Consists of instrument parameters, signal processor, sample look-up table
•Updates the dynamics and articulation when the section selector matches
•Outputs a stereo signal representing the location in the orchestra
•
Instrument Manager
USBInput Score Data
Score Manager
Conductor
Mixer
enb enbpitch pitchpitch
dynamicsarticulationsection_sel
video_out
USB245M
note_data
beat_numbusynote_data(no beat)
video_input
enb
tempo
beat
stereo_left,stereo_right stereo_left, stereo_right
stereo_left, stereo_right
audio_out
reprogram
22
12
10
7 77
8 83
2x16 2x16 2x16
2x24Compressor
signal_sum
reset
reset
reset
ZBTnote_data
address address1919
34 34
2x16
clk2 clk2
clk2
clk2
clk2clk2
clk1
ZBT ZBT
write_enbread_enb
VideoDecoder
video_data
Instrument Manager
reset
•Simply adds the left channel and right channel signals separately
•Produces a 24-bit stereo signal from adding 16-bit stereo signals
Mixer
USBInput Score Data
Score Manager
Conductor
Mixer
enb enbpitch pitchpitch
dynamicsarticulationsection_sel
video_out
USB245M
note_data
beat_numbusynote_data(no beat)
video_input
enb
tempo
beat
stereo_left,stereo_right stereo_left, stereo_right
stereo_left, stereo_right
audio_out
reprogram
22
12
10
7 77
8 83
2x16 2x16 2x16
2x24Compressor
signal_sum
reset
reset
reset
ZBTnote_data
address address1919
34 34
2x16
clk2 clk2
clk2
clk2
clk2clk2
clk1
ZBT ZBT
write_enbread_enb
VideoDecoder
video_data
Instrument Manager
reset
•Balances the output and prevents clipping or getting too quiet
•Applies gain < 1 for signals that would clip
•Applies gain > 1 for signals that are too quiet
•Sends a 16-bit stereo signal to the DAC, then to the speakers
Compressor
Memory and Resources
Memory and Resources
16 16 16 16 16 16 16
Sound sample lookup table
Memory and Resources
16 16 16 16 16 16 16
Sound sample lookup table
16 16 16 16 16 16 16
Sound sample lookup table
Memory and Resources
~4000 samples per instrument ~4000 samples per instrument
16 16 16 16 16 16 16
Sound sample lookup table
16 16 16 16 16 16 16
Sound sample lookup table
Memory and Resources
~4000 samples per instrument ~4000 samples per instrument
~15 instruments
16 16 16 16 16 16 16
Sound sample lookup table
16 16 16 16 16 16 16
Sound sample lookup table
Memory and Resources
~4000 samples per instrument ~4000 samples per instrument
~15 instruments
16 16 16 16 16 16 16
Sound sample lookup table
16 16 16 16 16 16 16
Sound sample lookup table
0.96Mbits out of 2.5Mbits on one FPGA
Finale
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Detect LED ColorWrite USB Input ModuleSample Audio & Playback
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Collect data and determine Beat and TempoWrite Score Data ModulePlayback audio at given pitch
Detect LED ColorWrite USB Input ModuleSample Audio & Playback
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Thanksgiving Break
Collect data and determine Beat and TempoWrite Score Data ModulePlayback audio at given pitch
Detect LED ColorWrite USB Input ModuleSample Audio & Playback
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Thanksgiving Break
Collect data and determine Beat and TempoWrite Score Data ModulePlayback audio at given pitch
Detect LED ColorWrite USB Input ModuleSample Audio & Playback
Write Score ManagerCreate Conductor Module for whole orchestra
Playback audio with Dynamics and Articulation Parameters
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Thanksgiving Break
Integration
Collect data and determine Beat and TempoWrite Score Data ModulePlayback audio at given pitch
Detect LED ColorWrite USB Input ModuleSample Audio & Playback
Write Score ManagerCreate Conductor Module for whole orchestra
Playback audio with Dynamics and Articulation Parameters
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Thanksgiving Break
Integration
Finishing Touches
Collect data and determine Beat and TempoWrite Score Data ModulePlayback audio at given pitch
Detect LED ColorWrite USB Input ModuleSample Audio & Playback
Write Score ManagerCreate Conductor Module for whole orchestra
Playback audio with Dynamics and Articulation Parameters
Sun Mon Tues Wed Thurs Fri Sat
Nov
Today
Dec
Timeline
30
2
25
18
17161514131211
9
19
26
3
10
20
27
4
11
21
28
5
12
22
29
6
13
23 24
7
14
1
8
15
Thanksgiving Break
Integration
Finishing Touches
Collect data and determine Beat and TempoWrite Score Data ModulePlayback audio at given pitch
Add orchstra sections and graphicsWrite MIDI to Bitfile ConverterWrite Mixer & Compressor Modules
Detect LED ColorWrite USB Input ModuleSample Audio & Playback
Write Score ManagerCreate Conductor Module for whole orchestra
Playback audio with Dynamics and Articulation Parameters
Thank you...