intelligent effects for automatic mixing2.pptjosh/documents/intelligenteffects...system polarity -...
TRANSCRIPT
![Page 1: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/1.jpg)
Josh ReissCentre for Digital MusicQueen Mary, University of London
INTELLIGENT DIGITAL AUDIO EFFECTS FOR AUTOMATIC MIXINGEFFECTS FOR AUTOMATIC MIXINGAll demonstrations available fromAll demonstrations available fromwww.elec.qmul.ac.uk/digitalmusic/automaticmixing/
![Page 2: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/2.jpg)
MotivationMotivation
There’s no reason why a
Musicians who’d rather get on with making music than
get too deep into yband recording using reasonably
conventional instrumentation shouldn’t be EQ’d and balanced
g pengineering[audio interfaces can] come
with a 'gain learn' mode... DAWs could optimise their
automatically by advanced DAW software.
DAWs could optimise their own mixer and plug-in gain
structure while preserving the same mix balance.
![Page 3: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/3.jpg)
ClassificationClassificationIs this art or engineering?engineering?
![Page 4: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/4.jpg)
Approachl h l kCreate tools that automate complex mixing tasks
Develop effects which dynamically adapt to multi‐channel input of a mixer
Analyse content of each channel with respect to otherAnalyse content of each channel with respect to other channels
Must be suitable for automatic livemixing tasksMust be suitable for automatic live mixing tasksOperate real‐time
Cross Adaptive EffectsPreserve system stability
Cross-Adaptive EffectsAlso called inter-channel dependent or pMIMO (multi-input / multi-output) effects
![Page 5: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/5.jpg)
Cross-Adaptive Effects
ch1 to chN are audio signals.
fv1 to fvN are features extracted from audio h lchannels
t t lcv1 to cvN are control vectors that drive digital
di
General cross‐adaptive effect diagram
audio processors
General cross adaptive effect diagram
![Page 6: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/6.jpg)
Automatic mixing host
Normalised-EQTime Offset Correction
Auto Panner Auto Fader
ChannelChannel Enhancer
Automatic Mixing Host
Hardware Control Surface
![Page 7: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/7.jpg)
Automatic stereo pannerAutomatic stereo panner
![Page 8: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/8.jpg)
Automatic Panner
Side chain processing for real time applicationstime applicationsMultiple channel processing dependencydependencyAccumulative spectral decomposition
M lti i tMulti-input processing
dependency
![Page 9: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/9.jpg)
Automatic Panner Stereophonic Masking
Di ti l t l kiDirectional spectral masking
Panning space directional masking i i i ti
Reducing directional spectral
minimization
Reducing directional spectral masking improves intelligibility.
![Page 10: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/10.jpg)
Automatic Panner Results
Lissajous plot of sterophonic outputj p p pof automatic pannerPanning factor for 4 input channels using
the automatic panner
Spectral decomposition rules manage to converge and adapt to inputs.Reduction in directional masking maximizes stereophonic separation.g p pThe automatic panner requires a better priority scheme
![Page 11: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/11.jpg)
Automatic Panner Evaluation
![Page 12: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/12.jpg)
DEMONSTRATIONDEMONSTRATIONAutomatic Panner
![Page 13: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/13.jpg)
Spectral EnhancerSpectral Enhancer
![Page 14: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/14.jpg)
ObjectiveMinimising signal masking through the characterisation of spectral channel interdependenceof spectral channel interdependence
What do we define as spectral masking?
Spectral masking is the loss of spectral content perception of one or more channels when they are mixed together.
not the same as psychoacoustic maskinglt f i l d i i th f biresults from signal and noise occupying the same frequency bins
noise spectrum masks spectrum of the signal
![Page 15: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/15.jpg)
Inter‐channel dependent spectral enhancer
Master channel ‘writes’ spectral enhancing contour and has no signal processing applied to it.applied to it.
Slave channels ‘read’ spectral contour andspectral contour and receive a processing
t l lcontrol value.
M t h l h ti i f ti di•Master channel has continuous information regarding properties of all channels
![Page 16: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/16.jpg)
Understanding the displayUnderstanding the display
Spectral enhancing Filter bank division =k2contour
AttenuationAttenuation
Freq encFrequency
![Page 17: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/17.jpg)
Attenuation Q
+ Affects all Ch l j t-+ Channels just like a normal ducker would do
Bypass
+Affects Only Channels just+- Channels just with same spectral content
![Page 18: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/18.jpg)
DEMONSTRATIONl hSpectral Enhancer
Un-masked
MaskingLimitLimit
Masked
![Page 19: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/19.jpg)
A i i i i iAutomatic gain maximisation and feedback preventionand feedback prevention
![Page 20: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/20.jpg)
Feedback preventionp
Feedback Suppressors: Frequency shiftingFrequency shifting
Auto‐notch filtering
...
Goal of mixing engineer to adjust frequency response while staying below threshold for feedbackFlatten responseDon't Treat the Symptoms-Treat the Cause!p
Maximise gain
Need gain limiter which does not introduce artifacts or
Treat the Cause!
Need gain limiter which does not introduce artifacts or prevent EQ
![Page 21: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/21.jpg)
Feedback explained
H ( ) i f d f d
Given the following acoustic model:HETOT(x) is feed‐forward transfer function of the systemsystem
product of individual transfer functions of microphone, equaliser, amplifier and speaker.
HATOT(x) is acousticHATOT(x) is acoustic feedback transfer function of the system
The system will introduce undesired acoustic feedback if:HATOT(x)·HETOT(x)>1
![Page 22: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/22.jpg)
Feedback preventionGiven the following acoustic model:
if equalizer gain is 0dB when flat and system is on marginal condition before undesired feedbackfeedback,
can still use the equalizer by forcingequalizer by forcingHeEQ(x) to remain below 0dB.
![Page 23: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/23.jpg)
![Page 24: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/24.jpg)
DEMONSTRATIONDEMONSTRATIONFeedback prevention / Gain maximisationp
![Page 25: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/25.jpg)
Multi-Channel Time Offset Correction for Mixing
![Page 26: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/26.jpg)
Undesired destructive interference of i l k b flitt i isignals, known as comb-flittering, is
common in day-to-day audio mixing.Comb-filtering caused by:
U i th i h t d i l•Using more than one microphone to record a single sound source
•Mixing an electric instrument direct box signal with its acoustic amplified signalacoustic amplified signal.
•Use of parallel signal paths with different latency timesp g p y
•Digital implementations which do not compensate for difference in latency time of different algorithms
![Page 27: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/27.jpg)
Comb Filter
Peaks at frequencies1/d, 2/d, 3/d…
Notches at frequencies1/(2d ), 3/(2d), 5/(2d)…
Where d is delay betweenWhere d is delay between signals
Comb Filtering effect: a white noise signal added to a copy of itself, delayed by 1ms
![Page 28: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/28.jpg)
Problem
++
![Page 29: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/29.jpg)
Aim
++
Delay / PolarityCorrection
++Delay / PolarityCorrection
![Page 30: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/30.jpg)
Implementation
Delay - distance between impulse response absolute maxima and the instant pwhen the impulse was input into the system
Polarity - whether the impulse response is positive or negative will determine the polarity of the system
General algorithm flow diagram for an automatic mixture cross-adaptive time offset corrector.
![Page 31: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/31.jpg)
Results
Impulse response of signal before correction (top) and after the correction (bottom) for a weakly correlated signal.( ) y g
![Page 32: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/32.jpg)
Results
Impulse response of signal before correction (top) and after correction (bottom) for a highly correlated signal.( ) g y g
![Page 33: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/33.jpg)
DEMONSTRATIONDEMONSTRATIONTime offset and polarity correctionp y
![Page 34: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/34.jpg)
Automatic fadersAutomatic faders
![Page 35: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/35.jpg)
Auto-gain
Auto-faders
![Page 36: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/36.jpg)
Auto‐faders
SPL Application
![Page 37: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/37.jpg)
DEMONSTRATIONDEMONSTRATIONAutomatic Faders and Gain Control
![Page 38: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/38.jpg)
DEMONSTRATIONPutting it all togetherNO automatic toolsAuto‐Gain ONAuto‐Fader ONAuto‐Pan ONVarying Auto‐Pan control Normalise Equaliser ONqTransfer function of EqualiserEqualiser FlatqEqualiser BoostALL Automatic tools Runningg
![Page 39: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/39.jpg)
Conclusions• Automatic Mixing is underdeveloped
• can be achieved by using side processingy g p g
• Automatic stereo panning reduces spatial/spectral masking
• Cross‐channel gain maximisation can avoid feedbackCross‐channel gain maximisation can avoid feedback
• Fader settings can be automated for equal loudness
• E h h l i i t d d t t l f t• Enhance a channel using inter‐dependent spectral features• Minimises spectral masking of a channel by other channels
ll l f ff b d f d• Allow user control of effect by modifying Q and attenuation parameters
• Time offset corrector reduces comb‐filtering due to delays betweenaudio signals in a mixeraudio signals in a mixer• correct delays with +/‐4 sample accuracy, optimize polarity for all channels
N d d i i h l d d ff hNeed a cross‐adaptive, inter‐channel dependent effect host‐VST3?
![Page 40: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/40.jpg)
Current WorkA t ti M it Mi i t d f AES J lAutomatic Monitor Mixing – accepted for AES Journal
Reverse engineering the mix – submitted to DAFX
Adaptive noise gating – submitted to DAFX
Automatic fader and gain control – submitted to WASPAA
Turing Test
FutureWorkHardware and VST3 implementations
d i i
Future Work
Targeted mixing
Auto‐EQ
automatic ambisonics source placementbased on musician position tracking
B d t tBased on source content
Feedback prevention with environmental measurements
![Page 41: Intelligent effects for automatic mixing2.pptjosh/documents/Intelligenteffects...system Polarity - whether the impulse response is positive or negative will determine the polarity](https://reader035.vdocuments.mx/reader035/viewer/2022081621/612ed2791ecc515869430e43/html5/thumbnails/41.jpg)
ThanksThanksEnrique Perez Gonzalez
Yonghao WangMichael TerrellMichael Terrell
Martin Morrell
Daniele Barchiesi
Demonstrations: www.elec.qmul.ac.uk/digitalmusic/automaticmixing/Research group:
www.elec.qmul.ac.uk/digitalmusic/q / g /automaticmixing/audioengineering.html