![Page 1: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/1.jpg)
1
J. T. Lindgren / OpenViBE 1
Jussi T. Lindgren, PhD [email protected]
HYBRID / Inria Rennes
HACKING
BOXES
IN A HURRY
![Page 2: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/2.jpg)
2
J. T. Lindgren / OpenViBE 2 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 2
Boxes?
Making a prototype box
C++ boxes, anyone?
Inside your C++ box
Everything is not a box
Contents
![Page 3: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/3.jpg)
3
J. T. Lindgren / OpenViBE 3 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 3
Boxes?
![Page 4: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/4.jpg)
4
J. T. Lindgren / OpenViBE 4 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 4
Recall: signal processing chains --
It’s made of boxes
Signal processing boxes
offered by OpenViBE...
...potentially
allow you to do
1001 things already
OpenViBE Designer
![Page 5: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/5.jpg)
5
J. T. Lindgren / OpenViBE 5 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 5
• Try to use existing boxes in a clever way
• Make a wish
• Hire an engineer
• Custom order from a company...
• Do a new box yourself (this talk)
But you want to do thing #1002...
![Page 6: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/6.jpg)
6
J. T. Lindgren / OpenViBE 6 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 6
Input(s)
Output(s)
What is a box
Box Streams have types
(signal, stimuli, etc...)
Box::process()
{
// get input
// do things
// put output
}
Signal processing happens here
(in C++)
![Page 7: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/7.jpg)
7
J. T. Lindgren / OpenViBE 7 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7
• 1. Prototype box with Matlab or Python wrapper
Or, in C++
• 2. Modify an existing box
• 3. Use Skeleton Generator to get a bare-bones box
Three ways to a box
![Page 8: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/8.jpg)
8
J. T. Lindgren / OpenViBE 8 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 8
Making a prototype box
![Page 9: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/9.jpg)
9
J. T. Lindgren / OpenViBE 9 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 9
• Use existing wrapper boxes
• Pass data between OpenViBE Matlab or Python
• Actual processing by custom .py or .m script
• Box::process() calls script for each chunk of data
• Pros & cons • + fast to prototype, no compilation or restart needed
• + no need for compiler
• - depends on matlab or python
• - potential issues if matlab/python versions change
Prototyping with Scripting Boxes
![Page 10: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/10.jpg)
10
J. T. Lindgren / OpenViBE 10 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 10
• Make three scripts: initialize.m, process.m, uninitialize.m
• Init/uninit called at the start/end of the playback
• Example of process.m:
• box_in is data from Designer, box_out goes to Designer
Prototyping: Matlab scripting box
function box_out = process(box_in)
disp(’process called.')
% do things to box_in
box_out = box_in;
end
![Page 11: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/11.jpg)
11
J. T. Lindgren / OpenViBE 11 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 11
Prototyping: Matlab scripting box, contd.
% do things to box_in
% we iterate over the pending chunks on input 1 (here SIGNAL)
for i = 1: OV_getNbPendingInputChunk(box_in,1)
% pop the first chunk to be processed, box_in used as output
[box_in, start_time, end_time, matrix_data] =
OV_popInputBuffer(box_in,1);
% signal is in matrix_data, [channels X samples] orientation
% …
% replicate header
box_in.outputs{1}.header = box_in.inputs{1}.header;
% The chunk dates are the same as the input chunk
box_in = OV_addOutputBuffer(box_in,1,start_time,end_time,matrix_data);
end
box_out = box_in;
![Page 12: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/12.jpg)
12
J. T. Lindgren / OpenViBE 12 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 12
• Examples & tutorials: • box-tutorials/python/
• box-tutorials/matlab/
• http://openvibe.inria.fr/tutorial-using-python-with-openvibe/
• http://openvibe.inria.fr/tutorial-using-matlab-with-openvibe/
Prototyping: more info
![Page 13: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/13.jpg)
13
J. T. Lindgren / OpenViBE 13 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 13
C++ boxes, anyone?
![Page 14: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/14.jpg)
14
J. T. Lindgren / OpenViBE 14 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 14
• Pros & cons • + C++ is the implementation language of OpenViBE
• + no external dependencies needed
• + code can be as fast as you can make it
• + no added python/matlab compatibility issues
• - needs a compiler (free ones are ok)
• - modifications require compilation and restart
• - C++ prototyping can be slow
• Getting started with C++
http://openvibe.inria.fr/build-instructions
C++ boxes
![Page 15: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/15.jpg)
15
J. T. Lindgren / OpenViBE 15 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 15
Simple:
• Decide an old box that is close to the idea of your new box
• Locate its source code from the source tree (’plugins/’)
• Hack away
Your starting point:
• Has some ’interface glue’ already in place and works
• May not have everything you need
• May have things you don’t need
A. Modifying an existing C++ box
![Page 16: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/16.jpg)
16
J. T. Lindgren / OpenViBE 16 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 16
- A box needs 30+ lines of ’glue code’
- More for boxes having parameters, inputs, outputs, etc
- Skeleton Generator can make the basic box code
- You get a .cpp/.h pair
- Add your custom processing code to - ::initialize() { // ... }
- ::process() { // ... }
- ::uninitialize() { // ...}
B. Skeleton Generator tool
![Page 17: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/17.jpg)
17
J. T. Lindgren / OpenViBE 17 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 17
Tabs to specify
various properties
of your box
When done:
Check, then
Generate!
![Page 18: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/18.jpg)
18
J. T. Lindgren / OpenViBE 18 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 18
• Step-by-step tutorial
http://openvibe.inria.fr/tutorial-1-implementing-a-signal-
processing-box
Time to hack into the box...?
B. Skeleton Generator tool
![Page 19: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/19.jpg)
19
J. T. Lindgren / OpenViBE 19 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 19
Inside your C++ box
![Page 20: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/20.jpg)
20
J. T. Lindgren / OpenViBE 20 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 20
• Each box needs to be described with e.g.
• Box identifier (random 64bit number)
• Box name
• Author
• Inputs & outputs
• Box parameters
• ...
• These are in the box .h file
• The header may also define a ’listener’ : code that reacts to box
parameter changes when made by the user
Box description
![Page 21: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/21.jpg)
21
J. T. Lindgren / OpenViBE 21 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 21
Box settings
Settings registered in the .h
get a GUI automatically
Save & load to scenario xml
also automatic
Getting value of an uint64 param at slot 3 (index from 0):
uint64 order
= FSettingValueAutoCast(*this->getBoxAlgorithmContext(),2);
![Page 22: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/22.jpg)
22
J. T. Lindgren / OpenViBE 22 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 22
- Data needs to be encoded to travel between boxes
- Codecs capsulate the data in box input/output
- Each stream needs its own codec
Codecs
Box Encoders
Decoders
In from kernel
Out to kernel
![Page 23: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/23.jpg)
23
J. T. Lindgren / OpenViBE 23 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 23
Codec example: initializing
myBox::initialize() {
// TSignalDecoder< myBox > m_signalDecoder; // in .h
// TSignalEncoder< myBox > m_signalEncoder; // in .h
m_signalDecoder.initialize(*this);
m_signalEncoder.initialize(*this);
// make decoder and encoder share buffer & sampling rate
m_signalEncoder.getInputMatrix().
setReferenceTarget(m_signalDecoder.getOutputMatrix());
m_signalEncoder.getInputSamplingRate().
setReferenceTarget(
m_signalDecoder.getOutputSamplingRate());
}
![Page 24: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/24.jpg)
24
J. T. Lindgren / OpenViBE 24 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 24
Codec example: processing
myBox::process() {
for(uint32 i=0; i<nChunks; i++) {
// decode the chunk i on input 0
m_signalDecoder.decode(0,i);
if(m_signalDecoder.isBufferReceived()) {
IMatrix* l_pMatrix = m_signalDecoder.getOutputMatrix();
// do stuff with the data in l_pMatrix, [chns x samples]
// ...
// send out
m_signalEncoder.encodeBuffer(0);
l_rDynamicBoxContext.markOutputAsReadyToSend(0,
l_rDynamicBoxContext.getInputChunkStartTime(0, i),
l_rDynamicBoxContext.getInputChunkEndTime(0, i));
}
}
![Page 25: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/25.jpg)
25
J. T. Lindgren / OpenViBE 25 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 25
Everything is not a box
![Page 26: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/26.jpg)
26
J. T. Lindgren / OpenViBE 26 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 26
• Device driver is not a box • Put it in Acquisition Server instead
• Algorithm is not a box • If a routine is shared by several boxes, consider it an ’Algorithm’
• Stimulators may not want to be boxes • Should be external apps outside Designer
• Send their triggers to Acquisition Server
• For timing reasons, avoid making stimulators as boxes
![Page 27: BOXES IN A HURRY - OpenVibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfJ.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7 • 1. Prototype box with Matlab or Python wrapper](https://reader034.vdocuments.mx/reader034/viewer/2022042123/5e9ecf9d4ce3ed360f672326/html5/thumbnails/27.jpg)
27
J. T. Lindgren / OpenViBE 27 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 27
Here we have skipped a lot of details
Box coding is best learned by doing
More about at
http://openvibe.inria.fr
That’s it!