using galileo and raspberry pi written in python script language semesterial project students :...
TRANSCRIPT
Smart Home DesignBased On Voice Recognition
Using Galileo and Raspberry PiWritten in Python script language
Semesterial project
Students : Matan Cohen & Sefi CohenSupervisor : Mony Orbach
Spring 2014
Most smart homes today require a remote instead of being hands-off
We want to automate home life
We want to control appliances from afar without remotes
Smart home products are expensive
Smart home products aren’t extendible
Motivation
Project GoalsTo build an infrastructure of an interactive home
It’ll need to be hardware extendible – Can add more components in the future easily
It’ll need to be software extendible – Can add more software features easily
To build a prototype which turns on a led with the right command.
Voice recognition based commands Since most products out there require a remote (which is a hassle),
ours will be hands-off.
Communicate with the user – Using TTS we can have the system speak back and give the user indications on actions being performed
To experience designing hardware and software product from scratch.
To get our hands dirty with both hardware and programming at the hardware & software environments.
Project Vision Sketch (for example)
LIVINGROOMINTERNET
INTERNET
INTERNET
BEDROOM
WHEREVER
Raspberry Pi
Raspberry Pi
SERVER(RP or PC)
Galileo
Electric
Blinds
MIC
MIC
Galileo
TV
Hardware EnvironmentRaspberry Pi Galileo
• Based on ARM• Linux based• Open Source• On board GPU• On board sound card• Ethernet
• Based on Intel Quark• Arduino based• Open Source• Ethernet
Software EnvironmentRaspberry Pi - Linux environment
Using Python script language.
Galileo – Linux environmentUsing implemented library of Arduino in Python.
Both are controlled via SSH protocol.
Hardware Architecture
Linked through internet
Software ArchitectureRaspberry Pi - Linux environment
Listener program – record the voice from the microphone and send the wav file to voice recognition unit .
Voice recognition unit – based on WIT learning speech recognition web service online.
RabbitMQ SERVER – Linux environment Server SW– receive and send the analyzed messages from
client to client. Command Parser– parse the received string and send a
signal to the right Galileo to run the right add-on script
Galileo – Arduino environment Add-ons scripts written in python using Arduino library . Scripts like lights and blinds are already implemented.
Configuration fileWe need to initialize the HW component with
information that it needs to function properly.When we add a new component (recording like Rpi or
commands invoker like Galileo) we have to write a “system config” file which contain the following : Mic – insert microphone dry details : Format, Rate,
Channels number and Chunk size. Voice Energy Threshold – which the microphone will
start recording from this energy and above. WIT Key – that serial number is needed for connecting
to the WIT web service. Name – The component's name. It takes part in
routing the message to the right component via RabbitMQ.
Commands – The addons that are supported on this component.
External SW ComponentsRabbitMQ – Server Implementation
There is a lot of different implementation of servers – we choose “Routing” which in this implement every target has a queue of his own and he gets only the messages that’s addressed to him
In the code the RabbitMQ is called “AMQP” (Advanced Message Queuing Protocol) which is the name of the protocol which RabbitMQ implements .
External SW Components cont.wit.ai – Voice recognition web service
enables developers to add a language interface to their app or device in minutes.
It’s a learning system and as one we need first to provide a few examples of queries to get the expected responses.
We use their API to send queries and get the response as structured information back (JSON)
Py Galileo Library – Python library for Galileo board Helps us write code in python that control the pins on
the board like Arduino does.
Main Software ComponentsText To Speech Module – Given a string, create
an audio file using a Google TTS service that says that string aloud. Plays that file.
Wit Query Analyzer – using WIT web service. Given an audio recording of a command, the WIT service returns a JSON that can be worked with containing information about what was said.
Client - Manages sending parsed commands to the correct HW component.
Main Software Components cont.
Receiver - Receives a command from another HW components and starts processing it by sending it to the Addon Manager
Addon Manager – when the Receiver gets a command the Addon Manager is responsible for checking the validity of the command and executing the right script
Flow of events - System POVVoice message is received in Raspberry Pi
The message is analyzed by WIT web service online and a json file returns as a string
This json file is wrapped by private object called wit_response – locally on Raspberry Pi
From Rpi the analyzed message is sent to the right target by the RabbitMQ server which send the message according to the target’s name
Flow of events - System POV cont.The target (Galileo or other Rpi) receives the wit
response and transfers it to the add-on manager
The add-on manager that runs on the target checks if (in the system configuration file) the target supports the command. It does this by checking if the "intent" field in the wit response object is found in the "commands" field list in the component's JSON configuration file.
If there is a match the add-on manager executes the right script (and pass to the add-on the original wit response since the wit response includes information needed by the addon itself as well.)
Demonstration Equipment1 Raspberry pi
2 Galileo
1 PC (the SERVER)
1 SD card (for Rpi)
2 mini SD card (for Galileo)
1 USB sound card
3 Ethernet cables
2 USB microphone
1 Speakers
1 Led circuit
1 Buzzer circuit
Demonstration Environment
RP Galileo
LED
MIC
SPK
INTERNETRabbitMQ SERVER
Galileo
BUZZ
SSH Connection over TCP/IP
Ethernet Connection
USB Connected Electric
CircuitElectric Circuit
Starting DemonstrationBurning the Linux version on SD cards for all
the hardware components.Connecting the all the Demo environment as
the sketch in the last slide.Connect the Rpi and Galileo through direct
Ethernet connection.Run the command Nmap from any PC in order
to find the hardware IP’sConnected to all the hardware components
through SSH protocol over TCP/IP.
Starting Demonstration cont.Write a configuration file for each
component : Measure the energy threshold using script. Insert the Microphone factory properties . Inset the WIT key. Give a name to the component. Enter the commands that are available for this
“room”/HW component.
Execute the RabbitMQ server on PC or on one of the Raspberry Pi and “tell” the other components his IP.
Execute on each Raspberry Pi in the all system the client “audio_client” program.
Execute on each Galileo the “receiver_client” program.
FAQWhy python ?
It’s a strong language with all the libraries we need. It’s easier to write code in.
Why do we need the server in the middle ? Because we want to make Hardware extendible infrastructure and we
need a server to control all the units in the house .
How can we configure new microphone ? In the system configuration file we can edit the mic factory settings
(format, rate, channels number and chunk size).
How can we configure energy threshold in the config file ? Special script written for that matter ,named energy_threshold.py,
which measure the silence and loud energies in the room (will be shown in demo).
FAQ cont.What about the delay of the all system?
We add to our software “performance counters” for getting information about the time of each action.
The voice recognition proccess (using WIT) is taken between 6 to 12 seconds depend on the network speed and the load on WIT server.
The sketch upload to the right galileo device is taken between 1 to 2 seconds.
How can we do the initialize step hands off ? At our home we could give the server static IP and write the IP in
the configuration file so that all the other devices will know who is the server.
Then we need to define in each device the right scripts that’s needs to be uploaded from the start.
Future UpgradesMake the voice recognition SW run locally
It will shorten the delay significantly. “Sphinx” – library that implement voice recognition locally.
Use Zero-configuration networking In our project its important that each device is getting an IP
address. In this methodology DHCP server is not necessary for
network configuration and there is no need in manual operator intervention or other special configuration servers.
Make the all system wireless
Future Possible ProjectsAdd camera and use facial recognition
algorithms If its not someone familiar turn on the alarm. If its someone familiar turn on the A/C and his favorite
music.
Add a web application of user interface Easy to configure hardware components. Easy to add software add-ons. Control away from home will be enabled.
Questions?
Thank you all very much!!!