Java and the KinectFRC 2012 Kickoff

Team 1279 ColdFusionJanuary 7, 2012

Presentation Overview

• Kinect with Java focused– C/C++ looks similar

• FRC focused– Not the full Kinect SDK

• Quick and Dirty– Practical Example, no Bells or Whistles

• Simple End-to-End Application– Driver, WPILibJ, Application

Java Application

• Sample Application– Xylophone1279

• Demo• Code walkthrough

– Resources:• Source & Presentation

– www.coldfusion1279.com

• Contact Info– programming@coldfusion1279.com

Java Application

• Beta KinectStick Class– Y-axis Only

• Standard -1.0 - 1.0 range (-128 to 127)

– Wrist – Shoulder Angle• Solves Some Issues• Restrictive

• Modified KinectStick Class– X, Y, and Z axes

• Standard -1.0 - 1.0 range (-128 to 127)

– Skeleton Joint Values

Java Application

import edu.wpi.first.wpilibj.KinectStick;

public class Player implements XyloConstants {public KinectStick rStick, lStick;

public Player(){rStick = new KinectStick(2);lStick = new KinectStick(1);

}}

Java Application

void playKinectStick(){double lAngle, rAngle;

while (ds.isEnabled() && ds.isOperatorControl()){

lAngle = 0.0;rAngle = 0.0;lAngle += (78.0 + lStick.getY() * 40.0);rAngle += (62.0 + rStick.getY() * 40.0);

…}

}

Java Application

• Conceptualization– Analog & Real-time

• Map Movements to Actions– Natural– Full Analog Spectrum

– Digital & Parsing• Encode Commands

– Un-natural– Precise

Java Applicationvoid playKinectRaw(){ double rX = 0, rY = 0, lX = 0, lY = 0;

while (ds.isEnabled() && ds.isOperatorControl()){ rX = rStick.getX();

if (rX <= -0.2) rElbow.setAngle(RNOTES[0]); else if (rX > -0.20 && rX <= -0.05)

rElbow.setAngle(RNOTES[5]);… else if (rX > 0.70 && rX <= 0.85)

rElbow.setAngle(RNOTES[11]); else if (rX > 0.85) rElbow.setAngle(RNOTES[12]);

if (rStick.getY() < 0.25) rWrist.setAngle(RWRIST_DOWN); Timer.delay(STRIKE); rWrist.setAngle(RWRIST_UP); Timer.delay(0.25);

}}

Java Applicationvoid playKinectNL(){

while (ds.isEnabled() && ds.isOperatorControl()){ rX = rStick.getX(); rY = rStick.getY(); lX = lStick.getX(); lY = lStick.getY();

if (rX <= 0) rxIndex = 0; else rxIndex = 1;

index = (rxIndex << 3) | (ryIndex << 2) | (lxIndex << 1) | lyIndex;//merge into 0-15

switch(index){ case 0: //0000 r quadrant 3, l quadrant 3 rElbow.setAngle(RNOTES[RSAFE]); lElbow.setAngle(LNOTES[C0]); play = true; break; … default: //unknown symbol, don't play anything lElbow.setAngle(LNOTES[LSAFE]); rElbow.setAngle(RNOTES[RSAFE]); play = false; } if (play && (index < 6)) lWrist.setAngle(LWRIST_DOWN);

Timer.delay(STRIKE); lWrist.setAngle(LWRIST_UP); } }

Java Application

• Implementation– Bounds

• Don’t Map to the Screen

– Those Pesky Humans• Size• Distance• Sneezing

– Noise• Filtering• Averaging

WPILibJ

• Kinect class• KinectStick class

– Axes• X, Y, Z, Throttle, Twist, Custom

– Buttons• Standard 15• Button 9 reserved

– Data Representation• Built for sbyte to double• Granularity vs. Noise

WPILibJ

• Modifying Source Code– NetBeans Considerations

• WPILibJ as a project

– Compiling• Don’t Clean, Build ALL takes time!

– Save Copies!• If you break something…• Plug-in Updates

WPILibJ byte joystick1[] = new byte[6]; short button1; byte joystick2[] = new byte[6]; short button2;

public void read() { backingNativeMemory.getBytes(0, joystick1, 0, 6); button1 = backingNativeMemory.getShort(6); backingNativeMemory.getBytes(8, joystick2, 0, 6); button2 = backingNativeMemory.getShort(14); }

WPILibJ public double getY(Hand hand) { byte tmp; getData();

if (m_id == 1) { tmp = tempOutputData.data.joystick1[1]; return normalize(tmp); } else { tmp = tempOutputData.data.joystick2[1]; return normalize(tmp); } }

private double normalize(byte rawValue) { if(rawValue >= 0) return rawValue / 127.0; else return rawValue / 128.0; }

FRC Kinect Server

• Kinect Overview– Depth Capture

• Skeleton with X, Y, and Z (distance)

– Video Capture• Built-in Camera, no WPI Kinect Image API (yet?)

– Audio Capture• Microphone Array, FRC Friendly?

FRC Kinect Server

• Modifying Source Code– Gesture file

• Program Files (x86)\FRC Kinect Server\Source\Kinect Server

– Writeable Joystick Class• Axes• Buttons

– Compiling and replacing• Kill the Service!

– Save Copies!

FRC Kinect Server

• Limitations– Noise

• Apps ‘jumpy’

– Bandwidth• 6 Bytes, and a 16-bit Integer (bit 9 reserved)• Full Skeleton X, Y, Z coming

– Data Types• sbyte

FRC Kinect Server

• Programming Environment– Visual Studio/Express (free)

• IDE similar to NetBeans or Eclipse

– C# (free)• Indistinguishable from C++ (need C#!)• Reasonable Learning Curve from Java• http://www.microsoft.com/visualstudio/en-us/products/2010-editions/

visual-csharp-express

– Kinect SDK beta v.2 (free)• http://kinectforwindows.org/

FRC Kinect Server

x16 = (Int16)(skeleton.Joints[JointID.HandRight].Position.X * 128); if (x16 > 127) x16 = 127; if (x16 < -128) x16 = -128;

rightAxis[(uint)WritableJoystick.Axis.X] = (sbyte)x16; rightAxis[(uint)WritableJoystick.Axis.Y] = (sbyte)y16; rightAxis[(uint)WritableJoystick.Axis.Z] = (sbyte)z16; rightAxis[(uint)WritableJoystick.Axis.Throttle] = 0; rightAxis[(uint)WritableJoystick.Axis.Twist] = 0; rightAxis[(uint)WritableJoystick.Axis.Custom] = 0;

//Use "Button 9" as Kinect control enabled signal buttons |= (ushort)WritableJoystick.Buttons.Btn8;

joy[0].Set(leftAxis, buttons); joy[1].Set(rightAxis, buttons);

Questions?

?

Thanks!

Best of Luck to All!