Ideas for Improving Pixy’s Color Classification Performance Contents Pixy’s signature definition and color classification .......................................................................................... 1

The color space pixy uses for color signature classification ......................................................................... 1

Signature Definition Algorithm ................................................................................................................... 4

Color Classification ..................................................................................................................................... 4

Color lookup table (LUT) based preselection .............................................................................................. 5

A less efficient color model is used for the color LUT .............................................................................. 5

Weaknesses and Improvements .................................................................................................................... 6

The minimum brightness limit acts differently on pure and mixed colors ................................................... 6

Surprising effects of PixyMon's signature tuning sliders ............................................................................. 6

Mean value as signature anchor is not robust against outliers .................................................................... 8

A more symmetric color model for signature definition and LUT based preselection .................................. 9

Appendix ......................................................................................................................................................13

Example code for the approximated nYUV color model (for LUT preselection) ..........................................13

Pixy’s signature definition and color classification The color space pixy uses for color signature classification I was quite surprised that Pixy does the color detection not in the intuitive Hue-Saturation-Value (HSV) color model, as documented in the wiki, but in the chrominance u-v-plane of some kind of luminance normalized YUV model, which is actually quite different from the original YUV color model definition … but better suited for color classification.

u=(r-g1)/(r+g1+b) v=(b-g2)/(r+g2+b)

The corresponding code can be found in function Bool IterPixel::nextHelper(…) defined in ...\common\colorlut.cpp. Assuming that the two Bayer green filters g1 and g2 are not that different1, this simplifies to y=r+g+b u=(r-g)/y v=(b-g)/y

1 I don't know if this assumption is correct. But if it is not, then also the final signature check in Blobs::runlengthAnalysis() defined in file ...\common\blobs.cpp in might have a problem, as there the u and v values are divided by a common y (line 158). Why are the u and v values calculated differently in the signature generation and application?

Plotting some HSV spectra to this u-v plane illustrates the beneficial feature of this color model for color classification. First the full spectrum, HSV/RGB {[0..1],[0..1],[0..1]}. Left side: The chrominance u-v-plane on the left (u,v=[-1..1]) Right side: y plotted verses r=sqrt(u²+v²)

The nice feature is that the color saturation grows from 0 in the center of the u-v-plane to 1 at the edges of the distribution. The darker colors are not visible in the left plot, because they are overlaid by the bright ones. If one plots only the 100% saturated HSV color the distribution looks like this.

The full saturated colors, Pixy is looking for, stacks at the edges of the color distribution, independently of being gloomy or bright. Plotting only the low saturated HSV colors, i.e. S<50%, shows that all uninteresting hard to be classified colors, highlights and shadows, are all located in the center region of the u-v-plane.

This nice separation of the high and low saturated colors makes this model quite suitable for Pixy's color classification. It is an effect of the "normalization", i.e. the division of u and v by y. The original YUV model

y = wr*r + wg*g + wb*b u = (b-y)/(1-wb) v = (r-x)/(1-wr) with wr=0.299 wb=0.114 wg=1-wr-wb=0.586

doesn't provide this feature: Even though being more symmetric, the low and high saturated colors are kind of mixed up in the u-v-plane. The first plot shows the distribution of the full saturated HSV colors, the second one the distribution of HSV colors with a saturation lower than 50%. The gloomy but full saturated colors mix up with all kinds of low saturated colors. Thus reliable color classification in gloomy regions would be rather difficult.

Signature Definition Algorithm A color signature is defined by a simple rectangular region in the u-v plane of Pixy's distinct "normalized" YUV color model described above.

Given a set of continuous pixels Pixy performs the following steps to generate a color signature. It calculates for each of the pixels the u, v and y values. It skips all pixels with a luminance y lower than a fixed lower limit yMin=0.05*3*255=38, It calculates from the remaining set the u and v mean values uMean and vMean. It determines with a binary search algorithm upper and lower limits that encloses 90% of the

remaining pixels: uMin90, uMax90, vMin90 and vMax90 Using these values and the signature range sRange (adjustable in PixyMon) Pixy defines the final

selection box applied in the u-v-plane as follows: o uMin = uMean + (uMin90-uMean)*sRange o uMax = uMean + (uMax90-uMean)*sRange o vMin = vMean + (vMin90-vMean)*sRange o vMax = vMean + (vMax90-vMean)*sRange

The corresponding code can be found in the call tree of following functions defined in ...\common\colorlut.cpp: ColorLUT::generateSignature(…)

ColorLUT::iterate(…) ColorLUT::calcRatios(…)

ColorLUT::updateSignature(…) and the IterPixel functions called in there (e.g. the yMin cut applied in IterPixel::nextHelper(…) )

Color Classification The final color classification is simple:

The y-value of the color object has to be greater than a lower limit that is adjusted with PixyMon's "Min Brightness"-slider: y>minBright*3*255

The u and v values have to be inside the signature selection box described above

This happens in function Blobs::runlengthAnalysis() defined in file ...\common\blobs.cpp in line 161:

But before getting to this point a pixel has to pass a pre-selection that is done by the assembler code that also reads in the pixel data from the sensor. This assembler code, that can be found in ...\libpixy\rls_m0.c, is running on the CortexM0 co processor.

I haven't delved into the assembler code. My last assembler excursion is quite some time ago: Z80 poked into memory. … I assume that it does the same as CBlobModule::handleLine defined in \src\host\pixymon\cblobmodule.cpp.

Color lookup table (LUT) based preselection The color preselection is realized by using a lookup table. This lookup table is a 64x64 lut[u,v] array that spans the u-v-chrominance plane and contains the signature IDs assigned to the corresponding position in the u-v-plane. The color table is generated in function ColorLUT::generateLUT():

With three nested loops Pixy walks through the RGB color space Colors with a luminance y=r+g+b smaller than minBright*3*255 are skipped. "minBright" is

adjustable via PixyMon's "Min Brightness" slider. Pixy calculates the u and v values of the color, loops through all signatures and checks if the color is

compatible with each signature. In case of a signature match the signature ID is written to the corresponding lut[u,v] bin.

In case of collisions, i.e. the color is compatible with more than one signature or the corresponding lut[u,v] bin already contains a signature ID from another color, the signature with the smallest ID wins. The signatures are prioritized!

A less efficient color model is used for the color LUT But for the LUT based preselection the u and v values are calculated differently than described above. The color model used is simpler than the one used for the final color classification:

y = r+g+b u = r-g v = b-g

The "normalization", i.e. the division of the u and v values by y has been omitted. This is a pity, because this model does not provide the nice saturation gradient from the center of the u-v-plane to the edges that allows reliable classification of gloomy but well saturated pixels. In this color model they are mixed up with low saturated colors, dark and bright ones. That’s why the rather aggressive cut on the minimum brightness has to be applied in order to suppress false positives. This is illustrated with the following two plots. The first plot shows the full saturated colors and the second one the colors with a saturation smaller than 50%.

I guess that this less suitable color model has been chosen, because the cortex M0, the LPC43xx is equipped with, doesn't provide divisions in HW. It is not even able to do fast multiplication in HW. A multiplication takes 32 cycles.

Weaknesses and Improvements The minimum brightness limit acts differently on pure and mixed colors The "Minimum Brightness" slider applies a global lower limit to Pixy's distinct non-weighted "luminance" y=r+g+b. Unfortunately it acts quite differently and pure and mixed colors. E.g. the y of a magenta is twice the y of a pure red of similar brightness. The y of a grey is even three times greater than a similar red, blue or green.

A more intuitive approach might be a "Minimum Brightness" cut on max(r,b,g).

By setting the "minimum brightness" to values greater than 33% => 0.33*3*255=255, it should be even possible to suppress pure colors completely. But Pixy reacts in a different and rather strange way on this setting, as shown in the following test.

Surprising effects of PixyMon's signature tuning sliders The following test has been done with PixyMon 2.0.6 and Pixy FW 2.0.8. Pixy has been powered via USB, Servos and everything else has been disconnected. A "Restore default parameter values" has been executed before the test. The following two PixyMon console commands activates Pixy's color bar test pattern: cam_testPattern(1) cam_setWBV(0x404040)

The HSV hue value within the bars is constant. In each bar the HSV value ranges from 60 to 255 and the HSV saturation ranges from 164 to 240. The saturation variation is caused by the nonzero black level of rgb_black=(21,21,21).

To get a reasonable fraction of the color bars selected by Pixy, one has to choose the signature training boxes rather large. I have chosen the center half of the magenta and red colored bar. The result is shown in the left plot (signature ranges set to 2.5 default, Min Brightness to 0.1 default). In the middle picture the Min Brightness setting has been increased to 0.3. OK, as expected the Min Brightness cut has a stronger impact in the red than in magenta color bar. But at 0.3 the red signature should already have almost completely vanished. Increasing the Min Brightness further just below its maximum setting results in the strange situation shown in the third picture. The brightest red pixels are suppressed, while some darker ones still passes the Min Brightness filter.

I don't have any clue why Pixy behaves like this. Maybe this is a side effect of utilizing a different color model for the preselection or something is wrong with the calculation of y. After reading the following in the WIKI: "… Pixy calculates the color (hue) and saturation of each RGB pixel from the image sensor and uses these as the primary filtering parameters. The hue of an object remains largely unchanged with changes in lighting and exposure. Changes in lighting and exposure can have a frustrating effect on color filtering algorithms, causing them to break. Pixy’s filtering algorithm is robust when it comes to lighting and exposure changes …", one would expect that Pixy selects the whole color bar, independently of where one places the signature training window, because the hue is a constant of each color bar and only the brightness is changing significantly in the color bars. This is not the case, at least if one selects only a small region of the color bar for signature generation. E.g. if one defines a red signature based on a quadratic region located at e.g. 2/3 of the bar height, only a region slightly bigger than the definition box is selected. Even if one adjusts the signature range to maximum and the minimum brightness to zero, Pixy doesn't select the lower darker part of the bar. I agree to the WIKI, that this is kind of frustrating. With this test pattern you can provoke some amazing effects. Just try the following with default settings:

Define signature 1 by a narrow box at the top of one color bar. Define signature 2 in the center of same color bar by a box that has half the height of the bar.

The results looks like shown in the picture below. You might argue that this is a nasty artificial hack. But I have observed quite similar looking color object fragmentation, when trying to train Pixy to detect some differently colored legos.

Mean value as signature anchor is not robust against outliers Using the u and v mean values as position anchor for the signature selection box in the u-v-plane is not a good idea, as some outliers might significantly bias the mean value. If e.g. the signature training box contains some white highlight the resulting selection box is shifted to the low saturation center of the u-v-plane. Because in the suboptimal color model used for the preselection LUT the white colors mixes with the gloomy but saturated colors, the result might be a very intrusive signature creating lots of false positives.

Example: If one defines in Pixy's test pattern in the upper half of the yellow bar a signature and adds just a tiny little bit of the neighbored grey bar, one ends up with a very intrusive yellow signature. The left picture shows the result with default parameter values. If one increases the signature range from default 2.5 to 3.5 Pixy takes even a clear blue as yellow (right picture).

Using the

median of the u and v distribution, the geometric middle between the min90 and max90 values, or the mean of the 90% subsample

is a better choice for the signature selection box position in the u-v-plane.

In the LUT generation algorithm the dark colors are already masked out. It might be a good idea to do the same with the very bright colors.

An additional cut on the color saturation that masks out all highlight and shadow pixels from the signature training sample would also help.

A more symmetric color model for signature definition and LUT based preselection Playing around with your color models I noticed that also the nicely symmetric classic YUV color model could be "normalized". This makes it very well suited for color classification. If there are some cycles left on the M0 it could be even used for the preselection LUT as it can be approximated very well without any expensive multiplication or division. It needs an additional division LUT of 255 byte, some shifts, adds and a few branches.

The model:

y = wr*r + wg*g + wb*b u = (b-y)/(1-wb) / max(r,g,b) v = (r-x)/(1-wr) / max(r,g,b) with wr=0.299 wb=0.114 wg=1-wr-wb=0.586

The difference to the original YUV color model is that the u and v values are divided by the maximum of the r,g and b values. The result is illustrated in the following plots: Left side: The chrominance u-v-plane on the left (u,v=[-1..1]) Right side: y plotted verses r=sqrt(u²+v²) First the full spectrum, HSV/RGB={[0..1],[0..1],[0..1]}.

Just the full saturated colors: HSV={[0..1],1,[0..1]}:

The low saturated colors: HSV={[0..1],[0...0.5],[0..1]}.

The high saturated colors: HSV={[0..1],[0.5..1],[0..1]}:

Just one single hue: HSV={0,[0..1],[0..1]}:

This model has the following nice features:

The colors are arranged quite symmetrically in the u-v-plane. The saturation grow linearly from the center to the edges A large fraction of the u-v-plane is covered by colors. (Important for the LUT) The position of a color in the u-v-plane is independent of its brightness

=> It's very well suited for color selection! It would be great if it could also be used for the LUT. I'm confident that this would significantly improve Pixy's detection performance. Then Pixy could even be trained to detect white goalposts used in the RoboCup (if min/max brightness would be adjustable separately for each signature). Documented example C code for a division and multiplication free approximation of u and v for the LUT based preselection on the cortex M0 can be found in the appendix (The approximation core is in the box.) The results of the approximation are shown in the plots below: First the full spectrum, HSV/RGB={[0..1],[0..1],[0..1]}.

Just the full saturated colors: HSV={[0..1],1,[0..1]}:

The low saturated colors: HSV={[0..1],[0...0.5],[0..1]}.

The high saturated colors: HSV={[0..1],[0.5..1],[0..1]}:

Just one single hue: HSV={0,[0..1],[0..1]}:

Appendix Example code for the approximated nYUV color model (for LUT preselection) // This is actually a C++ ... script // running in the C interpreter of root // https://root.cern.ch/drupal/ // A lookup table for a silly approximation of a division // of a signed int16 (not too big) by an unsigned uint8 // Usage Example a/b: // unsigned char lut=divLUT[b]; // unsigned char s1=lut&0xf; // if(s1) a+=a>>s1; // unsigned char s2=lut>>4; // a>>=s2 unsigned char divLUT[256]; // fills the division approximation LUT void fillDivLut(){ // just debug stuff float maxErr=0.0f; // an reference nominator for error estimation // seems to be a good reference value. strangely higher vals lead to higher errors const short nomi = 12240; divLUT[1]=0; for(unsigned int dnomi=2; dnomi<256; ++dnomi){ // loop through all denominators float refQ = float(nomi) / dnomi; // the "exact" result unsigned char msb = 1; while(dnomi>>msb) ++msb; // pos of most significant denominator bit, used as loop limit below unsigned char s1MinErr = 0; // stores first and second shift unsigned char s2MinErr = 0; // for the float minErr = 23.0f; // lowest quotient error found so far short qMinErr = 0; // just debug stuff // loop through reasonable bit shift distances // and search for the combination with the lowest quotient approximation error for(unsigned char s2=msb-1; s2<=msb+1; ++s2){ for(unsigned char s1=0; s1<8; ++s1){ short apprxQ = nomi; if(s1) apprxQ += nomi>>s1; apprxQ >> =s2; float err = apprxQ/refQ-1.0f; float aerr = fabs(err); if(aerr<minErr){ minErr = aerr; s1MinErr = s1; s2MinErr = s2; qMinErr = apprxQ; } } } // store the best s1/s2 combination to the LUT divLUT[dnomi]=s1MinErr|(s2MinErr<<4); printf("%3u %2u %u %2x %5i %5.1f %2.1f%% %3.0f\n", dnomi, s1MinErr, s2MinErr, divLUT[dnomi], qMinErr, refQ, minErr*100.0f, qMinErr-refQ); if(minErr>maxErr)maxErr=minErr; } printf("%.1f\n",maxErr*100.0f); } pimpLUTPreselect(float hueMin, float hueMax, float satMin, float satMax, float valMin, float valMax, float yMin=0.0, int steps=50){ fillDivLut(); // graphics initializations const int sz=255; const int psz=5; TCanvas* cnvs = new TCanvas("pl2YUV","",sz*4,sz*2); cnvs->Divide(2,1); cnvs->Draw(); TASImage* imgUV = new TASImage(sz+psz+1, sz+psz+1); TASImage* img_rUV_Y = new TASImage(sz+psz+1, sz+psz+1); // just some debug stuff short uMin=0.0; short uMax=0.0; short vMin=0.0; short vMax=0.0; // loop through the HSV color space

for(float hue=hueMin; hue<=hueMax; hue+=360.0/steps) for(float sat=satMin; sat<=satMax; sat+=1.0/steps) for(float val=valMin; val<=valMax; val+=1.0/steps) { float r,g,b; TColor::HSV2RGB(hue<0.0 ? hue+360.0 : hue,sat,val,r,g,b); // classic YUV weights wr=0.299 wb=0.114 wg=1-wr-wb=0.587 // float y = wr*r + wg*g + wb*b; // float u = r-y; // float v = b-y; // translate the float rgb vals [0..1] to shorts [0..255] short r16 = r*255.0+0.5; short g16 = g*255.0+0.5; short b16 = b*255.0+0.5; // Approximate classic YUV luminance Y and boost to maximum range that fits into int16 // 2:4:1 isn't a that bad approximation of 0.299:0.587:0.114 const unsigned char rs=5; const unsigned char gs=6; const unsigned char bs=4; short y = (r16<<rs) + (g16<<gs) + (b16<<bs); // Extend the r and b ranges accordingly // Do similar shifting as above as the LPC43xx's M0 doesn't support fast multiplication short r16s = (r16<<rs) + (r16<<gs) + (r16<<bs); // r16*(32+64+16)=r16*112 short b16s = (b16<<rs) + (b16<<gs) + (b16<<bs); // Calculate classic YUV chrominance values u and v short u = b16s-y; short v = r16s-y; // tune the chroma vals to correct a bit the weight approximation errors above // and - more important - to cover as many bins of the LUT as possible u+=(u>>3)+(u>>5); v+=(v>>2)+(v>>4)+(v>>5); // "Normalize" the chroma vals to the maximum of the rgb vals // This moves u/v vals of the gloomy but well saturated pixels from // the center to the bright edges of uv-plane, leaving only the low // saturated (gloomy and bright) in the center region. // first get the maximum of rgb ... short rgbm = r16; if(g16>rgbm) rgbm=g16; if(b16>rgbm) rgbm=b16; // // ... and approximate the division u/rgbm and v/rgbm // As division is even slower than multiplication // Thus do the following shift/add/shift approximation instead of a real division // shift values are taken from a look up table unsigned char lut=divLUT[rgbm]; unsigned char s1=lut&0xf; if(s1){ u+=u>>s1; v+=v>>s1; } unsigned char s2=lut>>4; u>>=s2; v>>=s2; // debug stuff if(u<uMin)uMin=u; if(u>uMax)uMax=u; if(v<vMin)vMin=v; if(v>vMax)vMax=v; // plot the point in its HSV colors // in the u/v plane // and the r(u,v) / y plane int U=((u<<1)+255)>>1; int V=(255-(v<<1))>>1; char c[23]; sprintf(c,"#%02x%02x%02x", r16, g16, b16); imgUV->DrawBox(U, V, U+psz, V+psz, c, 1, TVirtualX::kFilled); float uf=u*2; float vf=v*2; int ruv = sqrt( uf*uf+vf*vf); img_rUV_Y->DrawBox(ruv, 255-y/112, ruv+psz, 255-y/112+psz, c, 1, TVirtualX::kFilled); } cnvs->cd(1); imgUV->Draw(); cnvs->cd(2); img_rUV_Y->Draw(); printf("%i %i\n",uMin,vMin); printf("%i %i\n",uMax,vMax); }