PROPEL: Probabilistic Parametric Regression Loss for Convolutional NeuralNetworks

Muhammad Asad ∗1,2, Rilwan Basaru2, S M Masudur Rahman Al Arif2,3, and Greg Slabaugh2

1Imagination Technologies, 2City, University of London, 3Netherlands Cancer Institute

Abstract

Recently, Convolutional Neural Networks (CNNs) havedominated the field of computer vision. Their widespreadsuccess has been attributed to their representation learningcapabilities. For classification tasks, CNNs have widely em-ployed probabilistic output and have shown the significanceof providing additional confidence for predictions. How-ever, such probabilistic methodologies are not widely appli-cable for addressing regression problems using CNNs, asregression involves learning unconstrained continuous and,in many cases, multi-variate target variables. We propose aPRObabilistic Parametric rEgression Loss (PROPEL) thatenables probabilistic regression using CNNs. PROPEL isfully differentiable and, hence, can be easily incorporatedfor end-to-end training of existing regressive CNN archi-tectures. The proposed method is flexible as it learns com-plex unconstrained probabilities while being generalizableto higher dimensional multi-variate regression problems.We utilize a PROPEL-based CNN to address the problem oflearning hand and head orientation from uncalibrated colorimages. Comprehensive experimental validation and com-parisons with existing CNN regression loss functions areprovided. Our experimental results indicate that PROPELsignificantly improves the performance of a CNN, while re-ducing model parameters by 10× as compared to the exist-ing state-of-the-art.

1. IntroductionConvolutional Neural Networks (CNNs) are enabling

major advancements in a range of machine learning prob-lems [14, 16, 19, 21, 27, 28, 32]. Their success has beenwidely reflected by their application in a number of do-mains, most notably for image classification [16, 19, 28]and segmentation problems [14, 15, 27, 32], where they

∗This work was done when Muhammad Asad was with City, Universityof London.

HEATMAP-MSE [13] PROPEL

(a) (b)

Parameters Learned

1,248,932 126,890

Figure 1: Comparison of predictions from (a) HEATMAP-MSE [13] and (b) PROPEL (proposed). PROPEL learnsparametric probability distributions requiring 10× fewermodel parameters, while resulting in more confident pre-dicted distributions. The predictions from PROPEL are pa-rameters for a probability distribution (shown as a contin-uous distribution in (b)). In contrast, (a) directly learns anon-parametric distribution of size 20× 20.

have outperformed conventional machine learning methods.The existing state-of-the-art for CNN-based classificationand segmentation benefits from probability distributions fortarget class prediction [16, 19, 27, 28, 32]. These distribu-tions provide additional confidence information that can beuseful to determine the level of uncertainty in a given pre-diction. They can also help resolve ambiguous model pre-dictions [5]. Furthermore, for complex learning problemswhere dividing a given learning task into smaller subtaskscan help, probabilistic models provide a flexible frame-work for combining the predictions from multiple models[5, 12, 18]. Moreover, the probabilistic predictions enableincorporation of priors using Bayes’ rule for applicationswhere prior class distributions are known [6].

1

arX

iv:1

807.

1093

7v1

[cs

.CV

] 2

8 Ju

l 201

8

Despite widespread application of CNNs for probabilis-tic classification, little emphasis has been made for un-constrained probabilistic regression. Regression using aCNN has been restricted to target space learning using aMean Squared Error (MSE) that does not provide a prob-abilistic output [10, 23]. In some existing methods, non-parametric probabilistic target variables are learned usingMSE in the form of probability heatmaps [7, 13, 31]. Thesemethods make assumptions that the target continuous vari-ables can be constrained within a specified domain whichis useful for generating heatmap distributions. This lim-its the application of such methods to only a specific do-main, where they are unable to generalize well for uncon-strained multi-variate regression problems. Moreover, non-parametric probabilities require the output layer to representa significantly large dimensional heatmap. This contributesto the complexity of the model, requiring larger number ofparameters to be learned with significantly increased learn-ing time and training data.

We address the existing issues with CNN-based re-gression by proposing a novel PRObabilistic ParametricrEgression Loss (PROPEL) that enables a CNN modelto output parametric regression probabilities. This isachieved by using a loss function that models the CNNoutput as a parameterized Mixture of Gaussians in thetarget space. PROPEL is fully differentiable with a novelanalytic closed-form solution to integrals, allowing it to beused for end-to-end training of existing CNN regressivearchitectures. Moreover, PROPEL is generalizable formodeling different levels of complexity within predictionprobabilities as well as the number of dimensions foraddressing multi-variate regression problems. As theoutput is parameterized by a Mixture of Gaussians, theproposed layer requires fewer number of output parametersas compared to previously used probability heatmaps. Inthis paper, PROPEL is used to address the problem ofcolor image-based hand and head orientation regression[5, 11]. We show that PROPEL-based CNN outperformsstate-of-the-art regression losses. Moreover, it enablesthe model to generalize better and resolve ambiguitiesin a dataset, while using 10× fewer model parametersas compared to the existing state-of-the-art (as shown inFig. 1).

Our Contributions. The main contributions of this paperare:

• To the best of our knowledge, we are the first to pro-pose PROPEL, a fully differentiable loss layer for en-abling unconstrained probabilistic parametric regres-sion using CNNs. A novel derivation is provided forthe proposed loss using Mixture of Gaussians distribu-tion;

• We present a generalizable framework that handlesany number of target dimensions and has the abilityto model complex multimodal probabilities with addi-tional Gaussians in the mixture;

• We show that the proposed PROPEL can help addressambiguities in predictions by enabling a model to out-put expressive predictions in the form of parametricprobabilities;

• We provide comprehensive experimental validationand comparisons with existing methodologies and re-port that the proposed loss outperforms existing state-of-the-art by a large margin, while using 10× fewermodel parameters.

2. Related WorkProbabilistic Regression using CNN. Previous work hasmostly been focused on exploring non-parametric proba-bilistic regression for CNNs [2, 7, 13, 25, 30, 31]. Thesemethods first generate the ground truth probability heatmapdistributions in the target space. A CNN is then trained us-ing a Mean Squared Error (MSE) loss to learn the mappingof input images onto the heatmaps. Tompson et al. [31] gen-erated and employed single-view heatmaps for hand jointslocalization using depth images as input. Ge et al. [13] ex-tended [31] to use multi-view CNN, where heatmaps fromthree projections of depth images were used. Three separateCNNs were trained to infer the heatmaps of different jointlocations in each projection. Similar methods were also pro-posed for human pose estimation using a CNN [7, 25]. Re-cently, Al Arif et al. [2] proposed to learn heatmap distribu-tions of cervical vertebrae locations by utilizing the Bhat-tacharyya coefficient (BC). The use of BC resulted in betteraccuracy as it measured a probabilistic loss between twoheatmap probability distributions. Pathak et al. [24] em-ployed probabilistic interpretation of Euclidean loss to en-force a set of known constraints on the model outputs.

Lathuiliere et al. [20] jointly learned both a CNN model,for representation learning, and a Gaussian mixture of lin-ear inverse regressions by utilizing a modified ExpectationMaximization (EM) algorithm. Crucially, this approach uti-lized a pre-trained CNN model; and EM is known to sufferfrom sensitivity to initialization. Similar to PROPEL, thisapproach enabled probabilistic regression using CNNs. Themethod, however, required a carefully designed initializa-tion, that included a pre-trained CNN and clustering of data.Moreover, their initialization was based on the assumptionthat the data can be easily clustered with known number ofclusters. In contrast, our proposed PROPEL is a fully dif-ferentiable loss that enables end-to-end training of a CNNmodel. As PROPEL has a closed-form solution, it can beeasily incorporated within existing CNN architectures, andcan learn models from scratch using backpropagation. As

2

20× 20 40× 40 80× 80 160× 160

Figure 2: Heatmap size versus resolution of the probability density function (PDF) shows that a smaller size results in loweraccuracy, while larger size contributes towards better approximation of the PDF. The majority of the heatmap distributionconsists of values close to zero, which results in under-utilization of the allocated model parameters.

indicated by our experimental validation, PROPEL general-izes well from smaller dataset, while also helping to signif-icantly reduce the model parameters.

Non-parametric heatmap distributions only work inproblems where target space can be fully defined withina finite domain, such as hand or body joint localization[7, 13, 25, 31]. While such assumptions prove useful for in-troducing and benefiting from the probabilistic regression inCNNs, they limit generalization of such methods and theirapplication to other regression problems. Furthermore, theparameters required to learn a non-parametric probabilitydistribution are much higher as such methods directly learnthe distributions. Achieving high accuracy from these meth-ods often requires high-resolution heatmap distributions, re-sulting in higher number of model parameters (see Fig. 2and Fig. 3). In addition, the majority of a non-parametricheatmap distribution contains values close to zero. This re-sults in under-utilization of the allocated model parametersas most of the output parameters are set to zero during in-ference. Learning higher dimensional target variables alsoresults in rapid increase in model parameters, where ex-isting methods split the problem into learning multiple 2Dheatmaps [7, 13, 31].

The proposed PROPEL method addresses the existinglimitations of non-parametric probabilistic regressionby utilizing a parametric loss function that is fully dif-ferentiable. In addition to significantly reducing modelparameters, PROPEL does not require the target spaceto be constrained. Furthermore, the proposed solution isgeneralizable in terms of target variable dimension andprediction probability complexity. As a result, PROPELcan be easily applied to existing regression problems withany number of target dimensions without constraining the

Figure 3: Heatmap size versus number of required modelparameters for CNN architecture in Fig. 5, where only thenumber of neurons within the fully connected layers Fc1and Fc2 are changed to match the size of a given heatmapdistribution. The number of model parameters increases ex-ponentially with increased heatmap size. In contrast, PRO-PEL uses parameterized probabilities where modeling com-plex distributions only require adding more Gaussians (I)to the Mixture of Gaussians. This is linearly related byd = 2× n× I to the model parameters, and hence requiressignificantly less model parameters.

learned probabilities. We provide PROPEL along withits partial derivatives, which enables its application forend-to-end training of existing CNN architectures usingbackpropagation.

Orientation Estimation. In this paper, we demonstrate the

3

ability of the proposed method to learn and outperform ex-isting methods for image-based hand and head orientationregression.

Image-based hand orientation regression has only beenapplied in [3, 4, 5]. [3] utilized two single-variate RandomForest (RF) regressors based on an assumption that the ori-entation angles vary independently. This method, evaluatedon a subset of hand orientation angles, showed the signif-icance of inferring hand orientations from 2D uncalibratedcolor images. Later, [4] used a multi-layered Random For-est regression method that utilized multi-variate regressorsto regress the orientation angles together. Similarly, [5]presented a staged probabilistic regressor, which learnedmultiple expert Random Forests in stages and employed amulti-layered regressor [5]. These image-based hand orien-tation regression methods do not require camera calibrationwhich renders them suitable for a wider array of applica-tions across different devices. The datasets used for trainingcome from multiple people, which enables these methods tonaturally handle person-to-person hand variations.

All existing image-based hand orientation regressionmethods utilize hand-crafted features. In this paper, weshow that the task of hand orientation regression can benefitfrom the representation learning capability of CNN. Utiliz-ing MSE, we show that a CNN model, without any proba-bilistic output, can significantly outperform existing meth-ods that use hand-crafted features. We use this model asbaseline for our experimental validation and demonstratethat the proposed probabilistic parametric regression losscan outperform existing regression loss functions for CNNs.This is particularly due to PROPEL’s ability to handle ambi-guity in hand orientation datasets by utilizing probabilisticparametric distributions to comprehensively represent thepredictions.

A number of existing methods address head orientationregression using color images [8, 1, 22, 20]. Drouard etal. [8] proposed Gaussian mixture of locally-linear map-ping model. This method learned the mapping of HOG-based features onto head orientation, additionally providingprobabilistic output. Later, Lathuiliere et al. [20] improved[8] by utilizing a pred-trained CNN for feature extraction.This method, however, relied on a modified EM algorithmfor fine-tuning CNN and learning the mapping of featuresonto target head orientations. CNN-based head orientationregressors were proposed in [1, 22]. Both methods learnedusing MSE loss, where [22] utilized an additional syntheticdataset for improving the model. In our experimental vali-dation section, we compare against these existing methodsand show that the proposed PROPEL can outperform exist-ing state-of-the-art for head orientation estimation that uti-lize CNNs.

3. MethodWe now describe the details of our proposed PROba-

bilistic Parametric rEgression Loss (PROPEL). First, we in-troduce the loss function and the corresponding parametricprobabilities. This is followed by the derivation of a novelanalytic closed-form solution to the integrals within our lossfunction that are required for computing the loss over anunconstrained real space. Lastly, we provide details of thepartial derivatives of our loss function which allows PRO-PEL to be integrated with existing CNN architectures usingbackpropagation.

3.1. Probabilistic Parametric Regression LossWe expand on [26] by introducing a regression loss

function for learning a CNN model. The existing mea-sures, such as Bhattacharyya coefficient (BC) and Kull-backLeibler (KL) divergence, are tractable when distribu-tions are unimodal, i.e. each consisting of a single Gaus-sian. However, it is well-known that these metrics be-tween multimodal Mixture of Gaussians have no analyticsolution [17, 9] and at best, one must resort to approxi-mation. However, to address modelling of complex multi-modal probabilistic distributions, and handling ambiguouspredictions (see Fig .7), Mixture of Gaussians are essential.This motivates the use of metric from [26] for proposingPROPEL, as this metric has an analytic closed-form so-lution when distributions are Mixture of Gaussians. Weshow, for the first time in this paper, how to analyticallyaddress neural network regression using a Mixture of Gaus-sians loss. PROPEL is fully differentiable, enabling us todetermine both the loss and gradient of a model distributionPm with respect to a ground truth (GT) distribution Pgt. Letx = {x1, x2, · · · , xn}ᵀ ∈ Rn define the target predictionspace with n dimensions, the proposed loss can be definedfor x as:

L = − log

[2∫PgtPm dx∫

(Pgt2+Pm

2)dx

], (1)

where Pgt is the n-dimensional GT PDF defined by a Gaus-sian distribution as:

P kgt =

e− 1

2

[(x1−µx1k )2

σx1k+···+ (xn−µxnk )2

σxnk

]

(√2π)n

√σx1k · · ·σxnk

, (2)

where k represents a sample selected from a givendataset, µx1k

, µx2k, · · · , µxnk are the GT observations and

σµx1k , σµx2k , · · · , σµxnk are the GT variances associatedwith Pgt. The numerator in Equation 1 evaluates similar-ity between a GT PDF Pgt and a model PDF Pm, whereasthe denominator eliminates bias towards specific shapes ofprobability distributions.

In addition to the GT distribution Pgt, PROPEL requiresa model distribution Pm. The choice of Pm determines theability of a trained model to handle complex variations intarget space for a given dataset. We recognize that any

4

Figure 4: Symmetry problem due to depth ambiguity in hand orientation dataset shows image pairs with different orien-tations (shown with green arrows) but similar hand shapes. Such ambiguiuties motivate the use of multimodal probabilitydistributions as the model distribution Pm in PROPEL.

parametric PDF can be utilized as the model distribution.Our work is motivated by a limitation of existing regressiontechniques for addressing ambiguity within a given dataset.Consider the problem of learning hand orientations fromuncalibrated color images. As noted in existing literature[4] and shown in Fig. 4, the absence of depth informationresults in ambiguity, where multiple symmetrically oppositehand orientations have similar color images. In cases wheresuch ambiguities exist, a unimodal probabilistic distributionproves insufficient (the same is true for probabilistic inter-pretation of MSE). In contrast, utilizing a multimodal distri-bution enables the regressor to address such ambiguities bylearning to infer multiple hypotheses [4, 5]. In this work,we use a Mixture of Gaussians as the model distributionPm because it facilitates in keeping our derivations sim-ple, while also enabling us to model complex multimodaldistributions. Moreover, in comparison to existing state-of-the-art which uses non-parametric heatmaps, a Mixtureof Gaussians only requires a fraction of parameters (meansand variances) for modeling a distribution. This reduces thenumber of model parameters as well as the overall complex-ity of a CNN model. The number of Gaussians in the Mix-ture of Gaussians also enable us to learn complex probabil-ity distributions that can handle ambiguous predictions, re-sulting in better accuracy while keeping the derivation con-sistent. For a target space with n dimensions, Pm is definedas:

Pm =1

I

I∑

i=1

Pi =1

I

I∑

i=1

e− 1

2

[(x1−µx1i )

2

σx1i+···+ (xn−µxni )

2

σxni

]

(√2π)n

√σx1i · · ·σxni

,

(3)where Pi represents an individual ith Gaussian within aMixture of Gaussians, µx1i

· · ·µxni and σx1i· · ·σxni are

model parameters inferred as the output of a CNN networkand I is the total number of Gaussians in Pm. Note that theweights for each Gaussian in the mixture are fixed to 1

I inour method. Our model distribution does not include covari-ances as in our experimental validation we found variancesto be sufficient for outperforming the state-of-the-art meth-

ods. The next section provides a novel analytic closed-formsolution of the loss function L such that we can evaluateintegrals over the continuous domain −∞ to +∞.

3.2. Analytic Solution to IntegralsGiven the loss function L, the model PDF Pm and GT

PDF Pgt, we can substitute these PDFs in Equation 1 andsimplify.

L = − log

[2∫PgtPm dx∫

(Pgt2+Pm

2)dx

], (4)

= − log

[2

∫PgtPm dx

]+log

[∫Pgt

2 dx+

∫Pm

2 dx

],

(5)

= − log

[2

I

I∑

i=1

G(Pgt,Pi)

]

︸ ︷︷ ︸T1

+ log

[H(Pgt)+

1

I2

I∑

i=1

H(Pi)+2

I2

I∑

i<j

G(Pi,Pj)

]

︸ ︷︷ ︸T2

, (6)

where the functions G(Pi, Pj) and H(Pi) represent the an-alytic solutions to the integrals

∫PiPj dx and

∫Pi

2 dx, re-spectively. Pi and Pj are two multi-variate Gaussian distri-butions. Both G(, ) and H() are defined as follows 1:

G(Pi,Pj) =

∫PiPj dx, (7)

=e

[ 2µx1iµx1j

−µx1i2−µx1j

2

2(σx1i+σx1j

)+···+

2µxniµxnj

−µxni2−µxnj

2

2(σxni+σxnj

)

]

(√2π)n

√(σx1i +σx1j ) · · ·(σxni +σxnj )

.

(8)

H(Pi) =

∫Pi

2 dx =1

(2√π)n√σx1i · · ·σxni

. (9)

1Complete derivation of integrals in functions G(, ) and H() is pro-vided in the accompanying Appendix A.

5

Next, we show how the loss L from Equation 6 can beused alongside existing CNN architectures that use back-propagation algorithm for training.

3.3. OptimizationIn this section we present the partial derivatives ofLwith

respect to model parameters µxni , σxni . These can be usedfor end-to-end training of a CNN using backpropagation.We can derive partial derivatives of L as:

∂L

∂µxni= − 1

T1

[∂G(Pgt,Pi)

∂µxni

]+

1

T2

2

I2

I∑

i<j

∂G(Pi,Pj)

∂µxni

, (10)

∂L

∂σxni= − 1

T1

[∂G(Pgt,Pi)

∂σxni

]+

1

T2

1

I2∂H(Pi)

∂σxni+

2

I2

I∑

i<j

∂G(Pi,Pj)

∂σxni

.

(11)

The partial derivatives ∂G(Pi,Pj)∂µxni

, ∂G(Pi,Pj)∂σxni

and ∂H(Pi)∂σxni

are defined as:

∂G(Pi,Pj)

∂µxni

=(µxnj −µxni)

(σxni +σxnj )G(Pi,Pj), (12)

∂G(Pi,Pj)

∂σxni

= [.]G(Pi,Pj), where, (13)

[.] =(µxni

2+µxnj2−2µxniµxnj −σxni −σxnj )

2(σxni +σxnj )2

, (14)

∂H(Pi)

∂σxni

=−1

2σxni

H(Pi). (15)

At each training iteration, the proposed PROPEL losslayer computes the loss in the forward pass using Equation 6on the output from the CNN model. For the backward pass,the partial derivatives are used along with the GT distribu-tion to backpropagate the error and optimize the parametersusing RMSProp [29].

4. Experimental ValidationWe perform experimental validation of PROPEL by ad-

dressing image-based hand and head orientation regressionproblems. Given a dataset U = {(Ck,ok)}Kk=1 with K un-calibrated color images Ck of hands or heads, and the corre-sponding target orientation vectors ok, both hand and headorientation regression problems involve learning the map-ping of color images Ck onto the target orientations ok. Forhands, the orientation vector ok = {φk, ψk}ᵀ ∈ R2 con-tains Azimuth (φk) and Elevation (ψ) angles, resulting frompronation/supination of the forearm and flexion/extensionof the wrist, respectively [5]. Whereas head orientation isdefined by yaw (φk), pitch (ψ) and roll (χk) angles, i.e.ok = {φk, ψk, χk}ᵀ ∈ R3 [11]. We note that the problemof hand orientation regression is specifcally challenging asthere may be similar hand shapes in color images that maponto multiple orientations. We evaluate PROPEL’s perfor-mance on the hand orientation dataset from [5], which con-tains 9414 images collected from 22 participants. For head

12

8×

12

8×

3

64

×64

×16

32

×32

×32

16

×16

×32

8×

8×

32

20

48

50

d =

2×

n×

I

d =

2×

n×

I

Co

nv1

Co

nv2

Co

nv3

Co

nv4

R1

Fc1

Fc2 S1

PR

OP

EL

3x3 Conv + Batch

Norm + ReLU +

2x2 Max Pool

Reshape Sigmoid PROPELFully Connected

Figure 5: Flowchart shows the CNN regression architectureused for experimental validation of the proposed PROPELloss (details in Section 4.1). The green arrow (↑↑↑) shows thetarget GT hand orientation that is only used during training.Scalar d represents the dimension of the network output.The comparison methods, described in Section 4.2, use thesame overall architecture, where only the components in thehighlighted box are replaced to match output dimensionsrequired by a comparison loss function.

orientation, we evaluate the performance on publicly avail-able BIWI dataset [11], which contains 10k images from 20participants. The range of hand orientation angles capturedby the hand orientation dataset are defined within a circularspace with

√φ2 + ψ2 ≤ 45◦.

4.1. Network Architecture

The CNN network utilized for experimental validation ofPROPEL is inspired from VGG networks [28] (Fig. 5). Wekeep the number of filters in each layer lower than in [28],as this does not have a significant impact on our model’sperformance to the orientation regression problems. Forhand orientation regression, the input to our network is anRGB color image Ck of size 128 × 128 × 3, where themethod learns to infer parameters for the model distributionin Equation 3. The maximum likelihood estimate (MLE)of the inferred parametric model distribution is used to getpredicted hand orientation angles. The representation learn-ing part of our network has four 3× 3 convolutional layers,where each layer is followed by batch normalization, a rec-tified non-linear unit (ReLU) and a 2×2 max pooling layer.The output from the convolutional layers are resized and fedinto two fully connected layers, Fc1 and Fc2 with 50 andd = 2 × n × I neurons, respectively. In our model distri-bution Pm, each dimension n requires two parameters, i.e.the mean and the variance of a Gaussian. Pm may have IGaussians resulting in d = 2 × n× I parameters as outputof Fc2.

The head orientation validation is done with the samenetwork, however as the size of head images in BIWIdataset is small, the input image has size 64 × 64. We alsomodify Fc2 to enable inference of three-dimensional headorientation.

6

Table 1: Mean Absolute Error (MAE) in degrees alongwith number of model parameters for single-fold and 5-foldcross-validation.

Single-fold Validation

Method Azimuth(φ)

Elevation(φ) CMAE No. of

Parameters

PROPEL (proposed) 5.27◦ 3.99◦ 4.63 126, 890HEATMAP-MSE [13] 7.22◦ 6.07◦ 6.65 1, 248, 932

MSE 8.23◦ 7.18◦ 7.71 126,584

SPORE [5] 8.49◦ 7.26◦ 7.88 -MtR [4] 9.67◦ 7.97◦ 8.82 -

5-fold Cross-validation

PROPEL (proposed) 11.96◦ 10.00◦ 10.98 126, 890HEATMAP-MSE [13] 13.81◦ 10.42◦ 12.12 1, 248, 932

MSE 14.16◦ 11.51◦ 12.84 126,584

SPORE [5] 15.73◦ 12.95◦ 14.34 -MtR [4] 16.16◦ 12.83◦ 14.50 -

4.2. Comparison Methods

The proposed method is compared with existing state-of-the-art methods for hand orientation regression, namely,Marginalization through Regression (MtR) and StagedPrObabilistic REgression (SPORE) [4, 5]. Both these meth-ods employ hand-crafted features along with probabilis-tic learning frameworks for learning a multi-layered Ran-dom Forest regressor. We also compare PROPEL with twoCNN regression loss methods, namely, Mean Squared Er-ror (MSE) and probability heatmaps (HEATMAP-MSE).As PROPEL addresses probabilistic regression for a CNN,we compare it with the probability heatmap (HEATMAP-MSE) prediction framework inspired from [13, 31]. Fol-lowing the methodology of [13, 31], the heatmap distri-butions for HEATMAP-MSE are generated by evaluatingEquation 2 within a 20 × 20 grid covering the domainφ ∈ [−45◦,+45◦] and φ ∈ [−45◦,+45◦]. Both MSEand HEATMAP-MSE use the same network architectureas PROPEL, except for the changes in the highlighted boxin Fig. 5 to match the output dimensions of each compar-ison methods. MSE uses Fc2 with two neurons, whereasHEATMAP-MSE requires 500 neurons for Fc1 and 400neurons for Fc2. The additional neurons are required to en-able HEATMAP-MSE to learn 20× 20 = 400 dimensionaltarget heatmap distributions. MSE and HEATMAP-MSEboth use Mean Square Error as the loss function. A single-fold and 5-fold cross-validation is used to evaluate the per-formance of PROPEL in comparison to the hand orientationregression methods.

We also compare the performance of PROPEL on image-based head orientation regression task with existing litera-ture. Performance of PROPEL is compared to existing CNNmethods [20, 22] and a hand-crafted feature-based method[8]. [22] also utilizes synthetic data, however we only in-

0 5 10 15 20 25 30 35 40 45

Error in Predicted Angles

0

10

20

30

40

50

60

70

80

90

100

Pe

rce

nta

ge

da

ta

PROPEL

HEATMAP-MSE

MSE

SPORE

ML-RF MtR

Figure 6: Percentage data versus prediction error shows thepercentage data that lies below an error threshold for single-fold validation.

clude their results from real data for a fair comparison.MSE and HEATMAP-MSE are independently

trained for experimental validation. Both PROPELand HEATMAP-MSE require additional ground truth (GT)variances to be defined in Equation 2. A large variance canover-smooth the GT PDF, producing underfitted predictionPDFs. However, a small variance yields a GT PDFthat captures unwanted variations, resulting in overfittedmodel PDFs. We empirically found that a variance of(9◦)2, for both Azimuth (φ) and Elevation (ψ) angles, canoptimally capture the variations within the dataset, whileenabling both PROPEL and HEATMAP-MSE to generalizewell. Additionally PROPEL also requires the number ofGaussians I in the model PDF Pm. We found that I = 2results in outperforming existing methods while showingthe significance of having multiple components in Pm.Following the approach in [5], we utilize Mean AbsoluteError (MAE) and Combined Mean Absolute Error (CMAE)for evaluating the overall performance of all comparisonmethods.

4.3. Experimental Validation for Hand OrientationRegression

We perform single-fold experimental evaluation by ran-domly dividing the dataset into training (70%), testing(20%) and validation (10%) sets. This evaluates the gener-alization performance of the comparison methods against ascenario where the training dataset is large enough to covervariations in hand shape, color, size and orientations. Ad-ditionally, we also evaluate the performance of our methodusing 5-fold cross-validation, where the folds are created bygrouping multiple participants’ data.

Table 1 shows Mean Absolute Error (MAE) along withthe number of model parameters for CNN methods in thesingle-fold validation. We observe that the proposed PRO-

7

-40 -30 -20 -10 0 10 20 30 40

Azimuth ( ) angle

-40

-30

-20

-10

0

10

20

30

40

Ele

va

tio

n (

) a

ng

le

1

2

3

4

5

6

7

8

9

1010-4

(a)

-40 -30 -20 -10 0 10 20 30 40

Azimuth ( ) angle

-40

-30

-20

-10

0

10

20

30

40

Ele

va

tio

n (

) a

ng

le

2

4

6

8

10

1210-4

(b)

-40 -30 -20 -10 0 10 20 30 40

Azimuth ( ) angle

-40

-30

-20

-10

0

10

20

30

40

Ele

va

tio

n (

) a

ng

le

2

4

6

8

10

1210-4

(c)

Figure 7: Ability of PROPEL to handle ambiguity in predictions, showing input hand images and the corresponding predictedPDFs from PROPEL. Hand orientation angles from GT (in green ↑↑↑ ×××), and predictions from PROPEL (in blue ↑↑↑×××) and MSE(in red ↑↑↑×××) are shown. (a) - (b) show cases where PROPEL’s PDFs successfully handle ambiguities, whereas MSE predictionscontain large error as they do not have ability to resolve ambiguities. (c) shows a case where PROPEL prediction contains alarge error. We note that even in such case, the PDF contains useful information regarding the correct prediction.

PEL method outperforms existing state-of-the-art in bothhand-crafted feature-based as well as CNN-based hand ori-entation regression. Probabilistic regression in both PRO-PEL and HEATMAP-MSE outperforms the widely usedMSE loss. This is due to the ability of probabilistic methodsto provide better generalization while addressing ambigui-ties in cases where multiple hand orientations can result insimilar projected hand shape [5]. Furthermore, it can be ob-served from Table 1 that Azimuth (φ) angles have greaterMAE as compared to Elevation (ψ) angles for all compar-ison methods. This is due to the variations in inter-fingerseparation and styles for performing pronation/supinationof the forearm across different participants in the hand ori-entation dataset. Fig. 6 provides further insight by showingthe percentage of data that lies under a given error thresholdfor all comparison methods. We note that with PROPEL95% of the data lies below 10◦ error, while HEATMAP-MSE has an error of 15◦ for the same percentage data.This demonstrates PROPEL’s ability to learn even with 10xfewer model parameters as compared to HEATMAP-MSE,while generalizing well using parametric probabilities.

In Fig. 7, we show some of the ambiguous cases andcompare the probabilistic output from PROPEL against pre-dictions from MSE. The PROPEL method has the ability tolearn multiple hypotheses, due to the presence of multipleparameterized Gaussians in the model PDF. During train-ing, the error is minimized for ambiguous targets wherethese Gaussians are able to model multiple hypotheses in-stead of being forced to a single hypothesis (as in MSE).On the other hand, MSE prediction tries to fit into the tar-

-50 -40 -30 -20 -10 0 10 20 30 40 50

Predicted azimuth ( ) angle

-50

-40

-30

-20

-10

0

10

20

30

40

50

GT

azim

uth

()

angle

PROPEL: Single-fold for Azimuth angle

-50 -40 -30 -20 -10 0 10 20 30 40 50

Predicted elevation ( ) angle

-50

-40

-30

-20

-10

0

10

20

30

40

50

GT

ele

vation (

) angle

PROPEL: Single-fold for Elevation angle

-50 -40 -30 -20 -10 0 10 20 30 40 50

Predicted azimuth ( ) angle

-50

-40

-30

-20

-10

0

10

20

30

40

50

GT

azim

uth

()

angle

HEATMAP-MSE: Single-fold for Azimuth angle

-50 -40 -30 -20 -10 0 10 20 30 40 50

Predicted elevation ( ) angle

-50

-40

-30

-20

-10

0

10

20

30

40

50

GT

ele

vation (

) angle

HEATMAP-MSE: Single-fold for Elevation angle

Figure 8: GT versus predicted angle plot using PROPEL(in blue ◦) and HEATMAP-MSE (in red ◦) shows Azimuth(φ) angles (left column) and Elevation (ψ) angles (right col-umn). A good regressor predicts angles close to GT an-gles, resulting in a diagonal line. PROPEL outperformsHEATMAP-MSE by accurately predicting angles, whereasthe main source of error in HEATMAP-MSE is quantizationdue to 20× 20 heatmaps.

get space (see Fig 7 (a)-(b)). Fig. 7 (c) shows a failure casewhere PROPEL prediction results in a large error. We ob-

8

serve that even in such case, the predicted model PDF con-tains useful information regarding the correct prediction.

The most closely related method to PROPEL isHEATMAP-MSE as it also provides probabilistic regres-sion by learning probability heatmaps. Table 1 and Fig. 8show a comparison of these methods. We make two ob-servations from this comparison. First, PROPEL uses ap-proximately 10× fewer model parameters as compared toHEATMAP-MSE, while providing significantly better ac-curacy. This is due to the parametric probabilistic learn-ing capability of PROPEL. Secondly, from Fig. 8 (secondrow), quantization due to 20 × 20 heatmaps becomes themain source of errors in HEATMAP-MSE. Furthermore,it should be noted that with higher dimensional targetsHEATMAP-MSE will require an exponentially larger num-ber of model parameters, whereas our proposed PROPELmethod will only result in increasing n which is linearly re-lated by d = 2×n×I to the model output parameters. Fur-thermore, unlike HEATMAP-MSE where predictions areconstrained by the domain defined in heatmap distributions,PROPEL is capable of learning unconstrained target predic-tions without the need of heatmap distribution generationstep.

The 5-fold cross-validation helps in understanding thegeneralization capability of each comparison method forinferring hand orientation for unseen participants’ handshape, color and size. Table 1 shows results from this exper-iment. PROPEL again outperforms all comparison methodsby achieving least MAE. This experiment shows that utiliz-ing the proposed PROPEL results in a CNN model that isable to generalize better than the existing state-of-the-art forCNN regression loss.

4.4. Experimental Validation for Head OrientationRegression

For this comparison, we use the estabished experimentalprotocols from [20] and leave-one-out validation [8]. Ta-ble 2 shows comparison of PROPEL with existing state-of-the-art methods. Note that PROPEL is trained withoutany data augmentation, and the model is learned end-to-end unlike [20] that uses a pre-trained CNN. We did notrun HEATMAP-MSE in this experiment as it becomes un-wieldy in 3D, and BIWI requires estimation of three angles.In contrast, PROPEL can learn to infer probabilities for anyhigher dimensional target. PROPEL achieves competitiveresults with other methods and outperforms all techniques(including the state-of-the-art) in one of three angles, inboth evaluation protocols. We also note, from Table 2, thatPROPEL significantly reduces the model parameters, whileachieveing accuracy comparable to the state-of-the-art.

Table 2: MAE for experimental validation on BIWI dataset(*indicates the use of evaluation protocol from [20])

Method Pitch Yaw Roll CMAE No. ofParameters

Unseen Faces (training=21 users and testing=3 users)*

PROPEL (proposed) 3.44◦ 4.02◦ 3.28◦ 3.58◦ 50, 294MSE 9.20◦ 6.30◦ 5.15◦ 6.88◦ 49,835

Lathuiliere et al. [20] 4.68◦ 3.12◦ 3.07◦ 3.62◦ 135, 847, 232Liu et al. [22] 6.10◦ 6.00◦ 5.94◦ 6..01◦ 595, 573

Drouard et al. [8] 5.43◦ 4.24◦ 4.13◦ 4.6◦ -

Unseen Faces (Leave-one-out validation)

PROPEL (proposed) 5.93◦ 5.81◦ 4.53◦ 5.42◦ 50, 294MSE 7.70◦ 6.70◦ 5.81◦ 6.74◦ 49,835

Liu et al. [22] 6.00◦ 6.10◦ 5.70◦ 5.17◦ 595, 573Drouard et al. [8] 5.90◦ 4.90◦ 4.70◦ 5.93◦ -

5. ConclusionWe proposed a novel PRObabilistic Parametric rEgres-

sion Loss (PROPEL) which enabled learning paramet-ric probabilities for addressing regression problems us-ing CNN. PROPEL is fully differentiable with an analyticclosed-form solution to integrals, which allows it to be usedwith existing CNN architectures. A complete generalizablederivation for different level of complexity of predictionprobabilities and multi-variate target dimensions was pro-vided using Mixture of Gaussians. Comprehensive exper-imental validation showed that PROPEL outperforms pre-vious state-of-the-art by a large margin. It shows bettergeneralization capabilities while reducing the number ofmodel parameters by a factor of 10. We also showed theusefulness of parametric probabilities for ambiguous caseswhere PROPEL handled prediction by providing multiplehypotheses. Our contribution can enable CNN regressionmodels to have better prediction capabilities for addressinga range of problems. Our future work will explore the use ofPROPEL in other challenging regression problems. We alsoaim to study the incorporation of other parametric probabil-ity distributions within PROPEL.

References[1] B. Ahn, J. Park, and I. S. Kweon. Real-time head ori-

entation from a monocular camera using deep neural net-work. In Asian Conference on Computer Vision, pages 82–96. Springer, 2014.

[2] S. M. M. R. Al Arif, K. Knapp, and G. Slabaugh. Probabilis-tic spatial regression using a deep fully convolutional neuralnetwork. In British Machine Vision Conference. The BritishMachine Vision Association, 2017.

[3] M. Asad and G. Slabaugh. Hand orientation regression usingrandom forest for augmented reality. In International Con-ference on Augmented and Virtual Reality, 2014.

[4] M. Asad and G. Slabaugh. Learning marginalization throughregression for hand orientation inference. In Computer Vi-sion and Pattern Recognition (CVPR) Second Workshop on

9

Observing and Understanding Hands in Action (HANDS),2016.

[5] M. Asad and G. Slabaugh. Spore: Staged probabilistic re-gression for hand orientation inference. Computer Vision andImage Understanding, 2017.

[6] C. M. Bishop. Pattern recognition and machine learning.springer, 2006.

[7] A. Bulat and G. Tzimiropoulos. Human pose estimation viaconvolutional part heatmap regression. In European Confer-ence on Computer Vision, pages 717–732. Springer, 2016.

[8] V. Drouard et al. Head pose estimation via probabilistic high-dimensional regression. In ICIP, 2015.

[9] J.-L. Durrieu et al. Lower and upper bounds for approxi-mation of the kullback-leibler divergence between gaussianmixture models. In ICASSP, 2012.

[10] D. Eigen, C. Puhrsch, and R. Fergus. Depth map predictionfrom a single image using a multi-scale deep network. InAdvances in neural information processing systems, pages2366–2374, 2014.

[11] G. Fanelli, M. Dantone, J. Gall, A. Fossati, and L. Van Gool.Random forests for real time 3d face analysis. InternationalJournal of Computer Vision, 101(3):437–458, 2013.

[12] S. R. Fanello, C. Keskin, S. Izadi, P. Kohli, D. Kim,D. Sweeney, A. Criminisi, J. Shotton, S. B. Kang, andT. Paek. Learning to be a depth camera for close-range hu-man capture and interaction. ACM Transactions on Graphics(TOG), 33(4):86, 2014.

[13] L. Ge, H. Liang, J. Yuan, and D. Thalmann. Robust 3d handpose estimation in single depth images: from single-viewcnn to multi-view cnns. In Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition, pages3593–3601, 2016.

[14] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea-ture hierarchies for accurate object detection and semanticsegmentation. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 580–587,2014.

[15] K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask r-cnn.arXiv preprint arXiv:1703.06870, 2017.

[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn-ing for image recognition. In Proceedings of the IEEE con-ference on computer vision and pattern recognition, pages770–778, 2016.

[17] J. R. Hershey et al. Approximating the kullback leibler diver-gence between gaussian mixture models. In ICASSP, 2007.

[18] C. Keskin, F. Kırac, Y. E. Kara, and L. Akarun. Hand poseestimation and hand shape classification using multi-layeredrandomized decision forests. In European Conference onComputer Vision, pages 852–863. Springer, 2012.

[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenetclassification with deep convolutional neural networks. InAdvances in neural information processing systems, pages1097–1105, 2012.

[20] S. Lathuiliere, R. Juge, P. Mesejo, R. Munoz-Salinas, andR. Horaud. Deep mixture of linear inverse regressions ap-plied to head-pose estimation. In IEEE Conference on Com-puter Vision and Pattern Recognition, volume 3, page 7,2017.

[21] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature,521(7553):436–444, 2015.

[22] X. Liu, W. Liang, Y. Wang, S. Li, and M. Pei. 3d head poseestimation with convolutional neural network trained on syn-thetic images. In Image Processing (ICIP), 2016 IEEE Inter-national Conference on, pages 1289–1293. IEEE, 2016.

[23] T. Narihira, M. Maire, and S. X. Yu. Direct intrinsics: Learn-ing albedo-shading decomposition by convolutional regres-sion. In Proceedings of the IEEE International Conferenceon Computer Vision, pages 2992–2992, 2015.

[24] D. Pathak, P. Krahenbuhl, S. X. Yu, and T. Darrell. Con-strained structured regression with convolutional neural net-works. arXiv preprint arXiv:1511.07497, 2015.

[25] T. Pfister, J. Charles, and A. Zisserman. Flowing convnetsfor human pose estimation in videos. In Proceedings of theIEEE International Conference on Computer Vision, pages1913–1921, 2015.

[26] G. Sfikas, C. Constantinopoulos, A. Likas, and N. Galat-sanos. An analytic distance metric for gaussian mixturemodels with application in image retrieval. Artificial NeuralNetworks: Formal Models and Their Applications–ICANN2005, pages 755–755, 2005.

[27] E. Shelhamer, J. Long, and T. Darrell. Fully convolutionalnetworks for semantic segmentation. IEEE transactions onpattern analysis and machine intelligence, 39(4):640–651,2017.

[28] K. Simonyan and A. Zisserman. Very deep convolutionalnetworks for large-scale image recognition. In InternationalConference on Learning Representations, 2015.

[29] T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Di-vide the gradient by a running average of its recent magni-tude. COURSERA: Neural networks for machine learning,4(2):26–31, 2012.

[30] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bre-gler. Efficient object localization using convolutional net-works. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 648–656, 2015.

[31] J. Tompson, M. Stein, Y. Lecun, and K. Perlin. Real-timecontinuous pose recovery of human hands using convolu-tional networks. ACM Transactions on Graphics (ToG),33(5):169, 2014.

[32] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,Z. Su, D. Du, C. Huang, and P. H. Torr. Conditional randomfields as recurrent neural networks. In Proceedings of theIEEE International Conference on Computer Vision, pages1529–1537, 2015.

10

Appendix A. Derivation of Analytic Solution to IntegralsThis section includes derivations of analytic solution to integrals for function G(, ) and H() presented in the Section 3.2.

Let x = {x1, x2, · · · , xn}ᵀ ∈ Rn define the target prediction space with n dimensions. In order to facilitate analytic solutionof our proposed loss function, we write the function G(Pi, Pj) between two Gaussian distributions Pi and Pj as:

G(Pi, Pj) =

∫PiPj dx =

∫· · ·∫PiPj dx1 · · · dxn, (16)

=

∫· · ·∫ [

e− 1

2

[ (x1−µx1i )2

σx1i+···+ (xn−µxni )

2

σxni

]

(√2π)n

√σx1i · · ·σxn1

][e− 1

2

[ (x1−µx1j )2

σx1j+···+

(xn−µxnj )2

σxnj

]

(√2π)n

√σx1j · · ·σxnj

]dx1 · · · dxn, (17)

=

∫· · ·∫

e− 1

2

[ (x1−µx1i )2

σx1i+

(x1−µx1j )2

σx1j+···+ (xn−µxni )

2

σxni+

(xn−µxnj )2

σxnj

]

(2π)n√

(σx1i · · ·σxni )(σx1j · · ·σxnj )dx1 · · · dxn, (18)

=

∫· · ·∫

e

− 12

[ (x21−2x1µx1i+µx1i

2)σx1j+(x21−2x1µx1j

+µx1j2)σx1i

σx1iσx1j

+

···+ (xn−µxni )2

σxni+

(xn−µxnj )2

σxnj

]

(2π)n√

(σx1i · · ·σxni )(σx1j · · ·σxnj )dx1 · · · dxn, (19)

=

∫· · ·∫

e

[−σx1i

+σx1j2σx1i

σx1jx2+

µx1iσx1j

+µx1jσx1i

σx1iσx1j

x− 12

[ µx1i2σx1j

+µx1j2σx1i

σx1iσx1j

+

···+ (xn−µxni )2

σxni+

(xn−µxnj )2

σxnj

]]

(2π)n√

(σx1i · · ·σxni )(σx1j · · ·σxnj )dx1 · · · dxn. (20)

Given G(Pi, Pj), we can now analytically evaluate the Gaussian for∫[.] dx1 using the following theorem 2:

∫ +∞

−∞e[−ax2

1+bx1−c]dx1 = e[b2

4a−c]

√π

a. (21)

We compare left hand side of Equation 21 with Equation 20 to determine a, b and c as:

a =σx1i + σx1j

2σx1iσx1j

, (22)

b =µx1iσx1j + µx1jσx1i

σx1iσx1j

, (23)

c = [µx1i

2σx1j + µx1j2σx1i

2σx1iσx1j

+ · · ·+ (xn − µxni )22σxni

+(xn − µxnj )2

2σxnj]. (24)

Replacing a, b and c in the right hand side of Equation 21 to get Equation:

G(Pi, Pj) =

∫· · ·∫

e

[2µx1i

µx1j−µx1i

2−µx1j2

2(σx1i+σx1j

)− 1

2

[ (x2−µx2i )2

σx2i+

(x2−µx2j )2

σx2j+

···+ (xn−µxni )2

σxni+

(xn−µxnj )2

σxnj

]]

(2π)n−1√

2π(σx1i + σx1j )(σx2iσx2j · · ·σxniσxnj )dx2 · · · dxn. (25)

We can apply similar analytic solution for∫· · ·

∫[.] dx2 · · · dxn to arrive at the Equation:

G(Pi, Pj) =e

[ 2µx1iµx1j−µx1i2−µx1j2

2(σx1i+σx1j

)+···+

2µxniµxnj

−µxni2−µxnj

2

2(σxni+σxnj

)

]

(√2π)n

√(σx1i + σx1j ) · · · (σxni + σxnj )

. (26)

2Gaussian Integral (accesseed on 20-07-2018) http://mathworld.wolfram.com/GaussianIntegral.html

11

The second function we evaluate for helping our derivation is H(P1) defined as:

H(Pi) =

∫Pi

2 dx =

∫· · ·∫Pi

2 dx1 · · · dxn, (27)

=

∫· · ·∫e− 1

2

[ (x1−µx1i )2

σx1i+···+ (xn−µxni )

2

σxni

]

(√2π)n

√σx1i · · ·σxni

2

dx1 · · · dxn, (28)

=

∫· · ·∫

e−[ (x1−µx1i )

2

σx1i+···+ (xn−µxni )

2

σxni

]

(2π)nσx1i · · ·σxnidx1 · · · dxn, (29)

=

∫· · ·∫

e−[ x21−2x1µx1i

+µx1i2

σx1i+···+ x2n−2xnµxni

+µxni2

σxni

]

(2π)nσx1i · · ·σxnidx1 · · · dxn. (30)

Applying Equation 21, we reduce Equation 30 to:

H(Pi) =1

(2√π)n√σx1i · · ·σxni

. (31)

12