On Many-to-Many Mapping Between

Concordance Correlation Coefficient and

Mean Square Error

Vedhas PanditZD.B Chair of Embedded Intelligence for Health Care and Wellbeing,

University of Augsburg, Germany

Bjorn SchullerZD.B Chair of Embedded Intelligence for Health Care and Wellbeing,

University of Augsburg, GermanyGroup on Language, Audio, and Music (GLAM),

Imperial College London, United Kingdom

November 10, 2019

Abstract

The concordance correlation coefficient (CCC) is one of the most widelyused reproducibility indices, introduced by Lin in 1989. In addition to itsextensive use in assay validation, CCC serves various different purposesin other multivariate population-related tasks. For example, it is oftenused as a metric to quantify an inter-rater agreement. It is also oftenused as a performance metric for prediction problems. In terms of thecost function, however, there has been hardly any attempt to design oneto train the predictive deep learning models. In this paper, we presenta family of lightweight cost functions that aim to also maximise CCC,when minimising the prediction errors. To this end, we first reformulateCCC in terms of the errors in the prediction; and then as a logical nextstep, in terms of the sequence of the fixed set of errors. To elucidateour motivation and the results we obtain through these error rearrange-ments, the data we use is the set of gold standard annotations from awell-known database called ‘Automatic Sentiment Analysis in the Wild’(SEWA), popular thanks to its use in the latest Audio/Visual EmotionChallenges (AVEC’17 and AVEC’18). We also present some new and in-teresting mathematical paradoxes we have discovered through this CCCreformulation endeavour.

1

arX

iv:1

902.

0518

0v1

[cs

.LG

] 1

4 Fe

b 20

19

1 Introduction

The need to quantify inter-rater, inter-device or inter-method agreement arisesoften in the field of biometrics. The ‘inter-rater agreement’ and ‘inter-deviceagreement’ scenarios refer to the situations, where quantifying the extent ofagreement between two or more examiners or devices is of primary interest. Forexample, before any new instrument can be introduced to the market, it is nec-essary to assess first whether the new assay reproduces the measurements consis-tent with the traditional gold-standard assay [21, 22, 43]. Inter-rater agreementis for example important, when severity of a specific disease is evaluated bydifferent raters during a clinical trial, and the reliability of each of their subjec-tive evaluation is to be determ ined by measuring agreement among the raters[13]. Inter-rater agreement evaluations are further popular in the fields suchas psychology, psychobiology, anthropology, and cognitive science, where sub-jectivity coupled with a multitude of factors directly influences the assessmentof the variable under observation [41]. Inter-method comparisons are made,when a bivariate population results from two disparate methods. For example,comparing the gold standard sequences (e. g., device measurements) against theprediction sequences from a trained machine learning model, or the annotationsequences from an indepedent observer.

A quest for a summary statistic, that effectively represents the extent ofassociation between two variables, has led researchers to invent several differentindices. As for the nominal and ordinal classification tasks, Cohen’s kappa coef-ficient (κ) [36, 12, 6], and intraclass correlation coefficient (ICC) [9, 17] are themore commonly used reproducibility indices. The Pearson correlation coefficientor simply the ‘correlation coefficient’ (ρ) [10, 11, 30], concordance correlationcoefficient (ρc) [21] are arguably the most popular performance measures whenit comes to the regression tasks and the ordinal classifications. In this paper,we focus on the problem involving only the bivariate population featuring thecontinuous regression values, or the ordinal classes.

While there exist multiple ways to interpret a correlation coefficient (ρ)[20, 38], ρ essentially represents the extent to which a linear relationship betweentwo variables exists. ρ is the covariance of the two variables divided by theproduct of their standard deviations.

2

Thus, given a bivariate population X := (xi)N1 and Y := (yi)

N1 ,

ρ =cov(X,Y )

σXσY=

σXYσXσY

, (1)

=

n∑i=1

(xi − µX)(yi − µY )√√√√ n∑i=1

(xi − µX)2

√√√√ n∑i=1

(yi − µY )2

, (2)

where, µX =1

N

N∑i=1

xi, σX =

√√√√ 1

N

N∑i=1

(xi − µX)2,

µY =1

N

N∑i=1

yi, σY =

√√√√ 1

N

N∑i=1

(yi − µY )2,

and cov(X,Y) = σXY =1

N

n∑i=1

(xi − µX)(yi − µY ).

Covariance is a measure of the strength of the joint variability between twovariables. When the greater values of one are often associated with the lesservalues of anoother, the covariance is negative. When they grow together, thecovariance is positive. The magnitude of the covariance is harder to interpret, asit depends largely on how deviated the magnitudes of the two variables are fromtheir respective mean values. It is thus normalised by the standard deviationsof both the variables to effectively yield ρ. To compute the covariance in thenumerator in Equation (2), X and Y are first centred around zero by subtract-ing the mean of each of the variables separately, before the sum of products ofthe centred variables is obtained. The scales of the variables, too, are then nor-malised by the denominator. ρ is, therefore, a centred and standardised sum ofinner-product of the two variables. The magnitude of the denominator can onlybe greater than or equal to that of the numerator owing to the CauchySchwarzinequality. Thus, ρ ∈ [−1, 1] [20].

While ρ signifies a linear relationship, the ρ measure fails to quantitativelydistinguish between a linear relationship and an identity relationship. The ρmeasure also fails to quantitatively distinguish between the linear relationshipwith a constant offset, the one without any offset, and an identity relationship.In summary, it fails to capture any departure from the 45◦ (slope = 1) line, i. e.,any shifts in the scale (slope) and the location (offset). Thus, while successfulin capturing the precision of the linear relationship, the ρ measure completelymisses out on the accuracy.

Lin in 1989 proposed CCC or Lin’s coefficient (ρc), which is a product of ρwith the term Cb that penalises such deviations in the scale and the location [21].The Cb component captures the accuracy, while the ρ component represents the

3

precision. Formally,

Cb =2(

v +1

v+ u2

) , (3)

where, v =σXσY

= the scale shift, (4)

and u =(µX − µY )√σXσY

= the location shift relative to the scale. (5)

Substituting Equation (4) and Equation (5) in Equation (3), we get

Cb =2(

σXσY

+σYσX

+

(µX − µY√σXσY

)2) ,

=2σXσY

σ2X + σ2

Y + (µX − µY )2. (6)

ρc is, thus, given by the following:

ρc = ρCb,

=2ρ

(v +1

v+ u2)

,

=2ρ

σXσY

+σYσX

+(µX − µY )2

σXσY

,

=2ρσXσY

σ2X + σ2

Y + (µX − µY )2. (7)

The concordance correlation coefficient (ρc) has the following characteristics:

−1 ≤ − |ρ| ≤ ρc ≤ |ρ| ≤ 1.

ρc = 0 if and only if: ρ = 0.

ρc = ρ if and only if: σ1 = σ2 and µ1 = µ2.

ρc = ±1 if and only if:

(µX − µY )2 + (σX − σY )2 + 2σXσY (1∓ ρ) = 0,

i. e., if and only if:

ρ = ±1, σX = σY , and µX = µY ,

i. e., if and only if:

xi and yi are in perfect (ρc = 1) agreement, or

xi and yi are in perfect reverse (ρc = −1) agreement.

Since Lin’s pioneering work, numerous articles have been published advanc-ing the field. The CCC-measure is based on the expected value of the squared

4

difference between X and Y . In terms of the distance function used, a moregeneralised version has since been proposed [16, 15], also establishing its similar-ities to the kappa and weighted kappa coefficients. Some others have extendedthe applicability of CCC to more than two measurements by proposing new reli-ability coefficients, e. g., the overall concordance correlation coefficient [3, 1, 2].Alternative estimators for evaluating agreement and reproducibility based onthe CCC have also been proposed [31, 37]. Comparing the CCC against thepreviously existing four intraclass correlation coefficients presented in [35, 24],Nickerson presents a strong critique of the contributions of the CCC-measure inevaluating reproducibility [26]. The usability and apparent paradoxes associatedwith the reliability coefficients have been thoroughly and vehemently debatedupon [8, 44, 18]. However, CCC remains arguably one of the most popular re-producibility indices, used in a wide range of fields, e. g., cancer detection andtreatment [27, 25], comparison of the analysis method for fMRI brain scans [19],or Magnetic resonance fingerprinting [23]. The popularity of the measure hasencouraged researchers to publish macros and software packages likewise [4, 7].When it comes to instance-based ordinal classification, regression, or a sequenceprediction task, the machine learning community likewise has begun adaptingCCC as the performance measure of choice [39, 28, 29]. Take the case of the‘Audio/Visual Emotion Challenge and Workshops’ (AVEC) for example. Theshift is noticeable, with early challenges using RMSE as the winning criteria,to now CCC in those recently held [32, 40, 33, 34]. Almost without exception,the winners of these challenges have lately used deep learning models, which aretrained to model the input to output (the raw data or features to prediction)mapping through minimisation of a cost function. A cost function nominallycaptures the difference between a prediction from a model and the desired out-put; its job, consequently, is to encourage a model to drive the prediction of themodel close to the desired value. While the shift in the community to use theCCC measure as a performance metric is definitely underway, no attempts havebeen made to design a cost function specifically tailored to boost CCC, barringa lone exception [42].

The cost function used in [42] is directly the CCC, which is computationallyexpensive to use at every training step. This is because, the computation ofCCC necessitates computation of standard deviations of the gold standard andthe prediction, covariance between the gold standard and the prediction, andthe difference between the mean values at every iteration. Also, because CCCis being used as a cost function, the partial derivative of CCC with respect tothe outputs needs to be recalculated as well, to propagate the error down to theinput layers using the backpropagation algorithm in neural networks – again,at every step in the training iteration. In this paper, we therefore identify andisolate workable lightweight functions which directly have an impact on the CCCmetric. We achieve this by dividing the CCC into its constituent componentsthrough reformulation of the CCC in terms of the prediction errors. Recognisingthe terms that are affected by the error or the prediction ‘sequence’ alone, wepropose a family of candidate cost functions.

We present next the overall organisation of the paper. In Section 3, we

5

formally define the problem we attempt to tackle. In Section 4 and Section 5,we rework the CCC formulation through two slightly different substitutions interms of the prediction sequences. Interestingly, we arrive at mutually contra-dictory requirements in terms of the redistribution of error values. We presentthese interesting insights and paradoxes in Section 6. We then go on to establishconditions that help us conclusively determine which of the two reformulationsyields us a better CCC in Section 7. We supplement our findings with illustra-tions in Section 8. Learning from these insights, we present a family of candidatecost functions in Section 9. In Section 10, we summarise our motivation, theresulting findings and our planned future work as we conclude.

6

2 Many to Many Mapping between MSE andCCC In General

∵ σ2X + σ2

Y − 2σXY =1

N

N∑i=1

(xi − µX)2 +1

N

N∑i=1

(yi − µY )2 − 2

N

N∑i=1

(xi − µX)(yi − µY )

(8)

=1

N

N∑i=1

[(xi − µX)2 + (yi − µY )2 − 2(xi − µX)(yi − µY )

](9)

=1

N

N∑i=1

(xi − µX − yi + µY )2 (10)

=1

N

N∑i=1

[(xi − yi)− (µX − µY )]2

(11)

=1

N

N∑i=1

[(xi − yi)2 + (µX − µY )2 − 2(xi − yi)(µX − µY )

](12)

= MSE+1

N

N∑i=1

[(µX − µY )2 − 2(xi − yi)(µX − µY )

](13)

= MSE+1

N

N∑i=1

(µX − µY )2 − 2

N

N∑i=1

(xi − yi)(µX − µY )

(14)

= MSE+1

N·N · (µX − µY )2 (15)

− 2

N·(x1 − y1 + x2 − y2 + · · ·+ xN − yN

)·(x1 + x2 + · · ·+ xN

N− y1 + y2 + · · ·+ yN

N

)= MSE+

1

N·N · (µX − µY )2 − 2

N

(NµX −NµY

)(µX − µY

)(16)

= MSE−(µX − µY )2 (17)

∴ σ2X + σ2

Y + (µX − µY )2 = MSE+2σXY (18)

7

∴ ρc =2σXY

σ2X + σ2

Y + (µX − µY )2(19)

=2σXY

MSE + 2σXY=

1

1 + MSE2σXY

(20)

=

(1 +

MSE

2σXY

)−1

(21)

3 Many to Many Mapping for a Fixed Set ofPrediction Errors

Inspired by the discovery that the predictions with identical mean square errorcan result in different values of CCC, we attempt to decouple the components ofCCC that are dependent on merely the magnitudes of errors, from those directlyimpacted by the sequence of errors. We, thus, formulate our problem as follows.

Given (1) a gold standard time series, G := (gi)N1 , and (2) a fixed set

of error values, E := (ei)N1 (thus a fixed mean square error (MSE)), find the

distribution(s) of error values that achieve(s) the highest possible CCC.Let the prediction and the gold standard sequence be X := (xi)

N1 and Y :=

(yi)N1 , not necessarily in that order. As the formula for ρc is symmetric with

respect to X and Y , which variable represents what sequence does not matter,as far as ρc is concerned.

From Equation (2) and Equation (7), we note:

ρc =2σXY

(µY − µX)2 + σ2X + σ2

Y

, (22)

where, σXY =1

N

N∑i=1

(xi − µX)(yi − µY ), (23)

µX =1

N

N∑i=1

xi, σX =

√√√√ 1

N

N∑i=1

(xi − µX)2, (24)

and µY =1

N

N∑i=1

yi, σY =

√√√√ 1

N

N∑i=1

(yi − µY )2. (25)

Now, let di := xi − yi, and µD :=1

N

N∑i=1

di. (26)

∴ µD = µX − µY , ∵ Equations (24) to (26) (27)

∴ MSE :=1

N

N∑i=1

d2i , RMSE :=

√√√√ 1

N

N∑i=1

d2i =√

MSE, MAE :=1

N

N∑i=1

|di| .

(28)

8

∴ ρc =

2

N

N∑i=1

(xi − µX)(yi − µY )

(µY − µX)2 + σ2X + σ2

Y

. ∵ Equations (22)and (23) (29)

=

2

N∑i=1

(xi − µX)(yi − µY )

N(µD)2 +

N∑i=1

(xi − µX)2 +

N∑i=1

(yi − µY )2

. ∵ Equations (24)to (26)

(30)

4 Formulation 1: Replacing (xi) with (yi + di)

ρc =

2

N∑i=1

(yi + di − µY − µD)(yi − µY )

N(µD)2 +

N∑i=1

(yi + di − µY − µD)2 +

N∑i=1

(yi − µY )2

. (31)

Rewriting the individual terms in terms of (yi − µY )

ρc =2[∑N

i=1(yi − µY )2 −∑Ni=1 µD(yi − µY ) +

∑Ni=1 di(yi − µY )

]Nµ2

D + 2∑Ni=1(yi − µY )2 − 2

∑Ni=1 µD(yi − µY ) + 2

∑Ni=1 di(yi − µY ) +

∑Ni=1 (di − µD)2

.

(32)

Now,

N∑i=1

µD(yi − µY ) = µD

N∑i=1

(yi − µY ) = µD(NµY −NµY ) ∵ Equation (25).

(33)

which cancels out a term from both the numerator and the denom-inator.

ρc =2[∑N

i=1(yi − µY )2 −(((((((((∑N

i=1 µD(yi − µY ) +∑Ni=1 di(yi − µY )

]Nµ2

D + 2∑Ni=1(yi − µY )2 −(((((

((((2∑Ni=1 µD(yi − µY ) + 2

∑Ni=1 di(yi − µY ) +

∑Ni=1 (di − µD)2

(34)

=2Nσ2

Y + 2∑Ni=1 di(yi − µY )

Nµ2D + 2Nσ2

Y + 2∑Ni=1 di(yi − µY ) +

(∑Ni=1 µ

2D − 2

∑Ni=1 diµD +

∑d2i

) .(35)

9

The terms underlined in Equation (35) above, thus, sum to zero. That is,

Nµ2D +

N∑i=1

µ2D − 2

N∑i=1

diµD = Nµ2D +Nµ2

D − 2Nµ2D = 0. (36)

∴ ρc =2Nσ2

Y + 2∑Ni=1 di(yi − µY )

���Nµ2D + 2Nσ2

Y + 2∑Ni=1 di(yi − µY ) +

(���

��∑Ni=1 µ

2D���

����

−2∑Ni=1 diµD +

∑Ni=1 d

2i

) .(37)

=

2Nσ2Y + 2

N∑i=1

(yi − µY)di

2Nσ2Y + 2

N∑i=1

(yi − µY)di +N(MSE)

∵N∑i=1

di2 = N(MSE). (38)

= 1−

(N(MSE)

2Nσ2Y + 2

∑Ni=1 yidi − 2NµY µD +N(MSE)

)∵

N∑i=1

di = NµD.

(39)

For a given gold standard Y and a set of {di} values, maximisation of ρc requires

that

N∑i=1

(yi)di is maximised.

5 Formulation 2: Replacing (yi) with (xi − di)

ρc =

2

N∑i=1

(xi − µX)(xi − di − µX + µD)

N(µD)2 +

N∑i=1

(xi − di − µX + µD)2 +

N∑i=1

(xi − µX)2

∵ Equation (30).

(40)

Continuing as in Section 4 (cf. Appendix 1), we get

ρc =2Nσ2

X − 2∑Ni=1 (xi − µX)di

2Nσ2X − 2

∑Ni=1 (xi − µX)di +N(MSE)

(41)

= 1−

N(MSE)

2Nσ2X − 2

N∑i=1

xidi + 2NµXµD +N(MSE)

. (42)

10

For a given gold standard X and a set of {di} values, maximisation of ρc requires

that

N∑i=1

(xi)di is minimised.

6 The Paradoxical Nature of the Conditions onthe Error-set

The concluding remarks of Section 4 and Section 5 imply that we arrive atmutually contradictory requirements in terms of the rearrangement of values inthe error set. Likely because, given a set of error values and a gold standardsequence, two different candidate sequences may be generated yielding a highCCC, for a given MSE.

However, when devising a cost function in terms of the predicted sequenceitself, the requirements implied by both formulations converge to being the samefortunately. We discuss this rediscovery of consistency, arising interestinglyout of contradictory insights next. To this end, we formally (and this timeunambiguously) redefine the symbols for prediction sequence, gold standardsequence and the error sequence – instead of using X and Y interchangeably,albeit in different sections (Sections 4 and 5).

Let G := (gi)N1 , E := (ei)

N1 , and P := (pi)

N1 be the gold standard sequence,

the error sequence and the prediction sequence respectively. Let µG, µE , andµP be the arithmatic means of the sequences G, E, and P respectively.

6.1 Paradoxical requirements in terms of the(∑N

i=1 giei

)summation

As noted in Sections 4 and 5, the formulations 1 and 2 end up in exactly thecontradictory requirements in terms of the product summation

(∑Ni=1 giei

).

Specifically, to maximise ρc:

• The formulation in the Section 4 requires maximisation of the(∑N

i=1 giei

)quantity.

• The formulation in the Section 5 requires minimisation of the(∑N

i=1 giei

)quantity.

6.2 Paradoxical requirements in terms of redistribution ofthe values in the error set

• The requirement posed by the formulation in the Section 4, i. e., the max-imisation of

(∑Ni=1 giei

), necessitates that the error values are in the same

sorted order as of the gold standard values (cf. Appendix 2).

• That is, a larger ei needs to get multiplied with the larger gi.

11

• The requirement posed by the formulation in Section 5, i. e., the minimisa-tion of

(∑Ni=1 giei

), necessitates that the error values are in the opposite

order as of the gold standard values (cf. Appendix 2).

• That is, a smaller ei needs to get multiplied with the larger gi.

In other words, taking into account not only the magnitudes, but also thesigns, the error values need to be sorted in the same order as of the elements ofthe time series, when it comes to the first ρc formulation. In case of the secondformulation, the errors need to be sorted in exactly the opposite order as of theelements of the time series.

Thus, there exist two prediction sequences that correspond to identical setof error values when compared against the gold standard sequence, but likelymaximise ρc.

6.3 Consistent requirements in terms of the(∑N

i=1 gipi

)summation

While it has been established that there exist two distinct prediction sequenceswith identical mean square error and error value set, and despite the existence ofthe contradictory conditions on the underlying error redistribution, the condi-tions for the ρc maximisation in terms of the product summation

∑Ni=1 gipi are

consistent in both the formulations. In summary, the formulations 1 and 2 endup in identical requirements in terms of the product summation

(∑Ni=1 gipi

).

The consistency in the requirement can be proven as follows.According to the first formulation,

• E = P − G, i. e., (ei)N1 = (pi)

N1 − (gi)

N1 , and the quantity

(∑Ni=1 giei

)needs to be maximised.

• That is, the quantity(∑N

i=1 gi(pi − gi))

needs to be maximised,

• implying the quantity(∑N

i=1 gi(pi))

needs to be maximised, since(∑N

i=1 g2i

)is constant for any given gold standard.

According to the second formulation,

• E = G − P , i. e., (ei)N1 = (gi)

N1 − (pi)

N1 , and the quantity

(∑Ni=1 giei

)needs to be minimised.

• That is, the quantity(∑N

i=1 gi(gi − pi))

needs to be minimised.

• implying the quantity(∑N

i=1 gi(pi))

needs to be maximised, since(∑N

i=1 g2i

)is constant for any given gold standard.

12

7 The Quest for Highest Achievable CCC

Having obtained the two prediction sequences corresponding to the same set oferror values, let us now compare the two concordance correlation coefficientswe obtain. Our attempt here is to determine conclusively which of the twoprediction sequences will correspond to the higher of the two CCCs. If this isnot possible, our aim here is to, at the very least, establish the conditions underwhich one can choose the prediction sequence conclusively.

The Chebyshev’s sum inequality [14] states that

if a1 ≥ a2 ≥ · · · ≥ an and b1 ≥ b2 ≥ · · · ≥ bn,

then1

n

n∑k=1

ak · bk ≥

(1

n

n∑k=1

ak

)(1

n

n∑k=1

bk

), (43)

and if a1 ≤ a2 ≤ · · · ≤ an and b1 ≥ b2 ≥ · · · ≥ bn,

then1

n

n∑k=1

akbk ≤

(1

n

n∑k=1

ak

)(1

n

n∑k=1

bk

). (44)

Let1

E := (1ei)

N1 , and

2

E := (2ei)

N1 denote the two error sequences which relate

to the optimal redistributions per the formulations in Section 4 and Section 5

respectively, such that indexing of1

E and2

E is consistent with the sorted rear-rangement of G, i. e., G := (gi)

N1 .

If for G, g1 ≥ g2 · · · ≥ gN , (45)

then for1

E,1e1 ≥ 1

e2 · · · ≥ 1eN , (46)

and for2

E,2e1 ≤ 2

e2 · · · ≤ 2eN , (47)

i. e.,1

E,1eN ≥ 1

eN−1 · · · ≥ 1e1, (48)

∵2ej =

1eN−j ∀ j ∈ N : j ∈ [1, N ] (49)

From Equations (39) and (42),

ρc1 = 1−

(N(MSE)

2Nσ2G + 2

∑Ni=1 gi

1ei − 2NµGµE +N(MSE)

), (50)

ρc2 = 1−

(N(MSE)

2Nσ2G − 2

∑Ni=1 gi

2ei + 2NµGµE +N(MSE)

). (51)

As per the Equations (46) and (47), the denominators for expressions for bothρc1 and ρc2 are strictly-non negative (referring to the Chebyshev’s sum inequalityfrom the Equations (43) and (44)). Therefore, both the error redistributionsresult in the sequences that are guaranteed to be non-negatively correlated. ρc

13

approaches 0 only when the MSE is large.

Suppose ρc1 ≥ ρc2

This is true if and only if

N(MSE)

2Nσ2G + 2

∑Ni=1 gi

1ei − 2NµGµE +N(MSE)

≤ N(MSE)

2Nσ2G − 2

∑Ni=1 gi

2ei + 2NµGµE +N(MSE)

,

which is true only if

2Nσ2G + 2

N∑i=1

gi1ei − 2NµGµE +N(MSE) ≥ 2Nσ2

G − 2

N∑i=1

gi2ei + 2NµGµE +N(MSE),

which is true only if 2

N∑i=1

gi1ei − 2NµGµE ≥ − 2

N∑i=1

gi2ei + 2NµGµE ,

which is true only if

N∑i=1

gi1ei +

N∑i=1

gi2ei ≥ 2NµGµE ,

which is true only if

1

N

N∑i=1

gi1ei +

1

N

N∑i=1

gi2ei ≥

(1

N

N∑k=1

gi

)(1

N

N∑k=1

(1ei +

2ei

)),

which is true only if

1

N

N∑i=1

gi

(1ei +

1eN−i+1

)≥

(1

N

N∑k=1

gi

)(1

N

N∑k=1

(1ei +

1eN−i+1

)).

(52)

We note that, the scatter plot of the series (x, y) =(i,(1ei +

1eN−i+1

))is sym-

metric with respect to x = N+12 . Because the sequence (

1ei +

1eN−i+1) features

both the similarly ordered and the oppositely ordered error components, noguarantees can be made in terms of veracity of Equation (52) using either theChebyshev’s sum inequality (Equations (43) and (44)) or the rearrangement in-equality (Appendix 2). One can only compute the two sides of the Equation (52)to determine which redistribution of error values would result in the predictionsequence with the highest CCC.

8 Examples and Illustrations

Figure 1 illustrates what it means to ‘redistribute the errors’, and kinds ofprediction sequences this translates to when using each of the two CCC refor-mulations. The data used to generate Figure 1 are the gold standard sequencesof arousal levels from the SEWA database. The database has recently gained alot of popularity, as it was used in the recent AVEC challenges [34, 33].

Prediction 1 in Figure 1 corresponds to maximisation of ρc1(Equation (50)),while prediction 2 corresponds to ρc2 maximisation (Equation (51)). Because

14

Figure 1: Illustration of two candidate prediction sequences having an identicalset of prediction errors (thus, the same mean square error), when comparedagainst a common gold standard. Different ordering of these errors gives riseto different prediction sequences, resulting in different CCCs against the verysame gold standard sequence. In the figure above, gold standard sequence usedis the arousal level for one of the subjects from the SEWA German database(more specifically, from the challenge data of and ), shown in Green. The twoprediction sequences are shown in red and blue in the first row of plots.

the error distribution we chose in this simulation happens to be strictly positive,the errors carry an identical sign and the minumum of the errors is close to zero.Correspoding to this minimum error, we note that the first prediction sequenceattempts to closely follow the lower values in the gold standard, while the laterclosely follows the larger values present in the gold standard, thanks to the errorrearrangement discussed in Section 6 and Appendix 3.

We also note that the shape of the two prediction sequences is quite sim-ilar to one another, but far from being identical. The overall shape is non-linearly stretched in the vertical direction, i. e., corresponding to the differenterror values. The two plots at the bottom provide a better insight into thisvertical stretching in a comparative sense. Comparing the magenta and thegreen-coloured plot, it can be readily seen that the difference between the twoprediction sequences is at its highest at the extremities of the gold standard.This is expected, as per the Equations (45) to (47).

15

9 Designing a cost function

Both the reformulations imply that, in order to achieve the highest possible ρc, itis necessary to minimise the Lp norm of the error values (i. e., minimise MSE =1N

∑Ni=1(gi − pi)

2, while simultaneously maximising the DotProduct(G,P ) =DP =

∑Ni=1 gipi (also called the Hadamard product).

In order to design the cost function φ(pi, gi), the only constraint we have,therefore, is that

∂

∂piMSE < 0 and

∂

∂piDP > 0, (53)

i. e.,∂

∂pi

N∑i=1

(gi − pi)2 < 0 and∂

∂pi

N∑i=1

gipi > 0,

should imply that∂

∂pif(gi, pi) < 0 ∀pi ∈ R (54)

A family of cost functions, such as the following, can now be easily designed.

(1) f(gi, pi) =

N∑i=1

(gi − pi)2 − αN∑i=1

gipi, where α > 0;

(55)

or more generally, (2) f(gi, pi) =

N∑i=1

(gi − pi)2 − αN∑i=1

(gipi)β , where α, β > 0;

(56)

even more generally, (3) f(gi, pi) =

N∑i=1

(gi − pi)2 −∑j

αj

N∑i=1

(gipi)βj , where αj , βj > 0.

(57)

Such a cost function, attempting to maximise∑Ni=1 gipi, i. e., the dot product

between the predictions and the gold standard, makes intuitive sense as well.We essentially dictate the neural network to raise the prediction values as largeas possible when dealing with large values in the gold standard sequence, anddiminish the predictions to as small as possible corresponding to the smallervalues in the gold standard sequence. The mean square error as a componentof the cost function, too, attempts to achieve this. However,the inner workingsare slightly different. The mean square component drives the prediction closerto the gold standard by the amount that is proportional to the error. The innerproduct component drives the prediction in the direction of gold standard byan amount proportional to the gold standard itself. Another way to look at thedot product term is that, we now effectively weigh the individual errors by thecorresponding gold standard.

16

10 Concluding Remarks

Keeping the deep learning models in perspective – that are capable of mappingcomplex and non-linear input to output relationships, we propose a family ofcost functions, whereby the model can also aim to maximise the concordancecorrelation coefficient, while simultaneously minimising the errors in the predic-tion.

Our proposed cost function consists of two componets. The classical costfunction, such as MSE that reduces the difference between the prediction andthe gold standard by an amount that is proportional to the difference itself inevery training iteration. Our newly introduced component of the cost functiondrives every prediction closer to the corresponding gold standard by an amountthat is proportional to the value of the gold standard itself. We derive theseproperties of the desired cost function by rigorously reformulating the CCC mea-sure in terms of the error coefficients, i. e., the difference between the predictionand the gold standard population, in two different ways.

It is a well known fact that, while each of the MAE, MSE, and RMSE costfunctions aim to reduce the difference between the prediction and the desiredoutput, they deal with the outliers in the data differently. RMSE gives arelatively high weight to large errors when compared to MAE. For XGBoostalike frameworks, the twice differentiable functions are more favourable (unlike,e. g., as in the case of MAE) [5]. In the same vein, we hope to accelerate the

model convergence when maximising∑Ni=1 gipi by introducing −α

∑Ni=1 (gipi)

β

to the cost function (β ∈ R : β > 0).With the mathematical foundations formally laid out in this paper, we now

intend to rerun our emotion recognition experiments and relevant examplesof application with newly invented cost functions. It would be interesting toinvestigate the effect of the hyperparameters present in the cost function. Weintend to optimise a model’s predictive power and time-to-convergence by tuningthe hyperparameters such as αj , βj , presenting the run-time benchmarks as ourfuture work.

11 Acknowledgements

This research was partly supported by the European Commission’s SeventhFramework Programme under grant agreement No. 338164 (ERC Starting GrantiHEARu) and the Horizon 2020 Programme through the Innovative ActionNo. 645094 (SEWA).

References

[1] Huiman X Barnhart and John M Williamson. Modeling concordance cor-relation via GEE to evaluate reproducibility. Biometrics, 57(3):931–940,2001.

17

[2] Huiman X Barnhart, Michael Haber, and Jingli Song. Overall concordancecorrelation coefficient for evaluating agreement among multiple observers.Biometrics, 58(4):1020–1027, 2002.

[3] Josep L Carrasco and Lluis Jover. Estimating the generalized concordancecorrelation coefficient through variance components. Biometrics, 59(4):849–858, 2003.

[4] Josep L Carrasco, Brenda R Phillips, Josep Puig-Martinez, Tonya S King,and Vernon M Chinchilli. Estimation of the concordance correlation co-efficient for repeated measures using SAS and R. Computer Methods andPrograms in Biomedicine, 109(3):293–304, 2013.

[5] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting sys-tem. In Proceedings of the 22nd ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining, pages 785–794, San Francisco,CA, 2016. ACM.

[6] Jacob Cohen. A coefficient of agreement for nominal scales. Educationaland Psychological Measurement, 20(1):37–46, 1960.

[7] Sara B Crawford, Andrzej S Kosinski, Hung-Mo Lin, John M Williamson,and Huiman X Barnhart. Computer programs for the concordance corre-lation coefficient. Computer Methods and Programs in Biomedicine, 88(1):62–74, 2007.

[8] Alvan R Feinstein and Domenic V Cicchetti. High agreement but lowkappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology,43(6):543–549, 1990.

[9] Ronald A Fisher. Statistical methods for research workers. Edinburgh:Oliver and Boyd, 1925.

[10] Francis Galton. Typical laws of heredity. Nature, 15:492–5, 512–4, 532–3,1877.

[11] Francis Galton. Typical laws of heredity. Proceedings of the Royal Institu-tion, 8:282–301, 1877.

[12] Francis Galton. Fingerprints. Macmillan and Company, 1892.

[13] Ying Guo and Amita K Manatunga. Nonparametric estimation of theconcordance correlation coefficient under univariate censoring. Biometrics,63(1):164–172, 2007.

[14] Godfrey Harold Hardy, John Edensor Littlewood, and George Polya. In-equalities. Cambridge University Press, 1988.

[15] Tonya S King and Vernon M Chinchilli. A generalized concordance correla-tion coefficient for continuous and categorical data. Statistics in Medicine,20(14):2131–2147, 2001.

18

[16] Tonya S King and Vernon M Chinchilli. Robust estimators of the concor-dance correlation coefficient. Journal of Biopharmaceutical Statistics, 11(3):83–105, 2001.

[17] Gary G Koch. Intraclass correlation coefficient. Encyclopedia of StatisticalSciences, 4:213, 1982.

[18] Klaus Krippendorff. Commentary: A dissenting view on so-called para-doxes of reliability coefficients. Annals of the International CommunicationAssociation, 36(1):481–499, 2013.

[19] Nicholas Lange, Stephen C Strother, Jon R Anderson, Finn A Nielsen,Andrew P Holmes, Thomas Kolenda, Robert Savoy, and Lars Kai Hansen.Plurality and resemblance in fMRI data analysis. NeuroImage, 10(3):282–303, 1999.

[20] Joseph Lee Rodgers and W Alan Nicewander. Thirteen ways to look at thecorrelation coefficient. The American Statistician, 42(1):59–66, 1988.

[21] Lawrence I-Kuei Lin. A concordance correlation coefficient to evaluatereproducibility. Biometrics, pages 255–268, 1989.

[22] Lawrence I-Kuei Lin. Assay validation using the concordance correlationcoefficient. Biometrics, pages 599–604, 1992.

[23] Dan Ma, Vikas Gulani, Nicole Seiberlich, Kecheng Liu, Jeffrey L Sunshine,Jeffrey L Duerk, and Mark A Griswold. Magnetic resonance fingerprinting.Nature, 495(7440):187, 2013.

[24] Kenneth O McGraw and Seok P Wong. Forming inferences about someintraclass correlation coefficients. Psychological Methods, 1(1):30, 1996.

[25] Muhammed Murtaza, Sarah-Jane Dawson, Dana WY Tsui, Davina Gale,Tim Forshew, Anna M Piskorz, Christine Parkinson, Suet-Feung Chin,Zoya Kingsbury, Alvin SC Wong, et al. Non-invasive analysis of acquiredresistance to cancer therapy by sequencing of plasma DNA. Nature, 497(7447):108, 2013.

[26] Carol AE Nickerson. A note on “A concordance correlation coefficient toevaluate reproducibility”. Biometrics, pages 1503–1507, 1997.

[27] Satoshi Nishizuka, Lu Charboneau, Lynn Young, Sylvia Major, William CReinhold, Mark Waltham, Hosein Kouros-Mehr, Kimberly J Bussey, Jae KLee, Virginia Espina, et al. Proteomic profiling of the NCI-60 cancer celllines using new high-density reverse-phase lysate microarrays. Proceedingsof the National Academy of Sciences, 100(24):14229–14234, 2003.

[28] Vedhas Pandit, Nicholas Cummins, Maximilian Schmitt, Simone Hantke,Franz Graf, Lucas Paletta, and Bjorn Schuller. Tracking authentic andin-the-wild emotions using speech. In Proc. 1st ACII Asia 2018, Beijing,P. R. China, May 2018. AAAC, IEEE.

19

[29] Vedhas Pandit, Maximilian Schmitt, Nicholas Cummins, Franz Graf, LucasPaletta, and Bjorn Schuller. How Good Is Your Model ‘Really’? On ‘Wild-ness’ of the In-the-wild Speech-based Affect Recognisers. In Proceedings20th International Conference on Speech and Computer, SPECOM 2018,Leipzig, Germany, September 2018. ISCA, Springer.

[30] Karl Pearson. Note on regression and inheritance in the case of two parents.Proceedings of the Royal Society of London, 58:240–242, 1895.

[31] Hui Quan and Weichung J Shih. Assessing reproducibility by the within-subject coefficient of variation with random effects models. Biometrics,pages 1195–1203, 1996.

[32] Fabien Ringeval, Bjorn Schuller, Michel Valstar, Roddy Cowie, and MajaPantic, editors. Proceedings of the 5th International Workshop on Au-dio/Visual Emotion Challenge, AVEC’15, co-located with MM 2015, Bris-bane, Australia, October 2015. ACM, ACM.

[33] Fabien Ringeval, Michel Valstar, Jonathan Gratch, Bjorn Schuller, RoddyCowie, and Maja Pantic, editors. Proceedings of the 7th InternationalWorkshop on Audio/Visual Emotion Challenge, AVEC’17, co-located withMM 2017, Mountain View, CA, October 2017. ACM, ACM.

[34] Fabien Ringeval, Bjorn Schuller, Michel Valstar, Roddy Cowie, HeysemKaya, Maximilian Schmitt, Shahin Amiriparian, Nicholas Cummins, Den-nis Lalanne, Adrien Michaud, Elvan Ciftci, Huseyin Gulec, Albert AliSalah, and Maja Pantic. AVEC 2018 Workshop and Challenge: BipolarDisorder and Cross-Cultural Affect Recognition. In Proceedings of the 8thInternational Workshop on Audio/Visual Emotion Challenge, AVEC’18,co-located with the 26th ACM International Conference on Multimedia, MM2018, Seoul, South Korea, October 2018. ACM, ACM.

[35] Patrick E Shrout and Joseph L Fleiss. Intraclass correlations: uses inassessing rater reliability. Psychological Bulletin, 86(2):420, 1979.

[36] Nigel C Smeeton. Early history of the kappa statistic. Biometrics, 41:795,1985.

[37] Roy T St. Laurent. Evaluating agreement with a gold standard in methodcomparison studies. Biometrics, pages 537–545, 1998.

[38] Richard Taylor. Interpretation of the correlation coefficient: A basic review.Journal of Diagnostic Medical Sonography, 6(1):35–39, 1990.

[39] George Trigeorgis, Fabien Ringeval, Raymond Bruckner, Erik Marchi, Mi-halis Nicolaou, Bjorn Schuller, and Stefanos Zafeiriou. Adieu Features?End-to-End Speech Emotion Recognition using a Deep Convolutional Re-current Network. In Proceedings 41st IEEE International Conference onAcoustics, Speech, and Signal Processing, ICASSP 2016, pages 5200–5204,Shanghai, P. R. China, March 2016. IEEE, IEEE.

20

[40] Michel Valstar, Jonathan Gratch, Bjorn Schuller, Fabien Ringeval, RoddyCowie, and Maja Pantic, editors. Proceedings of the 6th InternationalWorkshop on Audio/Visual Emotion Challenge, AVEC’16, co-located withMM 2016, Amsterdam, The Netherlands, October 2016. ACM, ACM.

[41] Joseph P Weir. Quantifying test-retest reliability using the intraclass cor-relation coefficient and the sem. The Journal of Strength & ConditioningResearch, 19(1):231–240, 2005.

[42] Felix Weninger, Fabien Ringeval, Erik Marchi, and Bjorn Schuller. Dis-criminatively trained recurrent neural networks for continuous dimensionalemotion recognition from audio. In Proceedings of the 25th InternationalJoint Conference on Artificial Intelligence, IJCAI 2016, pages 2196–2202,New York City, NY, July 2016. IJCAI/AAAI.

[43] James O Westgard and Marian R Hunt. Use and interpretation of commonstatistical tests in method-comparison studies. Clinical Chemistry, 19(1):49–57, 1973.

[44] Xinshu Zhao, Jun S Liu, and Ke Deng. Assumptions behind intercoderreliability indices. Annals of the International Communication Association,36(1):419–480, 2013.

21

A 1: Formulation 2: Replacing (yi) with (xi−di)

ρc =2∑Ni=1(xi − µX)(xi − di − µY + µD)

N(µD)2 +∑Ni=1(xi − di − µX + µD)2 +

∑Ni=1(xi − µX)2

. (58)

Rewriting the individual terms in terms of (xi − µX)

ρc =2[∑N

i=1(xi − µX)2 +∑Ni=1 µD(xi − µX)−

∑Ni=1 di(xi − µX)

]Nµ2

D + 2∑Ni=1(xi − µX)2 + 2

∑Ni=1 µD(xi − µX)− 2

∑Ni=1 di(xi − µX) +

∑Ni=1 (di − µD)2

.

(59)

Now,

N∑i=1

µD(xi−µX) = µD

N∑i=1

(xi−µX) = µD(NµX−NµX) ∵ Equation (25).

(60)which cancels out a term from both the numerator and the denominator.

ρc =2[∑N

i=1(xi − µX)2 +(((((((

((∑Ni=1 µD(xi − µX) −

∑Ni=1 di(xi − µX)

]Nµ2

D + 2∑Ni=1(xi − µX)2 +(((

(((((((

2∑Ni=1 µD(xi − µX) − 2

∑Ni=1 di(xi − µX) +

∑Ni=1 (di − µD)2

(61)

=2Nσ2

X − 2∑Ni=1 di(xi − µY )

Nµ2D + 2Nσ2

X − 2∑Ni=1 di(xi − µX) +

(∑Ni=1 µ

2D − 2

∑Ni=1 diµD +

∑d2i

) .(62)

The terms underlined in Equation (62) above, thus, sum to zero. That is,

Nµ2D +

N∑i=1

µ2D − 2

N∑i=1

diµD = Nµ2D +Nµ2

D − 2Nµ2D = 0. (63)

∴ ρc =2Nσ2

X − 2∑Ni=1 di(xi − µX)

���Nµ2D + 2Nσ2

X − 2∑Ni=1 di(xi − µX) +

(��

���∑Ni=1 µ

2D���

����

−2∑Ni=1 diµD +

∑Ni=1 d

2i

) .(64)

=2Nσ2

X − 2∑Ni=1 (xi − µX)di

2Nσ2X − 2

∑Ni=1 (xi − µX)di +N(MSE)

∵N∑i=1

di2 = N(MSE).

(65)

= 1−

(N(MSE)

2Nσ2X − 2

∑Ni=1 xidi + 2NµXµD +N(MSE)

)∵

N∑i=1

di = NµD.

(66)

For a given gold standard Y and a set of {di} values, maximisation of ρc requires

that

N∑i=1

(yi)di is maximised.

22

B 2: The Rearrangement Inequality

The rearrangement inequality is a theorem concerning the rearrangements oftwo sets, to maximise and minimise the sum of element-wise products.

Denoting the two sets (a) = {a1, a2, · · · an} and (b) = {b1, b2, · · · bn}, let (a)and (b) be the sets where elements of (a) and (b) are arranged in the ascendingorder respectively such that

a1≤ a2 ≤ · · ·≤ an (67)

and b1 ≤ b2 ≤ · · ·≤ bn (68)

The rearrangement inequality states that

n∑j=1

aj bn+1−j ≤n∑j=1

ajbj ≤n∑j=1

aj bj . (69)

More explicitly,

anb1 + · · ·+ a1bn ≤ aσ(1)b1 + · · ·+ aσ(n)bn ≤ a1b1 + · · ·+ anbn,(70)

for every permutation {aσ(1), aσ(2), · · · , aσ(n)} of {a1, · · · , an}.

For the unsorted ordered sets (a) and (b), the two sets are said to be ‘similarlyordered’ if (aµ − aν)(aµ − aν) ≥ 0 for all µ, ν, and ‘oppositely ordered’ if theinequality is always reversed. With this notion of ‘similar’ and ‘opposite’ order-ing, the maximum summation corresponds to the similar ordering of (a) and(b). The minimum corresponds to the opposite ordering of (a) and (b).

23