use of information-theory methods in the analysis of measurement devices
Post on 12-Aug-2016
Embed Size (px)
USE OF INFORMATION-THEORY
OF MEASUREMENT DEVICES
Ya. M. Vyrk
METHODS IN THE ANALYS IS
UDC 681.2.001.5 : 519.92
In recent years a number of criteria based on the notion of quantity of information introduced by Shannon [1, 2] have been suggested for evaluating measurement means and processes. The principal concept of information the- ory of measurements is the proposition that the performance index of a measurement device is the value of the aver- age quantity of information  contained in the instrument readings (z) on the values of the measured quantity (x). The quantity of information is defined as I (x, z) = H(x ) -H(x /z ) , where H(x) and H(x/z) are respectively the en- tropy and conditional entropy of the measured discrete quantity. For a continuous x the value I(x, z) is defined as the difference between the differential and conditional differential entropies.
Heretofore, however, such an approach has been applied only in the analysis of static characteristics of mea- surement devices.
In analyzing the dynamics of the measurement process by information methods the basic criteria are the quan- tity of information per measurement and the quantity of information per unit time as defined on this basis.
It is assumed that the measured process is stationary. Such an approach is convenient in analyzing digital and cyclic methods of measurement.
In many cases both the input and output process of the measurement device are continuous stationary random processes . Below we shall consider the possibilities of applying information theory in estimating the dynamics of the measurement process in such a case.
The original model is shown in Fig. 1, where x (0 is the input (measured) signal which is a stationary random process; k(t) is the impulse transient function of the device for the input process; y(t) is the additive error referred to the output of the device and is a continuous stationary random process which is uncorrelated with x(t); z(t) is the output signal, the relationships
being valid .
X1 ( [ ) =
j " x ( t - -x ) k(x) aT, z(0 =x l (0+Y( t ) --co
As in the simplest problems in statistical dynamics of linear automatic conuol systems, let us assume that ~(t) and y(t) have zero mathematical expectations and that their spectral densities Sx(W ) and 8y(W) are known and may be represented by rational-fraction functions of w 2. The transfer function of the device, which corresponds to the impulse transient function k (t), is K (j w). The device is linear with constant parameters.
As far as the distributions of the random processes are concerned, we shall restrict the analysis to a normal dis- tribution. This, of course, narrows the range of problems considered, but since in a number of cases one may assume that both the input process and the error of the device have a normal distribution , the study of the problem in the given plan is substantiated.
The basic information characteristic of the device is assumed to be the quantity of information on the input process which is contained in the output signal. On the basis of  we have
Translated from Izmeritel 'naya Tekhnika, No. 3, pp. 5-7, March, 1971. Original article submitted Novem- ber 4, 1968.
9 1971 Consultants Bureau, a division of Plenum Publishing Corporation, 227 West 17th Street, New
York, N. Y. 10011. All rights reserved. This article cannot be reproduced for any purpose whatsoever without permission of the publisher. A copy of this article is available from the publisher for $15.00.
Sx (co) Sz (co) Sx (~o) S... (,.o) - - i Sxz (~)?'
where Sz(W) is the spectral density of the output signal; under these conditions it follows that in accordance with  we have
Sz too) ~ S~ (o) I K (/o>)i2 + Sv (:~o),,
while Sxz0O) is the mutual spectral density of the processes x (t) and z (t), and
r 2 q~2 Sx~ (to) !2 =- [ g (/o~) ~ ~ .,: (co).
Finally we obtain
S (x, z) = 2~- In 1 2_ , ' S v (o ) 0
From (1) it is evident that the quantity of information obtained from the measurement device depends on the dynamic properties of K( jw) and on the error Sy(W) of the device, as well as on the measured process Sx(CO).
In order to estimate the accuracy obtained, we use the mean-square error. The output signal with no error is assumed to be the signal K (0) x (t), where K (0) is the static transmission factor of the device. The mean-square er- ror  averaged over the entire spectrum is
~ = [[ K (o) - - K (ico)? s,< (~) + S v (o~)] do. (2) 2=
In accordance with  we have
H~ (x) ----- ~ [ K 2 (0) Sx (o) 1]
in max I.- 0~ ' do, (a)
where 0 z is determined from the equation
-- 2n min [K ~ (0) Sx (co), 021 do. (4) ~ o o
By analogy with the analysis of random quantities by the methods of information theory of measurement de- vices  we define the information eff iciency as
H e (xj
~1I -- l (x , z )
Low values of 7/I indicate that it is expedient to perform further processing of the output signal of the device. Under these conditions the achievable minimum mean-square error and the maximum information eff iciency 7/I' are of interest.
It is well known that for the case considered (stationary signals having a normal distribution) linear fi ltering is optimal  (i.e., no other method of processing the output signal, for example, nonlinear filtering, can produce a smaller mean-square error),
This means that by connecting an opt imal Wiener fi lter at the output of the device we obtain an improved signal at the filter output:
zl (t) = .t z (, - - ~) h (~) dr, - -oo
a)n 2 CO
i r I(x,Z) r
+ \ / -,;
0,1 l 10 100 c
Fig. 2 Fig. S
where h(T) is the impulse transient function of the filter and reproduces the measured process x(t) with the mini- mum mean-square error e~. Knowing ~, we use (3) and (4) to determine the corresponding s-entropy I~ 1Cx), and find the maximum information efficiency
Iq" I -- He, (x) I (x, z)
Since the minimum quantity of infon-nation He(x) required for reproducing the signal x(t) with a stipulated accuracy ~2 depends s~lely on the properties of the signal [see (3) and (4)], while the quantity of information I(x, z) obtained from the measurement device also depends on the dynamic properties and the error of the device, it follows that qI characterizes the degree to which the dynamic properties of the device and the measured process are matched. The index ~I', which interrelates the quantity of information I (x, z) and [via Hq (x)] the minimal mean-square er- ror, is the measure of the value of the information obtained from the device. Let us remember that one of the prob- lems in the theory of the value of information is finding the extremal value of the mathematical expectation of the penalty function (in our case the minimal mean-square error) for a fixed value of the quantity of information . The value of ~I' determines that portion of the overall quantity of information I(x, z), which may be used in the best case for the reproduction of x (t).
The question naturally arises: what would be the mean-square error if it were to be possible to use all of the information optimally (i.e., we are interested in the mean-square error for the case Hq(x) = I(x, z) or ~I' -- 1)? Let us call the corresponding error the information-theory minimal mean-square error, and let us denote it by ~. In order to determine ~20 we substitute I(x, z) for Hs(x) in (3), whence we find 02; then we find ~ from (4). Based on ~, we define the dispersion efficiency as
l i t - 8s
and the maximum dispersion efficiency as
Since in performing the measurements we are interested primarily in the error obtained, it follows that ~, ~T, and ~T' turn out to be clearer characteristics of the measurement device than I(x, z), 7li,and 7li'. Both ~I and ~I' characterize the degree to which the properties of the device and the measured process are matched. The value of ~T', which couples the quantity of information I(x, z) and (via g~) the mean-square error ~ , is the estimate of the value of the information obtained from the device.
The proposed information characteristics-the quantity of information I(x, z), the information efficiency ~I and 7/I' (or the corresponding accuracy characteristics, which are the information-theory minimal mean-square
error ~0 z and the dispersion efficiencies r/T and 0T ' ) may turn out to be useful in analyzing the dynamics of mea- surement devices for continuous processes and their sections. They may be used as the criteria in comparing devices of various types, and provide the possibility of estimating the degree to which individual devices are matched when they are used jointly; they also estimate the degree to which the dynamic properties of a device and the measured process are matched, etc.
In connection with the extremality of the normal distribution for the entropy (for a stipulated mean-square value of the reproduction error) one may obtain limiting estimates in the form of inequalities for a given device when the distributions are not normal.
Since the quantity of information I (x, z) depends on the properties of the device and the elements used in it via K(jw) and Sy(ca), it is obviously expedient to use the information criteria I(x, z), 7/I and ~/I' (or gz0, ~T and ~T') for the purpose of maximizing or minimizing these criteria as early as in the design stage. Under these con- ditions it is necessary to consider the statis