Anticipation Problems in Aerospace Psychology: Application of Delphi Method

The Delphi Method (DM)

Useful information of a very broad nature can be often obtained, as is known, in a timely and cost-effective fashion about an event, an issue, a possibility, a situation, a methodology or a parameter of interest by processing, in a systemic way, opinions (feedback) of a selected group of experts, using DM. In some applied science and engineering undertakings such an approach could be implemented instead of conducting costly and time consuming actual experiments. The DM is the best known and the most widely used technique of this type [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. The method has been proven to be useful in numerous decision making problems in forecasting, auditing, planning and business management. The experts do not interact and express their opinions independently, by answering sets of highly focused questions from the beforehand developed questionnaires. The information obtained from a particular expert is treated as an independent sample of a random (uncertain) variable (characteristic) of interest, and more or less sophisticated statistical methods for analyzing random data are being subsequently used to process and interpret the obtained information.

The DM is a probabilistic predictive modeling technique that is based on an assumption that the deviations of expert opinions (estimates) from the “objective truth” are due to random causes, and that the application of the DM is able to “restore” such an “objective truth” with a certain degree of confidence.

The DM assesses the meaningfulness of the estimates, the degree of the agreement of the expert opinions, and can be even used to select and form the right team of experts by assessing their suitability and the levels of their expertise. After the experts estimates are processed, each expert might be given the resultant estimate. The expert might be provided also with some additional information. Using these data, he/she might re-visit and re-examine his/hers estimates. The entire procedure is repeated until the experts’ estimates become reasonably consistent. The method is most suitable, when there is a need to provide an insight into an undeveloped subject area, when a long range forecast is needed, when the past data are not relevant to the future, and when consistent historical data do not exist or are not available.

The major strengths of the DM are the use of sound reasoning; applicability to a wide range of issues; absence of the interference from group social pressures; and ease in bringing in experts. The DM’s drawbacks are, in effect, common to all judgmental forecasts: judgmental nature of such forecasts; difficulties in selecting suitable experts; evaluating their expertise and other human qualities important for obtaining an objective and consistent information; difficulties in estimating how accurate the obtained information is; numerous challenges in creating effective and goal-oriented questionnaires. The typical problem and complaint is the difficulty of evaluating the actual expertise and/or the conscientiousness of the experts. In the analysis below we show how three more or less well known methods of processing DM data can be applied in some anticipation problems in avionic psychology.

The following particular problems are addressed in this analysis: 1) Selection of experts; 2) DM based estimates using Student’s distribution; 3) DM based decision making support using Fischer criterion

Selection of Experts

Prior to starting questioning experts using the developed questionnaire the DM analyst has to determine the actual expertise of the experts, the thoroughness of their thinking (“human capacity factor”) and the appropriate number of experts [20]. Apart from competence in the field, a good expert should possess some important additional human qualities: 1) Creativity, i.e. the ability to address problems, for which methods of solution are unknown; 2) Heuristic ability, i.e., the ability to identify nontrivial problems; 3) Intuition (“gut feeling”, “educated guess”), i.e. the ability to see a solution without knowing why; 4) Predicativity, i.e., the ability to anticipate a solution or a potential pitfall; 5) Independence, i.e., the ability to withstand the opinions of others, even if these opinions are in majority; 6) Versatility, i.e., the ability to see a problem from a different point of view. Formalized information about the source for the experts’ answers and the degree of influence due to each source on his/her answer can be obtained, e.g., by asking the experts to complete the following Table 1:

As to the number of experts, it should be sufficient to ascertain that the essential features of the problem are captured. Too many experts may lead to significant differences of opinions, as well as to difficulties in the organization of the expert procedure and processing of the obtained data. It is advisable that the group of experts consists of not less than 20 and not more than 50 individuals. Once the number of experts is decided upon, the selection of particular individuals can be carried out in several stages. For instance, a small group of specialists capable of providing opinions could be initially nominated by the project manager (coordinator). Then the questionnaires are given to each of the specialists named. They, in turn, give the names of specialists competent in the topics under consideration. The process continues until new names do not appear any more. Such a procedure will most likely result in a complete list of competent persons in the organization (agency). The questions in a questionnaire should be formulated so as to exclude ambiguous answers. Humans answer qualitative questions (“better/worse”, “more/less”, “higher/lower”) better than quantitative questions. Questionnaires should therefore involve questions requiring qualitative answers, even if the ultimate goal is to obtain quantitative information.

Example: Do you think that Iran’s attacking Israel is impossible, highly unlikely, unlikely, less likely than more, has a 50%/50% chance, more likely than less, likely, highly likely, almost certain (circle the appropriate answer). These responses could be interpreted as 0, 12.5%, 25.0%, 37.5%, 50.0%, 62.5%, 75.0%, 87.5%, and 100% probability that the attack will occur. The questions in the questionnaires should be arranged hierarchically, i.e., more general questions should be asked first, and the more specific questions should follow.

Estimate using Student’s Distribution

Student’s t-distribution is a family of continuous distributions that arises when estimating the mean values of a normally distributed random variable in situations where the sample size is small and the standard deviation of the population is unknown. Student’s t-distribution is symmetric and bell-shaped, like the normal distribution, but has “heavier tails”, meaning that it is more prone to producing values that fall far from its mean. This makes it useful for understanding the statistical behavior of certain types of ratios of random quantities, in which variation in the denominator is amplified and may produce outlying values when the denominator of the ratio falls close to zero. In the example that follows we assess, using DM data and Student’s t-distribution the most likely value of the duration of the anticipation time. The general procedure is as follows [18].

Let a parameter (event) X , such as, in the case in question, short term anticipation (STA) time in a particular flight situation, be estimated by N independent experts. The effective estimate α of the parameter X is N = = ∑ x x x x α α φ α k k k N N (1) ( , ,..., )

1 1 2 = ∑ k k

1 = And the extent of disagreement among the expert opinions about the parameter X can be characterized by the normalizes variance N − = ∑

2 ( ) x D α α k k k N

1 = (2) ∑ α k k

1 = Here is the competence (“weight”) of the κ -th expert, and

kx is the estimated value by this expert. The statistical

significance of the obtained result can be defined as an

$$ \bar {\alpha} - \Delta \leq \alpha \leq \bar {\alpha} + \Delta \tag {3} $$ In which the value of interest can be found with the given (required, desired, specified) probability1 Q − , where Q is the probability of error. In the inequality (3), α is the mean value of the estimate and is the deviation of the actual (random) value of the estimate α from its mean value. If the variable α has a normal distribution with the mean value α and the variance D then the deviation of the actual value of the estimate α from its mean value is t $$ \Delta = \frac {t \sigma}{\sqrt {N}} \tag {4} $$ N where the t value has Student’s distribution with

1 N −

degrees of freedom, and

$$ \cdot \sigma = \sqrt {D} \cdot $$ Let the experts’ estimates of the number of seconds that a pilot needs to anticipate a particular situation of importance during a certain flight are as shown in Table 2:

All the experts have the same “weight”

$$ ^ {\prime \prime} \alpha_ {k} = 1, \mathrm {t h e} $$ acceptable probability of error is 5%, Q = and one wants to establish the confidence interval for the estimate

35.5 α =

determined by the formula (1) and the degree of

$$ d = 4. 9 0 $$

found from the formula (2). The

tables for the Student’s distribution for the number

$$ N - 1 = 9 $$ of degrees of freedom, the probability of error 0.05 Q = , and the standard deviation

4.90 2.2136 D σ = = = , yield: 2.262 t =

Then the formula (4) yields:

$$ \Delta = 1. 5 8 3 \mathrm {F r o m} $$

the inequality (3), we conclude that the actual value of the

duration of the anticipation time falls within the range

between 33.917 and 37.083 seconds, with the probability

95%.

Decision making Support using Fischer Criterion

The DM can also be applied for the evaluation, using opinions of experts (reviewers), of a particular choice (proposal, strategy, approach, technology, plan, item, product, mission, initiative, service, software, weather forecast, sea state condition, individual, algorithm, etc.), and for making a decision (choice) based on such an evaluation [18]. First, the experts are requested to fill out questionnaires aimed at the selection of the choice to be made. Then they are requested to answer another questionnaire, in which, based on the results of the analysis of the first questionnaire, the selected characteristics are ranked, i.e., are listed in order of their significance. The objective of the third questionnaire is to apply the established characteristics to evaluate the possible choices and to make the decision about the best choice that possesses the most valuable characteristics. Here is how it could be done.

Let the number of experts be $(i=1,2,...,n)$, the number of characteristics of the choice under consideration be $(j=1,2,...,m)$, and the significance numbers in the developed ranking matrix be $x_{ij}$, where refers to an expert and to a particular characteristic. The sums of all the columns and all the lines in the ranking matrix should satisfy the condition:

$$\sum_{j=1}^{m} \sum_{i=1}^{n} x_{ij} = \sum_{i=1}^{n} \sum_{j=1}^{m} x_{ij}$$

Using the ranking matrix, one can assess the degree of correlation of the expert opinions. The discrepancies can be due to different qualifications of the experts and/or to their different opinions about the particular characteristic because of different knowledge about this characteristic. If there are no equal elements (ranks) $x_{ij}$ in the ranking matrix, the concordance coefficient $C$ can be calculated as

$$C = \frac{12S}{n^2 m(m^2 - 1)}$$

where

$$S = \sum_{j=1}^{m} \left( \sum_{i=1}^{n} x_{ij} - \frac{1}{m} \sum_{j=1}^{m} \sum_{i=1}^{n} x_{ij} \right)$$

is the correlation factor. If some of the elements (ranks) in the ranking matrix are the same, then the formula

$$C = \frac{12S}{n^2 m(m^2 - 1) - n \sum_{i=1}^{n} T_i}$$

for the concordance coefficient should be used. In this formula

$$T_i = \sum_{j=1}^{m} t_i (t_j^2 - 1)$$

and $t$ is the number of ranks of the j-th type in each of the lines of the ranking matrix. If the calculated $C$ value is small (say, smaller than 0.1) this could be an indication that the experts have been chosen wrongly and/or that their knowledge about the subject is not sufficient. If the calculated $C$ value is very large (say, larger than 0.9), one should conclude that the analysis has been carried out too superficially, too formally, without an in-depth study of the attributes of the object of interest. In both extreme cases the process should be revisited and repeated.

The deviation of the concordance coefficient from zero can be checked using the following Fischer criterion:

$$Z = \frac{1}{2} \ln \left[ \frac{(n-1)C}{1-C} \right]$$

The calculated $Z$ value is compared with the $Z_\alpha$ value determined for a low level $\alpha = 0.01 - 0.05$, and the degrees of freedom $\nu_1$ and $\nu_2$. These are calculated as $\nu_1 = m - 1 - \frac{2}{n}$, $\nu_2 = (n-1)\nu_1$. If $Z < Z_\alpha$, then one should conclude, with the probability $Q \geq 1 - \alpha$, that there is no agreement among the experts. In this case one should conduct new analysis or substitute the experts with new ones, who would agree better. This can be done, for instance, by excluding one of the experts from the team and by evaluating the coefficient $C_1$ for the remaining experts. If $C_1 < C$, this expert should be excluded from the team. If then $C_1 < C$ he/she should remain on the team. Such evaluations should be conducted for each expert. As a result of these evaluations, the degree of agreement among the experts remaining on the team increases.

At the next step one assesses the difference in the significance of different characteristics. This can be done on the basis of the computed variances. The significance of the rank is due to the following three independent components (inputs): 1) the component (input) that is due to the given expert; 2) the component (input) that is due to the given characteristic; and 3) the remainder that can be treated as a normally distributed random variable with zero mean and nonzero variance.

Hence, the total variance of the given characteristic can be represented as a sum of three components:

$$\sum_{j=1}^{m} \left( x_{ij} - \bar{x}_{ij} \right)^2 = \sum_{j=1}^{m} \left( \bar{x}_{ij} - \bar{x}_j \right)^2 + \sum_{j=1}^{m} \left( \bar{x}_j - \bar{x}_j \right)^2 + \sum_{j=1}^{m} \left( x_{ij} - \bar{x}_j + \bar{x}_j \right)^2$$

Here

$$\bar{x}_{ij} = \frac{1}{mn} \sum_{j=1}^{m} x_{ij}$$

is the total mean rank.

is the mean rank for the i-th expert, and

$$\bar{x}_i = \frac{1}{n} \sum_{i=1}^{n} x_{ij}$$

is the mean rank for the i-th expert. The assessment of the differences in the significances of different characteristics are carried out by comparing the variances

$$D_1 = \frac{1}{m-1} \sum_{i=1}^{m} \left( \bar{x}_i - \bar{x}_{ij} \right)^2 = \frac{1}{m-1} \left( n \sum_{j=1}^{m} \bar{x}_i^2 - mn \bar{x}_{ij}^2 \right)$$

between the different characteristics, with the remaining variance being

$$D_r = \frac{1}{(m-1)(n-1)} \sum_{i=1}^{m} \left( x_i - \bar{x}_i + \bar{x}_{ij} \right)^2 = \frac{1}{(m-1)(n-1)} \left( \sum_{j=1}^{m} \bar{x}_i^2 - m \sum_{j=1}^{m} \bar{x}_{ij}^2 + mn \bar{x}_{ij}^2 \right)$$

The significance of the difference in the variances $D_1$ and $D_r$ can be evaluated on the basis of the Fischer criterion:

$$Z = \frac{1}{2} \ln \left( \frac{D_1}{D_r} \right)$$

for the degrees of freedom $\nu_1$ and $\nu_2$, calculated as $\nu_1 = m-1$, and $\nu_2 = (m-1)(n-1)$. If $Z \geq Z_\alpha$, where $Z_\alpha$ is determined for a low enough $\alpha$ value (say, $\alpha = 0.05$), then one concludes that the difference between the variances $D_1$ and $D_r$ is significant. This means that the difference in the roles of different characteristics is substantial, and the influence of the chosen characteristics is significant. If, however, $Z \prec Z_\alpha$, then the difference between the variances $D_1$ and $D_r$ is insignificant, the difference in the roles of different characteristics is small and the influence of the given characteristic is not essential. If this is the case, one should broaden the number of characteristics and start a new Delphi process.

When conducting statistical analyses, one might be willing to assess the probability distributions of different characteristics and the differences in these distributions. The assessment of the significance in the distributions of different characteristics can be substituted by the comparison of the mean values of the corresponding ranks. Let us examine now how the difference in the mean values of the ranks can be assessed.

When the ranks for each criterion form random samples are evaluated, each of these samples is characterized by its mean $\bar{x}_{ij}$ value and variance

$$D_j = \frac{1}{n-1} \sum_{j=1}^{n} \left( x_{ij} - \bar{x}_{ij} \right)^2$$

The significance of the difference in the mean values of different ranks for different characteristics can be evaluated on the basis of the Student distribution tables by using a sequential comparison of the mean ranks for different characteristics:

$$t_{\bar{x}_i - \bar{x}_j} = \frac{\bar{x}_k - \bar{x}_l}{\sqrt{D_{\bar{x}_i - \bar{x}_j}}}, \quad k, l = 1, 2, \dots, m, \quad k \neq l$$

and the variance $D_{\bar{x}_i - \bar{x}_j}$ can be computed as

$$D_{\bar{x}_i - \bar{x}_j} = D_{\bar{x}_k} + D_{\bar{x}_l}$$

and

$$D_{\bar{x}_j} = \frac{D_j}{n}, \quad j = 1, 2, \dots, m$$

The corresponding number of the degrees of freedom is $\nu = 2(n-1)$

If the table for the Student distribution results in the absolute value $|t_{\bar{x}i - \bar{x}_j}|$ that exceeds the value $t\alpha$ determined on the basis of the Student distribution table, then one can conclude, with the probability $Q \geq 1-\alpha$, that the difference in the mean values $\bar{x}_k$ and $\bar{x}_l$ is insignificant and that the characteristics of interest are not different from the standpoint of their significance (influence), and can be put therefore in the same group. The next step is to evaluate the distributions of these characteristics from the factors affecting these characteristics.

Let, e.g., an analyst is interested in assessing the validity

of a possible choice of an anticipation algorithm using a set of

8 12 12 8 $$ \sum_ {j = 1} ^ {8} \sum_ {i = 1} ^ {1 2} x _ {i j} = \sum_ {i = 1} ^ {1 2} \sum_ {j = 1} ^ {8} x _ {i j} = 3 $$

1 1 1 1 387 ij ij j i i j x x

so that the Table 3 data are consistent. The calculated sums of the columns indicate that the characteristic #3, with the sum

$$ \left| \sum x _ {i 3} = 2 1 \right| $$

is the most important one and the characteristic #8, with the sum 8 80 ix = ∑ is the least important. Since in the Table 3 matrix some of the ranks are equal, the formula (8) should be used to evaluate the concordance coefficient. The formula (9) yields:

1 3 4 6 7 8 9 11 6 T T T T T T T T = = = = = = = = ; 2 12 T = ; 5 12 12 T T = = ; The correlation factor computed in accordance with the formula (7), is 2190 S = . The concordance coefficient found on the basis of the formula (8) is

0.370 C =

. In order to

determine how far this coefficient is from zero we calculate

the Z value by the formula (10). It yields:

$$ : Z = 0. 9 3 0. \mathrm {T h e} $$ degrees of freedom are: 1 2 0.930. 8 1 6.833 7 12 Z ν = = −− = ≈

2

(12 1) 6.833

$$ v _ {2} = (1 2 - 1) x 6. 8 3 3 \approx 7 5. \mathrm {F o r} \alpha = 0. 0 5, $$

, with these degrees of freedom, we find: 0.35 Zα = . Since this value is smaller than

0.930 Z =

we conclude, with the probability

$$ Q \geq 0. 9 5 \mathrm {t h a t} $$

there is a nonrandom agreement among the experts that the

selected characteristics reflect well the value of the choice of

interest.

Let us assess now, using the formulas (15) and (16), the differences in the roles of different characteristics on the algorithms of interest and the significance of these roles.

The mean values of the ranks of the different characteristics are: 1 3.08 x = ; 2 4.16 x = ; 3 1.75 x = ; 4 4.25 x = $$ ; \bar {x} _ {5} = 3. 0 0; \bar {x} _ {6} = 4. 3 3; \bar {x} _ {7} = 5. 0 0; \bar {x} _ {8} = 6. 6 7. $$

. The mean values of the ranks, as determined by the experts, are: 1 4.63 x = ;

2 4.75 x = ; 3 4.38 x = ; 4 4.00 x = ; 5 3.25 x = ; 6 3.75 x = ;

7 3.75 x = ; 8 3.88 x = ; 9 3.75 x = ; 10 4.50 x = ; 11 3.88 x = ;

The total (averaged) mean rank is $\bar{x}_j = 4.03$. The formulas (15) and (16) yield: $D_1 = 26.1$; $D_2 = 3.1$. The significance of the difference in the variances and can be checked using the formula (17). It yields: $Z = \frac{1}{2} \ln \left( \frac{26.1}{3.1} \right) = 1.06$.

With $v_1 = 7$ and $v_2 = 75$ for $\alpha = 0.05$, we find: $Z_\alpha = 0.35$.

Since $Z \succ Z_\alpha$ we conclude, with the probability $Q \geq 0.95$ that the difference in the influence of the characteristics of interest is statistically significant, and the selected characteristics are suitable to evaluate the choice under consideration.

Determine now the influence of the role of the selected characteristics on the general evaluation of the choice under consideration. Different groups of the selected characteristics can be addressed based on the assessment of the difference in the mean values of the ranks of different criteria. The standard deviations of the criteria can be found, using the formula (18), as follows: $\sqrt{D_1} = 1.44$; $\sqrt{D_2} = 1.69$; $\sqrt{D_3} = 0.87$; $\sqrt{D_4} = 2.30$; $\sqrt{D_5} = 1.54$; $\sqrt{D_6} = 2.10$; $\sqrt{D_7} = 2.13$; $\sqrt{D_8} = 1.30$.

The highest mean value of $\bar{x}8 = 6.67$ has been given by the experts to the characteristic $x_8$. The standard deviation of this characteristic is $\sqrt{D_8} = 1.30$. Then follows the characteristic $x_7$ with $\bar{x}_7 = 5.00$ and The formulas (20) and (21) yield: $D{x_8-x_7} = \frac{1.3^2 + 2.13^2}{12} = 0.5184$, $\sqrt{D_{x_8-x_7}} = 0.72$. Using the formulas (19) and (22), we have: $t_{x_8-x_7} = \frac{6.67 - 5.00}{0.72} = 2.32$, $\nu = 2(12-1) = 22$. For $\alpha = 0.05$ and $\nu = 22$ the Student table yields: $t_\alpha = 1.72$. Since $t = 2.32$ is greater than $t = 1.72$, we conclude, with the probability $Q \geq 0.95$, that the difference in the mean values of characteristics #8 and #7 is statistically significant, and therefore these characteristics should be placed, based on their roles, in different groups. The other values can be calculated in a similar fashion: $t_{x_8-x_7} = 0.77$, $t_{x_8-x_1} = 1.76$, $t_{x_8-x_4} = 0.09$, $t_{x_8-x_1} = 0.13$, $t_{x_8-x_2} = 0.22$, $t_{x_8-x_3} = 2.45$, $t_{x_8-x_4} = 1.68$. Comparing the calculated $t$ values with the $t_\alpha = t_{0.05} = 1.72$ one can select the following groups of characteristics: 1) $x_6, x_4, x_2, x_1$; 2) $x_5, x_1$; 3) $x_3$. As a result of our calculations, we found that the characteristic belongs to two groups. The comparison of the values $t_{x_8-x_1} = 0.13$ and $t_{x_8-x_2} = 1.68$ shows however, that the characteristic $x_1$ should belong to the same group as the characteristic $x_5$.

The final grouping is as follows: 1) $x_8$; 2) $x_7, x_6, x_4, x_2$; 3) $x_5, x_1$; 4) $x_3$. The characteristic is the most influential and is followed by the characteristics $x_8, x_9, x_1, x_2, x_4, x_6, x_7, x_8$. Hence the most suitable anticipation algorithm should be selected considering the obtained importance (priority) of the algorithm’s characteristics.