Creating predictive models with intended, calibrated bias

In most situations where predictive models are applicable, the demand is that the predicted values are as
close as possible to those actually observed (which in theory determines the accuracy of the model).
However, there are many cases where such precision is not the primary objective, and a "pessimistic" or
"optimistic" bias is more convenient and useful, especially if it can be fine-tuned to show up in the upper
or lower ends in the value spectrum of the variable being modeled.

Consider, for example, the case of predicting a patient's stay in a hospital. In some cases, especially in
those where there is enough availability (small or no waiting lines) of beds, a "pessimistic" model, which
eventually slightly overestimates (being conservative) the most serious cases, may be useful to hospital
administration, thereby avoiding risk of ICU overbooking, for example.

On the other side of the same situation, a health care insurer, already having indications that hospitals
are unnecessarily prolonging the patient stay (especially in the case of longer stays) before discharging
them, may wish to work with a more optimistic estimate and start a continuous follow-up with patients
and hospitals, when the hospitalization period takes longer than what was predicted.

Switching to a completely different application, airline companies, depending on the policy they practice
with their customers, may also prefer forecasting models with a more pessimistic or more optimistic bias
of the no-show percentage, to use as input to their overbooking strategy.

Here we will present, in a simplified way, for illustration purposes of the basic concepts involved, the
methodology that we use to treat these cases, based on the Ward Systems’ Chaos Hunter software tool.

I – Creating intentionally biased predictive models based on artificial neural networks (ANNs):
The model presented here aims to predict the percentage of the expected loss for a credit portfolio with
high risk debtors, using 8 independent variables, the dependent variable being the percentage of
expected loss.

The data of the independent variables (columns 2 to 9) Optimization by Genetic Algorithm (GA) only, population size
and the dependent variable (expected loss) to be = 100 and seed = 10. Fitting criterion: maximization of R2
predicted by the trained ANN.

Pág. 1 de 8
 As a structure of the model, we selected all the
possible topologies allowed in Chaos Hunter,
but without emphasizing the use of sigmoid
(logistic) or hyperbolic (tgh) functions.

We performed a training cycle long enough to make R2 exceed 80%, as shown below.

Applying the trained / optimized network to the "out-of-sample" test data, the dispersion plot is relatively
well balanced, exhibiting a slightly pessimistic bias (predicted values higher than observed) in the
intermediate range of expected loss, while behaving slightly optimistically for the cases in the extremes,
which may be quite convenient for such an application (predicting the expected loss of a homogeneous,
high-risk credit portfolio).

Pág. 2 de 8
Then, to produce a clearly optimistic bias for the extreme (low and high) expected loss cases, we changed
the structure of the model, increasing the emphasis on sigmoid and hyperbolic functions.

As before, the training was extended until R2 exceeded 80%.

Pág. 3 de 8
Applying the trained / optimized network to the "out-of-sample" test data, the graph shows a trend
markedly optimistic bias (predicted values lower than those observed) at both ends of expected loss
spectrum, being balanced in the intermediate values, where most of the population lies.

Pág. 4 de 8
II – Creating intended bias in Predictive Models Based on Analytical Functions:
The Chaos Hunter allows optimization by genetic algorithms (GA) or Particle Swarm Optimization (PSO),
not only of models based on neural networks, but also of those that use regression methods based on
conventional analytical mathematical functions, which also find a lot of real-world applications. In the
illustrative example that follows, we will present how to induce intentional and calibrated bias for specific
ranges of values. The model in question aims to predict the evolution of the severity of an inpatient (a
continuous measure, like an inverted "health score"). There are 16 "independent" variables in the model
(columns 2 to 17).

Optimization by GA only, population size = 100 and seed = 1. Functions to be used in the model include
Fitting criterion: maximization of R2 transcendental and trigonometric, besides algebraic,
simple arithmetic and polynomials

Pág. 5 de 8
As in the previous examples, the training / optimization cycle is conducted until R2 exceeds 80%.

The model, composed only of conventional functions, is applied to all data ("in-sample" and "out-of-
sample" for better visibility), showing a clear optimistic bias, especially for the most serious cases. Such a
model should be applied with extreme caution, and a second medical evaluation should be undertaken
before making commitments n treatment approaches.

Then, we change only the pre-selected functions, excluding from the model "basket" the trigonometric
and transcendental functions.

Pág. 6 de 8
The optimization is run again and stopped after R2 exceeds 80%.

The application of the new model to the "out-of-sample" test data shows that it presents mild pessimism
for the most severe cases, and optimism for those with less severity, being more suitable for hospital
administration than the previously biased example (a better fitting for health care insurance).

Pág. 7 de 8
The figure below illustrates the performance of the model for all data ("in-sample" and "out-of-sample"),
showing that the predictive capacity is slightly higher (R2 of 90.4% versus 86.9%), attesting that the GA
optimization has produced a highly generalizing model.

In addition to adjusting the "baskets" of elementary math functions (building blocks of the model), or
ANN architectures, the fine-tuning of the seed and maximum population size parameters are also very
important.

All of this critical fine-tuning is as much art as science, and over the past 15 years of our experience in
developing AI applications for many purposes, we have a lot of these parameter sets that allow us to build
high-performance predicting and classifying models.

References: http://www.chaoshunter.com/

Pág. 8 de 8