PredVariance for NN

Hi,

We're building a model using the NN algorithm, and had a question about the how the PredVariance value is computed. Our testing data set has ~28K cases, but for some reason when we run our prediction query there are only 10 unique PredVariance values generated. Why doesn't each case with a distinct predicted value have its own PredVariance value?

For e.g. here are 3 different PredValues that all have the same PredVariance (229985900) and each has a TrueVal of 0:

15307.6681537296
17759.1791905724
1843.85682442577

If you need more specific info please let me know.

Thanks.

Hello

PredictVariance outputs the error variance for the subnetwork used in executing the prediction. This is detected during training for each subnetwork and, for all predictions executed on that respective subnetwork, PredictVariance will return the same value (same applies to PredictStdev).

If your target variable is a continuous one, then there are at most 2 subnetworks built for the variable (one for the value, one for the probability of Missing state, assuming your training data contains missing values).

If your target variable is discrete or discretized, then one subnetwork is trained to predict the probability of each individual state (including Missing).

Assuming that a prediction returns TargetValue1, the associated PredictVariance should return always the same value (the error variance of the TargetValue1 subnetwork)

You mention that there are 10 unique PredVariance values generated -- it seems that your target variable is discrete with at least 10 distinct states, is this correct?

thanks

|||

Thanks Bogdan,

Our target variable is actually continuous. That's what initially prompted the question, as we expected a PredVariance to be computed for all distinct target variable values (of which there are as many as there are cases).

Does that clarify the question?

Thanks.

|||

The error variance is not computed for each distinct target variable value, but for each subnetwork.

Here are the steps for one continuous target:

- partition the training set in two blocks (training and holdout -- the HOLDOUT_PERCENTAGE and SAMPLE_SIZE parameters control the size of the partitions)

- Iteratively train one subnetwork based on the training partition, and estimate the error (at each step) based on the holdout partition

- at the end -- compute the error variance of the trained subnetwork over all the cases the holdout partition

Therefore, there is a single variance value for the whole subnetwork that predicts one continuous target.

At prediction time, Predict(Target) runs the subnetwork for the target and returns the result, PredictVariance simply returns the (pre-computed) variance for the respective subnetwork (mapped from the normalized training space to the original input space). Therefore, the value returned by PredictVariance should always be the same.

You mentioned that there are 10 distinct variance values being returned?

|||i'll read your feedback more thoroughly this evening, but I wanted to answer your question...yes, there are 10 distinct PredVariance values.|||

Hi,

It's now clear, based on your response, why we have 10 distinct PredVariance values. We were using 10-fold cross-validation and each fold has a corresponding PredVariance.

Going to follow-up with one additional post/question before we close it out....stay tuned.

Thanks,

mike

Prediction Accuracy

Hi ,

I am a novice Data Mining Programmer.

I am using Time series algorithm for forecasting.

We are Quite concerned about the accuracy of Prediction output.

For Example Our Data is like this

StudIdDatePerf

00101/01/200190

00102/01/200189

00103/01/200187

00201/01/200159

00202/01/200170

00303/01/200147

If I write my Prediction Query to predict for 100 th time step.Its giving me out put like

DatePerf

03/01/201547.000000115

We are not sure about the accuracy of the values. Is it possible to use trend information as input to my model and make my prediction based on that.

I don’t know how to do that? Can anyone help?

Thanks,

Karthik.

The time series algorithm in SQL Server 2005 - ARTxp is designed for near term prediction accuracy, not far term - e.g. 100 steps. You can get details on the research behind the algorithm at http://research.microsoft.com/~dmax/publications/dmart-final.pdf

PredictCaseLikelihood

I'm working with the cluster analysis algorithm (EM) in SQL 2005. I have tried to find documentation on the function PredictCaseLikelihood without luck. Is there any reference on how this function is defined?

Here's an excerpt from my book Data Mining with SQL Server 2005

PredictCaseLikelihood

PredictCaseLikelihood returns a measure from 0 to 1 that indicates how likely an input case is to exist considering the model learned by the algorithm.This measure is very good for use in anomaly detection as it quickly and easily tells you if new data is similar to any data seen before.This function operates in two modes, normalized and nonnormalized.

In the nonnormalized mode, the value of the measure is the raw probability of the case, that is, the product of the probabilities of each of the attributes in the case.For instance, if the probability of Home Ownership = ‘Yes’ is 40% and the probability of Occupation = ‘Craftsmen’ is 10% then the probability of the case is 40% x 10% = 4%.

Nonnormalized likelihoods can be useful, but due to the nature of the probabilities, as you increase the number of attributes in a case the probability of the case becomes increasingly small.Additionally, as a user, you can not understand if a 4% probability for a certain combination of attributes is a good thing or a bad thing.The normalized likelihood divides the probability of the case as provided by the model by the probability computed without the model using raw statistics.This provides a “lift” number that is normalized between 0 and 1 using the formula (lift)/(lift + 1).This is interpreted that cases with likelihood values greater than 0.5 have positive lift and are more likely than random to occur and that values less than 0.5 have negative life and are less likely than random to occur.

For continuous attributes, the probability distribution is used for this computation.

This query returns the normalized case likelihood for each case in the input set.

SELECT t.id, PredictCaseLikelihood()

FROM CustomerClusters

NATURAL PREDICTION JOIN <Input Set> AS t

This query returns the nonnormalized case likelihood for each case in the input set.

SELECT t.id, CaseLikelihood(NONNORMALIZED)

FROM CustomerClusters

NATURAL PREDICTION JOIN <Input Set> AS t