Statistics Evolved

This article will attack two problems in statistics. The two problems are the selection bias in sampling and the arbitrary nature of common alpha levels.

Most statisticians know that the ideal way to sample is a simple random sample, but what if that simple random sample is arguably not representative of the whole population? The solution to that problem has been to use corrective modeling or some form of stratified sampling. This does not solve anything. Corrective modeling and stratified sampling introduce selection bias into the process which only makes things worse. First I will give an example of this occurring and then I will emphasize why this occurs in theory.

Last week <a href="http://afterecon.com/politics-and-government/why-cuccinelli-win-virginia/">I wrote about the Virginia governors race and chose to use a particular poll which gave results very different from the results other polls giving. The Emerson found that Cuccinelli was losing by about a 2% margin while other polls were calling for a loss of 7-12%. I used the Emerson poll rather than other polls for two key reasons. First, the poll was current. Using old poll or even averages of old and new polls distorts and dilutes the effects of change over time, and in the political world things change quickly so you shouldn't mess with those effects. The second reason, possibly even a more critical reason, was that the Emerson poll was based on a simple random sample by an automated call system while most other polls were based on stratified sampling. The other poll taken in that same time frame called for a 7%+ loss by Cuccinelli. It was shown to be an absolute joke, and the Emerson method validated, when Cuccinelli only lost by about 2.5%. The inaccurate poll was based on a stratified sample, corrective modeling and a personal phone interview data collection method.

The poll which was not done by Emerson was done by <a href="http://www.realclearpolitics.com/epolls/2013/governor/va/virginia_governor_cuccinelli_vs_mcauliffe_vs_sarvis-4111.html">Christopher Newport University. That poll had three layers of bias, but these three layers are accepted commonplace in the statistical research communities both in practice and in academia. Selection bias enters in through the process of stratification. If I ensure that a political poll will be corrected for the black vote and blacks generally vote democrat then the overall effect is that I am ensuring democratic votes, meanwhile creating risk for republicans. The solution to this has been to account for not only blacks, but also whites and hispanics and so on. The problem is that this modelling cannot be carried out correctly because racial correction in general usually throws off an entire other set of demographics such as marriage composition or gender composition. You may use corrective modelling to ensure that race is properly represented, but that hardly makes the sample a good representation.

Even if we correct for everything we can think of this is still a kind of selection bias. We are only correcting for the things we are thinking of, but there are an infinite number of potential characteristics which need to be corrected for. This is fundamentally impossible, except through use of a simple random sample. So next time someone criticizes you for having a sample which is not representative, do not respond by use of corrective modelling. Instead, tell them that you accept the criticism but any sort of cure would be worse than the problem. The real way to improve representation is simply to improve sample size because as N increases the sample approaches the population.

Selection bias occurs through selection of strata and the selection of factors to correct for and, although this one is already widely acknowledged, interview bias occurs when gathering data through a personal interview. This is why the Emerson SRS by robo method is superior to the UNC stratafied corrected method by personal call. Ironically, the Emerson poll was probably much cheaper to conduct as well, not that being cheap is ever a good excuse for conducting biased and poor analysis.

There is an entirely other set of problems in statistics today which has to do with the arbitrary selection of the alpha level. Arbitrary selection leads to inconsistent, inaccurate and biased methodology. Two confused reasons for this which are often given are that no objective standard exists and that this is done because scientists want to be very conservative and cautious with their analysis. In reality, there is an objective standard and the scientific community is well known as one of the most liberal institutions that exist.

The objective standard is the 50% alpha level. 50% is the genuine indifference mark for an objective statistician. If you are 50% certain that a relationship exists, you are objectively not confident. If you are 51% sure that a relationship exists, or if you are even 50.000...1% sure that a relationship exists, you are objectively more confident that the relationship exists than you are sure that the relationship does not exist. If P( A ) > P( A' ) then A should be identified as more likely to be true.

One reason this hasn't been adopted is that statisticians do not have a firm grasp for a sense of urgency. They would rather say let me go do some more research and figure out if I can be more confident and blah blah blah. What if there was a gun to your head? You could do nothing, in which case you figure that you have a 49.9999...% chance of living, or you can try to smack the gun out of the person's hand, in which case you estimate you have a 50.000...1% chance of living. In this hypothetical scenario the options are mutually exclusive and collectively exhaustive. The objectively rational thing to do would be to try and smack the gun away. This kind of common sense application makes sense to the normal person, but not to the academic who is often more concerned with pleasing his peers and maintaining a certain academic process than he is concerned with the implications of his work in the real world. To be fair, there are some academics who are pleasantly able to think like a normal person, but they seem to be undesirably few in my opinion.

I would certainly agree that a larger degree of certainty in a fact is better ceteris paribus, but to be better is a meaningless thing when there is no firm minimum. The academic and professional statistical communities have settled on a bit of a firm minimum, the 5% alpha level, but it is an incorrect minimum.

Lastly, I would point out that simply because something is more likely to be true than not true does not mean people should act as though it was true. This is an interesting but key point. Let's say there is a 51% chance that a stock will increase in value and a 49% chance that it will decrease in value. Furthermore let's say the standard error is sufficiently small so as not to jeopardize this hypothetical scenario. This does not imply that we should invest in the stock. This is related to expected value. Perhaps we expect the stock to go up by $5 per share if it goes up at all and decrease by $10 per share if it decreases at all. We would find expected value this way:

($5)*(.51)+(-$10)(.49) = expected value = $2.55 - $4.9 = -$2.35

Even though we are more certain that the stock will go up than that the stock will go down there is an overall negative expectation. Even the objective expectation doesn't determine what should be done. Perhaps you are personally willing to accept more or less risk for whatever reason.

In conclusion, simple random sampling is always the way to go, impersonal data gathering reduces bias, the objective alpha level is 50% not 5% and the fact that statistics provides evidence for or against a statement does not imply that the statement should be accepted as a basis for human action or thought process.