Statistics for everyone: ch.34 Meaning of sample size (power) calculation

easier R than SPSS with Rcmdr : Contents

ch.34 Meaning of sample size (power) calculation

When I was asked about ‘sample size calculation’, or ‘power calculation’, during the lecture, I had a strong feeling that the students did not understand the meaning very well. In addition, sometimes the descriptions of sample size calculations in books and materials often lack a clear explanation of their meaning.

In fact, to explain this, we have to deal fundamentally with ‘what is the value of p’, ‘what is a test’, and ‘what is it like to compare two groups’. This is something you should know, but it is often omitted.

It would be nice to take this opportunity to understand the fundamental concepts.

When we say that there are two groups that have a treatment called A and a treatment called B, and we compare the results of it, what we are interested in is not the result of the ‘sample group A’ and the ‘sample group B’. What really matters is the ‘whole group that received the treatment of A’ and the ‘whole group that received the treatment of B’. It is an infinitely large group, a group that spans the past, present, and future. Because it spans South Korea and the United States, and even Japan and the rest of the world, they are all imaginary groups that cannot be measured.

But let’s say we are only aware of the results of ‘sample group A’ and ‘sample group B’ at the moment, respectively, purely randomized.

We can estimate the nature of the entire infinite population from a randomized selected sample group. The mean of the sample group resembles the mean of the entire infinite population, and the standard deviation is likely to be something similar. However, this is not always the case. In some cases it may come out bigger, in some cases it may come out smaller.

Let’s visit https://goo.gl/5V59WO. It looks similar to Excel. Let’s enter 0, 50, and 1 in the yellow cells. This is a kind of simulation that compares two sample groups drawn in increments of 50 from an infinite population with a mean of 0 and a standard deviation of 1. What if we looked at the mean and standard deviation of the two groups? Perhaps both have an average near 0 .

Inside the red square there is the mean and standard deviation of the two randomly sampled groups. That is, the row 5 are the mean, and , the row 6 are the standard deviation. And the two groups are compared: columns B and C, columns E and F, and columns H and I, respectively. You will see different numbers than the picture above. Because this is a experiment that is designed to be different every time. Although the averages are near zero, in some cases they are quite far from zero. The case of column E is pretty large, around 0.15. In fewer cases, but there are cases like that. If you randomly enter a number in cell A1, a new experiment will take place each time.

There will be cases where the average on one side is large and the average on the other side is small. It’s a small probability, but in that case, the two sample groups would show a pretty large difference. So we came up with an indicator that tells us the difference between these two groups, and a long time ago someone named William Sealy Gosset devised this and it is shown in line 4. Later, people called the T.

If the T value is large, the difference is large, and if the T value is small, the difference is small.

This is from https://en.wikipedia.org/wiki/Student%27s_t-test, which subtracts the mean of the two sample populations from the molecule, so you can see that the greater the difference in the mean, the greater the T value. The denominator has a standard deviation, so the larger the standard deviation, the smaller the T value. The number of samples in both groups is 50 in this case, which is also involved in the denominator. The larger the number of samples, the smaller the denominator, so the T value has the effect of being larger.

In any case, each time you experiment over and over again, a new T value is obtained. So if you do 1000 experiments, 1000 T values are obtained, and randomly you get different T values, which you can list from large to small. A large T value means that the difference between the two groups is large. The smallest one has an number of 1 and the largest one becomes 1000, all of which are 1000, so if you express it as %rank, it can be expressed as 0.001 through 1.

This is the p value, with infinite repetition, which is what we did for the experiment, up to 1000 theoretically. If you list them, it will look like the following figure.

Now if you draw the equivalent number in this experiment, %rank on the x-axis and the p value on the y-axis, you can see that it is in a straight line. (imagine that all kind of ranks are straight, and all kind of %rank are straight too)

As you continue to move the cell to the right, you will rarely encounter things where the T value is excessively large and the result is an excessively small p value.

The probability of encountering something less than 0.5 is 0.5, or 50%. The probability of encountering a value less than 0.1 is 0.1, or 10%, and the probability of encountering a value less than 0.05 is 5%. Because the p value is the % value of rank.(the p is from probability)

Occasionally, you’ve probably heard in the news that the top 10% and bottom 5% of the 100,000 examinees are in the news. Just as you can enumerate them in order and understand what their % value means, you can understand this p value as a % value of that rank.

Now, let’s take another look at the basic assumptions. There can also be a large difference by chance in two groups with the same mean and the same standard deviation, that is, two sample groups drawn from two groups with no differences. If you decide to describe 5% of cases where the difference is large, as unusual, strange, or large, then you say that you have set the significance level to 5%. In other words, even if you pulled from two groups with no difference, you’d always say that there was a difference of about 5%. In other words, this is called a ‘type 1 error’. The probability that even if there is no difference in the infinite populations, there is a difference in the randomly sampled two groups.

Even in a sample taken from two populations with no differences, the difference can be unusually large 1 in 20 times(5%), and the degree to which we usually set the criterion. Then it’s a statistical test to look at two sample populations from an unknown population, two populations that are too numerous to measure, and conclude that if there is a bigger difference than that criterion, the original two populations would not have been the same population.

Now, let’s imagine two groups that are clearly different. Enter 0.4 in cell B7 to assume two populations with a difference of 0.4 in the mean. Then the resulting p value is in row 3, in some cases small and in some cases large. It can still come out big by accident, or it can come out small. Instead, as you can see by looking at the graph, a small value coming out comes even more frquently. This does not always mean that the value of p is small .

If you draw a red line roughly around 0.05, almost half of the blue dots are placed above the red line.

If you zoom in only to the part where the y value is less than 0.05 at the bottom, almost 55% of the points will have a higher p value than 0.05. In other words, in this case, despite the obvious difference in the mean of the two populations, the p value is greater than 0.05 in about 55%. This is a ‘2 type error’, i.e. a failure to identify that there is a difference despite the difference. On the other hand, if there is a obvious difference in the mean of the two populations, there is a 45% chance that there is a difference in randomly sampled groups, which is called power.

Now let’s say you sample 60 people for each groups. This will result in a p value less than 0.05 to nearly 60%.

Now we need to sample about 100 people, so that the power reaches almost 0.8.

This means that if you extract two groups of 100 people and do a statistical test with an mean difference of 0.4 in the population and a standard deviation of 1, the value of p will be less than 0.05 about 4 times out of 5(80%).

Even if the differences in the population are clear, we cannot conclude that there is a difference of about 20%. This is the type 2 error. You cannot reduce the type 2 error to 0. Unless you are examining all the populations anyway. For the same reason, you can’t reduce the type 1 error to 0.

So, let’s get back to our actual situation. We don’t know the value of the real population. The two populations may or may not be different. If you know if the two populations are different or not, you don’t need to test them. However, based on the observations so far , when you think there is a difference in the population, you try to do a statistical test to see if that is the case.

When we expect to see some degree of difference in population, we run the experiment expecting a certain degree of success. If the success rate to uncover the difference is around 50%, should we run an experiment or not?

That experiment is money, time, and people. Therefore, in order to run the experiment with the highest success rate possible, that is, to proceed with the high power, it is necessary to plan well in advance.

But in order to increase the power, it makes no sense to hold the difference in the estimated mean value a little higher. This is because only the calculated power goes up, not the actual power.

It’s as if you set the expected distance to your destination at 10 km and the walking speed is estimated to be 4 km/h, and the time it takes is 2 and a half hours, you have to increase the speed to shorten the time, not reduce the expected distance.

The mean difference in the population is not known exactly, because it is the value of the infinite population. However, it is necessary to make a rationale and make predictions so that we can make the most accurate estimates so that we can calculate the actual power.

Next Part is for Uploading and Editing Data

easier R than SPSS with Rcmdr : Contents

=================================================