With a large sample- you are virtually guaranteed of getting significance.

Many students are glued to the idea of significance testing. They want to design a dissertation that will lend itself to being “rigorously tested” using statistical analyses.

Statistical testing is anything but “rigorous.” If you design a study with a large sample – you are virtually guaranteed of getting significance. If you design a study with a small sample, your chances of finding significance are slim indeed. Now that does not sound very rigorous does it?

To make matters worse – the large sample study may show significance but the size of the effect could be very, very small. The small sample study will not show significance, but the size of the effect could be huge. It is the size of the effect that counts.

What if you want to design a study with a small sample? Present it as a qualitative study. In my book of “rigorous” designs, the qualitative study beats the quantitative study every time.

Robert Rodgers, Ph.D.
Consultations