The purpose of this exercise is to check your understanding of significance level, p-values and power.
If you have a complex test statistic it can very well happen that you don't know its distribution when the null hypothesis is true, nor do you know its distribution for some specific alternative hypothesis. As long as your test statistic is reasonably sound this should not prevent you from using it. With fast computers of today you can easily simulate the null hypothesis distribution as well as the distribution under some specific alternative.
Below is the outcome of such simulations. For illustration I have only used 100 simulations, in real life I would have done perhaps 100 000. The scenario is as follows:
The effect is described by some positive parameter λ. You test H0: λ = 0 against H1: λ > 0.
The test statistic T is such that H0 shall be rejected only for large values of T.
Table: Simulation of test statistic T under H0 (i.e. when there is no effect)
|
0,00
|
0,01
|
0,01
|
0,02
|
0,02
|
0,02
|
0,04
|
0,05
|
0,13
|
0,13
|
|
0,17
|
0,19
|
0,19
|
0,20
|
0,21
|
0,21
|
0,24
|
0,27
|
0,27
|
0,27
|
|
0,29
|
0,30
|
0,30
|
0,31
|
0,38
|
0,41
|
0,42
|
0,44
|
0,44
|
0,47
|
|
0,50
|
0,52
|
0,54
|
0,54
|
0,57
|
0,62
|
0,63
|
0,65
|
0,65
|
0,69
|
|
0,70
|
0,70
|
0,72
|
0,74
|
0,77
|
0,77
|
0,78
|
0,80
|
0,82
|
0,85
|
|
0,86
|
0,88
|
0,89
|
0,91
|
0,91
|
0,94
|
0,96
|
0,96
|
1,05
|
1,06
|
|
1,09
|
1,16
|
1,16
|
1,17
|
1,18
|
1,19
|
1,21
|
1,24
|
1,24
|
1,27
|
|
1,27
|
1,45
|
1,46
|
1,47
|
1,49
|
1,65
|
1,86
|
1,95
|
1,99
|
2,02
|
|
2,04
|
2,07
|
2,13
|
2,20
|
2,27
|
2,35
|
2,54
|
2,54
|
2,56
|
2,62
|
|
2,69
|
2,70
|
2,87
|
3,00
|
3,33
|
3,62
|
3,77
|
4,03
|
4,15
|
5,29
|
Table 2 Simulation of test statistic T for λ=2 (i.e. for some specific nonzero effect)
|
0,08
|
0,14
|
0,21
|
0,29
|
0,34
|
0,47
|
0,52
|
0,54
|
0,93
|
0,93
|
|
1,01
|
1,07
|
1,48
|
1,61
|
1,66
|
1,71
|
1,80
|
2,08
|
2,16
|
2,20
|
|
2,25
|
2,46
|
2,50
|
2,53
|
2,57
|
2,65
|
2,79
|
2,91
|
2,99
|
3,22
|
|
3,36
|
3,37
|
3,49
|
3,50
|
3,65
|
3,70
|
3,85
|
4,02
|
4,03
|
4,09
|
|
4,38
|
4,80
|
4,81
|
5,08
|
5,57
|
5,82
|
6,25
|
6,32
|
6,46
|
6,48
|
|
6,68
|
6,75
|
6,79
|
6,79
|
7,39
|
7,41
|
7,66
|
7,68
|
7,72
|
8,34
|
|
8,72
|
9,30
|
9,57
|
10,25
|
10,54
|
10,71
|
11,33
|
11,66
|
11,80
|
12,65
|
|
13,58
|
13,69
|
13,75
|
13,85
|
14,03
|
14,85
|
15,77
|
16,06
|
16,54
|
16,60
|
|
16,70
|
18,32
|
18,34
|
19,50
|
19,56
|
20,38
|
21,77
|
24,21
|
26,30
|
26,76
|
|
28,76
|
31,85
|
34,80
|
35,65
|
37,36
|
39,99
|
40,34
|
46,92
|
57,24
|
87,89
|
Based on the simulated data above you shall:
a) Make an estimate of the critical value c for a test at 5% significance level, i.e. estimate c such that P (T ≥ c) = 0.05) under H0.
b) Give an estimate of the p-value for T = 2.
c) Estimate the power, i.e. estimate P (T ≥ c) when λ = 2.