Statistical significance
Help - Canadian station weather data and extreme winter events
Canada main | Help main | Return to previous page
Statistical significance is determined for the ratios of extreme weather events during extreme high index days and extreme low index days. The testing for statistical significance was done for all ratios, but the less complete the data record used to figure the ratio, the less meaningful the result of the significance testing. This is explained further below in an important note on interpreting ratios and their statistical significance in this analysis.
The test of significance aims to answer the question of whether any given ratio different from 1 is robust. Can it be said with 95% confidence that the ratio is different from 1 with more event days occurring during the index extreme as observed? Or more simply, is the ratio really the color (red or blue) that is shown?
- A ratio different from 1 means that the type of weather event of interest occurs more frequently during one of the extremes of the index of interest, while a ratio of exactly 1 means that the type of weather event of interest occurs equally during the two extremes of the index of interest.
- However, ratios different from 1 may arise in the data simply because the sample sizes are finite and not because there is any true relationship causing more weather events to occur during one of the index extremes.
- Basically, the fewer weather event days there were from which to obtain a ratio, the greater the ratio must be to be considered significant.
The following procedure was used to determine whether the ratios are significant.
- A test distribution of 7563 days was created to match the number of NDJFM winter days over the 1948-1997 interval. Further matching the actual distribution, extreme high and low index days were defined in this test distribution in the same number as in the real distribution for each of the three indices: 1260 high/low for NAM and PNA, 1209/1210 high/low for CTI.
- From this test distribution a number of "event days" was randomly selected. This is analogous to selecting the weather event days of interest. From the days chosen the ratio of extreme high index days to extreme low index days was calculated, giving one test ratio for that number of event days and that index. That process was repeated to produce a total of 100,000 test ratios.
- For that number of event days a threshold value corresponding to the 97.5 percentile of the test ratios was attained - assuming unique values 97.5% of the test ratios are smaller than this value, and 2.5% are greater.
- If an actual ratio derived from the same number of event days is greater than or equal to the threshold value for that number of event days, the ratio is considered "statistically significant."
- The 97.5 percentile allows for 97.5% confidence the ratio is different from 1 (two-sided test), but it allows 95
- See plots of the significance threshold values versus number of event days (figures open in new windows)
- The curve has an inverse exponential shape {1/exp(x)}, asymptoting toward ratio=1 for very many event days and toward ratio=infinity for very few event days.
- From the definitions used to determine event days, temperature data have around 300 days of each extreme (less if the data record is not complete, and perhaps 200 more or less depending on the skewness of the distribution). Some stations receive measureable precipitation almost every day, so event days with any observed precipitation may be very large. Extreme precipitation days defined as the top 20% of days with measureable precipitation, so the number of those event days may range anywhere from zero to about 800.
- Some threshold values of interest:
- less than 28 event days: no ratio is considered significant
- The significance thresholds are the same for the NAM and PNA days since there are equal numbers of extreme days under those indices. The thresholds are slightly greater for the CTI since by how extreme of that index were defined there are fewer of those days. However, this difference for the CTI is only a few percent and not important except in certain cases with very few event days.
Figures showing event ratios for all stations versus the number of extreme weather days
- Select figure to open in new window from grid below by the weather type and index.
- Data for all stations shown by grey dots. Significance threshold values shown by orange.
- To limit scaling, all ratio values greater than 10 are shown as 10 in figures.
These figures are only intended to give an overall impression of how the station ratios compare to the significance thresholds, not to show where any one station stands.
| Extreme
Low | Extreme
High | Any |
| TMAX | | | |
| TMIN | | | |
| TEMP | | | |
| PRCP | | | |
| SNOW | | | |
| RAIN | | | |
Important consideration when judging ratios and statistical significance testing results
When evaluating the results of statistical significance testing (as well as the ratios themselves) calculated in cases where the data record is missing a great number of days, care must be taken not to draw unwarranted conclusions. A complete data record allows for equal numbers of high and low index days to possibly go into the extreme weather days ratios. However, a data record without data for a great number of days opens the likelihood of the number of high and low index days not being equal thus affecting the expected ratio results.
For example, consider a station with TMIN data from only 3 winters, and by chance those all 3 of those happened to be El Niño winters. That means there could be many high CTI days but no low CTI days, resulting in infinite TMIN CTI ratios like 15:0 or 18:0. Those large ratios may appear meaningful and significant, yet because they were based on data exclusively from high CTI index days it cannot be concluded that extreme TMIN (or anything) occur more frequently on high CTI index days.
A less extreme example would be a station with an approximately half-complete data record where most of the data is from either the earlier or later halves of the 1948-1997 period. Though over the whole 50 years the number of high and low index days are equal, that is not the case for a shorter segment of the period. For instance, there are more high NAM days later in the period than earlier, and half the low CTI days (La Niña) are from 1950-1956. So a SNOW record with many observations from the 1980s and 1990s and few from the 1950s and 1960s may result in ratios that indicate SNOW occurs much more frequently at that station on high NAM days, but that apparent relationship may only be the result of there being more high than low NAM days with data in the record.
On the station pages the popup window showing the weather variable observations by year along with the extreme index days by year are useful for helping determine for individual cases with much missing data whether ratios are strongly influenced by from when the weather data was available.