I'm a big fan of data. Showing some empirical justification for a claim is generally a good thing, even in blogs and especially when the data is available. I'm not talking about phD level statistics, but some basic facts based on data to support claims would be appreciated. For my part, I like to do original analysis for this blog as often as I can. Sure, I like to share my opinions and qualitative thoughts as much as the next guy, but when it comes to empirical topics, I feel data should be a requirement.

So it was with much interest that I read all these blogs and articles about how airlines were canceling so many flights during this winter's snowstorms and at much higher rates than in the past. Some authors even took it the next step and cited a cause for this phenomenon: The DOT's tarmac rule, which admittedly doesn't take effect until April, but still was apparently the cause of this conservative operational approach.

Here's a sample:

It is noteworthy that none of these articles quoted anyone from an airline that would suggest airlines have actually changed their approach. They simply cited FlightStats facts about the sheer volume of canceled flights and assumed it was meaningful.

Well it turns out, when you look at real data, those claims are unsubstantiated at least through the end of 2009. In past winter storms of similar magnitude, roughly the same percentage of flights were canceled before, during, and after the storms.

Before I give numbers, I'll caveat this by pointing out the weaknesses of my analysis:

1) Not all storms are created equal. I found 2 other storms of similar magnitude based on duration and snowfall. But still, there are differences between any two storms (including time of day and day of week they strike).

2) Flight volumes and schedules were different. I'll use only percentages to show the magnitude of cancellations and delays, but I recognize this could have impact..

3) Airport operations are different today than in the past. This surely impacts operational approach.

4) I don't include WHEN flights were canceled, only IF. This is an additional piece of analysis for which data is trickier. I realize some of the experts quoted in these articles were basing their premise on this difference alone, but the tone of the article and conclusions drawn fail to make this difference clear. Alas, if I can come up with data to do this analysis additionally, I'll give it a go.

Despite these (and other) weaknesses, I think the story still holds. At the very least, it should give one pause before writing another story based on non-existent data.

1) January 7-8 1996, 20 in. @ IAD, 23 in. @ BWI, 20in. @ JFK

2) February 16-17 2003, 19in. @ IAD, 25in. @ BWI, 22in. @ JFK

3) December 19-20 2009, 16 in. @ IAD, 21in. @ BWI, 16in. @ JFK

(Source: Weather Underground)

IAD, DCA, BWI, JFK, LGA, EWR

(I didn't do the recent Feb 2010 storm because the DOT/BTS won't release the detailed data on flights until early April. FlightStats has some data to help this, but not at the granularity that I needed. I figured I'd just start with 2009 and I'll fill in 2010 when it becomes available, even if it proves me wrong!)

Over the course of 4 days, (the day before the storm, the 2 days of primary impact, and the day after the storm), here are the cancellation and delay statistics:

Overall cancelations

1996: 67.9% of flights

2003: 51.3% of flights

2009: 29.3% of flights

Overall flights that operated but were delayed 15 or more minutes:

1996: 15.1%

2003: 21.3%

2009: 31.1%

Overall flights that operated but were delayed 3 hours or more:

1996: 1.0%

2003: 1.3%

2009: 2.8%

Number of flights with tarmac delays of 3 hours or more:

1996: 4

2003: 0

2009: 1

While this data is the combination of stats from 6 airports, similar trends hold for each airport individually.

So what does this tell us? Namely the cancelation rates for the day before the storm were not higher in 2009. Cancelation rates the days of the storm were also not higher, as airlines in the past have also canceled every single flight from affected airports. Also, delays and long delays on flights that did go were higher in 2009 -- signaling that airlines were not optimizing to avoid long delays and would send planes if they thought they could complete the flight.

The one thing that is noticeably different: Cancelations for the day AFTER the storm were lower in 2009. Meaning airlines were able to get their schedules back to normal much faster than in the past. Their irregular ops programs are more sophisticated today and the way they shuffle crew and think longer-term about where their planes are allows them to get back on their feet faster.

Finally, I wanted to note that long tarmac delays were NOT an epidemic in these winter storms, even in the past. Sure, there were 4 such instances in the great blizzard of 1996, but none in 2003, and only 1 this time around. I think this challenges the notion that airlines have changed their approach to eliminate these -- they weren't a big problem to begin with. Yes, Jetblue had the disastrous Valentine's Day storm of 2007 -- but that was a JetBlue problem uniquely. Aside from that, major tarmac delays are not often the result of big winter storms, but rather thunderstorms and other things that are harder to predict and plan for. As such, it's hard to conclude that airlines made major changes to how they handle snow in response to a DOT rule that hasn't taken effect and history would show requires little, if any, changes from how they've handled similar storms in the past.

This is by no means a statistical or scientific study. But it is empirical data that suggests airlines are NOT canceling more flights today than in the past. What they might be doing is canceling the flights earlier (the day before, even), allowing passengers to stay at home rather than wait at the airport. That might have created enough media buzz to lead to these biased conclusions about airline operations.

Perhaps the February 2010 storms will prove this analysis wrong, but in the meantime, this is the conclusion that the data suggests. Yes, there are holes in it, but it's a start.