UPDATE: I reached out to the researchers for this study with questions and they were kind enough to reply to them. These responses are at the bottom of the page.
This morning amidst the WWDC news I saw an article circulated on Twitter claiming that a new study from the University of Illinois discovered that hurricanes given female names kill more people than hurricanes with male names. The reason given for this was that “People imagining a ‘female’ hurricane were not as willing to seek shelter,” Shavitt said. “The stereotypes that underlie these judgments are subtle and not necessarily hostile toward women – they may involve viewing women as warmer and less aggressive than men.” Studies with surprising and novel results always set off my ‘Turns Out’ alarm so I decided to take a closer look at the study.
I was pleased to see that the authors made their supporting information freely available and include their Excel dataset and a PDF describing their methods. Reading to the bottom of the methods, this caught my eye: “Participants: 109 Amazon Mechanical Turk users participated for cash compensation (ages 20–68 y, 54 females).” Mechanical Turk aren’t specifically required to be US residents but they do have to provide a US billing address and US based payment method. At first the idea of using Mechanical Turks for scientific research seemed like it wouldn’t give you a fair sample but after looking at the literature it seems like with care Mechanical Turk can give an appropriate and representative sample. After reading the methods and looking at the data I had some doubts about the validity of the research. The full journal article wasn’t available for free online so I purchased a copy to study further. The article describes the tests in more detail and shows that there were actually three tests done by students at University of Illinois and three by Mechanical Turkers.
Before we analyse this research further I need to briefly go over the history of hurricane naming. In 1950 hurricanes began to be given names of woman (the first three names were Baker, Easy, and King, all masculine names. But after that they were feminine). Before that hurricanes either had no name or were named after the area that they affected. Starting in 1979 hurricanes were given male names as well. Hurricane names are decided ahead of time and given out alphabetically to major Atlantic hurricanes. Only hurricanes that touched down in the US were included in the study. An implication of this is that hurricanes are named alphabetically. I didn’t examine this angle further but I would be interested to see the damage done by hurricane based on time of year when they strike, rather than by their randomly assigned gender.
The research took two parts, looking at historical data for hurricane fatalities, and conducting a series of laboratory experiments to understand people’s assessment of hurricane risk based on name. I don’t have a strong enough statistical background to talk about the lab experiments and whether the statistics behind them are valid. However from what I could understand and check, they seemed to be reasonable. I’d appreciate feedback from statisticians on this part.
In the historical analysis the researchers excluded hurricane Katrina (2012, 1833 deaths) and Audrey (1957, 416 deaths) because they were large outliers that skewed the model. This left 92 hurricanes that hit the US between 1950 and 2012 (the data for this study was taken in January 2013). The key phrase that the study turns on is:
The analyses showed that the change in hurricane fatalities as a function of MFI (Masculine-feminine index) was marginal for hurricanes lower in normalized damage, indicating no effect of masculinity-femininity of name for less severe storms. For hurricanes higher in normalized damage (they chose 47 of 92), however, this change was substantial, such that hurricanes with feminine names were much deadlier than those with masculine names.
In plain english they split the set of hurricanes into two parts, damaging and non damaging. It’s not clear in the dataset which ones went into which category or why 47 was chosen. This splits the dataset almost in half but not along a clean boundary.
After they split the hurricanes in half they found that
Of the 47 most damaging hurricanes, the female-named hurricanes produced an average of 45 deaths compared to 23 deaths in male-named storms, or almost double the number of fatalities. - Washington Post
What isn’t mentioned is that 19 of the storms included were from before 1979 when all hurricanes were given women’s names. Including hurricanes from before 1979 didn’t make a lot of sense to me as male hurricanes didn’t even exist before then.
I then summarised their data (they used more advanced statistics than this) and came up with the following charts:
|Hurricane gender||Average # of deaths||Count of hurricanes|
At first glance their shocking result seems to be borne out by the data, namely that female named hurricanes are more deadly than male hurricanes. However lets repeat the analysis, but this time only use hurricanes from 1979 onwards. I’ve shown here
|Hurricane gender||Average # of deaths||Count of hurricanes|
Now the two are much closer together. I suspect there are other factors that would explain the death toll more convincingly than gender, for example population density of the path that the hurricane took. If we summarise the hurricanes by grade of severity then we see that there no longer seems to be any major difference in deaths from gendered hurricanes. The chart below is actually a little misleading because it makes it look like male hurricanes are more deadly. Hurricane Sandy (2012) was the most deadly hurricane from 1979-2012 but was only a grade 2 and was averaged with other less dangerous hurricanes. Nevertheless I think it is a reasonable comparison.
|Grade||Average # of deaths||# Hurricanes||Average # of deaths||# Hurricanes|
While I was reading and researching this I kept thinking that I must have something wrong. My results seem to contradict their historical analysis that says that female hurricanes are more dangerous than male hurricanes. I encourage you to reach out if I’ve missed something obvious here or I’ve analysed the results incorrectly.
The wider issue here is that I was able to do this research in a few hours with just a copy of Numbers and some critical thinking. I didn’t see any news articles question the results of the study or check this themselves, instead just reprinting the press release.
In case it needs to be said, I’m not trying to minimise all of the other aspects of society where there is a strong gender bias but I’m a little sceptical about this study.
UPDATE: Responses from the researchers below.
Q: It wasn’t clear why it was valid to include hurricanes from 1950-1978 in your study as all of them had female names. Can you explain why you did this?
A: Our main analysis focused on Masc-Fem Index, a continuous predictor, not on counts of male and female hurricanes. Importantly, the modeling controls for other variables relevant to the storm’s severity, something that a tally does not do. The modeling shows an interaction between normalized damage and the femininity of the name. This is explained in the paper and here: http://publish.illinois.edu/shavitt/files/2013/07/PNAS-Reply.pdf Note that we have not gone back to check your tally because this is not the appropriate analysis to begin with.
Q: Were other factors like population density of areas affected by hurricanes more predictive than the gender of the hurricane?
A: Normalized Damage partly reflects population density, among other factors, in that densely populated areas have more infrastructure — roads, bridges, buildings— that can be damaged. Thus, in the paper we indicated that total deaths had the strongest bivariate association with normalized damage. Specific population densities in affected locations were unavailable from NOAA. However, we later added the U.S. population at the time of the hurricane to model 3, and that didn’t change any conclusions we drew from the previous analysis. No significant parameter changes were observed. The same was true for years elapsed since the hurricanes, which was reported in the paper.
Q: How did you choose the 47 most dangerous hurricanes and which ones were they? Looking at the data, there didn’t seem to be a clear cutoff point around this number.
A:For illustrative purposes, in response to others’ questions, we median-split the dataset by normalized damage, and those 47 hurricanes you referenced were at or above the median of $1.65B in normalized damage. Also for illustrative purposes we split the post-1979 data. See the description at the above link. Note that we did not publish these analyses, they came later in response to questions that specifically asked for tallies of male and female-named storms. However, as noted above, there are several shortcomings in simply tallying storms. Even using the cutpoints of Category 1-5, as you did in one simple tally, would not control for how damaging the storms actually were, nor adequately consider substantial differences within the categories in storm severity. This point was made in the paper (see Table S1 and relevant discussion in the Methods section). Normalized damage and minimum pressure are more predictive of deaths than storm category.