Adding Diversity to the Hugo Awards

[daveon]

One of the claims the various puppies have made is they're getting more works into consideration for the Hugo Awards...  I thought this was an interesting claim, especially given the effectiveness of a strong slate, so I thought I'd look at the data.  I can only find details back to 2011, anybody who has links to the earlier nomination data, I'd quite like it.

So 2011 is a clearer pre-puppy year, back in the bad old days when either middle-aged priviliged white men, or a sekrit cabal of Social Justice Warriors ruled the Hugos - it was also a smaller con, being in Australia.

As our pre-puppy, cSWM/SJW dominated reference we had 833 Ballots Cast for Best Novel, with 23 works up for consideration.

Each year, since then, the number of ballots cast has increased - interestingly, the number of books nominated has not, in fact, in each of the following years just 16 works have been nominated... meaning that the ratio of ballots to works has gone from 36:1 in 2011, to 100:1 last year...  strike 1 for the puppies.

I next looked at the ratio of Best Novel nominations to nominations for the Short Story categories - interestingly, by and large they're relatively static - more or less half the people who nominate a book for Best Novel nominate a short.

But here's another thing, in the last Post Puppy year we had the following numbers:

Novelette: 24
Novella: 22
Short Story: 17

Let's look at the state of play post puppy?

Novelette: 17-16-15
Novella: 15-15-16
Short Story: 18-17-15

Apart from Short Story which hasn't changed, the number of individual works has actually fallen too.

One of the original puppy claims was that works were not getting onto the Hugo Ballot, this is certainly true, but the implication of the claim was they were not being nominated - the data doesn't support that.  They were being nominated, they just didn't have enough people doing so.

The drive to get more people to nominate Puppy books has actually reduced the diversity and range of options being nominated, NOT, increased it.  Which rather dismisses the idea that the puppies do not and have no voted the slate as a bloc and read their own things and made their own decisions - if they have been doing that, then actually they've been therefore reading pretty much the same stuff as the cSWM/SJW 'cabal'.

I'd be interested to look at the data going back, but on the face of it, the drive to get more people to nominate and increase numbers has actually had the reverse effect and that strong slates actually make things even worse.

EDIT 1: A puppy supporter has pointed out to me that 2012 was also pre-puppies too - I'd need somebody to check if that was around the time a lot of Book Bloggers were getting upset about cSWM domination of the event.  Either way, it's interesting that despite an increase in the numbers of nominations, the numbers of works being nominated hasn't changed a lot with or without the Puppies.  I would like pre-2011 data so we can see what the normalized ratios were pre Aussiecon.

EDIT 2: The data I'm seeing seems to cut off at about 3% of votes cast - so I don't actually have full nominations here, but the numbers making that cut off seem very very consistent year in and year out.  Having more works spread out doesn't indicate much of a conspiracy.  Just people tend to like the same things.

EDIT 3: It's really hard to pull out the numbers of unique works because they're not always given or not given in a format I can be arsed to pull out.  However, interestingly a couple of things emerge:

1. The ratio of ballots to unique works in Best Novel is constant at about 2 - you can increase the voters, they all seem to behave the same way - the number of works that make the 3% of the votes cast list that is conventionally shown dropped in 2011 and has stayed the same


2. The Short Story categories have changed - if the ratios of votes to unqiue works had stayed constant then they'd have been 300+ unique works in one of the categories last year, instead there were only 200

In short, the more people we have taking part, the easier the numbers suggest it becomes to actually log roll a category with a strong slate.  I wasn't expecting that.

There is also precious little evidence that there has really been any real concerted activity in the past to game the awards - but we knew that.

Comments

Unclear

[(Anonymous)]

There are bunch of things that confuse me in this post, but the 2011 Worldcon was in Reno. It was the 2010 Worldcon that was in Australia.

You can find the voting statistics for 2010 at http://web.archive.org/web/20100917163329/http://aussiecon4.org.au/hugoawards/files/2010HugoVotingReport.pdf. (The Aussiecon 4 web site has been taken down, but it was archived.)

The WSFS Constitution (section 3.11.4) requires a Worldcon to publish nomination statistics "including in each category the vote counts for at least the fifteen highest vote-getters and any other candidate receiving a number of votes equal to at least five percent (5%) of the nomination ballots cast in that category, but not including any candidate receiving fewer than five votes." So the number of works that get listed as runners-up for nominations in the Hugo statistics will vary slightly from year to year, but will also depend on how much work the Hugo administrators want to do in publishing the list.

For example, there were 1,827 nomination ballots submitted for Best Novel this year. If the administrators wanted to publish only the nomination count for the top 15 novels and any other novel receiving at least 5% (that is, at least 91 nomination votes), they could do that. Or they could go deeper down the list, and publish the nomination counts for all novels receiving at least 0.3% of the nomination votes (that is, at least 5 nomination votes). Or they could go somewhere in between.

But, in any event, they have already revealed that 587 different novels were listed on those 1,827 ballots. (See https://chaoshorizon.wordpress.com/2015/04/06/how-many-puppy-votes-breaking-down-the-hugo-math/.) Many of them were probably listed by only one to four people and thus will not be listed in the final statistics in any event.

In summary, I'm not sure that the data you are using supports the conclusion you are trying to draw from it.

--J. Kreitzer

Re: Unclear

[daveon.livejournal.com]

See the latest edits - I have gone back and looked for the unique nominated works in total for each year, not easy because it's not always been tabulated. The numbers for Best Novel seem to remain constant, the Shorts don't.

[despotliz.livejournal.com]

I have data going back to 2009 in a spreadsheet, and some stuff back to 2005 in more scattered form - it's all on the pages for each year linked from the Hugo history page.

When you say number of works nominated, you mean number of works which are in the extended stats, right? Unfortunately all they have to do is list at least the top 5 works or ones which got more than 15%, so all we can really say from these stats is that the nominations may be getting slightly more spread out and slightly fewer items are hitting the 5% threshold. Here's a graph of top 20 nominees from the past 6 years:
http://i.imgur.com/3uys3Lv.png
and it's actually pretty consistent what proportion of the vote the top nominees are getting, even as voter numbers increase. Except for dramatic presentation, which is really weird (I've truncated the scale at 50%, but in BDP you get films nominated by >50% of nominators).

I don't think there's that much historical data on total number of works nominated in a category, although there's a few scattered bits and pieces in the stats in recent years. Looking at novel, in 2015 we have 1827 ballots and 587 works,in 2014 there were 1595 ballots and 648 works, in 2013 there were 1113 ballots and 475 works, and in 2009 639 ballots and 335 works.

[daveon.livejournal.com]

There isn't that much on the total nominations but I've found it for 3 years which was interesting, and I'll pull things out in a later post.

The Novel category stays remarkably consistent in relation to the number of people nominating. That's probably why it's harder to completely game.

The Short Story categories? Yeah, the more people involved the easier it is for a relatively small number of people doing the same thing to own them... that is assuming that the majority are voting according to their interests.

The number of short stories up for consideration is actually going down relative to the number of voters which I wasn't expecting. The Novels are staying more or less the same.

[drplokta]

2013 is also effectively pre-puppy, as their "slate" that year only consisted of one novel by Larry Correia.

[(Anonymous)]

I was wondering about that, but work to increase the number of people nominating had been underway in reaction to the charges of the Hugos being run by a bunch of old men...

I just find it interesting that as near as I can tell, relative to the numbers involved, the range of works was proportionately higher with what I'd describe as the 'core' voters. I suspect we have got more people, but the new people have come with a narrower agenda.

I'll be interested to see the nomination stats for this year and the 'awsom powa' of the 'dread ilk' and the like - I suspect that we'll see the same thing.

[daveon.livejournal.com]

That was me that was, not paying attention.