Another conference organizer recently asked the !!Con team what our anonymization process was, and we realized that we didn’t have a good source for them. While we tout our anonymization as a major component of our quest to eliminate implicit bias during the talk selection process, we haven’t written what our anonymization and talk selection process actually looks like in a while.
Julia Evans wrote a great post about our anonymization outcome after the first !!Con in 2014, and Lindsey Kuper has an update from a year later that goes into a lot more detail. We’ve changed some processes since (we no longer use github issues to track the proposals, for example).
One thing I want to point out is that if you’re submitting a talk proposal to !!Con, you shouldn’t worry about how your proposal will be anonymized! Just submit your talk like you want it to be listed, and we’ll take care of the anonymization process.
Here’s the blurb about anonymization from this year’s Call for Talk Proposals! page:
Proposals will be anonymized to avoid bias. Although we ask for your name, email address, and so on in the proposal submission form, only one or two organizers who serve as anonymizers will actually see this information, and they won’t review your proposal. The rest of the organizing team will review your proposal without knowing who you are.
I also think we have a pretty cool way of doing the talk selection, so I’ll cover that as well in this post.
Here’s the secret sauce:
- Each year, before we open the Call for Proposals, we decide who is going to be on the anonymization team, and who is going to be on the talk selection team. Generally the anonymization team is one or two organizers plus potentially one or two outside helpers who we recruit. The talk selection team is the remaining four or five organizers. This year, Alex Clemmer and I are the anonymizers, and the rest of the organizing team is the talk selection team.
- The anonymization team is the only team with access to the google form (and resulting google spreadsheet) that we use to collect talk submissions. Then we open the Call for Proposals!
- It’s tradition that the anonymizer who is supposed to close the google form falls asleep before midnight and forgets to close the form until the next morning. This year it was me and I didn’t close the form until 9am. Whoops!
- The “chief anonymizer” saves a copy of the original spreadsheet and begins going through it to anonymize it. Each talk is given an ID (a random unique integer, so that we don’t give away when the talk was submitted), so that we can track and re-aggregate our data at the end of the review process.
Let’s talk about the anonymization process. Here’s an example talk proposal (by me right now for demonstration, this probably wouldn’t be accepted):
I work at FiniteAutomata Corp, so my talk is about how knowing how finite automata work gives you regex superpowers! People will look at me and say, “wow he is so cool with his regex” and I’m like “heck yes I am good because I know how finite automata work let me teach you”. Here’s a link to me giving a longer version of this talk at FiniteAutomataCon 2011: http://example.com
The anonymizer takes this proposal and does a couple of things:
- Remove/Obscure any names of people or companies. We generally leave in product names unless the talk submitter is the main author of that product, in which case we will remove the product name as well. For example, “FiniteAutomata Corp” in the example above would become “[Large Tech Firm]”, in square brackets (or similar).
- Remove/Obscure any pronouns or other identifiers. For example, my proposal above says “he” and “his”, which gives away my gender. We would replace these pronouns with “[they]” or “[author]”.
- We also try our best to watch / read any associated material and give a quick synopsis of it. For example, the link in the proposal above would become “[Video of the author giving a 30-minute version of this talk at a well-known conference. The talk is well received by the audience.]”
During anonymization, we try not to remove aspects of the speaker’s identity that are essential to the message of the talk. For example, the abstract for Sina Bahram’s 2016 talk about screen readers begins, “I use a computer very differently than most people, because I’m blind.” It wouldn’t have made sense for the anonymizer to remove this fact, because it is Sina’s personal experience that makes the talk proposal especially compelling. We try to use our best judgement here (and probably get it wrong sometimes!)
Once we’ve created the anonymized version of all the talks, we create a copy of the anonymized spreadsheet for each of our reviewers. If I were on the reviewing team, I would get a spreadsheet called “!!Con 2018 – Talk Proposals [Erty]”. This is my personal copy of the anonymized spreadsheet.
The next step is for each reviewer to give each talk a grade. !!Con co-organizer Lindsey Kuper introduced this grading scheme, based on the Identify the Champion process (Specifically Make Champions Explicit):
A – I really like this talk and will advocate for it to get into the conference
B – I like this talk, but not enough to advocate for it
C – I don’t like this talk, but not enough to advocate against it
D – I don’t like this talk and will advocate against it being included in the conference
X – I think I know who submitted this talk, and abstain from giving it a grade
The chief anonymizer then creates a spreadsheet that aggregates these grades into a single place (did you know that you can include cells from one spreadsheet in another using google spreadsheets? It’s pretty nifty!). We look at the best grade and the worst grade for each talk.
Some examples:
A talk that got grades A B B C would have a final grade of AC (A is the best and C is the worst)
A talk that got grades X A A X would have a final grade of AA
A talk that got grades X D C A would have a final grade of AD
If a talk got 0 or 1 grades due to abstain votes, we would handle that as a special case. I’m pretty sure that hasn’t happened in the 5 years we’ve been running !!Con.
We all get on a conference call at this point and start going through the talks! This is really fun! AA talks generally get in immediately. AB talks are usually also admitted as well, if we still have room. AC talks we usually discuss, but they’re generally not admitted, and AD talks are really interesting, since it means someone will advocate for it and someone else will advocate against it! Generally, someone misunderstood the proposal, or two people have differing ideas about what kind of talks should get into !!Con. It’s usually a no, but these talks will sometimes get in (especially if they got AAAD).
More generally, this means that only talks that get at least one A are considered. Your talk needs to convince one of our reviewers to become a champion for it. As Lindsey noted in her blog post Scaling !!Con, last year (2017) we had 86 talks with an A out of 215 total, which we then had to narrow down to just 30!
Remember that all of this is still being done with anonymized talks. The anonymizer(s) will usually sit in on this call and answer any questions about the proposals that the reviewers have (usually around things like the descriptions of videos that were with the proposal). But never speaking up to sway a decision in one way or another. This is difficult! Last year I spoke up because I was really excited about a talk proposal (that ended up not getting in, anyway) but I definitely shouldn’t have done that, since I knew who the speaker was and therefore had implicit bias.
Once we’ve decided which talks we’ll approve for the slots we have available, we take our next favorites (generally about five of them, but it varies) and hold those as backups in case our acceptances aren’t accepted (it happens!).
The anonymizer then goes through and makes sure that we didn’t select the same speaker twice (some people have submitted as many as six talks in a single year!) If we did, we bring some more people off the backup list, then replace them to keep our backup list full. Usually if someone submitted two talks that were accepted, we let them choose which one they’d rather give. Although, if there’s a clear favorite, we might suggest that they choose that one 🙂
At this point we do the dramatic reveal and show the review team who got in! Then we email these people. This is really fun 🙂
One important thing to note: If you didn’t get accepted, then your name is never revealed, not even to our review team. The anonymizer emails out our famous rejection letter after we’ve figured out our lineup. This is not so fun 🙁
That’s it! I hope this is useful information, especially if you’re starting a conference yourself.
We were originally working at 512×512 (scaling up x4 for the final image), but eventually realized that Galactic Castle works at about 150×150, so we scaled down. The resulting pixelation is much more pleasing. Added trees and improved the mountain cross-hatching.
