Not your Type

In any field that uses quantitative analysis to make decisions you’ll hear about Type 1 and Type 2 errors.

  • Type 1 errors are false positives. We thought there were weapons of mass destruction but in reality there weren’t any.
  • Type 2 errors are false negatives. We thought there was no Russian meddling in the election but there actually was.

I’m using these exagerrated examples to explain the concept. The substance of Type 1 and Type 2 errors–you make a wrong call, or you miss out on an opportunity–is something I think we can learn a lot from in the industry.

You end up with a Type 1 or Type 2 error when you use data that wasn’t good enough to make your judgments, or because you asked the wrong questions to begin with.

I run into this Catch-22 all the time, especially on teams where there isn’t a thriving critique culture. A designer presents work, the team doesn’t feel great about it, but people don’t know how to articulate why, or don’t feel like their input would be valuable. They punt to testing the design at some hypothetical point in the future. Quant or qual, it doesn’t matter. Results will probably be inconclusive and late. It’s hard to test things that are said in a design critique.

Testing single features can easily get into Type 1 and Type 2 territory. Your signup flow looks great and is usable, but it doesn’t give you the customer results you wanted. Type 1 error. You were asking a question about customer conversion/retention, or did not have the full picture.

Or you test a single feature by measuring how much usage it gets. It doesn’t end up being very popular, so the team calls it unsuccessful. But if the feature wasn’t findable, or you had a poor understanding of the user need for the feature, you might be in a Type 2 error situation. The feature could be quite valuable.

Where the analogy really gets spicy is when we apply it to how we hire, develop, or put designers in positions of leadership.

You hire a person who talks the talk but didn’t walk the walk. Type 1 error–they were a bad fit for reasons that you didn’t or couldn’t measure before extending an offer. I wonder if this one is tempting to fall into when hiring someone famous or from an “It” company.

Or, you pass over a person for a position or promotion, and the person leaves your company, and goes somewhere else to thrive in the role. Type 2 error. This trap is easy to fall into when job descriptions and everyday responsibilities don’t match—which is often. Does the designer really need to know how to code to be successful? Why do we put people in management positions based off of how productive they were as an IC?

This is where the analogy gets serious–when we look at the low representation of marginalized folks in the industry and even lower numbers in positions of leadership.

Type 1: You hire a person that seemed like they’d do a fine job. It becomes apparent reasonably early on that it’s not going to work out, but you let them continue. The white guy, all the time.

Type 2: You pass over a woman of color for a leadership position because [most reasons]. Except she would do a fine job. You just never let her.

Whether it’s in our day-to-day work, or how we build teams, or how we face up to systemic problems in our industry, it makes a difference when we ask ourselves:

“Are these the right questions?”
“Are these questions capable of giving the right answers?”

It’s always nice when I get to put that fancy psych background to use 😄.