[Paraphrasing] There are lots of things we don’t know that we don’t know about [the project we were working on]. Agile is how we optimize our process for learning those things. – Ward Cunningham

One of the things that really changed how I thought about building software was learning about the Dunning-Kruger Effect. I didn’t know that’s what I was called, mind you, but I certainly had witnessed it and once it was explained, it made perfect sense. What is the Dunning-Kruger effect?

“people reach erroneous conclusions and make unfortunate choices but their incompetence robs them of the metacognitive ability to realize it.”  – http://en.wikipedia.org/wiki/Dunning-kruger_effect

What does that have to do with Agile? Well, some of the seeds for Agile, at least from Kent Beck and Ward Cunningham’s influence came from the days when those two guys shared a lab and were partners in R&D at Tektronix. I’ve written about this before, but basically, their job was to come up with, every set period of time, some idea that Tektronix should pay a team of guys to develop out into a product or tool. So, KB and WC would come into work every morning and one of them would propose an idea and they would spend the morning trying to hack the idea into working software. If it wasn’t working, they’d go back to working on whatever was the last best idea, and then at the end of the time period, they always had something that worked that they could turn over as an idea. This is where some of the seeds of TDD, iteration, and other ideas that are now staples of Agile came from.

But why would you do that? Why would you essentially start over every morning? Well, as it turns out, humans are really bad at knowing what they don’t know. So why not come up with an idea and just work on that idea until you had something? Let’s explore that. . .

BDUF is for those that don’t understand the Dunning-Kruger Effect

I don’t feel like this is a very controversial topic (that might also be the other end of the K-D Effect) because I think this one has already won out. It certainly has in the ruby community. I’ll go ahead an explain it though. BDUF stands for “Big Design Up Front” and most non-enterprisy, non-corporate software teams know, you can’t design all of your code before you write your code. And the agilists out there will even say that the only way to write truly simple clean code is to write a test first. I hope this isn’t a big shock to anyone. If we examine why this is true, we pretty quickly get to a point where you say “we don’t know enough about the code to design the code without first writing some code”. In other words, we don’t know what we don’t know about the code yet, we have to learn what we don’t know and adapt our application architecture as we learn. I guess if you get a CS degree, you’ll run up against a whole set of folks that don’t believe this, but I’ll go out on a limb and say that most of those folks also don’t write much software. Anyway, suffice to say, we’ve got a whole set of practices, red-green-refactor, continuous integration and deployment, to optimize against doing BDUF (among other things, surely).

Specification Documents are for those that don’t understand the Dunning-Kruger Effect

Another example (I think more important and less widely accepted, and the real reason I wrote this post) of the optimization for all the “things you don’t know that you don’t know” in Agile is that we don’t write spec docs. We don’t start out a project with a giant document that spells out everything we’re going to build. That’s not to say no one knows what we’re building, or has a vision of what it looks like when it’s done, au-con-trair, that’s definite a recipe for disaster (and, I think, most people’s biggest fear that haven’t participated in an Agile project). We spend the time to write all of this down in whatever format we would need to give it to a developer and walk away. Why? Well it’s that damn Dunning-Kruger effect. You *can’t* actually write a good spec, and not only that, those that don’t know this are the ones that precisely think it’s possible. No matter how many lines of code you’ve written, or UI’s you’ve designed, or businesses you’ve started, you will never know enough about what you don’t know to write a document that will cover every edge case, angle, possibility, challenge and decision that has to be made to successfully create quality software.

I hear this lot when I tell people this: “But you can change your spec” or “Good specs allow for learning”. I’m sure you can and I’m sure they do. Change and learning is precisely what agile is all about and not just embracing change, but optimizing for it. You’re *always* going to change that spec. Every project I’ve ever been on, at one point or another, something was too hard or expensive to be worth it and we’ve always changed what we’re going to do. Most of the time, this something was a surprise. It wasn’t what we thought would be hard or even something we thought we needed to do! I’ll even go further to say that some of the more successful projects I’ve been on, not only did we optimize for learning things early, we optimized for being able to change things easily.

The problem with writing a spec and then changing it, however, is that usually this renders part of the spec useless, or worse yet, it renders useless some “real work” you have already done. There’s a million quips and anecdotes people use for this, “Fail fast. . .”,  “If you’re not embarrassed when you launch your project, you waited too long”, etc. The reason people say these things is precisely because they’ve experienced the Dunning-Kruger effect. They’ve written a spec or built software thinking they had their bases covered, that they knew things, but they didn’t end up knowing everything and not all of what they thought they knew ended up being true. And many of these people have come to the conclusion that if they would have done less planning, and got to implementing even faster, they would have learned important information that they didn’t know they didn’t know earlier and saved themselves work. XP is about optimizing for this in a technical setting, “The Lean Startup” is about optimizing for this in a marketing and product development setting.