Agile, Waterfall, Whatever: Make Development Transparent

A while back I made the migration from our print side to the Web team. Once aboard with the so-called "webmonkeys" I found myself involved in a project to use an open source tool to create our new blogs. I'm no programmer, but I'd played around with some blogging/content management tools and had some half-baked opinions, which made me an expert at the time.

Thus I found myself on the dark side of the development process.

Back in 2002, Joel Spolsky wrote an excellent (if somewhat snarky) article about how end users perceive software development.  The gist was that end users don't understand that 90 percent of software development is "underwater," meaning that creating a slick UI--or worse, a slick demo of a UI--is but a tiny part of the overall development process.

Along the way he stops to slag the extreme programming process of having end users along for the development ride and talks about how people really want to hire a good architect to design their kitchen and then leave all the rest to them. (Personally, if someone's going to be building the kitchen of my dreams, they'd better have regular conversations with me. And the architect who assumes I'm not bright enough to learn the differences between Viking and Aga ranges is an architect who needs to find a new client.)

But the real gem is at the very end of the piece: Spolsky insists that developers must manage user expectations, particular about scheduling.  I'd amplify it even more:

Having a transparent schedule is the most
important factor in any development process.

More important than a slick user interface. More important than hitting all the functionality goals. Even more important (to a point--not to stupidity) than delivering bug-free code.

Our development process on the blogs was rapid, seat of the pants, sparsely documented and--from our point of view--a great success. We went from a creaking old hand-coded pseudo blog tool to an open-source, modular, PHP-based tool in just a few months.  Yeah, we slipped a few deadlines as weird behaviours cropped up in the final days. But when we were done, we had something that--we felt--took us from 1996 to 2006 in a heartbeat.

But while we jogged along feeling the sun in our eyes and the breeze in our hair, the actual users were practically in an isolation tank. We were using a project management tool (Basecamp) to keep tabs on our progress internally, but the end users didn't know anything about it. To them, we went from dead silence to a trumpet call of  "Hey! Here's a test system! Try it out and let us know what you think!" and back to silence until the day we launched. And while we expected hosannahs to ring when we went live, we got as good as we gave--silence.  Silence, that is, quickly followed by feature requests, bug reports and all the day-to-day stuff you'd expect for some tool that had been in production for five years, not for our super-slick new blogging engine on which we'd all worked so hard

The problem--at least as I see it now--is that we didn't deliver transparency. A simple task list on the outside wall of a cubicle would have helped. Weekly emails detailing our progress--and anything that was impeding progress--would have been welcomed. We could have used the opportunity to make people aware of how things were going and simultaneously reset expectations when features were inevitably lopped off by the scheduling machete.

Yeah, it might have seemed like make-work for someone on the project (lord knows our project manager had more than enough to keep him busy), but it would have helped our credibility in the long term. (We're working to fix this, with a different project management tool and regular reports to the users. It's taking some time, and other, seemingly more critical projects constantly get in the way. But we will get there.)

A development backlog is a stack (some might argue that it's a pile--or perhaps even a mouldering heap in the worst cases, but that's another issue.) New things get dropped on top. Higher priority pieces get shoved to the middle. But without an end-user window onto that stack, it might as well be a black hole.