What's the value of an experiment or a prototype?

There are all kinds of ways to have impact. A feature can improve user experience; a hardening project can reduce risk of a production outage; refactoring or test coverage can improve velocity or make software easier and safer to maintain. And good engineers care a lot about impact. While it's not the only thing that matters (the "how" is important too), if you start with impact, you'll generally do well.

An engineer's job to put ideas into practice, to make things. But sometimes we're not sure what to make. Or we think we know, but aren't sure it'll work. The best way to figure that out is often running a set of experiments, or maybe building a prototype (an n=1 experiment).

But crucially, an experiment doesn't have value itself. An experiment is successful only if we've learned something. The intent of the test rig or prototype isn't to live on. Indeed, knowing that we plan to throw it away is part of what makes it fast and cheap to build, and it shouldn't have all the trappings of production-quality software, like test coverage and code reviews.

So how do we ensure that value gets delivered? When you work in a team or a company people turn over. It's not just enough to do the experiment, you need to write it up and share your results. To produce a good writeup, you should:

1. Figure out the hypothesis(es) you're testing. Often this is in the form of one or more questions. For prototypes, it might be a boolean, i.e. we can build X that will work. But even then, consider what "done" means. Stating your hypothesis in terms of a metric is often easiest. NB I find the goal/driver/guardrail framework from Thanks Diane's book helpful, Trustworthy Online Controlled Experiments.

2. State your assumptions and method. This is where you usually get the most feedback. Note that this usually isn't a project plan, as your reviewers usually don't care how long it takes or what happens when.

3. Seek feedback from your peers. Publish the doc stating the method to have smart people poke holes in your plan and make sure what you're measuring will actually address the hypothesis. And then when the experiment is done, get it reviewed by someone senior to ensure that your work supports your conclusion. This also spreads knowledge about this work (both that you're doing it, and the results) so the overall organization benefits.

The artifact produced has many benefits. It's useful for you as you discuss follow-on work; it's useful come performance evaluation time. But most importantly, it benefits the organization. Contemporary and future peers can learn from this work.

You'll benefit from taking the time to write it up, the reviewers learn from reading, and it'll live on past your time with the team.

Trick-or-treaters volume at our Menlo Park house this Halloween was basically flat compared to last year. Last year we had 208 trick-or-treaters, this year 211. We remain down quite a bit from our 2012 peak. Maybe this is the new norm?

Everything happened later this year. Our first trick-or-treater didn't show up until 6:30, and peak wasn't until 8:45. That's thirty minutes or more later than prior years. I speculate it's because it was a warm weekend night. Why not stay out a bit later, no school tomorrow. And with the fall-back DST change everyone would be looking foward to a "free" hour of sleep.

As usual, the full story can be seen in the numbers. Check it out!

This year's Halloween tally was 208. I don't know why we have 32% fewer trick-or-treaters than we had last year, which was down 20% from the year before that. Maybe the rain earlier in the day kept people home. Maybe because it was Friday people opted for parties instead of going door to door. I don't know.

Thanks to my friend Stuart for being on this year's data gathering crew. As usual, the full story is in the numbers.

For the past five years I've gathered statistics on how many trick-or-treaters have come by on Halloween. If you want to read about that, check out posts from last year or the year before. This post is about how I track those stats, and how I don't.

Every year I'm tempted to build some fancy system to collect and manage these statistics. Wouldn't it be fun, say, to wire up some Raspberry Pi sensor that automatically counts and tweets running totals? It wouldn't be that hard and sounds like fun.

The problem is making something like that reliable. You'd have to do all the un-fun stuff, like testing and contingency planning. If your baseline is a clipboard, paper, and a ball point pen, your bar for failure is basically "never". Even if I did build something fancy I'd still end up doing backup tallies by hand. At this human scale, the tech ends up being a fun gimmick, not required.

It reminds me of a story from friend [Tony]. Tony and his brother Tom run a giant gaming convention every year, the Evolution Championship Series (Evo for short). It's a multi-day convention in Las Vegas that attracts something like ten thousand participants. They run the whole thing with their two other founders and some friends — I'm sure they have some paid help now, but the four guys are the main ones. It's impressive.

Given that Tony and Tom are strong engineers, I figured this would be a slick high-tech operation. Not so.

Tony said they've tried tech at various points and it wasn't worth it. It's easy to see why that is tempting: they have multiple mobile coordinators that need access to changing, shared information, like brackets and schedules. But what they've tried has let them down. Usually it's not the hard parts that fail, but the basics, like batteries and wireless connectivity. So they still run this off of printouts and voice communications (cell phones/walkie-talkies) and periodic data dumps.

And so, this year I'll be gathering my Halloween stats like I always have: clipboard, pen, and a hand-held tally counter. The data will still be timely and accurate.

For the curious few, check out my Halloween Traffic Spreadsheet.

Two postscripts. First, Please stop spreading that NASA Space Pen story. I'm sure you've heard it: how do you write in zero G? the wasteful Americans commissioned a multi-million dollar space pen project; the scrappy can-do Russians used pencils. Well, this story has been debunked by the good people at Snopes.

And second, I'd like to plug Tony and Tom's "day job", Stonehearth. I think of it as Starcraft meets Minecraft. I am so eager to play it when it lands.

This year we had a sizable number of trick-or-treaters at our house in the Willows neighborhood of Menlo Park. The 303 we saw was down from our high last year, but abou the same as the year before that.

So here are the totals:

The rate at peak was comparable to last year.

         

I don't have an explanation why we are down a bit. The weather was beautiful, indeed a little better than last year since rain started at 8:30 last year. Maybe the forecast rain coming last year got people out earlier who might have missed altogether?

It was outstanding having my friend Amy LaMeyer helping out. She operated the clicker that was new this year, and so kept me company, which was a ton of fun.

As always, the Google Spreadsheet with the graphs and raw data is publicly available here. Check it out!

Update with actuals from Halloween 2012:  It was a banner year.

You may be giving out candy later today. What can you expect? Let's look at some data.  This post summarizes the past three Halloweens.

As you can see we live in a pretty popular neighborhood.  Each year has its own story.

• 2009 - our first year in our new neighborhood. We had no idea that this was such a popular trick-or-treating spot. I ran out of candy at 8:00, turned out the lights, and hid in the back of the house. Shameful.
• 2010 - a fine year.  No complaints.
• 2011 - we moved to a new house just around the corner.  I figured the quieter street would mean fewer kids -- not so! What I didn't appreciate was the attractive power of my next door neighbor's insane decorations. Luckily my wife came back with emergency supplies just in time.

And how busy do things get?  Darn busy.

During the busiest 15 minute period last year I was serving a kid every twenty seconds or so.  When bursting this is close to my max current candy-dispensing throughput.

If you come by my house this year you'll see me again, handing out candy with one hand and scribbling hash marks with the other.   I'll update the data in my public spreadsheet.