My Next Job


I left my last job a few weeks back and it's high time to look for a new one. If you're working on something interesting and think I could help, let me know!

It's nice to not have a day job while looking for another. I was lucky enough to do this once before in 2012 which turned out great. I learned then that time and flexibility lets you talk to lots of friends and learn about a breadth of projects. I found a fun project in a new domain (online education), something I doubt I'd have found the normal way.

Maybe I'll get lucky again.

Enough small talk, what am I looking for?

I'm looking for some flavor of line manager. I'm a good senior manager and code-every-day engineer; but I'm exceptional leading a team and running a project. That's what line managers do: lead engineers, not other managers or departments or matrix-anything. Also, if you're some kind of executive then coding is an indulgence, and I'd rather it just be part of my job. Mostly I'm talking to small companies, say 10-100 people (fun-size).

I want to build on my experience. I know infrastructure and cloud, SaaS and enterprise, and online education. I'm probably not the best person for your storage, security, gaming, e-commerce, or cryptocurrency company. I want to stay working on Internet technology. I like the (micro)services model. For my own projects I choose Python, JavaScript (frontend and backend), and Java. I know web operations, especially the Amazon stack.

Location is important: I don't want to do a daily Menlo Park to San Francisco round-trip. I'd like to work with friends if possible. And I want to do something worthwhile.

You can always get to my resume from the header here, or via this short link. I'm open to a bunch of things, just no kick boxing. Let's have coffee/drink or take a walk.

Lessons from Three Years in AWS

AWS Logo

I've spent the last three years building and operating web sites with Amazon Web Services and here are a few lessons I've learned. But I first have to come clean that I'm a fan of AWS with only casual experience with other IAAS/PAAS platforms.

S3 Is Amazing. They made the right engineering choices and compromises: cheap, practically infinitely scalable, fast enough, with good availability. $0.03/GB/mo covers up for a lot of sins. Knowing it's there changes how you build systems.

IAM Machine Roles From The Start. IAM with Instance Metadata is a powerful way to manage secrets and rights. Trouble is you can't add to existing machines. Provision with machine roles in big categories (e.g. app servers, utility machines, databases) at the start, even if just placeholders.

Availability Zoness Are Only Mostly Decoupled. After the 2011 us-east-1 outage we were reassured that a coordinated outage wouldn't happen again, but it happened again just last month.

They Will Lock You In And You'll Like It. They secondary services work well, are cheap, and are handy. I'm speaking of SQS, SES, Glacier, even Elastic Transcoder. Who wants to run a durable queue again?

CloudFormation No. It's tough to get right. My objection isn't programming in YAML, I don't mind writing Ansible plays, it's the complexity/structure of CloudFormation that is impenetrable. Plus even if you get it working once, you'd never run it again on something that is running.

Boto Yes. Powerful and expressive. Don't script the CLI, use Boto. Easy as pie.

Qualify Machines Before Use. Some VMs have lousy networking, presumably due to a chatty same-host or same-rack neighbor. Test for loss and latency to other hosts you own and on EBS. (I've used home-grown scripts, don't know of a standard open-source widget, someone should write one).

VPC Yes. If you have machines talking to each other (i.e. not a lone machine doing something lonely) then put them in a VPC. It's not hard.

NAT No. You think that'll improve security, but it will just introduce SPOFS and capacity chokepoints. Give your machines publicly routable IP's and use security groups.

Network ACLs Are A Pain. Try to get as far as you can with just security groups.

You'll Peer VPC's Someday. Choose non-overlapping subnet IP ranges at the start. It's hard to change later.

Spot Instances Are Tricky. They're only For a very specific use case that likely isn't yours. Setting up a test network? You can spend the money you save by using spot on swear jar fees.

Pick a Management Toolset. Ansible, Chef, all those things aren't all that different when it comes down to it. Just don't dither back and forth. There's a little bit of extra Chef love w/ AWS but not enough to tip the scales in your decision I'd reckon.

Tech Support Is Terrible. My last little startup didn't get much out of the business level tech support we bought. We needed it so we could call in to get help when we needed it, and we used that for escalating some problems. It was nice to have a number to call when I urgently need to up a system limit, say. But debugging something real, like a networking problem? Pretty rough.

...Unless You Are Big. Stanford, on the other hand, had a named rep who was responsive and helpful. I guess she was sales, but I used her freely on support issues and she worked the backchannels for us. Presumably this is what any big/important customer would get, that's just not you, sorry.

The Real Power Is On Demand. I'm reaffirming cloud koolaid here. Running this way lets you build and run systems differently, much better. I've relied on the cloud this to bring up emergency capacity. I've used it to convert a class of machines on the fly to the double-price double-RAM tier when hitting a surprising capacity crunch. There are a whole class of problems that get much easier when you can have 2x the machines for just a little while. When someone comes to you with that cost/benefit spreadsheet arguing why you should self-host, that's when you need your file of "the cloud saved my bacon" stories at the ready.

Don't Say No By Email


When I have to tell someone no, I pick up the phone. I hate talking on the phone, but I do it anyway.

When you're answering no to someone, you're disappointing them even if just a little bit. So you owe it to them to talk instead of sending an email. It's the polite thing to do.

But there are two other reasons, selfish reasons, for making the call. First, you get immediate feedback on how they took the news. If they're upset then you can do damage control straight away. And at least you know! And second, delivering bad news directly and respectfully is an important skill to develop. We can all use the practice. And it's never as bad as I think it will be.

Fewer Trick or Treaters This Year

This year's Halloween tally was 208. I don't know why we have 32% fewer trick-or-treaters than we had last year, which was down 20% from the year before that. Maybe the rain earlier in the day kept people home. Maybe because it was Friday people opted for parties instead of going door to door. I don't know.

Thanks to my friend Stuart for being on this year's data gathering crew. As usual, the full story is in the numbers.

In Praise of the Hand Tally

Tally Marks

For the past five years I've gathered statistics on how many trick-or-treaters have come by on Halloween. If you want to read about that, check out posts from last year or the year before. This post is about how I track those stats, and how I don't.

Every year I'm tempted to build some fancy system to collect and manage these statistics. Wouldn't it be fun, say, to wire up some Raspberry Pi sensor that automatically counts and tweets running totals? It wouldn't be that hard and sounds like fun.

The problem is making something like that reliable. You'd have to do all the un-fun stuff, like testing and contingency planning. If your baseline is a clipboard, paper, and a ball point pen, your bar for failure is basically "never". Even if I did build something fancy I'd still end up doing backup tallies by hand. At this human scale, the tech ends up being a fun gimmick, not required.

It reminds me of a story from friend [Tony]. Tony and his brother Tom run a giant gaming convention every year, the Evolution Championship Series (Evo for short). It's a multi-day convention in Las Vegas that attracts something like ten thousand participants. They run the whole thing with their two other founders and some friends — I'm sure they have some paid help now, but the four guys are the main ones. It's impressive.

Given that Tony and Tom are strong engineers, I figured this would be a slick high-tech operation. Not so.

Tony said they've tried tech at various points and it wasn't worth it. It's easy to see why that is tempting: they have multiple mobile coordinators that need access to changing, shared information, like brackets and schedules. But what they've tried has let them down. Usually it's not the hard parts that fail, but the basics, like batteries and wireless connectivity. So they still run this off of printouts and voice communications (cell phones/walkie-talkies) and periodic data dumps.

And so, this year I'll be gathering my Halloween stats like I always have: clipboard, pen, and a hand-held tally counter. The data will still be timely and accurate.

For the curious few, check out my Halloween Traffic Spreadsheet.

Two postscripts. First, Please stop spreading that NASA Space Pen story. I'm sure you've heard it: how do you write in zero G? the wasteful Americans commissioned a multi-million dollar space pen project; the scrappy can-do Russians used pencils. Well, this story has been debunked by the good people at Snopes.

And second, I'd like to plug Tony and Tom's "day job", Stonehearth. I think of it as Starcraft meets Minecraft. I am so eager to play it when it lands.

Leaving Stanford Online Ed


Friday was my last day building Stanford's open-source online education platform. What started as a lark turned out to be one of the most fun and rewarding times in my career. So I'll use this opportunity to reflect back on the project and what we accomplished.

First a Little History

    "MOOCs are just information technology happening to higher education." -- George Siemens

I joined Stanford in 2012, coming off of my self-imposed sabbatical. My goal then had been to get hands-on again, to sharpen the technical tools. I did a few online classes and hobby projects, but my friend Jane suggested I could do some good and have more fun working in a group back on the Stanford campus (I'm an alum, after all). So I joined her and a small group of engineers in a conference room in the fourth floor of Gates.

Recall what was happening in education in 2012. Stanford's first three big Massive Open Online Courses (MOOCs) had just made big splashes. Later that year the New York Times would famously declare 2012 to be the Year of the MOOC. What could consumer-grade web tech could do for higher education? Or even do to higher ed? The low cost and ease of cloud computing removed many of the barriers to trying new things out. You approach experimentation completely differently when things get 100x or 1000x cheaper. Profs were literally getting out their credit cards and buying Wordpress blogs or throw up virtual machines for automated grading. Wild stuff.

So this was the environment when I joined Stanford. That first summer we built Class2Go to host free online courses. We went from empty buffers to a live Python/Django site for hundreds of thousands of enrolled students in eleven weeks. Man, that was a fun ride.

We built Class2Go for a few reasons. First, we feel strongly that Stanford needs to control its destiny. There is no "file format" for online ed content, so every course development online is a bet on a platform. And we didn't want to be beholden to one platform vendor (still don't). In 2012 the technology that had powered the early MOOCs were becoming "platforms" and moving off-campus. While we were happy with the success of Udacity, Coursera, and (a year later later) NovoEd, we were also a bit wary. They were going to have to repay their investors at some point. The last thing we wanted was for online education to look like textbook publishing or academic journals. We know what happened there.

Second, we wanted to have a broad impact. We were lucky enough to have an engineering team at Stanford, but it makes no sense for every school to develop their own platforms. This kind of project is tailor made for open source development, and I've advocated strongly for that since the beginning. Not only would this mean many others could benefit form the software, Stanford would benefit from many more developers and see many more use cases (and we have).

And the third reason is the need to modify the platform. There are many reasons why you need to get hands dirty in the code. It could be just developing a point feature, like tracking changes in peer evaluation. Or it could enable a whole new use case, like authenticating on-campus students. Our teachers and researchers want to do interesting work online, not just put up courses. They have great ideas about using the things that make MOOCs unique (different cultures, scale, data-gathering, etc.) as powerful tools, not obstacles to overcome. (To hear more about interesting online ed projects, follow the Stanford VPOL Signal Blog).

Scaling Up With EdX

Come early 2013 we started seeing some interest from others to use Class2Go outside Stanford. While exciting, we were concerned about the quality of the code. There were giant missing features that were going to be hard to write (e.g. peer evaluation). And the code was quick and dirty, with nearly zero tests and other things a "real project" needs. We could have backfilled all of that, but it sounded like a lot of work.

It's around that time that we started talking with edX. Our academic ties to MIT, Harvard, and other edX consortium members are strong. We liked their philosophy and approach. And our technologies were similar: both Python/Django stacks running in Amazon, etc. We decided to work together. Rob Rubin and I got the deal done quickly, mostly between sessions at PyCon. In April 2013 we announced that we'd shutter Class2Go and adopt the Open edX platform.

Stanford didn't join the XConsortium. Rooted in our belief in the power of open source, we made open-sourcing their platform a condition of us working together. EdX didn't need convincing. But Stanford was of a forcing function to do it then, and it did take some work. They pushed the button on June 1 2013.

For the past year and a half the Stanford team has operated as a virtual member of the extended edX engineering team. They've been a fun group to work with, generous with their time and good collaborators. We've been running the platform successfully now for over a year, supporting dozens of courses for Stanford students, MOOCs, online high school students, and many other uses. We've contributed back many features, like theming, course email, instructor analytics to name a few.

I've spent a lot of my own time helping to make sure the Open edX project a healthy open source project. It's not enough to just open up the code, to have a thriving community you have to conduct your development out in the open. Beyond helping other institutions get up and running I've worked to drive the open-source agenda overall. With my friend Nate Aune last May we published recommended changes to technology, governance, and community. And with Paul-Olivier Dehaye this past June we hosted the first Open edX workshop, in Zürich. I think those efforts have made a difference.

Moving On

I've heard the siren's song of the startup. Last Friday August 29th was my last day at Stanford. I'll post about my next gig when the time is right.

But I do feel good about moving on. The Stanford engineering team is solid and productive. The engineers work well with each other and have a good breadth of skills. The course operations team is dedicated and strong. Jane Manning will do great running the team in addition to her day job as of product management -- she's done this before and knows the product inside and out. And Jason Bau, the Stanford Open edX tech lead, will continue to anchor eng and ops.

And what about the open source community? I do feel like the flywheel is starting to spin up. EdX is fostering this in several ways, I'll mention two. First, they are in the process of opening up their bug tracking database and sprint planning (Jira). That's a key step to doing true open development. And second, I'm really excited to see the first full-on Open edX conference happening this upcoming November. I expect that to be well attended.

EdX has asked me to continue on as a member of the Technical Advisory Board, which I am happy and honored to do. I look forward to staying plugged into the project and working with my friends on the board (Armando, Ike, Jim, Phil, Ross...)

Thanks to my many friends on the project: the Open edX engineering team, especially Jason and Jane; the team in the Office of the Vice Provost for Online Learning, especially Professor John Mitchell who made this possible; and my many friends at edX. There's a lot to be proud of over the past two years, and a lot of good work ahead.

Arthroscopy Is Amazing

My Left Knee

Three days ago I was hobbling around on a sore knee with a torn meniscus. But after just three days of icing and Ibuprofen and resting on the couch already my knee is almost better than it was before. This arthroscopic surgery stuff is amazing.

My knee had been getting progressively worse for the past few months. But the clincher was six weeks ago, when I was chasing around a dog that was loose in our front yard. A twist, pop, and I was down. The doctor I saw the next day confirmed that it wasn't anything "serious" like a torn ligament, but something was definitely wrong. When we went up to a week of family camping a few weeks back, I couldn't go on any hikes and could barely play ping pong. It was a real bummer.

So I met with a sports medicine doctor at Palo Alto Medical Foundation and got an MRI. He saw a few things wrong in there. A few days later, I was getting the procedure done. Viola.

Afterwards I was surprised that it didn't hurt. They did prescribe some pain medication (Hydrocodone plus Acetaminophen) that I've been taking when I go to bed, but haven't needed during the day. I've been using a machine that pumps ice water through a pad around my knee all day to keep it cool, and that's worked really well. So much better than ice packs. Totally recommend the ice machine.

Thanks to the good surgical team at Palo Alto Medical Foundation especially Dr. Colin Eakin. Their surgery center on Willow Road made the whole experience smooth and reassuring.

But one pro tip for you out there considering this. Don't read the wikipedia article on General Anesthesia the night before surgery. It's crazy stuff. Especially the parts about how we're still not really sure how it works, or the part about how the level of anesthesia where it is safe to operate is right between "excitement" where you vomit and twitch, and "overdose" where you stop breathing. Don't read that part.

I'm back to work tomorrow, and I expect to be back on my bike by next week!