I've developed a nice little way to manage my work in progress folders.

I keep a bunch of folders around where I can stick things that are in-flight: one for home projects, one for blogs, etc. I like accumulating ideas in there, one idea per file, that I can come back to and fuss over until something is ready.

The folder is flypaper for ideas and notes.  My blog folder has about twenty files right now.

Using "ls", and relying on file and directory modification times, is a useful way to keep track of all these files.  Here's how.

As projects are finished I move them into a "done" folder. This gets them out of the way.  A useful side effect is the modification time of the done directly itself gets updated.

The in-flight projects I care about most are the ones most recently updated: as an idea gets older it becomes less and less interesting.  When looking at this directory I use the "ls -ltr" command.  "l" means long (so you can see dates), "t" means sort by modification time, and "r" reverses that sort, so you see latest modified on the bottom.

The old files are at the top where I don't worry about them.  Some even scroll off the top of the screen, that's fine.  The done folder is a nice visual line for when something was last completed (say, published).  Having a separate color for directories makes this visually apparent.  And the bottom of the list are those most recently modified. Generally I work on the bottom up. Sometimes I go back to an old idea and add some notes. That bumps it to the top of the line -- it is interesting again.

I probably type "ls -ltr" at least 5 times an hour.  I've never aliased it since my muscle memory is so strong (but maybe I still should).  

Every engineer has their "the day I did something very very bad" story. This is mine.

It was my last week at Ning. For about a year I ran the Infrastructure and Operations group. It was the end of a good stretch, working with a strong team on hard problems.

In early 2010 (before I joined) Ning had successfully switched from a premium to paid model. Of Ning's two million social networks some were thriving onine communities, but many weren't. Trading those in for one hundred thousand good paying customers had been the right thing for the business. We kept around the 1.9M orphaned sites for some time in case someone wanted to assume ownership and start paying. By late 2011 the revival rate had dropped low enough that it was time to clear out the cruft.

I volunteered for the cleanup. I enjoyed little projects like this because it kept my hands in the code and took away a task that would be a chore and distraction for the engineers. The job was some SQL queries, scripting, load testing, checking, and double-checking. The hardest part was ensuring we had the right list of networks to delete.

It took about two months to write and run the scripts. I felt good that it had gone through uneventfully.

There was a legacy corner of our system that hadn't been included in my cleanup pass. I hadn't worried about it at first since it was just a few thousand networks. But now that I was leaving Ning I wanted to tie off my loose end projects, and this was one. I assembled the list of stragglers and ran my scripts. The 2,000 deletes went quickly. This was 11:00 AM on regular workday -- my last Wednesday on the job.

I immediately knew that I had done something very wrong. Since I sat close to the Customer Advocacy group I heard their IM clients and phones light up. Every Ning page was a "We're Sorry" 404 page. Oh crap.

It only took a couple of minutes to find the problem. One of the networks I had marked for deletion had core Javascript and CSS needed by every Ning network -- important stuff. The network, "socialnetworkmain" was known to everyone as being important, and if I had done my spot checks better, I would have seen it straight away. Instead, I had been sloppy.

The backdoor method I was using bypassed the safeties we had in place to protect our "really important" networks.

Luckily, Phil, one of our top-notch infrastructure engineers was sitting close by and quickly restored the network through the backend. Things returned to normal almost immediately. That feeling of panic started to back off. The jokes started and we started to relax.

It took us another few hours to find and fix second and third order effects of my snafu. The network creation flow was broken -- fixed. An in-house monitoring system was dying silently -- fixed. And so on.

I learned four lessons:

1. Undo is your friend. The tooling or features to revert your change will take time, but you will be happy to have it if you ever need it.
2. Make changes during the day. Resolving a hairy problem is hard if you don't have the right people, and when the people you do have are tired and foggy. This incident might have taken hours to resolve instead of minutes in the middle of the night.
3. It's not over until it's over. Once in crisis mode, you're eager to be get out of it. Once the main problem is solved it is tempting to exhale, go to lunch, go back to bed, whatever. But aftereffects might take a while to show up, or may just be harder to find.
4. Disproportionate effects. We all know that our complex systems nonlinear failure modes. Incidents like these are reminders that small changes have big effects. Said another way, even small things should be taken seriously.

My time at Akamai had prepared me well for managing a crisis. But it's nice to have a refresher course every now and then.  

I've joined a research project at Stanford University (my alma mater).  I am working with a small team to build a platform for on-line education. This post explains what we're building, my part in it, and why this is an interesting area right now.

Class2Go

We are building Class2Go, an application to put Stanford classes on line.  Envision a video-driven web site with exercises and tests.  It will run much like a class today (professors, TA's, lectures, homework, tests, schedules) but with everything happening on-line.  For some kinds of classes this could improve the classroom experience for enrolled students; the excitement comes when you can bring in the a much larger set of non-enrolled students.

The professors who have expressed interest so far are those who want to flip their classrooms or host a Massive Open On-Line Course (MOOC). Others have ideas on experimentation with the learning process itself. For example, do students learn better when they see slides, a talking head, or both? A large enough student population makes meaningful studies possible and cheap.

There are a bunch of features needed over and above basic basic video.  First, can we devise a good system for students to grade each others work? This is an absolute must to do anything in the humanities at large scale.  We have some ideas of how to do peer evaluation, but it's an area that needs much more experimentation. A second important feature is enabling off-line use from laptops and tablets.  When we think of disconnected students we typically think of commuters, but we've heard it's just as important (or maybe more so) for students in developing countries where bandwidth is precious.

While there are commercial offerings to do this, and even things from other established universities, Class2Go will be different in three important ways:

• Research will be first and foremost. Hey prof, you want to do a wacky experiment? We can help.
• Value produced by the course is retained and controlled by the professor and the university.  That value comes from the assets (video, homework) but even more so the community and technology.
• The professors will have quick and ready access to their data. No waiting in lines for reports.

Stanford Professor Rob Reich made a very coherent argument for this in a recent blog post. (Getting to work with smart people like him is a major job perq, by the way.)

Part of what makes this doable is that there are so many great building blocks available.  We plan to use YouTube for video and the Khan Academy framework for exercises.   Piazza has great discussions and forums to start with. Because of all this good stuff, we feel good about getting something strong out by the Fall for a couple of courses, and then expanding from there.

And me?  I'm the line manager. It's fun to work on a project of this size where I can keep my hands dirty. For example, this week I'll be working on authentication (oauth).

Why Education, Why Now?

Two things brought me to education.

Firstly, education is uniquely important. I challenge you to name another human pursuit that is as important, consumes so many resources (for good reason), and can be so transformative when done well.  The other two that come to mind are health and agriculture. But I'd say education is right up there.

And second, education is changing right now. Of the meetings I've had while on sabbatical  some of the most interesting have been about education. Why can't information technology have as big an impact on how people learn as it has had on everything else? Technology could mean making it available to to many more students, in different places and of different means. Educators could spend less time lecturing (dry stuff, to be sure) and more time working with students. Better measurement and analysis and could give educators near-real-time feedback and students tailor-made homework. Those are just a few of the examples that I heard about that got me excited.

I look forward to doing something that will make even a little difference here. I'll blog about my experiences along the way.

PS - Peter Norvig's TED Talk captured the reasons and potential nicely.  That seven minutes is worth your time.

You probably wouldn't hire an engineer that you hadn't seen code. But people tend to hire  engineering managers without seeing them manage. I contend that's a big oversight: you should see them manage on their feet, at least a little.  But how?

I'm thinking along the lines of the "FizzBuzz" class of questions. I believe they got some fame with this article in 2007, and Google seems to support this. Anyway, they are simple programming problems that should be no-brainers for any decent engineer. But they are useful exactly because of the surprisingly large number of candidates who flame out on them. By some accounts, as high as 50%.

So what we want are something similar for engineering managers:  practical questions that can be done within the constraints of an interview. Here are a few that have been useful for me.

Role Playing

Instead of asking them how they would do something, just ask them to do that thing in front of you. Role playing can be awkward if you've never done it before, but give it a try. Once you get over the hump and try it, it can be fun. Here are three good scenarios I've used.

• "I'm a product manager. Customer Foo (say, one of our biggest customers) wants a feature right before the release is to go out. You're the VP of Engineering. You feel like this will destabilize your release.  How do you say no?"  Sometimes it can be fun to play out the natural follow-up, when they go over your head to the CEO.
• "I'm a slacking employee. I have the skills, but for some reason I haven't been productive lately. It's our weekly one-on-one time -- what do you say?"
• "You have to lay someone off. How do you prepare? What do you say?"

If they have follow up questions, or dispute the premise, then play it out. Usually those end up being the most insightful and fun interviews. Also, one warning: questions like these can take you far down the rabbit hole and use up the whole hour if you're not careful.  Remember that its your time.  You should cut them off and switch to the next question, abruptly if need be, to get to everything you need to cover.

Nuts and Bolts

Ask candidates about the mechanics of how they do their job and you can get insight into their experience and values. What tools (wikis, bug tracking, source control, continuous testing) do they use? Do they run team meetings or daily standups? If so, how do these go?

What you're looking for are thoughtful answers. Anyone who has used these tools a lot should have opinions about the them. If they don't then they don't care (bad) or they haven't done the work (bad). Are they negative about everything? Are they flexible?

Motivation

For me this often is the clincher. Why did you get into engineering management in the first place? Why are you doing it now? I won't say what I consider to be good answers, since this varies a lot and should be pretty personal. Again, what you're looking for is a thoughtful answer.

This can be most useful for turning up red-flag responses. You don't want the person who did it for power ("I wanted to boss people around") or ambition ("I wanted to make more money") or just fell into it.  Trust your gut for answers that don't sound genuine and ask probing follow-ups.

I find questions like these can lead to more insightful conversations and get the candidate off-script faster than the typical resume walkthrough.  

Good engineers are attracted to places with a strong engineering culture. But how can you see what the culture is really like from the outside? Here are my two quick-and-dirty indicators.

First a word about what I mean by an engineering culture. It means engineers are valued and important. Some implications:

• How are decisions made? In an engineering culture, technical people have input into what gets built, when, and by whom.  Not signoff, but a real say.
• Is there respect for the craft of making software? Coding is still creative work that requires the right time and space. Some projets are tough to predict how long they will take, and that's needs to be OK.
• Infrastructure. How hard is it for the people who know (engineers, managers) to justify to their bosses when work is needed on non-feature driven stuff? This could be in the runtime system (like scaling work on the message queue) or back office (like build systems or version control).

Unfortunately, teasing this out in an interview can be tricky unless you have someone you really know and trust on the inside.

How big are the monitors?

A story from a prior company. I was an engineering manager that had a retention problem. One of the engineers on my team quit to go to a smaller, hipper company. This was from my exit interview:

Me: why are you leaving?

Him: because they have bigger monitors.

Me: (incredulous) are you kidding? we can get you a bigger monitor.

Him: it's not just me -- everyone has big monitors.

Me: why is that so important?

Him: because it shows how much they value my time. The extra money to cram that many more pixels into my retina must be worth it to them.

And now I understand that this is totally true. Places that value their people consider equipment expenses small compared to the productivity (and happiness) of their people.  The best engineers are given the best tools to do their jobs. Big monitors are a very visible sign of this.

Can people choose their own email addresses?

Non-engineers sometimes don't appreciate how important an email address is. It's your identity on line. A strict naming convention (first name last initial, or worse, last name first initial) indicates place that values conformity over engineer happiness. Worse, its a great way to make their people feel like cogs or "human resources," not the cool individuals that they are.

(Aside: let's do away with the term Human Resources.  It's horrible.)

This one is important for me personally since I have a weird first name. If you don't let me be [email protected] then you get major demerits in my book. And no, clunky alias tricks, like a mailing list with one member in it, doesn't count. It's what you see on your shell prompt that matters; it's what whoami returns that matters.

One final word: this isn't a slam on you hardworking IT guys and gals who keep important things running and have to enforce the rules you're given. Instead, I'm speaking to the bad policies (usually stemming from bad cultures) that can put you into bad positions. If you are at such a place, hunker down and pray for daylight.  

I have two distinct use cases for web browsers: browsing and web apps.  I contend browsers are fine for browsing, but are clumsy for web apps.  This blog proposes how to improve the web app use case.

Case 1: Browsing. I interact with a variety of web sites for a variety of purposes. The common activities here are searching, moving between pages, filling out forms. What is most important to me are rapid navigation, working with multiple pages, even advanced things like saving images to disk.

Case 2: Web Apps. These I use in very limited ways: aside from an authentication flow, I'm often interacting with a single page. Browser features like tabs and search bars are inappropriate and confusing. I want these to feel as much like a native apps as possible.

A web app could be something as simple as embedding a little flash player.  I've done this for the Radio Paradise player, a little station that I like a lot.

A better web app example is Google Calendar. I don't want forward/back buttons, URL bar, or tabs. I want right click to either do nothing, or something very app-specific (new appointment?).  I want "quit" to just quit Google Calendar and not affect any my other work.

In my day-to-day life as a Mac user I use Fluid to get pretty close to what I want. With Fluid I get one window per site, a so-called site-specific browser (SSB). Fluid removes a lot of the browser features, like bookmarks, URL bar, or the search bar, because it doesn't need them. The app get its own top-level icon so it can be in my dock. It's great, I totally endorse Fluid.

Here's a screenshot of another web app that I love: Remember the Milk.  Excellent web app.  Just don't clutter it up with all that browser junk.

But it's tricky enough to get this working that I can't recommend it to "normal" users.   Here are two examples.

The definition of what URL's make up this app can be a bit fussy, and changes over time.  URL's in the app open need to open in the same window, outside the app should open a new full-fledged browser.  This bit me when I started using Google's two-factor authentication.  This additional step in the auth flow, implemented with a new URL, spawned a new browser.  Since Fluid uses Safari, and my default browser is Chrome, the auth cookie got set in the wrong browser, effectively breaking authentication altogether.

The second complication is finding a good app icon. This isn't as acute as broken authentication, but don't trivialize good icons either.  Fluid's default is to use the favicon (32x32 pixels I belive) which is just lousy. Luckily people have sets of icons they share on sites like Flickr, but you have to find them, unzip, etc. That's doable for me, but not for everyone.

I'm glad to see Chrome and Firefox making this easier, albeit slowly. Chrome has the "Make Application Shortcut" feature, but it's been Windows only for a long time. (hint hint Chrome team).

Here's a neat idea: a crowdsourced recipes database for Fluid that would encapsulate all the good settings to make it work and keep them current. As the apps change, the recipes would update and your local apps would get those updates. You could even take that one step further to enable features like limiting geometries and font sizes to presets that make sense for that app.  For example, Remember the Milk (screenshot above) looks good tall and narrow.  Fluid does seem to have a premium feature to allow Userscripts and Userstyles, which could help a bit.

Hmm, maybe I'll see if the Fluid guys want to work on something together on this.

Like pretty much everyone I know, I have moved the data that I care about into a cloud. Actually a few clouds. Mostly this has resulted in lots of goodness: multiple device access, reliability, backup, sharable. But it's also caused a problem.

Where did I put that list of frequent flier numbers? Where are those notes on running a Minecraft server? Will Spotlight help me find it, or do I need to open up a browser?

The challenge is that I need a mental index of what I've stored where (and sometimes when). That index is getting larger and more difficult to maintain. On top of that, all hell breaks loose when two clouds rub together and data is synced/reconciled between them. Witness the mess that my once-pristine address book has become.

Granted, this is a bit of a high-class problem to have. But it does cause friction and confusion, especially as more "regular people" (ie. non early-adopters) switch to cloud storage.

Here's a sampler of the clouds that I have some stake in, to some degree:

• Dropbox
• Google Docs
• iCloud
• Instapaper (only for its own app, but still)
• Box.com (50GB free!)
• Evernote (no longer)

(For this discussion I'm not even broaching things like relationships. Let's keep "data" simple.)

Each stack has its special wins. For Dropbox, it is elegance, always-works simplicity, and cross-platform. Google Docs' real-time collaborative editing is so awesome, it feels like science fiction whenever I use it. And so on. Either you forego those features, or you live with fragmentation.

I look suspiciously at a new application that brings its own sync and store. Is its trick so compelling to offset the complexity? Take Evernote, for example. Their iPhone amd desktop apps are super; data synchronization is quick, clean, and reliable; they have nice integration with apps and a growing ecosystem. But it's yet another cloud.

I wish this would shake out by itself, but I fear it will not. What I want is these stacks to begin to use other backends in addition to (or instead of) their own.  Instapaper on Dropbox; Evernote on iCloud.  But there are at least two big headwinds: first, the big consumer companies will always go it on their own (Google, Microsoft, Apple); second, even the small guys realize that data is sticky and what really matters, so they will hug that data as long as they can.

So in the meantime I hope for integration / consolidation, and try to keep things simple.

An interesting corollary. The consumer backup problem is becoming simpler by devolving into specific types (and sizes) of data. The free clouds big enough to house documents and high-value data. Gmail is large enough for all but the biggest email junkpiles. That leaves two classes of data: music and video libraries (big) and personal movies and photos (also big). With iTunes Match Apple has a unique take on the first set, I wonder who will have a good solution to the second?

I tend to obsess a bit about creating a nice work environment. Software environment, that is.  This blog post focuses on the setup that I have that works for me, specifically those little glue utilities that are so helpful. So I won't cover the big local apps (email, browsers, productivity, dev/test) or web apps (todos, photos), that's for a future post.

Witch ($14, Many Tricks) - Maybe the one I use the most.  I find that the default OS X cmd-tab switching to be annoying by cycling through apps instead of windows. Maybe that's just because I spent so many years living in windows and this mimics the Windows behavior, I don't know, but this is what I consider natural.

Adium (free) - Best IM and IRC client I know of.

iTerm 2 (free) - Replacement for built in terminal.app. Terminal isn't bad, but the one feature this has which I miss from old terminal programs is auto copy selection contents when switching to another app. I do that all the time. Doesn't just save a few keystrokes -- seems every time I expect my selection to be in the clipboard, and when it's not, I find myself hunting around.

Fluid (free) - I like web apps I use a lot (gmail, Remember the Milk, Google Calendar) to function like real apps: own icon, startup and quit behavior distinct from other browser windows, etc.  I've now come to appreciate those as Site Specific Browsers (SSB's), and Fluid is the best way of doing those.  It's a little strange as that relies on Safari, and I use Chrome for everything else, so those SSB's don's share cookies or anything else, but that's not a big deal.  Fluid even has some nice little special cases, like adding unread mail badges to your gmail icon (neat!).

Homebrew (free) - the missing Mac command line package manager.  There are others (most notably MacPorts) but I've tried both and this one seems cleaner and easier.

My current package list is:

• autojump, multitail -- surprising gems
• macvim -- worthy of its own blog
• watch, wget, figlet, dos2unix/unix2dos, wget -- how can they not be included standard?
• fping, wireshark -- network hacking
• gnupg -- crypto