I’ve seen and read a lot of discussions about hiring programmers. Most people claim to want great programmers, but few people know how to even marginally separate the good from the bad. “Great” is a loaded term, and it means different things to different people. What I want to focus on is how to hire competent programmers who can in fact produce a solution and produce working code. Then I leave it to the reader to determine how high they want their threshold set.
Let me start with a story about what I consider the best hiring process I’ve ever been through. Something I’ve modeled subsequent hiring processes on. I don’t necessarily think this process was perfect, but I think they are erring on the right side of the equation. First I’ll give the narrative of the technical screening for the hiring process and then I’ll go back and explain what I liked and didn’t like about it.
I heard through a friend of mine that a prominent web analytics company was hiring in my area. The position sounded very interesting and with my frustration at my current job running at an all time high, I decided what the heck, let’s give it a go. I submitted my resume and within a day or so I had a recruiter on the line talking to me about the position.
The first stage of the interview process consisted of a set of verbal brain teasers given to me over the phone. I managed to work through them and come up with the right answers. After that we talked for a while about what the position would entail and how I would fit into the new company. The next day I received another call from the recruiter, he told me they would be moving me to the next level of the recruiting process. The next step it turned out was another set of brain teasers, this time a bit harder, but still administered over the phone. After making it through that set of brain teasers he told me they wanted to schedule a face to face interview.
So he scheduled an interview with the development manager I would be working for. When I showed up on the appointed day at the appointed time, the manager took me to a conference room with glass windows, a table, chairs, and a white board. He then handed me a printout of a technical problem and asked me to pseudo code the solution on the white board. He told me he would pop in every 10 minutes or so and answer any questions I had about the system. He told me up front not to worry about syntax issues or compiler errors or the like. He wasn’t concerned with that level of detail, just pseudo code a solution to the problem, but make sure it fundamentally solves the problem. After I was finished, we would talk through my solution. With that he left me alone in the room with the printout and the blank white board staring back at me.
I have to admit, my initial feeling was pure unadulterated panic. How hard was this problem? Were they measuring how long it would take me? What if I took too long? Was I good enough to solve this on a white board like this? Was I good enough to solve it at all? I took a minute and tried to calm my heart and slow my breathing. Make sure you are breathing in and out is some of the best advice I’ve ever heard 1.
So after calming myself down, I began reading through the problem. I had never seen this particular problem before, but it was an ACM problem one month. Here is a link to the problem. I’ll be assuming you’ve read the problem from here on. After reading through the problem a few times, and making sure I understood it, I sat there and thought through the problem for a few minutes. After thinking about it, and having a chance to ask my first questions I started pseudo coding a solution on the board.
After about 30 minutes my solution was complete and the development manager came back in and looked over my pseudo code. As a side note, I do my pseudo coding in Ruby because the syntax is so clean and concise, so I had to do some explaining to the manager about a few of the character symbols I used. We talked for a while about my solution and then he quizzed me on how it worked.
Once the development manager was satisfied that my solution would indeed work, he started asking me optimization questions. “How can you make it run faster?” We went through a few things and tightened up my code. Then he asked me again how I could make it run faster? I had already been through a first round of optimization, so now I had to think for a bit. After thinking for a little bit, I came up with another set of changes that would make the code run more quickly. We talked about that for a while, then he asked the question I was afraid he would ask again. “How can you make it run faster?”
There is that panic making a come back, what was I missing? Then he asked me a leading question, “Do you think you could use the results of your first 12 hour run to do some optimizing?”. D’oh, as soon as he said that the solution was blindingly obvious. Of course you can, the first run would define a series of shifts for the balls, after you’ve run it the first time you can just apply the same shift pattern again without running the full simulation. So we talked through how I would implement that for a while.
Then he said “ok, I’m going to ask you for one final level of optimization. No one has ever gotten this, so I don’t expect you to, but I like to see if anyone can figure it out”. After setting the stage like that I was fully involved. Panic? Hah, now I’m having delusions of grandeur, maybe I can figure it out!
“What if we made this into kiosk software and it has to run in the time it takes for someone to type in the number of balls and get an immediate response. How would you make it run that fast?”. Knowing this would take a while to compute especially at the upper bound, I asked him if it would still be bounded by the 27 to 127 constraint as the initial problem. He answered affirmatively. After thinking about it for a moment, the obvious solution is to pre-compute the number of 24 hour periods for each number 27 to 127 and simply ship the kiosk software with a built-in lookup table.
He then asked “What if we removed the 27 to 127 bounds and said it has to be able to solve any entry entered, but still retain the same speed and responsiveness as our last solution for anything that’s already been solved.” From pre-computing to this step was pretty easy. Now you just need your storage mechanism to be extendable so you can lazy compute any value and store the results once it’s already been computed.
After getting through all of that, we talked a little more about the position, and then he showed me around the place. In the end they offered me the job.
The reason I went through that process in as much detail as I did was to give people some inkling of what a solid technical interview is like. I left a lot of detail out because it’s too cumbersome to type it all and my memory isn’t perfectly accurate. However let’s cover the good and the bad of that process.
First, brain teasers. A lot of companies use them, but I don’t think they have any place in a modern programming interview. Solving brain teasers is more about whether you are good at brain teasers than how good of a programmer you are. Case in point, I know several people who are great at brain teasers but are terrible programmers. Solving a brain teaser has no predictive value when it comes to programmers. That being said brain teasers are a great thing to give your HR guy to do if they require you to let them do the first round of screenings. The drawback is you could alienate or lose possible candidates who can’t solve brain teasers but are still good programmers.
Second, the real gem in this process was actually solving a technical problem. It seems obvious that when you are going to buy a car you test drive it. When you are buying a pair of shoes you try them on for fit. When hiring a programmer, you need to see how they program.
Ultimately that is the real point of this long winded blog post. There are two ways to administer programming problems to applicants. The easy way is to mail the problem to them and have them e-mail back a solution, then if you like what you see, bring them in and discuss it with them. There are several benefits to the e-mail option. First, your applicant will freak out less because they aren’t on the spot. Second, you get to see actual code. The biggest draw back though is how easy they can cheat on the test. Most of these problems are just one Google search away. The other problem is they may not even solve it themselves. They may just cut and paste someone elses solution or have a friend do it for them. Most of those risks can be alleviated by bringing them in to talk about their code. Ask them why they designed it the way they did. Talk through the implementation details with them. Ask them about alternative designs.
One key to this process for my last company was having an in-house programmer competition. We all solved the same problem the applicants were solving, then we had to solve it different ways. The fastest solution, the least amount of code, the most elegant, etc. Doing the in-house competition made all of our programmers understand the problem the applicants were being asked to solve in great detail. That way when an applicant came in, everyone could ask intelligent questions about their solution. It also meant we could pass around e-mailed solutions to the team and everyone had a solid opinion about whether the solution was good or not.
If you don’t like the e-mail option you can have applicants solve the problem on the white board, but it’s important that you emphasize the pseudo code approach. If you expect syntactic perfection on a white board, you are probably setting the bar too high.
The ball clock isn’t the only problem to give to candidates. There are a number of places you can find less publicized problems to use. You can search for ACM Problems or you can go to TopCoder and take some of their problems. Before you give it to any applicants though, make sure some of your internal programmers solve it, some of those problems are deceptively hard and may not be suitable for an interview.
The key to a good programming problem though is something you can talk with the applicant in detail about. You and your team need to come up with some really hard questions to ask them about this problem. If they don’t get them all right that’s ok, usually you and your team have had a lot more time to think about this problem than an applicant. It’s just important for you to probe their thinking process.
A quick digression on some other interviewing tactics people use. Don’t expect programmers to have memorized every little technical detail in every corner of the language or platform. I’ve often seen interviewers ask details about incredible minutae, and then get a self-satisfied smirk when the applicant can’t answer it. Yet I know for a fact that if the roles were reversed the applicant could stump the interviewer with some technical minutae as well, particularly if they got the chance to prepare for it. This is a relatively sticky point because many technical details are important and any programmer that works in a technology all the time should know them. Others are arcane bits of lore that even the creators of the language have to look up. There are no hard answers here, but always try to avoid those arcane knowledge questions. They generally have little predictive power when assessing an applicant and usually just instill resentment.
In concluding you need to have a real technical screening process in place. Many applicants will judge you just as harshly as you judge them. If I interview somewhere that doesn’t have a real screening process, I know I will be working with incompetent programmers. As a programmer, the last thing I want to do is walk into a company where I’m constantly fixing someone else’s mess. So unless I’m being brought in to build the technical screening process, I’m going to pass on a company that doesn’t do good screening of applicants. Hopefully after reading this, you’ll be able to conduct a screening that will at the very least sift out the bad programmers or the programmers incapable of producing working code. The sad fact of the matter is, most managers wouldn’t believe just how many incompetent programmers there are in the world.