Tag Archives: interviews

A Tale of Two Interview Loops

by - Reading Time: 16 minutes

It was mainly the worst of times, when a comedy of errors left me on a prolonged job search, alternating between the spring of hope and the winter of despair. But there is lemonade to be made from the almost three dozen lemons interview loops I went through, and this article is that lemonade. Because, while astronomical amounts of money go into recruiting, and even though we live in the age of wisdom, most of the interviewing process is sorely broken at the majority of companies; and so we also live in the age of foolishness.

From my experience, most Big Tech companies more or less copy each other, and because they are giant ships that are impossible to turn quickly, change comes slowly. Many startups and smaller companies then look to the giants for inspiration, out of some combination of familiarity (because they used to work in Big Tech) or imitation (“we’re also Big Tech!”) or just assuming that anything Big Tech does must be good, because one hundred billion dollars can’t be wrong.

And so we end up with the typical tech interview loop:

  1. Sourcing: usually from a referral, but could be a recruiter reaching out after a LinkedIn search or, rarely, actually responding to an unsolicited application
  2. Recruiter screen: to do a sanity check and vet logistics (salary range and other expectations, like remote work)
  3. A general aptitude screen: coding or system design or leadership chat or some combination, depending on the role
  4. A panel, or “virtual on-site”: somewhere around 5 interviews scheduled all at once — but maybe not happening on the same day — with a mix of roles, from engineers to product managers to engineering managers to designers to QA
  5. If that went well, a final interview, with a senior leader, and then an offer

Some variant of this describes almost all of the ones I’ve come across, but: the devil is absolutely in the details, and those details are the difference between a great process and a terrible one. So let’s look at two hypothetical loops at those extremes.

The Bad Loop

If you’re hiring, and elements of the bad loop sound like your company, please think about improving them. If you’re looking, and you run into one a little too close to this: I’m sorry.

Sourcing

The bad interview loop usually starts off badly from the beginning. It’s obvious they don’t invest in candidate experience up front, and it rarely gets better further down the process.

There are a few ways to screw up sourcing, and the most common is just never getting back to candidates. No rejection on merit, no timeout rejection, no insulting GIF — just silence. This happens a lot.

Still better than nothing

Marginally better are companies that do reply, but after way too long. The worst of these for me was a startup that got back to my application — which was a referral — after 47 days. They set up a recruiter screen for the next day, which went well, and then an interview with the VPE for the following week. Unfortunately, that got rescheduled for the next week… and again, for the next week, and then again for the next week, and when it got to the 4th time, I decided to bow out of the process. After 84 days total.

The third common way to turn people away at this step is to send clearly automated emails or LinkedIn messages that are supposed to sound tailored to you. Like the recruiter is channeling a horoscope writer.

Hi Gabriel,

Your profile looks great. It looks like you are thriving at Hillrom, so I know it is a long shot that you’ll be interested, but hope springs eternal!

[Recruiting company] has been retained to recruit an Embedded Systems Manager at [hiring company]. This person will lead the embedded systems team comprised of electrical and software engineers while also serving as a software lead on connected care and digital health projects.

I’d welcome the opportunity to talk with you. Please let me know if you are interested to learn more. Thanks in advance and I look forward to your response.

An actual email I got. I’ve never worked with embedded systems.

In short, the sourcing is bad if they:

  1. Don’t value candidates, and
  2. Don’t value the company’s image

Recruiter Screen

This step is actually pretty hard to do badly, but it generally involves the recruiter being out of their depth and/or not caring about their craft. I’ve spoken to both (a) recruiters who had no idea there was a difference between Java and Javascript, and (b) ones who did, but treated our conversation like the most annoying part of their day — like their Zoom background might as well have a giant “I’d rather be doing literally anything else” banner. And of course, the worst is (a) and (b) combined:

Recruiter: yeah hey, I’m in a rush, but this won’t take long

Me: okay…

Them: so I saw your resume, I don’t have it in front of me right now, but can you just answer a few questions?

Me: sure

Them: ok so this is a management role, you have 5+ years of experience managing, recruiting, and mentoring a team of diverse engineers?

Me, realizing they have, in fact, not seen my resume: yeah

Them: do you have 8 years of Java?

Me: uh… [trying hard to make it past the syntax errors] … yeah

Them: do you thrive in a fast-paced environment?

Me: sure

Them: can you speak good English?

Me: I think you said the quiet part out loud.

General Aptitude Screen

After the boilerplate kind of checks, comes the first true test of the candidate’s mettle: do they actually have a notion of how to do this job, or are they a complete fraud? For engineers, this is almost always a somewhat basic (sadly, often leetcode-y) technical interview. For managers, it’s one of three things:

  1. A purely technical interview, because the thinking is managers should also know how to leetcode
  2. A leadership interview
  3. A bit of both

The bad versions of this are the ones that don’t test aptitude required for the job. We’re hiring a staff engineer for their background in designing APIs and scalable architectures, but first: let’s make sure they can figure out the trick to finding the longest palindromic substring. Because that’s the thing they’ll constantly trip over in their day-to-day job: figuring out solutions to neat little puzzles that can be done in 20 minutes.

And that’s really the crux of the problem, because this kind of exercise filters out a few kinds of traits that, in my experience, a lot of good engineers have:

  1. Not performing well while being watched
  2. Not having done algorithmic puzzles in years
  3. Not particularly caring to spend dozens of hours preparing for interviews
  4. Not having slept well the night before the interview

The arguments for leetcode interviews don’t hold water either:

  1. It’s fair: everyone from the new bootcamp grad to the Sr. Fellow gets the same questions. Is that fair though? It’s fair for the new grad, because they have no work experience and it’s all they can be tested on. And early in their career, they should still remember this stuff. But it’s not fair to someone with a decade of experience that (correctly) hasn’t thought about sorting anything beyond typing my_list.sort() in almost exactly as long.
  2. It shows motivation: the thinking is that if you’re willing to put in the hours to dust off long-forgotten skills, that means you’re a go-getter. Either that, or you have a ton of free time. Or are willing to work hard for 3 months so you can rest and vest.
  3. It’s a great weeding-out tool: “here at Hooli, we get far too many yahoos applying and we need a simple and consistent way to halve the middle of the funnel.” This I can’t argue with. If you have too many applicants and just need an efficient way to politely send scores of them away, while making it look like meritocracy, then this loop design does in fact meet those requirements.

Panel

The next, and sometimes final step of the process tends to be an interview panel. It’s kind of like a death panel, except it’s very much real, and only decides your professional and financial future. It’s a holdover from the days of in-person interviews, which you can tell because it’s often called a “virtual on-site”.

Back in the before times, companies used to fly people over to their headquarters, even across country, for a gauntlet of interviews held over a full day. You’d talk to 3-7 people across ~4 interviews — because sometimes they’d double up, or you might even have several people in one interview. Like much in-office culture, most companies have lifted and shifted this process into the Internet, and now offer the same experience, but worse because it’s over Zoom. Also better though, because you don’t have to fly to every interview.

The idea of having several people interview a candidate is obviously fine. The problems here are more around how they’re structured, and how much freedom the candidate has to tailor this step so they can make a good impression. Because to a lot of people, if not most, this step is very stressful. Some would rather do it in one day and get it over with (even if they’d perform better otherwise) and some would rather split it into two or three days. Companies that don’t care, also don’t care about candidate’s preferences here.

The more general problem though, that many companies are guilty of, is not adapting this step to the modern world. It’s not great for the company to sign up for more than ten people-hours of time (adding in the prep, write-up, and debrief time) just based on the one aptitude screen. Especially if it’s a low-signal one, like leetcode. Data is king here, but I would bet the panel pass rate at most quality companies is not that high, and so that adds up to a lot of waste all around: effort and stress for the candidate, plus time and energy for the interviewers.

The last thing to mention here is the companies not investing in quality tools, especially for the technical interviews: having the candidate write code in something like Google Docs instead of a purpose-built thing, like CoderPad or HackerRank. Same goes for whiteboarding, for like system design exercises. And using Microsoft Teams.

Final Interview

This step, mirroring the recruiter screen, tends to be more of a formality or sales tool. By now, the candidate has passed the panel gauntlet, and this final hurdle gives good candidates the opportunity meet with a senior leader in the organization, while also giving that leader the opportunity to meet with promising hires before the offer goes out.

Where this is done poorly, it’s either not done at all, or it’s a very rigorous interview. Not doing one at all can be fine, but — especially in a more talent-driven job market — having this step can sway a candidate with multiple options. It’s also great, from an organizational health perspective, for the most senior leader to either be part of the interview loop, or schedule a 1:1 shortly after the candidate starts; doing both is even better.

This interview should be real, of course, with questions that give the interviewer insight into the candidate; but if it’s too much more than formal, it can both put the candidate (who has already been through the gauntlet) off, while also causing an internal problem if the candidate fails a hard interview, after having already passed the panel. Can the more senior leader just override the panel? Because that doesn’t sound like a healthy org.

Surprise Step!

Sometimes, panels are split. Lots of soft thumbs up and down. And the job req has been open for months, so they don’t want to pass on a potentially good candidate. And so they decide to tack on another interview — or maybe repurpose the Final Interview above — in order to break the tie. This is a smell of a poor interview process. One that is not designed to produce strong yeses and nos. Or one that is, and has failed. Most well-run processes will explicitly forbid this, and for good reason: an 8th person won’t add that much value to the chorus.

The Good Loop

Now, for the fun one! Sorely few of the interview loops I’ve experienced have been good, but for the ones that have, it provided so much signal that it would’ve been worth taking a lower offer. A lot of the anxiety of committing to a role comes from the company’s culture, and a great culture usually comes out through the interview loop.

Sourcing

Great sourcing is genuine. Also the things I said earlier about valuing the candidate and the company’s image. But great places to work do it with care. The messages are personal, not form- or AI-generated. They show that the sourcer has actually read and understand your resume, as well the job description. And they make a good case for why they think you’d be a good fit and choose to go through their particular interview gauntlet.

But good sourcing is hard. To do this well, it obviously takes a lot more time, and therefore people, and therefore money. So it’s not a bad signal if a company is just okay at this. BUT, if the sourcing really is great, that’s great signal: it shows the company has committed actual dollars to candidate experience, because they care about attracting the best employees into a great environment.

Recruiter Screen

A great recruiter screen gives the candidate a positive glimpse into what the rest of the process will be like. The recruiter is prepared, understands what the candidate’s resume or LinkedIn says, and has good follow-up questions. The chat is friendly but organized and the run-down is provided up-front: “first I’ll give you an overview of the company, talk a little more about the role and give you a chance to ask questions, and then talk about your experience and how it might be a good fit.”

It’s mostly proforma, but value comes out of it: the recruiter gains more clarity than the resume provides, makes sure the logistics are lined up, and the candidate asks questions about the team/org, responsibilities, tech, etc that aren’t in the job description. Even if it turns out that something doesn’t line up, like salary expectations, everyone leaves feeling like it was a productive call.

General Aptitude Screen

An effective aptitude screen tests just that: how well this person is suited for this role. This means that the defining characteristics of the role have been thought through and distilled into discoverable skills; and then questions have been created which are good at discovering those skills. But an important piece of the puzzle is where the confidence comes from, that those questions illuminate the correct things. In most places, it’s just someone’s gut feeling.

We make social networks here at Hooli, so we need to make sure people understand Dijkstra’s algorithm. We’ll ask a question where the obvious solution is the algorithm, and pass people that either use it in the solution; or they mention it, have a good reason for not using it, and still come up with a good solution.

Hypothetical, but I’m sure someone somewhere has written interview questions like this.

Some problems with the above:

  1. Do successful employees really need to understand Dijkstra? Is there a correlation between that and job performance? Is it important for this particular role and experience level?
  2. To most candidates that are familiar with Dijkstra, is it obvious that it’s the right solution here? Does the wording and the problem statement lead them there? Or is it just obvious to the people that came up with the question?
  3. Can the question actually be favorably solved in another way? Are we too narrowly targeting a particular way of thinking?

Good questions have been purposely designed all the way through. Their goal has been validated by data as being good predictors of employee performance. The questions themselves have been vetted by trying them out on actual people, getting their feedback, and looking at false positives and false negatives. Ideally, those people aren’t just employees — though this, of course, is difficult.

The above applies to questions asked as part of the panel below. The essence here is that a great aptitude test is effective at predicting job performance.

Panel

I’m not sure what percentage of rejections in the interview process are false negatives — meaning a perfectly good employee has been turned away — but I would bet a small amount of money that it’s significant. As in, more than a quarter. And I’d also bet that a lot of that has to do with the candidate being nervous. And I’m not alone.

Chris Parnin of NCSU has a great paper in ACM on the matter, pointing out two things we all already know, intuitively:

  1. An estimated 40% of adults suffer from performance anxiety, and
  2. The typical software interview “has an uncanny resemblance to the Trier social stress test“

For those of you who haven’t clicked on that link, that stress test was created to induce change in physiologically measurable stress markers, like heart rate, cortisol, and other things. The test makes those markers increase, in most people, anywhere from 30-700%. And how does it do that? It’s a simple, three part process:

  1. You have 5 minutes to prepare a 5 minute presentation for a job interview
  2. Ok, it’s time for the presentation, but: surprise! Your presentation materials have been taken away.
    • Also, the judges will be silent and maintain only neutral expressions the whole time
  3. Count down from 1022, by 13s. If you make a mistake, start again from 1022.

“Uncanny resemblance” indeed, right? It’s basically the same thing as software interview, which gives the candidate a complex problem — much more so than counting down by 13s — except they don’t even have time to prepare! And the typical interviewer is cold and silent. (As an aside, please don’t be that interviewer. Read my Art of Interviewing, if you are.)

What’s more is that he says “scientific evidence finds that women experience disproportionately more negative effects from test and performance”, and while I’m not saying that this is the only reason tech might be so heavily male-dominated, I wouldn’t be at all surprised if it turns out to be a big factor.

So what’s a company to do? The above paper has one suggestion — and I think it’s a special case of a more general solution — which is the “private interview”. That is, rather than give the candidate a problem and watch them solve it, give them the problem and go away, letting them solve it in private. When they ran this experiment, the results were undeniable:

In the public setting 61.5% of the participants failed the task compared with 36.3% in the private setting. […]

Interestingly, a post-hoc analysis revealed that in the public setting, no women (𝑛 = 5) successfully solved their task; however, in the private setting, all women (𝑛 = 4) successfully solved their task—even providing the most optimal solution in two cases. This may suggest that asking an interview candidate to publicly solve a problem and think-aloud can degrade problem-solving ability.

from “Does Stress Impact Technical Interview Performance?”, in Association of Computing Machinery

The private interview is a great idea, and an improvement on the “take home” challenge, which suffers from the fact that some people will spend an hour on it, and others will spend ten, with two friends helping. But, there are problems with it. One that the paper mentions is that some people were frustrated by not having access to the proctor, to ask clarifying questions and be kept from drifting. So it’s not a one-size-fits-all solution.

But what’s the philosophy behind the private interview? That most people get too stressed out by a stress test an interview, and doing it in private helps. Most people though: not all people. And that, I think is the key insight: what the paper calls “provide accessible alternatives”.

Great companies will tailor their process to help the candidate succeed. They will allow the panel to be split up over as many days as there are interviews. They will allow the candidate to pick the programming language. They will have a “live” option and a “take home” option — I have yet to see the “private interview” option, though it’s by far my favorite. They will train their interviewers to be kind, to be expressive, and to focus on the person, not the question.

In short, as with all good science, they will make sure they have adjusted the process to eliminate as many confounding factors as they can, in order to minimize false negatives and false positives, and give everyone a fair chance.

Final Interview

There’s not much left to say about the final interview other than that a good leader will leverage them to motivate a promising candidate. So instead, I’ll summarize the two loops:

  • bad: disinterested people going through the motions of a process that will usually produce some result
  • good: thoughtful, kind, well-trained people assessing how well this candidate might fit into this role

The Art of Interviewing

by - Reading Time: 10 minutes

The job interview process is a high-stakes dance that’s notoriously difficult and full of missteps — on both sides. (At least, in software engineering it is. If you care about another industry, <Jedi hand wave/> this is not the article you’re looking for.) Avoiding the missteps is an art, just like it is in dancing; we get better with practice, just like dancing; it’s a lot of fun when it goes well, just like dancing; and leading with dexterity is paramount. This article focuses on that last bit: not on what questions to ask or how to structure the time, but rather on artfully leading an interview. First, however — because it’s a crucial part of the interview context — we should start with how the job candidate views the process. Just like dancing.

Because while for Larry the Interviewer, it’s just a small fraction of his day — maybe an annoying one that pulls him from the dynamic programming problem he was definitely not working on — for Jane the candidate, it’s one of the most important events of her life. The median job tenure is about 4 years, so people will only have somewhere around 10 jobs their whole professional life. And each of those jobs is going to significantly affect Jane’s life both during, and after, it. The CV builds on the shoulders of the previous job, yes, but work is also where we spend the bulk of our weekday, make friends, and develop a large part of our identity.

So what does Jane go into this high stakes dance equipped with? If the company is large, maybe she can get a sense of the general culture from Glassdoor, or news articles, or social media. Otherwise, maybe a sense of how it wants to be seen, from its marketing. But even then: what will her daily life be like there? Would she like her new boss? Her teammates? The bureaucracy? The tasks she’ll be working on over the next year? Four years? Unless she knows someone on the inside, these questions mostly won’t get answered until after the start date. Sure, she’ll get to peek in now and then through cracks in the process and through the five minutes of each interview in which she can ask stuff, but: largely unanswered. Which means it all adds up to a big gamble.

The gamble is better for more mercurial and adaptable personalities than those more averse to change, but your resume can only tolerate a couple of quick stints before it starts getting tossed aside. So it’s a big gamble for everyone. Or rather, I should say “for every job searcher”, because well… it’s a small gamble for Larry the Interviewer. Worst case for him? It isn’t even giving his “thumbs up” to a terrible employee — because someone truly terrible will get fired before too long. No, the worst case is that he gives his thumbs up to a really annoying Taylor that ends up on his team and really annoys him for the next 4 years. And also produces mediocre work that Larry then has to constantly deal with. But Taylor isn’t annoying enough and the work just isn’t bad enough to get fired or even be put on a PIP. Taylor kind of coasts just baaaarely on the good side of the policy. And in doing so, makes Larry’s work life feel like a chirping smoke alarm that he can never find. That is the worst case for Larry.

The best case? He ends up with a really awesome coworker. Which, depending on the existing coworkers, may or may not matter a lot. But in any case except for the narrow and unlikely one of Taylor, the stakes for Larry are much, much, much lower than for Jane. And yet, this is who decides Jane’s fate.

The irony is that all of us will — at different times — be the Jane, and most of us will also be the Larry at other times. And when we are the Larry, interviewing a job candidate, do we act as the interviewer we’d like to have interview us, when we’re searching for a job? When I grudgingly leave the house, I usually drive, and sometimes I walk, and sometimes I bike. And what I find fascinating is that when I’m a driver, I get mad at other drivers, when I’m a cyclist I get mad at drivers, and when I’m a pedestrian, I get mad at everyone. Even though I know exactly what challenges they’re all going through.

Through this lens, how should Larry — all of us, when we are the Larry — conduct his interviews? How can we be the Larry we want to see in the world?

Larry does have a concrete output from the interview; he needs to answer one simple question: is Jane likely to be successful in this role? “But wait,” you ask, “can Jane’s ideal Larry even answer that question? Can he both be empathetic and figure out if she should be hired?” To which I say: “not only can he, but that’s the best way to do go about it!” Let me illustrate by looking at some questions Larry should not be trying to answer:

  1. Did Jane answer all my questions correctly?
  2. Was Jane quick on her feet and graceful under pressure?
  3. Did Jane pick up on my algorithmic hints?
  4. Was her solution complete?
  5. Do I like Jane, as a human being? Could we be friends?

Those are certainly signals one can get in an interview, but are they relevant to her success in the role? Does her answering all of Larry’s questions correctly mean there’s a good chance she’ll get a high performance rating next year? It depends on the questions, right? Well then, what about the inverse: does getting answers wrong correlate with bad performance reviews? Or does it correlate with nervousness? Or miscommunication? Or ignorance of a concept that Jane could learn in the first week on the job?

The hard truth is that while most interviews test something, that something is more likely to be “ability to pass our interview” than “ability to do the job well”. Which — if the abstract interview is indeed a conscious choice, and more often it is not — then the argument for it goes like this:

We want people that will do what they need to in order to succeed; if they study hard for our arbitrary and irrelevant interview process and pass it, it means they (a) really want to work here, and (b) can do the same for a real-world project

Even for well-known companies that can attract enough people to run their gauntlet, this is of dubious value, because they’re filtering for a specific criteria: studiousness; and simultaneously filtering out many that are at odds with it, starting with people that don’t have the luxury of time to study for their interview.

Instead, I think interview questions should be relevant for the job and tailored to illuminate whether Jane will be successful in it. Does it matter if Jane came up with a coding solution in 10 minutes versus 20? How much time would she have in the course of her job for that problem? Does the job require her to be well-versed in algorithms? Even if Stack Overflow didn’t exist. Is it relevant that she didn’t finish the last part, even though she explained how it would work? Are the people we like more productive than the ones we don’t? Do these rhetorical questions make my point?

Which is that the interview is not a test and it’s certainly not an interrogation. It is above anything else a conversation, ideally between equals, in which both parties are trying to figure out if an employment arrangement would be a win/win scenario. And as the interviewer, by leading it with empathy, you accomplish two things:

  1. Have a much better chance at arriving at the truth
  2. Leave the candidate with a great impression

So how should Larry approach the interview? I like to think about role models, because we humans are great at mimicking, and in this case I think the right archetype is a podcaster. They have a guest on their show and they generally try to make the experience pleasant, to keep the content interesting, and to really get at what makes their guest tick in their particular way. If they have an actor on, they try to figure out what makes them a great actor; if it’s a business tycoon, what makes them great at business; if it’s a scientist, what makes them great at science. And similarly, Larry’s job is to figure out what makes his guest great at programming.

In order to successfully do that, the guest has to first and foremost be comfortable. People won’t let you in unless they’re comfortable. And if they won’t let you in, it is a giant barrier to understanding them. So Larry should spend some time in the beginning of the interview breaking that ice. He should make small talk — genuine small talk, not awkward conversation about the weather. Tell Jane a little about himself: what he does at the company, what he’s passionate about in his work, and what his background is. Things that will give Jane an understanding of him and help them find common ground. It’s not only worth the investment, but it’s what makes the rest of the interview worthwhile.

Once a good rapport has been established — or Larry’s given up on such happening — only then should he go into the topics he needs to cover. And he should always remember to treat Jane as he would a valued guest in his home. To be kind, to be forgiving, and to give her the benefit of the doubt. If she says something that sounds incorrect, he should make sure it’s not due to a simple mistake or misunderstanding. He should ask polite questions that help him understand how much Jane knows and understands about graph traversal or whatever — not merely that she does (or doesn’t) know that those words are the answer to his question. Because in the end, Larry’s job isn’t that of a proctor, to simply grade Jane on her performance as if this were an audition or a midterm exam. Larry’s job is actually much more difficult: it’s to create, in 45 to 60 minutes, a sorely incomplete mental model of Jane from a certain angle — be it programming ability or cultural fit or leadership style or what have you — and to then decide if that mental model is a good match to fill the open role.

Again: this is hard and it takes a lot of practice to do well, just like dancing. But taking shortcuts will just lead to lots of false positives or negatives. Tech companies love to industrialize the process and create complex questions with “objective” answers and rubrics that tally up those answers into a simple pass/fail exam and then also point to that process when talking about fairness. But in truth, there’s nothing fair about standardized tests — which is why higher education is finally moving away from the SATs.

They add a veneer of objectivity on top an industrialized process that answers not so much what a person understands, but what they’ve been exposed to and what they can quickly recall in a stressful situation. It’s like being tested on “ticking time bomb diffusal” for a job making watches. And the opportunities for subjectivity still abound: from how comfortable or nervous the candidate is, to how many hints they’re given, to how much sleep they’ve gotten the night before, to whether they’ve seen a similar question recently, and to the leniency of the proctor.

What sets Oxford and Cambridge apart from most other universities is that they use the Tutorial System, in which students learn the subject matter in whatever way makes sense, meet in very small groups with a tutor and have a discussion — in which it will become readily apparent how well they understand the subject.

It has been argued that the tutorial system has great value as a pedagogic model because it creates learning and assessment opportunities which are highly authentic and difficult to fake.

from Tutorial System at Wikipedia

Software interviews of a similar ilk have the same value.

But the one thing to remember is that regardless of the questions we ask when we’re being the Larry — no matter how good or fair or comprehensive they are — the questions are just a means to an end, and not the end. They are a conversation starter, and it’s up to us to guide that conversation in such a way that will allow us to understand our guest to such a degree that we can answer the only question that matters: “is this person likely to be successful in this role?”

Lessons Learned on the Job Search

by - Reading Time: 13 minutes

I’m starting what I’m sure will be a great new job at an awesome company soon, but before I got that offer (and four other great ones), I was on the interview circuit for a few months. When I started looking, I figured that in this boiling job market (of mid-late 2021) it would take like around a month. For better or worse though, I hit a string of bad luck on what turned out to be an already overly-optimistic timeline: two hiring freezes (both at the offer stage in the interview loops) and a conflict of interest (after an offer).

I say “for better”, because this bad luck did come with a silver lining: it prolonged the process enough that I ended up in a lot of interviews: 79, over 34 interview loops. Which, on the one hand, was exhausting; but on the other hand, it gave me huge exposure to the current landscape of interview processes for engineering leadership at tech companies. I’ll write another article on how I think these processes can be improved, but this one is about how to deal with them — as a job seeker. Specifically, an engineering leader, such as management, or staff-level IC and higher.

This experience was so radically eye-opening for me, because aside from a random interview now and then, I’d never been exposed to it on this level. And in the pre-pandemic world, this gauntlet would’ve been impossible, due to the travel alone. My previous jobs came to me — which is a completely different dynamic — so I’m writing this to share what I’ve learned, mainly for others that are looking for a change, and are as unfamiliar with the landscape as I was.

The main takeaway: interviewing is a skill onto itself. It’s largely unrelated to the day-to-day of the job, it matters a great deal, and it’s absolutely something that can be learned. It’s like dating that way: the first few dates are crucial, and making a good impression is paramount, but the impressions you get on those early dates contain a lot of not only false signals — like being nervous or interested in everything your date loves — but also superfluous signals, that don’t matter a year into the relationship — like your favorite band.

And unfortunately, both dating and interviewing are optimized for charming (and good looking) people, who will always win out if everything else is equal. Fortunately, being a polished interviewee is a skill that I think most people can master with practice. Not unlike the premise of the movie Hitch. Because you could be the most amazing engineering leader in the world but, if you don’t interview well, no one will ever know that — aside from the people who’ve worked with you.

Before we get into the details, some more context: I interviewed with a wide range of companies, from Big Tech like Meta/Facebook and Amazon to smaller ones like like Etsy and Reddit, to startups at various stages. I applied for only remote roles, roughly evenly split between Engineering Manager (EM), Sr. EM, and Director — with a few senior IC roles added in — and all my interviews were remote, via Zoom or the like.

And I kept statistics.

But don’t worry, there’s a TL;DR section at the end that you can always skip to, if it gets too dry.

Anatomy of a Loop

Most companies have a 4±1 step process:

  1. Screening call with an internal recruiter
    • 30% of my interview loops skipped this step
  2. A first round, which is generally a role-fit discussion with the hiring manager (HM)
    • I consider this to be the first round of the loop
    • For me, 76% of the 25 of these I had were just a casual discussion with the HM
    • Of the remaining six:
      • four were a panel (two of which had technical components)
      • one was a coding challenge
      • one was a design challenge
  3. A second round, consisting of one or more people, sometimes as a panel (meaning multiple interviewers are in the video call), sometimes as a string of video calls with individual interviewers. But all pre-scheduled together.
    • Eight of the 13 I went through had a technical bent:
      • four included a technical chat
      • three design challenges
      • one coding challenge
    • Six of them were also panels:
      • one of the design challenges, and three technical chats
  4. A third round, which is usually the opposite of the second: if they did a panel first, then this is not a panel; and vice-versa. If they have a technical screen and the second round was technical, then this won’t be; and vice-versa.
    • 29% of my interview loops skipped this step
    • Three of my five were panels, and one of those was technical
    • The other two were informal chats with a CTO or VP
  5. Hopefully, an offer
    • Most of my interview loops skipped this step 🙂
    • All five of the offers I got had a fairly consistent base, plus 10-20% bonus, but big variation on equity — not just in terms of value, but also type: from no equity to private options to public RSUs. Equity ended up being the deciding factor for me.

For me, the average time from resume submission to some kind of decision was 32 days, but the max was 93 days, and 18% took over 45 days. Most of this was just waiting a long time for the initial response, and then for the next steps to be scheduled, each of which generally took a week or two.

The average time from resume submission to some kind of response was 17 days, but the max was 70.

What Worked at Each Stage

Resume

As you can see above, this was the biggest rejection stage, by a Grand Canyon-sized margin. A full quarter of my resume submissions were rejected, but even more (39%) were ghosted, and I never heard anything about them. I put a lot of work on my resume — and it always helps for it to be short (one page, if possible, because no one has time to screen them), to include the buzz words you’re skilled with and interested in (to make it past recruiters), and to be visually attractive (you can find lots of great templates for Google Docs on Etsy, for under 5$) — but that’s not what made the difference, as I found:

It basically all hinged on referrals, which increased my chances of getting to a recruiter or hiring manager from 5% to a whopping 63%! That’s a 12x improvement, which is impossible to get any other way. I’m sure it helped that my resume looked sharp, but I’m also sure that even with my previous, plain and long-ish resume, the referral success rate would still be multiples of the cold application.

So if you’re looking for a job: tell everyone you know in the industry, and hopefully they know of an opening or know someone who does and even if they can’t vouch for you personally: if all they do is pass your resume along with a “this person might be good for this role”, that itself does wonders. Also, if you find a job posting you really like, try to find a recruiter or someone in your network from that company. If you don’t know anyone directly, look at 2nd connections on LinkedIn, and then ask your mutual connection to introduce you.

Ironically though, in spite of all of this, I ended up at one of the 3 companies in the bar graph above where I had no network connection. And for all three of those, I wrote a thoughtful message in the freespace of the application. Now, I also did that for probably at least a dozen others, so not a great success rate there, but: if you’re applying cold, it definitely helps to write the recruiter a note about why they should choose your resume out of the pile. None of my other cold applications made it out alive.

The other big category is getting recruited. Most of the ones that reached out to me were either Big Tech companies or startups I’d never heard of, and most of those were clearly automated messages. I entertained almost all of them, if for nothing other than the interview experience. And here, LinkedIn is key: using the appropriate phrasing and buzz words and highlighting the experience relevant for the kind of job you want. Recruiters use various tools to search through the hundreds of thousands of potential candidates, and a little SEO goes a long way.

Recruiter Screen

At this stage I did pretty well: 3 rejections out of 18 conversations, for a pass rate of 83%.

What the recruiter wants is to make sure you’re not going to embarass them. They’ve seen your resume and decided that based on your history and so forth, you’d probably be a good candidate. So all you really have to do is sound competent, back up your resume with your voice, and fit within the logistical parameters for compensation, location, time zone, etc.

Hiring Manager Screen

Of the 19 HM screens I was in, my pass rate was 68%, and I have to say that most of the rejections at this stage were surprising to me. I thought almost all of them went well, and in fact I had at least two of the HMs tell me they’d like to move me forward, only to get a Dear John email a few days later from the recruiter. I imagine they talked to someone they liked more the next day, but one of the more frustrating parts of the process is that rejection feedback is exceedingly rare. The unfortunate truth is that the overwhelming majority of companies do not provide feedback, largely for legal reasons.

But like the recruiter, the HM largely wants to make sure the person behind the resume is a solid candidate, that they seem personable and would fit on the team, and at least part of this conversation is them selling you on the job. They ask more poigniant questions than the recruiter obviously, and have more details around the position and what they’re looking for and have a better sense of how you’d succeed in the position, but largely this stage shouldn’t be hard, and if you don’t pass, you probably wouldn’t have fit in on the team anyway — not that you’re not qualified, but team dynamic is a real thing.

Behavioral Interview

The vast majority of leadership interviews I was in were what has become the norm the tech industry: storytelling. I can’t tell you how much PTSD I have over the phrase “tell me about a time when …”. The idea is that an experienced manager can (a) demonstrate that experience by recollecting tales from a storied career, and (b) knows to emphasize the parts that the interviewer wants them to emphasize. In the beginning of my job search, I would fail miserably at the second part with the mind-reading, because I didn’t realize there was a hidden agenda to the question.

So they would ask me to tell them about a time when I promoted someone, and I’d literally tell them about such a thing and move on. When in fact, they also wanted me to tell them about my philosophy (a.k.a framework) around promotions, and maybe even around career development in general, and emphasize how that’s an important part of a manager’s job, etc, etc. And because of that, my pass rate in the second round (where behavioral interviews would often take place) fell to 54%, and was my lowest. And that’s after I cracked the code. It helped when reviewers would nudge me in a certain direction, but that didn’t happen often, and certainly not with Big Tech, where the rubric is king.

If you’re interested in seeing examples of this tactic, watch some videos on ExponentTV (and paying for Exponent is probably a great return on investment as well) to get a better sense of the kind of answer interviewers are looking for. And make a list of stories you can cycle through quickly, on the spot. The answers are supposed to given in the STAR format. There are various lists out there, but here are some of the ones I actually got:

Tell me about a time when…

  1. you had to manage someone out
  2. you had to deal with a difficult employee
  3. you had a disagreement with a peer
  4. you promoted someone
  5. your project was late
  6. you failed
  7. you dealt with a DEI issue
  8. you motivated your team
  9. you affected change without using authority
  10. you were part of a large undertaking

One prevalent theme was essentially #6 above, and that was also something I wasn’t quite sure how to answer. It felt like the old joke to me, about the interviewer asking what the candidate’s greatest weakness is, who replies that he works too hard.

But the reason people ask this is twofold: first, because experience implies failure, since no one has a perfect batting average. And so the idea is that if you’re able to talk about a time you’ve failed, you’re more likely to be an experienced leader. Second, being able to talk about failure in a way that shows humility is important in that it demonstrates some good leadership qualities such as empathy, the ability to learn from mistakes, and the ability to turn things arounds.

And herein is why getting good at interviewing and storytelling is so important, because it’s not enough to merely have those qualities: you have to realize when you’re being subtly prompted to reveal them and to do so in a tactful way. But again: it’s definitely a skill that can be learned.

Technical Interview

Only 20% of my post-recruiter-screen interviews were rigorous technical ones, and I think about half of the loops had such a stage. So it’s definitely possible to get a job without running into one, but they’re very common in Big Tech.

Of my nine technical interviews, five were design questions — and one of those was a take-home exercise. Another two were presentations of a project I’d done in the past, and the last two were coding challenges.

One of the coding challenges consisted of two leetcode-type algorithmic/data structure questions. There’s nothing to be done about this except grinding on leetcode until that part of the brain that lay dormant since your CS college courses has sprung back awake. Yes, it has nothing to do with anything in a real job, but such is life.

For the presentations, I gave essentially the same one both times: passed one, failed the other.

The design questions generally ask you to design some kind of system you’ve likely dealt with before: a search engine, a messaging service, a URL shortener. Here, it’s important to start at a high level — the main components and connections — make sure you’ve covered enough there (and take hints in the form of questions from the interviewer) and then go a level lower and talk about technologies, protocols, tradeoffs, and so on. Anything to make the interviewer feel comfortable that you have the technical chops to design things. Again, Exponent has some great videos to give you an idea of what a good one looks like.

TL;DR

Interviewing processes are very broken at most companies, and as a job seeker, there’s not much you can do about it except refusing to take part in broken processes — such as live coding challenges — if you have the luxury of doing so. For many people though, especially in the new remote world of SF/NY salaries, this would mean walking away from a significant pay raise.

Otherwise, the things I wish I had known at the onset:

  1. It will likely be a multi-month process, due to the speed at which most companies operate; I would prepare for roughly 3 months
  2. The best thing to get your resume noticed is to make use of your network
  3. When you do have to cold apply, writing a paragraph to the recruiter is worth the investment
  4. A polished, short, beautiful resume is worth the investment
  5. Optimizing your LinkedIn profile for recruiter searches is worth the investment
  6. Feedback, about why you were rejected, is an endangered bald eagle: rare to come across and wondrous to behold
  7. Practicing storytelling for the behavioral interview, and looking for the question behind the question, are both crucial
  8. Practicing leetcode is needed for many interviews at Big Tech companies
  9. Good PowerPoint skillz might come in handy
  10. There are lots of mock interviews available to watch — both behavioral and design — and they help a lot

All of this can be boiled down to, as Patrick McKenzie put it, “becoming aware of how power operates (versus how virtue is generally described) and choosing to join it”. Like with many concepts in the tech world (microservices, cloud computing, Agile project management) this fairly narrow interview process has evolved and largely caught on, and being familiar with it is a shibboleth that’s as important as anything else in a job search, in the early-2020s tech world anyway.