The NASA Technique That Changed How I Hire Executives

How a space program decision-maker's framework helps you assess expertise in any domain, even ones you don't understand

Hi there, it’s Adam. 🤗 Welcome to my weekly newsletter. I started this newsletter to provide a no-bullshit, guided approach to solving some of the hardest problems for people and companies. That includes Growth, Product, and company building. Subscribe and never miss an issue. If you’re a parent or parenting-curious I’ve got a Webby Award Honoree podcast on fatherhood and startups - check out Startup Dad.

---

Questions you’d like to see me answer? Ask them here.

---

About 18 months ago I had the pleasure of attending a dinner for a select group of growth leaders in enterprise companies. Attendees were from companies like OpenAI, Atlassian, Postman, Glean, and Workato amongst others. The conversation topic turned to hiring executives at many of these fast-growing companies. That’s when I met Sarah Charlton. She had a specific point of view and a framework for hiring executives at “startups” (broadly defined as any fast-growing, scrappy company - including hers at the time, Asana). She expertly captured and articulated the essence of what I had experienced as an interviewer in the executive hiring process. I was so intrigued by her style and this no-bullshit approach to interviewing that I invited her to write a guest post for this newsletter.

Sarah Charlton is currently an AI company growth advisor. She was previously the Chief Business Officer for a generative video AI company, GM of SMB and Self Serve at Asana, Director of Marketing at Dropbox, has been a GTM advisor to several startups and has led strategy teams around the world.

Today’s newsletter is for founders, executives and all interviewers who are hiring outside of their primary area of expertise.

Sarah covers:

• How she developed her framework with an assist from a NASA leader

• A step-by-step guide to the Depth Method for interviewing executives

• Real examples of this approach applied to her interview experience

• Interviewing pitfalls and areas to watch out for

---

Becoming a CEO, department head, or a manager of managers is a tough transition because it’s typically the first time you’re running functions that you aren’t excellent in yourself.

This is triply true for founders, who are often in their most senior role ever, running the largest team they ever have, and face large systemic consequences for hiring and building entire functions badly. But it also applies to anyone making this jump. You’ve been promoted because you crushed it in your domain. Now suddenly you’re also responsible for sales when you trained as a marketer (a transition I’ve had to make!), or operations when you’re a product person (editor: been there).

The most overwhelming step is hiring someone to do something you probably couldn’t do even with 20-hour days and some killer AI agents. How do you identify people who have the skills you lack and can get the results you want when you can’t even evaluate their expertise?

This is where most people get burned. Experienced executives are really good at interviewing but might actually be really bad at the job you need them to do. They excel at presentation and storytelling, but may not excel at building. They’ll seem unflappable and all-knowing. They’re impressive and inspiring. At the same time, they might not know the limitations of their own knowledge, where they need support to be successful, or whether they’re a good fit for your company and role.

Most interviewers don’t know how to uncover this because they don’t go particularly deep. A talented storyteller can weave and dodge their way through an interview and the interviewer walks away without a deep sense of whether they can roll up their sleeves and do what is required.

I’ve found a solution to this problem that I learned from an unlikely source: NASA. I call it the Depth Method.

I developed the Depth Method after listening to an interview with Dr. Thomas Zurbuchen, the longest running Associate Administrator for NASA’s Science Mission Directorate. His job was to oversee the launch of NASA missions and make a “go” or “no-go” decision to bring a mission to the launch pad. If a launch goes way off budget and off track he’s the guy who got hauled in front of Congress for questioning (see: James Webb Space Telescope budget overruns 😬). An important job with big technical scope and real-world consequences.

Dr. Zurbuchen had to evaluate whether a mission was ready and whether he could trust what people were saying about their projects, but it wasn’t possible to be an expert in every relevant technical domain. His elegant solution: create a “secret” checklist of the key areas that needed to be rock solid, then listen to mission teams discuss their projects while specifically noting areas they weren’t covering from his checklist. He would ask questions in those gaps and continuously drill down question after question in those areas to determine if they really knew what was going on in their missions and had surfaced and addressed the most critical problems.

When I listened to this podcast, I realized he wasn’t just sharing a good project management technique. Zurbuchen had developed a systematic framework for gathering reliable information in an environment without complete knowledge or deep personal trust relationships. I saw that this framework could help navigate any interpersonal uncertainty and decision-making so I began testing and using it in my day-to-day work.

The most surprising place it proved useful, and where it’s now my go-to approach, is interviewing and hiring senior leaders and executives.

Step-by-Step Guide to the Depth Method of Interviewing

There are three phases to this interview method: Map the Territory (pre-interview prep), Dive Below the Surface (interview), and Surface (evaluate).

Step 1: Map the Territory

If you’re completely naive to a function (engineers hiring for marketing is the most common pain point I hear about from founders), you have to start by acknowledging you don’t know the domain expertise areas for this function, but you do know something about it.

You can start with skill maps common to all executives: are they good speakers who can rep your company (Communications), do they seem like good team adds (Culture), can they manage people well (Management). These areas are critical in every function, though the expectations for each role are different.

But for functional expertise, your first step is to gain an understanding of the function. I like to talk to people who have “graduated” out of it, because they tend to have a higher level and more philosophical understanding of it. A former CMO who is now COO, a former sales AE who now runs all of enterprise, a former customer success leader who is now CCO. Friends who do the job and are not trying to impress you are also good resources as are VC talent partners.

Create a checklist that maps the different areas that someone has to be great at in this particular role.

If you’re interviewing for a Growth leader, your checklist might include

• Growth Loops

• Forecasting

• Experimentation

• User Understanding

• Data Strengths & Limitations

[Editor’s note: I wrote about the Growth Competency model here].

While If you’re interviewing a Product executive, you can include:

• Data Fluency

• Product Strategy

• Product Vision & Roadmapping

• Delivering Business Outcomes

• Customer Insight Development

• Product Execution (specification, delivery, quality)

• Influencing

To support the creation of the checklist you’re going to want to ensure that the entire interview team is collaborating on its development. Because executives need to excel across multiple competencies, each individual interviewer should have their own set of ~3-5 areas that they’re going to go deep on.

[Editor’s note: For an in-depth guide to interview preparation you can read this].

Step 2: Go Below the Surface

The key to conducting the interview is getting below the surface. As the name implies you might only get a surface-level view of the executive unless you dig deeper. To do this you need to continue digging down with each subsequent question.

Let’s say that you’re interviewing a Head of Growth and one of the areas you want to explore is building growth models. You might start with a question like:

“Tell me about your experience building a new growth model at [company X].”

Then they might say something like:

“Well, it’s really important that you start with a solid foundation of data.”

You’d follow-up with:

“Oh, that’s great – how do you go about establishing that foundation?”

They’d say:

“We had to instrument our entire product and send that data into our data warehouse in addition to the product analytics tool.”

You’d follow up with:

“Ahh, getting instrumentation right can be really challenging. How did you do it? Which analytics tool did you choose, why did you choose it?”

…and so on.

The key is to keep drilling deeper with more specific questions until you reach a place where the candidate stumbles or can’t go any deeper.

Alternatively, you might also find that a candidate will launch into a detailed explanation from the first question. Especially in a long answer, listen carefully to what they’re saying and not saying. For example, if they leave out the process of instrumentation, you can choose to ignore the rest of their answer and dig into this area after they finish their initial explanation. Be conversational, but don’t let conversational distraction dissuade you from what your checklist requires you to evaluate.

What you’re looking for here is twofold:

1. Knowledge boundaries

2. Expertise signals

Knowledge boundaries are where the sparkle starts to come off the diamond. Through your continued, specific questions you run into the edge of their expertise. Here you’re evaluating not just their knowledge but also honesty about their own limitations. Do they acknowledge their limit, and if so do they act defensive? What happens at this critical point will show you how they perform at any knowledge limit in the future.

Expertise signals are areas where they light up and are able to continue elaborating for days. As Dr. Zurbuchen says,

“You know, there’s some people that have never missed. I’ve never got to the end of my questioning. There’s some people that were two questions deep and I got them. And it’s like, OK, they are a two question kind of person. You know, there’s the other one that I’ve never found the end. After six questions, he’s still going. And I’m like, OK, don’t worry about him.”

This might look something like:

You: “Ahh, getting instrumentation right can be really challenging. How did you do it?”

Them: “It was challenging, but we figured it out. First we spec’d out all of the events we wanted to fire based on the critical paths through the product. We built it out in a giant table and specified the syntax for the event call so we could make it super easy and templatized for our engineers…”

You’ll notice in the above that this person clearly knows what they’re talking about and has done the actual work. First, they’re excited to talk about it and second, they can articulate with great detail the process they went through. And with every follow up, you get more information and depth. After a few more questions you can just stop: they’ve got what you need in that area.

A Real Example: Customer Success Leader

I was once on the interview panel for a Customer Success exec. I was the GM and Head of Growth for the self-service and SMB customers. Given my role, my checklist included:

• Management: standard in any functional leader

• Customer love: obviously expected in this role, I wanted to see someone who knew and named their customers, their roles, and pain points

• White-glove large customer management: what’s their strategy and techniques for handling the whales?

• Long tail, small dollar value customer management: did they also have strategies and plans for handling my book of more than 100k customers?

• Technical stack knowledge: did they know what systems and tools they and their teams needed to be successful

In one interview, I asked a candidate how they handled the long tail. Here’s the exchange:

Me: “How do you handle long-tail customers, making sure they’re successful without spending too much money?”

Them: “It’s all about A/B testing.”

Me: Oooh, I’m a growth person, this is catnip for me… “I love A/B testing. Tell me about a specific A/B test you ran and what you’ve learned about small customers?”

Them: “We ran some trials on email that showed it helped change behavior at scale.”

Me: pushing another level deeper… “Wow, I love email. It’s always been really mixed for me. Tell me about an email test you ran that was really impactful?”

Them: Long pause… followed by a very small experiment from ~4 years prior.

What I uncovered was that this candidate knew that A/B testing was an important thing to do, they knew it worked at scale… but they actually hadn’t done it. Maybe they’d overseen a team doing it, but clearly without paying that much attention to what was going on. In this particular area they would need a lot of help.

It happens that I know a lot about A/B testing given my background in growth so this area was easy to dig into. Let’s take an area I knew less about but was equally important: white-glove customer management. Make sure to probe especially in areas where you don’t know the answer: digging deeper not only assesses the candidate, it also gives you an opportunity to learn something meaningful and tangible from the conversation.

Here’s how that line of questioning went with another candidate:

Me: “Beyond our huge book of small customers we have some very large customers that are very important to us. What does it look like to do a great job with customers like this?”

Them: “You need to build the support they require to be super successful into the package so that they pay for it at the beginning of the contract, and you can afford to provide them with the higher-level of engagement they need to be successful and happy with you as a vendor.”

Me: “That’s really interesting; how do you make sure they’re willing to pay enough?”

Them: “Well, most enterprises expect there to be onboarding and installation fees. But sometimes they aren’t willing to pay the whole value of the package. In this case, you can take some of the charges you impose on mid-market customers to help subsidize operations at larger customers. But you can also…”

Me: “What happens at the sale if they’re not willing to pay for support but you know that they actually need it?”

Them: (Super detailed answer about what to do in this situation and how to make it work.)

So here I am not knowing much about implementation of a white glove customer engagement program but I’m following my own curiosity and staying on that specific topic. I kept going deeper and deeper, down to as many levels as possible. At the end of that part of the interview I had learned something about how to run a white-glove team with big name customers. After three of four of these, I had a good sense of the top candidates even in this area where I had less experience, and I had learned an incredible amount about building a great white-glove customer success organization.

The best part about this approach is that even though the areas you’re probing are things someone should know to do the job well, the specific line of questioning looks different every time. They can’t rehearse for it.

You need at least 45-60 minutes for this method, because otherwise you can’t cover your entire checklist while going really deep in a few areas. There will be pressure to do shorter interviews for candidate experience. Push back on this. Hiring an executive is one of the most important things you can do and also most fraught with false positives. If ~50% of exec hiring fails you owe it to yourself to spend the time up front.

Got an interview horror story or success story? Let us know below.

Step 3: Surface

After the interview, you need to do three things: assess how deep their knowledge actually goes, identify any gaps where they’ll need support, and compare notes with other interviewers.

During and right after your interview, you can note how deep the candidate could go in each of your critical areas. Make sure you leverage your actual conversation content, not their resume and supposed experience here.

Remember that the goal isn’t to find someone who knows everything, but to identify their gaps where they’ll need strong support and to have a sense of how they react when coming up against their own limitations. A sales leader might be brilliant at hiring and training but weak on sales operations. If they are aware of it and can work around it, there’s no problem. If they’re sensitive or arrogant around their limitations this will be a harder person to take on.

Most importantly, compare insights across your interview panel. Other interviewers might have been impressed by the candidate’s presence and missed the shallow spots you uncovered. They may have gone deep and gotten signals in areas that conflict with your insights. Compare the checklist, share specifics, and then make a call.

Common Pitfalls and Areas to Watch Out For

There are four primary traps you can fall into when interviewing executive hires:

1. Being swayed by (over)confident delivery or resume

2. Conversation redirects

3. Time

4. Preparation

Avoid being swayed by (over)confident delivery

You can’t let someone’s smooth delivery or paper resume trick you into thinking that they know something. Confidence is unfortunately largely uncoupled from actually knowing things at the executive level. Leveraging the Depth Method helps you push through this because you’re increasing the required specificity and depth until you have a personal assessment of their knowledge and capability boundaries.

Being aware of executives who redirect the conversation

If you come across an interviewing pro - especially true of people who have done thorough press training - they can try to turn the interview questions around to you when they hit their limits. It’s a jujitsu move: you ask something and they’ll flip it and say “Yeah, that is really important, I’m fascinated by [you, your company’s] approach to that?”

Watch out for this; it’s not a huge red flag, but you’re letting them dictate the terms of the conversation and you’re losing the thread of what you’re trying to assess and learn. Stay friendly and interested, but save your opinions and the conversational back-and-forth for after you’ve finished assessing capabilities.

Giving yourself enough time

As I mentioned above, you need more than a standard 30-minute conversation to run this interview method. Without at least 45-60 minutes you won’t have time to go really deep and cover the 3-5 competencies that you need to get to. But sometimes you’ll hear arguments for shorter interviews for the sake of candidate experience. Too bad. Push back on this when it’s a senior critical hire.

Preparation

You really have to thoughtfully create your checklist and interview plan. Winging it leads to the common strategic pitfall I call “SQUIRREL!”

Have you ever taken a dog for a walk, and then a squirrel crosses your path? If the dog’s on a leash you’re often ok after a brief struggle, but off-leash the dog is gone chasing the squirrel up trees and around the park a long time, while you’re left standing there wondering where things went wrong.

Prepare for squirrels by organizing your checklist and interview plan so you can stay on track and avoid exciting distractions.

Final Thoughts

The Depth Method has become my default approach for any senior hire, but it’s particularly crucial when you’re scaling into new functions. I’ve used it to hire growth leaders, partnerships execs, and even technical roles where I needed to evaluate architectural thinking despite not being an engineer myself.

What I’ve learned is that the best leaders actually enjoy this process. They light up when you ask them to go deep on something they genuinely understand and love. The ones who get defensive or try to redirect? That tells you something important about how they’ll be on the job as well.

The method also scales your own learning as a leader. Every time I’ve used it, I’ve walked away understanding the function better than before. That customer success interview where I learned about white-glove operations? That knowledge directly improved how we later structured our enterprise support packages six months later.

The most surprising aspect of this approach is it has made me better at assessing expertise across the board. Once you default to wide-ranging questions with strategic deep-diving, you start gaining insights in every conversation, from vendor pitches to board meetings. You develop an instinct for when someone actually knows their stuff versus when they’re performing expertise, and you can choose your own reactions accordingly.

The stakes are too high to wing executive hiring. A bad senior hire doesn’t just fail: they take six months to fail while burning cash and potentially damaging your team’s confidence in both you and the company. The Depth Method gives you a systematic way to cut through the presentation skills and get to the substance.

Most importantly, it gives you confidence to hire outside your domain expertise. You don’t need to become a marketing expert to hire a great CMO. You just need to map the territory, go deep, and trust the process.