Unbiased Hiring

10 Sep

When hiring for technical positions, people often use high-precision heuristics conditional on relevance to decide who to interview. Otherwise, the interviewing burden would be too much. The advent of GPT makes use of high-precision heuristics yet more important as interviewing and resumes are less trustworthy now. The heuristics people often rely on are ~ network, fancy pants school and company, and international student (for junior applicants). But this likely leads to bias against certain groups, etc. There are at least three potential solutions that improve upon the status quo:

  1. Use test scores from exams that are widely taken that are correlated with performance (after confirming the relationship), e.g., IQ, SAT, GRE, and use that to filter.
  2. Use testing companies to offer proctored exams in topical areas. Companies like Karat are well placed to offer such testing. And working with companies like Coursera, which have strong incentives to make their certifications meaningful, may be a good idea. The larger idea has data behind it. Advanced software certifications offered by major companies are widely viewed as good signals of competence.
  3. Create a paid show-me-your-work period. The solution is the least attractive as it is expensive for the company and for most applicants (except for fresh graduates or unemployed people).

p.s. Many of the job postings are too generically worded and plausibly cause more unqualified people to apply. Making job descriptions more precise, e.g., “proficiency in pandas,” may help.

Why do We Fail? And What to do About It?

28 May

I recently read Gawande’s The Checklist Manifesto. (You can read my review of the book here and my notes on the book here.) The book made me think harder about failure and how to prevent it. Here’s a result of that thinking.

We fail because we don’t know or because we don’t execute on what we know (Gorovitz and MacIntyre). Of the things that we don’t know are things that no else knows either—they are beyond humanity’s reach for now. Ignore those for now. This leaves us with things that “we” know but the practitioner doesn’t.

Practitioners do not know because the education system has failed them, because they don’t care to learn, or because the production of new knowledge outpaces their capacity to learn. Given that, you can reduce ignorance by 1) increase the length of training, b) improving the quality of training, c) setting up continued education, d) incentivizing knowledge acquisition, e) reducing the burden of how much to know by creating specializations, etc. On creating specialties, Gawande has a great example: “there are pediatric anesthesiologists, cardiac anesthesiologists, obstetric anesthesiologists, neurosurgical anesthesiologists, …”

Ignorance, however, ought not to damn the practitioner to error. If you know that you don’t know, you can learn. Ignorance, thus, is not a sufficient condition for failure. But ignorance of ignorance is. To fix overconfidence, leading people through provocative, personalized examples may prove useful.

Ignorance and ignorance about ignorance are but two of the three reasons for why we fail. We also fail because we don’t execute on what we know. Practitioners fail to apply what they know because they are distracted, lazy, have limited attention and memory, etc. To solve these issues, we can a) reduce distractions, b) provide memory aids, c) automate tasks, d) train people on the importance of thoroughness, e) incentivize thoroughness, etc.

Checklists are one way to work toward two inter-related aims: educating people about the necessary steps needed to make a decision and aiding memory. But awareness of steps is not enough. To incentivize people to follow the steps, you need to develop processes to hold people accountable. Audits are one way to do that. Meetings set up at appropriate times during which people go through the list is another way.

Growth Funnels

24 Sep

You spend a ton of time building a product to solve a particular problem. You launch. Then, either the kinds of people whose problem you are solving never arrive or they arrive and then leave. You want to know why because if you know the reason, you can work to address the underlying causes.

Often, however, businesses only have access to observational data. And since we can’t answer the why with observational data, people have swapped the why with the where. Answering the where can give us a window into the why (though only a window). And that can be useful.

The traditional way of posing ‘where’ is called a funnel. Conventionally, funnels start at when the customer arrives on the website. We will forgo convention and start at the top.

There are only three things you should work to optimally do conditional on the product you have built:

  1. Help people discover your product
  2. Effectively convey the relevant value of the product to those who have discovered your product
  3. Help people effectively use the product

p.s. When the product changes, you have to do all three all over again.

One way funnels can potentially help is triage. How big is the ‘leak’ at each ‘stage’? The funnel on the top of the first two steps is: of the people who discovered the product, how many did we successfully communicate the relevant value of the product to? Posing the problem in such a way makes it seem more powerful than it is. To come up with a number, you generally only have noisy behavioral signals. For instance, the proportion of people who visit the site who sign up. But low proportions could be of large denominators—lots of people are visiting the site but the product is not relevant for most of them. (If bringing people to the site costs nothing, there is nothing to do.) Or it could be because you have a super kludgy sign-up process. You could drill down to try to get at such clues, but the number of potential locations for drilling remains large.

That brings us to the next point. Macro-triaging is useful but only for resource allocation kinds of decisions based on some assumptions. It doesn’t provide a way to get concrete answers to real issues. For concrete answers, you need funnels around concrete workflows. For instance, for a referral ‘product’ for AirBnB, one way to define steps (based on Gustaf Alstromer) is as follows:

  1. how many people saw the link asking people to refer
  2. of the people who saw the link, how many clicked on it
  3. of the people who clicked on it, how many invited people
  4. of the people who were invited, how many signed up (as a user, guest, host)
  5. of the people who signed up, how many made the first booking

Such a funnel allows you to optimize the workflow by experimenting with the user interface at each stage. It allows you to analyze what the users were trying to do but failed to do, or took a long time doing. It also allows you to analyze how optimally (from a company’s perspective) are users taking an action. For instance, people may want to invite all their friends but the UI doesn’t have a convenient way to import contacts from email.

Sometimes just writing out the steps in a workflow can be useful. It allows people to think if some steps are actually needed. For instance, is signing-in needed before we allow a user to understand the value of the product?

Technical Leadership: Building Great Teams and Products

31 Aug

Companies bet on people. They bet that the people they have hired will figure out the problems and their solutions. To make the bet credible, the golden rule is to hire competent people. Other virtues matter, but competence generally matters the most in technical roles.

Once you have made the bet, you must double down on it. Trust the people you hire. And by being open, empathetic, transparent, thoughtful, self-aware, good listeners, and excellent at our job, earn their trust. Once people are sure that you are there to enable them and grow them, increasing their productivity and growing them becomes a lot easier.  

After laying the foundation of trust, you need to build culture. The important aspects of culture are: 

  1. inclusion
  2. intellectual openness 
  3. technical excellence 
  4. focus on the business problem
  5. interest in doing the right thing
  6. accountability
  7. hard work  

To build a culture of technical excellence and intellectual openness, I use the following techniques:

  • Probe everything. Ask why. Ask why again.
  • Ask people to explain things as simply as possible.
  • For anything complex, ask people to write things down in plain English.
  • Get perspective from people with different technical competencies.
  • Frontload your thinking on a project.
  • Lead a meeting to ‘think together’ about projects where tough, thoughtful, probing is encouraged.
  • Establish guidelines for coding excellence and peer review each major PR. 
  • Lead a meeting where different members of the team teach each other — I call it a ‘learn together.’

To make sure that the team has a place to share and reflect on the understanding, I also work hard to build a shared understanding. Building a shared understanding of the space helps drive clarity about actions. It also provides a jumping-off point for planning. Without knowing what you know, you start each planning cycle like it is groundhog day. 

I actively use Notion to think through the broader problem and the sub-problems and spend time organizing and grooming it. Build something that inspires others to build that shared understanding.

I encourage active discussion and shared ownership. We own each page as a team, if not as a company. But eventually, one person is listed as a primary point of contact who takes the responsibility of keeping it updated.

To drive accountability, I drive transparency. For that, I use the following methods:

  1. Share folders. Shared team Google drive folder with project-level folders nested within it. I dump most documents that are sent to me in the shared folder and organize the folder periodically. There is even a personal growth folder where I store things like ‘how to write an effective review,’ etc.
  2. Be a Good Router: I encourage people to err toward spamming. I catch whatever failures by 1. forwarding relevant emails, 2. Posting attachments to Google Drive, and 3. posting to the relevant section on Notion.
  3. Visibility on intermediate outputs: I encourage everyone to post intermediate outputs, questions, concerns, etc. to the team channel, and encourage people to share their final outputs widely.
  4. Use PPP: Each member of the team shares a Progress, Plans, and Problems each week along with links to outputs. 

Culture allows you to produce great things if you have the right process and product focus. There are some principles of product and process management that I follow. My tenets of product management are: 

  • Formalize the business problem 
  • Define the API before moving too far out.
  • Scope and execute on a V1 than before executing on the full solution. Give yourself the opportunity to learn from the world.
  • Do less. Do well.

The process for product development that I ask the team to follow can be split into planning, execution, and post-execution.

Planning: What to Work On?

  1. Try to understand the question and the use case as best as you can. Get as much detail as possible.
  2. Talk to at least one more person on the team and ask them to convince you why you shouldn’t do this followed by if you are doing this, what are the other things I am missing here.
  3. For larger projects, write a document and get feedback from all the stakeholders.
  4. As needed, hold a think together with a diverse, cross-functional group. Keep the group small enough. Generally, I have found ~5—8 is the sweet spot.
  5. Get an ROI estimate and rank against other ideas.

How to execute?

  1. Own your project on whatever project management software you use.
  2. Own communication on where things are on the project, new discoveries, new challenges, etc.
  3. Ask people to review each commit. Treat reviewing seriously. Generally, the review process that works well is people explaining each line to another person. In the act of explaining, you catch your hidden assumptions, etc.
  4. Come up with at least a few sanity checks for your code.

Communicating Within and Outside What You Learn

  1. Share discoveries aggressively. Err on the side of over-communication.
  2. Contribute to Notion so that there is always a one-stop-shop around how we understand X. 
  3. Create effective presentations

(Software) Product Development Cycle

6 Mar
  1. Solicit Ideas
    Talk to customers, analyze usage data, talk to peers, managers, think, hold competitions, raffles,…
    For each idea:
  2. Define the idea
    Write out the idea for clarity — at least 5–10 sentences. Do some due diligence to see what else is there. Learn and revise (including abandon).
  3. Does the idea make business sense? Does it make sense to the developers (if the idea is mechanistic or implementation related)?
    Ideally, peer review. But at the minimum: Talk about your idea with the manager(s) and the developers. If both manager(s) and developer(s) agree (for some mechanistic things, only developers need to agree), move to the next step.
  4. Prototype
    This is optional, and for major, complex innovations that can be easily prototyped only. Write code. Produce Results. Does it make sense? Peer review, if needed. If not, abandon. If it does, move to the next step
  5. Write the specifications
    Spend no less than 20% of the entire development time on writing the specs, including proposed functions, options, unit tests, concerns, implications. Run the specifications by developers, get this peer reviewed, improve, and finalize.
  6. Set Priority and Release Target
    Talk to the manager about the priority order of the change, and assign it to a particular release cycle.

  7. Who Does What by When?
    Create JIRA ticket(s)
  8. Code
    General cycle = code, test -> peer review -> code, test -> peer review …
    MVP of Expected Code or Aspects of good code: ‘Code your documentation’ (well-documented), modular, organized, tested, nice style, profiled. Do it once, do it well.