Sign in Subscribe

Topic

Statistics

A collection of 112 issues

Learning From the Future with Fixed Effects

Say that you want to predict wait times at restaurants using data with four columns: wait times (wait), the restaurant name (restaurant), time and date of observation. Using the time and date of the observation, you create two additional columns: time of the day (tod) and day of the week

Rehabilitating Forward Stepwise Regression

Forward Stepwise Regression (FSR) is hardly used today. That is mostly because regularization is a better way to think about variable selection. But part of the reason for its disuse is that FSR is a greedy optimization strategy with unstable paths. Jigger the data a little and the search paths,

Faites Attention! Dealing with Inattentive and Insincere Respondents in Experiments

Respondents who don’t pay attention or respond insincerely are in vogue (see the second half of the note). But how do you deal with such respondents in an experiment? To set the context, a toy example. Say that you are running an experiment. And say that 10% of the

Maximal Persuasion

Say that you want to persuade a group of people to go out and vote. You can reach people by phone, mail, f2f, or email. And the cost of reaching out f2f > phone > mail > email. Your objective is to convert as many people as possible. How would

The Value of Bad Models

This is not a note about George Box’s quote about models. Neither is it about explainability. The first is trite. And the second is a mug’s game. Imagine the following: you get hundreds of emails a day, and someone must manually sort which emails are urgent and which

Snakes on Ladders: Encouraging People to Climb the Engagement Ladder

Marketers love engagement ladders. To increase engagement with a product, many companies segment their users based on usage, for instance, into heavy (super), medium (average), and light, and prod their users to climb the ladder by suggesting they do things that people in the segment above them are doing and

Wanted: Effects That Support My Hypothesis

Do survey respondents account for the hypothesis that they think people fielding the survey have when they respond? The answer, according to Mummolo and Peterson, is not much. Their paper also very likely provides the reason why—people don’t pay much attention. Figure 3 provides data on manipulation checks—

Interview with InfoQ

I recently gave an interview to InfoQ about my paper (and associated open source software) on predicting the race and ethnicity of a person using the sequence of characters in a name. Here a relevant excerpt: InfoQ: Can you discuss how we can learn from names? What ML/DL algorithms

Estimating the Trend at a Point in a Noisy Time Series

Trends in time series are valuable. If the cost of a product rises suddenly, it likely indicates a sudden shortfall in supply or a sudden rise in demand. If the cost of claims filed by a patient rises sharply, it plausibly suggests rapidly worsening health. But how do we estimate

What Clicks With the Users? Maximizing CTR

Given a pool of messages, how can you maximize CTR? The problem of maximizing CTR reduces to the problem of estimating the probability that a person in a specific context will click on each of the messages. Once you have the probabilities, all you need to do is apply the