Probabilities from classification models can have two problems: 1. Miscalibration: A p of .9 often doesn’t mean a 90% chance of 1 (assuming a dichotomous y). (You can calibrate it using isotonic regression.) 2. Optimal cut-offs: For multi-class classifiers, we do not know what probability value will maximize the

Say that you train a model to predict who will click on an ad. Say that you deploy the model to only show ads to people who are likely to click on them. (For a discussion about the optimal strategy for whom to show ads to, see here.) And say

Let’s say that we want to measure the effect of a phone call encouraging people to register to vote on voting. Let’s define compliance as a person taking the call (like they do in Gerber and Green, 2000, etc.). And let’s assume that the compliance rate is

Say you want to measure how often people visit pornographic domains over some period. To measure that, you build a model to predict whether or not a domain hosts pornography. And let’s assume that for the chosen classification threshold, the False Positive rate (FP) is 10\% and the False

Ad targeting is often useful when you have multiple things to sell (opportunity cost), when the cost of running an ad is non-trivial, when an irrelevant ad reduces your ability to reach the user later, or any combination of the above. (For a more formal treatment, see below.) targetingtargeting.pdf133

Say that we want to find people to whom a product is relevant. One way to do that is to launch a small campaign advertising the product and learn from people who click on the ad, or better yet, learn from people who not just click on the ad but