Sign in Subscribe

Topic

Statistics

A collection of 112 issues

ML (O)Ops! Improving and Deploying On-Device Models With Confidence (Part 1)

With Atul Dhingra. Part 1 of a multi-part series. It is well known that ML Engineers today spend most of their time doing things that do not have a lot to do with machine learning. They spend time working on technically unsophisticated but important things like deployment of models, keeping

Build Software for the Lay User

Most word processing software helpfully point out grammatical errors and spelling mistakes. Some even autocorrect. And some, like Grammarly, even give style advice. Now consider software used for business statistics. Say you want to compute the correlation between two vectors: [100, 2000, 300, 400, 500, 600] and [1, 2, 3,

Superhuman: Can ML Beat Human-Level Performance in Supervised Models?

A supervised model cannot do better than its labels. (I revisit this point later.) So the trick is to make labels as good as you can. The errors in labels stem from three sources: 1. Lack of Effort: More effort people spend labeling something, presumably the more accurate it will

Too Much Churn: Estimating Customer Churn

A new paper uses financial transaction data to estimate customer churn in consumer-facing companies. The paper defines churn as follows: There are three concerns with the definition: 1. The definition doesn’t make clear what is the normalizing constant for calculating the share. Given that the value “can vary between

94.5% Certain That Covid Vaccine Will Be Less Than 94.5% Effective

“On Sunday, an independent monitoring board broke the code to examine 95 infections that were recorded starting two weeks after volunteers’ second dose — and discovered all but five illnesses occurred in participants who got the placebo.” Moderna Says Its COVID-19 Vaccine Is 94.5% Effective In Early Tests The data

Fat Or Not: Toward 'Proper Training of DL Models'

A new paper introduces a DL model to enable ‘computer aided diagnosis of obesity.’ Some concerns: 1. Better baselines: BMI is easy to calculate and it would be useful to compare the results to BMI. 2. Incorrect statement: The authors write: “the data partition in all the image sets are

Not So Robust: The Limitations of "Doubly Robust" ATE Estimators

Doubly Robust (DR) estimators of ATE are all the rage. One popular DR estimator is Robins’ Augmented IPW (AIPW). The reason why Robins’ AIPW estimator is called doubly robust is that if either your IPW model or your y ~ x model is correctly specified, you get ATE. Great! Calling something

Instrumental Music: When It Rains, It Pours

In a new paper, Jon Mellon reviews 185 papers that use weather as an instrument and finds that researchers have linked 137 variables to weather. You can read it as each paper needing to contend with 136 violations of the exclusion restriction, but the situation is likely less dire. For

Unmatched: The Problem With Comparing Matching Methods

In many matching papers, the key claim proceeds as follows: our matching method is better than others because on this set of contrived data, treatment effect estimates are closest to those from the ‘gold standard’ (experimental evidence). Let’s side-step concerns related to an important point: evidence that a method

Optimal Recruitment For Experiments: Using Pair-Wise Matching Distance to Guide Recruitment

Pairwise matching before randomization reduces s.e. (see here, for instance). Generally, the strategy is used to create balanced control and treatment groups from available observations. But we can use the insight for optimal sample recruitment especially in cases where we have a large panel of respondents with baseline data,