Zach and Jack* write:
What sort of papers best serve their readers? We can enumerate desirable characteristics: these papers should
(i) provide intuition to aid the reader’s understanding, but clearly distinguish it from stronger conclusions supported by evidence;
(ii) describe empirical investigations that consider and rule out alternative hypotheses [62];
(iii) make clear the relationship between theoretical analysis and intuitive or empirical claims [64]; and
(iv) use language to empower the reader, choosing terminology to avoid misleading or unproven connotations, collisions with other definitions, or conflation with other related but distinct concepts [56].
Recent progress in machine learning comes despite frequent departures from these ideals. In this paper, we focus on the following four patterns that appear to us to be trending in ML scholarship:
1. Failure to distinguish between explanation and speculation.
2. Failure to identify the sources of empirical gains, e.g. emphasizing unnecessary modifications to neural architectures when gains actually stem from hyper-parameter tuning.
3. Mathiness: the use of mathematics that obfuscates or impresses rather than clarifies, e.g. by confusing technical and non-technical concepts.
4. Misuse of language, e.g. by choosing terms of art with colloquial connotations or by overloading established technical terms.
Funnily Zach and Jack fail to take their own advice, forgetting to distinguish between anecdotal evidence (they claim a ‘troubling trend’ without presenting systematic evidence for it). But the points they make are compelling. The second and third points are especially applicable to economics though they apply to a lot of scientific production.
* It is Zachary and Jacob.