Platform to reduce low-effort conference submissions and reviews

ICLR reviews are out today. I’ve seen both authors and reviewers, including myself, frustrated by the poor quality of certain papers and reviews. Here are my thoughts. As people often say, 𝗛𝗮𝗽𝗽𝗶𝗻𝗲𝘀𝘀 = 𝗥𝗲𝗮𝗹𝗶𝘁𝘆 / 𝗘𝘅𝗽𝗲𝗰𝘁𝗮𝘁𝗶𝗼𝗻𝘀. I think much of the frustration comes from our expectations about the distribution of paper and review quality. Paper quality does 𝗡𝗢𝗧 follow a normal distribution, while we often aim to assign scores that do. The lower bound extends a lot farther from the mean than the upper bound. The same applies to review qualify. But because we expect a normal distribution, we also expect that we should encounter low-quality papers or reviews only rarely, which is not the reality. This mistaken assumption of normal distributions appears in many areas of life, as I learned from Taleb’s The Black Swan recently. My suggestion: we should recalibrate our expectations. Celebrate the completion of a 𝘀𝗼𝗹𝗶𝗱 project, rather than its recognition by a few peers. The true milestones in science are: (1) discovering or inventing something, and (2) having those findings make an impact to the community or society by being adopted by others. Paper reviews and acceptance are usually just steps along the way, though good feedback can certainly help refine our work. At the same time, our community needs better mechanisms to reduce low-effort submissions and reviews, so we can bring back some joy to the conference cycle.