The Unofficial Google Data Science Blog

 by MICHAEL QUINN, JEREMY MILES, KA WONG As data scientists, we often encounter situations in which human judgment provides the ground truth. But humans often disagree, and groups of humans may disagree with each other systematically (say, experts versus laypeople). E ven after we account for disagreement,  human ratings may not measure exactly what we want to measure. How do we think about the quality of human ratings, and how do we quantify our understanding is the subject of this post. Overview Human-labeled data is ubiquitous in business and science, and platforms for obtaining data from people have become increasingly common. Considering this, it is important for data scientists to be able to assess the quality of the data generated by these systems: human judgements are noisy and are often applied to questions where answers might be subjective or rely on contextual knowledge. This post describes a generic framework for understanding the quality of human-labeled data, based arou

 by AMIR NAJMI & MUKUND SUNDARARAJAN Data science is about decision making under uncertainty. Some of that uncertainty is the result of statistical inference, i.e., using a finite sample of observations for estimation. But there are other kinds of uncertainty, at least as important, that are not statistical in nature. This blog post introduces the notions of representational uncertainty and interventional uncertainty to paint a fuller picture of what the practicing data scientist is up against. Data science and uncertainty Data Science (DS) deals with data-driven decision making under uncertainty . The decisions themselves may range from "how much data center capacity should we build for two years hence?" or "does this product change benefit users?" to the very granular "what content should we recommend to this user at this moment?" This kind of decision making must address particular kinds of uncertainty. Wrestling with uncertainty characterizes the