101 Statistical Ideas I wish I learned Sooner

I was in my graduate program when I finally learned/internalized what a “statistic” actually was. I have later asked the question “what is a statistic?” to many other very well educated peers who have some experience with statistics, often through AP statistics, and rarely do they know the actual answer. What better way to start a list about statistics concepts I wish I learned earlier than a question about the eponymous “statistic”.

The definition is simple, given some set of data, a statistic is simply some function on this set of data. The mean takes a set of data and computes the arithmetic mean over it, that makes it a statistic, but technically, a function that takes in data always outputs “42” is also a statistic, though not a very helpful one. THe real trick of statistics, is figuring out what statistics are actually helpful, and how we can use them to come to conclusions about the world.

Statistics and probability can often sound very similar, especially when they share a lot of verbiage. However, the best analogy I can use is that math is to science and probability is to statistics.

Probability and statistics both deal with data and some random process. Probability is a field of mathematics, where we deal in the abstract, and our we clearly define how our random process functions. For example, a coin flip is a random process, and in probability we assign a certain probability of heads and tails and use that to answer questions about “what is the proobability that if I flip this coin $200$ times, it will land on heads more than $150$ times.” This is a probability question, because we know the random process, and our goal is to learn something about the data generated by this process.

In statistics, like science, we take measurements and make observations, and we very rarely have any idea about the random processes that give rise to these observations. In statistics the data, originating from the latin “datum” meaning “to give” is immutable, it is given. We use the data and observations to try and learn about the random processes that might have given rise to the observed data. So while in probability, one might ask the probability of flipping $5$ out of $10$ heads on a fair coin, a statistician might observe $5$ heads out of $10$ flips, and wonder how likely is it that the coin is fair?

The short version is that statistics uses known data to gain information about unknown random processes; probability uses known random processes to give us information about unknown data.

Because we insist on teaching statistics in such an applied way, the nuiance of probability theory is typically thrown out of the window, leading to immense confusion later on. You have likely heard about random variables, and fairly assumed that you should try to treat them like any other variable. However, a random variable is more like a function than it is a variable.

In probability, there is a set that contains all possible outcomes, called the sample space denoted "$/Omega$". Every possible event is within this set, and a random variable is a mapping, from events in this set, to some other set of real numbers. Say you flip a coin twice, the sample space would be \Omega = \{HH, HT, TH, TT}, if we are concerned with the number of heads flipped, we might define some random variable $X$, that maps the events in this sample space to the real number corresponding to the number of heads flipped like so:

HH -> 2 HT -> 1 TH -> 1 TT -> 0

The “Random Variable” $X$, represents this mapping or function from the sample space to some numerical space. You can of course chain functions together, so something like $(X+1)^2$ can of course still be used, and would represent the following mapping:

HH -> 9 HT -> 4 TH -> 4 TT -> 1