Data Collection: How to Gather Evidence Without Tricking Yourself

You now know the language of statistics. You know how to identify variables, and you know how to classify your data using the NOIR framework.

But all of that knowledge is useless if you don’t have any actual numbers to work with. You have to go out into the real world and collect them.

There is a famous saying in data science: Garbage In, Garbage Out (GIGO). A statistical model is like a high-performance engine. If you pour mud into the gas tank, it doesn’t matter how well-engineered the engine is; it’s going to break. If your data collection methods are flawed, your mathematical conclusions will be completely wrong—even if you did the math perfectly.

Before you can run a single test, you need to know where your data comes from and how to avoid the hidden traps of bias.

Primary vs. Secondary Data

The first choice you have to make is whether you are going to collect the data yourself, or borrow it from someone else.

• Primary Data: Data that you collect firsthand for your specific question.
  • Example: You stand outside a grocery store and ask 100 people what their favorite fruit is.
  • Pros: You control the exact quality and formatting.
  • Cons: It takes a massive amount of time, effort, and money.
• Secondary Data: Data that already exists because someone else collected it for a different reason.
  • Example: You download a spreadsheet from the US Census Bureau to see the average income of your city.
  • Pros: It is fast, cheap, and often contains huge sample sizes.
  • Cons: You have to trust that the original researchers didn’t make mistakes.

The Three Main Methods of Data Collection

If you decide to gather Primary Data, you have three main tools at your disposal.

1. Surveys and Questionnaires

This is the most common method. You write a list of questions and send them out via email, phone, or in person.

• Best for: Gathering opinions, preferences, or demographic data quickly.
• The Danger (Response Bias): People lie. If you send out a survey asking, “Do you wash your hands every time you use the restroom?”, almost 100% of people will say yes. If you rely purely on self-reported surveys, your data will often reflect who people want to be, rather than who they actually are.

2. Observational Studies

Instead of asking people what they do, you simply watch them do it without interfering.

• Best for: Discovering true behavior in a natural setting.
• The Danger (The Hawthorne Effect): If you stand in a factory with a clipboard to measure how fast workers assemble a product, they will naturally work faster simply because they know they are being watched. True observation requires the observer to blend in seamlessly.

3. Controlled Experiments

This is the scientific gold standard. You create a controlled environment where you actively manipulate the Independent Variable while keeping everything else perfectly frozen (Control Variables).

• Best for: Proving absolute cause-and-effect.
• The Danger (Artificiality): Labs are not the real world. A mouse might behave one way in a sterile, temperature-controlled plastic maze, but behave completely differently in a messy, chaotic city alleyway.

The Ultimate Trap: Selection Bias

No matter which method you choose, you have to avoid the deadliest trap in statistics: Selection Bias.

Selection bias happens when the data you collect does not accurately represent the real world because of how you collected it.

Imagine you want to know what percentage of adults use the internet every day. You decide to run a poll on Twitter. The results come back: 99.9% of people use the internet daily.

Your math is right, but your conclusion is garbage. By choosing to host the poll on the internet, you automatically excluded anyone who doesn’t use the internet. You accidentally rigged the game before it even started.

Whenever you look at a dataset, your first question should never be, “What does the math say?” Your first question must always be, “How was this collected, and who was left out?”

Congratulations! You’ve Completed Path 1.

You have officially mastered the foundations of data. You know what statistics is, the types of data that exist, how variables interact, the NOIR levels of measurement, and how to gather your evidence cleanly.

You are now ready to graduate to Path 2: Descriptive Statistics. In the next module, we will finally start doing some math, beginning with the famous trio: Mean, Median, and Mode.