Judge the Evidence Quality

In the last chapter you learned to investigate who made a source and whether they have a stake in what you believe. But even an authoritative, independent source is only as good as the evidence it rests on. This chapter gives you a way to judge that evidence: to tell strong support from weak, and to avoid being dazzled by words like “study” or “peer-reviewed” that sound convincing but do not, on their own, make a claim true.

What Makes Evidence Strong?

A claim is only as trustworthy as the evidence underneath it. The hard part for most beginners is that weak and strong evidence often look the same on the surface: both can be written confidently, both can cite numbers, and both can use the word “study.” What separates them is underneath.

There are four things worth looking at:

1. The first is the kind of evidence. One person saying “this worked for me” is an anecdote, the weakest kind. A survey is stronger. Strongest is a controlled experiment that compares a group who got the thing against a group who did not.

2. The second is sample size. A claim built on five people is far shakier than one built on five hundred, because small samples can show an effect purely by chance.

3. The third is method: how the evidence was gathered. Were people just asked how they felt, or was something actually measured? Was there a comparison group?

4. The fourth is whether the result holds up. Peer reviewed means other experts checked the work; replicated means someone ran the study again and got the same result. One study is a start, but a result that keeps showing up is far stronger.

So do not tick a single box. Ask all four questions together: what kind of evidence, how much of it, gathered how, and does it hold up. A confident claim built on one small self-funded survey is weak, however polished. A modest claim built on a large, independent, replicated study is strong, even if it sounds less exciting.

Why “Peer-Reviewed” Doesn’t Mean “True”

You will run into the phrase peer-reviewed constantly, so it is worth understanding what it does and does not mean. Peer review is the process where, before a study is published, other experts in the field read it and check the work for obvious flaws. It is a genuine quality filter, and a claim that has been through it generally deserves more weight than one that has not.

But peer review is a signal, not a guarantee. Reviewers can miss things. A peer-reviewed study can still be built on a sample, the group of people or cases it looked at, that is far too small to support its conclusion. It can use a weak method. It can be funded by a party with a stake in the result. And plenty of peer-reviewed findings have later failed to replicate, meaning that when other researchers ran the study again, they did not get the same outcome.

So treat “peer-reviewed” as one box among several, not a trump card. A peer-reviewed study resting on a tiny, company-funded sample can be weaker than a large, independent one. What matters is the whole picture: the kind of evidence, the sample, the method, whether it holds up, and who paid for it.

Run the Evidence-Strength Checklist

You will not always have an instructor to talk you through a source. What you need is a short list of questions you can run on any piece of evidence, in order, from the most basic to the most demanding, from a single story (an anecdote, one person’s experience offered as proof) up to a controlled experiment. The table below is that checklist. Work down it, and stop with a clear sense of how much weight the evidence can bear.

Compare Two Sources on the Same Claim

Let’s watch the checklist do its work. Imagine you are deciding whether to buy a standing desk, and you want to know one thing: do standing desks actually reduce back pain? You find two sources making that same claim. Both say yes. The checklist is what tells them apart.

Source A: a desk maker’s blog. The page says, “In our own trial, 8 out of 10 users told us their back felt better after a week.” Run the checklist. The kind of evidence is a survey of feelings, not a measured outcome. The sample is tiny, just ten people. The method is self-reported, with no comparison group, so we cannot tell whether the desk helped or the users simply expected it to. It has not been reviewed or replicated. And it is funded by the company selling the desk. On every question, this is weak.

Source B: a study in an occupational-health journal. It is a randomized controlled trial, a study where people are split by chance into a group that gets the thing and a group that does not, then compared. It followed 200 office workers for six months, measured their reported pain on a standard scale, and was funded by a university with no stake in desk sales. A later study found a similar result. Run the checklist: a controlled experiment, a large sample, a real method with a comparison group, peer reviewed and replicated, independently funded. On every question, this is strong.

The verdict. Both sources make the identical claim, yet they are worlds apart. Source B can carry real weight; Source A can support very little. Notice that you did not need to be a scientist to tell them apart. You only needed to ask the five questions in order.

Your Turn!

Time to weigh evidence yourself. Below are three sources, all supporting the same claim: that a particular air purifier reduces household allergy symptoms. Your job is to rank them by evidence strength, from strongest to weakest, using the checklist.

The three sources

1. Customer reviews. On a shopping site, dozens of buyers leave five-star reviews saying their allergies improved after using the purifier.

2. The manufacturer’s website. It reports an internal study: “In our survey of 50 customers, 70% said they noticed fewer symptoms within a month.”

3. A peer-reviewed study in an environmental-health journal: a randomized controlled trial of 300 homes that measured actual airborne allergen levels and symptom scores, funded by a public research grant.

Your deliverable. Rank the three sources from strongest to weakest evidence, and write one sentence for each explaining your placement using the checklist questions. When you are done, check your answer against the model solution at the end of this chapter.

Let’s Recap!

• Evidence varies in strength, and a confident tone or the word “study” does not make a claim true.

• Judge evidence with five questions: what kind it is, how big the sample, how it was gathered, whether it holds up, and who funded it.

• The strongest evidence tends to be a controlled experiment with a large sample, measured outcomes, replication, and independent funding.

• Peer review is one quality signal, not proof, since a peer-reviewed result can still rest on a small sample, a weak method, or conflicted funding.

• Weighing these questions together is how you will score each source in your project’s Evaluation Matrix.

You have now finished investigating and evaluating sources from the inside: who made them, and how strong their evidence is. But the surest test of a claim is not to stare harder at the page in front of you. It is to leave it. In Part 2 you will learn to check a source against the wider world, trace its claims back to where they began, and catch the tricks that make weak evidence look strong.

Model Solution

This is the solution to the “Your Turn!” activity above. Rank the three sources yourself before reading on.

Strongest: Source 3, the peer-reviewed study. It is a controlled experiment with a large sample of 300 homes, it measured actual allergen levels rather than only asking how people felt, it was independently funded by a public grant, and it passed peer review. It is strong on every question in the checklist.

Middle: Source 2, the manufacturer’s internal study. It is more structured than scattered reviews, with 50 customers surveyed, but the evidence is self-reported, there is no comparison group, the sample is modest, it has not been reviewed or replicated, and it was funded by the company that sells the purifier. Better than an anecdote, but conflicted and weak in method.

Weakest: Source 1, the customer reviews. Dozens of positive reviews feel persuasive, but they are anecdotes: self-selected, uncontrolled, unmeasured, and impossible to verify. People who felt no effect often simply do not post. This is the weakest evidence of the three.

So the ranking is 3, then 2, then 1. Notice that the number of voices did not decide it: dozens of reviews still rank below a single well-run study, because strength comes from method and independence, not volume.