Evaluate, Troubleshoot, and Decide from Start to Finish

Across Part 2 you built three moves: evaluate an output against clear criteria, troubleshoot a failure back to its root cause, and decide what to do about it. This final chapter runs all three together, first as one complete worked case you can follow move by move, then as a turn of your own. Watch closely how the moves connect, because seeing them chain into a single flow is what turns three separate skills into one dependable habit.

Meet the Fairhaven Summer Reading Challenge

Each summer, volunteers at the Fairhaven Library run a reading challenge for local children. A child logs every book they finish, and at the end each child receives a warm congratulations message while the library posts a leaderboard of totals. Nobody on the team is a technology expert, and everything runs on free, everyday tools.

Here is the workflow, briefly. A parent or child submits each finished book on an online form, giving the child’s name and the book title. A volunteer’s shared spreadsheet holds one row per child with a running count of books read. A free AI assistant drafts a congratulations message for each child from their row. A volunteer sends the messages by email, and once a month someone exports the spreadsheet to a printed leaderboard posted by the front desk.

With the closing event a day away, two outputs have landed on the table, and both need judging before they go public.

• Output A is the AI-drafted congratulations message for a child named Priya. It reads warmly, correctly says she read eighteen books, and adds, “we loved seeing your passion for mystery novels.”

• Output B is the printed leaderboard now pinned by the front desk. It shows Priya at twelve books, and it lists a child named Sam Rivera twice, once with nine books and once with fourteen.

Both look finished. Neither has been checked. That is exactly where the method begins.

Evaluate the Two Outputs Against the Criteria

Start where Part 2 told you to start: run each output against the criteria instead of trusting how it looks. Judge each one against something outside it, never against its own wording.

Take Output A first. Its correctness holds on the number, since eighteen books matches Priya’s row, but it fails on the genre: to check that claim you look at what Priya actually logged, and her form entries carry no genre at all. The message is complete and its tone is appropriate for a child, yet a warm message built around a fact the child never supplied is not fit to send. A fluent, friendly draft has still failed the check that matters.

Now Output B, which is a data output, so it earns the two extra checks. Its currency fails: the printed sheet says twelve for Priya while the live spreadsheet says eighteen, so the posted list is an old snapshot. Its consistency fails too: Sam Rivera cannot truthfully have both nine books and fourteen, so the same child disagrees with himself across two rows. Two outputs, two failures, and not one of them was visible from the polished surface.

Troubleshoot Each Failure to Its Root Cause

An evaluation tells you something is wrong. Troubleshooting tells you why. Run the five steps on each failure so you fix the cause, not the surface.

Begin with Priya’s message. Because an AI assistant is stochastic, re-running the draft might not repeat the mistake, so instead of chasing an identical repeat you characterize the failure: save the exact message and record its input, which is Priya’s row plus the instruction “add a personal touch.” Then isolate the failing step by walking the map, form entry to spreadsheet row to AI draft. The form and the row hold no genre field anywhere, yet the finished message names one, so the failing step is the AI draft. Separate the symptom from its possible causes: the genre could have come from a wrong row, from a second Priya entered by mistake, or from the assistant inventing it. Confirm with evidence, and the comparison settles it: the source has no genre to draw from, so the assistant filled the gap with a confident guess. The root cause is a hallucination, invited by a vague instruction that asked for a personal touch the data could not supply.

Now the leaderboard, which hides two separate problems. The stale total is the first: the symptom is that the printed twelve disagrees with the live eighteen, you isolate it to the export step, and the root cause is plain once you check the export date, since the sheet was printed three weeks ago and never refreshed. The duplicate Sam is the second: the symptom is one child in two rows with two totals, and comparing those rows against the form shows two volunteers each created a row for the same child, so the root cause is a consistency break from two people logging one child separately.

Decide What to Do with Each Result

A confirmed cause still leaves a decision. Walk the four choices across the three findings, then close the loop so none of it happens again.

The congratulations messages that checked out completely need nothing, so the team accepts them. Priya’s message cannot go as written, and with the event a day away there is no time to re-engineer the whole drafting step, but the volunteers can supply the right result themselves, so they override the draft, delete the invented genre, and send a corrected note by hand. The stale leaderboard is a clear, in-reach repair, so they fix it by re-exporting from the live sheet. The duplicate Sam is trickier: if the two rows can be reconciled against the form with confidence, they fix it by merging into one correct total, but if they cannot tell which entries are real, they escalate to the challenge coordinator with the two rows and the form in hand, because guessing a child’s total is worse than asking.

Yes, which is why the last move is prevention. Overriding rescued one message; a preventive workflow improvement stops the whole class of failure. Here the team makes three: they rewrite the AI instruction so it draws only on fields the row actually contains and keep a human reading each draft before it sends, they give the leaderboard a single owner with a rule to re-export it the morning of any posting, and they add a check so two volunteers can no longer open a second row for the same child. Each change removes or guards a fragility point you first learned to spot back in Part 1. One afternoon’s judgment becomes a challenge that runs cleaner every summer after it.

Your Turn! Evaluate, Troubleshoot, and Decide

Now you run the whole method. Read the scenario, then work the three tasks in order using only the moves from Part 2: evaluate against the criteria, troubleshoot to a root cause, and make a judgment call.

Context. Volunteers run the Sunnyside Toy Drive, collecting donated toys and handing them out to local families. A family submits an online request form naming a child’s age and a wished-for toy. A shared spreadsheet matches each family to a toy. A free AI assistant drafts pickup-confirmation messages, a volunteer emails them, and a printed “toys still needed” list sits on the drop-off table. No one on the team is a technology expert.

With pickup day tomorrow, two outputs need judging before they go out.

• Output 1 is the AI-drafted confirmation to the Nguyen family. It reads warmly and gives the correct pickup time, but it tells them to collect a “science kit for age ten,” while their form asked for a board game for a seven-year-old.

• Output 2 is the printed “toys still needed” list on the table. It shows four dolls still needed, but the live spreadsheet shows zero, since all four arrived yesterday. The same list credits one donor, the Okafor family, twice, once with three items and once with five.

Your task. Work these three moves, writing your answers as you go.

1. Evaluate each output against the criteria, naming which criteria each one fails.

2. Pick one of the failures and troubleshoot it with the five steps to a single root cause, then say whether you would fix or escalate.

3. Make a judgment call on each output, choosing accept, fix, escalate, or override, and name one preventive workflow improvement for the drive.

Deliverable. A short evaluation of both outputs, one troubleshooting chain ending in a named root cause, and a judgment call for each output plus one preventive improvement.

Then check your answer against the model solution found at the end of this chapter.

Let’s Recap!

• The full method chains three moves into one flow: evaluate an output against the criteria, troubleshoot the failure to its root cause, then decide what to do ;

• Judge every output against something outside it, never against its own fluent surface, since an output cannot reveal its own errors ;

• The same five troubleshooting steps work on an AI draft and a printed sheet alike, ending in a cause you can back with evidence ;

• Accept, fix, escalate, and override each fit a different situation, and choosing the right one is the skill rather than defaulting to a favorite ;

• Closing with a preventive improvement turns a single afternoon’s fix into a workflow that fails less often every time it runs.

You have now built the whole skill, from mapping a workflow and finding its weak points to judging its outputs, tracing failures to their source, and deciding what to do. A short Part 2 quiz is the last step: use it only to confirm what has landed and to spot anything worth a second read, then carry this method to the real digital and AI workflows you already depend on.

Model Solution

Here is one sound way to judge the Sunnyside Toy Drive outputs. Your wording will differ, and that is fine; what matters is that each judgment rests on a criterion and each root cause rests on real evidence.

Evaluate both outputs.

• Output 1, the Nguyen confirmation, fails correctness: the toy and the age are both wrong, since the form asked for a board game for a seven-year-old and the message names a science kit for a ten-year-old. Its tone is appropriate and its pickup time is complete, but the wrong item makes it unfit to send ;

• Output 2, the printed list, is a data output and fails two data checks. It fails currency, because it shows four dolls needed while the live sheet shows zero after yesterday’s donations. It fails consistency, because the Okafor family is credited twice with two different totals, three items and five.

Troubleshoot one failure to its root cause. Taking Output 1: characterize the failure first, since the AI draft is stochastic, so save the message and record its input, the Nguyen row. Isolate the failing step by walking the map, form to row to draft, and check the row against the form. If the row correctly says “board game, age seven” but the message says “science kit, age ten,” the failing step is the AI draft and the root cause is a hallucination. If instead the row itself already says “science kit,” the root cause sits earlier, in a manual re-keying error when the form was copied into the spreadsheet. Either way you confirm with a direct comparison rather than a guess. If the fix is in your reach, correct the source and the message; if you cannot tell which item the family truly needs, escalate to the coordinator with the form in hand.

Make the judgment calls and prevent recurrence.

• The Nguyen confirmation: override it for now, sending a corrected message by hand so the family is not sent to the wrong toy, and accept the other confirmations that checked out ;

• The stale list: fix it by re-printing from the live sheet ;

• The duplicate Okafor entry: fix it by merging the two rows after confirming the real donation against the records, or escalate if the totals cannot be reconciled with confidence ;

• One preventive workflow improvement, any of these earns the check: rewrite the AI instruction to use only the fields in the row and keep a human reading each draft before it sends; give the “toys still needed” list a single owner with a rule to reprint it fresh each morning; or add a check so one donor or family cannot be entered twice.

If you named the failed criteria for both outputs, traced one failure to a single evidence-backed root cause, and made a clear accept, fix, escalate, or override call with a preventive improvement, you have run the complete evaluation method on your own. That is the whole skill this course set out to build.