Debug AI-Written Code and Test the Behavior That Matters

Use a reproducible failure to guide the repair, then keep a test that would catch its return.

AI-generated code can be clear, plausible, and wrong at a boundary that the demonstration never reaches. A list limit may work for one item and fail at the maximum. A parser may accept normal data but silently mishandle a missing field. A retry loop may succeed in a happy-path demo while duplicating work after a timeout.

Debugging begins by making the failure specific. What input produced it? What should have happened? What actually happened? Which version of the code was running?

Those questions are more useful than asking an assistant to “fix all bugs.” A focused report gives both you and the assistant an observable target.

Start with a requirement independent of the code

The catalog specification permits at most 20 entries. When the list already contains 20 titles, adding another must fail and leave the original list unchanged. When it contains 19, adding one valid title must produce a list of 20.

That pair of cases comes from the product requirement. It does not depend on how the programmer chose to implement the limit.

Consider this complete, deliberately faulty function:

def add_title_buggy(existing, raw):
    title = raw.strip()
    if not title:
        raise ValueError("Enter a title")
    if len(title) > 60:
        raise ValueError("Title too long")
    if any(item.lower() == title.lower() for item in existing):
        raise ValueError("Duplicate title")
    if len(existing) > 20:
        raise ValueError("Catalog full")
    return existing + [title]

The function rejects a list only when its length is already greater than 20. A list of exactly 20 passes that check, and the returned list has 21 entries. The defect is a single comparison, but it changes a user-facing guarantee.

The function assumes existing contains previously validated strings and raw is a string, as supplied by the application's text-input boundary. If those assumptions change, add validation at the appropriate boundary. This lesson focuses on the documented title rules and capacity behavior.

Reproduce the smallest useful failure

A minimal reproduction does not need a browser, server, model, or database:

existing = [f"Item {number}" for number in range(20)]
result = add_title_buggy(existing, "Extra")
print(len(existing), len(result))

Place those lines below the faulty function to run them. The defect produces 20 21. The original list stays at 20 because the function creates a new list; the returned list violates capacity.

A good bug report would say: “With 20 distinct existing titles, adding Extra returns 21 titles. The specification requires rejection at capacity, with the original list unchanged.” Include the short reproduction and the expected exception.

Avoid sending an entire private project when a synthetic reproduction captures the problem. Smaller examples reduce irrelevant context and make the claimed failure easier to verify independently.

Write a regression test before repairing the function

A regression test records a behavior that must continue to hold after a fix. The most direct test here expects a capacity error for a full list. A neighboring test checks that a list of 19 still accepts its twentieth entry.

The corrected implementation and test suite below are complete. Save them together as test_catalog.py and run python3 test_catalog.py. They use Python's built-in unittest framework; its official documentation describes test discovery and assertions.

import unittest


def add_title(existing, raw):
    title = raw.strip()
    if not title:
        raise ValueError("Enter a title")
    if len(title) > 60:
        raise ValueError("Title too long")
    if any(item.lower() == title.lower() for item in existing):
        raise ValueError("Duplicate title")
    if len(existing) >= 20:
        raise ValueError("Catalog full")
    return existing + [title]


class CatalogTests(unittest.TestCase):
    def test_full_catalog_rejects_addition_without_mutation(self):
        existing = [f"Item {number}" for number in range(20)]
        before = existing.copy()
        with self.assertRaisesRegex(ValueError, "Catalog full"):
            add_title(existing, "Extra")
        self.assertEqual(existing, before)

    def test_nineteen_items_accept_twentieth_without_mutation(self):
        existing = [f"Item {number}" for number in range(19)]
        before = existing.copy()
        result = add_title(existing, "Final")
        self.assertEqual(result, before + ["Final"])
        self.assertEqual(len(result), 20)
        self.assertEqual(existing, before)
        self.assertIsNot(result, existing)

    def test_blank_title_rejected(self):
        with self.assertRaisesRegex(ValueError, "Enter a title"):
            add_title([], "   ")

    def test_case_and_whitespace_duplicate_rejected(self):
        with self.assertRaisesRegex(ValueError, "Duplicate title"):
            add_title(["Field notebook"], " FIELD NOTEBOOK ")

    def test_sixty_code_points_accepted(self):
        self.assertEqual(add_title([], "x" * 60), ["x" * 60])

    def test_sixty_one_code_points_rejected(self):
        with self.assertRaisesRegex(ValueError, "Title too long"):
            add_title([], "x" * 61)

    def test_success_trims_edges_and_preserves_internal_space(self):
        self.assertEqual(add_title(["Map"], " Field  notebook "),
                         ["Map", "Field  notebook"])


if __name__ == "__main__":
    unittest.main(verbosity=2)

Prove that the test can detect the defect

A test that passes only after you write it may still be checking the wrong thing. For a regression, demonstrate that it fails against the known faulty behavior and passes after the intended repair.

During preparation, this seven-test suite was run against a version using len(existing) > 20. Exactly one test failed: the full-catalog rejection test. The same suite passed all seven tests after the comparison changed to >= 20.

That result supports a narrow conclusion: the suite detects and verifies the repair of this capacity defect while checking several neighboring title rules. It does not prove that the website has no bugs, because the Python function is a separate implementation of part of its specification.

The browser application's JavaScript still needs its own interaction checks. A Python test cannot establish that the page announces errors correctly, restores keyboard focus, or displays literal text safely. Keep each claim connected to the layer actually exercised.

Choose tests by the consequence of being wrong

The test suite emphasizes boundaries and preserved state. At 19 entries, addition succeeds. At 20, it fails. At 60 code points, a title is accepted. At 61, it is rejected. A duplicate remains a duplicate after trimming and case normalization.

Each test has a reason to exist. It represents a requirement, a previously observed failure, or a consequential edge case. Adding hundreds of examples that all follow the same easy path may increase test counts without improving confidence.

Use different kinds of checks for different questions:

A mock or fixture replaces part of the real system. That is useful when you need repeatable failures, but it is also the reason a fixture-based pass cannot establish that the replaced system works.

Read errors as evidence

An error message usually identifies where execution stopped, not necessarily where the original mistake began. A missing dictionary key may result from an earlier parser accepting the wrong structure. A timeout may reflect a slow service, a network problem, or a client deadline that is too short for the task.

Read the exception type and the relevant traceback frame. Compare the actual input with the function's documented assumptions. Reduce the failing case until you can explain which condition triggers it.

When asking AI for help, include the smallest relevant code, the exact non-sensitive error, the input, expected behavior, and what you already checked. Remove credentials and unrelated personal content. Ask for a hypothesis tied to the evidence, not a confident story about unseen infrastructure.

If several causes remain plausible, choose a check that distinguishes them. For example, an offline parser fixture can separate a response-format problem from a network problem. A direct call to the validation function can separate a rule defect from a browser event-handling defect.

Review the proposed repair as a change in behavior

An assistant may fix a capacity test by deleting the capacity requirement, changing the test expectation to 21, or catching every exception and returning an empty list. Those changes can make a test run look better while making the product worse.

Compare the patch with the specification. In this case, the intended repair changes one comparison. It should not alter duplicate handling, title length, or whether the input list is mutated.

Run the failing regression and the nearby checks affected by the change. Broaden testing when the patch touches shared behavior or leaves a concrete risk unresolved. Repeating unrelated checks indefinitely does not make a narrowly understood change more correct.

Keep the test after the repair. A future refactor should be able to reorganize the implementation while preserving the requirement. That is why the test describes what happens at capacity rather than asserting that a particular source-code line contains >=.

Distinguish deterministic bugs from model variability

The catalog capacity rule is deterministic: the same valid input and code should produce the same result. A model-generated explanation can vary while remaining acceptable. Its evaluation needs criteria that distinguish valid variation from a failure.

For a model-backed catalog answer, you might require the correct item identifier, a location supported by the returned record, and no disclosure of inaccessible records. Exact wording may be unimportant. For a machine-readable field, exact type and value may be essential.

Use stable fixtures to debug the surrounding software. Then evaluate model behavior separately on representative tasks, including missing evidence and conflicting instructions in retrieved content. Record the model version, configuration, and evaluation cases for comparisons.

Do not treat a single fluent demonstration as an evaluation, and do not treat a single alternative phrasing as a defect unless the output contract requires that wording.

Keep a short repair record

A useful repair note can be only a paragraph:

“Adding to a full 20-entry catalog returned a list of 21. The capacity comparison rejected only lists already above the limit. Changed it to reject lists at or above 20. The new regression failed on the original version and passed after the change; all seven title-rule tests passed. Browser interaction behavior was not tested by this Python suite.”

That note tells a reviewer what was wrong, why the change addresses it, how it was checked, and where the evidence stops. It avoids both unsupported certainty and a long chronological transcript of every attempt.

A reusable prompt

Debug this specific failure: [input, expected behavior, actual behavior, and relevant code]. First identify the smallest reproduction and the requirement being violated. Propose a regression test that fails on the current version. Make the smallest justified repair, explain its behavioral effect, and report the checks actually run. Keep unrelated behavior intact and distinguish local fixtures from live integration evidence.

For students: explain the defect before showing the patch

In a programming assignment, a useful debugging submission includes the reproduction, the violated requirement, the failing test, the change, and the result after repair. Explain why the original condition admitted a twenty-first item.

If AI suggested the fix, demonstrate that you can predict both versions' behavior. Ask a classmate to change a neighboring boundary and see whether your tests detect the new problem. This is a stronger learning exercise than collecting a screenshot of a green test run.

Students in other fields can apply the same method to spreadsheet formulas, data-cleaning rules, or research scripts. A grade calculation at a cutoff and a file organizer with a name collision both benefit from explicit expected outcomes and small reproducible cases.

Practice: preserve a failure as a test

Run the corrected suite. In a disposable copy, change >= 20 back to > 20 and confirm that the capacity test fails. Restore the fix and confirm that the same test passes.

Then choose one real rule from your own project. Write its expected behavior in ordinary language, identify a boundary case, and create a check that would fail if the rule were broken. Explain what the check does not cover.

Completion check: You can show a test failing for the intended reason before a repair and passing afterward, explain why the patch satisfies the requirement, and avoid claiming that local tests verify untested interfaces or services.

Stretch: Build a small evaluation set for a model-backed feature. Separate exact requirements from acceptable wording variation, include unsupported-answer cases, and record enough configuration to compare later versions meaningfully.