Skip to content

Module 05 / Lesson 02

Review a diff in a fresh session with Semgrep

Module 05 · Real-World ContextStep 02 of 03

The AI will always tell you the code is ready. Only empirical verification and human judgment decide when to release.

It is Friday afternoon. The AI agent reports: "All changes implemented cleanly, ready for deployment!" But when a real visitor visits the live site, they receive stale cached information. Furthermore, a hidden security regression was introduced in an unreviewed helper function. Shipping without proof turns an AI shortcut into an outage.

Narrated lesson · 1:39
Read the narration transcript

A green summary leaves room for questions

The agent says its changes are safe and its tests passed. You can thank it for running the checks. The next question is which rules those tests actually exercised.

Open a fresh review with the issue and the diff. Ask for a read-only review of the public visibility rule and any changed paths. Give the reviewer the real checks already run.

Use Semgrep and ask for a fresh review

I run Semgrep Community Edition with the team’s local rule file. The auto configuration can fetch rules through the network. Either way, I look at what was scanned and review the actual findings.

Semgrep Skills gives the coding agent instructions for using the scanner and reading the results. I ask it to identify the relevant findings and cite the source. The command output is still the evidence.

A separate chat gets the diff, the actual requirement, and the checks we have. It reviews the public visibility rule and whether repeated submissions can duplicate an update. The final call belongs to the reviewer.

Turn the report into a reviewable finding

The public query changed. In the review, state that the published-state restriction no longer appears where it used to. It might be handled in another helper. Trace the actual code path.

Check the visitor session with a fixture that includes both kinds of data. If the rule is still unproven, say so and hold the approval.

Record what would change the decision

A command can exit successfully and still list results. Inspect the report and configure CI deliberately if findings should block the job.

Once the visibility check passes, attach it to the review and state what it proves. This is the same discipline you'll use for the launch decision.

The everyday problem

The author’s summary says the status app patch is safe because it uses validation and the tests pass. A second reviewer wants to know which behavior those tests cover and what the diff touches outside the form. A short review that starts with evidence will catch more than asking the authoring conversation whether it did a good job.

A fresh review can help because it starts from the requested behavior and the patch rather than the author’s earlier decisions. That is an organization choice, not a promise of independent judgment. Another session may use the same model and miss the same issues. Keep a human reviewer responsible for prioritizing findings and checking conclusions.

Why the agent got it wrong

Static scanning matches configured patterns and, depending on the tool and engine, can follow some information through code. It cannot infer every business rule or guarantee that every file and path was analyzed. A browser test covers an execution path. A reviewer can reason about behavior outside that path and ask for more evidence. All three have a different scope.

Your scan command can also mislead you if it returns exit zero while still reporting findings. Verify the documented behavior for the mode you’re using, review explicit findings and coverage, and configure any CI gate deliberately. A single line that says pass is weaker evidence than the rules and scope it represents.

The tool and technique

Give a fresh review the current requirement, the diff, and the checks already run. Ask the reviewer to list findings by severity and provide file and behavior evidence. Ask it to tie each finding to a requirement or failure mode and suggest the smallest additional check. Keep it read-only for the review pass.

Use Semgrep Community Edition as a defensive signal where its rules cover the code. It can run local rules. Fetching rules from the registry may use the network and can enable metrics under the default behavior. Check the documented settings and use approved data and rules in the project environment.

Pair a local static scan with a fresh review

Use Semgrep Community Edition for a repeatable scan. The companion repository’s beta code-security skill gives the reviewer a checklist for configuration, data handling, and authorization. A fresh session receives the diff, intended behavior, and existing evidence. It checks whether the status app protects private notes and handles repeated submissions. Treat scan results and skill suggestions as leads the reviewer must verify.

# Optional: review the code-security skill before adding it
npx skills add semgrep/skills
semgrep scan --config ./semgrep.yml --metrics=off

Use code-security to review this diff and the scan output.
Check for secrets in logs and confirm public queries hide private notes.
Cite the file and line for each concern. Identify the test that covers it.
Check if two valid submissions can record the same update.

Use a reviewed local rule file for the example. Registry presets such as --config=auto fetch rules over the network. Read coverage and findings even when the process exits successfully. The human owner decides which results block release.

On screen in the status app

In the first review below, the summary treats passing tests as a blanket approval. In the second, the reviewer notices the public query no longer makes the published-state restriction obvious. The next step is to trace the actual data path and run the approved role and visibility tests against synthetic incidents. If the filter moved safely, update the review note with evidence. If the public response can include an internal note, hold release until that behavior is corrected and checked.

For a separate finding, the scanner flags a pattern in a mock test fixture. Inspect its path, rule, and runtime relevance. Confirm that the fixture cannot become part of the shipping build. A finding might be a useful fixture check, a true issue, or an irrelevant match. Keep a reason for your decision in the review.

Pull request review

Before

All tests are green. The agent says the new query is safe. Approve.

After

Finding: the public query’s published-state guarantee is unclear.
Evidence: the previous filter is absent from the changed call site.
Check: trace the existing data helper and verify that the public fixture excludes an internal note.
Decision: hold until the returned content matches the public rule.

What to watch for

A new thread doesn’t make test reports trustworthy by itself. A model can cite a file or a check that wasn’t run. Follow the exact path, see the result, and reproduce any release-blocking issue from controlled fixtures. Use human review of authentication, data access, and migrations even when automated tools are quiet.

Treat explanations and tool outputs as inputs to review. Avoid putting private source or sensitive records into unapproved services. Confirm what the automated rule covers and write test cases that match the release risk. More review loops help only when each adds a different source of evidence.

How you know it worked

Ask a teammate to choose one high-impact finding and show the changed behavior, how to observe it safely in the local fixture, and whether it is covered by a check. Confirm the underlying rule still holds after the fix, rather than relying on a scanner match disappearing. Keep false positives documented narrowly.

End the review with a decision and what would change it. For example, hold until a published-state check passes in the visitor context, then recheck the diff. A review with one concrete blocker is more useful than a long list of unverified suspicions.

References