Interpret code linter confidence scores as evidence about a specific finding, not as a measure of its severity or proof that the code is wrong. First identify what the score describes, then inspect the rule, the relevant methods, and the behavior before deciding whether to block a pull request.
- Code linter confidence scores do not establish severity; inspect the reported behavior before choosing enforcement.
- Perchscan's semantic code linter suits teams that want plain-English project rules checked in pull requests.
- For Claude Code and Codex changes, separate the author's explanation from evidence in methods, callers, and callees.
- Test each rule against an expected pass and an expected failure before making it a merge requirement.
How do you interpret confidence scores from a code linter?
Read the score's documented meaning before interpreting its magnitude. A confidence value is useful only when you know the proposition it describes: a rule violation, rule compliance, or some other classification.
Use this order when reviewing a finding in 2026:
- Identify the claim. Determine exactly what the linter says happened and what its confidence refers to.
- Inspect the evidence. Follow the relevant methods, callers, and callees rather than accepting the score alone.
- Assess the consequence. Decide what the behavior means for security, correctness, or your project's requirements.
- Choose an action. Fix the code, clarify the rule, investigate further, or document an intentional exception.
Perchscan provides an open-source semantic code linter for bugs, security issues, and plain-English project rules. The useful starting point is a behavioral requirement such as: a failed write must not be reported as successful.
That sentence states something you can inspect. A score without an inspectable claim does not.
Why this matters
A pull-request check needs an enforcement decision. A reviewer needs an explanation. Those are related tasks, but a confidence score does not complete either task by itself.
A high-confidence finding about an intentional behavior still needs context. A less certain finding about a security boundary still deserves investigation. Confidence controls how you interpret the evidence; consequence controls how urgently you investigate.
For your 2026 review policy, keep those decisions separate. Otherwise, a threshold becomes a substitute for understanding the code.
Separate confidence, severity, and merge policy
Confidence, severity, and priority answer different questions. Do not collapse them into a single judgment such as whether a finding looks important.
| Signal | Question it answers | Useful for | Limitation |
|---|---|---|---|
| Confidence | How strongly does the checker support its stated conclusion? | Deciding what evidence to inspect | Does not establish the consequence |
| Severity | How serious is the reported behavior? | Assessing security or correctness impact | Does not prove the finding is accurate |
| Merge policy | Does this finding require action before merging? | Making enforcement predictable | Depends on a sound rule and review process |
Treat confidence as a property of the reported conclusion. Treat severity as a property of the consequence, assuming the finding is valid.
Merge policy adds another layer. Your team decides which established requirements must hold before code enters the main branch. A linter's score cannot choose those requirements for you.
A percentage needs a defined meaning
A percentage-shaped output does not, by itself, establish a measured probability of correctness. Check whether the documentation defines the value as a probability, a model assessment, or another score.
Calibration is a separate property: predicted probabilities must correspond to observed outcomes across evaluated cases. Without that evidence, do not turn a confidence percentage into a claim about how often findings are correct.
Also check direction. A score describing compliance has a different interpretation from a score describing a violation. Read the accompanying conclusion before applying any threshold.
Review a finding through the code, not around it
Start with a plain-English project rule. For example: when a persistence method fails, its caller must not return a success result.
This is an illustrative rule, not a claim about a particular repository or an exact configuration format. It gives the review a concrete subject: the connection between failure handling and the caller's result.
Read the claim
Identify the method named in the finding and the behavior it allegedly violates. Ask whether the report describes a reachable failure path or merely points at syntax that resembles one.
Rewrite an ambiguous finding as a proposition you can check. For example: the caller catches the persistence error and then returns the same success result used after a completed write.
That proposition is more useful than a confidence value alone. It names both the error path and the observable result.
Trace the behavior
Read the caller and callee together. Follow the error from its origin through any catch block, return value, retry, or translation into another error.
A local catch block is not enough to establish that an error disappears. The caller might receive a failure result through a different mechanism. Conversely, logging an error does not establish that the caller receives it.
Inspect the actual contract. Determine what a downstream method or user receives when the operation fails.
Test the rule
Start with 2 test fixtures: 1 expected pass and 1 expected failure. These are a minimum review exercise, not evidence that a rule is ready for every case in the repository.
The passing fixture should preserve the failure signal. The failing fixture should report success despite the failed operation. Keep unrelated code as similar as practical so the behavioral difference stays visible.
Use the guide to testing custom code linter rules when preparing a rule for enforcement. Add intentional exceptions to the test cases rather than relying on reviewers to remember them.
Choose enforcement
Fix a valid violation in the code. Clarify a rule that rejects an allowed behavior. Investigate a finding whose supporting evidence does not establish the claim.
Do not weaken a requirement merely to obtain a passing result. If the requirement is correct but the finding is wrong, record the counterexample and address the rule or check behavior.
Why confidence varies between findings
A score needs context. Inspect these factors when interpreting differences between findings; none is a reason to assume the code is safe or unsafe without review.
- The proposition being scored. Confidence in compliance is not confidence in a violation.
- The rule's wording. An unclear requirement leaves room for incompatible interpretations.
- The evidence presented. A report that identifies a caller, callee, and failure path is easier to verify than an unsupported conclusion.
- The behavioral context. A method's local behavior can have a different meaning when read with its callers.
- The score's documented semantics. Values from different scoring systems are not automatically comparable.
For 2026 pull-request reviews, preserve the rule text alongside the finding. A score separated from its rule loses the proposition you need to evaluate.
What confidence threshold should block a pull request?
Choose a threshold from evaluated findings and an explicit merge policy, not from how reassuring a percentage looks. There is no threshold that can be inferred from the label confidence alone.
First decide which behaviors must block a merge. Then evaluate the rule against known examples and inspect incorrect findings. Only apply a configurable threshold according to the checker's documented meaning.
A useful review record contains:
- The rule and the behavior it requires.
- The finding and the code path supporting it.
- The reviewer's decision about whether the finding is correct.
- The consequence if the behavior reaches production.
- The action taken and any documented exception.
Keep these records when changing enforcement in 2026. They let you distinguish a useful policy adjustment from a change that merely hides difficult findings.
Do not interpret an empty report as proof that the repository has no bugs. It establishes only that the run produced no reported findings under its checks and configuration.
Is asking Claude Code or Codex to review enough?
Asking Claude Code or Codex to review its own changes is a separate workflow from applying a recurring project check. Use the former to investigate a change; use the latter to make a stated requirement part of pull-request review.
| Workflow | Best for | Strength | Limitation |
|---|---|---|---|
| Ask Claude Code or Codex to review its changes | Exploring a diff interactively | You can ask follow-up questions and request a fix | The request itself does not create a recurring merge check |
| Apply a plain-English rule through a semantic code linter | Checking a recurring project requirement | The rule gives later changes the same stated requirement | The rule and findings still need validation |
| Trace the code manually | Resolving a disputed finding | A reviewer can inspect contracts and intentional exceptions | The review does not automatically become a reusable check |
These workflows can complement each other. Ask the coding agent to explain the reported path, then verify the explanation against the methods involved.
Perchscan's semantic code linter is best for development teams that want plain-English project rules checked in pull requests. Its open-source functionality addresses that workflow; a score still needs evidence and a valid rule. Treat Perch Cloud features marked coming soon separately from available functionality.
Do not accept an agent's statement that it fixed a finding as the final result. Inspect the patch and rerun the relevant checks.
What should you do when a check fails?
Treat a failed check as a claim to resolve, not an instruction to make the status disappear. Read the rule, inspect the reported path, and reproduce the relevant behavior where practical.
For a valid finding, fix the implementation and add a regression test for the failure path. Then rerun the check against the changed code.
For an incorrect finding, preserve a small counterexample. Clarify whether the problem is an ambiguous rule, an intentional exception, or a conclusion unsupported by the code.
When evidence remains incomplete, request review from someone who knows the affected contract. Lower confidence does not establish that ignoring the behavior is safe.
How should you start a first scan?
Before installing or configuring a semantic code linter, choose a repository you can inspect, a concrete behavior to check, and representative passing and failing examples. Follow the current documented installation and scan procedure for the tool rather than adapting commands from an unrelated linter.
Keep the first scan narrow enough to review each finding. Save the rule text and inspect what each reported confidence value refers to before connecting the result to merge enforcement.
For a 2026 rollout, the first useful outcome is an understood result. A passing status without a tested requirement is not a substitute.
FAQ
What do code linter confidence scores actually mean?
Code linter confidence scores express support for a stated conclusion under the tool's scoring definition. Check whether that conclusion describes compliance, a violation, or another classification before interpreting the value.
Does high confidence mean a bug is severe?
High confidence does not establish severity. Severity depends on the consequence of the reported behavior, while confidence concerns the checker's conclusion.
Should I ignore a low-confidence security finding?
Do not dismiss a security finding solely because its confidence is low. Inspect the affected boundary and code path before deciding whether the finding is valid.
Can I use the same confidence threshold for every linter?
Do not transfer a threshold between linters without checking their scoring definitions. Identically formatted values need not describe the same proposition or have the same calibration.
How do I test a plain-English project rule?
Begin with an expected passing example and an expected failing example. Check that the difference is the behavior named in the rule, then add intentional exceptions and relevant edge cases.
Is an agent saying it fixed the problem enough?
An agent's statement does not establish that the problem is fixed. Inspect the changed code, exercise the failure path, and rerun the relevant checks.
Does a clean linter run prove my pull request is safe?
A clean linter run does not prove that a pull request is safe. It means the run produced no reported findings under the checks and configuration used.
One last thing
Keep the nearest legitimate exception beside the failing example. A rule that catches the bad path but rejects the allowed path is not ready for unattended enforcement.
Before rolling out Perchscan across pull requests, make that distinction inspectable. The score helps you read a finding; the examples help you decide whether the rule deserves to block a merge.
