Case study
Agentic AI Security Testing With Scope and Review Controls
Automated testing within explicit authorization boundaries.
The challengeRepeatable testing of vulnerabilities and unsafe AI behavior.
The solutionScoped execution with review and structured evidence.
An authorized-scope gate before testing.
Human review for risky test proposals.
Approximately 340 penetration tests per day reported.
Engagement
Overview
Odva AI’s testing workflow is designed to identify vulnerabilities and unsafe behavior in AI applications. The system brings test planning, execution, findings, and evidence into a repeatable workflow with explicit authorization and review steps.
I built an agentic process that starts with target and permission information in a test-management console. An authorized-scope gate defines the permitted testing context, and a language-model endpoint supports AI-assisted test planning. Risky proposals pass through human review before the approved test plan reaches execution.
The executor performs scoped interactions against the authorized AI application. Observed findings pass through Pydantic-based schema validation, with results and evidence retained for reporting. The workflow distinguishes generating a test from authorizing its execution and structuring the resulting finding.
Usage figures are internally reported. Reporting periods and user definitions remain unconfirmed.
- Reported users
- 2,380
- Reported penetration tests / day
- ≈340
Challenge
Automated testing needed controls around where it could act and how its findings were recorded.
Authorized scope
Testing needed to operate against permitted targets and within an approved scope.
Risk-sensitive execution
Risky proposed tests needed a human review step.
Consistent findings
Observations needed a defined structure for downstream reporting.
Retained evidence
Results needed an evidence store rather than only a generated narrative.
Solution
I connected agentic test planning to controlled execution and structured reporting.
Scope authorization
Use target and permission information to establish the approved testing scope.
AI-assisted planning
Connect the test planner to a language-model endpoint for planning and test generation.
Human review
Route risky proposals through the documented review step.
Scoped execution
Run approved test interactions against the authorized AI application.
Schema validation
Use Pydantic to validate the structure of observed findings.
Evidence and reporting
Store test results and evidence and produce structured security findings and reports.
Separate planning, authorization, and validation
AI-assisted planning is one part of this system, not permission to execute any proposed action. Scope authorization and risky-test review sit before execution, while finding-schema validation sits afterward. These are distinct controls: a schema-valid finding is structured for reporting, but schema validation alone does not establish exploitability.
Impact
Reported activity shows recurring use of an automated security-testing workflow with authorization, review, and evidence handling built in.
Reported adoption
2,380 users.
Reported activity
Approximately 340 penetration tests per day.
Implemented capability
Agentic planning, scoped execution, human review, schema-validated findings, and retained evidence.