Skip to content
  • factory
  • ai-agents
  • qa
  • testing
  • validation

QA & Validation Agents

This document provides a comprehensive overview of QA & Validation agents in the ConnectSoft AI Software Factory. It is written for QA engineers, developers, and architects who need to understand how the Factory ensures code quality and architectural compliance.

QA & Validation agents review generated artifacts (code, tests, docs, pipelines) and identify issues, missing tests, and non-compliance with standards. They suggest improvements and trigger re-runs when needed, ensuring that only high-quality code reaches production.

Important

QA agents don't just generate tests—they enforce quality gates and ensure that only high-quality code is merged. They validate architecture, code structure, test coverage, and compliance with standards.

QA Agent Cluster Composition

The Security, Compliance, and QA cluster consists of 10+ specialized agents organized into sub-clusters:

Testing Agents

  • QA Engineer Agent - QA oversight, planning, and approvals
  • Test Generator Agent - Generates .feature and .cs tests from prompts
  • Test Case Generator Agent - Emits unit/integration test classes from trace metadata
  • Test Automation Engineer Agent - Executes tests across roles, editions, and traces
  • Test Coverage Validator Agent - Detects missing tests, retries, role-edition gaps

Security Agents

  • Security Engineer Agent - Validates security patterns and best practices
  • Security Penetration Testing Agent - Performs penetration testing and vulnerability scanning
  • Privacy Compliance Agent - Validates compliance with GDPR, HIPAA, and other regulations

Code Review and Bug Investigation

  • Code Reviewer Agent - Validates QA metadata and coverage in PRs
  • Bug Investigator Agent - Validates bugs are protected by regression scenarios

Advanced Testing Agents

  • Load & Performance Testing Agent - Measures throughput, latency, system resource limits
  • Resiliency & Chaos Engineer Agent - Applies retry policies, latency injection, resiliency breakers

Mission and Scope

QA & Validation agents are responsible for:

  • Reviewing Generated Artifacts - Code, tests, docs, pipelines
  • Identifying Issues - Bugs, missing tests, architectural violations
  • Enforcing Standards - Clean Architecture, DDD, observability patterns
  • Suggesting Improvements - Propose fixes and enhancements
  • Security Validation - Ensuring security patterns and compliance
  • Performance Testing - Validating performance and scalability
  • Chaos Engineering - Testing resiliency and fault tolerance

What They Do: - Review code structure and architecture compliance - Generate and validate tests - Check test coverage and quality - Validate pipeline configurations - Review documentation completeness - Identify security and compliance issues - Perform penetration testing - Execute load and performance tests - Run chaos engineering tests - Investigate bugs and regressions - Suggest improvements and fixes

What They Do NOT Do: - Execute tests in production environments (except monitoring) - Make architectural decisions - Generate production code

Inputs and Outputs

Inputs

Input Source Description
Repositories Engineering Agents Generated code repositories
Pipeline Definitions Engineering Agents CI/CD pipeline configurations
ADRs Architect Agents Architecture decision records
Documentation Engineering Agents README, API docs, code comments
Test Suites Engineering Agents Generated test code

Outputs

Output Consumer Description
Review Reports Engineering Agents, Human Reviewers Quality assessment reports
Suggested Changes Engineering Agents Proposed fixes and improvements
Quality Metrics All Code coverage, quality scores
Human Review Flags Human Reviewers Issues requiring human attention
Test Coverage Reports Engineering Agents Test coverage analysis

Core Capabilities

Static Code Analysis

  • Structure Analysis - Analyze code structure and organization
  • Pattern Compliance - Check adherence to Clean Architecture and DDD
  • Layer Violations - Identify dependency violations
  • Code Quality - Check code quality metrics

Test Generation and Validation

  • Test Generation - Generate unit, integration, and acceptance tests
  • Test Coverage Analysis - Analyze test coverage
  • Test Quality - Validate test quality and completeness
  • BDD Specs - Generate behavior-driven development specifications

Architecture Validation

  • Architecture Compliance - Validate adherence to architectural patterns
  • Bounded Context Validation - Check bounded context boundaries
  • Integration Validation - Validate API and event contracts
  • Pattern Validation - Check pattern application

Security and Compliance Checks

  • Security Patterns - Check security pattern implementation
  • Compliance Checks - Validate compliance with standards
  • Vulnerability Scanning - Identify security vulnerabilities
  • Secret Detection - Detect hardcoded secrets

Typical Workflows

Workflow: Post-Generation QA

  1. Receive Generated Code - Get code from Engineering agents
  2. Review Structure - Check project structure and organization
  3. Validate Architecture - Check Clean Architecture compliance
  4. Analyze Tests - Review test coverage and quality
  5. Check Observability - Verify logging, tracing, metrics
  6. Generate Report - Create quality assessment report
  7. Flag Issues - Identify issues requiring fixes
  8. Suggest Improvements - Propose fixes and enhancements
  9. Trigger Re-Run - Request Engineering agent fixes if needed

Workflow: Refactor QA

  1. Receive Refactored Code - Get refactored code
  2. Compare with Original - Compare with previous version
  3. Validate Changes - Ensure changes maintain functionality
  4. Check Test Updates - Verify tests are updated
  5. Validate Architecture - Ensure architecture is maintained
  6. Generate Report - Create refactor quality report

QA Agent Collaboration Flow

flowchart TD
    Code[Generated Code] --> TestCaseGen[Test Case Generator Agent]
    Code --> TestGen[Test Generator Agent]
    Code --> CodeReviewer[Code Reviewer Agent]

    TestCaseGen --> TestAutomation[Test Automation Engineer Agent]
    TestGen --> TestAutomation

    TestAutomation --> CoverageValidator[Test Coverage Validator Agent]
    CoverageValidator --> QAEngineer[QA Engineer Agent]
    CoverageValidator --> BugInvestigator[Bug Investigator Agent]

    Code --> SecurityEngineer[Security Engineer Agent]
    Code --> PrivacyCompliance[Privacy Compliance Agent]
    SecurityEngineer --> PenetrationTester[Penetration Testing Agent]

    Code --> LoadTester[Load & Performance Testing Agent]
    Code --> ChaosEngineer[Chaos Engineer Agent]

    QAEngineer -->|Gaps Found| TestGen
    BugInvestigator -->|Regression Found| TestGen
    CoverageValidator -->|Issues| Engineering[Engineering Agents]

    style Code fill:#e1f5ff
    style TestAutomation fill:#fff4e1
    style CoverageValidator fill:#e8f5e9
    style QAEngineer fill:#f3e5f5
Hold "Alt" / "Option" to enable pan & zoom

Visual Flow: Complete QA Process

flowchart LR
    A[Generated Code] --> B[Test Generators]
    B --> C[Test Automation]
    C --> D[Coverage Validator]
    D --> E{Issues Found?}
    E -->|Yes| F[Engineering Agents]
    E -->|No| G[Security Review]
    G --> H[Performance Testing]
    H --> I[Chaos Testing]
    I --> J[Approve]
    F --> A

    style A fill:#e1f5ff
    style B fill:#fff4e1
    style D fill:#e8f5e9
    style J fill:#f3e5f5
Hold "Alt" / "Option" to enable pan & zoom

Collaboration with Other Agents

With Engineering Agents

Receive: - Generated code and tests - Pipeline definitions - Documentation

Provide: - Code review feedback - Quality violations - Test coverage gaps - Suggested fixes

Trigger: - Re-runs for fixes - Additional test generation - Code improvements

With Architect Agents

Receive: - Architecture blueprints - ADRs - Design decisions

Provide: - Architecture validation feedback - Risk identification - Compliance concerns - Pattern violations

With Vision & Planning Agents

Provide: - Testability concerns - Quality risks - Edge case identification - Feasibility feedback

Individual QA Agent Details

Testing Agents

QA Engineer Agent

Role: QA oversight, planning, and approvals

Responsibilities: - Plan QA strategy and test coverage - Approve test plans and scenarios - Trigger prompt-based test generation - Manage QA prompt lifecycle - Review QA metrics and reports - Coordinate with other QA agents

Inputs: - Product requirements - Architecture blueprints - Test coverage reports

Outputs: - QA plans - Test approval decisions - QA metrics reports

See: Detailed QA Engineer Agent specification in Factory Documentation

Test Generator Agent

Role: Generates .feature and .cs tests from prompts

Responsibilities: - Translate QA prompts into executable tests - Generate BDD .feature files (Gherkin) - Create scenario-based tests - Generate tests for role × edition combinations - Create regression tests from bug traces

Inputs: - QA prompts - Bug traces - Coverage gaps - Scenario requirements

Outputs: - .feature files - Test code - Test specifications

See: Detailed Test Generator Agent specification in Factory Documentation

Test Case Generator Agent

Role: Emits unit/integration test classes from trace metadata

Responsibilities: - Generate unit tests for handlers - Create integration tests for adapters - Generate DTO validation tests - Create arrange/act/assert structured tests - Ensure handler-level path coverage

Inputs: - Trace metadata - Handler definitions - Port definitions

Outputs: - .cs test classes - Unit test methods - Integration test methods

See: Detailed Test Case Generator Agent specification in Factory Documentation

Test Automation Engineer Agent

Role: Executes tests across roles, editions, and traces

Responsibilities: - Execute test matrix (role × edition × scenario) - Run tests in parallel - Handle retry logic for flaky tests - Integrate with CI/CD pipelines - Report test results to Studio - Emit execution metadata

Inputs: - Test suites - Test matrix definitions - Execution requirements

Outputs: - Test execution results - Execution summaries - Retry reports

See: Detailed Test Automation Engineer Agent specification in Factory Documentation

Test Coverage Validator Agent

Role: Detects missing tests, retries, role-edition gaps

Responsibilities: - Validate test coverage completeness - Detect missing role × edition combinations - Identify scenario gaps - Compare expected vs. actual coverage - Trigger test regeneration for gaps - Report coverage metrics

Inputs: - Test execution results - Expected coverage matrix - Trace metadata

Outputs: - Coverage reports - Gap analysis - Regeneration requests

See: Detailed Test Coverage Validator Agent specification in Factory Documentation

Security Agents

Security Engineer Agent

Role: Validates security patterns and best practices

Responsibilities: - Review code for security patterns - Validate authentication and authorization - Check input validation - Verify secret management - Validate encryption usage - Review security configurations

Inputs: - Generated code - Security requirements - Security standards

Outputs: - Security assessment reports - Security violations - Security recommendations

See: Detailed Security Engineer Agent specification in Factory Documentation

Security Penetration Testing Agent

Role: Performs penetration testing and vulnerability scanning

Responsibilities: - Perform automated penetration tests - Scan for vulnerabilities - Test security boundaries - Validate security controls - Generate security test reports - Identify security weaknesses

Inputs: - Deployed services - Security test requirements - Threat models

Outputs: - Penetration test results - Vulnerability reports - Security recommendations

See: Detailed Security Penetration Testing Agent specification in Factory Documentation

Privacy Compliance Agent

Role: Validates compliance with GDPR, HIPAA, and other regulations

Responsibilities: - Validate GDPR compliance - Check HIPAA compliance - Verify data privacy controls - Validate consent management - Check data retention policies - Generate compliance reports

Inputs: - Code and configurations - Compliance requirements - Data handling policies

Outputs: - Compliance reports - Compliance violations - Compliance recommendations

See: Detailed Privacy Compliance Agent specification in Factory Documentation

Code Review and Bug Investigation

Code Reviewer Agent

Role: Validates QA metadata and coverage in PRs

Responsibilities: - Review pull requests for QA completeness - Validate test coverage in PRs - Check QA metadata presence - Ensure test references in PRs - Validate trace IDs in commits - Approve or request changes

Inputs: - Pull requests - Test coverage data - QA metadata

Outputs: - Code review comments - Approval/rejection decisions - Coverage validation reports

See: Detailed Code Reviewer Agent specification in Factory Documentation

Bug Investigator Agent

Role: Validates bugs are protected by regression scenarios

Responsibilities: - Investigate bug reports - Trace bugs to test coverage - Validate regression test coverage - Identify missing test scenarios - Generate bug investigation reports - Trigger test generation for gaps

Inputs: - Bug reports - Test coverage data - Trace metadata

Outputs: - Bug investigation reports - Regression test requests - Coverage gap analysis

See: Detailed Bug Investigator Agent specification in Factory Documentation

Advanced Testing Agents

Load & Performance Testing Agent

Role: Measures throughput, latency, system resource limits

Responsibilities: - Execute load tests - Measure system performance - Test scalability limits - Generate performance reports - Identify performance bottlenecks - Validate SLO compliance

Inputs: - Deployed services - Performance requirements - Load test scenarios

Outputs: - Performance test results - Performance reports - Bottleneck analysis

See: Detailed Load & Performance Testing Agent specification in Factory Documentation

Resiliency & Chaos Engineer Agent

Role: Applies retry policies, latency injection, resiliency breakers

Responsibilities: - Execute chaos engineering tests - Inject failures and latency - Test retry policies - Validate circuit breakers - Test fault tolerance - Generate resiliency reports

Inputs: - Deployed services - Resiliency requirements - Chaos test scenarios

Outputs: - Chaos test results - Resiliency reports - Fault tolerance analysis

See: Detailed Resiliency & Chaos Engineer Agent specification in Factory Documentation

QA Agent Workflow Summary

Phase Agents Involved Output
Test Generation Test Generator, Test Case Generator Test suites
Test Execution Test Automation Engineer Test results
Coverage Validation Test Coverage Validator Coverage reports
Security Review Security Engineer, Penetration Tester Security reports
Compliance Check Privacy Compliance Agent Compliance reports
Code Review Code Reviewer Review feedback
Bug Investigation Bug Investigator Investigation reports
Performance Testing Load & Performance Tester Performance reports
Chaos Testing Chaos Engineer Resiliency reports

Evaluation Criteria and Heuristics

QA agents evaluate code against these criteria:

Architecture Compliance

  • Layer Violations - Domain must not depend on Infrastructure
  • Bounded Context Boundaries - No cross-context dependencies
  • Pattern Adherence - Clean Architecture, DDD patterns followed

Code Quality

  • Code Structure - Proper organization and naming
  • Complexity - Cyclomatic complexity within limits
  • Duplication - No code duplication
  • Documentation - Code comments and documentation present

Test Coverage

  • Unit Tests - All domain logic has unit tests
  • Integration Tests - Critical paths have integration tests
  • Edge Cases - Edge cases are tested
  • Test Quality - Tests are meaningful and maintainable

Observability

  • Logging - Structured logging present
  • Tracing - Distributed tracing implemented
  • Metrics - Performance metrics collected
  • Health Checks - Health check endpoints present

Security and Compliance

  • Security Patterns - Security patterns implemented
  • Secret Management - No hardcoded secrets
  • Input Validation - Input validation present
  • Error Handling - Proper error handling

Important

Core Evaluation Heuristics: 1. Layer Dependencies - Domain → Application → Infrastructure (one-way) 2. Test Coverage - Minimum 80% coverage for domain logic 3. Observability - All operations must be logged, traced, and metered 4. Security - No hardcoded secrets, input validation, proper error handling 5. Documentation - README, API docs, and code comments present

KPIs and Evaluation Criteria

KPI Description How We Measure
Defect Density Issues found per unit of code Number of issues / lines of code
Issues Caught Early Issues caught by QA vs in production QA issues / production issues ratio
Time to Feedback Speed of QA feedback on Factory run Time from code generation to QA report
Test Coverage Percentage of code covered by tests Code coverage metrics
Architecture Compliance Adherence to architectural patterns Compliance check pass rate
False Positive Rate Incorrect issue identifications False positives / total issues

Risks and Anti-Patterns

Risk 1: Over-Reliance on QA Agents

Problem: Assuming QA agents catch all issues without human review.

Solution: QA agents are a safety net, not a replacement for human review. Always review critical code.

Risk 2: Low Signal-to-Noise

Problem: QA feedback is too generic or includes too many false positives.

Solution: Continuously improve QA heuristics. Focus on high-value checks.

Risk 3: Missing Critical Issues

Problem: QA agents miss important issues.

Solution: Combine automated QA with human review. Use QA as first pass, not final pass.

Risk 4: Slow Feedback

Problem: QA takes too long, slowing down development.

Solution: Optimize QA checks. Run fast checks first, detailed checks later.

Warning

Critical Risk: Over-reliance on QA agents without human review. QA agents are powerful but not perfect. Always have human architects and engineers review critical code, especially domain logic and security-sensitive areas.