Skip to content

Reporting and Analytics

This document defines key KPIs and analytics for the Personal Agents Platform domain. It is written for product managers, business analysts, and engineers building reporting and analytics capabilities.

Reporting and analytics provide visibility into platform performance, user engagement, agent effectiveness, and business metrics. This document defines KPIs, data sources, and dashboard requirements.

Note

Analytics are built on events from domain services and Audit Platform. Real-time dashboards provide immediate visibility, while historical reports enable trend analysis and cost optimization.

Key KPIs

KPI Description Calculation Target
Active Users Monthly active users Count of unique users with at least one agent run in period Track growth
Agent Runs per User Average agent runs per user per month Total runs / Active users 10-20 runs/month
Suggestion Approval Rate Percentage of suggestions approved (Approved / Total) × 100 > 60%
Pack Adoption Rate Percentage of users with at least one pack (Users with packs / Total users) × 100 > 70%
Agent Run Success Rate Percentage of successful agent runs (Successful / Total) × 100 > 95%
Average Run Duration Average time for agent execution Average(CompletedAt - StartedAt) < 30 seconds
Token Efficiency Average tokens per run Average(TokensUsed per run) Optimize based on cost
Cost per User Average AI costs per user per month Sum(Cost per run) / Active users Track and optimize
Free-to-Pro Conversion Percentage of free users upgrading to Pro (Upgrades / Free users) × 100 > 5%
User Retention (30-day) Percentage of users active after 30 days (Active after 30 days / New users) × 100 > 40%

Data Sources and Events

Domain Events

Events from Domain Services:

Event Source Data Captured
UserCreated Personal Identity & Profile Service User ID, email, subscription tier, created date
AgentRunCreated Agent Orchestration Service Run ID, user ID, agent ID, trigger type
AgentRunStarted Agent Orchestration Service Run ID, started timestamp
AgentRunCompleted Agent Orchestration Service Run ID, completed timestamp, tokens used, cost, status
AgentRunFailed Agent Orchestration Service Run ID, failed timestamp, error details
SuggestionCreated Suggestion & Approval Service Suggestion ID, user ID, run ID, type, priority
SuggestionApproved Suggestion & Approval Service Suggestion ID, approval timestamp, decision
SuggestionRejected Suggestion & Approval Service Suggestion ID, rejection timestamp
PackInstalled Pack & Workflow Service Pack ID, user ID, installation timestamp
ConnectorAuthorized Connector Service Connector ID, user ID, connector type, authorization timestamp
SubscriptionUpgraded Personal Identity & Profile Service User ID, from tier, to tier, upgrade timestamp

Audit Platform Events

Events from Audit Platform:

Event Type Description Use Case
User Actions User login, pack installation, suggestion approval User behavior analysis
Agent Run Events Agent execution, token usage, cost tracking Agent performance and cost analysis
System Events Pack installation, connector authorization System activity tracking
Privacy Events Data access, data export, account deletion Compliance and privacy reporting

See: Audit Platform for audit platform details.

Data Pipeline

ETL Flow:

  1. Domain services emit events to event bus (Azure Service Bus)
  2. Events consumed by Dashboard & Analytics Service
  3. Events transformed and stored in data warehouse (Azure SQL Database)
  4. Dashboards query data warehouse for metrics
  5. Real-time metrics calculated from event stream

See: Event-Driven Mindset for event patterns.

Dashboards and Views

User Dashboard

Purpose: User-facing dashboard showing their agents, packs, runs, and suggestions

Metrics:

  • Active packs count
  • Active agents count
  • Recent agent runs (last 7 days)
  • Pending suggestions count
  • Usage this month (runs, tokens, cost)
  • Usage vs. subscription limits

Widgets:

  • Packs Overview - List of installed packs with status and last run
  • Agents Overview - List of agent instances with status and last run
  • Recent Runs - Timeline of recent agent runs with status and results
  • Suggestions Inbox - Pending suggestions requiring approval
  • Usage Summary - Runs/month, tokens used, cost, vs. limits
  • Run History - Detailed run history with filters

Views:

  • Dashboard home (overview)
  • Packs detail view
  • Agents detail view
  • Runs detail view
  • Suggestions inbox
  • Usage and billing

Operations Dashboard

Purpose: Internal operations dashboard for platform management

Metrics:

  • Total active users
  • Total agent runs (today, week, month)
  • Total tokens used and costs
  • Suggestion approval rates
  • Agent run success rates
  • Subscription tier distribution
  • User retention metrics

Views:

  • Platform health overview
  • User engagement metrics
  • Agent performance metrics
  • Cost and usage trends
  • Subscription and revenue metrics
  • Error and failure analysis

Agent Performance Dashboard

Purpose: Dashboard for analyzing agent performance and effectiveness

Metrics:

  • Agent run success rate by agent type
  • Average tokens per run by agent type
  • Average cost per run by agent type
  • Suggestion generation rate by agent type
  • Suggestion approval rate by agent type
  • Most used agents and packs

Views:

  • Agent performance comparison
  • Cost analysis by agent type
  • Suggestion effectiveness
  • Popular agents and packs
  • Performance trends over time

Cost Analytics Dashboard

Purpose: Dashboard for cost tracking and optimization

Metrics:

  • Total AI costs (daily, weekly, monthly)
  • Cost per user by subscription tier
  • Cost per agent run
  • Token usage trends
  • Cost by agent type
  • Cost optimization opportunities

Views:

  • Cost overview and trends
  • Cost breakdown by user tier
  • Cost breakdown by agent type
  • Token usage analysis
  • Cost optimization recommendations

Analytics Queries

User Engagement Analytics

Query Examples:

  • Monthly active users (MAU)
  • Daily active users (DAU)
  • User retention cohorts
  • Agent runs per user distribution
  • Pack adoption by user segment
  • Suggestion interaction patterns

Agent Performance Analytics

Query Examples:

  • Agent run success rate by agent type
  • Average run duration by agent type
  • Token usage distribution
  • Cost per run by agent type
  • Suggestion generation and approval rates
  • Most effective agents (by approval rate)

Business Analytics

Query Examples:

  • Free-to-Pro conversion funnel
  • Subscription tier distribution
  • Revenue per user (RPU)
  • Customer lifetime value (LTV)
  • Churn analysis
  • User acquisition sources

Cost Analytics

Query Examples:

  • Total AI costs over time
  • Cost per user by tier
  • Cost per run trends
  • Token usage efficiency
  • Cost optimization opportunities
  • ROI by subscription tier

Reporting Requirements

User Reports

Available Reports:

  • Usage Report - Monthly usage summary (runs, tokens, cost)
  • Run History Report - Detailed run history with filters
  • Suggestion Report - Suggestion activity and approval rates
  • Cost Report - Cost breakdown and trends

Operations Reports

Available Reports:

  • Platform Health Report - System performance and reliability
  • User Engagement Report - User activity and retention
  • Agent Performance Report - Agent effectiveness and costs
  • Business Metrics Report - Subscription, revenue, conversion metrics

Compliance Reports

Available Reports:

  • Audit Trail Report - Complete audit log for compliance
  • Data Access Report - User data access logs
  • Privacy Report - Data retention and deletion logs