Sample AI Risk Assessment

Organization: Northstar SaaS (fictional company created for the program case study)

Industry: B2B Software-as-a-Service — Enterprise Resource Planning

Employees: ~1,200 globally

AI Use Case: Automated Customer Support Triage System ("NorthstarAssist")

NorthstarAssist is an AI-powered system that receives inbound customer support tickets, classifies them by severity and topic, suggests initial responses from the knowledge base, and routes tickets to appropriate support tiers. The system processes approximately 15,000 tickets per week across three product lines.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

This assessment covers the NorthstarAssist automated triage system, including:

• The machine-learning classification model and its training data pipeline
• Integration with the customer-facing support portal and internal ticketing system
• Suggested response generation from the product knowledge base
• Routing logic and escalation rules
• Data flows between NorthstarAssist and third-party infrastructure providers

Out of scope: Manual support workflows, legacy ticketing processes not connected to NorthstarAssist, and the company's separate sales-analytics AI system.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

Sample Portfolio Artifact — Northstar SaaS (fictional company)

NorthstarAssist uses a fine-tuned transformer-based classification model to analyze incoming support ticket text and metadata. The system performs three primary functions:

1. Severity classification: Assigns priority levels (P1–P4) based on ticket content, customer tier, and historical patterns.
2. Topic categorization: Maps tickets to product modules, known issue categories, and resolution playbooks.
3. Response suggestion: Generates draft responses by matching ticket content against the verified knowledge base, presented to human agents for review before sending.

The system does not autonomously send responses to customers. All suggested responses require human review and approval before delivery.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

• Customer names, email addresses, and account identifiers
• Support ticket text (may contain technical details, error messages, business context)
• Customer subscription tier and contract metadata
• Historical ticket resolution data (used for model training)
• Agent performance and response-time metrics
• Product usage telemetry linked to support tickets

Sensitive data note: Customers occasionally include screenshots, credentials, or personally identifiable information in ticket descriptions. The system processes this text without redaction during classification. A data-minimization review is recommended.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

Sample Portfolio Artifact — Northstar SaaS (fictional company)

Sample Portfolio Artifact — Northstar SaaS (fictional company)

• Model API endpoints require authentication and are accessible only from the internal network
• Training data is stored in an encrypted data lake with role-based access controls
• Input validation is limited; adversarial prompt injection through ticket text has not been formally tested
• Model weights are stored in a shared ML infrastructure environment without dedicated isolation
• Logging captures classification decisions but does not currently retain the full input text for audit purposes

Recommendation: Conduct adversarial testing against the classification model and implement input sanitization for known prompt-injection patterns.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

• The system processes personal data covered under the company's customer data processing agreements
• Historical ticket data used for training may contain personal information that was not explicitly consented for AI training purposes
• Data retention for training datasets exceeds the standard support-data retention period
• Cross-border data transfers occur when tickets from EU customers are processed by the US-hosted model
• Customer-facing privacy notices do not currently disclose the use of AI in support triage

Recommendation: Update privacy notices, review training-data consent basis, and implement data-minimization practices for model training pipelines.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

• Suggested responses require human approval before delivery — no autonomous customer communication
• Severity classifications are visible to agents, who can manually override the AI assignment
• No formal escalation trigger exists when the model's confidence score falls below a defined threshold
• Agents are not currently trained to recognize when AI classifications may be unreliable
• Override patterns are logged but not systematically reviewed for model improvement

Recommendation: Implement confidence-threshold alerting, add agent training on AI limitations, and establish a regular override-pattern review process.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

• The base model was provided by a third-party ML platform vendor under an enterprise license
• Fine-tuning was performed internally using Northstar's proprietary ticket data
• The ML platform vendor has access to model performance telemetry but not to raw customer data
• Cloud infrastructure for model hosting is provided by a separate IaaS vendor
• Vendor contracts do not currently include AI-specific governance requirements or audit rights for model behavior

Recommendation: Add AI-specific clauses to vendor agreements covering model transparency, data handling, audit rights, and incident-notification obligations.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

1. Severity misclassification represents the highest-impact risk, with potential SLA and contractual implications for enterprise customers.
2. Training data governance requires review — consent basis, retention periods, and data-minimization practices are not aligned with current privacy commitments.
3. Human-oversight mechanisms exist but lack formal confidence thresholds, agent training, and systematic override review.
4. Vendor agreements do not include AI-specific governance requirements, limiting visibility into model behavior and third-party data handling.
5. Security testing has not included adversarial or prompt-injection testing against the classification model.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

1. Implement confidence-score thresholds with automatic escalation to human review when the model is uncertain.
2. Conduct quarterly accuracy reviews of severity classification against actual ticket outcomes.
3. Perform adversarial testing and input-sanitization review before the next model update.
4. Update customer-facing privacy notices to disclose AI-assisted support triage.
5. Review and align training-data retention with the company's data-retention policy.
6. Add AI-specific governance clauses to ML platform and infrastructure vendor agreements.
7. Develop agent training materials covering AI classification limitations and override procedures.
8. Establish a monthly override-pattern review to identify systematic classification errors.
9. Implement input-text redaction for sensitive data before classification processing.
10. Create a model-change management process requiring risk re-assessment for significant updates.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

After implementation of the recommended controls, residual risk is expected to decrease from Moderate-High to Moderate-Low. Remaining risks include the inherent uncertainty of ML classification accuracy, the possibility of novel attack vectors not covered by current adversarial testing, and the dependence on agents correctly applying override procedures.

Residual risk should be re-evaluated quarterly and after any significant model update, data-pipeline change, or expansion of the system's scope.

Sample Portfolio Artifact — Northstar SaaS (fictional company)

Sample Portfolio Artifact — Northstar SaaS (fictional company)