Sample AI Vendor Review
This fictional sample demonstrates the type and quality of portfolio artifact participants learn to produce in the AI Governance Career Lab program.
Disclosure: This fictional sample is provided to demonstrate the type and quality of work participants will learn to produce. CogniFlow is a fictional vendor. This review is not an endorsement of or commentary on any real vendor, product, or service. It is not legal advice or a substitute for professional due diligence.
1. Vendor overview
Company: CogniFlow Inc. (fictional) — a cloud-based machine-learning platform provider offering model training, hosting, inference APIs, and model-lifecycle management tools.
Headquarters: San Francisco, CA (USA)
Data center locations: US-East (Virginia), US-West (Oregon), EU-West (Frankfurt)
SOC 2 Type II: Current (last audit completed Q1 of current year)
ISO 27001: Certified (scope includes ML platform and inference infrastructure)
AI-specific certifications: None at time of review
Northstar relationship: CogniFlow provides the ML platform used to host and serve the fine-tuned classification model for NorthstarAssist. Northstar performs model training internally using proprietary ticket data, then deploys trained models to CogniFlow's inference infrastructure. CogniFlow does not have access to raw training data — only the resulting model weights and inference-time inputs.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
2. Scope of this review
This review evaluates CogniFlow across seven governance domains relevant to Northstar's use of the platform for NorthstarAssist:
- Data handling and processing practices
- Security controls and infrastructure
- Model transparency and explainability
- Compliance and regulatory alignment
- Incident response and notification
- Contractual governance provisions
- Subprocessor and supply-chain management
This review is informed by CogniFlow's vendor security questionnaire responses, SOC 2 Type II report, published documentation, and a 90-minute review call with CogniFlow's Security Engineering and Customer Success teams.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
3. Data handling and processing
Understanding what data CogniFlow receives, how it processes that data, and where it stores or transmits it is the foundation of this review.
Data received by CogniFlow
| Data Type | Purpose | Contains PII? |
|---|---|---|
| Trained model weights | Deployment and inference serving | No (but may encode patterns from PII-containing training data) |
| Inference inputs (ticket text) | Real-time classification | Yes — customer names, emails, account context may appear in ticket text |
| Inference outputs (classifications) | Returned to Northstar systems | No |
| Platform telemetry | Performance monitoring, billing | No |
Finding: Inference inputs contain unredacted PII
Support ticket text passed to CogniFlow for classification frequently contains customer names, email addresses, and business context. CogniFlow processes this data in-transit for inference but states it is not persisted beyond the inference request lifecycle. However, the current data-processing agreement does not explicitly address inference-input data handling, retention exclusions, or deletion verification.
Data retention and deletion
- CogniFlow states inference inputs are processed in memory and not written to persistent storage
- Model weights are retained for the duration of the deployment and deleted within 30 days of model retirement
- Platform logs (excluding inference content) are retained for 90 days for operational purposes
- CogniFlow does not currently offer a customer-initiated data-deletion verification mechanism
Sample Portfolio Artifact — Northstar SaaS (fictional company)
4. Security controls and infrastructure
Encryption in transit
TLS 1.3 enforced for all API endpoints. Certificate pinning available for enterprise customers.
Encryption at rest
AES-256 for stored model weights. Customer-managed encryption keys (CMEK) available on Enterprise tier.
Access controls
Role-based access control (RBAC) with MFA enforcement. Service accounts use short-lived tokens. Audit logging for all administrative actions.
Network isolation
Dedicated VPC available on Enterprise tier. Northstar uses VPC peering for inference traffic, avoiding public internet traversal.
Penetration testing
Annual third-party penetration test (last completed Q4 prior year). Results shared under NDA upon request.
Model isolation
Models are deployed in containerized environments with namespace-level isolation. However, the underlying compute infrastructure is shared across customers on the same tier. Dedicated compute is available at additional cost but is not currently provisioned for Northstar.
Adversarial / prompt-injection testing
CogniFlow does not perform adversarial testing on customer models. This is treated as the customer's responsibility. No tooling or guidance is provided for adversarial evaluation within the platform.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
5. Model transparency and explainability
Model versioning and lineage
Full model version history with deployment timestamps, rollback capability, and A/B testing support. Each deployment is traceable to a specific model artifact.
Inference logging and auditability
Classification outputs and confidence scores are logged and accessible via API. However, input data is not retained in logs (by design for privacy), which limits post-hoc auditability of specific classification decisions.
Explainability tooling
CogniFlow does not provide built-in explainability features (e.g., SHAP values, attention visualization, feature attribution). Northstar would need to implement explainability independently before deployment or use third-party tools.
Drift detection
CogniFlow offers statistical monitoring for inference-volume changes and latency anomalies. It does not currently offer concept-drift or data-drift detection for classification accuracy. Northstar must implement its own accuracy monitoring by comparing model outputs to ground-truth labels.
Model change notifications
CogniFlow notifies customers of platform-level changes (infrastructure updates, API versioning) via email and status-page updates. Customer model changes are fully controlled by Northstar — CogniFlow does not modify customer models.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
6. Compliance and regulatory alignment
| Domain | Status | Notes |
|---|---|---|
| SOC 2 Type II | Current | Scope includes ML platform and inference infrastructure. Report reviewed — no material exceptions noted. |
| ISO 27001 | Certified | Certificate current. Scope covers platform operations and supporting IT infrastructure. |
| GDPR / Data Processing Agreement | Partial | Standard DPA is in place but does not specifically address inference-input data as a processing activity. EU data processing uses Frankfurt region. Standard Contractual Clauses are referenced but not individually negotiated. |
| EU AI Act readiness | Not addressed | CogniFlow has not published an EU AI Act compliance roadmap or documentation. No guidance provided on how customers using the platform for regulated use cases should approach compliance obligations. |
| ISO/IEC 42001 (AI Management) | Not pursued | CogniFlow stated this certification is "on the roadmap" but provided no timeline. No AI-specific management-system documentation was available for review. |
Sample Portfolio Artifact — Northstar SaaS (fictional company)
7. Incident response and notification
Incident notification timeline
CogniFlow commits to notifying affected customers of confirmed security incidents within 72 hours. However, the contract does not define 'confirmed' or specify notification requirements for suspected incidents, near-misses, or incidents affecting shared infrastructure that may indirectly impact Northstar.
Incident communication channels
Dedicated security-incident email alias and 24/7 on-call security engineering team. Named security contact assigned to Enterprise accounts.
Root-cause analysis sharing
CogniFlow provides post-incident summaries for incidents directly affecting a customer's deployment. Summaries are provided within 10 business days. Full root-cause analysis reports are shared only for Severity 1 incidents and only under NDA.
Customer audit rights
The current contract does not include audit rights for Northstar to independently verify CogniFlow's security controls, incident-response practices, or data-handling claims. CogniFlow positions the SOC 2 report as the primary assurance mechanism.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
8. Contractual governance provisions
The following table summarizes the current state of AI-relevant governance provisions in Northstar's agreement with CogniFlow and identifies gaps to address during contract renewal:
| Provision | Current Status | Renewal Recommendation |
|---|---|---|
| Data-processing agreement | Partial — does not cover inference inputs | Amend to explicitly include inference-input data as a processing activity with defined retention limits |
| Audit rights | Absent | Add contractual right to audit or commission third-party audit annually |
| Data deletion verification | Absent | Require written confirmation of data deletion upon model retirement or contract termination |
| AI-specific liability | Absent | Define liability allocation for model-performance failures, biased outputs, or classification errors caused by platform issues |
| Incident notification SLA | Partial — 72 hours, loosely defined | Tighten definition of "confirmed incident", add notification requirements for suspected incidents and shared-infrastructure events |
| Subprocessor notification | Present — 30-day notice | Maintain; add right to object before subprocessor change takes effect |
| Regulatory-change cooperation | Absent | Add clause requiring CogniFlow to cooperate with Northstar's compliance efforts related to evolving AI regulation (e.g., EU AI Act) |
Sample Portfolio Artifact — Northstar SaaS (fictional company)
9. Subprocessor and supply-chain review
CogniFlow disclosed the following subprocessors relevant to Northstar's deployment:
| Subprocessor | Service Provided | Data Access | Location |
|---|---|---|---|
| Major cloud IaaS provider | Compute, storage, networking infrastructure | Infrastructure-level access (encrypted at rest) | US-East, EU-West |
| Observability SaaS platform | Application monitoring, log aggregation | Platform telemetry only (no inference data) | US |
| CDN provider | API edge routing and DDoS protection | Request metadata (IP, headers) — no payload inspection | Global |
Assessment: Subprocessor list is reasonable and consistent with industry practice. The observability platform does not receive inference-content data, which was verbally confirmed and is consistent with the SOC 2 report data-flow diagrams. The 30-day subprocessor-change notification provision is adequate but should be supplemented with an objection right.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
10. Summary assessment
Overall Vendor Risk Rating: MODERATE
CogniFlow demonstrates strong foundational security practices (SOC 2, ISO 27001, encryption, RBAC) and provides a reliable platform for model hosting and inference. However, governance gaps exist in three areas: (1) contractual coverage of inference-input data processing, (2) absence of AI-specific compliance documentation and audit rights, and (3) limited explainability and adversarial-testing capabilities. These gaps are manageable and addressable through contract negotiation and compensating controls.
Domain-level summary
Data handling (Partial)
Strong in transit and at rest; gap in inference-input data governance and deletion verification
Security controls (Meets)
SOC 2 current, encryption strong, network isolation in place; shared compute is a residual risk
Model transparency (Partial)
Good versioning and lineage; lacks explainability tooling and drift detection
Compliance (Partial)
SOC 2 and ISO 27001 current; no EU AI Act readiness or AI-specific certification
Incident response (Partial)
Reasonable notification process; gaps in audit rights and incident-scope definitions
Contractual governance (Gap)
Multiple AI-specific provisions absent from current agreement
Subprocessor management (Meets)
Transparent subprocessor list; notification process in place
Sample Portfolio Artifact — Northstar SaaS (fictional company)
11. Recommendations
Based on this review, the following actions are recommended before contract renewal is finalized:
Priority 1 — Address before renewal
- Amend DPA to cover inference-input data. Define inference inputs as a processing activity. Specify that inputs are not persisted, retained, or used for any purpose beyond the immediate inference request. Require CogniFlow to provide written confirmation of non-retention upon request.
- Add contractual audit rights. Negotiate the right to conduct an annual audit or engage a mutually agreed third party to verify data-handling, security, and incident-response claims beyond the SOC 2 report.
- Add data-deletion verification. Require CogniFlow to provide written confirmation of complete data deletion (model weights, deployment artifacts, any cached data) within 30 days of model retirement or contract termination.
Priority 2 — Address within 90 days of renewal
- Implement input-data redaction. Deploy PII redaction on inference inputs before they leave Northstar's infrastructure. This is a Northstar-side control that reduces reliance on CogniFlow's data-handling assurances.
- Evaluate dedicated compute. Assess cost and feasibility of moving NorthstarAssist to dedicated compute infrastructure to eliminate shared-tenant risk.
- Request EU AI Act cooperation clause. Add contractual language requiring CogniFlow to cooperate with Northstar's EU AI Act compliance efforts and provide reasonable documentation support.
Priority 3 — Track for future reviews
- Monitor CogniFlow's AI certification roadmap. ISO/IEC 42001 certification and EU AI Act readiness are stated roadmap items. Revisit at next annual review.
- Evaluate alternative explainability solutions. Since CogniFlow does not provide built-in explainability, assess third-party tools that can be integrated into Northstar's model-evaluation workflow.
- Establish a vendor governance scorecard. Standardize the evaluation framework used in this review into a repeatable scorecard for use across all AI vendors.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
12. Conclusion and renewal recommendation
Recommendation: Proceed with renewal, conditional on Priority 1 contract amendments
CogniFlow provides a technically capable and operationally reliable platform for NorthstarAssist's model-hosting needs. The identified governance gaps are not disqualifying — they reflect the current maturity of AI-specific vendor governance across the industry. However, they must be addressed contractually to ensure Northstar's governance program can demonstrate adequate vendor oversight to regulators, auditors, and customers.
If CogniFlow is unable or unwilling to accept the Priority 1 amendments, the AI Governance Lead should escalate to the AI Governance Committee for a risk-acceptance decision before renewal is executed.
Sample Portfolio Artifact — Northstar SaaS (fictional company)
This fictional sample is provided to demonstrate the type and quality of work participants will learn to produce. CogniFlow Inc. is a fictional vendor created for the program case study. This review is not an endorsement of or commentary on any real vendor, product, or service. It is not legal advice, a procurement recommendation, or a substitute for professional due diligence. No real employer information, customer information, confidential material, or proprietary documents are represented.
One-time payment • 14-day satisfaction guarantee