AI in contract review: control, evidence and continuous improvement

For procurement, legal and IT teams, adopting AI for contract review raises practical questions: which service processes the data, whose records can it access, and how should an answer be checked?

Phoenix provides optional AI-assisted contract analysis, comparison and question answering. Our approach is to make the controls, limitations and evidence visible, including the work still required.

Be clear about the deployment

The current integration uses the OpenAI API directly. The configured model and version would be documented for an organisation’s technical approval.

A second approach is customer-managed Azure OpenAI, sometimes described as bring your own model, or BYOM. Phoenix would connect to an organisation’s approved Azure OpenAI deployment. This is proposed and requires integration and acceptance testing. Hosting the Phoenix application on Azure does not itself mean its AI requests use Azure OpenAI.

For the proposed Azure approach, acceptance would cover the selected model, deployment type, processing geography, retention and abuse monitoring. Regional, Data Zone and Global deployments have different processing-location implications.

Keep control of AI and data access

Phoenix’s built-in AI calls can be disabled through application configuration while retaining ordinary document, record and workflow functionality. The proposed Azure integration would retain this control. Separately connected external AI agents need their own access restrictions or revocation; switching off built-in AI does not automatically disconnect them.

For the built-in workflow, Phoenix applies workspace, project and record permissions before retrieving evidence. Client-scoped conversations further limit that evidence to the selected client and, where applicable, project. Application and database controls enforce these restrictions independently of model instructions.

Only evidence permitted through that workflow is submitted to the OpenAI API. The same Phoenix restrictions would apply before submission to a customer’s Azure endpoint, with Azure identity and resource permissions providing additional controls. Acceptance testing would cover cross-client isolation, privileged roles, permission changes and revocation. External agents require equivalent scope and access review.

Separate request context from model training

Phoenix uses a pretrained foundation model and does not currently train or fine-tune it. Authorised documents, records and selected standard positions provide context for individual analysis requests.

For the direct API route, OpenAI states that API data is not used for model training unless a customer opts in. Retention is a separate question: default abuse-monitoring logs can contain customer content and are generally kept for up to 30 days, with stated exceptions. Some API features also retain application state. Alternative retention controls depend on eligibility and approval. Account settings and processing arrangements would be confirmed during due diligence.

For Azure OpenAI, Microsoft states that inputs and outputs are unavailable to OpenAI and are not used to train foundation models without permission or instruction. Phoenix would supply authorised context without training the foundation model. The customer’s chosen configuration and applicable Microsoft agreement would need review.

Ground answers in evidence, then review them

Phoenix combines several controls to support reviewable answers:

• Evidence grounding: relevant, authorised records and documents, with instructions to base answers on that evidence.
• Explicit uncertainty: instructions to flag missing information and decline unsupported conclusions.
• Source validation: application checks on cited text and document versions, alongside references for human verification.
• Controlled comparisons: assessment against selected standard positions.
• Human review: checks of substantive conclusions, interpretation and completeness against source documents.

These controls do not remove the need for judgement. Microsoft’s guidance explains that trusted-source context can reduce inaccurate answers without eliminating them. A model can misinterpret real wording, attach evidence to the wrong claim or mishandle conversation context.

What the review has shown so far

In the AI responses reviewed so far, 94.5% were verified as correct without identified quality defects. The remaining 5.5% contained currency-labelling, citation or context-handling defects, recorded as answer-quality issues for correction.

No invented contract terms or fabricated source material were identified within those reviewed responses. Phoenix also demonstrated appropriate acknowledgement of insufficient evidence rather than supplying an unsupported answer.

These are limited observations from the reviewed online implementation. They do not establish a universal zero-hallucination rate, guaranteed production accuracy or a measured improvement attributable to prompting. They do not measure availability or the proposed Azure integration.

To measure the contribution of individual controls, we would compare configurations using the same representative questions, source documents and model version. Azure BYOM would require its own evaluation.

Be honest about training-data provenance

Phoenix does not maintain the foundation model’s training corpus. OpenAI describes its training sources as public internet material, information obtained through partnerships, and content supplied or generated by users, trainers and researchers. Those disclosures do not provide item-by-item licensing evidence.

We therefore cannot certify that every training item is licensed or open source. Public availability is not an open-source licence. Provider disclosures, applicable terms and contractual protections form part of an organisation’s approval process. Using Azure does not itself establish training-data provenance or make a model open source. Customer documents supplied for analysis must also come with appropriate rights and authority.

Make improvement measurable

Phoenix has implemented local tooling for citation-validation monitoring and reporting of human-reviewed unsupported claims, answer coverage and abstentions. Deployment and monitoring activation remain pending, alongside establishing a representative operational baseline.

Software verification recorded 263 passing tests, including 17 new checks, and successful builds. Those checks validate software behaviour and made no live AI calls. They are separate from the response-quality results above.

Our next steps are continuous evaluation and refinement: checking answers against sources, investigating defects, improving prompts and evidence retrieval, and retaining reviewed examples as regression tests. User feedback does not automatically retrain the foundation model. Completed answers, correct abstentions and quality defects will be reported separately.

Through the early-adopter programme, we will work with participating customers to assess accuracy and completeness against their source documents, including whether insufficient evidence is appropriately flagged. Those findings will guide improvements and regression testing as Phoenix develops.