10 Questions to Ask Before Your Firm Buys an AI Tool
The questions that matter for a legal AI purchase specifically — covering data processing, hallucination risk, jurisdictional coverage, liability, and supervision obligations. Get the answers in writing before you sign.
Most firms evaluate AI tools on features and price, then discover the real questions — where their data actually goes, what happens if the tool gets something wrong, what it costs to leave — only after signing. The questions below are the ones that matter for a legal AI purchase specifically, not a generic software purchase, because the product touches client confidentiality, professional liability, and work product in ways ordinary practice management software does not.
Ask a vendor these ten questions before you sign anything, and get the answers in writing.
1. Where is our data processed and stored, and is there a signed data processing agreement?
This is the first question, not an afterthought. Confirm the country or region where data is processed and stored, whether that location is covered by an adequacy arrangement or requires additional safeguards under your data protection law, and whether the vendor will sign a data processing agreement rather than pointing you to a standard terms of service page. A vendor that cannot answer this clearly in the sales process will not answer it better once you are a paying customer.
2. Does the vendor use our inputs or outputs to train its models, and can that be disabled?
Some AI products use customer data to improve the underlying model by default. For a law firm, this can mean client-confidential material becomes part of a training set that other customers’ outputs are drawn from, even indirectly. Ask specifically whether this can be contractually and technically disabled, not just whether the vendor “takes privacy seriously.” Get the answer in the contract, not the marketing page.
3. Is the output grounded in verifiable sources, or generated purely from training data?
Tools that retrieve from a live, verified source — case law databases, legislation, your own document set — before generating an answer produce output you can actually check against something. Tools that generate purely from training data produce fluent answers with no built-in way to trace where a claim came from. Ask the vendor to show you, live, how a citation or factual claim in the output links back to a source document. If they cannot demonstrate this, assume the tool carries the same hallucination risk as a general-purpose chatbot.
4. What is the tool’s actual accuracy track record, and against what benchmark?
Vendor accuracy claims are marketing until they are backed by independent testing. The Vals Legal AI Report, first published in February 2025, benchmarked several legal AI tools including Harvey Assistant and CoCounsel against a lawyer baseline across tasks including document Q&A, redlining, and chronology generation, and found meaningful performance differences between tools on the same tasks. Ask whether the vendor has been through independent benchmarking, and be sceptical of a vendor whose only accuracy evidence is its own internal testing.
5. What jurisdictions and legal sources does the tool actually cover?
A tool trained and tested primarily on US law will not perform the same way against English case law, Indian statutes, or Singapore judgments, even if the marketing describes it as a general legal research assistant. Ask specifically which case law databases, statute repositories, and regulatory sources the tool draws from for your jurisdiction, and ask for a live demonstration using a query from your own practice area, not a prepared demo script.
6. What security certifications does the vendor hold?
ISO 27001 and SOC 2 are the two most commonly referenced standards for evaluating whether a vendor has a systematic information security management practice, rather than an ad hoc one. Neither certification guarantees the product is right for your firm, but the absence of either — for a vendor handling confidential client data at scale — is a legitimate reason for caution.
7. Who is liable, contractually, if the tool produces an error that affects a client matter?
Your professional responsibility to the client does not transfer to the vendor no matter what the contract says. But the commercial question of who bears the cost when a tool error causes a loss — whether the vendor’s liability is capped, and at what amount — is a separate and important question for your firm’s own risk position. Read the limitation of liability clause specifically, not just the marketing claims about accuracy.
8. How is pricing structured, and what does that mean for client billing?
Per-seat, per-query, and enterprise flat-fee pricing models each create different incentives and different billing questions. If a tool materially reduces the time a task takes, your existing hourly billing approach may no longer reflect the work involved, which is a client communication issue as much as a pricing one. Understand the pricing model before you commit, and work out how it will be reflected — or not — in what you charge clients.
9. What happens to our data and our matters if we cancel?
Ask what format your data can be exported in, how long the vendor retains it after cancellation, and whether any matter-specific data, prompts, or outputs remain on the vendor’s systems after you leave. A vendor that cannot give a clear answer to this question is not a vendor you want your firm’s matter history locked into.
10. What training and onboarding does the vendor provide, and is it enough to meet our supervision obligations?
Regulators in multiple jurisdictions expect lawyers using AI tools to understand them well enough to catch when they are wrong, and expect firms to train staff before deployment, not after an incident. Ask what onboarding the vendor provides, whether it is tailored to legal use or generic software training, and whether it is enough on its own or whether your firm needs to build additional training around it. A powerful tool with no real onboarding shifts the entire competence burden onto your own supervision structure.
What to Do with the Answers
Do not evaluate these questions in the abstract. The same tool can present different risk depending on whether it is used for general research, internal knowledge search, contract analysis, or work involving highly confidential client material — so run this checklist separately for each intended use case, not once for the product as a whole. Keep a dated, written record of the answers for each tool your firm adopts. Insurers and regulators increasingly ask whether a firm can produce evidence of vendor due diligence, not just whether the due diligence happened informally in someone’s inbox.
For the governance policy that should sit alongside any tool you adopt, see What Is an AI Governance Policy for a Law Firm, and Do You Need One? For hands-on guidance choosing and using the right tool for a given task, see The AI Bar’s AI Foundations for Lawyers module.