Why does a law firm need its own way of evaluating AI tools?
Because in a law firm the first test of any tool is not ease of use but the lawyer's duty of confidentiality. Article 36 of the Turkish Attorneys Act (Law No. 1136) prohibits lawyers from disclosing matters entrusted to them or learned through their professional duties, and that duty is not suspended when a document is uploaded to an AI tool. A law firm's first question is not "does this speed us up?" but "which files can go into this tool, and under what safeguards?"
In August 2026 the Union of Turkish Bar Associations (TBB) published an advisory AI use guide for lawyers. It states plainly that confidentiality neither disappears nor weakens when data is entered into an AI tool, and sorts uses into four risk groups: free auxiliary use, limited and controlled use, use subject to special safeguards and, where needed, client consent, and prohibited use. Its tool selection criteria are data sources, data location, model training, security, transparency, support for human oversight and contractual safeguards. The ten questions below turn those criteria into questions you can put to a vendor and require answered in writing.
A disclosure first: Hatson Tech has its own LegalTech product for due diligence and contract review, Caseon. We have therefore written this list to favour no product; every tool, Caseon included, should face the same questions and scoring. We explain where Caseon fits at the end.
Which questions cover client confidentiality, KVKK and data location?
The first three questions decide whether a tool can be used on files containing client data at all; if one fails, the other scores no longer matter. Ask for the answers in the draft contract and technical documentation, not verbally during a demo.
1. Are our documents and questions used for model training, and who can access them?
The answer you want is a contract clause stating that prompts, uploaded documents and outputs are not used to train models or develop the product; the TBB guide asks for this commitment explicitly. A settings toggle is not enough on its own: settings change, a clause stays. Also ask which of the vendor's staff can access your data and when, where document content goes when you open a support ticket, how long data is kept and how deletion requests are handled. Free tools that staff use on personal accounts belong here too: without an approved-tools list, even a strong contract does not protect the firm.
2. What are our and the vendor's roles under KVKK, and is there a data processing agreement?
Usually the firm is the data controller and the vendor processing data on its behalf is the processor, but the role is decided by whose instructions the data is processed under and for what purpose, not by a contract's title. The Turkish Personal Data Protection Authority's generative AI guide, published in November 2025, explains the distinction and which decisions a controller can leave to a processor by contract. It matters because, under Article 12 of the Personal Data Protection Law (KVKK, Law No. 6698), a controller whose data is processed by someone else on its behalf is jointly responsible with them for security measures. Ask for the draft processing agreement and the sub-processor list, and find out in writing how changes to that list are notified, how quickly you will hear about a breach and how data is deleted when the contract ends.
3. In which country and on which infrastructure is the data processed and stored?
The answer should not be a single country name: document storage, model inference, backups, logs and support access should each be stated. As the TBB guide notes, cloud language model services often process data abroad. KVKK Article 9, amended in 2024, allows transfers abroad on the basis of an adequacy decision, appropriate safeguards or a narrow set of incidental cases; a standard contract, one of those safeguards, must be notified to the Authority within five business days of signing. For highly sensitive matters such as criminal, family, child, health, migration and trade-secret files, the guide recommends closed, auditable systems with a defined data location, and otherwise limiting use to general research. In practice the options are the vendor's cloud, an environment dedicated to the firm with its location written into the contract, or the firm's own servers. Which is enough depends on the classes of files you plan to put in.
How do you test answer accuracy and lawyer approval?
Measure accuracy on your own files, where you know the answers, not on the vendor's demo, and look for lawyer approval as a concrete step in the software rather than a sentence in a policy.
4. Does every answer show the passage it relies on?
A good tool shows the supporting paragraph next to each finding and opens the document at that place in one click. A general "sources" list under the answer is weaker: the lawyer has to work out again which sentence rests on which source. In the trial, check not only whether a source is shown but whether it actually supports the claim; an on-topic citation that does not support the conclusion is a common failure, and our article on verifying citations in RAG answers explains how to test for it. If the tool also offers a legislation and case-law database, ask where that data comes from, how often it is updated and under what licence.
5. What controls exist against hallucination, and how did you measure them?
Citing sources reduces hallucination but does not remove it. A 2024 study by Stanford researchers reported that widely used commercial legal research tools in the US, built on retrieval-augmented generation (RAG), produced fabricated or incorrect information in between 17% and 33% of the researchers' test queries. The KVKK guide illustrates hallucination with a lawyer meeting court decisions that do not exist, and under the TBB guide submitting fabricated case law may, depending on the facts, be treated as a serious breach of professional care. Ask the vendor: does the tool say "not found" when the documents offer no support? Are docket and decision numbers checked against a source that confirms they exist? Is quoted text separated from generated text? How did you measure your error rate, and on what data? Then repeat the measurement on your own question set.
6. How does lawyer approval work inside the software?
The TBB guide says AI output must not reach a client, court, authority or third party without a lawyer verifying it, and the software should make that visible. Each finding should carry a status such as pending, corrected or approved; the system should record who approved it, when, and what was corrected; and unapproved content should be clearly marked or blocked on export. Check whether a second approver can be assigned for the "second pair of eyes" review the guide recommends on high-risk files. The guide also says trainees and staff should not use AI on client files without a lawyer's explicit instruction and supervision, which requires role-based permissions.
7. Who asked what, and when: is there an audit trail?
For critical uses, the TBB guide recommends recording the file, purpose, tool, data type, verification and final decision. The tool should capture most of this itself: user, matter, documents used, question, answer version, sources shown, approver and timestamps. Logs should be exportable, retention configurable and access restricted, because the logs themselves contain privileged information. They let you trace a faulty output back to its user and source.
Why do Turkish legal language, integration and exit terms decide so much?
Because a tool can look good in an English demo, while its real cost depends on how it behaves on your Turkish files, inside your systems and when the contract ends.
8. How does it handle Turkish legal text?
Turkish agglutination, statute abbreviations (TBK, HMK, İİK, TTK), older terms that survive in legislation (such as "tevdi" and "muvafakat") and scanned, stamped or annotated pages can all trip up general-purpose models. Put these into the trial: a question that turns on the difference between two concepts (for example termination of a contract versus rescission), the current wording of a clause amended by an addendum, a date and amount from a scanned page, and data held in a table in a contract annex. Check whether Turkish characters (ı, ğ, ş) are read correctly from scans, and whether answers use Turkish legal concepts or another legal system's terms.
9. Does it fit our existing systems and permission structure?
Ask how it works with your document management system, email, word processor and single sign-on, whether documents arrive manually or automatically, and which formats of documents downloaded from UYAP, the national judiciary system, it can read. The point most specific to law firms is matter-level permission: a lawyer screened off a matter because of a conflict of interest must not reach its documents through the AI assistant either. Test it by asking the same question as two users. Hiding a document on screen is not enough; it must not appear in an unauthorised user's search results at all.
10. When the contract ends, how do we get our data back and have it deleted?
Exit terms have to be written before signing. Ask in which open formats (for example spreadsheets, editable documents, JSON) documents, review tables, notes, approvals and audit logs can be exported, how and when indexes, embeddings and backups are deleted, and whether deletion is confirmed in writing. Deletion, export and audit trails are also among the vendor questions in the TBB guide. Our guide to choosing an AI development partner covers exit and intellectual property clauses in general.
How do you use the evaluation scorecard?
Score each question from 0 to 3, multiply by its weight, and treat the first three questions as knock-out criteria too. Score evidence, not claims:
- 0: No answer, or an unclear one.
- 1: A verbal answer or marketing copy.
- 2: A written document or contract clause.
- 3: Verified in your own trial or a technical review.
| Question | Weight | Evidence to request | Knock-out |
|---|---|---|---|
| 1. Model training and access | 3 | Contract clause, access and retention policy | Yes, at least 2 |
| 2. KVKK roles | 3 | Processing agreement, sub-processor list | Yes, at least 2 |
| 3. Data location | 2 | Location by component, basis for transfer | Yes, at least 2 |
| 4. Source citation | 3 | Trial with your own documents | No |
| 5. Hallucination controls | 3 | Vendor's method and your own question set | No |
| 6. Lawyer approval | 2 | Live demonstration of the approval flow | No |
| 7. Audit trail | 2 | Sample log export | No |
| 8. Turkish legal language | 3 | Trial with Turkish and scanned documents | No |
| 9. Integration and matter permissions | 2 | Permission test with two users | No |
| 10. Exit and deletion | 2 | Contract clause, sample export | No |
The weights are a starting point; a firm with heavy criminal, family or health work might raise data location. The maximum total is 75, but a tool that fails any knock-out criterion should not be used on files with client data, whatever its total.
Hypothetical example: how do two tools compare on the same scorecard?
This example is hypothetical; the tools and scores are invented to show the method. A commercial and corporate law firm wants help with document review in acquisition due diligence. It scores two candidates: A, a general-purpose chat assistant on an enterprise plan, and B, a legal document review tool that works on the firm's own documents.
| Question (weight) | A score | A weighted | B score | B weighted |
|---|---|---|---|---|
| 1. Model training and access (3) | 2 | 6 | 3 | 9 |
| 2. KVKK roles (3) | 1 | 3 | 2 | 6 |
| 3. Data location (2) | 0 | 0 | 3 | 6 |
| 4. Source citation (3) | 1 | 3 | 3 | 9 |
| 5. Hallucination controls (3) | 1 | 3 | 2 | 6 |
| 6. Lawyer approval (2) | 1 | 2 | 3 | 6 |
| 7. Audit trail (2) | 1 | 2 | 2 | 4 |
| 8. Turkish legal language (3) | 2 | 6 | 2 | 6 |
| 9. Integration and matter permissions (2) | 2 | 4 | 1 | 2 |
| 10. Exit and deletion (2) | 2 | 4 | 1 | 2 |
| Total (maximum 75) | 33 | 56 |
Read the knock-out rows first. A fails on KVKK roles (1) and data location (0), so it cannot be used on files with client data, though it can stay on the approved list for what the TBB guide calls free auxiliary use, where no client data is entered: general legislative research, summaries of public texts, plain-language rewriting. B passes and scores clearly higher, but is weak on integration and exit. So before signing, the firm asks for export formats and written deletion confirmation in the contract, plus a written plan for connecting its document management system. That is the scorecard's real value: it does not just pick a winner, it shows what to negotiate.
How should the trial with our own files be run?
Run it on closed matters whose outcomes you know, after confidentiality and processing terms are in writing. Anonymise documents first if needed; as the TBB guide points out, anonymisation is more than deleting names and ID numbers, because dates, places, amounts, company names and unique facts can also identify a person.
- Choose two or three closed matters that reflect your real mix of work: contracts, annexes, scans, correspondence.
- Have lawyers write the answers to a question list in advance. Include questions the documents cannot answer; the right response there is "not found".
- Ask every candidate the same list in the same way, recording answers and sources.
- Have two lawyers independently mark each answer correct, partly correct or wrong, and check whether the source supports it.
- Test permissions with two users: a matter one of them cannot access must never appear in their answers.
- Actually run the export and deletion at the end.
- Note the lawyer time spent verifying answers. A fast tool whose every sentence needs rechecking may not reduce the workload.
Which vendor answers should raise a flag?
- Absolute claims such as "the tool does not hallucinate".
- Reluctance to name sub-processors or the country of processing in writing.
- Refusing a trial on your documents and offering only a prepared demo.
- Promises such as "predicts case outcomes" or "guaranteed results", which the TBB guide lists among uses that conflict with the advertising ban and the dignity of the profession.
- Export only in a vendor-specific format, or no access to audit logs.
- Offering a certificate as proof of security on its own; the guide states that accreditation does not mean a tool is absolutely secure or error-free.
Where does Caseon fit on this list?
Caseon is Hatson Tech's own LegalTech product for due diligence and contract review. It is still in development, which is why we suggest evaluating it with the same ten questions. It also brings contract drafting and reporting into the same workspace. Every answer is shown with the document excerpt it is based on, and the lawyer reviews the sources and approves the work. Those two features map directly to questions 4 and 6, but should only score 3 once verified in a trial with your documents. For KVKK roles, data location, audit trail and exit, ask us for written answers as you would any vendor. Our Caseon case study describes how the product works.
What should you do once the scorecard is filled in?
Complete it for two or three candidates, review the results with the partners, and introduce the chosen tool with a written firm policy. The TBB guide recommends a policy suited to the firm's size: an approved-tools list, prohibited data types, a permission matrix for trainees, assistants, lawyers and partners, a log of critical uses, and steps to follow after a faulty output or a data leak. Also confirm in writing with your insurer whether your professional liability policy covers AI-related errors. For a broader view of opportunities and risks, see our article on AI in law.
If you need less a packaged product and more an assistant that works on your own archive, permission structure and types of work, our RAG and enterprise semantic search page explains how we build systems that show the source behind every answer. If you would like to talk through which route fits, the first meeting is free.