Build a Document Extraction Agent with Claude: Tool Use, OCR, and a Validation Loop
To build a document extraction agent with Claude, give Claude one or two tools that do the pixel work (an OCR call that returns Markdown and one that returns typed JSON), run the Messages API tool use loop until Claude stops asking for tools, and then check the final record in ordinary code before you store it. Claude decides which fields a document needs, notices when a required value is missing, and retries with a different request. The OCR service reads the image. Your code has the last word on whether the output is good enough.
That split matters. Claude can read images on its own through vision, and for a single screenshot that is often fine (see Does Claude Have OCR?). An agent that processes a folder of invoices, ID cards, or inspection notices every night needs output that has the same shape every time, and that is easier to get from an endpoint built for it.
What the agent actually does
A useful extraction agent has a small job description:
- Look at an incoming file and decide what kind of document it is.
- Ask for the fields that document type should contain.
- If a required field comes back missing, decide whether the field is really absent or the request was wrong, and try again once.
- Return one JSON record, or a clear reason it could not.
Everything else (where the files come from, where the records go, retries on network errors) belongs in the code around the agent. Keeping the agent narrow keeps it cheap and predictable.
The tools
Claude’s tool use works by describing each tool with a name, a description, and a JSON Schema for its input. When Claude wants a tool, the response has stop_reason: "tool_use" and a tool_use content block with the arguments. You run the tool and send the result back in a tool_result block that carries the same tool_use_id.
Two tools cover most cases, both backed by OCRskill:
read_documentcallsPOST /ocrand returns Markdown. Claude uses it to classify a document it has not seen before.extract_fieldscallsPOST /ocr.jsonwith afieldslist. Appending?to a field name makes it optional. Leavingfieldsout entirely runs discovery mode, which returns whatever supported fields are present. A missing required field returns a400that includes the OCR text, so Claude can see what was read. The Structured OCR JSON API reference lists the supported fields.
Here are the definitions:
TOOLS = [
{
"name": "read_document",
"description": "OCR the current document image and return its text as Markdown. "
"Use this to identify the document type before extracting fields.",
"input_schema": {"type": "object", "properties": {}, "required": []},
},
{
"name": "extract_fields",
"description": "Extract typed fields from the current document image. "
"Pass field names from the OCRskill field reference. "
"Append ? to a name to make it optional. "
"Pass an empty list to run discovery mode.",
"input_schema": {
"type": "object",
"properties": {
"fields": {"type": "array", "items": {"type": "string"}}
},
"required": ["fields"],
},
},
]Notice that neither tool takes a file path. The code binds each run to one file, so Claude cannot wander off and read something else.
The loop in Python
This uses the official anthropic SDK and requests. Set ANTHROPIC_API_KEY, OCRSKILL_KEY, and CLAUDE_MODEL (any current Claude model from Anthropic’s models overview). You can get an OCRskill key with curl https://api.ocrskill.com/get-key.json.
import json, os, sys
import anthropic, requests
OCR = "https://api.ocrskill.com"
HEADERS = {"Authorization": f"Bearer {os.environ['OCRSKILL_KEY']}"}
client = anthropic.Anthropic()
def run_tool(name, args, path):
with open(path, "rb") as f:
files = {"file": f}
if name == "read_document":
r = requests.post(f"{OCR}/ocr", headers=HEADERS, files=files, timeout=120)
else:
params = {"fields": ",".join(args["fields"])} if args["fields"] else {}
r = requests.post(f"{OCR}/ocr.json", headers=HEADERS, files=files,
params=params, timeout=120)
# Pass 400 bodies back to Claude: they explain what was missing.
return r.text, r.status_code >= 400
SYSTEM = (
"You extract one JSON record from one document image. "
"Call read_document first if you do not know the document type. "
"Then call extract_fields with the fields that type should contain. "
"If a required field fails, retry once with it marked optional. "
"Finish by replying with only the final JSON object, or "
'{"error": "<reason>"} if extraction is not possible.'
)
def extract(path, max_turns=6):
messages = [{"role": "user", "content": "Extract the record from this document."}]
for _ in range(max_turns):
resp = client.messages.create(
model=os.environ["CLAUDE_MODEL"], max_tokens=1024,
system=SYSTEM, tools=TOOLS, messages=messages,
)
messages.append({"role": "assistant", "content": resp.content})
if resp.stop_reason != "tool_use":
text = "".join(b.text for b in resp.content if b.type == "text")
return json.loads(text)
results = []
for block in resp.content:
if block.type == "tool_use":
out, is_error = run_tool(block.name, block.input, path)
results.append({"type": "tool_result", "tool_use_id": block.id,
"content": out, "is_error": is_error})
messages.append({"role": "user", "content": results})
return {"error": "turn limit reached"}
if __name__ == "__main__":
print(json.dumps(extract(sys.argv[1]), indent=2))Run it with python agent.py invoice.png. On a typical invoice, Claude reads the Markdown, decides it is an invoice, asks for something like company_name,invoice_date, and returns the record. On an odd document it will often try discovery mode first, which is what you want.
The max_turns cap is not decoration. Without it, a confused agent can keep calling tools on a blurry scan and bill you for every round.
Validate outside the model
The agent’s final answer is still model output. Before it touches a database, check it with code that has no opinion about the document:
from datetime import date
REQUIRED = {"invoice": ["company_name", "invoice_date"]}
def validate(record, doc_type="invoice"):
if "error" in record:
return False, record["error"]
missing = [f for f in REQUIRED[doc_type] if not record.get(f)]
if missing:
return False, f"missing {missing}"
try:
date.fromisoformat(record["invoice_date"])
except ValueError:
return False, "invoice_date is not YYYY-MM-DD"
return True, "ok"Records that fail go to a review queue with the file attached. That queue is where you learn which document types need a better field list or a better scan, and it keeps one bad page from becoming a bad row three weeks later.
Limits worth knowing
- OCRskill takes images (PNG, JPEG, WebP, GIF), not PDFs. Convert each PDF page to an image first, for example with
pdftoppm -png, and run the agent per page or merge page results in code. - Every Claude turn costs tokens, and the Markdown from
read_documentis part of the context on later turns. If you already know a folder only holds one document type, skip the agent and call/ocr.jsondirectly with a fixed field list. It is cheaper and faster. - Let the agent choose fields, but keep the list of required fields per document type in your code. Otherwise the model quietly defines what “complete” means.
- Log the full message history for every failed record. It shows whether the OCR misread something or Claude asked for the wrong thing.
Where to take it next
The same loop handles very different paperwork once you change the system prompt and the validation table. The KYC form data extractor guide covers identity fields, and inspection notice OCR shows a document type where sources vary a lot. If you would rather run this inside Claude Code than as a script, the Claude Code OCR skill post shows how to wrap the same API calls in a skill.
The part that makes an extraction agent trustworthy is not the model. It is the small amount of code around it: one file per run, a turn cap, required fields defined by you, and a validator that rejects anything that does not fit.