Build a Tool-Calling Agent Framework

In this lab, you'll build the foundation of every agentic system: a tool-calling agent in Python. Your agent will maintain a registry of tools, use an LLM to decide which tools to call, validate parameters, execute tools safely, and handle errors gracefully.

This is the pattern used by every major agent framework (LangGraph, CrewAI, OpenAI Agents SDK). By building it from scratch, you'll understand exactly how tool calling works under the hood.

Architecture Overview

User Message
     ↓
Agent Loop ←──────────────────┐
     ↓                        │
LLM (decides: respond or     │
     call tool)               │
     ↓                        │
[If tool call]                │
     ↓                        │
Validate Parameters           │
     ↓                        │
Execute Tool (with timeout)   │
     ↓                        │
Inject Result → back to LLM ─┘
     ↓
[If final response]
     ↓
Return to User

Step 1: Tool Registry

Build a ToolRegistry class that manages tool definitions. Each tool has:

A unique name (string identifier)
A description (what the tool does — this is sent to the LLM)
A parameters JSON Schema (defines expected input types)
A handler function (the actual implementation)

The registry should support:

register(tool) — Add a tool to the registry
unregister(name) — Remove a tool by name
get(name) — Retrieve a tool by name
list_tools() — Return all tool definitions (for sending to the LLM)

Step 2: Parameter Validation

Before executing any tool, validate the provided arguments against the tool's JSON Schema:

Check required fields are present
Check types match (string, number, boolean, array, object)
Return a clear error message if validation fails

You can implement a simple validator or use the jsonschema library pattern.

Step 3: Tool Executor

Build a ToolExecutor that runs tool handlers with a bounded wait and structured results:

Execute the tool's handler function with the validated arguments
Apply a configurable wait budget (default: 30 seconds) using ThreadPoolExecutor
Catch exceptions and return structured error results
Track execution metadata: start time, end time, duration, success/failure

Read this before you implement it, because the obvious version does not do what its name says. A ThreadPoolExecutor timeout bounds how long you wait. It does not stop the handler. Python cannot kill a running thread, so future.cancel() returns False once the work has started, and the tool keeps going — it will finish, and any write it was going to make still lands, after your agent has already recorded a failure and moved on. Verify it yourself with a handler that sleeps and then appends to a list: the append happens.

Two consequences your implementation has to handle:

Do not create the pool inside a with block per call. ThreadPoolExecutor.__exit__ calls shutdown(wait=True), which blocks until the handler finishes — so a 0.2s timeout around a 1.5s handler returns after 1.5s, and the timeout bounds nothing at all. Create one long-lived pool in __init__.
Report the outcome honestly. A timeout is not "the tool did not run", it is "the tool's outcome is unknown". Give ExecutionResult a side_effects_possible: bool and set it True on the timeout path, so the agent loop and the audit log both record that a write may still be in flight.

The real fix for a tool that must be stopped is a boundary the OS can enforce — a subprocess you can kill, or a timeout inside the tool's own client, such as an HTTP client's request timeout — but that changes the Callable interface this lab is built on. Naming the limitation is the graded skill here, and it is also the answer in the interview.

Step 4: LLM Tool Selection

Build a ToolSelector that asks the LLM which tool(s) to call:

Format the available tools as a structured prompt for the LLM
Parse the LLM's response to extract tool call decisions
Support the LLM deciding to call no tools (direct response)
Support the LLM deciding to call multiple tools in sequence

Use structured output parsing: the LLM should return JSON with tool name and arguments.

Step 5: Agent Conversation Loop

Build the main Agent class that orchestrates everything:

Accept a user message
Send it to the LLM along with available tool definitions
If the LLM decides to call a tool: validate, execute, inject result, loop back
If the LLM provides a final response: return it to the user
Support multi-step reasoning (the LLM may call multiple tools before responding)
Enforce a maximum number of tool calls per turn (to prevent infinite loops)

Step 6: Audit Logging

Add an AuditLogger that records every action:

Tool call attempts (tool name, arguments, timestamp)
Tool execution results (success/failure, duration, output)
LLM decisions (tool call vs. direct response)
Error events (validation failures, timeouts, exceptions)

What to Submit

The editor has 6 file sections with TODO comments. Replace each TODO with your Python code. The AI grader will evaluate each section against the rubric.

Hints

For the JSON Schema validator, focus on type and required checks — you don't need a full JSON Schema implementation
For the timeout, use Python's concurrent.futures.ThreadPoolExecutor with a timeout parameter
For LLM tool selection, define a clear prompt format and expected JSON response structure
The agent loop should have a max_iterations parameter (default: 10) to prevent runaway execution

Instructions