Tool use & function calling

The model asks, your code acts

The central fact about tool use is that the model never runs anything. It has no network access and no ability to execute code. What it can do is emit a structured message that says, in effect, "I would like to call get_error_rate with service set to checkout". Your application receives that message, decides whether to honour it, runs the function, and sends the result back as a new message. The model then carries on, using the result.

1. You send:    the conversation + a list of tool definitions
2. Model says:  "call get_error_rate with {service: 'checkout', window_minutes: 15}"
                (a structured tool-call message, NOT the answer; generation stops here)
3. Your code:   validates the arguments -> runs the real function -> gets 0.042
4. You send:    the same conversation + the model's tool call + the tool result
5. Model says:  "The checkout error rate over the last 15 minutes is 4.2%, which is ..."
                (or it asks for another tool, and the loop continues)

Seeing it this way clears up most confusion. The model's tool call is a request, and an untrusted one. It may hold arguments that are wrong, invented, or planted by text somewhere in the conversation. Your code is the only thing with real power, so your code is where permissions, validation and limits belong. It also explains the cost: every tool call is a further round trip to the model, with the whole conversation sent again.

Providers differ in the exact shape of the messages, and the concepts are identical everywhere: tool definitions go in, the model returns either text or tool calls together with a reason for stopping, and you reply with tool results linked to the calls by an ID. The examples here use a neutral shape and a stand-in function, which you wire to your own provider.

Defining a tool

A tool definition has three parts: a name, a description, and an input schema written in JSON Schema. The model reads all three when deciding whether to call the tool and how, so together they are a prompt, and they deserve as much care as one.

{
  "name": "get_error_rate",
  "description": "Returns the fraction of failed HTTP requests (5xx) for one service over a recent time window, from production metrics. Use this when asked whether a service is healthy, or to confirm the impact of an incident. The result is a number from 0 to 1, where 0.02 means 2% of requests failed. Data is about one minute behind real time. For latency, use get_latency_percentiles instead.",
  "input_schema": {
    "type": "object",
    "properties": {
      "service": {
        "type": "string",
        "description": "Service name exactly as it is deployed, for example 'checkout' or 'orders-api'. Call list_services first if unsure."
      },
      "window_minutes": {
        "type": "integer",
        "minimum": 1,
        "maximum": 1440,
        "description": "How far back to look, in minutes. Use 15 for the current state, and 60 or more for trends."
      },
      "environment": {
        "type": "string",
        "enum": ["production", "staging"],
        "description": "Defaults to production if omitted."
      }
    },
    "required": ["service", "window_minutes"],
    "additionalProperties": false
  }
}

• The description is the most important part. Say what the tool does, when to use it, when not to, what it returns and in which units, and any limitation. Three or four sentences is normal. A one-line description is the commonest cause of tools that are called wrongly, or not called at all.
• Describe every parameter, with an example value and its format. "Date" is ambiguous. "ISO 8601 date, for example 2026-09-17" is not.
• Use enumerations, ranges and required fields to make invalid calls impossible instead of merely discouraged. Several providers offer a strict mode that guarantees the arguments will match the schema.
• Name tools clearly and consistently, with a verb and a noun: get_error_rate, list_services, create_ticket. With many tools, a common prefix per system helps, as in github_list_prs.
• Fewer, better tools. Every definition costs tokens on every request, and overlapping tools confuse the choice. If two tools are always used together, combine them. If a person could not say which of two tools to use, the model cannot either.
• Design for the model, not for your API. Do not expose forty endpoints one for one. Offer the handful of operations that correspond to tasks, with the parameters that a caller would naturally have to hand.

The agent loop

import json

MAX_TURNS = 10


def run_agent(user_request: str, tools: list[dict], handlers: dict) -> str:
    """call_llm is a stand-in: wire it to your provider. It takes messages and tool
    definitions, and returns {'text': str, 'tool_calls': [{'id', 'name', 'arguments'}]}."""
    messages = [
        {"role": "system", "content": "You help on-call engineers investigate production issues."},
        {"role": "user", "content": user_request},
    ]
    for _ in range(MAX_TURNS):
        reply = call_llm(messages, tools=tools)
        messages.append({"role": "assistant", "content": reply["text"],
                         "tool_calls": reply["tool_calls"]})

        if not reply["tool_calls"]:
            return reply["text"]                      # no tool requested: this is the answer

        for call in reply["tool_calls"]:              # there may be several in one turn
            result = execute_tool(call, handlers)
            messages.append({"role": "tool", "tool_call_id": call["id"],
                             "content": json.dumps(result)})

    return "I could not finish within the allowed number of steps."

The loop is short, and each line of it matters. The assistant's tool-call message must be added to the history before the results, and each result must carry the ID of the call that it answers, so that the model can match them up. The limit on iterations is not optional: without it, a confused model can call tools for ever, and you pay for every round.

import jsonschema


def execute_tool(call: dict, handlers: dict) -> dict:
    name, args = call["name"], call["arguments"]

    if name not in handlers:                                   # the model may invent a tool name
        return {"error": f"Unknown tool '{name}'. Available: {sorted(handlers)}"}

    handler = handlers[name]
    try:
        jsonschema.validate(args, handler["schema"])           # never trust the arguments
    except jsonschema.ValidationError as err:
        return {"error": f"Invalid arguments: {err.message}"}

    try:
        return {"result": handler["fn"](**args)}               # the function itself enforces timeouts
    except PermissionError:
        return {"error": "Not permitted for this user."}
    except Exception as err:                                   # report the failure; do not crash the loop
        return {"error": f"{type(err).__name__}: {err}"}

Notice the pattern: errors go back to the model as data. If the service name was wrong, the model reads "Unknown service 'chekout'. Did you mean 'checkout'?" and corrects itself on the next turn. If you raise an exception and abort instead, you lose that ability to recover. Write error messages for the model as you would for a colleague: say what went wrong and what to try next, and do not include stack traces or secrets.

Parallel calls, large results and tool choice

• Parallel tool calls. A model may ask for several tools in a single turn, for example the error rates of three services. Run them concurrently, then return all of the results together before calling the model again, each matched to its call ID. If one fails, return its error alongside the other results. Do not drop it, and do not abort the batch.
• Result size. Whatever a tool returns goes into the context window, and is paid for again on every later turn. A tool that returns ten thousand log lines will wreck both cost and quality. Return the smallest useful thing: summarise, paginate, truncate with a note saying that more exists, and let the model ask for detail.
• Result format. Return clean, labelled data with units. {"error_rate": 0.042, "window_minutes": 15, "requests": 18400} is better than a bare 0.042.
• Tool choice. Most APIs allow you to leave the decision to the model (the default), to forbid tools, or to require that a tool be used. Requiring a particular tool is one way to get structured output. Some newer reasoning models restrict the forced modes, so check what your model supports.
• Dates and context. The model does not know the current time, the current user or the default environment unless you tell it. Put such facts in the system prompt, or provide a tool that returns them.
• Many tools. Beyond a few dozen, accuracy of selection falls and cost rises. Group them, load only the set relevant to the task, or use a provider's tool search feature that reveals definitions on demand.

from concurrent.futures import ThreadPoolExecutor


def execute_all(tool_calls: list[dict], handlers: dict) -> list[dict]:
    """Run the tool calls of one turn concurrently, and keep each result with its call id."""
    with ThreadPoolExecutor(max_workers=8) as pool:
        results = list(pool.map(lambda call: execute_tool(call, handlers), tool_calls))
    return [{"role": "tool", "tool_call_id": call["id"], "content": json.dumps(result)}
            for call, result in zip(tool_calls, results)]

Designing tools that are safe to call

Once a tool can change something, you are no longer building a chatbot. You are building automation that is driven by a probabilistic component and steered by text, some of which you do not control. Apply the engineering judgement that you would apply to any automation with production access.

• Least privilege. Give the tool a credential that can do only what the tool is for. A read-only metrics token cannot delete anything, whatever the model is tricked into requesting. This is the same principle as in the secrets module, and it matters more here.
• Act as the user. Enforce the asking user's permissions inside the tool. The model must not become a way of reading data that the user could not otherwise read.
• Validate the meaning, not only the type. A schema confirms that replicas is an integer. Your code must confirm that 5,000 is not an acceptable value.
• Idempotency. The loop may retry, and the model may call a tool twice. Make writes safe to repeat, with idempotency keys, as the failure-modes module described.
• Timeouts and rate limits on every tool, so that one slow dependency, or a looping model, cannot hang the agent or flood a downstream system.
• Dry-run and preview. For risky actions, offer a tool that reports what would happen, and a separate one that does it.
• Log everything: the tool, the arguments, the result, the user, and the conversation that led to it. When an agent does something surprising, this record is how you find out why.

Testing tool use

Tool use can be evaluated like everything else, and the evals module applies directly. The cases are tasks, and the things to grade are partly mechanical.

• Tool selection: for a given request, was the right tool called? For a request that needs no tool, was none called?
• Arguments: were they valid and correct? These can be compared exactly with the expected values.
• Efficiency: how many turns, and how many tokens, did the task take? A rising number is a regression.
• Recovery: when a tool returns an error, does the model correct itself and continue?
• Outcome: was the final answer right, and did the expected side effects occur in a sandbox?
• Refusal: does it decline to call a tool that the request does not justify?

Test against fake or sandboxed implementations of the tools, never against production. When results disappoint, read the transcripts before you change the prompt. Most tool-use failures trace back to a vague description, to overlapping tools, or to a result that was too large or too cryptic for the model to use.