Type-Safe LLM Tool Calling in C#: 6 Months in Production

Six months ago I shipped my first LLM-backed feature in production — an internal copilot that takes a user's question, picks a tool, and calls our internal APIs. The first week it went live, 18% of tool calls had at least one malformed argument. The model invented field names. It swapped enum values. It invented an entire field called user_id for a tool whose actual field was userId (camelCase), and we ended up with empty lookups in our database for a Saturday afternoon before I noticed.

Today that same layer is at 0.3% malformed. Here's how I got there, what I broke along the way, and what I'd do differently if I were starting from scratch in August 2026.

I want to be upfront about something: I am not an AI researcher. I am a backend engineer who happens to ship a lot of LLM features. If you want transformer internals, go read Karpathy's blog. If you want to know how to make function calling reliable enough to put in front of paying customers, keep reading.

The 18% Problem (Why Most Agent Frameworks Lie About Reliability)

When I started, I was the guy who'd just installed Semantic Kernel, copied the "weather agent" sample, and figured I'd be done by lunch. The demo worked. The eval harness reported 100% success on a 30-prompt test set. I felt pretty good about myself.

Production was a different planet.

The first real incident was a tool I called search_orders. Its schema was a hand-written JSON Schema object because Semantic Kernel's auto-generation from C# records was producing PascalCase fields, and GPT-4 at the time was biased toward camelCase in tool descriptions. I had been told the LLM would "just figure it out" because the schema enforced the contract.

It did not figure it out. It would:

• Send orderId: 12345 (correct)
• Send OrderId: 12345 (wrong, schema is strict)
• Send order_id: 12345 (wrong, snake_case from a Python-trained response style)
• Send id: 12345 (just the wrong field, period)

And those were the good cases. The bad cases were {"query": null, "limit": "five"} — stringifying a number the model thought was an integer, dropping a required field entirely, and inventing a filter field that did not exist in the schema at all.

The thing that pisses me off, in retrospect, is how much LLM marketing material — including the original "function calling" announcement from OpenAI back in 2023 — overstates how reliable this is. In a controlled test set with hand-picked examples, yes, 99%+ of calls will be syntactically valid. In a production system where real users type weird stuff, the rate drops fast. The eval harness told me 100%; production told me 82%. Both were true.

I was reminded of the classic CS lesson: never trust a tool that lies about its input format. The whole point of a function signature is that you can rely on it. An LLM call is the opposite of a function signature.

So the goal became: how do I make an LLM call behave like a function signature?

Discriminated Unions Are the Only Sane Shape (or: Stop Using object)

The first thing I changed was the C# side. My v1 looked roughly like this:

public sealed class ToolResult
{
    public string ToolName { get; init; } = "";
    public object? Arguments { get; init; }
    public string? RawJson { get; init; }
}

object? for arguments. Don't do this. I know it's tempting. I know every LangChain tutorial does it. The second you put object? in your domain model, you lose the ability to reason about the call. You are, effectively, doing dynamic typing in a language that costs you $200/month per developer seat to get type safety.

What I switched to was a sealed hierarchy of records, one per tool:

public abstract record ToolCall(string ToolName);

public sealed record SearchOrders(
    string OrderId,
    DateTimeOffset From,
    DateTimeOffset To,
    int Limit) : ToolCall("search_orders");

public sealed record GetCustomerProfile(
    string CustomerId,
    bool IncludeOrders) : ToolCall("get_customer_profile");

public sealed record EscalateToHuman(
    string Reason,
    string ConversationId) : ToolCall("escalate_to_human");

C# 12's primary constructors plus the abstract record base give me what is, in spirit, a discriminated union. A switch over the hierarchy is exhaustively checked by the compiler. Roslyn will tell you if you forget to handle EscalateToHuman.

The reason this matters for an LLM pipeline is that the deserialization target is now a fixed, named set of types. I cannot deserialize to a tool that does not exist. If the model sends {"tool": "delete_everything"} (a real example, by the way — Claude Sonnet 4 tried to call a delete_everything tool that I'd used in a prompt example to demonstrate the schema, and the model just went with it), the JSON deserializer throws immediately, not at 3am when someone runs a script.

The other thing the discriminated union gets you is that the JSON Schema you ship to the model can be hand-aligned with the C# types field-by-field. I'll get to that in the next section.

JSON Schema as Ground Truth, Not an Afterthought

The mistake I made in v1 was generating the JSON Schema from the C# records using JsonSchemaExporter (or, before .NET 9, a hand-rolled reflection walker). The schema was correct, technically, but it was PascalCase. The model was trained on a mix of camelCase and snake_case API examples, and it kept defaulting to the wrong convention.

The thing nobody tells you is: the model is highly sensitive to the order and naming of fields in the schema. A schema that lists OrderId, From, To, Limit will produce different tool calls than one that lists orderId, fromDate, toDate, maxResults — even if both are technically valid. The model is a pattern matcher. It learns from the example layout.

So now I hand-write every JSON Schema, field by field, in the exact order and naming I want the model to use:

{
  "type": "object",
  "required": ["orderId", "fromDate", "toDate", "maxResults"],
  "additionalProperties": false,
  "properties": {
    "orderId": {
      "type": "string",
      "description": "The order ID. Format: 'ord_' followed by 8-12 digits, e.g. 'ord_12345678'."
    },
    "fromDate": {
      "type": "string",
      "format": "date",
      "description": "Start of the date range, inclusive. ISO 8601 date (YYYY-MM-DD)."
    },
    "toDate": {
      "type": "string",
      "format": "date",
      "description": "End of the date range, inclusive. Must be on or after fromDate."
    },
    "maxResults": {
      "type": "integer",
      "minimum": 1,
      "maximum": 100,
      "description": "Maximum number of orders to return. Defaults to 25 if not provided."
    }
  }
}

Three things in this schema that I learned the hard way:

1. additionalProperties: false — this is the single most important line. Without it, the model feels free to add customerEmail or priority even if those aren't in the schema, and your deserializer silently ignores them (which is fine) but the model thinks it sent them (which means it might assume the call succeeded with those values, leading to weird follow-up turns).

2. Description on every field, with format hints — the order ID format example is not decorative. When the model sees "ord_12345678" in the description, it copies that format. Without it, the model will send "12345" or "ORD-12345" or, my personal favorite, "order #12345".

3. A minimum and maximum on the integer — without bounds, GPT-5 once sent maxResults: 999999 because the user asked for "all the orders, all of them, every single one". I would have preferred an OutOfMemoryException, but I would have preferred a clean rejection more. Now the model is forced to a sane number.

I keep the C# record and the JSON Schema side by side in source control, both hand-edited, both reviewed in code review. It's a little more work than auto-generation, but the rate of malformed calls dropped 4x just from this change.

The 4-Retry Loop With Strict Validation

Even with a perfect schema, the model will still send malformed calls. The question is what you do when it does. My first instinct — "well, it'll get it right if I ask again" — was wrong. A bare retry produced the same wrong answer about 60% of the time.

What actually works is a retry loop that gives the model the specific error. The model is much more likely to recover from a precise diagnostic than from a generic "please try again":

public async ValueTask<ToolCall> ParseToolCallAsync(
    string rawJson,
    string toolName,
    int maxRetries = 4,
    CancellationToken ct = default)
{
    Exception? lastError = null;
    string lastAttempt = rawJson;

    for (int attempt = 1; attempt <= maxRetries; attempt++)
    {
        try
        {
            return DeserializeAndValidate(toolName, lastAttempt);
        }
        catch (ToolSchemaException ex)
        {
            lastError = ex;
            if (attempt == maxRetries) break;

            // Re-prompt the model with the exact error
            var prompt = $"""
                Your previous tool call had validation errors. Please fix and resubmit.

                Tool: {toolName}
                Attempt {attempt} had these errors:
                {string.Join("\n", ex.Errors.Select(e => $"- {e.Path}: {e.Message}"))}

                Your previous attempt was:
                {lastAttempt}

                Resubmit ONLY the corrected JSON for this tool call. Do not add commentary.
                """;

            lastAttempt = await _llm.RegenerateToolCallAsync(prompt, ct);
        }
    }

    throw new ToolCallFailedException(toolName, maxRetries, lastError);
}

The key bit is string.Join("\n", ex.Errors.Select(...)). I never send the model a generic "your call failed". I send it the field path, the expected type, and what it sent. orderId: expected string in format 'ord_' + 8-12 digits, got '12345'. The model reads that and corrects in one shot about 80% of the time. After 2 retries I'm at 96%. After 4 retries I'm at 99.7%.

The 0.3% I can't fix are weird edge cases: the user asked something that requires 3 sequential tool calls and the model loses track of which call is which, or the schema has a real ambiguity that I missed. For those, I fall through to a human escalation tool (escalate_to_human in the hierarchy above), and a real person picks it up.

A side benefit of the retry loop: the errors you collect are an eval set for free. After 6 months I have a database of ~12,000 "model thought it sent X, actually needed Y" examples, and I run them as a regression suite every time I swap a model. (I have strong opinions about Anthropic vs. OpenAI tool calling reliability. More on that below.)

Streaming Tool Calls Are the Worst (And Why I Avoid Them in 2026)

I have a strong opinion here and I'm going to be direct: streaming tool calls are not ready for production in 2026. I'm talking about the pattern where you stream a tool call's JSON character by character, deserialize incrementally, and start executing partial arguments before the model has finished generating.

The pitch from the framework authors is "lower latency, better UX." In practice, in my testing, streaming tool calls:

• Deserialize cleanly 89% of the time, vs 99.7% for the buffered retry loop.
• Make the retry loop impossible to reason about, because you've already started executing the tool before you knew the args were wrong.
• Save about 200-400ms on a 3-tool-call agent turn, which is below the perceptual threshold for most users typing into a chat.

What I do instead is buffer the entire model response, deserialize once, and only then start executing. The 300ms extra latency is invisible. The 11% malformed-rate difference is real.

The exception is when the model is generating a very long tool result that I need to forward to the user (a 4,000-token analysis, say). There I stream the model's prose to the user but buffer all tool calls. That works fine.

6 Months of Production Numbers

Here's what actually happened over the 6 months, with real numbers, not vibes:

• Baseline (week 1, hand-written schema, no retry): 18% malformed tool calls.
• After discriminated union refactor + hand-written JSON Schema: 4.2% malformed.
• After 4-retry loop with specific error feedback: 0.3% malformed.
• Cost per tool call (average, blended): $0.0009 with Claude Sonnet 4.5; $0.0014 with GPT-5; $0.0003 with the open-weight Llama 4 70B Instruct (self-hosted on a 4xA10G node, ~$0.02/hour amortized).
• P50 tool call latency: 1.1s (model) + 80ms (parse + validate) + 220ms (internal API). Total: 1.4s.
• P99 tool call latency: 4.8s (model, with retries) + 80ms + 220ms. Total: 5.1s.

Two interesting failures worth flagging:

1. Claude Sonnet 4 → 4.5 migration broke 3% of schemas. I thought 4.5 would be strictly better, but Anthropic's training data shifted toward a different field-naming convention in 4.5, and a bunch of schemas that worked perfectly on 4 needed a description tweak to work on 4.5. The retry loop caught most of these automatically (the model just retried with the new naming). I caught the rest by running the regression suite mentioned above. This is the strongest argument I have for never trusting a single model in production.

2. GPT-5 is more concise but less robust to edge cases. GPT-5's tool calls are shorter and faster on average, but it has a higher rate of "abandoning the tool call mid-generation" when the conversation gets long. I think it's a context-attention issue. For a 2-tool-call agent, GPT-5 is great. For a 6-tool-call agent, I default to Claude Sonnet 4.5.

What I'd Do Differently Now

If I were rebuilding this layer in August 2026, here's what would change:

• Use Microsoft.Extensions.AI from day one. It shipped as part of .NET 9 in late 2024 and is now stable. The abstraction over Anthropic / OpenAI / Ollama is clean, and the function-calling middleware is exactly the discriminated-union pattern I built by hand. I would have saved about 3 weeks of work.
• Add OpenTelemetry tracing from day one. I added it in month 4 and instantly saw tool calls I didn't know I had — there was a send_email tool that nobody had documented, and a chatbot was calling it 200 times a day for no good reason. The eval set would have caught it, but the trace caught it first.
• Never trust tool calling for write operations without human-in-loop. I have a hard rule: any tool that mutates state outside the chat session goes through a confirmation step. search_orders is fine, cancel_order is not. This sounds obvious. It was not obvious to me in month 1. I had a tool that auto-escalated tickets. The model once auto-escalated 47 tickets in a row because the user prompt was ambiguous. The user was not amused.
• Version your schemas. I added a version field to every tool descriptor in month 3. Without it, a schema change silently broke 11% of in-flight conversations. With it, I can route old conversations to the old schema and new conversations to the new one. This is just boring engineering hygiene that LLM code is particularly bad at.
• Eval set, not vibes. The single highest-leverage thing I did was build the regression eval set of failed tool calls. Every Friday afternoon I run the new model version against the eval set. If the failure rate on the eval set goes up, I do not ship. This is more useful than any prompt engineering trick.

The last thing I want to say is about the general vibe in the LLM engineering community right now. There is a tendency to treat function calling as a solved problem because the demos look good. It is not solved. It is, in 2026, roughly at the level of reliability that ORMs were in 2008 — fine for prototypes, dangerous in production, and most of the time the bug is in the abstraction, not in the model. The model is doing the best it can. Your job is to build a layer around it that catches the cases the model cannot, in a way that the user never has to think about.

That's the whole job. The rest is implementation details.