Iris

Streaming and AI backends

Build an AI backend in Go

Two packages ship with Iris v14 for this. sse carries an event stream in both directions, and ai renders a model's output on the wire. Neither ships a vendor SDK.

A service whose answers come from a model breaks HTTP habits. The answer arrives in fragments over seconds or minutes, the same endpoint has to serve it as a stream and as one JSON document, the thing producing it is a third party that fails in five distinct ways, and a failure can arrive after the response has already started with a 200.

None of that is hard to write badly. What is hard is the part nobody sees until production: a middleware that buffers the stream, a write deadline that ends it mid-sentence, a proxy that holds it, a client that vanished twenty minutes ago while the tokens kept being paid for.

Streaming that survives its own middleware

github.com/kataras/iris/v14/sse writes Server-Sent Events. The design rests on one decision:

sse.Open writes nothing. The protocol headers commit on the first Send, so an authorization check that fails after Open still returns an ordinary status code with an ordinary body.

Go
func chat(ctx iris.Context) {
    stream, err := sse.Open(ctx)
    if errors.HandleError(ctx, err) {
        return
    }
    defer stream.Close(nil) // the terminal frame, on success too.

    for token := range tokens {
        if err := stream.Send(sse.Event{Name: "token", Data: []byte(token)}); err != nil {
            return // the client is gone.
        }
    }
}

The error branch is rest/errors.HandleError, the same call every other handler on this site makes: it classifies the error, writes the canonical body and reports that it did, so the handler is one return.

On that first Send, and for that one response, five things happen that no middleware could guess were wanted:

  • every buffering response-writer wrapper is removed, so compression and the recorder an access log installs stop holding the events;
  • the server's absolute WriteTimeout is cleared, because it is a deadline on the whole response rather than on one write;
  • full duplex is enabled on HTTP/1, so a request body can still be read after the response has started;
  • Cache-Control: no-cache, no-transform and X-Accel-Buffering: no go out and any Content-Length is deleted, which is what stops nginx and the CDNs from buffering;
  • Connection: keep-alive is set on HTTP/1 only, where it means something.

A 15s heartbeat runs by default, which is the largest interval that survives nginx proxy_read_timeout at 60 seconds, an AWS Application Load Balancer at 60, Cloudflare returning a 524 at 100, and Heroku applying a rolling 55. Its larger job is noticing a client that disappeared: only a real FIN or RST, or a failing write, cancels a request context, so a closed lid leaves the work running for about fifteen minutes of TCP retransmission unless something writes and fails.

Close writes a done event on success as well as on failure, and that is the point. If the terminator is always present, its absence is how a client detects truncation. Nothing below it offers that signal: a proxy idle timeout closes a stream cleanly, and the client cannot tell a finished answer from a severed one.

One handler for both shapes

Every OpenAI-shaped API takes a stream field in the request body and serves both answers from one endpoint. github.com/kataras/iris/v14/ai takes that choice out of the handler body: when the request asks to stream, each part goes out as it is produced; when it does not, the parts are assembled and written as one JSON document. Nothing between Open and Close changes.

Go
o := ai.DefaultOptions()
o.Stream = &req.Stream // honour the request body.

res, err := ai.Open(ctx, o)
if errors.HandleError(ctx, err) {
    return
}

for delta := range generate(req) {
    if err := res.Send(ai.Text("m1", delta)); err != nil {
        res.Close(err)
        return
    }
}

res.Close(nil)

Options.Stream is a *bool rather than a bool because the zero value has to stay distinguishable from "not configured". Left nil it negotiates on the Accept header.

A vocabulary, not a vendor schema

A model does not only emit prose. Seven part kinds cover what actually travels: text, reasoning, tool, tool-result, usage, error, and data. Each has a constructor, and a Part's fields are unexported, so an invalid combination such as a tool call carrying reasoning text cannot be built and no encoder has to defend against one.

Three encoders render those parts: SSEJSON, NDJSON and PlainText. The first is the default and frames one part per SSE event; the second writes one JSON object per line for a server-to-server consumer; the third writes the answer and nothing else, for curl. Consecutive text and reasoning deltas are batched at 100ms by default, because one frame per token is one write, one flush, one TCP segment and one client-side render per token.

Vendor dialects are deliberately absent. The Encoder interface is the seam one enters through, and it points out of the framework: a shipped vendor schema is a commitment to track that vendor forever, and the best known of them changed its framing, its transport, its header name and its semantics inside eighteen months.

Relaying a provider, and knowing why it failed

A gateway reads an upstream stream, translates each event and forwards it. The translation is your code, because the framework has no business knowing what a given vendor calls a delta this month. The classification is not:

Go
resp, err := httpClient.Do(req)
if ue := ai.Classify(resp, err); ue != nil {
    return res.Close(ue) // typed: fatal, transient, rate limited, exhausted or canceled.
}

return res.Close(ai.Relay(res, resp, decoder, ai.RelayOptions{
    Read: sse.ReadOptions{Terminators: []string{"[DONE]"}},
}))

ai.Classify takes the response as well as the error, and that signature is the whole point. A well-behaved Go HTTP client returns a non-2xx response with a nil error, so a form that saw only the error would have to recover the status code from a string. The alternative is what AI gateways actually end up with, which is a regular expression over err.Error().

The five classes answer the only questions a caller has: retry, try a different credential, or stop. A 429 is its own class rather than a kind of exhaustion, because it is a routing signal and not a health signal: a credential pool that counts it as a fault quarantines a healthy key for being popular. And 401 and 403 mean exhausted rather than fatal, because in a gateway they almost always mean this credential is finished rather than this request is malformed.

A failure after the first byte cannot be rendered, so it goes out as an error event, and the default renderer runs it through the framework's own error pipeline: the route's mappings apply, the logger and the collector fire, and the text is redacted. A provider's message names the vendor, the model or the host, and it never reaches your caller by default.

Reading somebody else's stream

Every Go SDK that consumes a model's output reimplements event-stream parsing, usually by matching a "data: " prefix, and then breaks against a producer that omits the space. sse.Read implements the parsing algorithm instead, and is the framework's first iter.Seq2 API:

Go
for ev, err := range sse.Read(resp.Body, sse.ReadOptions{
    Terminators: []string{"[DONE]"},
}) {
    if err != nil {
        return err
    }
    // ev.Name, ev.Data
}

Multi-line data, comments, unknown fields, a leading byte order mark and CR, LF or CRLF line endings are all handled, and it never yields a partially framed event.

What these packages do not do

No agent loop: turns, stopping conditions and memory vary too much per application, and every framework that shipped one has spent the years since adding escape hatches to it. No LLM client, model catalog, token counter or price table. No conversation or vector store. The accounting hook hands you the assembled answer and the token usage after every turn, on success, on failure and on client disconnect alike, and the schema is yours.

Where to read more

Twenty-five chapters, about 85,000 words. Try BindJSON, graceful shutdown or error map.