How do you stream tool call results back to the client in real time?

Question

Accepted Answer

Streaming tool call results to the client in real time involves two key parts: capturing granular events on the backend as the agent executes tools, and pushing those events to the frontend over a persistent connection. On the backend, LangGraph's `astream_events` API provides a detailed stream of events, including `on_tool_start` and `on_tool_end`, which you can forward to the client via Server-Sent Events (SSE) or WebSockets. On the frontend, the `useStream` hook from `@langchain/react` automatically consumes this stream and provides a reactive `toolCalls` array that updates in real time as tools are invoked and completed. This approach gives you a smooth, token-by-token feel for tool execution without the complexity of managing raw WebSocket connections manually . The core of a real-time tool-call streaming implementation is LangGraph's `astream_events` API. This method streams every granular event that occurs during agent execution, including when a tool call starts (`on_tool_start`), when it ends (`on_tool_end`), and even individual text tokens from the LLM (`on_chat_model_stream`). To capture tool calls, your agent must be created with a model that supports tool calling and is configured with `streaming=True` (for providers like Anthropic or OpenAI) to ensure that events are emitted in a timely fashion . # streaming_service/fastapi_sse.py from collections.abc import AsyncIterator from fastapi import FastAPI from fastapi.responses import StreamingResponse from langchain.agents import create_agent from langchain_core.tools import tool from langchain_litellm import ChatLiteLLM @tool def get_weather(city: str) -> str: """Return the current weather for a given city.""" return f"The weather in {city} is sunny, 24°C." # Create an agent with a model that supports tool calling # streaming=True is crucial for capturing token-level events model = ChatLiteLLM(model="anthropic/claude-3-5-sonnet-20241022", streaming=True) agent = create_agent(model=model, tools=[get_weather]) app = FastAPI() def create_stream_response(query: str) -> AsyncIterator[bytes]: """Stream tool lifecycle and token events as SSE.""" async for event in agent.astream_events( {"messages": [{"role": "user", "content": query}]}, version="v2", ): if event["event"] == "on_tool_start": yield f"event: tool_start data: {event['name']} ".encode() elif event["event"] == "on_tool_end": yield f"event: tool_end data: {str(event['data']['output'])} ".encode() elif event["event"] == "on_chat_model_stream": chunk = event["data"]["chunk"].content if chunk: yield f"event: token data: {chunk} ".encode() yield "event: done data: [DONE] ".encode() @app.get("/chat") async def chat(query: str): """Endpoint for streaming agent responses.""" return StreamingResponse(create_stream_response(query), media_type="text/event-stream") # Server-Sent Events (SSE) is chosen for its simplicity and native compatibility # with HTTP/1.1 and HTTP/2, avoiding the need for WebSocket upgrades which can be # complex in serverless or autoscaling environments like Cloud Run . On the frontend, the `useStream` hook from `@langchain/react` integrates seamlessly with the streaming backend. It consumes the SSE stream and automatically populates a reactive `toolCalls` array. Each entry in this array has a state (`pending`, `completed`, or `error`) and contains both the tool call request (with its `name` and `args`) and its `result`. This allows you to build a UI that updates in real time, showing a loading indicator the moment a tool call is emitted and then displaying the result as soon as it arrives . // app/components/Chat.tsx import { useStream } from "@langchain/react"; export default function Chat() { const stream = useStream({ apiUrl: "/api/agent" }); return (

{stream.messages.map((message) => ( ))}

); } function Message({ message, toolCalls }) { if (message.type !== "ai") return

{message.content}

; // Filter tool calls that belong to this specific AI message const messageToolCalls = toolCalls.filter((tc) => message.tool_calls?.some((t) => t.id === tc.call.id) ); return (

{message.content}

{messageToolCalls.map((tc) => ( ))}

); } function ToolCallCard({ toolCall }) { if (toolCall.state === "pending") { return

🔄 Calling {toolCall.call.name}...

; } if (toolCall.state === "error") { return

❌ Error: {toolCall.result}

; } if (toolCall.call.name === "get_weather") { return

☀️ Weather: {toolCall.result.content}

; } return

📎 Tool Result: {toolCall.result.content}

; } // The toolCalls array from useStr

How do you stream tool call results back to the client in real time?

How do you stream tool call results back to the client in real time?

Scenario Questions

Follow-up Questions