How do you stream a response and display it token by token?

Question

Accepted Answer

Streaming allows you to display the model's response in real-time, character by character or token by token, rather than waiting for the entire output to be generated. This dramatically improves the perceived speed and responsiveness of your application, especially for longer responses. Both the OpenAI and Anthropic APIs support this by setting `stream: true` in the request body . Instead of waiting for a complete JSON response, the API returns a stream of Server-Sent Events (SSE). The client iterates over this stream, receiving data chunks as they are generated. For each chunk, you can parse the content, extract the new text fragment (often within a `delta` object), and append it to the UI. This process continues until the stream ends . This technique is essential for creating a fluid, chat-like user experience. import OpenAI from 'openai';
const openai = new OpenAI();

async function streamResponse() {
  const stream = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: 'Write a short story about a robot learning to paint.' }],
    stream: true,
  });

for await (const chunk of stream) {
    // Extract the text delta from the chunk
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      process.stdout.write(content); // or append to a div in a browser
    }
  }
}

streamResponse(); import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();

async function streamResponse() {
  const stream = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 1024,
    messages: [{ role: 'user', content: 'Write a short story about a robot learning to paint.' }],
    stream: true,
  });

for await (const event of stream) {
    // Anthropic's stream uses a `type` field
    if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
      process.stdout.write(event.delta.text); // or append to UI
    }
  }
}

streamResponse();

How do you stream a response and display it token by token?

Streaming responses are implemented by setting the `stream` parameter to `true` in the API request, which returns a stream of Server-Sent Events (SSE) that the client processes to display text incrementally.

How do you stream a response and display it token by token?

Streaming responses are implemented by setting the `stream` parameter to `true` in the API request, which returns a stream of Server-Sent Events (SSE) that the client processes to display text incrementally.

Scenario Questions

Follow-up Questions

How do you stream a response and display it token by token?

Streaming responses are implemented by setting the stream parameter to true in the API request, which returns a stream of Server-Sent Events (SSE) that the client processes to display text incrementally.

How do you stream a response and display it token by token?

Streaming responses are implemented by setting the stream parameter to true in the API request, which returns a stream of Server-Sent Events (SSE) that the client processes to display text incrementally.

Scenario Questions

Follow-up Questions

Streaming responses are implemented by setting the `stream` parameter to `true` in the API request, which returns a stream of Server-Sent Events (SSE) that the client processes to display text incrementally.

Streaming responses are implemented by setting the `stream` parameter to `true` in the API request, which returns a stream of Server-Sent Events (SSE) that the client processes to display text incrementally.