Haystack
Get Enterprise Support
Get Started
📘 How TELUS Agriculture & Consumer Goods Transformed Trade Promotions with Haystack Agents
Maintained by deepset

Integration: TogetherAI

Use the Together API for text generation models.

Authors
deepset
deepset-ai
GitHub Repo PyPI Package

Table of Contents

Overview

Once installed you will have access to [TogetherAIGenerator] and [TogetherAIChatGenerator] that allow you to call any LLMs available on TogetherAI, including:

  • OpenAI variants such as oopenai/gpt-oss-120B
  • deepseek-ai’s deepseek-ai/DeepSeek-R1
  • Other open-source models (Llama 2, Mixtral, etc.)

For more information on models available via the TogetherAI API, see the TogetherAI docs.

In order to follow along with this guide, you’ll need a TogetherAI API key. Add it as an environment variable, TOGETHER_API_KEY.

Installation

pip install togetherai-haystack

Usage

You can use TogetherAIChatGenerator as standalone, within a pipeline or with the Agent component.

Here’s an example of using it as a standalone component:

import os
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator

os.environ["TOGETHER_API_KEY"] = "YOUR_TOGETHER_API_KEY"

client = TogetherAIChatGenerator() # defaults to "meta-llama/Llama-3.3-70B-Instruct-Turbo",
response = client.run(
    [ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]
)
print(response["replies"])

{'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text='The capital of Vietnam is Hanoi.')], _name=None, _meta={'model': 'meta-llama/Llama-3.3-70B-Instruct-Turbo', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 8, 'prompt_tokens': 13, 'total_tokens': 21, 'completion_tokens_details': CompletionTokensDetails(accepted_prediction_tokens=None, audio_tokens=None, reasoning_tokens=0, rejected_prediction_tokens=None), 'prompt_tokens_details': PromptTokensDetails(audio_tokens=None, cached_tokens=0)}})]}

TogetherAIChatGenerator also support streaming responses if you pass a streaming callback:

import os
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator

os.environ["TOGETHER_API_KEY"] = "YOUR_TOGETHER_API_KEY"

def show(chunk):                              # simple streaming callback
    print(chunk.content, end="", flush=True)

client = TogetherAIChatGenerator(
    model="deepseek-ai/DeepSeek-R1",
    streaming_callback=show,
    generation_kwargs={"max_tokens": 100, "temperature": 0.7, "top_p": 0.9},
)

response = client.run([ChatMessage.from_user("Summarize RAG in two lines.")])
print (response)

License

togetherai-haystack is distributed under the terms of the Apache-2.0 license.