Tutorial: Preprocessing Different File Types

• Level: Beginner
• Time to complete: 15 minutes
• Goal: After completing this tutorial, you’ll have learned how to build an indexing pipeline that will preprocess files based on their file type, using the FileTypeRouter.

💡 (Optional): After creating the indexing pipeline in this tutorial, there is an optional section that shows you how to create a RAG pipeline on top of the document store you just created. You must have a Hugging Face API Key for this section

Components Used

• FileTypeRouter: This component will help you route files based on their corresponding MIME type to different components
• MarkdownToDocument: This component will help you convert markdown files into Haystack Documents
• PyPDFToDocument: This component will help you convert pdf files into Haystack Documents
• TextFileToDocument: This component will help you convert text files into Haystack Documents
• DocumentCleaner (optional): This component will help you to make Documents more readable by removing extra whitespaces etc.
• DocumentSplitter: This component will help you to split your Document into chunks
• SentenceTransformersDocumentEmbedder: This component will help you create embeddings for Documents.
• DocumentWriter: This component will help you write Documents into the DocumentStore

Overview

In this tutorial, you’ll build an indexing pipeline that preprocesses different types of files (markdown, txt and pdf). Each file will have its own FileConverter. The rest of the indexing pipeline is fairly standard - split the documents into chunks, trim whitespace, create embeddings and write them to a Document Store.

Optionally, you can keep going to see how to use these documents in a query pipeline as well.

Installing dependencies

%%bash
pip install haystack-ai huggingface-api-haystack
pip install sentence-transformers-haystack
pip install markdown-it-py mdit_plain pypdf
pip install gdown

Download All Files

Files that you will use in this tutorial are stored in a GDrive folder. Either download files directly from the GDrive folder or run the code below. If you’re running this tutorial on colab, you’ll find the downloaded files under “/recipe_files” folder in “files” tab on the left.

Just like most real life data, these files are a mishmash of different types.

import gdown

url = "https://drive.google.com/drive/folders/1n9yqq5Gl_HWfND5bTlrCwAOycMDt5EMj"
output_dir = "recipe_files"

gdown.download_folder(url, quiet=True, output=output_dir)

Create a Pipeline to Index Documents

Next, you’ll create a pipeline to index documents. To keep things uncomplicated, you’ll use an InMemoryDocumentStore but this approach would also work with any other flavor of DocumentStore.

You’ll need a different file converter class for each file type in our data sources: .pdf, .txt, and .md in this case. Our FileTypeRouter connects each file type to the proper converter.

from haystack.components.writers import DocumentWriter
from haystack.components.converters import MarkdownToDocument, PyPDFToDocument, TextFileToDocument
from haystack.components.preprocessors import DocumentSplitter, DocumentCleaner
from haystack.components.routers import FileTypeRouter
from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersDocumentEmbedder
from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore

document_store = InMemoryDocumentStore()
file_type_router = FileTypeRouter(mime_types=["text/plain", "application/pdf", "text/markdown"])
text_file_converter = TextFileToDocument()
markdown_converter = MarkdownToDocument()
pdf_converter = PyPDFToDocument()

From there, the steps to this indexing pipeline are a bit more standard. The DocumentCleaner removes whitespace. Then this DocumentSplitter breaks them into chunks of 150 words, with a bit of overlap to avoid missing context.

document_cleaner = DocumentCleaner()
document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50)

Now you’ll add a SentenceTransformersDocumentEmbedder to create embeddings from the documents. As the last step in this pipeline, the DocumentWriter will write them to the InMemoryDocumentStore.

document_embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
document_writer = DocumentWriter(document_store)

After creating all the components, add them to the indexing pipeline.

preprocessing_pipeline = Pipeline()
preprocessing_pipeline.add_component(instance=file_type_router, name="file_type_router")
preprocessing_pipeline.add_component(instance=text_file_converter, name="text_file_converter")
preprocessing_pipeline.add_component(instance=markdown_converter, name="markdown_converter")
preprocessing_pipeline.add_component(instance=pdf_converter, name="pypdf_converter")
preprocessing_pipeline.add_component(instance=document_cleaner, name="document_cleaner")
preprocessing_pipeline.add_component(instance=document_splitter, name="document_splitter")
preprocessing_pipeline.add_component(instance=document_embedder, name="document_embedder")
preprocessing_pipeline.add_component(instance=document_writer, name="document_writer")

Next, connect them 👇

preprocessing_pipeline.connect("file_type_router.text/plain", "text_file_converter.sources")
preprocessing_pipeline.connect("file_type_router.application/pdf", "pypdf_converter.sources")
preprocessing_pipeline.connect("file_type_router.text/markdown", "markdown_converter.sources")
preprocessing_pipeline.connect("text_file_converter", "document_cleaner")
preprocessing_pipeline.connect("pypdf_converter", "document_cleaner")
preprocessing_pipeline.connect("markdown_converter", "document_cleaner")
preprocessing_pipeline.connect("document_cleaner", "document_splitter")
preprocessing_pipeline.connect("document_splitter", "document_embedder")
preprocessing_pipeline.connect("document_embedder", "document_writer")

Let’s test this pipeline with a few recipes I’ve written. Are you getting hungry yet?

from pathlib import Path

preprocessing_pipeline.run({"file_type_router": {"sources": list(Path(output_dir).glob("**/*"))}})

🎉 If you only wanted to learn how to preprocess documents, you can stop here! If you want to see an example of using those documents in a RAG pipeline, read on.

(Optional) Build a pipeline to query documents

Now, let’s build a RAG pipeline that answers queries based on the documents you just created in the section above. For this step, we will be using the HuggingFaceAPIChatGenerator so must have a Hugging Face API Key for this section. We will be using the Qwen/Qwen3-4B-Instruct-2507 model.

import os
from getpass import getpass

if "HF_API_TOKEN" not in os.environ:
    os.environ["HF_API_TOKEN"] = getpass("Enter Hugging Face token:")

In this step you’ll build a query pipeline to answer questions about the documents.

This pipeline takes the prompt, searches the document store for relevant documents, and passes those documents along to the LLM to formulate an answer.

⚠️ Notice how we used sentence-transformers/all-MiniLM-L6-v2 to create embeddings for our documents before. This is why we will be using the same model to embed incoming questions.

from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersTextEmbedder
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator

template = [
    ChatMessage.from_user(
        """
Answer the questions based on the given context.

Context:
{% for document in documents %}
    {{ document.content }}
{% endfor %}

Question: {{ question }}
Answer:
"""
    )
]
pipe = Pipeline()
pipe.add_component("embedder", SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"))
pipe.add_component("retriever", InMemoryEmbeddingRetriever(document_store=document_store))
pipe.add_component("chat_prompt_builder", ChatPromptBuilder(template=template))
pipe.add_component(
    "llm",
    HuggingFaceAPIChatGenerator(
        api_type="serverless_inference_api",
        api_params={"model": "Qwen/Qwen3-4B-Instruct-2507", "provider": "nscale"},
        generation_kwargs={"max_tokens": 1024},
    ),
)

pipe.connect("embedder.embedding", "retriever.query_embedding")
pipe.connect("retriever", "chat_prompt_builder.documents")
pipe.connect("chat_prompt_builder.prompt", "llm.messages")

Try it out yourself by running the code below. If all has gone well, you should have a complete shopping list from all the recipe sources. 🧂🥥🧄

question = (
    "What ingredients would I need to make vegan keto eggplant lasagna, vegan persimmon flan, and vegan hemp cheese?"
)

pipe.run({"embedder": {"text": question}, "chat_prompt_builder": {"question": question}})

{’llm’: {‘replies’: [ChatMessage(_role=<ChatRole.ASSISTANT: ‘assistant’>, _content=[TextContent(text=‘To make vegan keto eggplant lasagna, vegan persimmon flan, and vegan hemp cheese, based on the provided context, here are the required ingredients for each dish:\n\n—\n\n### 🥘 Vegan Keto Eggplant Lasagna\n\nIngredients:\n- 2 large eggplants \n- A lot of salt (to draw out water; rub both sides) \n- 1/3 cup pesto (made with: 4 oz basil, 1/4 cup almonds, 1/4 cup nutritional yeast, 1/4 cup olive oil) \n- 1/2 cup store-bought vegan mozzarella (or optional layer of Violife shredded mozz or Miyoko’s) \n- 1 recipe spinach tofu ricotta (made with: 14 oz firm/extra firm tofu, 10 oz spinach, 1 tsp garlic powder, juice of 1/2 lemon, salt to taste) \n- Optional: 1 layer of store-bought vegan cheese on top \n\nNotes: \n- Eggplant is roasted, salted, and then layered with pesto, cheese, and spinach tofu ricotta. \n- The dish is keto-friendly due to the use of minimal carbohydrates (spinach and tofu are low-carb, and pesto/nutritional yeast are low in net carbs). \n- Do not freeze eggplant; store in fridge for up to a few days.\n\n—\n\n### 🥣 Vegan Persimmon Flan\n\nIngredients:\n- ½ cup persimmon pulp (strained from 2 average-sized fuyu persimmons; seeds removed) \n- 1 tbsp cornstarch \n- ½ tsp agar agar \n- 1 tbsp agave nectar (or to taste) \n- 2 tbsp granulated sugar \n- ¼ cup coconut cream \n- ½ cup almond milk \n- ½ tsp vanilla \n\nNotes: \n- Cook the mixture in a saucepan, bring to a boil, and simmer. \n- Cool slightly, stir in vanilla, then refrigerate for at least 4 hours or overnight. \n- Serves 2 (written during pandemic, so small batch). \n- This is a low-carb, keto-friendly dessert option.\n\n—\n\n### 🥛 Vegan Hemp Cheese (Note: This is a hemp-based cheese, not from the macadamia nut cheese)\n\nIngredients:\n- ½ cup sunflower seeds (soaked overnight or boiled for 15 minutes) \n- ½ cup hemp hearts \n- 1.5 teaspoons miso paste \n- 1 tsp nutritional yeast \n- ¼ cup rejuvelac \n- 1/4 tsp salt (or to taste) \n\nInstructions (from context): \n- Blend all ingredients in a blender or food processor. \n- Transfer to a clean glass bowl, cover with a dish towel and rubber band. \n- Culture at room temperature (not in direct sunlight) for 24–48 hours. \n- The cheese will become tangy, like sourdough bread. \n\nNote: This is a fermented, low-carb, plant-based cheese. It’s not keto-friendly in large amounts due to carbs from seeds and miso, but it can be consumed in small portions. \n\n—\n\n### ✅ Final Answer:\n\nIngredients needed:\n\n#### For Vegan Keto Eggplant Lasagna:\n- 2 large eggplants \n- Salt (a lot) \n- 1/3 cup pesto (basil, almonds, nutritional yeast, olive oil) \n- 1/2 cup store-bought vegan mozzarella (e.g., Violife or Miyoko’s) \n- 14 oz firm tofu \n- 10 oz spinach \n- 1 tsp garlic powder \n- Juice of 1/2 lemon \n- Salt to taste \n\n#### For Vegan Persimmon Flan:\n- ½ cup persimmon pulp (from 2 fuyu persimmons) \n- 1 tbsp cornstarch \n- ½ tsp agar agar \n- 1 tbsp agave nectar \n- 2 tbsp granulated sugar \n- ¼ cup coconut cream \n- ½ cup almond milk \n- ½ tsp vanilla \n\n#### For Vegan Hemp Cheese:\n- ½ cup sunflower seeds (soaked or boiled) \n- ½ cup hemp hearts \n- 1.5 tsp miso paste \n- 1 tsp nutritional yeast \n- ¼ cup rejuvelac \n- 1/4 tsp salt \n\n> ✅ All dishes are vegan. The eggplant lasagna is keto-friendly due to low-carb ingredients. The flan and hemp cheese are also low-carb, though the hemp cheese has some carbs from seeds and miso. \n\nThis combination gives a balanced, plant-based, and keto-friendly meal and dessert pairing.’)], _name=None, _meta={‘model’: ‘Qwen/Qwen3-4B-Instruct-2507’, ‘finish_reason’: ‘stop’, ‘index’: 0, ‘usage’: {‘prompt_tokens’: 2010, ‘completion_tokens’: 1023}})]}}

What’s next

Congratulations on building an indexing pipeline that can preprocess different file types. Go forth and ingest all the messy real-world data into your workflows. 💥

If you liked this tutorial, you may also enjoy:

• Serializing Haystack Pipelines
• Creating Your First QA Pipeline with Retrieval-Augmentation

To stay up to date on the latest Haystack developments, you can sign up for our newsletter. Thanks for reading!