Maintained by deepset

Integration: OpenSearch Document Store

Use an OpenSearch database with Haystack

Authors

deepset

GitHub Repo PyPI Package

You can use OpenSearch in your Haystack pipelines with the OpenSearchDocumentStore

For a detailed overview of all the available methods and settings for the OpenSearchDocumentStore, visit the Haystack API Reference

Installation

pip install farm-haystack[opensearch]

Usage

Once installed and running, you can start using OpenSearch with Haystack by initializing it:

from haystack.document_stores import OpenSearchDocumentStore

document_store = OpenSearchDocumentStore()

Writing Documents to OpenSearchDocumentStore

To write documents to your OpenSearchDocumentStore, create an indexing pipeline, or use the write_documents() function. For this step, you may make use of the available FileConverters and PreProcessors, as well as other Integrations that might help you fetch data from other resources.

Indexing Pipeline

from haystack import Pipeline
from haystack.document_stores import OpenSearchDocumentStore
from haystack.nodes import PDFToTextConverter, PreProcessor

document_store = OpenSearchDocumentStore()
converter = PDFToTextConverter()
preprocessor = PreProcessor()

indexing_pipeline = Pipeline()
indexing_pipeline.add_node(component=converter, name="PDFConverter", inputs=["File"])
indexing_pipeline.add_node(component=preprocessor, name="PreProcessor", inputs=["PDFConverter"])
indexing_pipeline.add_node(component=document_store, name="DocumentStore", inputs=["PreProcessor"])

indexing_pipeline.run(file_paths=["filename.pdf"])

Using OpenSearch in a Query Pipeline

Once you have documents in your OpenSearchDocumentStore, it’s ready to be used in any Haystack pipeline. For example, below is a pipeline that makes use of the “deepset/question-generation” prompt that is designed to generate questions for the retrieved documents. If our OpenSearchDocumentStore had documents about food in it, you could generate questions about “Pizzas” in the following way:

from haystack import Pipeline
from haystack.document_stores import OpenSearchDocumentStore
from haystack.nodes import BM25Retriever, PromptNode

document_store = OpenSearchDocumentStore()
retriever = BM25Retriever(document_sotre = document_store)
prompt_node = PromptNode(model_name_or_path = "gpt-4",
                         api_key = "YOUR_OPENAI_KEY",
                         default_prompt_template = "deepset/question-generation")

query_pipeline = Pipeline()
query_pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"])
query_pipeline.add_node(component=prompt_node, name="PromptNode", inputs=["Retriever"])

query_pipeline.run(query = "Pizzas")