Skip to content

Symfony AI - Store Component

Edit this page

The Store component provides a low-level abstraction for storing and retrieving documents in a vector store.

Installation

1
$ composer require symfony/ai-store

Purpose

A typical use-case in agentic applications is a dynamic context-extension with similar and useful information, for so called Retrieval Augmented Generation (RAG). The Store component implements low-level interfaces, that can be implemented by different concrete and vendor-specific implementations, so called bridges. On top of those bridges, the Store component provides higher level features to populate and query those stores with and for documents.

Indexing

Indexing is the process of converting documents into vector embeddings and storing them in a vector store. All indexers implement IndexerInterface and share a common index() method. Internally they use DocumentProcessor to run documents through the pipeline: filter → transform → vectorize → store.

There are three indexer implementations to choose from:

DocumentIndexer accepts documents directly (as DocumentInterface instances):

1
2
3
4
5
6
7
8
9
use Symfony\AI\Store\Document\TextDocument;
use Symfony\AI\Store\Document\Vectorizer;
use Symfony\AI\Store\Indexer\DocumentIndexer;
use Symfony\AI\Store\Indexer\DocumentProcessor;

$vectorizer = new Vectorizer($platform, $model);
$indexer = new DocumentIndexer(new DocumentProcessor($vectorizer, $store));
$document = new TextDocument('id-1', 'This is a sample document.');
$indexer->index($document);

SourceIndexer loads documents via a LoaderInterface from a runtime-provided source (file path, URL, etc.):

1
2
3
4
5
6
7
8
9
10
11
use Symfony\AI\Store\Document\Loader\TextFileLoader;
use Symfony\AI\Store\Document\Vectorizer;
use Symfony\AI\Store\Indexer\DocumentProcessor;
use Symfony\AI\Store\Indexer\SourceIndexer;

$vectorizer = new Vectorizer($platform, $model);
$loader = new TextFileLoader();
$indexer = new SourceIndexer($loader, new DocumentProcessor($vectorizer, $store));
$indexer->index('/path/to/document.txt');
// or index multiple sources at once:
$indexer->index(['/path/to/doc1.txt', '/path/to/doc2.txt']);

ConfiguredSourceIndexer wraps a SourceIndexer with a pre-configured default source, which is useful when the source is defined in configuration but should still be overridable at runtime:

1
2
3
4
5
use Symfony\AI\Store\Indexer\ConfiguredSourceIndexer;

$inner = new SourceIndexer($loader, new DocumentProcessor($vectorizer, $store));
$indexer = new ConfiguredSourceIndexer($inner, '/path/to/document.txt');
$indexer->index(); // uses the configured source

See the RAG Implementation cookbook for more advanced usage in combination with an Agent.

Retrieving

The opposite of indexing is retrieving. The Retriever is a higher level feature that allows you to search for documents in a store based on a query string. It vectorizes the query and retrieves similar documents from the store:

1
2
3
4
5
6
7
8
use Symfony\AI\Store\Retriever;

$retriever = new Retriever($store, $vectorizer);
$documents = $retriever->retrieve('What is the capital of France?');

foreach ($documents as $document) {
    echo $document->getMetadata()->getSource();
}

The retriever accepts optional parameters to customize the retrieval:

  • $options: An array of options to pass to the underlying store query (e.g., limit, filters)

All stores read the limit option, which sets the maximum number of returned documents:

1
$documents = $retriever->retrieve('What is the capital of France?', ['limit' => 5]);

Reranking

The RerankerListener sorts the retrieved documents again with a cross-encoder model and keeps the limit best ones. Without a limit option, it keeps the number given to its limit argument.

The reranker gives better results when it can choose from more documents than it keeps. Set the overFetchFactor argument and register the listener on both events. Then the store returns limit times overFetchFactor documents, and the reranker keeps the limit best ones:

1
2
3
4
5
6
7
8
9
10
11
12
use Symfony\AI\Store\Event\PostQueryEvent;
use Symfony\AI\Store\Event\PreQueryEvent;
use Symfony\AI\Store\EventListener\RerankerListener;

$rerankerListener = new RerankerListener($reranker, limit: 5, overFetchFactor: 4);
$dispatcher->addListener(PreQueryEvent::class, $rerankerListener);
$dispatcher->addListener(PostQueryEvent::class, $rerankerListener);

$retriever = new Retriever($store, $vectorizer, $dispatcher);

// the store returns 20 documents, the reranker keeps the 5 best ones
$documents = $retriever->retrieve('What is the capital of France?', ['limit' => 5]);

Note

When the listener is only registered on PostQueryEvent, the overFetchFactor has no effect. The over-fetched limit must stay within the limits of the store, for example 50 for Cloudflare Vectorize or 250 for Typesense.

Supported Stores

See Store Bridges for installation and configuration details for all supported bridges:

Document Loader

Creating and/or loading documents is a critical part of any RAG-based system, as it provides the foundation for the system to understand and respond to queries. Document loaders are responsible for fetching and preparing documents for indexing and retrieval.

To help loading documents and integrate them into your RAG system, you can use the provided document loaders or create your own custom loaders to suit your specific needs:

The DirectoryLoader scans a directory and delegates each file to a sub-loader chosen by its extension. The sub-loaders are injected as a map of file extension (without the leading dot) to a loader, and recursion into subdirectories can be toggled:

1
2
3
4
5
6
7
8
9
10
use Symfony\AI\Store\Document\Loader\DirectoryLoader;
use Symfony\AI\Store\Document\Loader\MarkdownLoader;
use Symfony\AI\Store\Document\Loader\TextFileLoader;

$loader = new DirectoryLoader([
    'md' => new MarkdownLoader(),
    'txt' => new TextFileLoader(),
], recursive: false);

$documents = $loader->load('/path/to/directory');

Files whose extension has no registered loader are skipped.

Create a Custom Loader

The main extension points of the Store component for document loaders is the LoaderInterface, that defines the method to load a document from a source. This leads to a loader implementing one method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
use Symfony\AI\Store\Document\LoaderInterface;
use Symfony\AI\Store\Document\Metadata;
use Symfony\AI\Store\Document\TextDocument;
use Symfony\Component\Uid\Uuid;

class MyDocumentLoader implements LoaderInterface
{
    public function load(?string $source = null, array $options = []): iterable
    {
        $content = ...

        yield new TextDocument(Uuid::v7()->toRfc4122(), $content, new Metadata($metadata));
    }
}

Document Transformer

Transformers change documents after loading and before vectorizing, for example to split them into smaller chunks. Pass them to the DocumentProcessor, which runs them in the given order. The transformers of the Store component live in the Symfony\AI\Store\Document\Transformer namespace.

Splitting Documents by Section

The TextSplitTransformer cuts a document into chunks of a fixed size, so a chunk can start in the middle of one section and end in the next one. To keep sections apart, run the RegexSplitTransformer first. It splits the content on each match of a regular expression, for example on Markdown headings:

1
2
3
4
5
6
7
8
9
10
use Symfony\AI\Store\Document\Transformer\RegexSplitTransformer;
use Symfony\AI\Store\Document\Transformer\TextSplitTransformer;
use Symfony\AI\Store\Indexer\DocumentProcessor;

$processor = new DocumentProcessor($vectorizer, $store, transformers: [
    // starts a new section at each Markdown heading
    new RegexSplitTransformer('/^#{1,6}[ \t]+.+$/m'),
    // splits sections longer than 1000 characters
    new TextSplitTransformer(chunkSize: 1000, overlap: 200),
]);

Each section becomes its own document. Its metadata holds the ID of the original document in _parent_id and the section text in _text. If the matched text is a Markdown heading, the metadata also holds the heading text without the # markers in _title and the heading level in _depth. Content before the first match becomes its own section. A section without a heading of its own keeps the _title and _depth of the original document.

If the TextSplitTransformer splits a long section into chunks, the _parent_id of these chunks holds the ID of the section. The section itself is not stored, so this ID doesn't point to a document in the store.

By default, each section keeps the matched text at its start. Pass keepDelimiter: false to remove it. To use both transformers as one, wrap them in a ChainTransformer.

Note

The strip_formatting option of the MarkdownLoader removes the # markers of all headings, so a pattern for Markdown headings finds nothing to split on. Don't use this option if you split the content on headings.

The heading pattern above also matches # lines inside fenced code blocks, for example a comment in a shell script. Such a line starts a new section and becomes its title.

Commands

While using the Store component in your Symfony application along with the AiBundle, you can use the bin/console ai:store:setup command to initialize the store and bin/console ai:store:drop to clean up the store. To remove all documents from a store without dropping it, use bin/console ai:store:clear:

1
2
3
4
5
6
7
8
9
10
11
# config/packages/ai.yaml
ai:
    # ...

    store:
        chromadb:
            symfonycon:
                collection: 'symfony_blog'
                # optional, a service implementing Codewithkyrian\ChromaDB\Embeddings\EmbeddingFunction.
                # Required to query the store with a TextQuery, which ChromaDB embeds client-side.
                embedding_function: 'app.chromadb.embedding_function'
1
2
3
$ php bin/console ai:store:setup symfonycon
$ php bin/console ai:store:clear symfonycon --force
$ php bin/console ai:store:drop symfonycon --force

Implementing a Bridge

The main extension points of the Store component is the StoreInterface, that defines the methods for adding, removing, querying and counting vectorized documents in the store. It extends \Countable, so a store reports the number of documents it holds and can be handed to count() directly.

This leads to a store implementing the following methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
use Symfony\AI\Store\Document\VectorDocumentInterface;
use Symfony\AI\Store\Query\QueryInterface;
use Symfony\AI\Store\StoreInterface;

class MyStore implements StoreInterface
{
    public function add(VectorDocumentInterface|array $documents): void
    {
        // Implementation to add a document to the store
    }

    public function remove(string|array $ids, array $options = []): void
    {
        // Implementation to remove documents from the store
    }

    public function clear(array $options = []): void
    {
        // Implementation to remove all documents from the store
    }

    public function query(QueryInterface $query, array $options = []): iterable
    {
        // Implementation to query the store for documents
        return $documents;
    }

    public function supports(string $queryClass): bool
    {
        // Return true if the given query class is supported
        return false;
    }

    public function count(): int
    {
        // Implementation to count the documents in the store
        return 0;
    }
}

Managing a store

Some vector store might requires to create table, indexes and so on before storing vectors, the ManagedStoreInterface defines the methods to setup and drop the store.

This leads to a store implementing two methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
use Symfony\AI\Store\ManagedStoreInterface;
use Symfony\AI\Store\StoreInterface;

class MyCustomStore implements ManagedStoreInterface, StoreInterface
{
    # ...

    public function setup(array $options = []): void
    {
        // Implementation to create the store
    }

    public function drop(array $options = []): void
    {
        // Implementation to drop the store (and related vectors)
    }
}

Clearing a store

To get rid of all documents in a store without dropping it, use clear():

1
$store->clear();

In contrast to ManagedStoreInterface::drop(), the store stays usable: documents can be added again right away, without calling setup() first, which makes clear() the method of choice for re-indexing:

1
2
$store->clear();
$indexer->index($documents);

Every store supports this operation, and almost all of them use the native mechanism of their backend to keep the table, index or collection in place - for example TRUNCATE TABLE for the SQL-based stores, _delete_by_query for Elasticsearch and OpenSearch, or deleteMany() for MongoDB.

The stores whose backend has no delete-all operation list the documents and remove them in batches instead, which is the case for Azure AI Search, Cloudflare, ChromaDB and S3 Vectors, as does Neo4j, which deletes its nodes in batched transactions. All of them use a sensible default batch size, which can be changed with the batch_size option:

1
$store->clear(['batch_size' => 250]);

The only exception is Vektor, which supports neither removing documents in bulk nor listing them. Its index is the storage directory itself, which is simply recreated.

Counting documents

To know how many documents a store holds, use count():

1
$total = $store->count();

Since StoreInterface extends \Countable, the store can be passed to count() as well, which also makes assertCount() work on a store in tests:

1
2
3
$total = count($store);

$this->assertCount(3, $store);

Every store supports this operation and uses the native counting mechanism of its backend - for example SELECT COUNT(*) for the SQL-based stores, _count for Elasticsearch and OpenSearch, an aggregation for Weaviate, or the exact count endpoint for Qdrant, which is asked instead of the estimated point count reported by the collection info.

Three stores have to read all their entries to count them, because their backend offers no count operation: Symfony Cache, S3 Vectors and Vektor.

Keep in mind that the number is as fresh as the backend makes it. Stores whose backend indexes documents asynchronously - Elasticsearch, OpenSearch, Meilisearch and Pinecone among them - can report a count that does not include documents added moments ago:

1
2
3
4
$store->add($documents);

// may still be 0 until the backend has indexed the documents
$store->count();
This work, including the code samples, is licensed under a Creative Commons BY-SA 3.0 license.
TOC
    Version