Symfony AI - Store Component

Installation

1

$ composer require symfony/ai-store

Purpose

A typical use-case in agentic applications is a dynamic context-extension with similar and useful information, for so called Retrieval Augmented Generation (RAG). The Store component implements low-level interfaces, that can be implemented by different concrete and vendor-specific implementations, so called bridges. On top of those bridges, the Store component provides higher level features to populate and query those stores with and for documents.

Indexing

Indexing is the process of converting documents into vector embeddings and storing them in a vector store. All indexers implement IndexerInterface and share a common index() method. Internally they use DocumentProcessor to run documents through the pipeline: filter → transform → vectorize → store.

There are three indexer implementations to choose from:

DocumentIndexer accepts documents directly (as EmbeddableDocumentInterface instances):

1
2
3
4
5
6
7
8
9

use Symfony\AI\Store\Document\TextDocument;
use Symfony\AI\Store\Document\Vectorizer;
use Symfony\AI\Store\Indexer\DocumentIndexer;
use Symfony\AI\Store\Indexer\DocumentProcessor;

$vectorizer = new Vectorizer($platform, $model);
$indexer = new DocumentIndexer(new DocumentProcessor($vectorizer, $store));
$document = new TextDocument('id-1', 'This is a sample document.');
$indexer->index($document);

SourceIndexer loads documents via a LoaderInterface from a runtime-provided source (file path, URL, etc.):

1
2
3
4
5
6
7
8
9
10
11

use Symfony\AI\Store\Document\Loader\TextFileLoader;
use Symfony\AI\Store\Document\Vectorizer;
use Symfony\AI\Store\Indexer\DocumentProcessor;
use Symfony\AI\Store\Indexer\SourceIndexer;

$vectorizer = new Vectorizer($platform, $model);
$loader = new TextFileLoader();
$indexer = new SourceIndexer($loader, new DocumentProcessor($vectorizer, $store));
$indexer->index('/path/to/document.txt');
// or index multiple sources at once:
$indexer->index(['/path/to/doc1.txt', '/path/to/doc2.txt']);

ConfiguredSourceIndexer wraps a SourceIndexer with a pre-configured default source, which is useful when the source is defined in configuration but should still be overridable at runtime:

1
2
3
4
5

use Symfony\AI\Store\Indexer\ConfiguredSourceIndexer;

$inner = new SourceIndexer($loader, new DocumentProcessor($vectorizer, $store));
$indexer = new ConfiguredSourceIndexer($inner, '/path/to/document.txt');
$indexer->index(); // uses the configured source

See the RAG Implementation cookbook for more advanced usage in combination with an Agent.

Retrieving

The opposite of indexing is retrieving. The Retriever is a higher level feature that allows you to search for documents in a store based on a query string. It vectorizes the query and retrieves similar documents from the store:

1
2
3
4
5
6
7
8

use Symfony\AI\Store\Retriever;

$retriever = new Retriever($store, $vectorizer);
$documents = $retriever->retrieve('What is the capital of France?');

foreach ($documents as $document) {
    echo $document->getMetadata()->getSource();
}

The retriever accepts optional parameters to customize the retrieval:

• $options: An array of options to pass to the underlying store query (e.g., limit, filters)

Supported Stores

• Azure AI Search
• Chroma (requires codewithkyrian/chromadb-php as additional dependency)
• ClickHouse
• Cloudflare
• Elasticsearch
• InMemory
• Manticore Search
• MariaDB (requires ext-pdo)
• Meilisearch
• Milvus
• MongoDB Atlas (requires mongodb/mongodb as additional dependency)
• Neo4j
• OpenSearch
• Pinecone (requires probots-io/pinecone-php as additional dependency)
• Postgres (requires ext-pdo)
• Qdrant
• Redis
• S3 Vectors
• SQLite (requires ext-pdo_sqlite)
• Supabase (requires manual database setup)
• SurrealDB
• Symfony Cache (requires symfony/cache as additional dependency)
• Typesense
• Vektor
• Weaviate

Document Loader

Creating and/or loading documents is a critical part of any RAG-based system, as it provides the foundation for the system to understand and respond to queries. Document loaders are responsible for fetching and preparing documents for indexing and retrieval.

To help loading documents and integrate them into your RAG system, you can use the provided document loaders or create your own custom loaders to suit your specific needs:

• CsvLoader
• DirectoryLoader
• InMemoryLoader
• JsonFileLoader
• MarkdownLoader
• RssFeedLoader
• RstLoader
• RstToctreeLoader
• TextFileLoader

The DirectoryLoader scans a directory and delegates each file to a sub-loader chosen by its extension. The sub-loaders are injected as a map of file extension (without the leading dot) to a loader, and recursion into subdirectories can be toggled:

1
2
3
4
5
6
7
8
9
10

use Symfony\AI\Store\Document\Loader\DirectoryLoader;
use Symfony\AI\Store\Document\Loader\MarkdownLoader;
use Symfony\AI\Store\Document\Loader\TextFileLoader;

$loader = new DirectoryLoader([
    'md' => new MarkdownLoader(),
    'txt' => new TextFileLoader(),
], recursive: false);

$documents = $loader->load('/path/to/directory');

Files whose extension has no registered loader are skipped.

Create a Custom Loader

The main extension points of the Store component for document loaders is the LoaderInterface, that defines the method to load a document from a source. This leads to a loader implementing one method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

use Symfony\AI\Store\Document\LoaderInterface;
use Symfony\AI\Store\Document\Metadata;
use Symfony\AI\Store\Document\TextDocument;
use Symfony\Component\Uid\Uuid;

class MyDocumentLoader implements LoaderInterface
{
    public function load(?string $source = null, array $options = []): iterable
    {
        $content = ...

        yield new TextDocument(Uuid::v7()->toRfc4122(), $content, new Metadata($metadata));
    }
}

Commands

While using the Store component in your Symfony application along with the AiBundle, you can use the bin/console ai:store:setup command to initialize the store and bin/console ai:store:drop to clean up the store. To remove all documents from a store without dropping it, use bin/console ai:store:clear:

1
2
3
4
5
6
7
8

# config/packages/ai.yaml
ai:
    # ...

    store:
        chromadb:
            symfonycon:
                collection: 'symfony_blog'

1
2
3

$ php bin/console ai:store:setup symfonycon
$ php bin/console ai:store:clear symfonycon --force
$ php bin/console ai:store:drop symfonycon --force

Implementing a Bridge

The main extension points of the Store component is the StoreInterface, that defines the methods for adding, removing and querying vectorized documents in the store.

This leads to a store implementing the following methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

use Symfony\AI\Store\Document\VectorDocument;
use Symfony\AI\Store\Query\QueryInterface;
use Symfony\AI\Store\StoreInterface;

class MyStore implements StoreInterface
{
    public function add(VectorDocument|array $documents): void
    {
        // Implementation to add a document to the store
    }

    public function remove(string|array $ids, array $options = []): void
    {
        // Implementation to remove documents from the store
    }

    public function clear(array $options = []): void
    {
        // Implementation to remove all documents from the store
    }

    public function query(QueryInterface $query, array $options = []): iterable
    {
        // Implementation to query the store for documents
        return $documents;
    }

    public function supports(string $queryClass): bool
    {
        // Return true if the given query class is supported
        return false;
    }
}

Managing a store

Some vector store might requires to create table, indexes and so on before storing vectors, the ManagedStoreInterface defines the methods to setup and drop the store.

This leads to a store implementing two methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

use Symfony\AI\Store\ManagedStoreInterface;
use Symfony\AI\Store\StoreInterface;

class MyCustomStore implements ManagedStoreInterface, StoreInterface
{
    # ...

    public function setup(array $options = []): void
    {
        // Implementation to create the store
    }

    public function drop(array $options = []): void
    {
        // Implementation to drop the store (and related vectors)
    }
}

Clearing a store

To get rid of all documents in a store without dropping it, use clear():

1

$store->clear();

In contrast to ManagedStoreInterface::drop(), the store stays usable: documents can be added again right away, without calling setup() first, which makes clear() the method of choice for re-indexing:

1
2

$store->clear();
$indexer->index($documents);

Every store supports this operation, and almost all of them use the native mechanism of their backend to keep the table, index or collection in place - for example TRUNCATE TABLE for the SQL-based stores, _delete_by_query for Elasticsearch and OpenSearch, or deleteMany() for MongoDB.

The stores whose backend has no delete-all operation list the documents and remove them in batches instead, which is the case for Azure AI Search, Cloudflare, ChromaDB and S3 Vectors, as does Neo4j, which deletes its nodes in batched transactions. All of them use a sensible default batch size, which can be changed with the batch_size option:

1

$store->clear(['batch_size' => 250]);

The only exception is Vektor, which supports neither removing documents in bulk nor listing them. Its index is the storage directory itself, which is simply recreated.