Symfony AI - Store Component
The Store component provides a low-level abstraction for storing and retrieving documents in a vector store.
Installation
1
$ composer require symfony/ai-store
Purpose
A typical use-case in agentic applications is a dynamic context-extension with similar and useful information, for so called Retrieval Augmented Generation (RAG). The Store component implements low-level interfaces, that can be implemented by different concrete and vendor-specific implementations, so called bridges. On top of those bridges, the Store component provides higher level features to populate and query those stores with and for documents.
Indexing
Indexing is the process of converting documents into vector embeddings and storing them in a vector store.
All indexers implement IndexerInterface and share a common index() method.
Internally they use DocumentProcessor to run documents through the
pipeline: filter → transform → vectorize → store.
There are three indexer implementations to choose from:
DocumentIndexer accepts documents directly (as DocumentInterface instances):
1 2 3 4 5 6 7 8 9
use Symfony\AI\Store\Document\TextDocument;
use Symfony\AI\Store\Document\Vectorizer;
use Symfony\AI\Store\Indexer\DocumentIndexer;
use Symfony\AI\Store\Indexer\DocumentProcessor;
$vectorizer = new Vectorizer($platform, $model);
$indexer = new DocumentIndexer(new DocumentProcessor($vectorizer, $store));
$document = new TextDocument('id-1', 'This is a sample document.');
$indexer->index($document);
SourceIndexer loads documents via a LoaderInterface from a runtime-provided source (file path, URL, etc.):
1 2 3 4 5 6 7 8 9 10 11
use Symfony\AI\Store\Document\Loader\TextFileLoader;
use Symfony\AI\Store\Document\Vectorizer;
use Symfony\AI\Store\Indexer\DocumentProcessor;
use Symfony\AI\Store\Indexer\SourceIndexer;
$vectorizer = new Vectorizer($platform, $model);
$loader = new TextFileLoader();
$indexer = new SourceIndexer($loader, new DocumentProcessor($vectorizer, $store));
$indexer->index('/path/to/document.txt');
// or index multiple sources at once:
$indexer->index(['/path/to/doc1.txt', '/path/to/doc2.txt']);
ConfiguredSourceIndexer wraps a SourceIndexer with a pre-configured default source,
which is useful when the source is defined in configuration but should still be overridable at runtime:
1 2 3 4 5
use Symfony\AI\Store\Indexer\ConfiguredSourceIndexer;
$inner = new SourceIndexer($loader, new DocumentProcessor($vectorizer, $store));
$indexer = new ConfiguredSourceIndexer($inner, '/path/to/document.txt');
$indexer->index(); // uses the configured source
See the RAG Implementation cookbook for more advanced usage in combination with an Agent.
Retrieving
The opposite of indexing is retrieving. The Retriever is a higher level feature that allows you to search for documents in a store based on a query string. It vectorizes the query and retrieves similar documents from the store:
1 2 3 4 5 6 7 8
use Symfony\AI\Store\Retriever;
$retriever = new Retriever($store, $vectorizer);
$documents = $retriever->retrieve('What is the capital of France?');
foreach ($documents as $document) {
echo $document->getMetadata()->getSource();
}
The retriever accepts optional parameters to customize the retrieval:
$options: An array of options to pass to the underlying store query (e.g., limit, filters)
All stores read the limit option, which sets the maximum number of returned documents:
1
$documents = $retriever->retrieve('What is the capital of France?', ['limit' => 5]);
Reranking
The RerankerListener sorts the retrieved documents again with a
cross-encoder model and keeps the limit best ones. Without a limit option, it keeps the number given
to its limit argument.
The reranker gives better results when it can choose from more documents than it keeps. Set the
overFetchFactor argument and register the listener on both events. Then the store returns limit
times overFetchFactor documents, and the reranker keeps the limit best ones:
1 2 3 4 5 6 7 8 9 10 11 12
use Symfony\AI\Store\Event\PostQueryEvent;
use Symfony\AI\Store\Event\PreQueryEvent;
use Symfony\AI\Store\EventListener\RerankerListener;
$rerankerListener = new RerankerListener($reranker, limit: 5, overFetchFactor: 4);
$dispatcher->addListener(PreQueryEvent::class, $rerankerListener);
$dispatcher->addListener(PostQueryEvent::class, $rerankerListener);
$retriever = new Retriever($store, $vectorizer, $dispatcher);
// the store returns 20 documents, the reranker keeps the 5 best ones
$documents = $retriever->retrieve('What is the capital of France?', ['limit' => 5]);
Note
When the listener is only registered on PostQueryEvent, the overFetchFactor has no effect.
The over-fetched limit must stay within the limits of the store, for example 50 for Cloudflare
Vectorize or 250 for Typesense.
Similarity Search Examples
- Similarity Search with Cloudflare (RAG)
- Similarity Search with Doctrine (RAG)
- Similarity Search with Manticore Search (RAG)
- Similarity Search with MariaDB (RAG)
- Similarity Search with Meilisearch (RAG)
- Similarity Search with memory storage (RAG)
- Similarity Search with Milvus (RAG)
- Similarity Search with MongoDB (RAG)
- Similarity Search with Neo4j (RAG)
- Similarity Search with OpenSearch (RAG)
- Similarity Search with Pinecone (RAG)
- Similarity Search with Qdrant (RAG)
- Similarity Search with SurrealDB (RAG)
- Similarity Search with SQLite (RAG)
- Similarity Search with Symfony Cache (RAG)
- Similarity Search with Typesense (RAG)
- Similarity Search with Vektor (RAG)
- Similarity Search with Weaviate (RAG)
- Similarity Search with Supabase (RAG)
Note
Both InMemory and PSR-6 cache vector stores will load all the data into the
memory of the PHP process. They can be used only the amount of data fits in the
PHP memory limit, typically for testing.
Supported Stores
See Store Bridges for installation and configuration details for all supported bridges:
- Azure AI Search
- ChromaDB (requires
codewithkyrian/chromadb-php) - ClickHouse
- Cloudflare Vectorize
- Doctrine (requires
doctrine/orm) - Elasticsearch
- InMemory & Symfony Cache (for development and testing)
- ManticoreSearch
- MariaDB (requires
ext-pdo, MariaDB 11.7+) - Meilisearch
- Milvus
- MongoDB Atlas (requires
mongodb/mongodb) - Neo4j
- OpenSearch
- Pinecone
- Postgres with pgvector (requires
ext-pdo) - Qdrant
- Redis Stack (requires
ext-redis) - S3 Vectors
- SQLite (requires
ext-pdo_sqlite) - Supabase
- SurrealDB
- Typesense
- Vektor
- Weaviate
Document Loader
Creating and/or loading documents is a critical part of any RAG-based system, as it provides the foundation for the system to understand and respond to queries. Document loaders are responsible for fetching and preparing documents for indexing and retrieval.
To help loading documents and integrate them into your RAG system, you can use the provided document loaders or create your own custom loaders to suit your specific needs:
- CsvLoader
- DirectoryLoader
- InMemoryLoader
- JsonFileLoader
- MarkdownLoader
- RssFeedLoader
- RstLoader
- RstToctreeLoader
- TextFileLoader
The DirectoryLoader scans a directory and delegates each file to a sub-loader chosen by its extension. The sub-loaders are injected as a map of file extension (without the leading dot) to a loader, and recursion into subdirectories can be toggled:
1 2 3 4 5 6 7 8 9 10
use Symfony\AI\Store\Document\Loader\DirectoryLoader;
use Symfony\AI\Store\Document\Loader\MarkdownLoader;
use Symfony\AI\Store\Document\Loader\TextFileLoader;
$loader = new DirectoryLoader([
'md' => new MarkdownLoader(),
'txt' => new TextFileLoader(),
], recursive: false);
$documents = $loader->load('/path/to/directory');
Files whose extension has no registered loader are skipped.
Create a Custom Loader
The main extension points of the Store component for document loaders is the LoaderInterface, that defines the method to load a document from a source. This leads to a loader implementing one method:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
use Symfony\AI\Store\Document\LoaderInterface;
use Symfony\AI\Store\Document\Metadata;
use Symfony\AI\Store\Document\TextDocument;
use Symfony\Component\Uid\Uuid;
class MyDocumentLoader implements LoaderInterface
{
public function load(?string $source = null, array $options = []): iterable
{
$content = ...
yield new TextDocument(Uuid::v7()->toRfc4122(), $content, new Metadata($metadata));
}
}
Document Transformer
Transformers change documents after loading and before vectorizing, for example to split them into smaller
chunks. Pass them to the DocumentProcessor, which runs them in the given order.
The transformers of the Store component live in the Symfony namespace.
Splitting Documents by Section
The TextSplitTransformer cuts a document into chunks of a fixed size, so a chunk can start in the middle of one section and end in the next one. To keep sections apart, run the RegexSplitTransformer first. It splits the content on each match of a regular expression, for example on Markdown headings:
1 2 3 4 5 6 7 8 9 10
use Symfony\AI\Store\Document\Transformer\RegexSplitTransformer;
use Symfony\AI\Store\Document\Transformer\TextSplitTransformer;
use Symfony\AI\Store\Indexer\DocumentProcessor;
$processor = new DocumentProcessor($vectorizer, $store, transformers: [
// starts a new section at each Markdown heading
new RegexSplitTransformer('/^#{1,6}[ \t]+.+$/m'),
// splits sections longer than 1000 characters
new TextSplitTransformer(chunkSize: 1000, overlap: 200),
]);
Each section becomes its own document. Its metadata holds the ID of the original document in _parent_id
and the section text in _text. If the matched text is a Markdown heading, the metadata also holds the
heading text without the # markers in _title and the heading level in _depth. Content before the
first match becomes its own section. A section without a heading of its own keeps the _title and _depth
of the original document.
If the TextSplitTransformer splits a long section into chunks, the _parent_id of these chunks holds the ID
of the section. The section itself is not stored, so this ID doesn't point to a document in the store.
By default, each section keeps the matched text at its start. Pass keepDelimiter: false to remove it.
To use both transformers as one, wrap them in a
ChainTransformer.
Note
The strip_formatting option of the MarkdownLoader
removes the # markers of all headings, so a pattern for Markdown headings finds nothing to split on. Don't use
this option if you split the content on headings.
The heading pattern above also matches # lines inside fenced code blocks, for example a comment in a
shell script. Such a line starts a new section and becomes its title.
Commands
While using the Store component in your Symfony application along with the AiBundle,
you can use the bin/console ai:store:setup command to initialize the store and bin/console ai:store:drop to clean up the store.
To remove all documents from a store without dropping it, use bin/console ai:store:clear:
1 2 3 4 5 6 7 8 9 10 11
# config/packages/ai.yaml
ai:
# ...
store:
chromadb:
symfonycon:
collection: 'symfony_blog'
# optional, a service implementing Codewithkyrian\ChromaDB\Embeddings\EmbeddingFunction.
# Required to query the store with a TextQuery, which ChromaDB embeds client-side.
embedding_function: 'app.chromadb.embedding_function'
1 2 3
$ php bin/console ai:store:setup symfonycon
$ php bin/console ai:store:clear symfonycon --force
$ php bin/console ai:store:drop symfonycon --force
Implementing a Bridge
The main extension points of the Store component is the StoreInterface, that defines the methods
for adding, removing, querying and counting vectorized documents in the store. It extends \Countable, so a store
reports the number of documents it holds and can be handed to count() directly.
This leads to a store implementing the following methods:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
use Symfony\AI\Store\Document\VectorDocumentInterface;
use Symfony\AI\Store\Query\QueryInterface;
use Symfony\AI\Store\StoreInterface;
class MyStore implements StoreInterface
{
public function add(VectorDocumentInterface|array $documents): void
{
// Implementation to add a document to the store
}
public function remove(string|array $ids, array $options = []): void
{
// Implementation to remove documents from the store
}
public function clear(array $options = []): void
{
// Implementation to remove all documents from the store
}
public function query(QueryInterface $query, array $options = []): iterable
{
// Implementation to query the store for documents
return $documents;
}
public function supports(string $queryClass): bool
{
// Return true if the given query class is supported
return false;
}
public function count(): int
{
// Implementation to count the documents in the store
return 0;
}
}
Managing a store
Some vector store might requires to create table, indexes and so on before storing vectors, the ManagedStoreInterface defines the methods to setup and drop the store.
This leads to a store implementing two methods:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
use Symfony\AI\Store\ManagedStoreInterface;
use Symfony\AI\Store\StoreInterface;
class MyCustomStore implements ManagedStoreInterface, StoreInterface
{
# ...
public function setup(array $options = []): void
{
// Implementation to create the store
}
public function drop(array $options = []): void
{
// Implementation to drop the store (and related vectors)
}
}
Clearing a store
To get rid of all documents in a store without dropping it, use clear():
1
$store->clear();
In contrast to ManagedStoreInterface::drop(), the store stays usable: documents can be added
again right away, without calling setup() first, which makes clear() the method of choice
for re-indexing:
1 2
$store->clear();
$indexer->index($documents);
Every store supports this operation, and almost all of them use the native mechanism of their backend
to keep the table, index or collection in place - for example TRUNCATE TABLE for the SQL-based
stores, _delete_by_query for Elasticsearch and OpenSearch, or deleteMany() for MongoDB.
The stores whose backend has no delete-all operation list the documents and remove them in batches
instead, which is the case for Azure AI Search, Cloudflare, ChromaDB and S3 Vectors, as does Neo4j,
which deletes its nodes in batched transactions. All of them use a sensible default batch size, which
can be changed with the batch_size option:
1
$store->clear(['batch_size' => 250]);
The only exception is Vektor, which supports neither removing documents in bulk nor listing them. Its index is the storage directory itself, which is simply recreated.
Counting documents
To know how many documents a store holds, use count():
1
$total = $store->count();
Since StoreInterface extends \Countable, the store can be passed to count() as well, which
also makes assertCount() work on a store in tests:
1 2 3
$total = count($store);
$this->assertCount(3, $store);
Every store supports this operation and uses the native counting mechanism of its backend - for example
SELECT COUNT(*) for the SQL-based stores, _count for Elasticsearch and OpenSearch, an aggregation
for Weaviate, or the exact count endpoint for Qdrant, which is asked instead of the estimated point count
reported by the collection info.
Three stores have to read all their entries to count them, because their backend offers no count operation: Symfony Cache, S3 Vectors and Vektor.
Keep in mind that the number is as fresh as the backend makes it. Stores whose backend indexes documents asynchronously - Elasticsearch, OpenSearch, Meilisearch and Pinecone among them - can report a count that does not include documents added moments ago:
1 2 3 4
$store->add($documents);
// may still be 0 until the backend has indexed the documents
$store->count();