Streaming JSON

7.3

The JsonStreamer component was introduced in Symfony 7.3 as an experimental feature.

Symfony can encode PHP data structures to JSON streams and decode JSON streams back into PHP data structures.

To do so, it relies on the JsonStreamer component, which is designed for high efficiency and can process large JSON data incrementally without needing to load the entire content into memory.

This component is ideal for handling APIs or interacting with third-party APIs. It transforms incoming JSON request payloads into PHP objects that your application can work with. Similarly, it converts processed PHP objects into a JSON stream for outgoing responses.

Installation

In applications using Symfony Flex, run this command to install the JsonStreamer component:

        1
        $ composer require symfony/json-streamer
    

Note

If you install this component outside of a Symfony application, you must require the vendor/autoload.php file in your code to enable the class autoloading mechanism provided by Composer. Read this article for more details.

Encoding Objects

JsonStreamer only works with PHP classes that have no constructor and are composed solely of public properties, like DTO classes. Consider the following Cat class:

        1
2
3
4
5
6
7
8
        // src/Dto/Cat.php
namespace App\Dto;

class Cat
{
    public string $name;
    public string $age;
}
    

To encode Cat objects into a JSON stream (e.g., to send them in an API response), first apply the #[JsonStreamable] attribute to the class. This attribute is optional, but it improves performance by pre-generating encoding and decoding files during cache warm-up:

        1
2
3
4
5
6
7
8
9
        namespace App\Dto;

use Symfony\Component\JsonStreamer\Attribute\JsonStreamable;

#[JsonStreamable]
class Cat
{
    // ...
}
    

Next, inject the JSON stream writer into your service. The service id is json_streamer.stream_writer, but you can also get it by type-hinting a $jsonStreamWriter argument with StreamWriterInterface.

Use the write() method of the service to perform the actual JSON conversion:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
        // src/Controller/CatController.php
namespace App\Controller;

use App\Dto\Cat;
use App\Repository\CatRepository;
use Symfony\Component\HttpFoundation\StreamedResponse;
use Symfony\Component\JsonStreamer\StreamWriterInterface;
use Symfony\Component\TypeInfo\Type;

class CatController
{
    public function retrieveCats(StreamWriterInterface $jsonStreamWriter, CatRepository $catRepository): StreamedResponse
    {
        $cats = $catRepository->findAll();
        $type = Type::list(Type::object(Cat::class));

        $json = $jsonStreamWriter->write($cats, $type);

        return new StreamedResponse($json);
    }
}
    

Tip

You can explicitly inject the json_streamer.stream_writer service by using the #[Target('json_streamer.stream_writer')] autowire attribute.

Decoding Objects

In addition to encoding, you can decode JSON into PHP objects.

To do this, inject the JSON stream reader into your service. The service id is json_streamer.stream_reader, but you can also get it by type-hinting a $jsonStreamReader argument with StreamReaderInterface. Next, use the read() method to perform the actual JSON parsing:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
        // src/Service/TombolaService.php
namespace App\Service;

use App\Dto\Cat;
use Symfony\Component\DependencyInjection\Attribute\Autowire;
use Symfony\Component\JsonStreamer\StreamReaderInterface;
use Symfony\Component\TypeInfo\Type;

class TombolaService
{
    private string $catsJsonFile;

    public function __construct(
        private StreamReaderInterface $jsonStreamReader,
        #[Autowire(param: 'kernel.root_dir')]
        string $rootDir,
    ) {
        $this->catsJsonFile = sprintf('%s/var/cats.json', $rootDir);
    }

    public function pickTheTenthCat(): ?Cat
    {
        $jsonResource = fopen($this->catsJsonFile, 'r');
        $type = Type::iterable(Type::object(Cat::class));

        /** @var iterable<Cat> $cats */
        $cats = $this->jsonStreamReader->read($jsonResource, $type);

        $i = 0;
        foreach ($cats as $cat) {
            if ($i === 9) {
                return $cat;
            }

            ++$i;
        }

        return null;
    }

    /**
     * @return list<string>
     */
    public function listEligibleCatNames(): array
    {
        $json = file_get_contents($this->catsJsonFile);
        $type = Type::iterable(Type::object(Cat::class));

        /** @var iterable<Cat> $cats */
        $cats = $this->jsonStreamReader->read($json, $type);

        return array_map(fn(Cat $cat) => $cat->name, iterator_to_array($cats));
    }
}
    

Tip

You can explicitly inject the json_streamer.stream_reader service by using the #[Target('json_streamer.stream_reader')] autowire attribute.

The examples above demonstrate two different approaches to decoding JSON data using JsonStreamer:

decoding from a stream (pickTheTenthCat)
decoding from a string (listEligibleCatNames)

Both methods handle the same JSON data but differ in memory usage and performance. Use streams if optimizing memory usage is more important. Use strings if performance is more important.

Decoding from a Stream

In the pickTheTenthCat method, the JSON data is read as a stream using fopen. This is useful for large files, as the data is processed incrementally rather than being fully loaded into memory.

To optimize memory usage, JsonStreamer creates ghost objects instead of fully instantiating them. These lightweight placeholders delay object creation until the data is actually needed.

Advantage: Efficient memory usage, ideal for very large JSON files.
Disadvantage: Slightly slower due to lazy loading.

Decoding from a String

In the listEligibleCatNames method, the entire JSON file is read into a string using file_get_contents. The decoder then instantiates all the objects immediately.

This approach is faster because all objects are created immediately, but it requires more memory.

Advantage: Faster, ideal for small to medium JSON files.
Disadvantage: Higher memory usage, unsuitable for large files.

Enabling PHPDoc Reading

The JsonStreamer component can read advanced PHPDoc type definitions (e.g., generics) and process complex PHP objects accordingly.

Consider the Shelter class that defines a generic TAnimal type, which can be a Cat or a Dog:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
        // src/Dto/Shelter.php
namespace App\Dto;

use Symfony\Component\JsonStreamer\Attribute\JsonStreamable;

/**
 * @template TAnimal of Cat|Dog
 */
#[JsonStreamable]
class Shelter
{
    /**
     * @var list<TAnimal>
     */
    public array $animals;
}
    

To enable PHPDoc parsing, run:

        1
        $ composer require phpstan/phpdoc-parser
    

Then, when encoding/decoding a Shelter instance, you can specify the concrete type information, and JsonStreamer will correctly interpret the JSON structure:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
        use App\Dto\Cat;
use App\Dto\Shelter;
use Symfony\Component\TypeInfo\Type;

$json = <<<JSON
{
  "animals": [
    {"name": "Eva", "age": 29},
    {...}
  ]
}
JSON;

// maps the TAnimal template in Shelter to the Cat concrete type
$type = Type::generic(Type::object(Shelter::class), Type::object(Cat::class));

$catShelter = $jsonStreamReader->read($json, $type); // will be populated with Cat instances
    

Configuring Encoding/Decoding

While it's usually best not to alter the shape or values of objects during serialization, sometimes it's necessary.

Configuring the Encoded Name

You can configure the JSON key for a property using the StreamedName attribute:

        1
2
3
4
5
6
7
8
9
10
11
12
        // src/Dto/Duck.php
namespace App\Dto;

use Symfony\Component\JsonStreamer\Attribute\JsonStreamable;
use Symfony\Component\JsonStreamer\Attribute\StreamedName;

#[JsonStreamable]
class Duck
{
    #[StreamedName('@id')]
    public string $id;
}
    

This maps the Duck::$id property to the @id JSON key:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
        use App\Dto\Duck;
use Symfony\Component\TypeInfo\Type;

// ...

$duck = new Duck();
$duck->id = '/ducks/daffy';

echo (string) $jsonStreamWriter->write($duck, Type::object(Duck::class));

// This will output:
// {
//   "@id": "/ducks/daffy"
// }
    

Configuring the Encoded Value

To transform a property's value during encoding, use the ValueTransformer attribute. Its nativeToStream option accepts a callable or a value transformer service id.

The callable must be a public static method or non-anonymous function with this signature:

        1
        $transformer = function (mixed $data, array $options = []): mixed { /* ... */ };
    

Then specify it in the attribute:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
        // src/Dto/Duck.php
namespace App\Dto;

use Symfony\Component\JsonStreamer\Attribute\JsonStreamable;
use Symfony\Component\JsonStreamer\Attribute\ValueTransformer;

#[JsonStreamable]
class Duck
{
    #[ValueTransformer(nativeToStream: 'strtoupper')]
    public string $name;

    #[ValueTransformer(nativeToStream: [self::class, 'formatHeight'])]
    public int $height;

    public static function formatHeight(int $value, array $options = []): string
    {
        return sprintf('%.2fcm', $value / 100);
    }
}
    

The following example transforms the name and height properties during encoding:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
        use App\Dto\Duck;
use Symfony\Component\TypeInfo\Type;

// ...

$duck = new Duck();
$duck->name = 'daffy';
$duck->height = 5083;

echo (string) $jsonStreamWriter->write($duck, Type::object(Duck::class));

// This will output:
// {
//   "name": "DAFFY",
//   "height": "50.83cm"
// }
    

Configuring the Decoded Value

To transform a property's value during decoding, use the ValueTransformer attribute. Its streamToNative option accepts a callable or a value transformer service id.

The callable must be a public static method or non-anonymous function with this signature:

        1
        $valueTransformer = function (mixed $data, array $options = []): mixed { /* ... */ };
    

Then specify it in the attribute:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
        // src/Dto/Duck.php
namespace App\Dto;

use Symfony\Component\JsonStreamer\Attribute\JsonStreamable;
use Symfony\Component\JsonStreamer\Attribute\ValueTransformer;

#[JsonStreamable]
class Duck
{
    #[ValueTransformer(streamToNative: [self::class, 'retrieveFirstName'])]
    public string $firstName;

    #[ValueTransformer(streamToNative: [self::class, 'retrieveLastName'])]
    public string $lastName;

    public static function retrieveFirstName(string $normalized, array $options = []): string
    {
        return explode(' ', $normalized)[0];
    }

    public static function retrieveLastName(string $normalized, array $options = []): string
    {
        return explode(' ', $normalized)[1];
    }
}
    

This will extract first and last names from a full name in the input JSON:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
        use App\Dto\Duck;
use Symfony\Component\TypeInfo\Type;

// ...

$duck = $jsonStreamReader->read(
    '{"name": "Daffy Duck"}',
    Type::object(Duck::class),
);

// The $duck variable will contain:
// object(Duck)#1 (1) {
//   ["firstName"] => string(5) "Daffy"
//   ["lastName"] => string(4) "Duck"
// }
    

Transforming Value Using Services

When callables are not enough, you can use a service implementing the ValueTransformerInterface:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
        // src/Transformer/DogUrlTransformer.php
namespace App\Transformer;

use Symfony\Component\JsonStreamer\ValueTransformer\ValueTransformerInterface;
use Symfony\Component\Routing\Generator\UrlGeneratorInterface;
use Symfony\Component\TypeInfo\Type;

class DogUrlTransformer implements ValueTransformerInterface
{
    public function __construct(
        private UrlGeneratorInterface $urlGenerator,
    ) {
    }

    public function transform(mixed $value, array $options = []): string
    {
        if (!is_int($value)) {
            throw new \InvalidArgumentException(sprintf('The value must be "int", "%s" given.', get_debug_type($value)));
        }

        return $this->urlGenerator->generate('show_dog', ['id' => $value]);
    }

    public static function getStreamValueType(): Type
    {
        return Type::string();
    }
}
    

Note

The getStreamValueType() method must return the value's type as it will appear in the JSON stream.

To use this transformer in a class, configure the #[ValueTransformer] attribute:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
        // src/Dto/Dog.php
namespace App\Dto;

use App\Transformer\DogUrlTransformer;
use Symfony\Component\JsonStreamer\Attribute\JsonStreamable;
use Symfony\Component\JsonStreamer\Attribute\StreamedName;
use Symfony\Component\JsonStreamer\Attribute\ValueTransformer;

#[JsonStreamable]
class Dog
{
    #[StreamedName('url')]
    #[ValueTransformer(nativeToStream: DogUrlTransformer::class)]
    public int $id;
}
    

Tip

Value transformers are called frequently during encoding and decoding. Keep them lightweight and avoid calls to external APIs or the database.

Configuring Keys and Values Dynamically

JsonStreamer uses services that implement the PropertyMetadataLoaderInterface to control the shape and values of objects during encoding/decoding.

These services are highly flexible and can be decorated to support dynamic configurations, providing more flexibility than attributes:

        1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
        namespace App\Streamer\SensitivePropertyMetadataLoader;

use App\Dto\SensitiveInterface;
use App\Streamer\ValueTransformer\EncryptorValueTransformer;
use Symfony\Component\DependencyInjection\Attribute\AsDecorator;
use Symfony\Component\DependencyInjection\Attribute\AutowireDecorated;
use Symfony\Component\JsonStreamer\Mapping\PropertyMetadata;
use Symfony\Component\JsonStreamer\Mapping\PropertyMetadataLoaderInterface;
use Symfony\Component\TypeInfo\Type;

#[AsDecorator('json_streamer.write.property_metadata_loader')]
class SensitivePropertyMetadataLoader implements PropertyMetadataLoaderInterface
{
    public function __construct(
        #[AutowireDecorated]
        private PropertyMetadataLoaderInterface $decorated,
    ) {
    }

    public function load(string $className, array $options = [], array $context = []): array
    {
        $propertyMetadataMap = $this->decorated->load($className, $options, $context);

        if (!is_a($className, SensitiveInterface::class, true)) {
            return $propertyMetadataMap;
        }

        // you can configure value transformers
        foreach ($propertyMetadataMap as $jsonKey => $metadata) {
            if (in_array($metadata->getName(), $className::getPropertiesToEncrypt(), true)) {
                $propertyMetadataMap[$jsonKey] = $metadata
                    ->withType(Type::string())
                    ->withAdditionalNativeToStreamValueTransformer(EncryptorValueTransformer::class);
            }
        }

        // you can remove existing properties
        foreach ($propertyMetadataMap as $jsonKey => $metadata) {
            if (in_array($metadata->getName(), $className::getPropertiesToRemove(), true)) {
                unset($propertyMetadataMap[$jsonKey]);
            }
        }

        // you can rename JSON keys
        foreach ($propertyMetadataMap as $jsonKey => $metadata) {
            $propertyMetadataMap[md5($jsonKey)] = $propertyMetadataMap[$jsonKey];
            unset($propertyMetadataMap[$jsonKey]);
        }

        // you can add virtual properties
        $propertyMetadataMap['is_sensitive'] = new PropertyMetadata(
            name: 'theNameWontBeUsed',
            type: Type::bool(),
            nativeToStreamValueTransformers: [fn() => true],
        );

        return $propertyMetadataMap;
    }
}
    

Although powerful, this approach introduces complexity. Decorating property metadata loaders requires a deep understanding of the internals.

For most use cases, attribute-based configuration is sufficient. Reserve dynamic loaders for advanced scenarios.

Marking Objects as Streamable

The JsonStreamable attribute marks a class as streamable. While not strictly required, it's highly recommended because it enables cache warm-up to pre-generate encoding/decoding files, improving performance.

It includes two optional properties: asObject and asList, which define how the class should be prepared during cache warm-up:

        1
2
3
4
5
6
7
        use Symfony\Component\JsonStreamer\Attribute\JsonStreamable;

#[JsonStreamable(asObject: true, asList: true)]
class StreamableData
{
    // ...
}
    

This work, including the code samples, is licensed under a Creative Commons BY-SA 3.0 license.

TOC

Version