Push it to the limits - Symfony2 for High Performance needs

Request flow through SOA (Service Oriented Architecture)

As we've said – the platform is structured into separate webservices serving as a REST APIs and Symfony2 application is one of the services.

Let's talk about three layers: Frontend >> Webservice >> Data Storage.

1. Frontend - when the end-client visits a frontend-type website, it communicates internally with a specific Web service (as described in this article).
2. Web service - web services handle large amounts of the business logic. Functionalities aren't divided on many sites, but everything is kept in a specific service. It's a well-known method of handling in software architecture separation of concerns principles
3. Data Storage - services are using their own data storages. Frontend websites mustn't connect directly to the database handled by a web service as it would lead to bypassing control over a platform logic.

(click on the image to expand it)

Application's server architecture

To maintain high performance and high-availability, the Symfony2 service uses several application's servers - in redundant configuration.

1. Requests are handled by a HAProxy load balancer which distributes them to Varnish reverse proxies.
2. We use Varnish only as a cache layer, not as another load balancer. There are separate Varnish instances on each of the application's servers. That way we don't end up with SPOF - Single Point Of Failure. It makes cache hits lower, but it favors availability over performance here (which is still not a problem). Single Varnish can handle up to 12,000 requests per second.
3. The last piece of the puzzle is an Apache2 server with Symfony2 based application. We're using PHP 5.4 running as PHP-FPM with APC enabled. That layer handles 700 req/s.

Storing data for High-Performance

Of course, every web-application uses some kind of data-storage. We've managed a nice setup here, too.

1. Redis - it's extremely fast, in-memory data storage. Think of Memcache on steroids - with a persistence mechanism, HA setup and so on. We can store 160,000,000 records, including 98% of the persistent ones i.e., won't disappear after server outage. Key space hits stay at the average level of 15,000 hits/s. Symfony2 is integrated using SncRedisBundle with Predis library.
2. MySQL - most known relational database. We're mostly using it as a last (third) cache layer, for non-expiring resources. We're using Doctrine DBAL for connections. Our MySQL stores 300,000,000 records - that's 400GB of pure data.

So, as you can see, some unusual things are happening here - Redis is our primary data store and MySQL is our last cache layer.

Clear project structure - Bundles

Symfony2 doesn't impose full structure of your project, it's actually quite flexible. Basically, you can structure your code into Bundles which maintains Symfony-related code and more generic Components which can handle common tasks, including those not necessarily strictly related to the Symfony2 ecosystem.

Following this concept, we're mostly using bundles to divide our Project into logically-connected parts. We barely modified the Symfony2 Standard structure, so our code base is very easy to understand for developers with Symfony2 experience.

Extending codebase - EventDispatcher Component

Do you need to change the response format in all your controllers? It's easy, just add new ResponseFormatListener and listen for kernel.response event.

We have seen so much spaghetti code in so many projects that we really liked the popularization of Events concepts in Symfony. It's nothing new in software patterns, just old Observer Pattern, but before, in legacy PHP frameworks it wasn't so commonly used.

In addition to origin Symfony2 events, we've also chosen to add our custom ones. Using Event Listeners we can keep code clean, methods dispatch their specific events and that way new parts of the code can connect to the existing parts without actually making in-code changes.

Because performance is crucial in the project, we're evaluating the performance impact. The nice thing is that this mechanism comes with hardly any performance overhead. Internally it's simple array that stores event listeners' instances. Virtually worry-free!

Retrieving requests' data - OptionsResolver Component

While designing this application, we're also considering the most efficient way of retrieving and validating data from request content. We needed to smoothly transform request's data into DTO (Data Transfer Object). That way we won't get stuck with hard-to-maintain associative arrays and we'll stick to the structure of Request classes.

Basically, we ended with passing queries with Query String and considered different Symfony2 mechanisms...

You could use Form Component, build request structure in a Form Type and pass Request to it. It's nice and has rich features, but comes with huge overhead. We also didn't need or want advanced options like having nested fields.

Another way would be to pass data in a request's content in a JSON structure. Use a Serializer component (like great JMSSerializer or even Symfony Serializer Component) and validate resulting DTO Request's objects. There are still two points that could lead to performance bottleneck: serializing and validating with the Validator Component.

Thus, we didn't need any advanced validation (just checking required options and some basic format validation). Requests' format structure is also designed to be simple, and we've chosen... OptionsResolver Component. It's the same one you use when making options for your forms. We pass to it the GET array and on the output we receive a nicely validated and structured model object (it's an array to DTO normalization).

One nice thing about handling validation, exceptions come with verbose messages so they're ideal for debugging purposes and for presenting readable exceptions for API Endpoints.

Example of a handling request

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

<?php

namespace Octivi\WebserviceBundle\Controller;

/**
 * @Route("/foo")
 */
class FooController extends BaseController
{
    /**
     * Sample requests:
     *  - OK:    api.com/foo/bars?type=simple&offset=10
     *  - OK:    api.com/foo/bars?type=simple
     *  - Wrong: api.com/foo/bars?offset=10
     *  - Wrong: api.com/foo/bars?limit=many
     *
     * @Route("/bars")
     */
    public function barsAction()
    {
        $request = new GetBarsRequest($this->getRequest()->query->all());
        $results = $this->get('main.foo')->getBars($request);
        return $results;
    }
}

<?php

namespace Octivi\WebserviceBundle\Request;
    
class GetBarsRequest extends AbstractRequest
{
    protected $type;
    protected $limit;
    protected $offset;

    public function __construct(array $options = array())
    {
        parent::__construct($options);
    }

    protected function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        parent::setDefaultOptions($resolver);

        $resolver->setRequired(array(
            'type',
        ));

        $resolver->setOptional(array(
            'limit',
            'offset'
        ));

        $resolver->setAllowedTypes(array(
            'limit' => array('numeric'),
            'offset' => array('numeric')
        ));
            
        $resolver->setDefaults(array(
            'limit' => 10,
            'offset' => 0
        ));

        $resolver->setAllowedValues(array(
            'type' => array('simple', 'extended'),
        ));    
    }

    // ...
}

Pretty straightforward, isn't it? And we've achieved everything we want:

• Basic validation

  • Setting required fields
  • Optional parameters
  • Handling data types (numeric) and allowed values
  • Default parameters
• Nice looking DTO representation of a request

Keeping configuration in-code - Annotations

Yes, we're using annotations in a high-performance application. Does that sound weird? Maybe, but we love this mechanism!

Annotations are just PHPDoc-like texts. They're parsed under a cache warm-up process and transformed to plain PHP files. As a matter of fact, it doesn't matter if you use XML, YAML or annotations, because they all end up transformed into plain PHP files.

We're using annotations as much as possible:

• Routing - as we've shown in previous barsAction, we declare our routes via @Route annotations. We have found it cleaner to keep such declarations within controllers' code as opposed to dividing it into separate YML/XML files. That way, developers don't have to jump from file to file searching for routes. Keep in mind, that we have declared loading of such annotations in a global app/config/routing.yml file.
• Services - it's the next thing you can place in- code. Using JMSDiExtraBundle we don't have to worry about maintaining YAML files with Service Container declaration. Everything ends in service classes. Still, configuration of DI for external classes is done via XML files.

Example event listener configured with Annotations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

<?php

namespace Octivi\WebserviceBundle\EventListener;

use JMS\DiExtraBundle\Annotation\Service;
use JMS\DiExtraBundle\Annotation\Observe;
use JMS\DiExtraBundle\Annotation\InjectParams;
use JMS\DiExtraBundle\Annotation\Inject;
use Symfony\Component\HttpKernel\Event\FilterResponseEvent;

/**
 * @Service
 */
class FormatListener
{

    /**
     * Constructor uses JMSDiExtraBundle for dependencies injection.
     * 
     * @InjectParams({
     *      "em"         = @Inject("doctrine.orm.entity_manager"),
     *      "security"   = @Inject("security.context")
     * })
     */
    function __construct(EntityManager $em, SecurityContext $security) {
        $this->em = $em;
        $this->security = $security;
    }

    /**
     * @Observe("kernel.response", priority = 0)
     */
    public function onKernelResponse(FilterResponseEvent $event)
    {
        $this->em->...;
    }
}

Making rich CLI commands - Console Component

Another extensively-used Symfony component is Console Component, our most used component since most of the new features for the application come as CLI commands.

In previous-gen PHP frameworks or other popular PHP-based software (e.g. Wordpress, Magento) making CLI commands gave us a serious headache. There weren't any standardized components or they were so lacking in features that everyone had to present his own solutions.

In Symfony2, we found a cool framework for making CLI commands. We can set command names, required options and arguments. We discovered a good practice to make accurate descriptions of the commands. They're self-documenting and there's no reason to add them to the standard form of documentation. It's especially great when you're developing with Agile methods. When new functionalities come fast, you'll end up with lots of out-dated versions of your paper-documentation.

We're using Console Component to create administrative tools and even long running processes. The longest one took about 6 days. Nice proof of lack of memory-leaks in the Symfony ecosystem!

1
2
3
4
5
6
7
8
9
10

$ php app/console octivi:test-command --help
Usage:
 octivi:test-command [-l|--limit[="..."]] [-o|--offset[="..."]] table

Arguments:
 table                 Database table to process

Options:
 --limit (-l)          Limit per SQL query. (default: 10)
 --offset (-o)         Offset for the first statement (default: 0)

Keeping an eye on the application - Profiler, Stopwatch & Monolog trio

To maintain control over potential performance-leaks, we strongly use HttpKernel's Profiler with Stopwatch Component. It's easier to spot Application methods that have un-expected, longer execution times as well as inefficient database queries (and we're talking here about MySQL and Redis).

When maintaining a bigger project, especially in an SOA environment, verbose logging is a must. Otherwise, you won't be able to track problems efficiently with e.g. API methods usage or connections with 3rd party Web services.

For that, we use rich-configured Monolog (all from app/config/config_prod.yml!). We're logging all un-expected things in a standard prod.log and leaving more verbose logs to specific channels or different files.

We've worked out some nice logging patterns with that config. Since we don't use Fingers Crossed handler, we always add a context as verbose as possible that we can to each log line. That way, calls to external Web services look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

$exception = null;

try {
    $response = $this->request($request);
} catch (\Exception $e) {
    $exception = $e;
}
 
if (null !== $exception) {
    $this->logger->error('Error response from XXX', array(
        'request'   => $request,
        'response'  => $response,
        'exception' => (string) $exception,
    ));

    throw $exception;
} else {
    $this->logger->debug('Success response from XXX', array(
        'request'  => $request,
        'response' => $response,
    ));
}

Doctrine ORM

We use MySQL mostly as a cache layer, storing in it serialized BLOBs. We don't use Doctrine ORM as it would affect performance unnecessarily. Instead, we just use Doctrine DBAL to retrieve associative arrays and than we have a Model layer which handles deserialization/normalization of records.

I must add that it's not as much about volume of stored data, but performance requirements for retrieving a single record. ORM adds big over-head while hydrating objects (deserialization). If you accept it, you can use Doctrine ORM even if you have +200 million records. ORM performance will be the same as if you have 1000.

Twig

The application returns responses mostly in a JSON format. Therefore we don't use Twig in a standard requests flow. However, we do keep it for rendering some administrative dashboards.

Who's behind the application?

Platform was designed by the Poland-based software house Octivi. We specialize in building scalable architectures focused on high-performance and availability. We'd also like to acknowledge the cooperation received from the client IT department.

Related articles

• Handling 1 Billion requests a week with Symfony2 - our overview of the whole application.
• Building A Growing Startup Architecture Using HAProxy, PHP, Redis And MySQL To Handle 1 Billion Requests A Week - details of the Hardware Architecture .