Dailymotion
Dailymotion, one of the world's largest independent video entertainment website, is now powered by symfony. Dailymotion is among the top 60 websites worldwide. In December 2008, more than 41,9 million unique users visited the website and they viewed more than 922 million online videos (source: comScore, December 2008). Dailymotion is also the number 2 French site in the world.
Founded in 2005, the Dailymotion website has always been powered by PHP and managed by a team of in-house developers. Over the years, a custom framework has been developed to power the platform.
But mid-2008, they decided to move away from their home-made framework and migrate to an Open-Source framework.
Dailymotion by the numbers
What to know all the nitty-gritty details of the Dailymotion architecture? Here are some interesting numbers:
- 50 PHP machines serving about 5000 PHP pages per second
- 17 DB machines
- 30 memcache machines for a total of 210 Gb of memory storage
- 60 Gb/s of bandwidth
- 1.2 petabytes of storage
And that does not account for the machines that encode the videos, host the search engine, ... All in all, they have more than 430 machines!
Why migrate to an Open-Source framework?
Even if Dailymotion has more than 10 developers dedicated to the website development, they wanted to be more focused on their business, instead of being forced to maintain a framework.
Over time, they started to feel the pain of maintaining their own framework: the maintenance of the framework itself started to take way too much time: adding new features, documenting them, training new developers, keep everything coherent, etc...
By choosing an Open-Source framework with a large community, they free themselves from everything other than their business.
Olivier Poitrey, one of the co-founder and CTO of Dailymotion, sums up the main advantages of moving to a framework:
- Don't reinvent the wheel
- Maintain less code
- Better compliance to coding conventions
- Implement new processes like unit/functional testing easily
Why symfony?
Thanks to the creativity of their marketing department, the Dailymotion developers add new features on a nearly everyday basis. So, rewriting the website from scratch was out of question. They just could not have afforded to stop all new developments, waiting for the new code base to be finished.
They needed a way to migrate step by step from the old code base to the new one, and keep adding new features at the same time. And symfony was the ideal candidate. Why?
First, because the symfony framework is one of the most well-known PHP frameworks, and one used by many high-traffic websites.
Then, the framework is well decoupled, meaning that each component can be used by itself, without the whole framework.
Eventually, the other big selling point of symfony was its flexibility. It is very easy to override the default behaviors or add new ones if needed. And that's exactly what they needed.
The migration process
The main challenge of the migration process was to avoid breaking too many things by introducing small doses of symfony.
The first change was to make symfony handle the request, dispatch it to the
old code, get back the response content, and send back the response to the
browser. Wrapping the existing code was made possible by using the symfony 2.0
sfRequestHandler class. We decided to use symfony 2.0 for its raw speed,
even if it is not public yet.
By the way, that is the classic process of the symfony framework evolution. Sensio Labs uses symfony for its customers and a lot of new features are added based on their needs and feedback. For the Dailymotion migration, Sensio Labs helped Dailymotion define the migration strategy and the target architecture.
The next step was the migration of the mod_rewrite rules to the symfony routing. Thanks to the great flexibility of the Symfony routing, Dailymotion has been able to extend the default behavior to fit its unique needs: object parameters, labeled variables, segment grouping, arbitrary ordered optional segments, centralized formats, ...
Along the way, the Dailymotion team contributed back code to the symfony community. Some code has already been committed and is already available as part of the symfony 1.2 release. Some other enhancements are proposed as tickets.
One of the other big changes was the introduction of massive unit and functional tests. To migrate with confidence, more than 5000 unit and functional tests have been written, using lime and the symfony test browser.
During phase 1, the following components of symfony has been used:
What's next?
Even if the Dailymotion website has only been powered by symfony for less than a week, the next migration steps are already planned:
- Introduction of symfony controllers
- Usage of symfony's facilities like
sfI18n,sfUser,sfCookieSessionStorage, ... - Usage of symfony's 2.0 service container
- Usage of symfony's model system (Doctrine)
Are you still using YouTube for your videos? I think it's time we all switch to Dailymotion ;)
It's a great achievements! But why is Symfony 2.0 not public?
@avorobiev: because symfony 2.0 is in a pre-alpha stage, where we test a lot of things. As soon as the main elements will be more stable, it will be published. But keep in mind that most components in symfony 2.0 are just the same as in symfony 1.2: the request, the response, the user, the form, the routing, ...
Great news ! I'm happy to see that this kind of web services use symfony ! A small step for the man, a big step for the humanity.
Symfony 2.0 is supposed to use PHP 5.3, no ? So their server runs an unstable PHP version ?
Good news for sf2, anyway ... seems that some parts are already quite stable.
That's absolutely fantastic news.
But it would be good to see SF 2.0 in SVN :)
@Tonio: As I said before, symfony 2.0 is still a moving target. And for now, it is still compatible with PHP 5.2.
Just i can say : GUAU !!!
Do you imagine, when we can say microsoft.com is powered by Symfony :D:D:D
My opinion is that is a weird way to announce symfony 2.0 to the community. Why no code and no specs are published?
But i'll stick on the good news ...
+1 for the Great news!
I'm happy to see the framework I choose, is also their choice ;)
Keep up the good work!
@tenshu: it is not the first announce on symfony 2.0. I have talked about it at symfony camp 2007 and symfony camp 2008. Nothing new here.
well done symfony! great news.
Congratulation!
This is splendid news - it renews my enthusiasm for checking out the latest stuff. I shall look forward to seeing what things the team have cooked up for us in s2.0!
Congratulations to the core team, and all sf contributors, for this excellent win.
This makes me even more excited about Symfony 2.0
What is sfServiceContainer about?
@15lloyd27
some dependency injection related stuff :)
@lloyd27: read more about it here: http://fabien.potencier.org/talk/17/symfony-camp-2008-beyond-1-2
Ok thanks :)
It is really excellent news that such a big site is making use of symfony. Perhaps we could get some more details about how the 2.0 changes are shaping up?
It's quite impressive, but i can see a few drawbacks to this news : 1/ they use a non-released version of symfony. 2/ they used consulting from sensio labs themselves (which is a web agency) 3/ they haven't released any code.
So, to sum it up, they're using something that nobody else can use, and they've been doing this work by paying a web agency.
I'm not sure it's such a great day for the "open source" spirit of this project (which i find remarkable and use myself for my projects).
@Ben - I think we'll see the non-DM parts of the effort added into symfony soon. True, we don't get the code at the same time, but I think we will get it - which is still a huge open source benefit.
Meanwhile this is great news, IMO, since it draws attention to the project and as a result may strengthen the size of the contributing community.
@Ben:
Some more information:
1/ As I have already stated in the comments, and as shown in the post schema, the Dailymotion code is based on symfony 1.2 libraries (request, response, routing, cache, ...), which are all free to uses, free to download, and available today. The only code they have used from symfony 2.0 is the sfRequestHandler class which represents 100 PHP line of codes. And this class will be released as part of the symfony 2.0 version.
2/3/ Sensio Labs does consulting for Dailymotion, helping them with the software architecture, but they write all the code. As far as contributions are concerned, they have already contributed code to the symfony routing system to make it more flexible. They have opened tickets with patches to add some new features to symfony. They have posted to the symfony developer mailing-list to suggest enhancements. And this is the spirit of the Open-Source philosophy. Not so many symfony share as much as they do. What would you expect? That they release their website code under an Open-Source license? Come on. Where can I download everything you have ever coded with symfony?
Dailymotion uses symfony, like many others, and they don't have to release anything they have done with it. That's the power of the MIT license. But thanks to people like Dailymotion, and many others, Sensio Labs have some money to work on the symfony project, add new features, add documentation, and release everything under a free license, with no string attached. I think that's huge!
bravo !
Indeed very good news! :)
As I expect my symfony-powered website to grow quickly, I'm happy to learn that the framework will be able to take the heat.
I'd switch to Dailymotion if they had an API :)