The resolutionfinder.org database aims to facilitate access to UN agreements with the purpose of improving the process of implementation. Within just 15 man days a team of 3 developers put together an easy to navigate frontend that encompassed the entire breadth of data that was collected by a volunteer team of graduate students and young professionals from all over Europe over the course of two years. The resulting application was one of the highlights at the „UN-connecting the World“ conference held in Geneva in late May 2010, where the application was launched as a useable concept and technology show case that will form the basis for future developments.

The rapid development of the application was only possible due to the powerful out of the box features of the popular symfony PHP framework with tight integration to Lucene and Solr via the sfSolrPlugin.

Customer Overview

The idea behind resolutionfinder.org was to create an instrument that facilitates access to UN agreements with the purpose of improving the process of implementation. Unlike other databases available at the moment, resolutionfinder.org not only compiles documents, but it also extracts clauses relevant for implementation and provides the evolution of documents and clauses. Right now, the database contains a substantial amount of information in four thematic areas: Clean Drinking Water, Malaria, Small Arms and Light Weapons and Women and Education.

The development of resolutionfinder.org has been supported in the past two years by the World Federation of United Nations Associations (WFUNA) and especially by the United Nations Association of Germany (DGVN). The project is currently in negotiations for a partnership with the International Security Network (ISN).

Key Search features of symfony

  • Powerful ORM capabilities through the integration of Doctrine ORM, with automatic generation of model and form classes
  • Admin generator to quickly setup powerful admin capabilities that require minimal work even if the underlying schema evolves
  • Tight integration with Solr through the sfSolrPlugin tying together the Doctrine model classes with minimal work
  • Availability of several more plugins like sfDoctrineGuardPlugin for user management, sfFormExtra for easy creation of web 2.0 forms, sfAdminDashPlugin for easy UI tweaking in the admin tool and several more
  • Great debugging environment and deployment tools to ease all steps of the maintenance process
  • Due to the Open Source nature of the entire application stack means that no licensing costs were incurred for the entire application

Challenges

Since resolutionfinder.org is an entirely volunteer effort that currently does not have an IT budget and is mostly comprised of experts in the domain of UN research and not application development, the available development resources were slim. It became clear that even with development time and hosting sponsored by the Liip AG of Switzerland, there would effectively be only 15 man days that could be dedicated to the development of the frontend. Especially as in parallel there was still work going on with migrating the excel sheets containing the last 2 years of research into the relational database.

The goal was to provide full text search capabilities with facet based filtering, displaying of documents and their containing clauses including their historical development. Users should also be able to register in order to bookmark and comment on clauses and documents. Furthermore as many aspects of the complex database schema should be administerable from the admin tool to ensure that issues in the data that was imported from the old excel sheets could be fixed by the researchers themselves without requiring interventions by the development team.

Solutions

The team already had an existing database schema and a symfony based administration tool which was to handle the excel sheet import. Due to the use of the Doctrine ORM and sfSolrPlugin loading the data into Solr was fully automated through just a small configuration file, which mapped properties and methods in the data model to fields in Solr. As a result after importing an excel sheet it was automatically available in Solr for searching without any additional code. The same configuration file also generated the main Solr configuration files. The Doctrine model classes also made it possible to hook in important denormalization logic in order to ensure that the complex relations between documents and clauses could be read out of the database with minimal overhead.

Finally sfSolrPlugin bundled a fully working Solr installation using Jetty as the servlet container including administrative scripts for Solr. Within just one day a test data set was imported into Solr and the first tests on text searches were implemented giving confidence to the entire team that the target was indeed possible. This meant that there was also zero time wasted having to install and configure Solr on each of the developers machines. Within just a few days an entire facet based filtering system was integrated that enabled users to click to reduce the result set along several dimensions without having to manually trigger a page reload. Via the native highlighting capabilities the user gets visual indication of why the given document is relevant. Additionally the results are color coded to give the users a better idea of the relative legal value of the document.

In parallel the admin tool was build up by using the admin generator. With just a few adjustments to the configuration files and the use of several sfFormExtra form widgets and the sfAdminDashPlugin UI theme, the admin tool provided editing capabilities for the bulk of the imported data.

User management and registration, including email verification an password recovery, were essentially provided out of the box by the sfDoctrineGuardPlugin and sfDoctrineApplyPlugin. As a result only a few lines needed to be added to the configuration and some optimization to the provided templates as done. Using the vjCommentPlugin also quickly added commenting to the site. Due to the integrated deployment tools it was easy to quickly deploy new versions of the code to enable the team of researchers to review the state of the development and provide feedback on the UI. All the while Doctrine migrations, which in most cases can be generated automatically by simply comparing the existing models classes with an updated schema definition file, take the pain out of managing data migration.

"The entire team was surprised how much was possible in such a short timeframe, even leaving time for additional polish where we expected to have to make due with an application that would just be a raw tech demo which would have been dependent on the imagination of users rather than showing a concrete version that is already useable for end users." Lukas Smith, resolutionfinder.org

Future developments

Over the next couple of months, the main focus is to improve the quality of the database and in the long run to extend it up to the point when it includes all thematic areas on the UN agenda. In this regard, research is ongoing for IT solutions in order to make the database universal in a more efficient way. Especially data mining tools to automate the parsing of PDF and HTML based UN documents into the database. Further work is also planned to enable different types of searches that focus more on chronological aspects or certain UN organisations or member states. Finally the admin tools will be improved to ease administrative workflows to enable not only content editing, but to also make it feasible to enter new documents and clauses inside the admin tool, rather than employing the excel sheet importer. This will require using the advanced form embedding capabilities provided by the symfony form system to load additional related forms via AJAX.

Software

  • Dedicated virtual host
  • Linux Debian

This case study was provided by a user of the symfony framework and published with the permission of all parties involved. Are you interested in having your case study published on the symfony blog? Feel free to contact our Community Manager Stefan Koopmanschap (stefan.koopmanschap - at - symfony-project - dot - com).

Published in #Case studies