Day 20: Administration and moderation

Previously on symfony

The askeet service should work as expected and without any bad surprises, thanks to our concern about performance before the initial release. But there is a much bigger problem: being an application open to contributions from anyone, it is subject to spam, excesses, or disturbing errors. Every service like askeet needs a way to moderate the publications, and accessing the database by hand is surely a bad solution. Should we add a backend application to askeet?

The advent calendar tutorials are supposed to talk about the development of a web application using agile methods. However, until now, we talked a lot about coding and not that much about application development, and the relations between the requirements of a client and the functionality implemented. The backend need will be a good opportunity to illustrate what comes before coding in agile development.

The expected result: what the client says

Today's job will consist of a few new actions, new templates and new model method, and we already know how to do that. The hardest part is probably to define what is needed, and where to put it. It is both a functional and usability concern, and it's a good thing that developers focus on something else than code every once in a while.

It will be the opportunity to illustrate one of the tasks of the eXtreme Programming (XP) methodology: the writing of stories, and the work that developers have to do to transform stories into functionality. XP is one of the best agile development approaches, and is usually applicable to web 2.0 projects like askeet.

Stories

In XP, a story is a brief description of the way an action of the user triggers a reaction of the application. Stories are written by the client of the website (the one who eventually pays for it - the web is not all about open source). Stories rarely exceed one or two sentences. They are regrouped in themes.

The stories are generally less detailed and more elementary than use cases. If you are familiar with UML, you might find the stories to not be precise enough, but we will see shortly that it can be a great chance.

Stories focus on the result of the action, not the implementation details. Of course, the client may have preferences concerning the interface, and in this case the story has to contain the demands and recommendations about the look and feel of the human computer interaction.

Stories have to be small enough to be evaluated easily by developers in terms of development time. Usually, a team of extreme programmers measure stories in units. The value of a unit is refined throughout the course of a project, and can vary from half a day to a few days.

Now, let's have a look at how the client would define the requirements for the askeet backend.

Story #1: Profile management

Every user can ask to become a moderator. In a user's profile page, a link should be made available to ask for this privilege. A person who asked to be moderator must not be able to ask it again until he/she receives an answer.

The persons entitled to accept or refuse a moderator candidate are the administrators. They must be able to browse the list of candidates, and have a button to grant or refuse the grade of moderator for each one of them. Administrators need to have a link to the candidate's profile to see if their contributions are all right.

Granting moderator rights must be a reversible action: Administrators must be able to browse a list of moderators, and for each, to delete the moderator credential.

Administrators can also grant administrator rights to other users. They have access to the list of administrators.

Story #2: Report of problematic questions or answers

Every user must be able to report a problematic question or answer to a moderator. A simple 'report spam' link at the bottom of every question or answer can be a good solution.

To avoid spam of reports, the report from a user about a specific question or answer can only be counted once. It would be great if the user had a visual feedback about the fact that his/her report was taken into account.

Story #3: Handling of problematic questions or answers

Moderators have two more lists available: the list of problematic questions, and the list of problematic answers. Each list is ordered according to the number of reports, in decreasing order. So the most reported questions will appear on top of the reported question list.

Moderators have the ability to delete a question, to delete an answer, and to reset the number of reports about either one. The deletion of a question causes the deletion of all the answers to this question.

Story #4: handling of problematic tags

Moderators have the ability to delete a tag for a question, whether the tag was given by them or not.

Moderators have access to a list of tags, ordered by inverse popularity, so that they can detect the problematic tags - the ones that don't make sense. By linking to the list of questions tagged with this tag, the list gives the ability to suppress the tags.

Story #4: Handling of problematic users

When a moderator deletes a user's contribution, it increments the number of problematic contributions posted by this user.

Administrators have a list of problematic users ordered by the number of problematic posts erased. Administrators must be able to delete a user and all his/her contributions.

Is that all?

Yes, that's all that the client needs to define about the functionality required for the askeet site management. It doesn't cover all cases as a functional specification would, it is not as accurate as a complete set of use case, and it leaves a lot of open ends that may lead to unwanted results.

But the job of the agile developers, which starts now, is to detect the possible ambiguities and lack of data, and to require the assistance of the client when it turns out that a story must be more precise. In a XP-style development phase, the client is always available to answer the questions of the development team.

So the developers meet up in pairs, and each pair chooses a story to work on. They talk a bit about what the story means, the unit test cases that would validate the functionality. They write the unit tests. Then, they write the code to pass these tests. When it's done, they release the code that they added in the whole application, and validate the integration by running all the unit tests written before. As it works, they take a cup of coffee, and split up. Then they form a new pair with someone else and focus on a new story.

What if the final result doesn't meet up with the desires of the client? Well, it only represents a few units of work (a few hours or days), so it is easy to forget it and try a new approach. At least, the client now knows what he/she doesn't want, and that's a great step towards determinism. But most of the time, as the developers are given the opportunity to talk directly with the client and read between the lines of he stories written, they get to produce the functionality in an even better way than the client would expect. Plus, it's the developer who knows about the AJAX possibilities and the way a web 2.0 can become successful. So giving them (us) the initiative is a good chance to end up with a great application.

If you are interested in XP and the benefits of agile development, have a look at the eXtreme Programming website or read Extreme Programming Explained: Embrace Change by Kent Beck.

Backend vs. enhanced frontend

The feedback of the developer on the client's requirements is often crucial for the quality of the application. Let us see what the developer, who knows how the application is built and how powerful symfony is, could say to the client.

The idea to add a backend application to askeet is not that good, and for several reasons.

First, a moderator using the backend might need a lot of the features already available in the frontend (including the list of latest questions, the login module, etc.). So there is a risk that the backend application repeats part of the frontend. As we don't like to repeat ourselves, that would imply a lot of cross-application refactoring, and this is much too long for the hour dedicated to it. Second, a new application would probably mean a new design to the site, with a custom layout and stylesheets. This is what takes the most time in application development. Last, to create the backend application in one hour, we would probably have to use the CRUD generator a lot, resulting in many unnecessary actions and long-to-adapt templates.

In the near future (it is planned for version 0.6), symfony will provide a full-featured back-office generator. All the functionality commonly needed to manage a website activity will be handled easily, almost without a line of code. This brilliant addition would have changed our mind about the way to build the askeet backend, but considering the current state of the framework, the best solution for the management features is to add them to the frontend application.

The base of the askeet frontend is a set of lists, and detail pages for questions and users in which certain actions are available. This is exactly the skeleton needed to build up site management functionality on.

Although it would be helpful to show how a project can contain more than one application, the client, impressed by this demonstration, goes for an integration of the site management features in the frontend application.

The functionality: what the developers understand

After the developers meet up and talk with the client about the stories, they deduce the modifications to be done to the askeet application. The developer transforms stories to tasks. Tasks are usually smaller than stories, because implementing a story takes more than a day or two, while a task can normally be developed within one or two time units.

1. The model has to be modified to allow efficient requests:

   • new table ReportQuestion to be created, with question_id, user_id and created_at columns
   • new table ReportAnswer to be created, with question_id, user_id and created_at columns
   • new column reports to be added to the Question and Answer tables
   • new columns is_administrator, is_moderator and deletions to be added to the User table

2. On every page, the sidebar has to provide access to new lists according to the credentials of the user:

   • All users: popular questions, latest questions, latest answers
   • Moderators: reported questions, reported answers, unpopular tags
   • Administrators: administrators, moderators, moderator candidates, problematic users

3. The question detail page (question/show) has to provide access to new actions according to the credentials of the user:

   • Subscriber: report question, report answer
   • Moderators: delete question and answers, delete answer, reset reports for question, reset reports for answer, delete tag

   The question detail has to give additional information according to the credentials of the user:

   • Subscriber: if the question has already been reported by the subscriber
   • Moderator: the number of reports about the question and answers

4. The user profile page (user/show) has to provide access to new actions according to the credentials of the user:

   • Subscriber on his own page: come forward as a moderator candidate
   • Administrators: delete the user and all his/her contributions, grant moderator credentials, refuse moderator credentials, delete moderator credentials, grant administrator credentials

   The user profile page has to give additional information according to the credentials of the user:

   • All users: credentials of the user, credentials being applied for
   • Administrators: number of erased posts

5. New lists with restricted access must be created:

   • Restricted to moderators:
     • question/reports: list of reported questions, in decreasing order of number of reports; For each, link to the question detail.
     • answer/reports: list of reported answers, in decreasing order of number of reports; For each, link to the question detail.
     • tag/unpopular: list of tags, in increasing popularity order; For each, link to the list of questions tagged with this tag
   • Restricted to administrators:
     • user/administrators: list of administrators, by alphabetical order; For each, link to the user profile
     • user/moderators: list of moderators, by alphabetical order; For each, link to the user profile
     • user/candidates: list of moderator candidates, by alphabetical order; For each, link to the user profile
     • user/problematic: list of problematic users, in decreasing order of deleted contributions; For each, link to the user profile

6. Two new credentials must be created: Administrator and Moderator.

7. At least one administrator has to be setup by hand in the database for the application to work.

Implementation

Once the task list is written, the way to implement the backend features on askeet with symfony is just a matter of work. Applying the XP methodology on this task list, including the writing of unit tests, would take at least a good day of work. For the needs of the advent calendar tutorial, we will do it a little faster, and we will just focus here on the new techniques not described previously, or on the ones that should help you to review classical symfony techniques.

New tables

For the question and answer reports, we add two new tables to the askeet database:

<table name="ask_report_question" phpName="ReportQuestion">
  <column name="question_id" type="integer" primaryKey="true" />
  <foreign-key foreignTable="ask_question">
    <reference local="question_id" foreign="id" />
  </foreign-key>
  <column name="user_id" type="integer" primaryKey="true" />
  <foreign-key foreignTable="ask_user">
    <reference local="user_id" foreign="id" />
  </foreign-key>
  <column name="created_at" type="timestamp" />
</table>
 
<table name="ask_report_answer" phpName="ReportAnswer">
  <column name="answer_id" type="integer" primaryKey="true" />
  <foreign-key foreignTable="ask_answer">
    <reference local="answer_id" foreign="id" />
  </foreign-key>
  <column name="user_id" type="integer" primaryKey="true" />
  <foreign-key foreignTable="ask_user">
    <reference local="user_id" foreign="id" />
  </foreign-key>
  <column name="created_at" type="timestamp" />
</table>

The combination of the question_id/answer_id and the user id is enough to create a unique primary key, so we don't need to add an auto-increment id for these tables.

We also add a new reports column to the Question and Answer table. In order to synchronize the number of records in the ReportQuestion and the number of reports in the Question table, we override the save() method of the ReportQuestion object to add a transaction, as we did during day 4:

public function save($con = null)
{
  $con = sfContext::getInstance()->getDatabaseConnection('propel');
  try
  {
    $con->begin();
 
    $ret = parent::save();
 
    // update spam_count in answer table
    $answer = $this->getAnswer();
    $answer->setReports($answer->getReports() + 1);
    $answer->save();
 
    $con->commit();
 
    return $ret;
  }
  catch (Exception $e)
  {
    $con->rollback();
    throw $e;
  }
}

Same for the ReportAnswer table.

Cascade deletion

When a question is deleted, all the answers to this questions must also be deleted, as well as all the interests about the question, the tags added to the question and the relevancy ratings about all the answers. We need a mechanism of cascade deletion to take care of all that for us.

During day two, we had the idea of using the InnoDB engine for the askeet database. This facilitates the cascade deletions. But the Propel layer can manage to do the cascade deletions even on a non-InnoDB enabled database, provided that we indicate in the schema that cascade deletion has to be taken care of. This has to be done when declaring a foreign key: add a onDelete="cascade" attribute to the <foreign-key> tag in a table definition. For instance, for the Answer table:

...
<table name="ask_answer" phpName="Answer">
  <column name="id" type="integer" required="true" primaryKey="true" autoIncrement="true" />
  <column name="question_id" type="integer" />
  <foreign-key foreignTable="ask_question" onDelete="cascade">
    <reference local="question_id" foreign="id"/>
  </foreign-key>
  <column name="user_id" type="integer" />
  <foreign-key foreignTable="ask_user">
    <reference local="user_id" foreign="id"/>
  </foreign-key>
  <column name="body" type="longvarchar" />
  <column name="html_body" type="longvarchar" />
  <column name="relevancy_up" type="integer" default="0" />
  <column name="relevancy_down" type="integer" default="0" />
  <column name="reports" type="integer" default="0" />
  <column name="created_at" type="timestamp" />
</table>
...

Once the model is rebuilt, cascade deletion is enabled for the relations bearing the onDelete attribute. When you delete a record in the Question table:

• if the database uses the InnoDB engine, the related answers will be deleted automatically by the database itself
• else, the Propel layer will automatically get the related answers, delete them, then delete the question.

All relations may not involve a cascade deletion. Deleting a user, for instance, should delete his/her interests and ratings for answer relevancies, but not his/her contributions (questions and answers). These contributions should be associated to the anonymous user after deletion.

So the onDelete attribute has to be set to cascade for the following relations:

• Answer/QuestionId
• Interest/QuestionId
• Relevancy/QuestionId
• QuestionTag/QuestionId
• ReportQuestion/QuestionId
• ReportAnswer/AnswerId

Add links in the sidebar for users with credentials

We create a new moderator module to handle all the moderator actions, and an administrator one to handle the administration actions.

During day seven, we used the component slot technique to store the code of the sidebar in the sidebar module. The links to the new lists will appear there, but they need to be conditionned to a credential. This is simply done by using the $sf_user->hasCredential() method, as seen during day six:

// in askeet/apps/frontend/modules/sidebar/templates/_default.php and _question.php:
...
<?php include_partial('sidebar/moderation') ?>
 
<?php include_partial('sidebar/administration') ?>
 
// in askeet/apps/frontend/modules/sidebar/templates/_moderation.php:
<?php if ($sf_user->hasCredential('moderator')): ?>
  <h2>moderation</h2>
 
  <ul>
    <li><?php echo link_to('reported questions', 'moderator/reportedQuestions') ?> (<?php echo QuestionPeer::getReportCount() ?>)</li>
    <li><?php echo link_to('reported answers', 'moderator/reportedAnswers') ?> (<?php echo AnswerPeer::getReportCount() ?>)</li>
    <li><?php echo link_to('unpopular tags', 'moderator/unpopularTags') ?></li>
  </ul>
<?php endif ?>
 
// in askeet/apps/frontend/modules/sidebar/templates/_administration.php:
...
<?php if ($sf_user->hasCredential('administrator')): ?>
  <h2>administration</h2>
 
  <ul>
    <li><?php echo link_to('moderator candidates', 'administrator/moderatorCandidates') ?> (<?php echo UserPeer::getModeratorCandidatesCount() ?>)</li>
    <li><?php echo link_to('moderator list', 'administrator/moderators') ?></li>
    <li><?php echo link_to('administrator list', 'administrator/administrators') ?></li>
    <li><?php echo link_to('problematic users', 'administrator/problematicUsers') ?> (<?php echo UserPeer::getProblematicUsersCount() ?>)</li>
  </ul>
<?php endif ?>    

The class methods QuestionPeer::getReportCount(), AnswerPeer::getReportCount(), UserPeer::getModeratorCandidatesCount() and UserPeer::getProblematicUsersCount() are to be added to the model. They are all based on the same principle:

public static function getReportCount()
{
  $c = new Criteria();
  $c->add(self::REPORTS, 0, Criteria::GREATER_THAN);
  $c = self::addPermanentTagToCriteria($c);
 
  return self::doCount($c);
}

AJAX report

We will provide a '[report to moderator]' link to report a question in all the places a question is displayed (in the question lists, in a question detail page). It would be nice if this link was an AJAX one, as in the day eight tutorial. So we add a new helper to the QuestionHelper.php file in the askeet/apps/frontend/lib/helper/ directory:

function link_to_report_question($question, $user)
{
  use_helper('Javascript');
 
  $text = '[report to moderator]';
  if ($user->isAuthenticated())
  {
    $has_already_reported_question = ReportQuestionPeer::retrieveByPk($question->getId(), $user->getSubscriberId());
    if ($has_already_reported_question)
    {
      // already reported for this user
      return '[reported]';
    }
    else
    {
      return link_to_remote($text, array(
        'url'      => '@user_report_question?id='.$question->getId(),
        'update'   => array('success' => 'report_question_'.$question->getId()),
        'loading'  => "Element.show('indicator')",
        'complete' => "Element.hide('indicator');".visual_effect('highlight', 'report_question_'.$question->getId()),
      ));
    }
  }
  else
  {
    return link_to_login($text);
  }
}

Now, the templates where the link has to appear (question/templates/showSuccess.php, question/templates/_list.php) can use this helper:

<div class="options" id="report_question_<?php echo $question->getId() ?>">
  <?php echo link_to_report_question($question, $sf_user) ?>
</div>

The @user_report_question rule has to be written in the routing.yml as leading to a user/reportQuestion action:

public function executeReportQuestion()
{
  $this->question = QuestionPeer::retrieveByPk($this->getRequestParameter('id'));
  $this->forward404Unless($this->question);
 
  $spam = new ReportQuestion();
  $spam->setQuestionId($this->question->getId());
  $spam->setUserId($this->getUser()->getSubscriberId());
  $spam->save();
}

And the result of this action, the user/templates/reportQuestionSuccess.php template, is simply:

<?php use_helper('Question') ?>
<?php echo link_to_report_question($question, $sf_user) ?>

The same goes for the reported answers.

New action links for users with credentials

In the question_body div tag of the askeet/apps/frontend/modules/question/templates/showSuccess.php, we add the question management actions for moderators only, so to be compatible with the AJAX report, we put them in a fragment:

...
<div class="options" id="report_question_<?php echo $question->getId() ?>">
  <?php echo link_to_report_question($question, $sf_user) ?>
  <?php include_partial('moderator/question_options', array('question' => $question)) ?>
</div>

The askeet/apps/frontend/modules/moderator/templates/_question_options.php fragment contains:

<?php if ($sf_user->hasCredential('moderator')): ?>
  <?php if ($question->getReports()): ?>
    &nbsp;[<strong><?php echo $question->getReports() ?></strong> reports]
    &nbsp;<?php echo link_to('[reset reports]', 'moderator/resetQuestionReports?stripped_title='.$question->getStrippedTitle()) ?>
  <?php endif ?>
  &nbsp;<?php echo link_to('[delete question]', 'moderator/deleteQuestion?stripped_title='.$question->getStrippedTitle()) ?>
<?php endif ?>
...

The same options are added in the askeet/apps/frontend/modules/answer/templates/_answer.php, with a link to a moderator/templates/_answer_options.php fragment.

The same kind of adaptation goes for the administrator action links in the user profile page.

 [php]
 <?php echo link_to('[delete answer]', 'moderator/deleteAnswer?id='.$answer->getId(), 'post=true') ?>

Access restriction

When a user with specific rights logs in, his sfUser object must be given the appropriate credential. This is done in the signIn method of the myUser class in askeet/apps/frontend/lib/myUser.class.php, that we created during day six:

public function signIn($user)
{
  $this->setAttribute('subscriber_id', $user->getId(), 'subscriber');
  $this->setAuthenticated(true);
 
  $this->addCredential('subscriber');
 
  if ($user->getIsModerator())
  {
    $this->addCredential('moderator');
  }
 
  if ($user->getIsAdministrator())
  {
    $this->addCredential('administrator');
  }
 
  $this->setAttribute('nickname', $user->getNickname(), 'subscriber');
}

Of course, all the moderator actions have to be restricted to moderators with appropriate settings in the askeet/apps/frontend/modules/moderator/config/security.yml:

all:
  is_secure:   on
  credentials: moderator

The same kind of restriction is to be applied for administrator actions.

New moderator and administrator actions

There is nothing new in the actions to be added to the moderator and administrator actions. We will just give the list here so that you know about them:

// administrator actions
executeProblematicUsers()     ->  usersSuccess.php
executeModerators()           ->  usersSuccess.php
executeAdministrators()       ->  usersSuccess.php
executeModeratorCandidates()  ->  usersSuccess.php

executePromoteModerator()     ->  request referrer
executeRemoveModerator()      ->  request referrer
executePromoteAdministrator() ->  request referrer
executeRemoveAdministrator()  ->  request referrer

// moderator actions
executeUnpopularTags()        ->  unpopularTagsSuccess.php
executeReportedQuestions()    ->  reportedQuestions.php
executeReportedAnswers()      ->  reportedAnswers.php

executeDeleteTag()            ->  request referrer
executeDeleteQuestion()       ->  @homepage
executeDeleteAnswer()         ->  request referrer

moderatorsSuccess:
  template: users

administratorsSuccess:
  template: users

moderatorCandidatesSuccess:
  template: users

problematicUsersSuccess:
  template: users

Log deletions

When a moderator deletes a question, we want to keep a trace of the deletion in a log file, in a warning message. To allow the logging of warning messages in the production environment, we need to modify the logging.yml configuration file:

prod:
  level: warning    

Then, in all the delete actions, add the code to log the deletion, as in this moderator/deleteQuestion action:

public function executeDeleteQuestion()
{
  $question = QuestionPeer::getQuestionFromTitle($this->getRequestParameter('stripped_title'));
  $this->forward404Unless($question);
 
  $con = sfContext::getInstance()->getDatabaseConnection('propel');
  try
  {
    $con->begin();
 
    $user = $question->getUser();
    $user->setDeletions($user->getDeletions() + 1);
    $user->save();
 
    $question->delete();
 
    $con->commit();
 
    // log the deletion
    $log = 'moderator "%s" deleted question "%s"';
    $log = sprintf($log, $this->getUser()->getNickname(), $question->getTitle());
    $this->getContext()->getLogger()->warning($log);
  }
  catch (PropelException $e)
  {
    $con->rollback();
    throw $e;
  }
 
  $this->redirect('@homepage');
}

If you want to know more about logging, you can have a look at the debug chapter of the symfony book.

We changed the try/catch statement to react only to PropelExceptions instead of all Exceptions. This is because we don't want the transaction to fail only because there is a problem in the logging of the deletion.

See you Tomorrow

As we took some time to think about the way to implement the backend features, and because there are quite a lot of them, today's tutorial probably lasted two hours rather than only one. But there is not many new things here, so the implementation should be a review of symfony techniques. You can have a good view of the total list of changes by browsing to the askeet timeline.

Tomorrow is the day of the mysterious feature. Numerous suggestions were sent to the forum, or even in the beta askeet site itself. You will see which one we decided to implement, and how symfony can be of great help to it.

Feel free to go to the forum if you have any problem with today's source, which, by the way, can still be downloaded from the SVN repository or browsed in the trac.