Symfony world

excellent modelling tool: cacoo

2012-02-13T00:17:00.003+01:00

Scene from the epic "Braveheart" by Mel Gibson (1995)

This time I want to present an excellent visual tool I've started using recently to improve my projects' documentation. It's the Cacoo online diagram editor, which is a professional flash tool.

The basic concept is very easy: you create graphic diagrams. You may structure them in folders and sheets (don't worry, each country should have its own translation already - I'm using Polish version):

There's a history of all changes done on each diagram:

Creating a diagram usually starts with creating your own, custom objects which represent something. You may use standard, predefined objects or import any image from the internet to combine new objects:

You may resize or change the angle of an object:

There are lots of predefined stencils to USE, including entire UML:

The interface is very intuitive - it took me only an hour to learn how to make the most out of cacoo.

There are dozens of easy tools you can use to make really good quality diagrams:

I won't try to list al those tools, I hope the screenshots I made speak for themselves (especially I like the drag n'drop stuff and moving objects across the diagram).

And the most comfortable and user-friendy thing is easy diagram update. If your IT project has changed and your documentatin is not up to date anymore - just log in, make a tiny change in the diagram and dump it again, replace the old diagram in the docs and you're done!

predefined objects to use

Just get a predefined stencil - just like a nice looking professor below:

This stencil is a group of smaller objects. You can choose some objects to be grouped to make moving easier. You can even resize a whole group of objects, including font size changes! And groups can consist of groups...

Anyway, a group member can still be edited separately:

There are really lots and lots of possibilities there. You can do whatever you can think of.

Finally, you can export your work to few file formats (I use ksnapshot - PNG file is enough for me).

try the demo version

The free plan gives you the possibility to hold 25 diagrams and it's enough to create useful diagrams, documenting your projects. I did - and I'm satisfied. Just take a look:

they didn't found me - I found them :)

Just to make it clear - I was not contacted by the Cacoo team and I was not asked if I could write a nice review (things like this happen - some time ago I was asked to review a pro-symfony IDE) ;-). So there was no bribe. One day I was just looking for a free UML-designing software (desktop app or online), because I was finding StarUML less and less satisfying. I found Cacoo and I didn't need to search any longer.

Try it.

svn blame: who made the bug?

2012-02-05T18:48:00.000+01:00

Scene from "12 Angry Men" by Sidney Lumet (1957)

In bigger projects it happens quite often that an unknown team member has made a modification that caused a lot of trouble. And no one wants to confess... It is quite uncomfortable to analyse distinct revisions and changes of a file to find who caused the problem, as it is time consuming. This is where svn blame is the perfect tool - you execute it upon a chosen target to display revision number, team member and content line by line. You can easily find who is the author of a bug and when he committed it. It's intereting, that despite being very useful, this tool is not very known.

blame me if you can find me quickly!

The usage is very simple:


tducin@workshop:/var/www/admin/config/doctrine$ svn blame schema.yml

and you have the entire file analysed:


   367     dennis connection: master
   362     tducin
   362     tducin options:
   362     tducin   charset: utf8
   362     tducin   collate: utf8_general_ci
   362     tducin
  3755     george Status:
  3755     george   comment: "Statusy"
   491     dennis   actAs:
  1980     dennis     SoftDelete: ~
  1793     harold     Timestampable: ~
  1793     harold     Signable: ~
   492     tducin   columns:
   492     tducin     name:
  3755     george       type: string(64)
   492     tducin       notnull: true
  2628     tducin       comment: "nazwa"
  3755     george     notify_mail:
  3755     george       type: boolean
  3755     george       notnull: true
  3755     george       default: false
  3755     george       comment: "klient ma być powiadamiany mailowo"
  1541     dennis     stock_mode:
  1541     dennis       type: enum
  1541     dennis       values: [ reserved, sent, cancelled ]
  1541     dennis       default: reserved
  2628     dennis       comment: "tryb magazynowy: reserved, sent, cancelled"
  3755     george     force_reminder:
   892     tducin       type: boolean
   878     tducin       notnull: true
  3755     george       comment: "Wymuszenie dodania wpisu w kalendarzu"
  3755     george     block_update:
  3755     george       type: boolean
  3755     george       notnull: true
  3755     george       comment: "Blokada zmiany statusu"
  4078     harold   indexes:
  4078     harold     softdelete_idx:
  4078     harold       fields: [ deleted_at ]

symfony range filter

2012-01-29T17:06:00.000+01:00

In symfony/doctrine, only the Timestampable behavior provides ranged filters (a combination of from and to values that are queried just like MySQL BETWEEN statement). But how about ranged values for other data types?

I thought it'd be both cool and very useful, so I set off to search the web, but found no solution. Hence, I've made one myself. We will create three new classes and modify two more files to make it work in the basic version.

new widget/validator classes

The first class to be added is the form widget, lib/widget/sfWidgetFormInputRange.class.php:

class sfWidgetFormInputRange extends sfWidgetForm
{
  protected function configure($options = array(), $attributes = array())
  {
    $this->addRequiredOption('from_value');
    $this->addRequiredOption('to_value');
 
    $this->addOption('template', 'from %from_value% to %to_value%');
  }
 
  /**
   * Renders the widget.
   *
   * @param  string $name        The element name
   * @param  string $value       The value displayed in this widget
   * @param  array  $attributes  An array of HTML attributes to be merged with the default HTML attributes
   * @param  array  $errors      An array of errors for the field
   *
   * @return string An HTML tag string
   *
   * @see sfWidgetForm
   */
  public function render($name, $value = null, $attributes = array(), $errors = array())
  {
    $value = array_merge(array('from' => '', 'to' => ''), is_array($value) ? $value : array());
 
    return strtr($this->translate($this->getOption('template')), array(
      '%from_value%' => $this->getOption('from_value')->render($name.'[from]', $value['from']),
      '%to_value%'   => $this->getOption('to_value')->render($name.'[to]', $value['to']),
    ));
  }
 
  /**
   * Gets the stylesheet paths associated with the widget.
   *
   * @return array An array of stylesheet paths
   */
  public function getStylesheets()
  {
    return array_unique(array_merge(
      $this->getOption('from_value')->getStylesheets(),
      $this->getOption('to_value')->getStylesheets()));
  }
 
  /**
   * Gets the JavaScript paths associated with the widget.
   *
   * @return array An array of JavaScript paths
   */
  public function getJavaScripts()
  {
    return array_unique(array_merge(
      $this->getOption('from_value')->getJavaScripts(),
      $this->getOption('to_value')->getJavaScripts()));
  }
}

The second one is the form filter widget, lib/widget/sfWidgetFormFilterInputRange.class.php:

class sfWidgetFormFilterInputRange extends sfWidgetFormInputRange
{
  /**
   * Configures the current widget.
   *
   * Available options:
   *
   *  * with_empty:      Whether to add the empty checkbox (true by default)
   *  * empty_label:     The label to use when using an empty checkbox
   *  * template:        The template used for from value and to value
   *                     Available placeholders: %from_value%, %to_value%
   *  * filter_template: The template to use to render the widget
   *                     Available placeholders: %value_range%, %empty_checkbox%, %empty_label%
   *
   * @param array $options     An array of options
   * @param array $attributes  An array of default HTML attributes
   *
   * @see sfWidgetForm
   */
  protected function configure($options = array(), $attributes = array())
  {
    parent::configure($options, $attributes);
 
    $this->addOption('with_empty', true);
    $this->addOption('empty_label', 'is empty');
    $this->addOption('template', 'from %from_value%<br />to %to_value%');
    $this->addOption('filter_template', '%value_range%<br />%empty_checkbox% %empty_label%');
  }
 
  /**
   * Renders the widget.
   *
   * @param  string $name        The element name
   * @param  string $value       The value displayed in this widget
   * @param  array  $attributes  An array of HTML attributes to be merged with the default HTML attributes
   * @param  array  $errors      An array of errors for the field
   *
   * @return string An HTML tag string
   *
   * @see sfWidgetForm
   */
  public function render($name, $value = null, $attributes = array(), $errors = array())
  {
    $values = array_merge(array('is_empty' => ''), is_array($value) ? $value : array());
 
    return strtr($this->getOption('filter_template'), array(
      '%value_range%'     => parent::render($name, $value, $attributes, $errors),
      '%empty_checkbox%' => $this->getOption('with_empty') ? $this->renderTag('input', array('type' => 'checkbox', 'name' => $name.'[is_empty]', 'checked' => $values['is_empty'] ? 'checked' : '')) : '',
      '%empty_label%'    => $this->getOption('with_empty') ? $this->renderContentTag('label', $this->translate($this->getOption('empty_label')), array('for' => $this->generateId($name.'[is_empty]'))) : '',
    ));
  }
}

Finally, the last class is the widget validator, lib/validator/sfValidatorInputRange.class.php:

class sfValidatorInputRange extends sfValidatorBase
{
  /**
   * Configures the current validator.
   *
   * Available options:
   *
   *  * from_value:   The from value validator (required)
   *  * to_value:     The to value validator (required)
   *  * from_field:  The name of the "from" value field (optional, default: from)
   *  * to_field:    The name of the "to" value field (optional, default: to)
   *
   * @param array $options    An array of options
   * @param array $messages   An array of error messages
   *
   * @see sfValidatorBase
   */
  protected function configure($options = array(), $messages = array())
  {
    $this->setMessage('invalid', 'The begin value must be before the end value.');
 
    $this->addRequiredOption('from_value');
    $this->addRequiredOption('to_value');
    $this->addOption('from_field', 'from');
    $this->addOption('to_field', 'to');
  }
 
  /**
   * @see sfValidatorBase
   */
  protected function doClean($value)
  {
    $fromField = $this->getOption('from_field');
    $toField   = $this->getOption('to_field');
 
    $value[$fromField] = $this->getOption('from_value')->clean(isset($value[$fromField]) ? array('text' => $value[$fromField]) : null);
    $value[$toField]   = $this->getOption('to_value')->clean(isset($value[$toField]) ? array('text' => $value[$toField]) : null);
 
    if ($value[$fromField]['text'] && $value[$toField]['text'])
    {
      $v = new sfValidatorSchemaCompare($fromField, sfValidatorSchemaCompare::LESS_THAN_EQUAL, $toField, array('throw_global_error' => true), array('invalid' => ('invalid')));
      $v->clean($value);
    }
 
    $value[$fromField] = $value[$fromField]['text'];
    $value[$toField] = $value[$toField]['text'];
 
    return $value;
  }
}

accustoming solution to a example project

For the purposes of this article, let's say we have an Offer model. We will add the offer.recommendations integer value filter in the offer admin module. The lib/filter/doctrine/OfferFormFilter.class.php shall be improved as below:

class OfferFormFilter extends BaseOfferFormFilter
{
  public function configure()
  {
    parent::configure();
 
    $this->setWidget('recommendations_range', new sfWidgetFormFilterInputRange(array(
      'from_value' => new sfWidgetFormInput(),
      'to_value' => new sfWidgetFormInput(),
      'with_empty' => false
    )));
 
    $this->setValidator('recommendations_range', new sfValidatorInputRange(array(
      'required' => false,
      'from_value' => new sfValidatorSchemaFilter('text', new sfValidatorNumber(array('required' => false))),
      'to_value' => new sfValidatorSchemaFilter('text', new sfValidatorNumber(array('required' => false)))
    )));
  }
 
  public function addRecommendationsRangeColumnQuery($query, $field, $value)
  {
    $rootAlias = $query->getRootAlias();
    if (isset($value['from']) && $value['from'])
      $query->andWhere($rootAlias.".recommendations >= ?", $value['from']);
    if (isset($value['to']) && $value['to'])
      $query->andWhere($rootAlias.".recommendations <= ?", $value['to']);
  }
}

Finally, add the filter inside the generator.yml file of the offer module:

  config:
    fields:
      recommendations_range:
        label: Liczba poleceń        
    filter:
      display: [ ..., recommendations_range]

result

Now you can use ranged filters anywhere in your project. Moreover, you can modify this mechanism to accustom it to your project needs.

symfony forward through LAN

2012-01-23T10:28:00.000+01:00

Scene from "Disney's Adventures of the Gummi Bears" by Jymn Magon & Art Vitello (introduced in 1985)

Ever thought of creating an action that would run different code depending on whether the client machine is inside or outside the LAN? It's really easy yet useful. I used it few times in one of my backend apps so far. For people outside the LAN, both links are hidden (use comparison below) and the redirected links are inaccessible. Thanks to it, company employees working from outside the office may not see inaccessible links.Below is the auxiliary method:

  /**
   * Checks if the client computer is inside the LAN with a
   * specific address. If so, redirects to a local address.
   *
   * @param String $ip - IP of the webpage to be forwarded to
   * @param String $port - port of the webpage to be forwarded to
   */
  protected function forwardThroughLan($ip, $port)
  {
    if ($_SERVER['REMOTE_ADDR'] == sfConfig::get('path_server_ip'))
      $this->redirect('http://'.$ip.':'.$port);
    else
      $this->setTemplate ('forwardThroughLanError');
  }

and here is direct usage:

  /**
   * Executes forward TRAC through LAN action. Forwards to TRAC webpage
   * inside LAN in the office building.
   *
   * @param sfWebRequest $request
   */
  public function executeForwardTracThroughLan(sfWebRequest $request)
  {
    $this->forwardThroughLan('192.168.1.99', '8080');
  }

SVN repository auto update

2012-01-16T19:08:00.000+01:00

Scene from "Brazil" by Terry Gilliam (1985)

I found an article (in Polish) describing a subversion hook that will automatically update a server working copy. This seems perfect - after commiting a new revision, all changes are available online (entire development team can test the most up to date version, your boss may see the newest results, etc).

Anyway, I don't see any reason to use C here, so I've made my own version updating the repository using shell command:

svn up /home/website/domains/dev.website.com/website
  --no-auth-cache --username user --password pass

The --no-auth-cache option is important here - authorization data is not going to be remembered here. Other two options define the authorization data.

troubleshooting

You may have problems with file permissions, if the user which holds the repository is different than the user which hold the working copy (this was in my case). The problem I got was permission denied for .../.svn/lock file, which is a temporary file, existing only when the revision is being checked/committed. Oops... There is no one good solution for this - it depends on the server configuration. It can be adding one user to the group of another, changing the owner of the entire working copy files (no matter in which /home subdirectory it is, as long as web server can handle requests, etc).

force to log message SVN python

2012-01-15T23:40:00.000+01:00

Scene from "It Happened One Night" by Frank Capra (1934)

Another blog post about Subversion. Few days ago someone asked me if I know how to block commiting a subversion revision that has no log message defined. I didn't, but I thought it's a good idea, because it could help to track bugs, changes, etc. There's no such configuration, but it can be achieved using SVN hooks.

tried to do this in PHP...

There are many resources in the web giving examples to do this using a windows batch file - but I'm a linux user. At first, I wanted to do this in PHP. Although my final solution is not PHP, maybe someone will find following code snippets useful.

So I started searching through many pages on the web to find PHP shell scripts that could block committing a revision with no log message. The first step is, of course, to create an executable pre-commit file in the hooks directory of the repository. Then make the shell use the PHP parser to execute our script. Next, define path to svnlook binary and execute it:

#!/usr/local/bin/php -q
<?php
 
$SVNLOOK = '/usr/bin/svnlook';
exec("$SVNLOOK log -t {$argv[2]} {$argv[1]}", $output, $result);

We have the output and the result of the command available. So we may just check if the message (which revision a user is just trying to commit) is empty (I guess there is no other way than to use svnlook). If the log message is empty, the revision shall be stopped - and this happens, when exit status code is different than 0 (of course, a message explaining the cause is appreciated):

if (empty($result))
{
  fwrite(STDERR, "Commit blocked, need to pass the log message.");
  exit(1);
}
else
  exit(0);

This all should work, but 99,9% of all servers will have the exec function disabled by default (in both CLI and web server PHP configuration; along with other system functions):

disable_functions = exec,system,passthru,shell_exec,escapeshellarg,escapeshellcmd,proc_close,proc_open,dl,popen,show_source

Even if you manage to remove those functions from the configuration, you lower the server security, which may risk dangerous script injection. So I forgot about using PHP...

...and finally moved to python

Finally, I managed to succeed using python:

#!/usr/bin/python
 
import sys, os, string
 
SVNLOOK='/usr/bin/svnlook'
 
def main(repos, txn):
  log_cmd = '%s log -t "%s" "%s"' % (SVNLOOK, txn, repos)
  log_msg = os.popen(log_cmd, 'r').readline().rstrip('n')
  if len(log_msg) < 10:
    sys.stderr.write ("\n> Revision comment must be defined\n")
    sys.exit(1)
  else:
    sys.exit(0)
 
if __name__ == '__main__':
  if len(sys.argv) < 3:
    sys.stderr.write("\n> Usage: %s REPOS TXN\n" % (sys.argv[0]))
  else:
    main(sys.argv[1], sys.argv[2])

svnsync tutorial

2011-12-15T23:07:00.000+01:00

Scene from magnificent "One Flew Over the Cuckoo's Nest" By Miloš Forman

I was forced to use this tool when I wanted to move my old project repository into a new server (until then, its SVN repository was hosted on a free SVN hosting). The hosting provider forced me to pay for the svnadmin dump, so I searched for an alternative solution and svnsync did the job.

In comparison to svnadmin dump/load, svnsync is migrating subversion repository with no svnadmin access. The migration can be done when you have a working copy (checkout) of the source repository and a clean (svnadmin create) version of the target repository. The repository synchronization is just reading all revisions one by one and copying them into the target repository, unchanged.

First, take a look at this article. Here you can find that a hook needs to be created in the target repository: hooks/pre-revprop-change (removing .tmpl and granting executable permissions for the script). What is not mentioned in the article above is that you should probably put

#!/bin/sh 
exit 0;

as the hooks/pre-revprop-change script content, to avoid the error, as written in another article.

Now, initialize the synchronization (example call below):

svnsync init [target repository] [source repository]
svnsync init file:///home/user/svn/test-repo svn://free-hosting/that-forced-me-to-pay/for-the-svn-dump

and copy all revisions, while being in the working copy of the source repository (example call below again):

svnsync --non-interactive sync [target repository]
svnsync --non-interactive sync file:///home/user/svn/test-repo

Be patient and wait until it's finished. Afterwards, remove the useless hooks/pre-revprop-change script - and you're done!

extending doctrine admin module: filtered sum

2011-11-08T20:58:00.000+01:00

Scene from "Breakfast at Tiffany's" by Blake Edwards (1961)

Another post for symfony beginners. I'll show how to display a sum (or any other function) of all elements filtered in an admin module. All elements - meaning the ones displayed on the current page (list pagination) and all the rest which is not currently visible. This is going to be really easy.

action

Let's start with calculations. We need to get the sum of all filtered elements and pass it to the View. As this article is not about MVC design pattern, I'll just override executeIndex action and put the calculations code inside (to make it as easy as possible, though calculations should be done in model, not controller).

class xxxActions extends autoXxxActions
{
  public function executeIndex(sfWebRequest $request)
  {
    parent::executeIndex($request);
    $query = $this->buildQuery()->copy();
    $root_alias = $query->getRootAlias();
    $total_data = $query
      ->limit(0)
      ->select("SUM({$root_alias}.cash_total) AS sum")
      ->fetchArray();
    $this->total_count = $total_data[0]['sum'];
  }
}

As you can see, we've got a Xxx model which holds the cash_total: decimal column, representing a sum of money. The $this->buildQuery()->clone() part does all the magic - we have the query with all filters set by the user in the interface. We will only tell doctrine to calculate the sum of all filtered elements for us ($total_count variable will be available in the indexSuccess.php template). The ->limit(0) clears the SQL limit clause, of course.

templates

This part is boring, actually. We need to override two more files: fetch them from cache and put them in the module/template directory. These files are: indexSuccess.php in which the line

<?php include_partial('event/list', array('pager' => $pager, 'sort' => $sort, 'helper' => $helper)) ?>

should be replaced with:

<?php include_partial('event/list', array('pager' => $pager, 'sort' => $sort, 'helper' => $helper, 'total_count' => $total_count)) ?>

and the _list.php partial which should have few lines added:

<tfoot>
  <tr><!-- added code starts here -->
    <th colspan="6">
      w sumie: <?php echo Tools::priceFormat($total_count, true) ?>
    </th>
  </tr><!-- added code ends here -->
  <tr>
    <th colspan="6">
       <?php if ($pager->haveToPaginate()): ?>
         <?php include_partial('event/pagination', array('pager' => $pager)) ?>
       <?php endif; ?>

As you can see, it's trivial, yet useful.

The sum of all filtered elements is visible in the footer of the doctrine admin module list table, as you can see below, but you can put it anywhere you want (as long as it's in the index action templates):

other functions

Of course, you can use other aggregate functions, such as average, minumum or maximum element - just take a look at the MySQL documentation. You may also create your own functions.

symfony basics: form default values

2011-11-06T19:10:00.000+01:00

This is some stuff for symfony beginners, who still want to learn symfony 1.4. You may set default form values for all kind of forms (including doctrine forms). Set one default value at a time:

class XxxForm extends BaseXxxForm
{
  public function configure()
  {
    $this->setDefault ('field', 'value');
  }
}

or set a whole array of them:

public function configure()
{
  $this->setDefaults(array(
    'field_1' => 'value_1',
    'field_2' => 'value_2',
    // ...
    'field_x' => 'value_x'
  ));
}

default values for new objects

Sometimes you want to set the default form values just before the object is created, because it'd be easier for the aplication user to fill in some data. For example, the owner/author of a blog post may be set default to the current logged in user - or the date of an event may be set to now - and so on. This can be achieved with the isNew method of the doctrine form class (lib/form/doctrine/XxxForm.class.php):

if ($this->isNew())
{
  $this->setDefault ('created_at'¹, date('Y-m-d 00:00:00'));
  $this->setDefault ('created_by'², sfContext::getInstance()->getUser()³->getId());
}

note ¹: Timestampable Doctrine behavior used in this example,
note ²: Signable Doctrine behavior used in this example,
note ³: check this blog post to avoid using sfContext::getInstance().

moreover

You may implement whatever complex conditions you want your doctrine form to follow. Look at some of examples below:

current time - php time function
language/localization (default country when registering a new user) - use Accept-Language HTTP
default settings set for a registered user - fetch individual user settings from database (doctrine query)
last used item (category/product/etc.) - a user inserts or updates a large amount of data, when he choses a specific item (category/product) it can be saved in its session ($user->set/getAttribute()) - when another record is processed, last used item is used as default (which, again, lowers time needed for the user to work)

A well designed interface includes lots of form default values, so that users don't have to waste their time on picking up the same values over and over again.

symfony dynamic max_per_page

2011-10-31T23:09:00.000+01:00

max_per_page

In this post I'll show a very easy and a really useful thing. It is dynamic max_per_page value of the list pager. Such feature gives you the possibility to change the number of elements displayed in a list just by one click. It can be used both in the frontend and backend (the pager is the same) - I'll use the doctrine generated admin module (and the interface will be placed inside the filters box):

templates

First, let's make it visible. Add the following entry to application config/app.yml file:


all:
  const:
    max_per_page: [ 10, 25, 50 ]

Now we may refer to app_const_max_per_page config value which holds few standard max_per_page values to be used (this can be used in many different admin modules). Let's say we've got an admin module for our custom MyModel model. Now, override the cached _filters.php template: fetch it from cache/admin/dev/modules/autoMyModel/templates and put it in apps/APP/modules/my_model/templates. Take a look at the following part of the code:

          </td>
        </tr>
      </tfoot>
      <tbody>
        <!-- insert here -->
        <tr>
          <td colspan="2">

and insert few lines of code (replace the comment) to get the following:


          </td>
        </tr>
      </tfoot>
      <tbody>
        <tr>
          <td colspan="2">
            set maximum elements per page:
            <?php foreach (sfConfig::get('app_const_max_per_page') as $value): ?>
              <span class="max_per_page_selector"><?php echo link_to($value, 'my_model/setMaxPerPage?max='.$value) ?></span>
            <?php endforeach; ?>
          </td>
        </tr>
        <tr>
          <td colspan="2">

controller/action

The interface to change max_per_page is ready, so we have to improve the controller now. Let's add an action which stores the number of elements to display per page in user session (symofny has nice set(get)Attributes methods). So here it comes:

  /**
   * Sets my_model list's max per page config value, using user session
   * attribute.
   *
   * @param sfWebRequest $request
   */
  public function executeSetMaxPerPage(sfWebRequest $request)
  {
    $this->getUser()->setAttribute('my_model.max_per_page', $max = $request->getParameter('max'));
    $this->getUser()->setFlash('notice', 'max_per_page has been set to: '.$max);
    $this->redirect('@my_model');
  }

configuration generator

And finally, tell the pager to look for the custom value each time the list is going to be rendered. We need to override the method in configuration generator of the admin module:

class my_modelGeneratorConfiguration extends BaseMy_modelGeneratorConfiguration
{
  /**
   * Returns max_per_page config value for my_model module. If it's not
   * defined manually by the user, default value is returned.
   *
   * @return Integer
   */
  public function getPagerMaxPerPage()
  {
    if ($max = sfContext::getInstance()->getUser()->getAttribute('my_model.max_per_page'))
      return $max;
    else
      return parent::getPagerMaxPerPage();
  }

It's all as easy as it could be. The controller searches the servers for the current user session data and returns either the custom data (if found) or the default value (which is taken from the generator.yml file).

Unfortunately, the sfContext::getInstance() is used here (this causes a lot of problems when the default context problem occurs). After a quick look I didn't find the better way to access the user from the configuration generator (but if you know how to - let me know ;).

(I'm wondering why it's not built in into symfony).

doctrine act as signable plugin - new releases

2011-08-16T07:56:00.000+02:00

After few months, new versions 1.2.2 and 1.2.3 of sfDoctrineActAsSignablePlugin have been released. In short, the plugin provides a Signable behavior which automatically stores information on who has created or updated a given object.

what's new

A fellow symfony developer, Daniel Möllenbeck, suggested that there are some options missing in the behavior configuration. Until version 1.2.1, the onDelete option was hardcoded as CASCADE, which - as Daniel emphasised - may cause problems when a given user is supposed to be deleted (physically from the database, not softDeleted).

ALTER TABLE customer
ADD CONSTRAINT customer_created_by_sf_guard_user_id
FOREIGN KEY (created_by)
REFERENCES sf_guard_user(id)
ON DELETE CASCADE;

ALTER TABLE customer
ADD CONSTRAINT customer_updated_by_sf_guard_user_id
FOREIGN KEY (updated_by)
REFERENCES sf_guard_user(id)
ON DELETE CASCADE;

Let's imagine user enters customers over and over, then leaves the company. Now the admin deletes the user - probably the hard way (directly on the database, maybe the user is auto-created from somewhere...) --> The constraint will trigger the deletion of all customers created by that user. I have some doubt anybody would be happy about that.

Having the constraint like that does the trick:

a) allow NULL values in created_by/updated_by columns:

ALTER TABLE customer
ADD CONSTRAINT customer_created_by_sf_guard_user_id
FOREIGN KEY (created_by)
REFERENCES sf_guard_user(id)
ON DELETE SET NULL;

ALTER TABLE customer
ADD CONSTRAINT customer_updated_by_sf_guard_user_id
FOREIGN KEY (updated_by)
REFERENCES sf_guard_user(id)
ON DELETE SET NULL;

b) forbid the deletion of the user:

ALTER TABLE customer
ADD CONSTRAINT customer_created_by_sf_guard_user_id
FOREIGN KEY (created_by)
REFERENCES sf_guard_user(id)
ON DELETE RESTRICT;

ALTER TABLE customer
ADD CONSTRAINT customer_updated_by_sf_guard_user_id
FOREIGN KEY (updated_by)
REFERENCES sf_guard_user(id)
ON DELETE RESTRICT;

c) do nothing

moreover

Another fellow symfony developer, Christoph Berg, helped me to trace the bug with fixtures on created_by/updated_by values. Now, the bug is fixed and all fixture data works perfectly.

the community

Taking the opportunity, I'd like to thank Daniel, Christoph and all other symfony developers who share their opinions on my work - they help to find bugs and suggest some good ideas on how to improve the code. Thanks to you, the plugin gets better and better all the time. Thanks, guys!

Please, feel free to share your opinion and comment on the plugin!

symfony custom configuration files

2011-08-15T18:11:00.000+02:00

dynamic and cross-application configuration

If your project gets very big and it has several applications, you may want to create cross-application configuration - store it in one place and let all applications use it (no matter how big it is). Or simply if your configuration is becoming really big and you want to arrange it somehow, custom config files is exactly what you are looking for!

how to do that

That's only three easy steps. First, let's create a custom configuration YAML file (config/something.yml, in the project main directory, not the app config directory) and put some data there:

prod:

  test:  ok



dev:

  test: ko



all:

  foo:  bar

  john: doe

The second step is to create/update a config handler YAML file (config/config_handlers.yml) with following example content:

config/something.yml:

  class:    sfDefineEnvironmentConfigHandler

  param:

    prefix: something_

The last step is to register the configuration file in all applications you want (after this step, the configuration will be available for an application). Use the application configuration class (e.g. apps/frontend/config/frontendConfiguration.class.php):

public function configure()

{

  ....

  require_once($this->getConfigCache()->checkConfig('config/something.yml'));

}

Now all config options defined in the new file are available in the application with the _something prefix. The best way to check if the new file is available for the application is to run it in the dev mode in the browser and check the web debug toolbar, config icon, settings section. All configuration available for the application is displayed there. If you followed all three steps above, new file configuration should be available.

sfAdminDashPlugin trick

This plugin provides an easy-to use menu divided into sections, which is configurable in YAML files. If you want to have different menus for different applications, you just have to put all configuration in the application config (by default, it's stored in the plugin config). One way is to put all menu config in the app.yml file, but, as mentioned before, it can make the app.yml file grow to an enormous size, so a better solution is to create a menu.yml file, which is loaded by the config handler. Everything is done the same way as before:

create apps/APP/config/menu.yml file and define sfAdminDash menu there
create or update (if exists) the config handler file (apps/APP/config/config_handlers.yml)
finally, update the APP configuration class (apps/APP/config/APPConfiguration.class.php) by registering new config file

I hope some of you will find this flexible configuration useful :).

Doctrine Event Listeners vs symfony fixtures

2011-06-26T22:54:00.000+02:00

introduction

This is just an easy tip on how to use symfony fixtures along with doctrine event listeners in a symfony project.

event listeners

Doctrine Event Listeners are the methods you define to trigger an activity always after/before a Doctrine Record is created, updated, deleted, etc. These can be: postInsert, preInsert, postDelete, preDelete and so on. They can be very useful to keep the code clean and logical. A good example of usage is uploading an image and creating many different files with different sizes, representing the same image (custom thumbnails). This could be done with overriding ImageForm upload mechanisms, but postInsert seems more clean and easier to maintain if uploading is done using more than one symfony form class.

fixtures

Fixtures are a built-in symfony feature which enables you to easily populate the database with some data. It is usually test data, but for some projects, some initial data can be held in fixtures (e.g. initial configuration values). They should be used in all projects, since it's a lot easier to spot any bug using data that pretends to be real.

the problem

The problem occurs when you combine those two features. Imagine you've got two models in your projects, A and B, related 1:1 (e.g. sf_guard_user with sf_guard_user_profile from sfDoctrineGuardPlugin). For each A record there has to exist the B record. This means, when the mother record is created (suppose A is the master model), the B record needs to be immediately created and related to A. A Doctrine Event Listener is what should be used - a A::postInsert() that creates the B record.

So far, so good - but we still want to use symfony fixtures. Let's say, we want to define 10 sample A records along 10 sample B records. So we write a fixture file with 10 A and 10 B fixtures. Next, we fire the

./symfony doctrine:build --all --and-load

command and what do we find? B records are doubled! And this can be quite difficult to spot!

Each time when a A fixture is saved, the postInsert is automatically executed, therefore new B record is created. So instead of creating records in the following order: A,A,A,...(10),B,B,B,...(10), we get A,B,A,B,A,B,...(10),B,B,B,...(10), having 10 A records and 20 B records. And this is not what we wanted. Fortunately, this can be quite easily fixed.

the solution

Just add one line to the Doctrine Event Listener:

public function postInsert($event)
{
  if ('cli' != php_sapi_name())
  {
    // some code here
  }
}

We're basing on the fact, that fixtures are loaded from a task, which uses command line interface (cli). Now, no fixtures use Doctrine Event Listeners.

data migration with symfony & doctrine

2011-05-08T14:54:00.000+02:00

Scene from "Apocalypse Now" by Francis Ford Coppola (1979)

Diving into 'outsourcing applications with symfony': migrating data

This post was inspired by Fizyk's comment on the outsourcing applications with symfony article. Fizyk suggested that it would be a good idea to wrap all data migration functionalities into a symfony task(s). Basically, migrating data between different versions of the same project is the topic I want to focus on in this article.

new version of a project, written from scratch

In the previous article, we've been discussing developing a new version of an old project from scratch (e.g. because the old one was so inflexible that it was unable to be extended). The first thing we should do is to design new database structure that will hold all data that the present project holds (I assume that the database structure will have to be different, because if it was good, no one would redevelop entire project from scratch). Depending on your luck, it can be an easy task or a nightmare [I've already had redeveloped 2 projects with databases that were not even in the first normal form... yes, there are projects like this. Check this out: one of these projects were made by a really bad developer who never used any type of version control system. He used no development environment (only production) and no localhost machines to test new features before they are added. He made all modifications on the production server, even if it took 2 weeks until they were finished. And there was only FTP access on the server... Oh, yes, there ae projects like that... and you may be asked to improve such projects :)].

data migration

Now when we have two parallel database structures, we need to copy all data from the old one to the new one. Depending on the data (its size, structure, relations, constraints, etc.), I use one of the following:

pure Doctrine (Doctrine in - Doctrine out)
mixed (Doctrine in - SQL out)
pure SQL (SQL query)

For each data structure, I choose one of the above migration types very carefully (detailed descriptions below). If the project is big (and I assume this is the case), the development phase takes a lot of time. It's very important that you can run entire migration job:

as fast as possible: this enables you to run the migration very often, because it doesn't take too much time. And this gives you the possibility to improve the scripts, test them carefully (each mistake made during the migration will be painful when new project is started in production environment). And, finally, you can download the production (old project) database and migrate it to the new structure to compare the frontend of both projects - this helps to develop the business logic,
with wrapped interface: click once - make yourself a cup of tea - come back in 10 minutes - click again - go out with your dog - come back in 15 minutes - the migration is done. Or otherwise: imagine you have a project with 50 tables and you have to run 50 actions (e.g. by a click), each one for each table - or call 50 symfony tasks, not to mention that you need to call them in appropriate order (because data is related in RDBMS) and you have to watch out not to break the order. Of course, there has to be 50 small jobs, deep inside the migration system, but your interface should do the work for you automatically (check an example below).
avoid repeating the same manual modifications. This is difficult to discuss in general - I'll give an example. I've had an action table which stored users activity. The old project database had a record which stated that user X has replied to topic Y with post Z. And neither topic Y nor post Z existed in the database (again, if the project is so bad that it has to be redeveloped from scratch, such bad mistakes are very common). Of course, the migration system failed each time when symfony tried to insert a record that was related to a non-existent record. What to do?
- bad solution: remove this action record from you old database local copy (you download the old project database from the production server, load it to your SQL localhost machine and everythng is 100% safe)
- good solution: remove this record from production database, because it already points some stupid and non-existing stuff.
Why to do so? Because if such mistakes were made in the old project (data inconsistency), there can be quite a lot of such paniful action table, which will make a single migration job really long. And after you have done the whole migration 2 or 3 times, it'll be enough or dumb work and you won't want to do it any more. Again, this is only a simple example. Each situation will probably need a different solution

There is one more important thing: no matter how complicated your migration job is, always create a detailed list of steps to make, describing what is being done at the moment, why such order and what is already migrated/what is still waiting to be migrated. With such list, performing data migration is really easy and unstressful :)

Here you have an example of a migration job order from a project I did recently (all .sql files are 'pure SQL' migrations):

where the action http://polonia/backend_dev.php/migrator/migrateAll calls the following (these are the only 'pure Doctrine' migrations):

MigrationManager::migrateConfig();
MigrationManager::migrateForbiddenPhrases();
MigrationManager::migrateQuotes();
MigrationManager::migrateArticles();
MigrationManager::migrateReadingCategories();
MigrationManager::migrateReadingTextbooks();
MigrationManager::migrateCountries();
MigrationManager::migrateRegions();
MigrationManager::migrateSubpages();

pure Doctrine migration

This is the only code example in the outsourcing applications with symfony article. And this is probably the only migration type that would be wrapped in a symfony task, as Fizyk suggested. This is really easy - just fetch data from the old system and save it in the new system.

used for small amounts of data

Below is another example of this data migration technique - private messages sent between users of a social website:

static public function migrateMessages()
  {
    $wiadomosci = Doctrine_Query::create()
      ->from('Wiadomosci w')
      ->orderBy('w.wiad_id')
      ->fetchArray();
 
    foreach ($wiadomosci as $row)
    {
      $message = new Message();
      $message->setId($row['wiad_id']);
      $message->setTitle($row['temat']);
      $message->setText($row['tresc']);
      $message->setFromId($row['wiad_od']);
      $message->setToId($row['wiad_do']);
      $message->setDisplayed($row['przeczytane']);
      $message->setProvoke($row['zaczep'] == 't');
      $message->setCreatedAt($row['kolumna_data']);
      $message->setUpdatedAt($row['kolumna_data']);
      $message->save();
    }
  }

pure SQL migration

You may ask, why native SQL INSERT queries are constructed to be executed immediately, instead of using Doctrine? The answer is obvious: time difference. Doctrine is nice, but in fact, it is really slow. The problem with pure Doctrine migration is that it takes a lot of time to load all data with PHP/Doctrine and then to insert it into new structure. The table from the example above held over 50'000 records. At my local machine, it took several minutes to finish the job. It's too much (data migration should be as fast as possible). The solution to this problem is simply to create a sql script that will do the job thousand times faster:

INSERT INTO NEW_DB.message (
  id,
  title,
  text,
  from_id,
  to_id,
  displayed,
  provoke,
  created_at,
  updated_at
)
SELECT
  wiad_id,
  temat,
  tresc,
  wiad_od,
  wiad_do,
  przeczytane,
  IF(zaczep = 't', 1, 0),
  kolumna_data,
  kolumna_data
FROM OLD_DB.wiadomosci
ORDER BY wiad_id

Sometimes the case may be even worse. The action table, mentioned before, held over 400'000 records and the PHP/apache configuration would need really big limits (and the migration would take several hours to finish). The following is another sql script, migrating action data:

INSERT INTO NEW_DB.action (
  created_at,
  created_by,
  type_id_external,
  target_profile_id,
  forum_post_id,
  forum_topic_id
)
SELECT kolumna_data, u_id,
CASE akcja
  WHEN "profil" THEN 1
  WHEN "logowanie" THEN 2
  WHEN "wylogowanie" THEN 3
  WHEN "pm" THEN 4
  WHEN "zaczep" THEN 5
  WHEN "post" THEN 6
  WHEN "watek" THEN 7
END,
  IF (akcja = 'profil' OR akcja = 'pm' OR akcja = 'zaczep', akcja_id, NULL),
  IF (akcja = 'post', post_id, NULL),
  IF (akcja = 'watek', akcja_id, NULL)
FROM OLD_DB.akcje ORDER BY kolumna_data ASC

Pure SQL migration may be a little bit more complicated (processing data in PHP is easier) - but you'll surely save a lot of time.

used for huge amounts of data, as this can be the only possibility to migrate it

mixed migration: Doctrine IN / SQL OUT

Sometimes it can be impossible to create a magnificent "INSERT INTO SELECT FROM" query. PHP, although being as slow as a dead snail (in comparison to SQL database management systems), still gives more possibilities just because being an imperative programming language. The idea of this migration type is to fetch the old system data using Doctrine, generate the SQL code and execute it. You may execute such generated script in any way you like - db console, tools like phpmyadmin or even forcing Doctrine to execute raw SQL. If someone prefers, this can be also done as a symfony task - but if the project is really big, you don't benefit from wrapping such mechanism into a symfony task.

To fully understand the idea of this data migration type, take a look at the example below:

static public function getInexistentProfileIds()
  {
    $u_ids = Doctrine_Query::create()
      ->from('Uzytkownicy u')
      ->select('u.u_id')
      ->orderBy('u.u_id')
      ->fetchArray();
 
    $ids = range(1, $u_ids[count($u_ids)-1]['u_id']);
 
    foreach ($u_ids as $u_id)
      unset($ids[$u_id['u_id'] - 1]);
 
    return $ids;
  }

static public function generateSqlToRebuildUsers()
  {
    $ids = self::getInexistentProfileIds();
    $sql = 'INSERT INTO profile (`id`) VALUES ('.implode('), (', $ids).')';
    echo $sql;
    echo '<hr />';
    $sql = 'INSERT INTO sf_guard_user (`id`, `email_address`, `username`,`is_active`) VALUES (';
    foreach ($ids as $id)
      $sql .= $id.',"NULL-'.$id.'","NULL-'.$id.'", 0), (';
    echo $sql;
  }

The reason for doing mixed migration in this case is data inconsistency in the old project. Users were creating forum topics and posts. And when a user was deleted, posts and topics still referred to it. Migrating forum data returned dozens of errors, all of them was failing constraint - assigning topic/post to a non-existent user. The solution was to create dead user records (with deleted_at set), just for the SQL not to break the constraint. The first method fetches all IDs of the users that has been deleted, the second part generates the SQL code creating those dead records into the database. This is only an example, of course, and the example above may be found controversial :), but, believe me, migrating an entire system (which was really, really badly developed) is a difficult task itself, the time you have is limited and sometimes you are forced to use somehow brutal solutions. Anyway, it's up to you how you want to grab the generated SQL code. For me it was the easiest way to display it in the browser, copy-paste to phpmyadmin and execute. But you can generate a sql file, wrap it into a symfony task or do whatever you think of. The main idea is just to make the data migration faster.

and that's it

I hope that some of my hints will be useful if you ever need to migrate data between different versions of the same project. Comments are welcome :)

faster queries: indexing tables

2011-04-18T01:11:00.000+02:00

When designing complex web applications, you have to pay attention to the project performance to make the framework handle your request as fast as possible. This involves optimising client side (clean CSS, clean HTML, fast Javascript, etc.) and server side (caching templates and queries, usage of the database and many others). We will concentrate on the database here. In short, the database should have such structure that all information fetched to handle any request should be accessible very fast. This short article will show you few facts and tricks about symfony projects performance.

built-in foreign key management

One of the brilliant features in symfony is creating indexes for foreign keys by default. This saves a lot of time for the developers and, surely, leverages the overall performance of all symfony applications. Below is an example schema:

Book:
  actAs:
    Timestampable: ~
    SoftDelete: ~
  columns:
    category_id:
      type: integer
      notnull: true
      comment: "kategoria książek"
    title:
      type: string(255)
      notnull: true
      comment: "tytuł"
    author:
      type: string(255)
      comment: "autor"
    description:
      type: string
      comment: "opis"
  relations:
    Category:
      class: BookCategory
      local: category_id
      foreign: id
      foreignAlias: Books

BookCategory:
  actAs:
    Timestampable: ~
    SoftDelete: ~
  columns:
    name:
      type: string(255)
      notnull: true
      comment: "nazwa"

Such schema will generate the following SQL code. Note that the book.category_id column is indexed (faster queries) and constrainted (no data inconsistency) at the same time, automatically:

CREATE TABLE book_category (id BIGINT AUTO_INCREMENT ...;
CREATE TABLE book (id BIGINT AUTO_INCREMENT ...
  INDEX category_id_idx (category_id) ...;
ALTER TABLE book ADD CONSTRAINT book_category_id_book_category_id
  FOREIGN KEY (category_id) REFERENCES book_category(id);

Obviously, you may create your custom indexes (and this will be discussed later). This section from The symfony and Doctrine book covers the topic of custom indexes.

optimising MySQL queries

Before you do anything to speed up your queries execution, you need to know what queries you have in your system. The obvious thing is to look at the powerful web debug toolbar. It's a great tool, but it won't tell you what to do when executing your queries takes too long. But it can point out which queries are definitely poorly written (they are highlighted with orange color then). Then it's time for you to solve the problem. Often, it may happen that you need to join tables (more about this is written in "less doctrine queries" article).

If the number of queries cannot be limited, probably you may need to add custom indexes on your tables. An index is an additional structure, bound to a table, that speeds up selecting the appropriate results (there are lots of good tutorials on this topic, such as the tizag tutorials). The database server, when executing any query, looks for the best structure that can be used to serve the results as fast as possible. We can ask our database server to analyse given any query to tell us how is it going to be executed. And the best tool to visualise this is the EXPLAIN statement (short reference here). We will optimise a heavy query executed on the homepage of a social website, using explain and adding custom index.

example - social website homepage problem

The manager of the social website wants the developers to emphasise the users who are the most active ones. For example, he wants to display last logged users at the homepage. The developers figured out that they need to create an action table that will store actions performed by users. Action and profile tables are related to each other - a simple JOIN will be used each time when the homepage action is executed: last logged x profiles are fetched from the database and displayed then.

The website has been set off. Many users have registered and the action table is growing bigger and bigger every day. After few months, it has over 300'000 records. The manager is very happy that his project is becoming popular, but he noticed that the homepage is being served few seconds slower than it was in the beginning. The developers tell him that they didn't run high performance tests and they have to spend some time on optimisation. The manager is not pleased that it was not considered before.

NOTE: always use test data when focusing on project performance

Symfony has a built-in fixture mechanism which allows you to easily generate lots of different data (see the jobeet tutorial). This is essential when you want to make sure that your project will manage with big popularity. Anyway, if you decide to generate really big amount of data, do NOT use any ORM. It consumes too much memory and generating fixtures takes a lot of your precious time. I'd suggest to generate raw SQL INSERT queries instead - they'll be a lot faster.

Okay, let's move on. Once you have got lots of data (either real or test), execute each action - one after another - and check its performance. First thing you should look at is the mentioned web debug toolbar in the top right corner of the screen when running dev application environment. You should be worried, when you see something like the following:

There is a big problem: at least one of the queries is unoptimal (orange color) and as a consequence, executing this action takes too much time (imagine, almost 5 secs per one execution is really long and it doesn't matter that I'm using my personal computer for testing). Left click on the query icon (the one to the right):

One query takes almost 4 seconds to be executed. This surely causes a serious performance problem! Don't panic, just let your database server analyse the query for you:

EXPLAIN
  SELECT a.id AS a__id, a.created_by AS a__created_by
  FROM action a
  LEFT JOIN profile p ON
    (a.created_by = p.id AND p.deleted_at IS NULL )
  WHERE (
    a.type_id_external = '2'
    AND p.avatar_id IS NOT NULL
    AND p.mode =4
  )
  ORDER BY a.created_at DESC;

Here we can see, that the query has to check at least 1690 p (profile) table rows. And each profile record stores a lot of text data, describing each website user. All this makes the query take such long time to execute. If we want to speed it up, we just have to read carefully the query and concentrate on all columns used (and the order of their appearance). The solution is to find the best index (this topic may be quite complex and independent on the framework you use - ask google about indexing database tables/optimising databases - and read those articles carefully).

In this case, the developers executed the following line in MySQL:

ALTER TABLE profile ADD INDEX deleted_at_idx (deleted_at);

which created an index on the deleted_at datetime column. Thanks to this index, the EXPLAIN command shows that only 10 different rows in the profile table have to be analysed for the query to be executed. And this is a great success - the execution time went down to 0,01 second. Imagine the 4 seconds difference for each homepage display. This is the benefit from optimising project databases.

By the way, I cannot understand why the deleted_at column in the SoftDelete behavior is not indexed by default, especially when you can turn on automatic not-null-check (sql callback):

$manager->setAttribute(Doctrine_Core::ATTR_USE_DQL_CALLBACKS, true);

Provided this line is present in the project configuration, each query that fetches a SoftDelete model will automatically add the "WHERE deleted_at IS NOT NULL" part. It's obvious that such column has to be indexed - the index can be complex though - and the deleted_at can be the last column in this index - but, anyway, default index on deleted_at is a good idea! As you can see, you have to pay attention to what queries are executed inside your projects!

Note: database server different versions use different indexes

Different database server versions may use totally different indexes to execute the same queries on the same database structure! Make sure you run your performance test in the (future) production environment. Otherwise, you may find your application execute unoptimised queries on the production server even if you spent a lot of time optimising it in your local dev machine.

In the example above, it turned out that the production server has a different db server than the developer's local machine. The developer didn't check it - he was not aware of the differences and their negative impact on project's performance. The index that has been built is useless in the production environment (so it should be deleted, because inserting each row is slowed down by this index). Moreover, it happedened, that the new index the developers needed to speed up the query should be build on the action table... pay attention to database server versions you work on!

how many indexes to create

Table indexes are really helpful and they speed up database performance. The more complex your application is, the more queries may be executed. In other words, the bigger your application is, the more queries it may need to provide good performance. But look out - do not create too much indexes and never create any index when you are not sure that it is used somewhere. Why? It's very simple - each index is an additional structure which uses some space and needs some time to be created and managed. When a record is inserted, updated or removed, each index has to be updated accordingly to the change made, which surely consumes time. If you create too many indexes, you may simply slow down your database. For example, each user login inserts new action record - then an action table with 10 indexes will be slower than an action table with only 3 indexes).

symfony validate database content with sfValidatorCallback

2011-03-29T20:34:00.000+02:00

Scene from "The three Musketeers" by Stephen Herek (1993)

In my last project, I needed to create a custom validation feature that would check if there are some specific records in database tables. You can read a lot about symfony validators at the official documentation, but I didn't found an exact solution to my problem there.

There are some customers with their names and their security codes and there are some offers that are bound to customers. There is a offer submission form in the frontend application, where the customer's name and security code have to be given. I found none of the simple validators useful. In the sfDoctrineGuardPlugin there is a sfGuardValidatorUser which is used to validate passwords in the signin form:

$this->setWidgets(array(
  'username' => new sfWidgetFormInputText(),
  'password' => new sfWidgetFormInputPassword(array('type' => 'password')),
  'remember' => new sfWidgetFormInputCheckbox(),
));
 
$this->setValidators(array(
  'username' => new sfValidatorString(),
  'password' => new sfValidatorString(),
  'remember' => new sfValidatorBoolean(),
));
 
$this->validatorSchema->setPostValidator(new sfGuardValidatorUser());

Take a look at the validator's class here. In short, the validator retrieves the user object and compares its password stored in the database with the one accessible from the request. As I mentioned, no simple validator can help, so a post validator has to be used on the entire validator schema:

$this->validatorSchema->setPostValidator(new sfGuardValidatorUser());

However, you can have the same effect without creating a custom validator. Symfony framework provides a callback validator:

$this->validatorSchema->setPostValidator(
  new sfValidatorCallback(array
    ('callback' => array($this, 'customer_code_callback'))));

The customer_code_callback has to be created now:

public function customer_code_callback($validator, $values)
{
  $customer_count = Doctrine_Query::create()
    ->from('Customer c')
    ->where('c.name = ?', $values['customer'])
    ->andWhere('c.code = ?', $values['code'])
    ->count();
 
  if (!$customer_count)
  {
    throw new sfValidatorError($validator, 'Niepoprawny kod lub firma');
  }
 
  return $values;
}

Following the principle the simplest is the best, I'm doing almost the same stuff as the sfGuardValidatorUser, but with less work. Of course, it is up to you to decide whether you prefer to create a simple callback function or a reusable, entire validator class (which is more complicated, more elegant but provides the same functionality). So this is enough for the form to validate the customer's security code, before the offer file can be uploaded.

Of course, apart from the post validator, all form widgets can be validated with simple validators:

$this->setWidgets(array(
  'customer' => new sfWidgetFormInputText(),
  'code' => new sfWidgetFormInputText(),
  'file' => new sfWidgetFormInputFile(),
));
 
$this->setValidators(array(
  'customer' => new sfValidatorString(array(
    'required' => true,
    'trim' => true
  ), array(
    'required' => 'Podaj nazwę firmy',
  )),
  'code' => new sfValidatorString(array(
    'required' => true,
    'trim' => true
  ), array(
    'required' => 'Podaj kod weryfikacyjny',
  )),
  'file' => new sfValidatorFile(array(
    'required' => true,
    'path'     => sfConfig::get('sf_upload_dir').'/offer/',
  ), array(
    'required' => 'Wybierz załącznik',
  ))
));

further usage

The example I gave seems quite similar to password check. But as you can see, you can just replace the example Doctrine query with any other queries, e.g.

check number of offers submitted by a customer, which are still being processed (not finished), if their total count exceeds 10, the validation shall fail
check if there is any free machine/employee that can handle a service, requested by the user; for example, all machines are busy, the validation shows an error you have to wait 36 minutes to wait for the first free machine and, optionally, store the IP to be handled as the first in the queue

less doctrine queries in symfony admin modules

2011-02-14T03:09:00.000+01:00

Scene from "The Damned" by Luchino Visconti (1969)

basic stuff

This short article is a hint for symfony newbies. I'll show how to configure admin modules to limit the number of SQL queries called to generate the list of result.

the problem

The problem occurs when we create an admin module for a model that has some related records which we want to show. By default, Doctrine is not smart enough to guess that joining those related records will generate less SQL queries, so - depending on the database structure - generating a list of 20 objects can generate hundreds of queries (or even thousands, if the there are so many related records to be displayed). The example below is not so tragic, there are only 20 useless queries.

By default, symfony calls a basic

SELECT * FROM model;

query first. Then, while generating the row for each record, it calls for all related objects separately. As we can see below, these are the same queries run for different ids. A great waste of resources.

Now, the thing is to make symfony call less SQL queries to do the same job.

the solution

Let's say, we have a custom model called Proc, representing a procedure.

Go to the generator.yml file in the custom admin module config directory and add the table_method option in the list section:

generator:
  class: sfDoctrineGenerator
  param:
    #...
    config:
      list:
        table_method: getListQuery

This means that the configuration tells the admin module controller that if it want to fetch objects to display in a list, it has to look for the getListQuery in the ProcTable class. So we shall define it:

public static function getListQuery()
{
  return Doctrine_Query::create()
    ->from('Proc p')
    ->leftJoin('p.Tags');
}

And it's all. Fast, easy and clean.

There's also a possibility to achieve the same goal by overriding the buildQuery method in the actions class of the custom admin module (apps/backend/modules/proc/actions.class.php), but it's the dirty solution (thus it's not advisable):

protected function buildQuery()
{
  $query = parent::buildQuery();
  return $query
    ->leftJoin($query->getRootAlias().'.Tags');
}

The result

Now, each Proc object has all related Tag objects fetched already. In the previous situation, when symfony was generating a row for an object (in the admin module list) it called the object for its related objects. None were fetched, so SQL queries were called. Now, when the Proc object is called for its Tags, they're already there, so no additional SQL queries are called.

This limits number of queries and is very important (ignoring this may cause the server to go down frequently). Each action (not only admin module) should be optimised to make symfony call as little SQL queries as possible. Of course, you may join all related models, not only one (the Proc model could have more than only Tags relation, for example, Category, Author, Department and so on - each of them would generate useless queries as well).

Enjoy ;)

EDIT: This post was updated thanks to fizyk's and sacri's opinions. Thanks, guys ;).

documenting database with symfony doctrine

2011-02-06T20:01:00.000+01:00

Scene from "Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb" by Stanley Kubrick (1964)

Symfony projects using Doctrine ORM have their schema defined in schema.yml files. There's a very nice & useful feature, setting comments to your table columns (which is probably not mentioned in the official docs).

It's very easy - just set a comment option to each of your columns, defining what this column holds, for example:

Customer:
  actAs:
    Timestampable: ~
    Signable: ~
  columns:
    firstname:
      type: string(32)
      notnull: true
      comment: "imię"
    lastname:
      type: string(64)
      notnull: true
      comment: "nazwisko"
    email:
      type: string(64)
      notnull: true
      comment: "adres E-mail"

The MySQL code generated for such schema is already with provided comments:

CREATE TABLE customer (
  id BIGINT AUTO_INCREMENT,
  firstname VARCHAR(32) NOT NULL COMMENT 'imię',
  lastname VARCHAR(64) NOT NULL COMMENT 'nazwisko',
  email VARCHAR(64) NOT NULL COMMENT 'adres E-mail',
  created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL,
  created_by BIGINT DEFAULT '1' NOT NULL,
  updated_by BIGINT DEFAULT '1' NOT NULL,
  INDEX created_by_idx (created_by),
  INDEX updated_by_idx (updated_by),
  PRIMARY KEY(id)
) DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci ENGINE = INNODB;

This is how phpmyadmin makes use of column comments:

And below, you can see another example of comments usage - an important database table that many other tables rely on (click to enlarge):

Short and precise description makes a table structure easier to be understood by a programmer.

It takes very little time to define those database comments. But it can save you a lot of time: either if you are working on a big project that has to be well-documented and need to review the structure or if the project is going to be extended by someone else who doesn't know what is where, or you may find it useful in many different situations.

outsourcing applications with symfony

2011-01-10T01:32:00.000+01:00

outsourcing applications

When you need to write from scratch an application that already exists (it's too old to be maintained anymore or simply someone else has written hopelessly unmaintainable junk - and you're the big guy that got the job) - symfony is a good choice! In the following article, I'll concentrate mostly on model and data migration.

overall

The first task is to prepare model for both (old & new) systems (multiple database connection used). Then, if the application already existed, the developer has to migrate all the date from the old structure to the new one, because it has to be available for the new system (e.g. already registered users). By migration I mean simply copying data between two different structures. Data migration will be run many times during new system development and finally it will be run last time when the old system becomes history and the new one is switched on - data has to be up to date. So a migration engine should be developed - this is the second task. And finally, the last one - a useful and handy tool to enable the developer to easily modify models. Each of the three tasks will be described in details below.

model for both systems

Start with creating your local copy of the database that is used by the webpage you are about to refactor. Set up both connections in config/databases.yml file. Now run

./symfony doctrine:build-schema

to creates a schema from an existing database (for the old one). Define your new schema as well. I'd suggest to put them in separate files (e.g. new.yml and old.yml; for Doctrine you only need to put it in config/doctrine directory, it doesn't have to be called exactly schema.yml, can be anything-else.yml).

And the problem starts at this moment... For sure, you'll have to rebuild entire new system (database and all classes) - to modify originally designed database, because of some unpredicted stuff. But this will damage the old system database if you want to create SQL as well (and if the old one is really big, it'd be very annoying to load database dumps anytime you need to change schema). But the part is done - schemas for both systems are created (schema of the old system will never change - we don't care about it anymore).

data migration

For such task it's always better to create a separate class that will handle all data migration, we'll call it MigrationManager.

Basically, one static method should perform data migration for one model (or more, if they're connected, like 1:n or m:n relations). If there'd be too many methods, you can always create more classes - it all depend on the project size. And if the old system data is really big, unfortunately, you won't be able to use Doctrine to insert new data. It consumes lots of memory to create Doctrine record objects. An example of what you can do to avoid the problem is to generate MySQL code (INSERT INTO ...) - then the migration is performed in two steps - generate a database dump file and simply load it.

If the amount of data doesn't force our local machine to use all it's memory (we can use Doctrine to insert data), it should be really easy:

  /**
   * Migrates data from old.aktualnosci to new.article.
   */
  static public function migrateArticles()
  {
    $aktualnosci = Doctrine_Query::create()
      ->from('Aktualnosci a')
      ->orderBy('a.news_id')
      ->fetchArray();
 
    foreach ($aktualnosci as $row)
    {
      $article = new Article();
      $article->setId($row['news_id']);
      $article->setTitle($row['tytul']);
      $article->setTextShort($row['skrot']);
      $article->setTextFull($row['pelny']);
      $article->setActive($row['aktywny']);
      $article->setCreatedAt($row['kolumna_data']);
      $article->save();
    }
  }

As you can see it's not difficult, even if you need to split or join the data or whatever. Then, enable using the migrateArticles() method from the controller - e.g. create a migrator module in the backend application, the module will have has distinct actions migrating different data. Calling any MigrationManager method should result in inserting data to the new system which is equivalent to the old system data.

easy schema/model modification tool

Finally, the last and the most complicated task. As mentioned before, there'll be no need to modify the structure of the old system, as long as no one is working on it at the same time. But there has to be an easy way to modify new system structure.

Assuming we won't have to modify old system structure, I suggest to move all old system files into separate subdirectories of model, form and filter directories (you'll do it only once). For example, let's move them to lib/doctrine/model/old, lib/doctrine/form/old and lib/doctrine/filter/old (and base Doctrine classes to lib/doctrine/model/old/base, lib/doctrine/form/old/base and lib/doctrine/filter/old/base respectively). Moving those files to separate directories gives us two advantages: [1] we'll be able to temporarily get rid of all old connection model classes to make symfony leave old database alone (when running doctrine:build --all) and [2] it simply separates two systems - it's more clear where to find which class.

We'll also need a versioning system like Subversion: after we temporarily delete class files, we'll retrieve them back.

To make it all easier, I created two batch scripts that handle most of the job. Rebuilding new system structure requires doing 5 things:

execute remove_old script - delete everything that is connected with the old system (so that old system data won't be erased)
clearing cache - old connection classes shall be removed from cache also
doctrine:build --all - run the main task
committing all structure changes - using a versioning system, all changes will be saved, new system classes will replace their previous version so the versioning system will automatically note file differences
execute model_reload script - get the original settings back

Examples of both scripts are provided below.

remove old - script

#!/bin/sh
 
echo 'deleting old system files'
rm config/doctrine/old.yml
cp config/databases.without-old.yml config/databases.yml
rm -fR lib/model/doctrine/old
rm -fR lib/form/doctrine/old
rm -fR lib/filter/doctrine/old
echo 'old system files deleted.'

old.yml file is the schema file holding old system structure. We delete it because we don't need those classes to be rebuild, especially that we moved them to a separate directory, so if they were rebuild, we would get two parallel set of classes (too bad...)
databases.without-old.yml is the same as databases.yml (two connections defined) with the only difference that the old connection is commented out (I prefer not to delete it just in case anything goes wrong with using versioning system).
removing class files - we don't have to list all classes separately (deleting entire directories is enough), because they've been separated from new system classes

model reload - script

#!/bin/sh
 
echo 'deleting new generated schema/model/form/filter'
rm -fR config/doctrine
rm config/databases.yml
rm -fR lib/model
rm -fR lib/form
rm -fR lib/filter
echo 'model/form/filter deleted.'
 
echo 'restoring svn schema.'
svn up config/doctrine
svn up config/databases.yml
echo 'svn schema restored.'
 
echo 'restoring svn model.'
svn up lib/model
echo 'svn model restored.'
 
echo 'restoring svn form.'
svn up lib/form
echo 'svn form restored.'
 
echo 'restoring svn filter.'
svn up lib/filter
echo 'svn filter restored.'

removing everything
retrieving it from the versioning system; the project has now acces to both databases and is up to date with both schema files!

Test it yourself. Any feedback is welcome!

Take a look at the data migration with symfony & doctrine article which is a continuation of this article, inspired by Fizyk's comment.

symfony sfAdminDashPlugin icons

2010-12-31T11:38:00.000+01:00

sfAdminDashPlugin is one of the most useful symfony plugins (it's the 8th most popular plugin at the moment). I want to show few ways how to easily extend your backend application with custom icons: where to get them from and how to make your project use them. It seems not very complicated - and it's not :)! But the way your website looks like is still very important.

sfAdminDashPlugin built-in login screen

sfAdminDashPlugin is provided with some standard icons, but probably you'll need more of them. Especially that it's really easy to add new ones - just follow these steps:

find icon(s) that have the folowing sizes: 16x16 and 48x48, should be in .png format and be transparent in the background
copy those files to the plugin directories: PROJECT/plugins/sfAdminDashPlugin/web/images/icons - for the 48x48 file and PROJECT/plugins/sfAdminDashPlugin/web/images/icons/small for the 16x16 file

use new icon(s) in your config/app.yml configuration file:

items:
  Articles:
    url:          article
    image:        CustomIcon.png

set of icons provided with sfAdminDashPlugin

desktop environment

The easiest place to look for your icons is your desktop environment resources directory. For example, I use KDE which stores its icons in

/usr/share/icons/
/usr/share/icons/default.kde4/16x16
/usr/share/icons/default.kde4/48x48

external resources

KDE has lots of icons added by users, which you can find here. I have downloaded several packages and whenever I need a specific icon, I look through them. Good thing about icon themes is that large amount of icons are created in the same style, which would make your backend look even better!

nuvola KDE icon theme

Of course, you may also find nice icons at special websites, such as

and many more (these two I use the most - to see the rest, ask google)...

icon usage example

drop-down menu example

icon usage example

drop-down menu example

icon usage example

simple mailing system with symfony - part I

2010-11-29T01:05:00.000+01:00

This article is the beginning of a tutorial presenting an easy way to implement database-templated-mailing system for symfony projects.

the basics

The basic version uses 2 tables: mail_queue and mail_template. For each type of mails, one template object is created.

mail_template

name, description - name & description of the template
template_code - (HTML, including variables marked as {variable})
template_data - what variables are defined, just a description, no calculations performed

For each single mail to be sent, one queue object is created.

mail_queue

template_id - which template to use
mail_data - serialized array: variables=>values
mail_recipent - formatted/serialized recipent information
mail_author - as above
mail_subject - string
send_at - time that this email should be sent
sent - boolean, already sent

E-mail is sent when sent=0 (and marked 1 afterwards, not to be sent again; in case of errors - mark as 2) and when actual time has gone beyond send_at. send_at column is especially useful, as you may create emails in a long time advance. For example, you want to send a reminder E-mail: create a new MailQueue object with its datetime value equal to current timestamp + two weeks. This would make the E-mail to be sent in two weeks time from now.

example usage

What kind of mails can we define? It depends on your project functionalities. For example, if your project is a stock managing software that stores information about products and their states, you can send important E-mails when one of the following situations take place:

status of a product is changing - other employees shall be informed about that,
quantity of a product reached alarming level - supply needed immediately,
inform customers when product becomes available again,
and so on...

A template object is created for each of the situations defined above. There's a MailTools class with static methods processing mail sending. Just like notifyProductStatusHasChanged method, taking two parameters: Product and status change. This method creates a mail_queue instance, defining specific data for this type of E-mail, using Article object passed as the parameter.

some details on implementation

Symfony Swift Mailer integration is provided with a ready to use mail queueing system. However, I decided to create a special task, defined to send E-mails that are marked with sent=0. The code is pretty obvious:

$mails = Doctrine::getTable('MailQueue')
  ->getMails2bSentQuery()
  ->execute();
 
foreach($mails as $m)
{
  $m->send();
}

Above task is run by a cron job (every few minutes). getMails2bSentQuery method looks for MailQueue objects that have sent=0 and send_at <= now. For each of those objects, the template code (HTML) is filled with variable data and sent afterwards. The mail data is unserialized first and each variable is injected into the HTML code using str_replace function:

protected function generateContent()
{
  $code = $this->getTemplate()->getTemplateCode();
  $data = unserialize($this->getMailData());
  foreach($data as $key => $value)
    $code = str_replace('{'.$key.'}', $value, $code);
  return $code;
}

Each mail template is defined using HTML/CSS code. It works similar to smarty template engine. Just define a list of variables of and all occurences of {VARIABLE} will be replaced with the parameter value you pass.

to be continued

This was just a brief overview. In the near future, precise code examples will be provided in tutorial's part 2.

Doctrine SoftDelete behavior usage

2010-11-01T02:28:00.000+01:00

Scene from "The Neverending Story" by Wolfgang Petersen (1984)

SoftDelete

Recently, I created a detailed list of Doctrine behaviors ready to use in symfony projects. Among them, there is the SoftDelete behavior I want to focus on this time. For a chosen model, it adds a deleted_at column, which defines if a record has been marked as deleted (and if so, when).

where to use it?

Suppose your system needs to store all the detailed data history, every modification has to be marked and stay there forever. And even if the application allows users to delete objects, they are not really deleted - it is just an abstraction layer - in fact, objects are marked as deleted, but they stay in the database (and the application treats them as if they were deleted). This is where SoftDelete comes handy.

the code

All you gotta do is just to state that a given model is SoftDelete:

GivenModel:
  actAs:
    SoftDelete: ~

After this, the deleted_at column is added (of course, both to SQL and PHP class). But that's not all the job. Unfortunately, you have to tell the application how to use this column manually. Fortunately, this is really easy. For example, if you have an admin generated module, soft-deleted objects shall never appear in the list. Add/modify the buildQuery action of the admin module:

protected function buildQuery()
  {
    return parent::buildQuery()
      ->andWhere('deleted_at IS NULL');
  }

Additionally, if you want to have a full protection, you should update your admin module edit action to disable executing it for a soft-deleted object. So this was for the backend. For frontend, you shall modify your eventual data retrieving table class methods, like the following:

public function getObjectByIdQuery($id)
  {
    return Doctrine_Query::create()
      ->from('GivenModel gm')
      ->where('gm.id = ?', $id)
      ->andWhere('gm.deleted_at IS NULL');
  }

Of course, soft-deleted objects can NEVER be accessible from any type of frontend applications. There is no way to give a detailed list of modifications you need to provide 100% data protection, because all applications have different structure - you have to go through your functionalities on your own. These code lines above are just examples you will probably use.

enable DQL callbacks

Alternatively, you may enable DQL callbacks for your models in each of your config/ProjectConfiguration.class.php files:

public function configureDoctrine(Doctrine_Manager $manager)
  {
    // Enable callbacks so that softDelete behavior can be used
    $manager->setAttribute(Doctrine_Core::ATTR_USE_DQL_CALLBACKS, true);
  }

this will dynamically add an andWhere DQL clause to all SELECT queries for SoftDelete models (Doctrine_Template_Listener_SoftDelete::preDqlDelete()). Thanks to it, you don't have to manually update all possible SELECT queries in your project (such as buildQuery mentioned above).

be careful!

As the official behavior docs say, SoftDelete overrides the delete() method. When delete() is called, instead of deleting the record from the database, a deleted_at date is set. This means, if you try to delete records without using Doctrine_Record::delete() method, SoftDelete won't work! To provide 100% data protection covering SoftDelete you should replace all code like

Doctrine_Query::create()
  ->delete()
  ->from('GivenModel gm')
  ->where('gm.id = ?', $id)
  ->execute();

with

$objects = Doctrine_Query::create()
  ->from('Manufacturer m')
  ->where('m.id = ?', $id)
  ->execute();
 
foreach($objects as $object)
  $object->delete();

to make the Doctrine_Record::delete() be called whenever an object is supposed to be deleted, which finally means that the desired SoftDelete behavior is fired always when it should be.

objects related to SoftDelete-able objects

But there comes another question - what to do with related records of a soft-deleted object? For example, we have two model classes with the SoftDelete behavior: Manufacturer and Supplier which have a m:n relation table, ManufacturerSupplier which defines who supplies which manufacturers. If some specific manufacturer and supplier objects are marked as soft-deleted, their corresponding ManufacturerSupplier object (if it exists, let's suppose it does) is left and no one knows how to treat it. It is not marked as soft-deleted, since it hasn't got SoftDelete behavior. And it still exists in the database, like a full-fledged record. Probably, it does not raise any application problems, but there is a data consistency question - what does such m:n table record represent, when at least one of its related objects are soft-deleted? Shall it be soft-deleted after the related master object is soft-deleted? Or maybe hard-deleted?

MySQL error on rename when dropping column with a constraint

2010-10-07T19:10:00.000+02:00

a nasty MySQL bug

Recently I wanted to remove a column from phpmyadmin that is covered with a constraint. Impossible to do so, MySQL throws following error:

ERROR 1025 (HY000): Error on rename of './optiner_prestadmin/#sql-6bc_3e6' to './optiner_prestadmin/order_history' (errno: 150)

You may get above error when executing a query removing index, e.g.

ALTER TABLE order_history DROP INDEX user_id;

Fortunately, there is a solution, not very easy to find in the internet (I used comments from this article).

execute the following in MySQL:

SHOW CREATE TABLE order_history;

You shall get something like:

order_history | CREATE TABLE `order_history` (
  `order_id` bigint(20) NOT NULL,
  `user_id` bigint(20) NOT NULL,
  `status_id` bigint(20) NOT NULL,
  `informed` tinyint(1) DEFAULT '0',
  `comment` text,
  `created_at` datetime NOT NULL,
  `created_by` bigint(20) NOT NULL DEFAULT '1',
  PRIMARY KEY (`id`),
  KEY `order_id_idx` (`order_id`),
  KEY `user_id_idx` (`user_id`),
  KEY `status_id_idx` (`status_id`),
  KEY `created_by_idx` (`created_by`),
  CONSTRAINT `order_history_created_by_sf_guard_user_id`
    FOREIGN KEY (`created_by`) REFERENCES `sf_guard_user` (`id`)
      ON DELETE CASCADE,
  CONSTRAINT `order_history_order_id_order_info_id`
    FOREIGN KEY (`order_id`) REFERENCES `order_info` (`id`),
  CONSTRAINT `order_history_status_id_order_status_id`
    FOREIGN KEY (`status_id`) REFERENCES `order_status` (`id`),
  CONSTRAINT `order_history_user_id_sf_guard_user_id`
    FOREIGN KEY (`user_id`) REFERENCES `sf_guard_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=28305 DEFAULT CHARSET=utf8

we want to delete the whole column user_id (with all corresponding indices, constraints, etc.) - need to drop the FOREIGN KEY first:
```
ALTER TABLE order_history
  DROP FOREIGN KEY `order_history_user_id_sf_guard_user_id`;
```
you'll find the name of the foreign key in the last constraint clause.

finally, drop the column:

ALTER TABLE `order_history` DROP `user_id`;

and that's it!

Query OK, 28065 rows affected (3,99 sec)
Records: 28065  Duplicates: 0  Warnings: 0

symfony & Doctrine behavior review

2010-10-05T22:56:00.000+02:00

Doctrine behaviors

The main idea of this article is to create a short summary on Doctrine behaviors (really powerful and useful tools) which speed up development significantly. I tried to mention the most important behaviors, providing brief description and links. It's quite common to reinvent the wheel, being just unconscious that someone has already implemented a feature you are just in need of. Moreover, I hope that both beginners and advanced symfony developers will find this article a useful source of brief information about Doctrine behaviors for everyday use.

Basically, a behavior lets you share some relations, algorithms and other features among your project's model. They are (or at least should be) highly configurable: providing you a mechanism that can be easily customized to fit in different projects. Customization usually includes: enabling/disabling some of the behavior features (you may use only some of them), renaming additional schema columns, configuring them (like adding 'not null'), etc. You can use many Doctrine behaviors in your projects: core behaviors, Doctrine extensions, symfony plugins - or if you've got nothing better to do late at nights - you can create your own behaviors ;)

Core behaviors

The Doctrine ORM is bundled with several behaviors (calles core behaviors), these are:

Versionable - add an entire XxxVersion table to store your Xxx model object versions as either only version numbers or numbers versions along with ALL column data - to see object data changes through time,
Timestampable - probably the most popular of all behaviors, adds created_at and updated_at timestamp columns to your model, automatically saving datetime when a record is created or updated, respecticely,
Sluggable - adds a slug column that stores a slug which is a unique index automatically and can be used along with sfDoctrineRoute class to refer to an object without passing its ID as a parameter; by default the update is disabled, since this may harm your search engines rating,
I18n - add an entire XxxTranslation table to provide Internationalization (I18n) for your Xxx model, essential when developing a multi-language project,
NestedSet - adds root_id, lft, rgt and level columns to your model to develop an advanced hierarchical data structure (such as product categories), nested sets is an alternative to adjacency model, more details on hierarchical data here,
Searchable - choose model columns you want to index and add an entire database table, speeding up a basic search engine development, more info about searching here,
Geographical - adds longitude and latitude columns storing geographical coordinates. Ever needed to use gmaps in your project, along with sfEasyGMapPlugin? Not only this behavior suits the data structure you need, but also provides you with getDistance() method to calculate distance between two Geographical objects,
SoftDelete - adds a deleted_at column which defines if a record has been marked as deleted (and if so, when). Useful when designing a highly complicated system where data consistency is important and even if some data should be invisible in the backend, it should still remain in the database.

extension behaviors

You may also use Doctrine extensions:

Blameable - adds an additional level of auditing capabilities to models, blameable allows you to track who created or last updated any model in an environment with many users, blameable is a great companion to Timestampable behavior,
EventLoggable, readme - logs any Doctrine Events (pre/post Delete/Insert/...) fired upon a record, log type may be chosen (e.g. file) and events may be limited to the chosen ones only,
Localizable - gives you functionality to convert units of measurement (e.g. kilometers to miles, etc.) or any other conversions desirable,
Locatable - gives you functionality using the Google Maps API to automatically populate your model with latitude and longitide information using Google, a fantastic tool to use along with sfEasyGMapPlugin,
Sortable - gives you sortable functionality to your models, enabling you to easily fetch next/previous objects, moving them up/down in a sorted list or swapping with another object,
Taggable - adds Tagging capabilities, creates a TaggableTag table and a XxxTaggableTag table for each taggable model, using refClass m:n db table relation, provides easy tag management: add, remove, remove all, set, etc, finally, gives you the possibility to get the most popular tags (for a tag cloud, for example) fetched either from chosen models or from all of them,

plugin behaviors

Finally, there are also symfony plugins providing custom behaviors:

sfDoctrineActAsSignablePlugin - a well developed alternative to blameable Doctrine extension,
sfDoctrineActAsTaggablePlugin - provides tag management (alternative to taggable Doctrine extension), provides a tag cloud generation,
csDoctrineActAsAttachablePlugin - associate various uploads with multiple models,
sfDoctrineJCroppablePlugin - add user-croppable images to your models with a javascript image cropper,
sfDoctrineNestedSetPlugin - a form and a validator providing easy NestedSet behavior management,
csDoctrineActAsCategorizablePlugin - categorize your models, mix, match or separate multiple models into multiple nested/flat categories,
sfDoctrineActAsRattablePlugin - rate Doctrine objects,
vjCommentPlugin - provides commentable behavior to comment news page, articles, etc,
ajDoctrineLuceneablePlugin - provides luceneable behavior with Lucene full text integration,

useful links about Doctrine behaviors:

notes

If you find any information missing, please, let me know. Hopefully, you'll find it useful, otherwise I'll lose the motivation to do such thing next time ;)

update record changes with sfDoctrineActAsSignablePlugin: Signable behavior

2010-10-03T18:18:00.000+02:00

sfDoctrineActAsSignablePlugin

The plugin is fairly simple, it contains only a Doctrine temlate and a template event listener (basic stuff to implement a new Doctrine Behavior). It was initially released by Vitaliy Tverdokhlib.

Signable Doctrine Behavior

The plugin provides a Signable behavior ready to be used in your models:

Model:
  actAs:
    Signable: ~

It works very similarly to Timestampable behavior: created_at and updated_at timestamp columns are added, storing information about the datetime that a record was created and last updated. Signable adds created_by and updated_by columns storing information about the user that created or last updated a record.

The plugin may be used along with sfDoctrineGuardPlugin. This gives you a possibility to refer to the sf_guard_user table. No additional configuration is needed, the plugin automatically checks if sfDoctrineGuardPlugin is installed.

configuration

The most important thing you can configure is to choose the type of the user field: it can be either string or integer:

Model:
  actAs:
    Signable:
      created:
        type: string
      updated:
        type: integer

If string type is chosen, you can simply display the name of the user (e.g. in frontend blog article display page), but can do nothing further. If you choose integer, you store ID of the user, which gives you a possibility to refer to the user in the user database table (which gives you all possibilities of using user Doctrine model).

You can configure the behavior to meet your needs. Let's take a look at some examples:

The created_by and updated_by column names are the default ones. You may change them:

Model:
  actAs:
    Signable:
      created:
        name: creator_id
        type: integer
      updated:
        name: update_username
        type: string

You want to know which user has created a record only (meaning that the user who updated the record is not important). In this case, disable udpated option:
```
Model:
  actAs:
    Signable:
      updated:
        disabled: true
```
You want to add the Signable behavior to models of an existing project that already maintaines lots of data. And you want the created_by/updated_by user to be not null, because there's always someone who creates or updates a record. In a big project (a CRM, ERP, etc.) it's a good solution to create a system user (with id=1 for example) who represents all system actions (a daemon creating/updating objects from cron tasks):
```
Model:
  actAs:
    Signable:
      created:
        options:
          notnull: true
          default: 1
      updated:
        options:
          notnull: true
          default: 1
```
In this case, adding created_by/updated_by database columns will fill them initially with 1 (system user) and you can refer to the user (e.g. using getUser() method), being sure that there's always a user referenced. And all further modifications will store real user ids.

real world examples

Blog articles are being added by application users. You add Signable behavior (with type: string) and you may use:
```
echo $article->getCreatedBy();
```
in the app frontend to display who submitted the article.
An E-commerce applications management system: users (employees) log in to the system and create orders that customers submit. You add Signable behavior (with type: integer). Whenever an order is created, the created_by is filled with the user ID. The application can generate statistics on which users added the most orders.

No matter if your project is a complex management system with extended backend - or a small company website with mainly frontend developed - you'll find this plugin useful!

new releases of sfApplicationMapPlugin

2010-09-13T21:52:00.000+02:00

application map

sfApplicationMapPlugin is an easily configurable documentation-generator tool. It generates a graphical map of your applications using GraphViz software. It is useful when you want to have an overwiev of a specific application or of a whole project - each module/admin-module is marked with all actions and their comments. Take a look at the plugin readme tab to see some examples. You need to have GraphViz installed on your system to use the plugin.

new release

New plugin versions supporting symfony 1.2-1.4 has been released. Thanks to bugs reported by Gordon Bazeley, the plugin should work correctly with no dependency on operating system.

an official GraphViz-based software

sfApplicationMapPlugin is listed as a Software Engineering Tool on the official Graphviz website.

todo and contribution

The main thing to be done is supporting plugins used in projects. Any contribution is welcome, as well as bug reports and questions.

Symfony framework instead of a well-known CMS

2010-09-13T00:15:00.000+02:00

Scene from "All Friends Here" aka "Our Folks" aka "Sami Swoi" by Sylwester Chęciński (1967)

Recently I got involved in developing and deploying a small website promoting a book that has just been released. My coordinator asked me to choose one of the most popular Content Management Systems (Wordpress, Joomla, Drupal) to deploy the site on. Since I have a very little experience with developing apps based on a full-featured CMS, I was not so happy to be forced to learn a CMS just for a single small project.

I was given a full, well-prepared layout. The first step was to replace the default layout with the one I have just received. I took a look at Wordpress (because it seems to be the easiest and the smallest CMS). A glance at the template files has given me an overview stating that I will really need to spend some time to fully understand the wordpress templating system. Of course, it doesn't mean that it's difficult or out of my reach - I'm just lazy, as most of us :) - and I prefer designing complex and complicated systems rather than studying boring wordpress engine...

I got an idea of giving up wordpress (after 20 minutes of reading documentation)... At first, it sounded quite crazy - to develop a small site with such a powerful tool like symfony. I asked myself questions like 'do I really need such an advanced tool to develop such a trivial project?' or 'how long is it going to take me to implement all mechanisms that are already implemented in wordpress (such as nice looking menu, panels, intuitive interface and so on)?' Maybe long... but the more I was analysing the time difference between designing the app in an unknown CMS and very well-known symfony framework, the more the symfony option was attractive.

Developing the frontend is only putting the new layout on. Backend is more complex. But still it's all very easy and extremely fast to use. Using sfAdminDashPlugin, sfJqueryReloadedPlugin and sfDoctrineGuardPlugin, the greatest 3 symfony plugins, gave me the basis of a backend engine with many features ready to be improved (e.g. I could define user credentials, I could organise the backend menu, etc). I use these 3 plugins in 99% of my projects (and suppose more people do so) and it takes me very little time to configure those plugins.

Developing backend is mainly working with generator.yml/module interface configuration, because this part is usually different in all web apps (meaning that you can copy lots of model/form/filter methods from your previous projects, to save time). If you implement any feature and use it in any project, you can always copy it later to another project (or make yourself a plugin, which is usually a better choice, because the whole community may reuse your work). Just like my AJAX backend activate/deactivate feature (see easy symfony ajax user friendly example) - after I managed to implement the feature (which originally could have taken me up to 3 hours, including studying entire symfony AJAX, ajax routing, sfJqueryReloaded and so on), now it takes me less than 5 minutes to copy it to a new place. Or to make a similar feature somewhere else. And imagine that some mechanisms, like taggable or commentable behavior can be put into plugins, which makes it even faster to use! This is one of the biggest symfony advantages - the speed on development. All of us shall use it as much as we can.

Choosing symfony for this project was a great decision. It saved me a lot of time, made me proud of how fast a small symfony project can be developed. Still, the code is comprehensible, it's kept well-documented (so the entire project is opened for any modifications). My experience I want to share with you is that symfony is a good choice also for small projects.

symfony file upload - leave original file name

2010-09-07T01:53:00.000+02:00

If you want your uploaded files to stay with the original file name (which is not randomly generated by symfony), just add following lines to the model class you want to upload files:

public function generateXxxFileName($file)
{
  return $file->getOriginalName();
}

where Xxx is the name of the column that stores the filename in the database model related table.

symfony crons and cron task logging

2010-09-06T00:51:00.000+02:00

Scene from "Frankenstein" by James Whale (1931)

this is really magic...

Recently, I found an update of the sfTaskLoggerPlugin. As the description says, the plugin allows you to run custom tasks and store the results. In fact, this is a highly customizable tool that provides you monitoring all cron tasks, checking their performance (objects processed, running time) and so on. Installing and configuring this plugin in my company CRM was one of the most challenging and most interesting things I have done since many months.

few words about cron tasks

The most important of my projects is a CRM tool that provides some tools for managing an E-commerce company. It is an application that stores lots of data in its own database, but it also has access to several other databases and connects external systems through APIs. Such a complicated system requires several cron tasks (the number is going to be doubled soon). Some of them have to be run once per day, but others have to be run every few minutes. There are few problems you have to face when using cron tasks in a big system:

Running each cron consumes server resources and estimating the best time period between the cron finishes his task and before he's run again is not easy. If the period is too small, lots of resources will be consumed with no sense (everything depends on the cron, of course), but if the crons are run too rare, other employees' work is a lot more dificult, because their data s not up to date.
Another important thing is task performance - how fast does it take until the cron finishes his task. When you implement a new task, you measure how long does it take to run the task once. But if your data grows incredibly fast during just few months, will you always remember to check if everything is ok with the application performance?
Finally, you need statistics to know what is going on in your system - how often a functionality is used, what is the traffic during specific part of the day or for different days of week, etc.
Just one more thing - all this information you would like to know about your own crons should be easily accessible. Until now, I had a basic logfile system which was really far away from meetig my expectations (look above).

so, is this plugin useful?

Extremely! As almost everything in life, the most useful things are usually the easiest. And it's the same right here. But I must admit, that at first sight the plugin's readme seemed complicated - but it's only the first sight (according to COil, the author of the plugin). In fact, my original task was handled by the plugin after 20 minute. COil replied to my mail within few hours and I got all the answers I needed to install his plugin into my apps.

few examples of where sfTaskLoggerPlugin is useful

There are some of my cron tasks:

migrating data between databases. Different www applications are run independently, each of them having it's own database. It's pretty clear, that an employee won't check several different admin panels each few minutes, so everything has to be in one place. And here comes migrating orders... A customer visits a webpage and submits his order. A system cron is waiting for this to happen and after at most 5 minutes he's run and copies the order data into the CRM database, where all tools to manage this order are available. But what is the time when most orders are submitted? How many, how often? From which of our applications? And how often such cron should be run? And how long does it take...
e-mail sending - similar as above.
redundant table columns speeding up other functionalities. In my project I have to generate lots of different XML files for external applications such as price comparators or catalogues. We're using the prestashop to deploy shops. Source prestashop database structure can be highly ineffective in our case, so some product data has to be copied between tables once per day (the ineffective queries are executed only once and the result is stored in the redundant columns which are very easily accessible). And the problem is that it has to be run on all products we have in our database. Running a time-consuming query may be dangerous for performance (it's expected that number of products we have is going to be 4-5 times more that it is now). So I need to watch and check if there's need to divide the process into phases (processing parts of data separately) to prevent a breakdown.
Another thing to look closely at is accessing external applications' WebAPI. It happens that the WebAPI may be developer-unfriendly :) by providing methods that are really poor. And by the same time, the external application generates great traffic to your company. Well, such integration will be difficult then. Not only you'll have to watch out for lots and lots of traps, problems (and finally - bugs), but also you need to know when to run the cron task and how often. And this is not only about the comfort of other employees' work, but mainly about data consistency, this time. Programming can be art, definitely :).

main sfTaskLoggerPlugin's features

Well, it's pretty easy - each cron run is stored in the database. You know the starting and the ending time of each task (so you know how long does it take to finish the task, how often such task is run, you can check if such task was already run today - and so on). You can define the count of the processed and unprocessed objects (if any error occurs or if just some amount of data has to be processe later - than you define the id of the last processed object). You define the error code, stating if there were any errors during runtime, you're able to check how many crons are running at the moment (or how many of them has broken and was never finished). Finally, you define your own comments for each cron. I use it to generate HTML code that is easily accessible by office workers (in case they need to find if some data was processed and what was the result).

It's easy to display/generate project cron statistics. I'm gonna use stOfcPlugin, for example, to display how many percent of cron task run migrates any data (and how many just checkes that there is nothing to migrate). There is plenty of examle statistics to generate. I'm gonna provide some real cron examples with potential stats soon...

why so small popularity?

To sum up, it's such a shame that there are so many good plugins that are not popular, including sfTaskLoggerPlugin. It's really difficult to break through and make your tool popular in the community. Well, using cron tasks is not the most important symfony feature - but I'm pretty sure that there are more than just 4 users who can make use of logging cron tasks... Or am I wrong? :) Just give this plugin a try!

By the way, we're looking for a Propel developer who is willing to contribute to the plugin. Any good contribution is welcome! ;-)

symfony popup with jQuery thickbox - beginner's tutorial

2010-08-29T13:26:00.000+02:00

Scene from "The Jetsons" by Hanna-Barbera (introduced in 1962)

Introducing jQuery thickbox pop-up

In this tutorial I will show you an easy way to create a thickbox using jQuery thickbox lib. At first, make sure you are familiar with the thickbox and take a look at the examples tab. We will create our pop-up feature basing on the iFramed Content section.

Install jQuery thickbox

The first thing you need to do is installing the software, of course. I want to make this tutorial as easy as possible, so just run the following commands from the project root directory:

wget http://jquery.com/demo/thickbox/thickbox-code/thickbox-compressed.js -O web/js/thickbox-compressed.js
wget http://jquery.com/demo/thickbox/thickbox-code/thickbox.css -O web/css/thickbox.css
wget http://jquery.com/src/jquery-latest.pack.js -O web/js/jquery-latest.pack.js
wget http://jquery.com/demo/thickbox/images/loadingAnimation.gif -O web/images/loadingAnimation.gif

After the files are downloaded, you need to make symfony include them into the generated output. One of the possible ways is to edit your application/config/view.yml file:

  stylesheets:    [ ..., thickbox.css ]
  javascripts:    [ ..., jquery-latest.pack.js, thickbox-compressed.js ]

The thickbox is already installed in your symfony project.

Basic usage

To make everything elegant, add 2 routes to your application/config/routing.yml file:

index_popup:
  url:   /popup
  param: { module: main, action: popup }
basic_popup:
  url:   /basic-popup
  param: { module: main, action: basicPopup }

The first route is just a demo action to demonstrate thickbox and you can the index_popup action with the following url:

http://local_sf_test/popup

The second route executes an action that shall be displayed inside the popup. Now we shall create the index_popup action inside the application/modules/main/actions/actions.class.php file:

/**
  * Executes popup action
  *
  * @param sfRequest A request object
  */
  public function executePopup(sfWebRequest )
  {
  }

and a link executing our basic popup - add the following code into your application/modules/main/templates/popupSuccess.php file:

<a class="thickbox" href="<?php echo url_for('@basic_popup') ?>?keepThis=true&TB_iframe=true&height=100&width=400">run basic popup!</a>

It's already working - if you press this link in the development mode, you'll get the symfony error Action "main/basicPopup" does not exist. - but generated in the thickbox. Now we're only left to create this action - create the file application/modules/main/templates/basicPopupSuccess.php file:

<h1>basic popup example</h1>

and the last thing - the basicPopup action inside the application/modules/main/actions/actions.class.php file:

/**
  * Executes basic popup action
  *
  * @param sfRequest A request object
  */
  public function executeBasicPopup(sfWebRequest )
  {
  }

The logic is very simple - only two things to be done:

The string ?keepThis=true&TB_iframe=true&height=100&width=400 is added to the end of the link's href attribute to pass some extra parameters to the thickbox. Those parameters can be easily modified, according to thickbox documentation.
The link has to have the CSS thickbox class

The screenshot below presents the output:

As we can see, the web debug toolbar is not fitting well here, therefore we may disable it by adding the following code to the executeBasicPopup action:

sfConfig::set('sf_web_debug', false);

This would display the popup without web debug no matter if the production or development controller is running:

Advanced usage

Above example could only display some static content. But it's not all that symfony thickbox integration can do - for example, you can embed symfony forms inside a popup. Create new action inside the application/modules/main/actions/actions.class.php file:

/**
  * Executes advanced popup action
  *
  * @param sfRequest A request object
  */
  public function executeAdvancedPopup(sfWebRequest $request)
  {
    sfConfig::set('sf_web_debug', false); // disable sf_web_debug
    $this->form = new ExampleForm(); // create the form object
 
    if ($request->isMethod('post'))
    {
      $this->form->bind($request->getParameter('example_form'));
      if ($this->form->isValid())
      {
        // do some business logic
        $this->getUser()->setFlash('notify', 'this message will be displayed inside the popup');
        return $this->renderPartial('main/message');
      }
    }
    return $this->renderPartial('example_partial');
  }

and a new route inside application/config/routing.yml file, corresponding to the new action:

advanced_popup:
  url:   /advanced-popup
  param: { module: main, action: advancedPopup }

update the popupSuccess.php file:

<a class="thickbox" href="<?php echo url_for('@basic_popup') ?>?keepThis=true&TB_iframe=true&height=100&width=400">run basic popup!</a>
<a class="thickbox" href="<?php echo url_for('@advanced_popup') ?>?keepThis=true&TB_iframe=true&height=100&width=400">run advanced popup!</a>

Now it's up to you to create a form class (the easiest way is to use a model form generated class) and display all its fields inside the _example_partial.php file:

<style>
#form {background: #d3e5f4; padding:5px 15px; color:#1f3037;}
/* some other CSS to be used by the popup form */
</style>

<div id="form">
  <form action="" method="post">
    <?php echo $form->renderHiddenFields() ?>
    <?php echo $form['example_field']->renderError() ?>
    <label class="label" for="example_form_example_field">example label</label>
    <?php echo $form['example_field']->render() ?>
    <input type="submit" value="submit" />
  </form>
</div>

The last thing to be created is the file to be used as the message partial, _message.php:

<span>example message</span>

Live app example

After clicking this button:

the following popup is displayed:

the form validation errors are displayed:

and finally the submission message is displayed:

Embedding videos in symfony projects using sfVideoPlugin

2010-08-16T23:52:00.000+02:00

What does the plugin do?

sfVideoPlugin is a plugin created by a small international symfony developers team. It's destined to give an easy interface for online video embedding. The plugin uses flowplayer software and uses files with the flv extension.

It's so easy

sfVideoPlugin comes provided with few ways of embedding a flash video player inside you project.

including a partial

Once the plugin:publish-assets task has been run, flv files can be accessed from the web. Just include the video partial from the sfVideo module (enable it first, of course) passing one obligatory parameter - file (it's taken from the default flv directory):

include_partial('sfVideo/video', array('file' => '01.flv'));

Take a look at the plugin demo site.

video widget

Use the sfVideoWidget class inside your forms to display the player inside a form (you may use predefined sfVideoForm class which has only one widget). An example study case using video widget is uploading video files in the backend - user should have the opportunity to display the video file before publishing it online.

multiple player embedding

A simple parameter has been added to the partial mentioned above to enable displaying more than one video widget on one page. If the player parameter is not set when including a partial, the default value is "player" - you need to override the default value to display more than one player. Put the following code into an action:

$this->players = array(
  array('file' => '01.flv', 'player' => 'player01'),
  array('file' => '02.flv', 'player' => 'player02'),
  array('file' => '03.flv', 'player' => 'player03'),
);

and include the partial for each of defined players:

<?php foreach($players as $p): ?>
<?php include_partial('sfVideo/video', $p) ?>
<?php endforeach; ?>

Take a look at a live demo.

configuration

At now, there are four attributes defined in the app.yml file that are common for all players in the plugin. These are:

width
height
autoplay
autobuffering

The first two attributes take an integer value and the rest takes true/false values. Access them using:

sfConfig::get('app_video_autoplay')

contribution

Feel free to comment on the post and to contribute to develop the plugin.

Mail testing configuration in symfony

2010-08-08T23:42:00.000+02:00

By default, the symfony mail delivery strategy is set to none. That means, that calling send method upon sfMailer object will perform no action.

Configuring mail delivery_strategy

All possible delivery strategies are described in the symfony docs. A good solution is to edit factories.yml in all application config directories and set production environment to realtime and development environment to single_address. The developer enters an E-mail address he's got access to, puts it in the delivery_address option and all mails sent from the project will be delivered to that address (so there's no worry that a customer/user will receive test E-mail). Put the following lines into factories.yml file to define mail configuration:

prod:
  mailer:
    param:
      delivery_strategy: realtime
dev:
  mailer:
    param:
      delivery_strategy: single_address
      delivery_address: your@address.com

Once we've set those values (even in the very beginning of a project development), we don't have to carry about mail configuration, hence symfony will carry out all mails to the correct addresses when working in the production environment.

Default form values for new objects

2010-07-25T23:04:00.000+02:00

what for?

Sometimes you may need to get some random values when creating a new object from the symfony generated form. This can happen in lots of situations, some of them are listed below:

populating article comments in a community website - a community website looks a lot better when each article has at least several comments. So when an article is created, few comments need to be created right afterwards. Choosing random names is quite easy, of course, but what if your system allows only logged in users to leave comments (which means, comment table row does not have author string value, but author_id integer foreign key)? In such case you have to look up a random user id in the database, which is pretty impossible. Default form values to the rescue!
E-shop product attributes - each product has some attributes that measure its quality, just like reliability, aesthetics, performance and so on. As we know, many information in advertisement is pure bull*hit nowadays, e.g. values of product attributes are probably fabricated. Suppose there are many attributes for each product (available inside the form or inside an embedded form, it doesn't matter). Defining each attribute costs time and an E-commerce shop employee will work slower if he needs to input each value separately. Again, default form values comes with assistance!

solution

Put the following code into the setup or configure method of a form class:

if ($this->isNew())
{
  $this->setDefault('name', Tools::getRandomName());
}

Above feature is really easy to implement and you'll find it really useful when generating some partially random data. Of course, you may also set random values after the form is submitted, when a form field is simply disabled - but then you don't have a posibility to change it during object creation.

Advanced SQL expressions in Doctrine

2010-07-21T20:25:00.000+02:00

Scene from "Irma la Douce" by Billy Wilder (1963)

missing element in symfony documentation

Symfony and Doctrine book (chapter 6) describes DQL API, but one very important SQL feature is missing: advanced expressions. Particularly, you may often need to use advanced logical expressions in WHERE clause.

example

For example, we run a cron task searching for all active posts which does not meet all SEO requirements - then a warning mail is sent to a particular employee of a company to do something with it. So the query needs to look for Post objects:

Doctrine_Query::create()->from('Post p')

which are active:

->where('p.active = 1')

and have invalid SEO data at the same time, let's assume that SEO is invalid when at least one of all SEO data columns is empty (title, keywords and description):

->orWhere('LENGTH(p.meta_description) = 0')
->orWhere('LENGTH(p.meta_keywords) = 0')
->orWhere('LENGTH(p.meta_title) = 0')

Now take a look at the above code - is that correct? Of course not! We want to generate the following query:

SELECT *
FROM Post p
WHERE p.active = 1
AND (
  LENGTH(p.meta_description) = 0 OR 
  LENGTH(p.meta_keywords) = 0 OR 
  LENGTH(p.meta_title) = 0)

Three SEO alternatives need to be enclosed in parenthesis. But Doctrine Query API does not provide specific methods doing that. Fortunately, we may use standard query methods (where, orWhere, andWhere, etc.). Unfortunately, we can use them only on the top-level of the query (meaning that all we write stays outside parenthesis). So the top-level of query is created by two arguments linked with AND logical operator - each of those arguments use one DQL API where method. But the second argument (invalid SEO) is internally ivided into an alternative of three subarguments, using OR logical operator. The final query looks like:

$objects = Doctrine_Query::create()
  ->from('Post p')
  ->where('p.active = 1')
  ->andWhere(
  'LENGTH(p.link_rewrite) = 0 OR '.
  'LENGTH(p.meta_description) = 0 OR '.
  'LENGTH(p.meta_keywords) = 0 OR ')
  ->execute();

Anyway, it works. Maybe the Doctrine API will be more friendly in Symfony 2.

Using SQL built-in functions in doctrine queries

2010-06-25T00:16:00.000+02:00

This short article describes a small trick that enables using SQL built-in functions inside doctrine queries. The trick is to use ticks inside the query.

example

We want to update all records of a table by setting one of its column to be rounded to 2 decimal points. By default, doctrine will ignore 2 sign which is after the comma inside the ROUND function.

return Doctrine_Query::create()
  ->update('Product')
  ->set('feature_b', 'ROUND(feature_a, 2)')

will generate the following query:

UPDATE product SET feature_b = ROUND(feature_a)

which is NOT what we want. We only need to add ticks surrounding the value to be rounded:

return Doctrine_Query::create()
  ->update('Product')
  ->set('feature_b', 'ROUND(`feature_a`, 2)')

and the generated query will satisfy our needs:

UPDATE product SET feature_b = ROUND(`feature_a`, 2)

Multiple database symfony configuration (doctrine)

2010-05-19T08:11:00.000+02:00

Scene from ingenious "Platoon" by Oliver Stone (1986)

It may happen that we'll need to connect to many databases inside one project. As this problem is not very common, I'll try to present the easiest way to manage multiple database connections.

First of all, edit the databases.yml file and set all database connections with appropriate labels. This means, the doctrine connection label means nothing and therefore it should be changed. Usually, in such case, apart from all external databases, the project itself has its own database which connection to is called master. Below is an example of multiple connection project databases.yml file:

  master:
    class: sfDoctrineDatabase
    param:
      dsn:      mysql:host=localhost;dbname=admin
      username: admin
      password: Mt5iA0Lcm0

  production:
    class: sfDoctrineDatabase
    param:
      dsn:      mysql:host=localhost;dbname=old_shop
      username: shop
      password: Mt5iA0Lcm0

  shop_one:
    class: sfDoctrineDatabase
    param:
      dsn:      mysql:host=localhost;dbname=shop_one
      username: shop
      password: Mt5iA0Lcm0

  shop_two:
    class: sfDoctrineDatabase
    param:
      dsn:      mysql:host=localhost;dbname=shop_two
      username: shop
      password: Mt5iA0Lcm0

This is an example database list of a project that migrates all data from one shop (which is, for example, too old and too bad to be mantained anymore) into two new distinct shops (each shop is deployed using open source software, like magento or prestashop) and one admin database (which is created for the project itself).

Now we need to apply a small change to the schema.yml files:

connection: master

This should be added in the beginning of each schema yml file, defining the connection name. It means, that all models defined in such schema file will be bound to chosen database. Remember to execute the clear cache command, because otherwise you may force difficulties with class files autoload (also when you're using plugins).

The main feature in connecting different classes with distinct databases is to have class-database connection binding set properly. In each base class, you will see

// Connection Component Binding
Doctrine_Manager::getInstance()->bindComponent('Product', 'master');

This line of code states, that all Product objects will be fetched from master database connection.

the structure to be chosen

From this moment, the model classes generated by the framework will include classes from all databases. And now here comes the main question to answer - how complex is my project and do I have to follow all symfony default mechanisms? Answer to this question is essential, because many implementation solutions will depend on the decision made. I'll briefly describe two alternative approaches:

Stick to the defaults. Suppose, you'll have about one hundred tables in each database. If we have 4 connections set, it gives 400 distinct model classes (to be precise: 400 base model classes, 400 model classes and 400 table classes, which is 1200).They all should have connection bindings set correctly, anyway, searching through 400 classes in a tree-view may be frustrating - this is the main disadvantage. Another one is that you'll have to generate all classes even if you won't need them (e.g. if you use open source software, you may need only some of them for your project). And one more disdvantage - you'll also have to keep all form and filter classes for all tables even if they're not needed. But the advantage is that if you modify anything in schema.yml file, base classes will be updated automatically and you don't have to care about that.
Choose your own lib directory structure. The task now is to separate all generate classes into different directories so that everything will be in order. The basic solution is to organise one directory for each database connection. The good things about this solution are: everything is more structured, easier to find; you don't have to keep all classes (if you need only few). And you don't have to keep all model/form/filter classes that doctrine generates - you choose them manually. The disadvantage is that all modifications made in schema files have to be updated manually (replace old version of your base model class in a special directory with a just generated new base model class). Subversion (or other versioning systems) helps greatly to keep everything up to date and user-friendly. After running symfony doctrine:generate-model you update the files you want (which means, destination base class files are replaced with their new versions) - then you commit your changes and afterwards clean it (remove model directory and update it from repository, so that all files you changed are saved in the repository and all the rest - which you don't want - is deleted).

Although the second option may seem too complicated or weird (or even both at the same time), I definitely prefer it. One of its great advantages I still haven't to mentioned is that it if you use your own lib classes structure, it allows you to do advanced and highly complex solutions (example: you've got few shops powered by the same OS software. And you need one tool to manage them all - an administration application. If you want to follow symfony default mechanisms, you have a very big problem, since table names will surely be the same in different shops. And connection binding is really basic in doctrine, since such problem is really rare. Even if you use distinct prefixes to set table names for distinct shop databases, you'll still force a big problem, since you'll need to copy-paste all methods between analogical classes, let's say, you define getFullName method in Product class - you'll have to copy this method into XxxProduct, YyyProduct, ZzzProduct and so on, if you want to stick to symfony defaults. In my opinion it's enough to prove, that symfony defaults are good as long as the project is easy, but if it's really advanced, non-standard solutions are usually the only way). Sticking to symfony defaults and being ready to regenerate model at the same time disables some higly complex features.

look out for the bug!

Unfortunately, there's a bug in symfony core, dealing with multiple database connections. It has been reported some time ago on trac, however, the bug is still there. In short, in some cases DoctrineManager takes the last connection from the databases.yml list, instead of the one you choose (you can read the details on the ticket page). To avoid problems with it, I pay attention to have all base model classes defined with the correct connection binding - and this is enough to get all project working.

As you can see, you're already ready to go - your site can handle multiple database connections!

Symfony world

excellent modelling tool: cacoo

predefined objects to use

try the demo version

they didn't found me - I found them :)

svn blame: who made the bug?

blame me if you can find me quickly!

symfony range filter

new widget/validator classes

accustoming solution to a example project

result

symfony forward through LAN

SVN repository auto update

troubleshooting

force to log message SVN python

tried to do this in PHP...

...and finally moved to python

svnsync tutorial

extending doctrine admin module: filtered sum

action

templates

other functions

symfony basics: form default values

default values for new objects

moreover

symfony dynamic max_per_page

max_per_page

templates

controller/action

configuration generator

doctrine act as signable plugin - new releases

what's new

moreover

the community

symfony custom configuration files

dynamic and cross-application configuration

how to do that

sfAdminDashPlugin trick

Doctrine Event Listeners vs symfony fixtures

introduction

event listeners

fixtures

the problem

the solution

data migration with symfony & doctrine

Diving into 'outsourcing applications with symfony': migrating data

new version of a project, written from scratch

data migration

pure Doctrine migration

pure SQL migration

mixed migration: Doctrine IN / SQL OUT

and that's it

faster queries: indexing tables

built-in foreign key management

optimising MySQL queries

example - social website homepage problem

NOTE: always use test data when focusing on project performance

Note: database server different versions use different indexes

how many indexes to create

further reading: this, this and this

symfony validate database content with sfValidatorCallback

further usage

less doctrine queries in symfony admin modules

basic stuff

the problem

the solution

The result

documenting database with symfony doctrine

outsourcing applications with symfony

outsourcing applications

overall

model for both systems

data migration

easy schema/model modification tool

remove old - script

model reload - script

symfony sfAdminDashPlugin icons

desktop environment

external resources

simple mailing system with symfony - part I