symfony custom configuration files

dynamic and cross-application configuration

If your project gets very big and it has several applications, you may want to create cross-application configuration - store it in one place and let all applications use it (no matter how big it is). Or simply if your configuration is becoming really big and you want to arrange it somehow, custom config files is exactly what you are looking for!

how to do that

That's only three easy steps. First, let's create a custom configuration YAML file (config/something.yml, in the project main directory, not the app config directory) and put some data there:

prod:
  test:  ok

dev:
  test: ko

all:
  foo:  bar
  john: doe

The second step is to create/update a config handler YAML file (config/config_handlers.yml) with following example content:

config/something.yml:
  class:    sfDefineEnvironmentConfigHandler
  param:
    prefix: something_

The last step is to register the configuration file in all applications you want (after this step, the configuration will be available for an application). Use the application configuration class (e.g. apps/frontend/config/frontendConfiguration.class.php):

public function configure()
{
  ....
  require_once($this->getConfigCache()->checkConfig('config/something.yml'));
}

Now all config options defined in the new file are available in the application with the _something prefix. The best way to check if the new file is available for the application is to run it in the dev mode in the browser and check the web debug toolbar, config icon, settings section. All configuration available for the application is displayed there. If you followed all three steps above, new file configuration should be available.

sfAdminDashPlugin trick

This plugin provides an easy-to use menu divided into sections, which is configurable in YAML files. If you want to have different menus for different applications, you just have to put all configuration in the application config (by default, it's stored in the plugin config). One way is to put all menu config in the app.yml file, but, as mentioned before, it can make the app.yml file grow to an enormous size, so a better solution is to create a menu.yml file, which is loaded by the config handler. Everything is done the same way as before:

• create apps/APP/config/menu.yml file and define sfAdminDash menu there
• create or update (if exists) the config handler file (apps/APP/config/config_handlers.yml)
• finally, update the APP configuration class (apps/APP/config/APPConfiguration.class.php) by registering new config file

I hope some of you will find this flexible configuration useful :).

Doctrine Event Listeners vs symfony fixtures

introduction

This is just an easy tip on how to use symfony fixtures along with doctrine event listeners in a symfony project.

event listeners

Doctrine Event Listeners are the methods you define to trigger an activity always after/before a Doctrine Record is created, updated, deleted, etc. These can be: postInsert, preInsert, postDelete, preDelete and so on. They can be very useful to keep the code clean and logical. A good example of usage is uploading an image and creating many different files with different sizes, representing the same image (custom thumbnails). This could be done with overriding ImageForm upload mechanisms, but postInsert seems more clean and easier to maintain if uploading is done using more than one symfony form class.

fixtures

Fixtures are a built-in symfony feature which enables you to easily populate the database with some data. It is usually test data, but for some projects, some initial data can be held in fixtures (e.g. initial configuration values). They should be used in all projects, since it's a lot easier to spot any bug using data that pretends to be real.

the problem

The problem occurs when you combine those two features. Imagine you've got two models in your projects, A and B, related 1:1 (e.g. sf_guard_user with sf_guard_user_profile from sfDoctrineGuardPlugin). For each A record there has to exist the B record. This means, when the mother record is created (suppose A is the master model), the B record needs to be immediately created and related to A. A Doctrine Event Listener is what should be used - a A::postInsert() that creates the B record.

So far, so good - but we still want to use symfony fixtures. Let's say, we want to define 10 sample A records along 10 sample B records. So we write a fixture file with 10 A and 10 B fixtures. Next, we fire the

./symfony doctrine:build --all --and-load

command and what do we find? B records are doubled! And this can be quite difficult to spot!

Each time when a A fixture is saved, the postInsert is automatically executed, therefore new B record is created. So instead of creating records in the following order: A,A,A,...(10),B,B,B,...(10), we get A,B,A,B,A,B,...(10),B,B,B,...(10), having 10 A records and 20 B records. And this is not what we wanted. Fortunately, this can be quite easily fixed.

the solution

Just add one line to the Doctrine Event Listener:

public function postInsert($event)
{
  if ('cli' != php_sapi_name())
  {
    // some code here
  }
}

We're basing on the fact, that fixtures are loaded from a task, which uses command line interface (cli). Now, no fixtures use Doctrine Event Listeners.

data migration with symfony & doctrine

Scene from "Apocalypse Now" by Francis Ford Coppola (1979)

Diving into 'outsourcing applications with symfony': migrating data

This post was inspired by Fizyk's comment on the outsourcing applications with symfony article. Fizyk suggested that it would be a good idea to wrap all data migration functionalities into a symfony task(s). Basically, migrating data between different versions of the same project is the topic I want to focus on in this article.

new version of a project, written from scratch

In the previous article, we've been discussing developing a new version of an old project from scratch (e.g. because the old one was so inflexible that it was unable to be extended). The first thing we should do is to design new database structure that will hold all data that the present project holds (I assume that the database structure will have to be different, because if it was good, no one would redevelop entire project from scratch). Depending on your luck, it can be an easy task or a nightmare [I've already had redeveloped 2 projects with databases that were not even in the first normal form... yes, there are projects like this. Check this out: one of these projects were made by a really bad developer who never used any type of version control system. He used no development environment (only production) and no localhost machines to test new features before they are added. He made all modifications on the production server, even if it took 2 weeks until they were finished. And there was only FTP access on the server... Oh, yes, there ae projects like that... and you may be asked to improve such projects :)].

data migration

Now when we have two parallel database structures, we need to copy all data from the old one to the new one. Depending on the data (its size, structure, relations, constraints, etc.), I use one of the following:

• pure Doctrine (Doctrine in - Doctrine out)
• mixed (Doctrine in - SQL out)
• pure SQL (SQL query)

For each data structure, I choose one of the above migration types very carefully (detailed descriptions below). If the project is big (and I assume this is the case), the development phase takes a lot of time. It's very important that you can run entire migration job:

• as fast as possible: this enables you to run the migration very often, because it doesn't take too much time. And this gives you the possibility to improve the scripts, test them carefully (each mistake made during the migration will be painful when new project is started in production environment). And, finally, you can download the production (old project) database and migrate it to the new structure to compare the frontend of both projects - this helps to develop the business logic,
• with wrapped interface: click once - make yourself a cup of tea - come back in 10 minutes - click again - go out with your dog - come back in 15 minutes - the migration is done. Or otherwise: imagine you have a project with 50 tables and you have to run 50 actions (e.g. by a click), each one for each table - or call 50 symfony tasks, not to mention that you need to call them in appropriate order (because data is related in RDBMS) and you have to watch out not to break the order. Of course, there has to be 50 small jobs, deep inside the migration system, but your interface should do the work for you automatically (check an example below).
• avoid repeating the same manual modifications. This is difficult to discuss in general - I'll give an example. I've had an action table which stored users activity. The old project database had a record which stated that user X has replied to topic Y with post Z. And neither topic Y nor post Z existed in the database (again, if the project is so bad that it has to be redeveloped from scratch, such bad mistakes are very common). Of course, the migration system failed each time when symfony tried to insert a record that was related to a non-existent record. What to do?
  
  • bad solution: remove this action record from you old database local copy (you download the old project database from the production server, load it to your SQL localhost machine and everythng is 100% safe)
  • good solution: remove this record from production database, because it already points some stupid and non-existing stuff.
  Why to do so? Because if such mistakes were made in the old project (data inconsistency), there can be quite a lot of such paniful action table, which will make a single migration job really long. And after you have done the whole migration 2 or 3 times, it'll be enough or dumb work and you won't want to do it any more. Again, this is only a simple example. Each situation will probably need a different solution

There is one more important thing: no matter how complicated your migration job is, always create a detailed list of steps to make, describing what is being done at the moment, why such order and what is already migrated/what is still waiting to be migrated. With such list, performing data migration is really easy and unstressful :)

Here you have an example of a migration job order from a project I did recently (all .sql files are 'pure SQL' migrations):

where the action http://polonia/backend_dev.php/migrator/migrateAll calls the following (these are the only 'pure Doctrine' migrations):

MigrationManager::migrateConfig();
MigrationManager::migrateForbiddenPhrases();
MigrationManager::migrateQuotes();
MigrationManager::migrateArticles();
MigrationManager::migrateReadingCategories();
MigrationManager::migrateReadingTextbooks();
MigrationManager::migrateCountries();
MigrationManager::migrateRegions();
MigrationManager::migrateSubpages();

pure Doctrine migration

This is the only code example in the outsourcing applications with symfony article. And this is probably the only migration type that would be wrapped in a symfony task, as Fizyk suggested. This is really easy - just fetch data from the old system and save it in the new system.

• used for small amounts of data

Below is another example of this data migration technique - private messages sent between users of a social website:

static public function migrateMessages()
  {
    $wiadomosci = Doctrine_Query::create()
      ->from('Wiadomosci w')
      ->orderBy('w.wiad_id')
      ->fetchArray();
 
    foreach ($wiadomosci as $row)
    {
      $message = new Message();
      $message->setId($row['wiad_id']);
      $message->setTitle($row['temat']);
      $message->setText($row['tresc']);
      $message->setFromId($row['wiad_od']);
      $message->setToId($row['wiad_do']);
      $message->setDisplayed($row['przeczytane']);
      $message->setProvoke($row['zaczep'] == 't');
      $message->setCreatedAt($row['kolumna_data']);
      $message->setUpdatedAt($row['kolumna_data']);
      $message->save();
    }
  }

pure SQL migration

You may ask, why native SQL INSERT queries are constructed to be executed immediately, instead of using Doctrine? The answer is obvious: time difference. Doctrine is nice, but in fact, it is really slow. The problem with pure Doctrine migration is that it takes a lot of time to load all data with PHP/Doctrine and then to insert it into new structure. The table from the example above held over 50'000 records. At my local machine, it took several minutes to finish the job. It's too much (data migration should be as fast as possible). The solution to this problem is simply to create a sql script that will do the job thousand times faster:

INSERT INTO NEW_DB.message (
  id,
  title,
  text,
  from_id,
  to_id,
  displayed,
  provoke,
  created_at,
  updated_at
)
SELECT
  wiad_id,
  temat,
  tresc,
  wiad_od,
  wiad_do,
  przeczytane,
  IF(zaczep = 't', 1, 0),
  kolumna_data,
  kolumna_data
FROM OLD_DB.wiadomosci
ORDER BY wiad_id

Sometimes the case may be even worse. The action table, mentioned before, held over 400'000 records and the PHP/apache configuration would need really big limits (and the migration would take several hours to finish). The following is another sql script, migrating action data:

INSERT INTO NEW_DB.action (
  created_at,
  created_by,
  type_id_external,
  target_profile_id,
  forum_post_id,
  forum_topic_id
)
SELECT kolumna_data, u_id,
CASE akcja
  WHEN "profil" THEN 1
  WHEN "logowanie" THEN 2
  WHEN "wylogowanie" THEN 3
  WHEN "pm" THEN 4
  WHEN "zaczep" THEN 5
  WHEN "post" THEN 6
  WHEN "watek" THEN 7
END,
  IF (akcja = 'profil' OR akcja = 'pm' OR akcja = 'zaczep', akcja_id, NULL),
  IF (akcja = 'post', post_id, NULL),
  IF (akcja = 'watek', akcja_id, NULL)
FROM OLD_DB.akcje ORDER BY kolumna_data ASC

Pure SQL migration may be a little bit more complicated (processing data in PHP is easier) - but you'll surely save a lot of time.

• used for huge amounts of data, as this can be the only possibility to migrate it

mixed migration: Doctrine IN / SQL OUT

Sometimes it can be impossible to create a magnificent "INSERT INTO SELECT FROM" query. PHP, although being as slow as a dead snail (in comparison to SQL database management systems), still gives more possibilities just because being an imperative programming language. The idea of this migration type is to fetch the old system data using Doctrine, generate the SQL code and execute it. You may execute such generated script in any way you like - db console, tools like phpmyadmin or even forcing Doctrine to execute raw SQL. If someone prefers, this can be also done as a symfony task - but if the project is really big, you don't benefit from wrapping such mechanism into a symfony task.

To fully understand the idea of this data migration type, take a look at the example below:

static public function getInexistentProfileIds()
  {
    $u_ids = Doctrine_Query::create()
      ->from('Uzytkownicy u')
      ->select('u.u_id')
      ->orderBy('u.u_id')
      ->fetchArray();
 
    $ids = range(1, $u_ids[count($u_ids)-1]['u_id']);
 
    foreach ($u_ids as $u_id)
      unset($ids[$u_id['u_id'] - 1]);
 
    return $ids;
  }

static public function generateSqlToRebuildUsers()
  {
    $ids = self::getInexistentProfileIds();
    $sql = 'INSERT INTO profile (`id`) VALUES ('.implode('), (', $ids).')';
    echo $sql;
    echo '<hr />';
    $sql = 'INSERT INTO sf_guard_user (`id`, `email_address`, `username`,`is_active`) VALUES (';
    foreach ($ids as $id)
      $sql .= $id.',"NULL-'.$id.'","NULL-'.$id.'", 0), (';
    echo $sql;
  }

The reason for doing mixed migration in this case is data inconsistency in the old project. Users were creating forum topics and posts. And when a user was deleted, posts and topics still referred to it. Migrating forum data returned dozens of errors, all of them was failing constraint - assigning topic/post to a non-existent user. The solution was to create dead user records (with deleted_at set), just for the SQL not to break the constraint. The first method fetches all IDs of the users that has been deleted, the second part generates the SQL code creating those dead records into the database. This is only an example, of course, and the example above may be found controversial :), but, believe me, migrating an entire system (which was really, really badly developed) is a difficult task itself, the time you have is limited and sometimes you are forced to use somehow brutal solutions. Anyway, it's up to you how you want to grab the generated SQL code. For me it was the easiest way to display it in the browser, copy-paste to phpmyadmin and execute. But you can generate a sql file, wrap it into a symfony task or do whatever you think of. The main idea is just to make the data migration faster.

and that's it

I hope that some of my hints will be useful if you ever need to migrate data between different versions of the same project. Comments are welcome :)