Dienstag, 25. März 2014

A general approach to migrating EMF models persisted into SQL based backends/stores

Introduction

With Eclipse EMF you can model your business domain (meta modeling) and persist instances of it into all kind of backends. As time goes by, you tend to add functionality to your application domain which involves changing the model (which is known as 'model evolution'). If you happen to change/evolve your model such that it is not compatible with its previous version, then, you need to prepare for complex migration procedures to transform one version (an old version) of your model into the a newer version.
For Eclipse EMF, there are several frameworks that are capable of persisting models into SQL based backends:
  • Eclipse CDO
  • Eclipse Teneo
  • Eclipse Texo
This article will demo a general approach to migrating EMF models that are persisted via Eclipse Teneo.
Eclipse Teneo is a framework which allows for connecting models via Hibernate to an SQL based database.
This is done internally by translating Eclipse Ecore meta models to Hibernate mappings which are translated themselves to SQL DDL statements.
The key point of the following approach is to grab the generated Hibernate mappings and use another tool that is capable of comparing two mappings (generated at two different points in time) of the same model. This tool is introduced in the next chapter: Liquibase.

Comparing Databases with Liquibase

Liquibase is capable of comparing two different 'Databases' with each other and generating a 'diff script' for execution in SQL backends.
A 'Database' can be a connection to a JDBC/SQL database or it can be some other sort of meta structure that describes a JDBC/SQL database. This is the key point of Liquibase for integrating Ecore meta models into it. As said, with Teneo we can generate mappings of our meta models which Liquibase can use as 'Database' for creating change sets which we can use to apply to another database for model evolution purposes.

A small demo

Just imagine a domain class called Book which looks like this (revision 0 of the meta model):


(This is borrowed from the 'Extended Library Model' available when installing the EMF SDK.)

Now, let us add some additional attributes to the book EClass (revision 1 of the meta model):
The changes include:
  • a new attribute of type float called 'price' (this is no incompatible change, but used to demonstrate some simple migration steps later)
  • changed the multiplicity of author to * and renamed it to 'authors'
Now, we need to compare both meta model revisions with each other and generate the 'diff script' (Liquibase migration script):
  1. Extract revision 0 from your source repository
  2. Extract revision 1 from your source repository
  3. Start Teneo to generate the Hibernate mapping of revision 0
  4. Start Teneo to generate the Hibernate mapping of revision 1
  5. Call Liquibase to compare mapping of revision 0 with mapping of revision 1 and generate the script
  6. Review and adapt the generated migration script
This is a very abstract algorithm, but was implemented for a customer and wrapped up in a JFace wizard for simplification:
The wizard collects all needed parameters from the user/developer and runs the above algoritm. In this case, the source repository was subversion and thanks to the Subversive plugin, no or minimal code was needed to choose and checkout the revisions.

The migration script looks like this (excerpt):



Liquibase handles 'migration steps' by the notion of 'change sets'. In our case, it has generated thre change sets (other change sets are not shown in the screenshot due to simplification of the demo):
  • an 'addColumn' instruction to add the new column 'price' to the table
  • a new junction table ('createTable' instruction) to reflect the many-to-many association of Book.authors and Writer.books (which are both set as eOpposites)
  • a 'dropColumn' instruction to remove the 'old author' column from the table 
In the review process of the generated migration script, we would have to review each instruction and check whether to keep it as-is, for example, the addColumn instruction which does not do any harm to the database, or to provide custom instructions for properly transitioning incompatible changes, for example the dropColumn instruction which leads to loss of information. In short, before executing the dropColumn instruction, we have to provide a new instruction to transform all rows of Book that contain an author value/reference into new rows in the new many-to-many table.
For example:

select i.book_id, i.author_id, 0 into book_authors from item i where i.author not null

(This is pseudo-sql and may have to be adjusted for the target DBMS, but you can tell Liquibase which DBMS is valid for each custom SQL.)

Summary/Conclusion

We have seen a possible approach to migrating Ecore meta model instances persisted into SQL backends using Teneo by leveraging the functionality of Liquibase.

As Liquibase operates on JDBC enabled databases, it can also be applied to the other EMF frameworks that persist models to SQL backends, for example CDO.

Another interesting point is to use EMF Compare (MPatch format) to generate Liquibase conditions to check whether each meta model change has been covered by the migration script.

Links

Liquibase: http://www.liquibase.org/
EMF: http://eclipse.org/modeling/emf/
EMF Teneo: http://wiki.eclipse.org/Teneo
EMF CDO: http://www.eclipse.org/cdo/
EMF Compare: http://eclipse.org/emf/compare