In this blog entry I want to give you an impression of an alternative solution by agimatec, that is slightly different. It is the solution we are using in our products for more than two years and with which we are satisfied. It would be interesting to hear your options on this.

The concept

Database:

We generate history tables for each table for which we need auditing. We generate database triggers (oracle or postgres) to automatically insert data in the history tables when entities are stored/deleted.

We do not store duplicate data, which means that our history tables contain only OLD data. All current data is ONLY stored in the application’s tables (usually mapped by hibernate/Ejb3).

Configuration:

We have a XML configuration, that contains the table names for which we want to enable our history solution. This configuration lets you specify:

• name of history table (or a default name will be used)
• which columns to exclude from history (default is that all columns are historised)
• optionally you can turn off history for insert and/or update, change trigger names etc.

The XML configuration controls the tables and triggers generated by freemarker templates.

Features:

It is not only important to version the changes, but also to store, the timestamp of a change and some context information about the change (e.g. who is the actor, which application, from which sessionID etc.). Any context information can be associated with a version record by storing a contextID in the history table’s rows.

This is also a way to separate changes from user-specific data, which might be a judical requirement. An extended Hibernate event handler stores the contextID into the database session by calling a stored procedure, so that the history triggers and access the contextID to store it together with the history data.

Each history table has additional columns:

• HIST_TIME (timestamp of change)
• HIST_CONTEXTID (ID of context providing additional information)
• HIST_TYPE (to distinguish INSERT, UPDATE and DELETION)

A simple java API exists to query historical data. It returns the same classes as you use for your hibernate entities, but it materializes the objects itself. Hibernate does not know anything about the history tables or mappings.

The benefits

There are some differences between the solution provided by Envers and agimatec:

• Envers needs a global _revision table, agimatec’s history rows are identified by the primary key and a version number (@Version) incremented from 1 for each entity/no global _revision
• You can change data with any SQL tool.Historisation does not rely on hibernate as the only API to change the database.
• Whenever you increment a row’s version, the history triggers will automatically write historical data. By this way, you can use migration scripts that can control whether history data will be stored.
• Envers stores current data in history tables, which makes it a bit easier to query the data, but with means redundancy (and potential data inconsistencies).
• Both framework are integrated with Hibernate by extending the Hibernate Events (to provide context information).

Examples

History configuration

<historyConfig>
<tableConfig>
<tableName>booking</tableName>
</tableConfig>
<tableConfig>
<tableName>customer</tableName>
<historyTable>h_cust</historyTable>
<insertTrigger>TR_I_cust</insertTrigger>
<updateTrigger>TR_U_cust</updateTrigger>
</tableConfig>


Generated tables

CREATE TABLE H_JOURNAL (
ID INTEGER NOT NULL,
HIST_TABLE VARCHAR(50) NOT NULL,
HIST_TIME TIMESTAMP NOT NULL,
HIST_CONTEXTID VARCHAR(40),
CONSTRAINT H_JOURNAL_PK PRIMARY KEY (ID, HIST_TABLE)
);

CREATE TABLE H_booking (
VERSION INTEGER NOT NULL,
booking_id INTEGER NOT NULL,
price NUMBER,
HIST_TIME TIMESTAMP NOT NULL,
HIST_CONTEXTID VARCHAR(40),
HIST_TYPE CHAR(1) NOT NULL,
CONSTRAINT H_booking_PK PRIMARY KEY (VERSION, booking_id)
);
CREATE TABLE H_cust (
VERSION INTEGER NOT NULL,
cust_id INTEGER NOT NULL,
first_name VARCHAR2(40),
last_name VARCHAR2(40),
HIST_TIME TIMESTAMP NOT NULL,
HIST_CONTEXTID VARCHAR(40),
HIST_TYPE CHAR(1) NOT NULL,
CONSTRAINT H_cust_PK PRIMARY KEY (VERSION, card_id)
);

Generated triggers

CREATE OR REPLACE TRIGGER TR_I_cust
AFTER INSERT ON customer
REFERENCING NEW AS NEW
FOR EACH ROW
BEGIN
setTATime();
INSERT INTO H_JOURNAL (ID, HIST_TIME, HIST_CONTEXTID, HIST_TABLE)
select :NEW.card_id, h.ts, h_session.getContextId, 'customer' from h_tatime h;
END;
/

CREATE OR REPLACE TRIGGER TR_U_cust
AFTER DELETE OR UPDATE
ON customer
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
BEGIN
IF(DELETING) THEN
setTATime();
INSERT INTO H_cust (
cust_id,
version,
first_name,
last_name,
HIST_TIME, HIST_CONTEXTID, HIST_TYPE)
select
:OLD.cust_id,
:OLD.version,
:OLD.first_name,
:OLD.last_name,
h.ts, h_session.getContextId, 'D' from h_tatime h;
ELSIF (UPDATING AND :OLD.VERSION != :NEW.VERSION) THEN
setTATime();
INSERT INTO H_cust (
cust_id,
version,
first_name,
last_name,
HIST_TIME, HIST_CONTEXTID, HIST_TYPE)
select
:OLD.cust_id,
:OLD.version,
:OLD.first_name,
:OLD.last_name,
h.ts, h_session.getContextId, 'U' from h_tatime h;
END IF;
END;
/

Java API

The implementation of class HibernateHistoryReader was rather complex, because of the need to resolve the relationships through the current tables and the history tables. All changes done inside a single transaction are joined together by using a timestamp in HIST_TIME that is unique inside the running database transaction.

Here is interface HistoryReader:

interface HistoryReader {

void open(EntityManager entityManager);
void close();

List<EntityRevision> getRevisions(Class entityClass, Object primaryKey, Timeframe timeframe);
EntityRevision getRevision(Class entityClass, Object primaryKey, int version);

Map<Object, List<EntityRevision>> getChildrenRevisions(Class parentEntityClass,EntityRevision parentRevision,String relationship);

<E> HistoryEntity<E> loadEntity(Class<E> entityClass, Object primaryKey,Timestamp time);
<E> List<HistoryEntity<E>> loadChildren(Class parentEntityClass, String relationship,
Object parentPrimaryKey, Timestamp time);

}

Where class HistoryEntity contains the entity instance for a given point of time and some additional information, such as the contextID and the foreignKey values to load some to-one-references.

If you are interested in more details contact us.

Can you guess the difference between

select * from dual d where  d.dummy <= ‘X’ /* {} */

and

select * from dual d where  d.dummy <= ‘X’

?

Just a comment? -

Well, the surprise is, that the driver throws

java.lang.NullPointerException  at oracle.jdbc.driver.T4C8Oall.getNumRows(T4C8Oall.java:876)

in the first case that contains the comment and executes the statement correctly in the second case.

Thanks, Oracle!

We could call merge(): ["MyEntity" is the name for a business entity, just for the examples.]

Example 1:

MyEntity persistentEntity = entityManager.merge(detachedEntity);

or do it the explicit way, when some more business logic is required before we save:

Example 2:

  MyEntity persistentEntity = entityManager.find(MyEntity.class, detachedEntity.getId());
  boolean isNew = false;
  if(persistentEntity == null) {
    persistentEntity = new MyEntity();
    isNew = true;
  }
  // ... some business logic here that works with detachedEntity and persistentEntity
  if(isNew) {
    entityManager.persist(persistentEntity);
  } // otherwise update happens implicitly

But: This does not solve the concurrency problems we get when two transactions are doing the same thing at (nearly) the same time!

What could happen is, that both transactions try to load the object first where find() returns null, because the object is not stored yet. Thus both transactions will try to insert the entity, which would create the entity twice in the database or would cause one of the transactions to fail (e.g. because of a constraint violation).

You do not need much code to solve this situation, but be careful, there are some pitfalls!

There is even a concurrency problem when the entity is already in the database:

When both transactions change the entity concurrently, this will result in lost updates or an OptimisticConcurrencyException. – I don’t like any of them!

The program must load and lock the entity (select-for-update), so that each subsequent transaction waits on the lock until the first transaction releases the lock (when the transaction ends with commit or rollback).

The simplest thing is to call

entityManager.lock(persistentEntity, LockModeType.READ);

but:

• according to EJB3-specifiation, lock() is only supported for entities with a @version attribute!
  The @version attribute normally is used for optimistic concurrency and I got bad surprises calling lock(): sometimes I got a StaleObjectStateException instead of a locked object!
  (Hibernate also supports lock() for entities without a @version attribute.)

• when the persistentEntity has not been inserted, there is no object to be locked!
  Lock() requires that the object has been saved before.

Let me suggest a solution:

I created an entity named “LockEntry”, that allows me to implement global locks.

@Entity
public class LockEntry {
  private String id;
  private int version;

  @Id public String getId() { return id; }
  public void setId(String id) { this.id = id; }

  @Version public int getVersion() { return version; }
  public void setVersion(int version) { this.version = version; }
}

Each kind of lock gets its own name and is inserted in LockEntry’s table when the database is created.

insert into Lockentry (id, version) values ('MyEntity', 0);

The algorithms of example (1) and (2) just start with two additional statements:

LockEntry entry = entityManager.getReference(LockEntry.class, "MyEntity");
entityManager.lock(entry, LockModeType.READ);

This places a lock on LockEntity with id=”MyEntity” to ensure that the transaction will not cause conflicts or lost updates.

But you must notice one more thing:

When working with different entityManagers (e.g. when using multiple databases or database schemas), the LockEntry entity must be locked with the same entityManager than your business entity (MyEntity)! Otherwise you can get lost updates or concurrency conflicts, because during the commit of the transaction the lock on “LockEntry” must be released in the same database transaction, in which the changes on MyEntity become visible to other transactions. (I am normally not using a two-phase-commit transaction manager.)

Optimization:

To improve throughput of concurrent transactions, you can decide to use the global lock only when you need to create the new entity:

MyEntity persistentEntity = loadAndLock(detachedEntity.getId());
boolean isNew = false;

if(persistentEntity == null) {
  LockEntry entry = entityManager.getReference(LockEntry.class, "MyEntity");
  entityManager.lock(entry, LockModeType.READ);
  persistentEntity = loadAndLock(detachedEntity.getId());
  if(persistentEntity == null) {
    persistentEntity = new MyEntity();
    isNew = true
  }
}

// ...

Note that loadAndLock() is called twice – double checked locking is required here!

The reason for the loadAndLock() method is that because of Hibernate’s implementation of lock(), calling find() and lock() can result in a StaleObjectStateException:

MyEntiy loadAndLock(long id) {
  Query query = entityManager.createNativeQuery("select e.* from myentity e where id=?1 for update");
  query.setParameter(1, id);
  MyEntity found = null;
  try {
    found = query.getSingleResult();
  } catch(EntityNotFoundException e) {} catch(NoResultException e) {}
  // this might be neccessary, too:
  if(found != null) entityManager.refresh(found);
  return found;
}

Weird, isn’t it? Or do you have a better solution?

Anwendungen oder Produkte werden mitunter jahrelang weiterentwickelt, während diverse Versionen auf unterschiedlichen Test- oder Produktivsystemen bereits im Einsatz sind. Solange noch kein Release produktiv eingesetzt wird, sind Änderungen noch problemlos möglich. Sammeln sich in Datenbanken aber erst einmal Kundendaten, so müssen diese auch beim Deployment einer neuen Software erhalten bleiben. Und darum geht es heute: Datenbank-Installation und Schemaänderungen.

Diese Probleme stellen sich in fast allen Projekten, die eine Datenbank einsetzen:

1. Installation der Software incl. Erstellen einer Datenbank mit Initialdaten, die je nach Installationsparametern variieren können – am liebsten automatisch bereits in der Testphase.

2. Upgrade der Software, wobei die Datenbankstruktur (neue Tabellen, Spalten, Refactoring des Schemas) sich ändern muss und die vorhandenen Daten erhalten bleiben.

3. Unterschiedliche Test/Acceptance/Produktivsysteme diverser Kunden, die auf verschiedenen Versionsständen stehen müssen auf eine definierte Version “hoch”-migriert werden. Wer blickt da noch durch?

4. Die bange Frage nach jedem Deployment: Haben wir an der Datenbank nichts vergessen? (Ist der Index XY angelegt? Können wir die Spalte YZ jetzt endlich löschen? Sind wirklich alle Tabellen vorhanden und keine View oder Trigger ungültig durch die Änderungen am Datenbankschema?) Oftmals sind es sehr viele Kleinigkeiten, an die gedacht werden muss!

Muss eine Anwendung gleichzeitig Datenbank verschiedener Hersteller unterstützen, werden diese Probleme noch weiter verschärft. Auch die Anforderung, dass bestimmte Views oder Tabellen aus anderen Tabellen automatisch erzeugt werden können (z.B. Historytabellen), ist nicht ungewöhnlich.

Wir nutzen in unseren Projekte dazu seit Jahren das Tool “dbmigrate“, das wir seit Kurzem als OpenSource freigegeben haben.

Was ist “dbmigrate”?

• Ein OpenSource Framework zur Ausführung von SQL- und Groovy-Scripten. (Enthält auch einen einfachen SQL-Parser.)
• Ein Tool, das mit Hilfe einer XML-Konfiguration das Setup bzw. die Migration von Datenbanken durchführen kann.

Welche Features sind enthalten?

• Erforderliche SQL-Scripte werden anhand einer Namenskonvention automatisch ausgeführt.
• Es können Verarbeitungsaktionen definiert werden, die immer vor oder nach der Migration laufen sollen.
• Ausführung von Groovy-Scripten oder Aufruf beliebiger Java-Klassen ist problemlos möglich.
• Bedingte Ausführung von Scripten oder Bedingungen innerhalb von Scripten erlauben eine Anpassung an die jeweilige Zielplattform.
• Datenimporten aus Textdateien oder mit DBUnit können eingebunden werden.
• Prüfungen auf Vollständigkeit des Schemas und Konsistenz (Triggers, Views, Functions) durchführen.
• Getestet mit Oracle und PostgresSQL, verwendbar für alle JDBC-Datenbanken.

Wie benutzt man “dbmigrate”?

1. Um “dbmigrate” einzusetzen sind Javakenntnisse nicht erforderlich, aber ggf. hilfreich.
2. Es muss daher kein Javaprojekt sein. “dbmigrate” kann für beliebige Datenbanken eingesetzt werden.
3. Um “dbmigrate” zu benutzen, muss eine Java-VM zur Verfügung stehen.
4. Die SQL-Scripte (CREATE TABLE, ALTER TABLE) müssen natürlich selbst geschrieben werden, oder werden mit einem anderen Tool (z.B. einem ER-Modeller oder UML-Tool) erzeugt.

Konkret, bitte!

1. Ein Beispielprojekt runterladen.
2. Zum Erzeugen der Datenbank aufrufen: “dbtool.bat setup.xml”
3. Zum Datenbank-Upgrade aufrufen: “dbtool.bat upgrade.xml”
4. Die Dokumentation anschauen

Wie geht’s weiter?

1. Das Beispielprojekt so abwandeln, dass der eigene Jdbc-Driver und die SQL-Scripte zur Erzeugung der eigenen Datenbank verwendet werden.
2. In Zukunft bei Änderungen am Datenbankschema nur die Setup-Scripte (CREATE TABLE) anpassen.
   Damit kann eine neue Datenbank erzeugt werden.
   Außerdem dienen sie zum Soll-Ist-Vergleich um die Vollständigkeit der Datenbank zu prüfen (für Oracle und Postgres).
3. Die erforderlichen upgrade-Sql-Scripte (ALTER TABLE, UPDATE …) erstellen und unter entsprechendem Dateinamen im Scriptverzeichnis abspeichern. Der Aufruf des Tools führt nur die Scripte aus, die erforderlich sind.
4. Wir freuen uns über Feedback, Fragen und helfen gerne beim ersten eigenen “dbmigrate” Projekt.