Storing Data in Automated Business Processes

This is data that identifies the process instance and data being handled in it. I don’t mean the process instance’s technical id, although you might use that at some point. Examples of identifying data are: the customer’s id, the id of the web shop order or the id of a document that’s stored in a repository. You can see the theme here: you don’t store the data itself in the process engine, you store a reference to it. The reference can be used to retrieve the data needed in subsequent steps, because it is stored somewhere else, where it belongs.

You can have multiple pieces of identifying data in a single process instance and together they make the instance unique. Think of an insurance claim that starts online (claim id) for a customer (customer id), gets processed internally which results in an assessment report (document id) and eventually is awarded by an outgoing payment (payment id). The complex data of the claim, the documents, etc. are not stored in the engine, but the identifying characteristics are, so they can be retrieved in the context of the process instance, when the process is running or after it’s been completed.

To perform the orchestration, the process engine needs access to control data that steer the process at decision points or determine which exact execution must take place. Examples of these kinds of data are human decisions (such as "award or reject"), derived data (the customer is a VIP customer) and the channel the process was started from ("mail order or internet order"). The nature of these data is that they influence the flow of the process instance (which makes them required for orchestration).

Control data can typically be stored in booleans or strings (e.g. derived from a enumeration). If they are more complicated, then it’s best to delegate to a service or a business rule engine to manage and provide the decision logic externally. The engine only needs the outcome, so send it a simple type to choose the right path.

Some data are only really needed to be able to orchestrate the process. This typically involves data that results from one step in the process and is used in the next, without being relevant outside the process execution. The subsequent steps have no other way of getting the data other than having it provided to them by the process engine, because the engine is the glue between the steps that has the overview.

An example of orchestration data is a balance total. Let’s say we have business process to add money to your savings account. At the end of the process, we send you an email with the new balance in your account. This balance total seems like something we might store in our application, but in fact we don’t store your balance total at all. We store transactions in and out of your account. The total is always calculated as a result of the whole history of transaction events (which is what we do store, in the application). So after processing the transaction, we invoke the "calculateBalanceTotal" method on some service. The result is a single amount, which is stored in the process engine for that process instance, because it belongs to this process instance only and is stored nowhere else. Once the process gets around to sending the email, the amount is retrieved by that process step.

Sometimes these types blur and a piece of data fulfills multiple roles. In any case, they are always just necessary because the process engine wouldn’t be able to orchestrate the flow without them.

In general, applications or services should not depend on what’s stored in the process engine or its history. That is an internal storage that you should be able to maintain without impact to the world outside the process engine. The engine might automatically clean up its history and should be able to do so without considering external factors. You don’t want to be hunted down in the car park because a business user couldn’t generate a monthly report that was based on data that was just deleted.

If anybody or any system needs the data during the process, it should be stored outside the process engine. Similarly, if the data has relevance after the process instance has completed, it should be stored outside the process engine. Your engine is not a view on your application’s state, but on the process' state.

If you don’t follow these guidelines, your data and the process become intertwined and it becomes cumbersome to change things because of dependencies resulting in higher impact. Your process doesn’t need to know about all the things that are being stored, it should only know about things that relate to the orchestration.

Since Activiti stores any Serializable object, you might have guessed that the way that works is by simply serializing the object and storing the byte array; you would be right. If you consider that many business processes are long-running and they may easily run longer than a development cycle, the objects that were stored in existing process instances may have changed class definitions in the next iteration of the software. Combined with the fact that many applications are developed with almost no regard for versioning of Serializables, it’s easy to see that your existing object blobs will no longer be able to be deserialized by the JVM. This will result in errors while querying for data, even if you’re not using those particular objects. You’ll be in a big mess before you know it and it won’t be easy to recover those existing instances. Trust me, I’ve been there.

If you are introducing a change to a process' data that makes older cases incompatible with new ones, you want to solve that problem where it arises, in the data or the services that manage it. If you need to start tackling compatibility problems at your process engine, you’re making life harder than need be.

Your processes will be much easier to change if they don’t store and/or rely on complex data structures. If you add a property to an order object, because you now want to track an additional characteristic, nothing changes in the orchestration. Older and newer orders can be just as easily maintained in the existing process, because you only keep a reference to them; what happens to be inside them or what’s used by services has no impact on the flow. If you add a decision option, which is handled the same way as an existing one, your process can remain unchanged.

Once you run into a change that does affect the process definition, it will be much easier to change because the data managed is still simple and most likely the older scenarios are still valid or easily changed.