BOP-4 BAM imperfections

  

Where the BOP-4 product stack generally looks very complete, mature and well designed, I have to admit I am still a little less enthusiastic about the Business Activity Monitoring (BAM) part.

In our latest project we have implemented a quiet extensive BAM implementation. We managed to implement a pretty useable dashboard, that satisfied the project and customer needs. However the BAM functionality did not feel quiet as “ready” as e.g. BPM.

BAM intends to monitor KPI-s that map upon a values used in a BPM. To get a graphical representation of these values in you dashboard a number of steps (and Cordys objects) is required.

1. 
   
   First the BPM values to reflect in BAM have to be explicitly exposed using the “Message filter” tab of the BPM. A selection is to be made of all values in the message map in the BPM.
2. 
   
   Next a Process Monitoring Object (PMO) must be created. This object defines the actual BAM data to collect in the BAM database, and the events when they are collected. It takes the BPM process as the input of the PMO. A selection of exposed data values and possible events must be made, and it is possible to process or enrich the data e.g. by using XPath or web services.
3. 
   
   Business Measure (BM) objects are intended to define the actual information from the PMO-s in the BAM database, to reflect in the report. A Business Measure is actually a SQL query that results in a aggregated measurements (averages, totals, counts) reported over a specific “dimension” (e.g. a Location, Person or Unit). This dimension ends up in a GROUP BY clause and it is possible to use a number of parameters in the WHERE clause. A Business Measure will result in a UI dashboard component and a related web service to get the specific data out of the BAM database. The web service can be used autonomously as well.
4. 
   
   The BAM dashboard is effectively an XForm, to which one or more BM -graphical components are added. There they can be linked to each other or to standard BAM components like “Time Selectors”. This way the output from one component  can be used in a filter of another. This way some kind of down drilling can be implemented. For a little advanced functionality scripting is required.

Though the functionality described above behaves correctly, there are a number of inconveniences and hence potential enhancements. I will name a number that come to mind:

1. 
   
   The number of fields to use in a PMO is limited. A PMO just offers a limited amount of fields for each data type (15x String, 10x Numeric, 10x decimal and 10x Date-Time). Exceeding one of these amounts results in an error message.
2. 
   
   Unlike other Cordys objects a PMO object must be manually unpublished before a new version can be published. There is no proper warning or error system available to prevent mistakes.  
3. 
   
   The possibilities to manipulate the looks of the dashboard graphs are very limited.
4. 
   
   Not types of graphical components are standard supported by BAM (like the gauge component). It requires intensive scripting.
5. 
   
   It is cumbersome to implement drill down features. Relations between data is now specified on graph level, and the business measures should already be prepared for the specific need. It would be more convenient to define these relations on data level.
6. 
   
   BAM does not support runtime packages; only staging packages
7. 
   
   BAM does not offer any standard reporting options (unlike Cordys Process Factory – CPF)
8. 
   
   In an ideal world PMO-s and BM shouldn’t be required. It would be nice to be able to specify data to expose on BPM level, and the relations too. This should be sufficient to define and populate a BAM database. When designing the dashboard standard graphical components should allow to select any data that should be displayed.  

At the recent Cordial event I was able to ask the development team after expected developments of BAM. They confirmed these imperfections and announced a number of enhancements that will be implemented with FP1 (Optimistic planning: end of 2010).

It is not hard to imagine I am looking forward very much to FP1, which I expect to solve a number of this issues!

Event-state modeling in Cordys

 

Case

Recently we have been working on a solution that can be considered as a message broker / hub:

It’s functionality is mainly to receive, validate and route messages between parties in a specific business segment over different channels.

 

All messages have a specific type, syntax and semantics. Messages may be interrelated; e.g. one message may be a response to another message. A business wide protocol-standard describes the complete interrelation between message types, and specifies the possible sequence of messages in a flow: A so called "conversation". In a conversation any message of a specific type can be only followed by a message of one or more other specific types.

In case we want to monitor conversations we need to model the messaging protocol. This protocol can be considered as an event-state diagram: The last received (and expected!) message in the message sequence reflects the state, and the reception of a new message reflects an event.

 

Multiple concurrent conversations may be active at a time. This requires a correlation id to link the reception of a new message (see  the previous blog “Events in Cordys”) to an active conversation (or to start a new one, if there is no flow active for that particular correlation id).

 

Implementation

What does it take to implement such an event-state diagram in Cordys BOP-4?

 

The core of the solution consists of three processes, a decision table and a database-table (with access web services):

• Monitor-process: A process that describes the state model
• HandleMessage-process: A process that is started on new receives new messages and that triggers events.
• WaitForEvent-process: A process that actually waits for an event to be triggered by HandleMessage, where the message is passed
• StateTransitions-decision table: A decision table containing all valid state transions that can be triggered by a message. 
• InstanceCorrelation- database table: A database table for storage of the instance id and correlation id

 

Monitor-process

Monitor is the main process. It describes the overal state model. At runtimes each instance reflects a conversation and it effectively keeps the current state of the “conversation”. A Correlation Id is required to correlate the messages to this conversation. The Monitor process simply consists of a number of states that are interrelated: From a particular state an occuring event may cause a transition to one of one or more possible new states. A state is in fact nothing more than a sub-process call to WaitForEvent. WaitForEvent waits for an event and after a correct (expected) message is received it returns the resulting (new) state to the monitoring process. Only in case a state may have more than 1 possible resulting states the monitoring process should use a decision to effectuate the correct transition. All other logic is handles in the WaitForEvent (by invoking the StateTransitions-decision table)

 

Because the Monitor-process is started with the first message of a new conversation it has an input message, having the same format as the messages received by WaitForEvent.

 

     Example of a monitor process

 

WaitForEvent-process

Monitor specifies two main parameters to the WaitForEvent-sub process: The current state of Monitor and the Correlation Id that belongs to the conversation.

The first step in the WaitForEvent is to store its Instance Id in a “InstanceCorrelation” table, together with the Correlation Id before invoking the “Receive Message” step. (See also my previous blog “Events in Cordys”).

Once an event has been triggeredby the HandleMessage-process the WaitForEvent event checks whether it is an event that it expected, depending on the current state. The logic for this check is implemented in the decision table "StateTransitions", that I will explain below.

 

If a message is received that is not an expected successor of the current state, an exception occurs that ought to be handled. (In our case we simply log an “Unexpected Message” error). After this the process waits for a next event to be triggered.

If the message that is received is expected however, the decision table returns the “new state”, which is returned to the monitoring process. Before returning to the monitor however first the InstanceCorrelation table is cleaned up by removing the record that was inserted at the beginning of the process.

 

    "WaitForEvent"-process  

 

HandleMessage-process

HandleMessage is triggered at reception of a new message. It first uses the correlation id in the message to look up a corresponding Instance Id of a waiting process in the InstanceCorrelation table. If no such record is found, we use the decision table to check if the event is an start-event for a new conversation. If so, a new process instance of Monitor is started -via the invokation of a generated Web Service on Monitor-  and passing the message.

In case the Instance Id is found, a notification to WaitForEvent is sent using ExecuteProcess, passing Instance Id and the message (as explained in my previous blog “Events in Cordys”).

 

    "HandleMessage"-process

 

StateTransitions-decision table

The StateTransitions decision table contains all valid "current"state - message combinations, and the resulting new state. (Similar logic may also be implemented e.g. in a matrix, implemented in a database table, see also my blog "logic in Cordys").

 

Note that actually all logic on state transitions is effectively implemented in this decision table, and not in the Monitor process: The Monitor process is only intended to keep the current state of a conversation: Not to execute any transition logic, apart from navigating to the correct new state based on the out-come of the StateTransitions decision table!

 

    "StateTransitions"-decission table

 

Later I will zoom in on the reporting over the monitoring process.

 

 

Regards,

 

Harald van der Weel,

e-mail: haraldvanderweel@gmail.com

Events in Cordys

  

In certain occassions BPM processes need to able to process events.

 

Consider the example of a sales process (e.g. CloseDeal) that sends a quote to a customer and waits for the response.

Handling client responses may be the task of a separate process (e.g. ReceiveQuotationResponse) which listens to a specific channel (e.g. SMTP) for customer messages. Once a response (e.g. a reply on a quote) is received it simply should be passed on to the waiting process (CloseDeal), to allow it to process the response and continue.

 

The good news is that BPMN provides an event mechanism via  “Send Message” and “Receive Message”  steps. Though implementation of a case similar to the one above in Cordys BOP-4 (CU9) is a bit more complicated than one would expect. The issue is that Cordys supports only the Send Message step for child-processes to trigger an event in a parent process.

In our case this is not sufficient, as ReceiveQuotationResponse is not an child but an independent process.

 

Fortunately Cordys offers a work-around:

It allows to start new and to continue existing (waiting) process instances by using an “ExecuteProcess” webservice.

 

How does it work?

The process to be notified of the event (CloseDeal in our example case) should include a “Receive Message” step at the position where the notification event is expectewd (e.g. just after sending the quotation).

In the properties of the “Receive Message” step the input message must be set. This is the (Process Specific defined) message that is expected with the notification.  

 

The notifying process (ReceiveQuotationResponse ) now may invoke the ExecuteProcess web service (found in BPM runtime references “Method Set Process Execution”) to send a notification to the waiting process.

 

This invocation has 3 mandatory input parameters to be set:

• The field “type” must contain (the literal) “instance”
• The field “message” must contain a correct (and complete) message instance of the type that is expected by the waiting process (and hence is having the same definition as the one in the “Receive Message).
• The field “receiver” should contain the correct instance id of the waiting process. The instance id can be obtained from the instance properties (instance:identifier) of the waiting process.

 

Passing the instance Id

The question is: How can the notifying process (ReceiveQuotationResponse) obtain the instance id of the process to be notified?

 

The solution I developed requires a correlation id. The correlation id is an identifying field that is available in both processes and equal in instances of of these processes that are related in this context. In case of the sales process the Quotation Id may be used for instance.

The waiting process stores the instance id together with the correlation id in a database table. The notifying process may use the correlation id (e.g. received in the customer message) to retrieve the instance id, and use pass this with the ExecuteProcess invocation.

 

In concrete the solution requires:

-          A database table, consisting of (at least) two fields: CorrelationId (primary key) and InstanceId (and an access web services)

-          Two steps (at least) in de the process that should receive the notification:

o   The step is to store the instance id with in de database together with the correlation id

o   The second is a “Receive Message” step

-          Two steps (at least) in the notifying process

o   The first to read the instance id based on the correlation id

o   The second to invoke the ExecuteProcess

 

Note that the approach may require additional processing like cleaning up the database table and error checking.

 

A future enhancement in Cordys to the Send en Receive message steps that allows to specify a correlation id would be a great improvement.

 

Implementing BPM logic in Cordys-BOP4

Introduction

Almost any Business Process has a conditional flow, which is determined by business logic. An example is to determine the approver for a purchase: In case the order amount does not exceed 50 dollars approval may not be required. However for amounts between 50 and 5000 euro the department-manager may need to approve. Above 5000 euros it may be the business line manager, and above 100000 euros even the CEO.

 

Cordys BOP-4 allows us to define Business Process logic in multiple ways. This flexibility is nice. However it raises the question: "When to use what"?

 

Considerations

There are several considerations to make before selecting one of the available alternatives:

 

Agility:

How frequent are the expected modifications? Should it be possible to perform them without restarting or redeploying the application?

 

Efficiency:

How much effort does it take to implement the rule?

 

Complexity/Maintanability:

How complex are the rules: How many steps and different conditions do they contain? Who should be able to perform the change: A business user or an IT engineer.

 

Functionality:

What functions and data do you need to be able to access? Do you need to be able to access database content? Perform difficult calculations or text operations?

 

Readability:

Should business be able to read and understand the logic to be able to grasp the business process.

 

Architecture:

It may be favorable to have your logic itself logically organized, for the following reasons so it is clear for business users and developers where to find certain logic, and to be able to package and deploy logic as much as possible in comprehensible units.

 

 

Lets first discuss the options:

 

Constructs to implement logic 

BOP-4 offers the following constructs to implement business logic:

(Note that most of the alternatives above may be used in conjunction with other alternatives.)

 

BPM Decision steps

The BPM modeling language BPMN allows to directly built logic into Business Process models using the construct "Decision". This step allows to select multiple alternative paths that may be chosen, based on a condition for each path. There may be one default condition that is executed when all other condition evaluate to false. The conditions are defined in "XPath".

 

+ The logic is visible in the process itself, which contributes to readability and architecture

- Complex logic may explode into an incomprehensible process model

- Changes should pass a complete DTAP cycle

- Advanced functionality will require knowledge of XPath

 

Recommended usage

 - Simple conditions in Business Process, e.g.:

 - Checking process inputs

 - Checking output from earlier steps

 

  

Example of  decision step

 

BPM Mappings

BPM mappings allows the assignment of values in steps in a BPM using Xpath. Xpath contains a number of functions that may be used to implement logic (e.g. Boolean functions, regular expressions and selection expressions).

 

+ The logic is stored in the BPM model. (Architecture)

- Changes should pass a complete DTAP cycle

- Logic is "hidden" in implementation details

- Limited functionality. More advanced functions require knowledge of XPath

- Logic is less visible / comprehensible to business users

 

Usage

 - Simple logic to influence attribute values

 

Example of mapping Xpath 

 

Transformation objects

Transformation object transform one data structure into another, using XSLT (which uses XPath underneath). Transformations may be used in BPM or turned into web services.

 

+ Supports complex logic

+ Reusable

- The logic is stored outside the BPM model. (Architecture)

- Changes should pass a complete DTAP cycle

- More complex logic requires XPath/XSL knowledge. 

- Logic is not visible / comprehensible to business users

 

Usage

 - More sophisticated logic used to transform a (business) object into another (business) object

 

 

Decision Tables

Decision Tables are a number of rules, containing one or more conditions and one or more (resulting) actions. The conditions use XPath-statements. These use the specified data interface, which is a predefined data structure. The actions concern assigning values to the data structure, or e.g. running a web service.

 

+ Supports complex logic

+ Reusable

+ The logic is stored in Business comprehensible (=maintainable?) objects.

+- Changes may be delivered in a staging packages, which allows online maintenance by business users. *)

- Best suitable for compact rules: More complexity or functionality may require XPath knowledge

 

Usage

 - More complex IF.. THEN conditions on the the same attributes

 - From 2 to 10+ different conditions

 - Rules that check values (e.g. amounts) against static numbers, that may be changed frequently (by business users?)

 

*) For advantages/disadvantages of staging packages, see:

https://wiki.cordys.com/pages/viewpage.action?pageId=164794477

 

Example of decision table 

 

Business Rules 

Business Rules are technically similar to decision tables, but are attached to events on business objects. (e.g. Insert in the database).

 

Usage

- Validations

- Triggering actions, comparable to stored procedures.

- Triggering BPM-s on specific events

 

 

 

Database queries

Logic may be implemented by executing SQL on the database e.g. via look-up tables, in order to retrieve specific values for specific objects. In order to use the queries with BPM they will have to exposed via a web service.

 

+ (Static) logic may be altered/extended during run-time. Screens may be offered to Business to alter the look-up tables.

+ May support large amounts of data. (E.g. customer bases)

- Logic is divided over 2 objects: data and query

- Logic is separated from business process.

 

Usage

 - Look-up actions of larger amounts of data

 - Logic that the user should be able to alter implicitly at run-time by changing the database content

  

Java

Java allows to write logic in java classes. In order to use it with BPM the java classes will have to be exposed via a WebService.

 

+ Maximum functionality offered. Suitable for complex rules

- Java requires more objects and more complex settings for deployment

- Changes should pass a complete DTAP cycle

- Require knowledge of Java

- Logic is not visible to business user

 

Usage

 - Complex manipulations of numeric, date or string values.

 - Complex logical structures (combined loops, nested IF..THEN-s, switch statements etc)

 

XForm/java script

Java script may be used in User Interfaces to react on user events and take appropriate actions. This may include validations or UI actions, but also invoking Web Services (which may include business processes and business rules).

 

 

Example of Java Script with XForms 

 

+ Directly usable in User Screens

- Not visible nor comprehensible for business users

- Requires java-script skills

- Executed in client, which may cause issues due to different browser types

 

 

Usage

 - Triggering actions from a User interface (e.g. starting a BPM by a press on a button)  

 - (Cross) field Validations

 

 

It is crucial to select the right option in order to meet the non-functional requirements as stated earlier in this blog.

Regards,

 

Harald van der Weel