Pierre Richelle

I am Client Technical Specialist in Integration at IBM

7 minute read

Introduction

IBM AppConnect has different ways to store data in memory. Depending of the data life time, you would use either variables or cache.

Event though there are different type of variables that have different lifetime, it is also important to keep in mind that variables can only be accessed from the same flow. If you need to share data across different flows you need to used the global or local cache.

The following section provides the different option to store data. The chapter on Embedded cache provides details on how to configure and access the embedded cache as well as how to monitor it.

Variables & Cache

Here are the different ways to store data in AppConnect according to their lifetime.

Node scoped

“normal” variables that are used within ESQL, Java … These variables only exists in the node where they are defined.

ESQL example:

DECLARE myVar CHARACTER 'aDefaultValue';
SET myVar = 'aValue';

Flow execution scoped

Those variables are defined and updated during the execution of a flow when a message is received. The variable cease to exist when the processing of the message has been performed by the flow. When a new message is received by the flow, the data hold by the variable from the previous execution is lost.

When a message is received by ACE, different logical trees are created and have a flow live time. There are three main message assembly: LocalEnvironment, Root, Environment These message assemblies are propagated from throughout the flow and can be accessed or modified in nodes.

[!NOTE] The LocalEnvironment and Root is reset if an exception is thrown and catch by a previous node
The Environment tree is not impacted.

Interfaces are provided to access those tree.

Flow start time

Those variables are defined at design time with the user defined properties or when a message is processed using SHAREDvariable.

The data stored in the variable can be accessed in the flow and is kept from one message to another message execution.

User defined properties

User defined properties (UDP) needs to be initialized at deployment time, can be changed using the administration REST API (or using the web console), can not be changed. The UDP has a global scope and can be accessed from any nodes in the flow.

This data can be accessed in ESQL by defining EXTERNAL variable.:

DECLARE myUDP EXTERNAL CHARACTER 'aDefaultVale';
SET OutputRoot.JSON.Data.MyUDP =  myUDP;

In java a method is provided: getUserDefinedAttribute(name)

More information can be found UDP documentation.

ESQL SHARED variable

Shared variable can be accessed in ESQL Compute node and data stored in this type of variable will be kept in memory until the flow is stopped.
ESQL example:

DECLARE myGlobalVar SHARED ROW;
SET myGlobalVar.Data.MyVar = InputRoot.JSON.Data.MyData;

Static Java variable

Static java variable can be accessed in Java Compute node and data stored in this type of variable will be kept in memory until the flow is stopped.

Integration Server life time

It is possible to store data in a cache that will be held by the IntegrationServer memory until it is restarted.
There are two type of embedded cache available: a local cache and a global cache.
The local cache is available to all the message flows running into the same Integration Server. The global cache is based on WebSphere eXtreme scale and can be distributed across different Integration Server and therefor be accessible to flow running on different Integration Server.

The global cache can also be stored on an external eXtreme scale server.

Cache can be accessed using mapping node or java code (using provided methods)

Embedded Cache

This section provides information on how to configure and access the embedded cache in ACE.

Configuration

Global Cache

Configuration for one integration server with the global cache Configuration is provided in the doc.

ResourceManagers:
  GlobalCache:
	cacheOn: true # Set to true to enable Global Cache functionality
	cacheServerName: 'MyCatalogServer1' # The name of this cache server component (a cache server component can be a catalog and/or a container); it must be unique in your global cache system
	catalogServiceEndPoints: 'localhost:2800' # Comma-separated list of hostnames and ports for the catalog servers to use, e.g. 'localhost:2800'
	catalogDomainName: 'WMB_MyCacheDomain' # Name of the shared global cache domain; this value should be shared by all catalog servers in the same domain
	catalogClusterEndPoints: 'MyCatalogServer1:localhost:2803:2801'
	enableCatalogService: true # Set to true to launch a catalog service cache server component in this integration server
	enableContainerService: true # Set to true to launch a container service cache server component in this integration server
	enableJMX: true # Allow admin access to this container service via JMX
	listenerHost: 'localhost' # Comma-separated list of hostnames for this cacheServer component, e.g. 'localhost,myserver.mycompany.com'
	listenerPort: 2800 # Port number this cache server listens on; it must be unique on this machine

Local Cache

ResourceManagers:
  GlobalCache:
   #cacheOn: true   
   defaultCacheType: 'local'

Access the Cache using Java

The only difference in term of accessing the local cache instead of global cache in Java is made by the following. Note that this is not required. The global cache code can also be used with the local cache !

Local cache:

MbGlobalMap gmap = MbLocalMapCache.getLocalMap(mapName, new MbGlobalMapSessionPolicy(20), null);

Global Cache:

MbGlobalMap gmap = MbGlobalMap.getGlobalMap(mapName);

Put

		    MbMessage message = assembly.getMessage();
		    // ----------------------------------------------------------
		    // Add user code below
		
		    MbElement jsonData = message.getRootElement().getLastChild().getLastChild();
		  
		    String mapName = jsonData.getFirstElementByPath("mapName").getValueAsString();
		    String mapKey = jsonData.getFirstElementByPath("mapKey").getValueAsString();
		    String mapValue = jsonData.getFirstElementByPath("mapValue").getValueAsString();
		    
			try {
			
				//MbGlobalMap gmap =   MbLocalMapCache.getLocalMap(mapName, new MbGlobalMapSessionPolicy(20), null);
	            MbGlobalMap gmap = MbGlobalMap.getGlobalMap(mapName); // new MbGlobalMapSessionPolicy(20)); 
				if (mapKey != null) {
					if (gmap.containsKey(mapKey)) {
						gmap.update(mapKey, mapValue);
					}else {
						gmap.put(mapKey, mapValue);
					}
				}
				
			} catch (MbException exp) {
				System.out.println("Exception occurs when trying to access the global cache: " + exp.getStackTrace());
				throw exp;
			}

Get

		MbMessage inMessage = inAssembly.getMessage();
		MbMessageAssembly outAssembly = null;
		String mapName = "myMap";
		
		try {
			// create new message as a copy of the input
			MbMessage outMessage = new MbMessage(inMessage);
			outAssembly = new MbMessageAssembly(inAssembly, outMessage);
			// ----------------------------------------------------------
			// Add user code below

			MbElement outRoot = outMessage.getRootElement();
			MbElement outJsonRoot =
					   outRoot.createElementAsLastChild("JSON");
			MbElement outJsonData =
					   outJsonRoot.createElementAsLastChild(MbElement.TYPE_NAME, MbJSON.DATA_ELEMENT_NAME, null);
			
			String mapKey = inAssembly.getLocalEnvironment().getRootElement().getFirstElementByPath("/REST/Input/Parameters/cache-key").getValueAsString();
			String cacheValue = "";			
		try {

			MbGlobalMap gmap =  MbGlobalMap.getGlobalMap(mapName);
			//MbGlobalMap gmap =   MbLocalMapCache.getLocalMap(mapName, new MbGlobalMapSessionPolicy(20), null);
			if (mapKey != null) {
				if (gmap.containsKey(mapKey)) {
					cacheValue = (String) gmap.get(mapKey);
				    outJsonData.createElementAsLastChild(MbElement.TYPE_NAME,MbJSON.DATA_ELEMENT_NAME, null)
    				.createElementAsLastChild(MbElement.TYPE_NAME_VALUE,"CacheContent", cacheValue);				}
			}
		} catch (Exception exp) {
			System.err.println("Exception occurs when trying to access the global cache: " + exp.getStackTrace());
			
		}

Monitoring

The following monitoring can only be traced, get if it is a global cache.

Service Trace can be enabled by setting trace in the server.conf.yaml file trace: ‘service’

Activity log

By activating the activity log and trace on a file it is possible to have an insight on the activity performed in the global cache (this doesn’t work for the local cache).

MessageNumber,Severity,Timestamp,ThreadID,FormattedMessage,Tags,Inserts
12082,I,"2025-01-08 12:25:13.758574",3516974,,"MAXIMUM_RATE=1,DELAY=0.95,MSGFLOW=gen.LocalCacheAPI,NODETYPE=INPUT,NODE=HTTP Input,",0.95,1,
11111,I,"2025-01-08 12:25:14.475688",3516870,,"RM=GlobalCache,CACHENAME=WMB,",MyCatalogServer1,(00000000-00000000-00000000),
11114,I,"2025-01-08 12:25:14.562124",3516870,,"RM=GlobalCache,CACHENAME=WMB,",MyCatalogServer1,(00000000-00000000-00000000),
11109,I,"2025-01-08 12:25:14.618049",3516872,,"RM=GlobalCache,CONNECTION_EVENT=CONNECTED,CACHENAME=WMB,",WMB,(00000000-00000000-00000000),
11516,I,"2025-01-08 12:25:28.927696",3516974,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,NODETYPE=INPUT,HTTP_METHOD=GET,",HTTP Input,REST,GET,/localcacheapi/v1/CacheContent,gen.LocalCacheAPI,(0035AA2E-677E6EB8-00000001),
11107,I,"2025-01-08 12:25:28.947881",3516974,,"RM=GlobalCache,CACHEMAP=myMap,CACHENAME=WMB,CACHEKEY=test,MSGFLOW=gen.LocalCacheAPI,NODE=getCacheContent (Implementation).Java Compute,",myMap,WMB,test,(0035AA2E-677E6EB8-00000001),
11517,I,"2025-01-08 12:25:28.948357",3516974,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Reply,NODETYPE=REPLY,HTTP_STATUS=200,",HTTP Reply,200,(0035AA2E-677E6EB8-00000001),
11506,I,"2025-01-08 12:25:28.948489",3516974,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,",(0035AA2E-677E6EB8-00000001),
11302,I,"2025-01-08 12:25:28.948519",3516974,,"RM=Parsers,MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,",12,(00000000-00000000-00000000),
11516,I,"2025-01-08 12:25:30.853315",3517666,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,NODETYPE=INPUT,HTTP_METHOD=PUT,",HTTP Input,REST,PUT,/localcacheapi/v1/CacheContent,gen.LocalCacheAPI,(0035ACE2-677E6EBA-00000001),
11107,I,"2025-01-08 12:25:30.855545",3517666,,"RM=GlobalCache,CACHEMAP=myMap,CACHENAME=WMB,CACHEKEY=test,MSGFLOW=gen.LocalCacheAPI,NODE=putCacheContent (Implementation).Java Compute,",myMap,WMB,test,(0035ACE2-677E6EBA-00000001),
11101,I,"2025-01-08 12:25:30.872809",3517666,,"RM=GlobalCache,CACHEMAP=myMap,CACHENAME=WMB,CACHEKEY=test,MSGFLOW=gen.LocalCacheAPI,NODE=putCacheContent (Implementation).Java Compute,",myMap,WMB,test,(0035ACE2-677E6EBA-00000001),
11517,I,"2025-01-08 12:25:30.873035",3517666,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Reply,NODETYPE=REPLY,HTTP_STATUS=200,",HTTP Reply,200,(0035ACE2-677E6EBA-00000001),
11506,I,"2025-01-08 12:25:30.873122",3517666,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,",(0035ACE2-677E6EBA-00000001),
11302,I,"2025-01-08 12:25:30.873154",3517666,,"RM=Parsers,MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,",9,(00000000-00000000-00000000),
12082,I,"2025-01-08 12:25:30.873285",3517666,,"MAXIMUM_RATE=1,DELAY=0.046,MSGFLOW=gen.LocalCacheAPI,NODETYPE=INPUT,NODE=HTTP Input,",0.046,1,
11516,I,"2025-01-08 12:25:32.600362",3516974,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,NODETYPE=INPUT,HTTP_METHOD=GET,",HTTP Input,REST,GET,/localcacheapi/v1/CacheContent,gen.LocalCacheAPI,(0035AA2E-677E6EBC-00000002),
11107,I,"2025-01-08 12:25:32.602113",3516974,,"RM=GlobalCache,CACHEMAP=myMap,CACHENAME=WMB,CACHEKEY=test,MSGFLOW=gen.LocalCacheAPI,NODE=getCacheContent (Implementation).Java Compute,",myMap,WMB,test,(0035AA2E-677E6EBC-00000002),
11103,I,"2025-01-08 12:25:32.603334",3516974,,"RM=GlobalCache,CACHEMAP=myMap,CACHENAME=WMB,CACHEKEY=test,MSGFLOW=gen.LocalCacheAPI,NODE=getCacheContent (Implementation).Java Compute,",myMap,WMB,test,(0035AA2E-677E6EBC-00000002),
11517,I,"2025-01-08 12:25:32.603887",3516974,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Reply,NODETYPE=REPLY,HTTP_STATUS=200,",HTTP Reply,200,(0035AA2E-677E6EBC-00000002),
11506,I,"2025-01-08 12:25:32.603981",3516974,,"MSGFLOW=gen.LocalCacheAPI,NODE=HTTP Input,",(0035AA2E-677E6EBC-00000002),

Service trace

Extended detail can be retrieved on the global cache when service trace is activated. Here is an example of a service trace:

PUT

2025-01-08 12:56:17.066950 3463207 { MbGlobalMap.put (0034D827-677E67E1-00000001) , 'key = test, value = hello'
2025-01-08 12:56:17.067182 3463207 { MbGlobalMapInternal.put (0034D827-677E67E1-00000001) , 'key = test, value = hello'
2025-01-08 12:56:17.069642 3463207 UserTrace BIP7170I: Operation 'put' on map 'myMap' has been completed successfully.
A client successfully completed operation 'put' on map 'myMap' in grid 'WMB'. The data processed by this action was 'Key = test'.
2025-01-08 12:56:17.070140 3463207 UserTrace BIP11101I: Put data into map 'myMap'
The message flow has successfully put data into map 'myMap', in cache 'WMB', using key 'test'.

GET

2025-01-08 12:56:24.853856 3465162 { MbGlobalMapInternal.get (0034DFCA-677E67E8-00000001) , 'key = test'
2025-01-08 12:56:24.854696 3465162 MbGlobalMapInternal.get (0034DFCA-677E67E8-00000001) file:/build/jenkins_swg/slot0/product-build/WMB/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp line:221 message:7170.BIPmsgs, 'Operation complete', 'get', 'myMap', 'WMB', 'Key = test'
2025-01-08 12:56:24.854696 3465162 UserTrace BIP7170I: Operation 'get' on map 'myMap' has been completed successfully.
A client successfully completed operation 'get' on map 'myMap' in grid 'WMB'. The data processed by this action was 'Key = test'.
None
2025-01-08 12:56:24.854894 3465162 UserTrace BIP11103I: Got data from map 'myMap'
The message flow has successfully got data from map 'myMap', in cache 'WMB', using key 'test'.
2025-01-08 12:56:24.855210 3465162 } MbGlobalMapInternal.get (0034DFCA-677E67E8-00000001) , 'result = hello'
2025-01-08 12:56:24.855566 3465162 } MbGlobalMap.get (0034DFCA-677E67E8-00000001) , 'actvalue = hello'

Tags:

Categories:

Updated:

Leave a comment

You may also enjoy

Threads limit for MQ on containers

4 minute read

MQ is a multithreaded solution that might spawn a relative high number of threads that might be incompatible with the default value of the operating system w...

Deploy ACE with remote BAR

2 minute read

Introduction There are different ways to deploy an Integration Server on a kubernetes environment: build your own image with the AppConnect flow already d...

KafkaConnect with EventStreams

8 minute read

Introduction This post is intended to explained how connectors for EventStreams can be deployed. The EventStreams connectors are provided with KafkaConnect. ...