Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Munira Baldiwala
Seema Deshmukh
Version History
TIBCO Confidential
Document Overview
This document discusses the performance of the TIBCO ActiveMatrix BusinessWorks Service Engine
5.7.0. The intent is to make the results of the comprehensive benchmark tests available to the users
to make informed architecture decisions.
Prior knowledge of TIBCO ActiveMatrix BW and TIBCO ActiveMatrix BWSE concepts is desirable for
the best use of this document.
Disclaimer: The performance results in this document are provided for the reference only.
They can be reproduced only under the exact environment and conditions as existed when the
tests were done. The performance numbers should be used only as a guideline and TIBCO
does not guarantee their accuracy.
http://www.tibco.com
TIBCO ActiveMatrix BWSE Performance Best Practices
Table of Contents
Version History...........................................................................................................1
Introduction................................................................................................................5
Use Cases.................................................................................................................30
SOAP (HTTP)-BWSE-Java Service............................................................................................30
References................................................................................................................33
References................................................................................................................33
Introduction
ActiveMatrix BusinessWorks Service Engine (BWSE) enables ActiveMatrix BusinessWorks to
participate in the ActiveMatrix environment which supports the development, deployment, and
management of applications that conform to a service-oriented architecture (SOA).
BWSE allows other ActiveMatrix components to discover and invoke ActiveMatrix BusinessWorks
services. It also allows ActiveMatrix BusinessWorks to consume services provided by other
components in the ActiveMatrix platform.
This document discusses the performance considerations for TIBCO BusinessWorks Service Engine
(BWSE). A brief overview of the BWSE message flow is provided to facilitate understanding of the
tuning parameters. A summary of the comparative performance is provided to help users understand
the performance characteristics of the product. A few use cases are discussed to illustrate the
application of tuning parameters.
All the tests in this document are performed on the configuration as stated in Appendix B:
Hardware Configuration
Step 2:
The SOAP (over HTTP) Requests from the client reach the BWSE Service (Service resource) in the
AMSG Environment. They are serviced by the Tomcat threads of the BW Engine. The MIN and MAX
size of the HTTP Server Thread pool can be set through the “bw.properties” file. The default values
of these properties are 10 and 75, respectively. An HTTP Server thread is allocated for each request
that is concurrently executing and is released only after the response is sent back (i.e. The HTTP
Server thread is blocked till the response is received). The HTTP Server thread routes the request for
further processing to the Engine thread (aka Job Courier thread)
Step 3:
The Job courier thread in BW picks up the message for processing. The max value of the Job courier
thread pool can be set in the bw.properties file using the property:
Engine.Threadcount
(This property determines the maximum Engine thread pool size. This property is applicable to that
specific BW service unit deployed on that node. An Engine thread pool is created for every service
unit deployed on the node.)
Step 4:
The response is then sent back by the Engine thread to the Tomcat thread (which is blocked waiting
for a response).
Step 5:
The Tomcat thread picks up the response and delivers it to the external SOAP Client.
In this scenario, the throughput depends on the latency of the service, the number of
concurrent users and the Engine thread pool size.
SOAP Service Unit and BWSE Service Unit are deployed on the same AMSG node
Step 2:
The SOAP (over HTTP) Requests from the client reach the SOAP Service in the AMSG Environment.
They are serviced by the HTTP Server Shared Resource threads (HTTP Resource Adapter
threads) which are initialized for the HTTP Server Shared Resource configured through the AMSG
Administrator. The MIN and MAX size of the HTTP Server Thread pool can be can be set through the
AMSG Administrator. The default values of these properties are 10 and 75, respectively. An HTTP
Server thread is allocated for each request that is concurrently executing and is released only after
the response is sent back (i.e. The HTTP Server thread is blocked till the response is received).The
HTTP Server thread routes the request for further processing to service provider (BWSE). The BWSE
provider is a Local Provider (residing on the same node).The message is routed directly to the
appropriate provider for further processing.
Step 3:
The max value of the BWSE provider thread pool, can be set in the Node’s TRA configuration file
using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
(This property determines the maximum thread pool size. This property is applicable to every service
unit deployed on that node. A separate thread pool is created every service unit deployed on the
node.)
Step 4:
The thread from the Engine Thread pool picks up the routed message for further processing. The max
value of the Engine thread pool can be set in the bw.properties file using the property:
Engine.Threadcount
(This property determines the maximum Engine thread pool size. This property is applicable to that
specific BW service unit deployed on that node. A Engine thread pool is created for every service unit
deployed on the node.)
Step 5:
The SOAP Consumer pool thread picks up the response and delivers it to the HTTP Server Thread
(which is blocked waiting for a response).
The max value of the thread pool can be set in the Node’s TRA configuration file using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
The default and min values of this property are 10 and 5, respectively.
(This property determines the maximum thread pool size. This property is applicable to every
Component Service, Component Reference, Composite Service and Composite Reference deployed
on that node. A separate thread pool is created for every Component Service, Component
Reference, Composite Service and Composite Reference deployed on the node.)
In a one node scenario, the throughput depends on the latency of the service, the number of
concurrent users and the Engine thread pool size. The maxpoolsize does not affect the throughput.
Here we take a simple use-case in which a Wired SOAP-BWSE Composite Service is invoked by
an external SOAP Client over HTTP. The following diagram depicts a typical message flow for an in-
out message exchange for an invocation of this service. The service implementation details are in
Appendix A: Light Weight Service
SOAP Service Unit and Java Service Unit are deployed on 2 different nodes.(SOAP on one
node and Java on another node)
Step 1:
Concurrent users send SOAP requests(over HTTP) to be serviced by a service provider in the
AMSG Environment.
Step 2:
The SOAP (over HTTP) Requests from the client reach the SOAP Service in the AMSG Environment.
They are serviced by the HTTP Server Shared Resource threads (HTTP Server threads) which are
initialized for the HTTP Server Shared Resource configured through the AMSG Administrator. The
MIN and MAX size of the HTTP Server Thread pool can be can be set through the AMSG
Administrator. The default values of these properties are 10 and 75, respectively. An HTTP Server
thread is allocated for each request that is concurrently executing and is released only after the
response is sent back (i.e. The HTTP Server thread is blocked till the response is received). The
Delivery Channel routes the request for further processing to the service provider (BWSE). The
BWSE provider, in this case, is a Remote Provider (residing on a different node).Hence, the Delivery
Channel routes the requests to the BWSE Component Queue on the EMS Server.
Step 3:
The max value of the BWSE provider thread pool, can be set in the Node’s TRA configuration file
using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
(This property determines the maximum thread pool size. This property is applicable to every service
unit deployed on that node. A separate thread pool is created every service unit deployed on the
node.)
Each provider pool thread acquires an available JMS session from the session-pool. It then gets the
request message from the BWSE Component queue (on EMS) and processes the request. The
number of concurrent sessions that can be created depends on the max value specified in the Node’s
TRA configuration file.
java.property.com.tibco.matrix.amxframework.messaging.sessioncount
(This property determines the maximum number of JMS sessions that may be created with the EMS
server. Like max thread pool property, this property also applies to every service unit deployed on that
node. A separate session pool is created for each service unit)
Step 4:
The thread from the Engine Thread pool picks up the routed message for further processing. At this
instant, the session is released and the BWSE provider thread is available for the next request. This
makes the BWSE Provider threads asynchronous in behavior. The max value of the Engine thread
pool can be set in the bw.properties file using the property:
Engine.Threadcount
Step 5:
The response is delivered by the Engine thread to the SOAP Service queue on EMS.
Step 6:
The SOAP Consumer pool thread picks up the response from the queue, the max value of which can
be set in the Node’s TRA configuration file using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
(This property determines the maximum thread pool size. This property is applicable to every service
unit deployed on that node. A separate thread pool is created every service unit deployed on the
node.)
Each SOAP Consumer thread acquires a session from the session pool with EMS and picks up the
response delivered in the SOAP Service queue. The response is then delivered to the HTTP Server
Thread that is blocked on the response. The number of sessions that can be created depends on the
max value specified in the Node’s TRA configuration file.
java.property.com.tibco.matrix.amxframework.messaging.sessioncount
Step 7:
The HTTP thread sends back the response to the external SOAP Client.
In a two node scenario, the throughput depends on the latency of the service, the number of
concurrent users and the Engine thread pool size. Each Provider pool thread creates its own session
with the EMS Queue to fetch/deliver the request/response. In case of a BWSE component, as soon
as the Engine Thread finishes processing the request, the provider thread is released (synchronous
behavior).
Here we take a simple use-case in which a Wired SOAP-BWSE Composite Service is invoked by
an external SOAP Client over HTTP. The following diagram depicts a typical message flow for an in-
out message exchange for an invocation of this service. The service implementation details are in
Appendix A: Light Weight Service
Legend (Threads)
HTTP Threads
C JobCourier Pool
BWSE Provider
Pool Threads
D InvokePartner Pool
Job Courier
Pool Threads
InvokePartner
Pool Threads
BWSE Consumer
Pool Threads
Diagram4: Architecture of a Wired BWSE Service with Invoke Partner (Same Node)
SOAP Service Unit, BWSE Service Unit and Java Service Unit are deployed on the same
node.
Step 1:
Concurrent users send SOAP requests(over HTTP) to be serviced by a service provider in the
AMSG Environment.
Step 2:
The SOAP (over HTTP) Requests from the client reach the SOAP Service in the AMSG Environment.
They are serviced by the HTTP Server Shared Resource threads (HTTP Server threads) which are
initialized for the HTTP Server Shared Resource configured through the AMSG Administrator. The
MIN and MAX size of the HTTP Server Thread pool can be can be set through the AMSG
Administrator. The default values of these properties are 10 and 75, respectively. An HTTP Server
thread is allocated for each request that is concurrently executing and is released only after the
response is sent back (i.e. The HTTP Server thread is blocked till the response is received). The
HTTP Server thread routes the request to service provider (BWSE) for further processing. In this
case, BWSE provider is a Local Provider (residing on the same node). The message is routed directly
to the appropriate provider for further processing.
Step 3:
The max value of the BWSE provider thread pool, can be set in the Node’s TRA configuration file
using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
(This property determines the maximum thread pool size. This property is applicable to every service
unit deployed on that node. A separate thread pool is created every service unit deployed on the
node.)
Step 4:
The thread from the Engine Thread pool picks up the routed message for further processing. The max
value of the Engine thread pool can be set in the bw.properties file using the property:
Engine.Threadcount
(This property determines the maximum Engine thread pool size. This property is applicable to that
specific BW service unit deployed on that node. A Engine thread pool is created for every service unit
deployed on the node.)
Step 5:
The Engine thread processes this message. Since BWSE has a reference partner, the request is then
forwarded and picked up by a thread in the InvokePartnerActivity Thread pool.
The max value of this thread pool can be set in the bw.properties file using the property:
InvokePartnerServiceActivityThreadPool.ThreadCount
The InvokePartner thread forwards the message to the Component it is wired to (in this case Java)
and waits for the response. Hence, the InvokePartner thread makes a synchronous call.
The Java provider pool thread picks up the request, the max value of which can be set in the Node’s
TRA configuration file using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
(This property determines the maximum thread pool size. This property is applicable to every service
unit deployed on that node. A separate thread pool is created every service unit deployed on the
node.)
The message is then processed in Java. After Java finishes processing the message, the response is
delivered by the Java thread to the BW Consumer thread pool
Step 7:
A thread from the BW Consumer thread pool picks up the response, the max value of which can be
set in the Node’s TRA configuration file using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
(This property determines the maximum thread pool size. This property is applicable to every service
unit deployed on that node. A separate thread pool is created every service unit deployed on the
node.)
BWSE consumer thread forwards this request to the Invoke Partner thread.
Step 8:
The InvokePartner thread which was waiting on this request picks up the response from the BW
Consumer thread. The response is then forwarded to the SOAP Consumer thread pool.
Step 9:
The SOAP Consumer pool thread picks up the response from the queue, the max value of which can
be set in the Node’s TRA configuration file using the property:
java.property.com.tibco.matrix.amxframework.messaging.maxpoolsize
(This property determines the maximum thread pool size. This property is applicable to every service
unit deployed on that node. A separate thread pool is created every service unit deployed on the
node.)
Step 10:
The HTTP thread sends back the response to the external SOAP Client.
In this scenario, the throughput depends on the latency of the service, the number of
concurrent users, the maxpoolsize, Jobcourier pool size and InvokePartner pool size value. The
maxpoolsize limits the number of requests that can be handled concurrently by the AMSG
Containers. The InvokePartner threads being synchronous, the throughput to a large extent depends
on their value.
The results for SOAP over HTTP Transport with increasing concurrent users for a 1K Request and 1K
Response document size are depicted below:
Graph1: Results for BWSE Unwired SOAP Service (Appendix A: Light Weight Service)
As the number of concurrent users is increased, the throughput increases linearly until it reaches a
peak (at 65% CPU utilization) and then drops from there.
2. As seen in the graph above, it is observed that as the concurrent users increase, the
throughput also increases until CPU utilization reaches 65%. It does not scale
beyond 65% with the below mentioned hardware.
The results for SOAP over HTTP Transport with increasing concurrent users for a 1K Request and 1K
Response document size are depicted below:
Graph2: Results for BWSE Wired SOAP Service (Appendix A: Light Weight Service)
As the number of concurrent users is increased, the throughput increases linearly until it reaches a
peak and then drops from there.
2. As seen in the graph above, it is observed that as the concurrent users increase, the
throughput also increases till a point where it reaches its peak (at 65% CPU
utilization).
3. For scalability with a very large number of concurrent users consider using an
external HTTP load balancer (for SOAP over HTTP), especially when the service is
4. For HTTP transport, increase the HTTP server threads as the number of concurrent
users accessing the service increase. If the HTTP server max threads are lower than
the concurrent users, the Node will throw connection errors and an error response
shall be returned to the clients those are unable to retrieve an HTTP server thread.
The HTTP server threads are synchronous in nature as hence merely increasing the
framework pool size will not increase throughput. The process address space
available for the process, thread stack size and the maximum heap size will
influence the maximum number of HTTP Server threads that can be created. So if
you have an extremely large (1000+) number of concurrent users, HTTP may have
some inherent limitations. Using JMS as the transport will help in this case. The
messages will accumulate on the EMS queue (this will impact latency) but at-least
they can be processed serially.
Graph3: Results for BWSE Unwired and Standalone BW Service (Appendix A: Light Weight Service)
The comparison results for SOAP Service (SOAP over HTTP Transport) with BW
Standalone, BWSE Unwired and BWSE Wired for a 1K Request and 1K Response
document size are depicted below:
Graph4: Performance Comparison BW, BWSE Unwired, BWSE wired for SOAP Service (Appendix A: Light
Weight Service)
Note: Results are for Implementation Used in Appendix A: Light Weight Service
1. Performance of the Wired scenario gets better as compared to the Unwired scenario as
the number of users increases. The above graph proves that BWSE Unwired does not scale
appropriately with increasing users, where BWSE Wired depicts better scaling.
2. As seen in the graph above, it is observed that as the concurrent users increase, the
throughput also increases till a point where it reaches its peak.
4. In case of the Unwired scenario, the maxpoolsize does not come into play, since the
service is directly accessed. Increase the Engine.Threadcount as the number of concurrent
requests increase. Though this may not always be required depending on the service
response time and hardware configuration. Also, here, the Tomcat threads of the BW Engine
itself are used to service the request.
All these tests have been run with the following settings:
Heap Settings:
[Maximum stack size depends on whether it is 32 bit/64 bit. With a 32 bit Windows OS, the maximum stack size is 256 KB and
with a 64 bit Windows OS, the maximum stack size is 512 KB. ]
On Node startup, (without having deployed any service assemblies), maximum heap memory
reaches 175 MB, whereas stable state memory is about 85MB. The total number of live threads are
about 53.
Footprint at Runtime
Following tests were conducted with varying the type of services/components/references and their
number:
The above tests were conducted with all components, services and references on the same node.
For details of the above threads per component, refer Appendix D: Thread and Memory Footprint.
NOTE:: The above memory values do not include the non-heap memory, which is a constant of 60 MB throughout.
The Max heap size of the Engine and Node are set to 512MB and 640MB,
respectively.
Observation
Graph5: Memory Graph with increasing concurrent Users and Doc Sizes
Analysis
For BW Standalone:
N = number of concurrent
requests in a system
For Example:
If, Request Doc size = 1 MB, Response Doc size= 2
MB, Number of concurrent requests = 5
Then, M= 2, N=5.
Please note: The above equations apply only to the above payload size range
indicated.
Two properties that are important for BW engine memory management are
MaxJobs and FlowLimit. If the expected workload would not exceed the available
memory resources, one can set MaxJobs=0 and FlowLimit=0. Calculating
MaxJobs and FlowLimit for a process engine depends on the following factors:
A best practice is to design the process deployment in such a way that the target
peak loads can be accommodated by the BW engine(s) without exceeding the
memory limit, and then set MaxJobs and/or FlowLimit to reflect the number of
concurrent jobs that are expected under this peak load.
Take a look at the table below to understand what this exactly means.
The scenario used below is a simple SOAP (HTTP) -> BWSE scenario which
echoes the request as the response. A payload size of 5MB is sent as data. The
input rate is 0.4 txn/sec.
Use Cases
The Java Component has a sleep of 1 sec between getting the request and
sending the response back, so the theoretical throughput for a single concurrent
user is 1 tps.
Scenario 1:
Considering that the SOAP Service, Java and BWSE Components are on the same
node. As explained in the Architecture above, being on the same node, the
requests do not pass through the EMS (hence, the maxsessioncount property
does not play a role here), but are directly routed to the Java and BWSE Provider
through the Delivery Channel. Following are the results with varying Thread Pool
settings:
The above results owe to the thread model designed for the One Node scenario.
Case 2:
Case 3:
Scenario 2:
Considering that the SOAP Service and each Java and BWSE Component are on
different nodes. As explained in the Architecture above, being on different nodes,
the requests pass through the EMS and are routed to the appropriate remote
BWSE and Java Provider through the Delivery Channel. Following are the results
with varying ThreadPool and Engine.Threadcount settings:
Case 1: When the maxpoolsize value of all the nodes is set to 10 and
Engine.threadcount is set to 32, the throughput is 10 tps as the number of
concurrent requests that can be processed are 10.
Case 2: When the maxpoolsize and Engine.threadcount value of the Java and
BWSE Nodes is set to 32 whereas that of the SOAP Node is set to 10, the
throughput is 32 tps. The maxpoolsize value of the SOAP Node does not affect the
throughput, because at the SOAP Node, the Consumer threads merely pick up the
requests and forward it to the HTTP Server threads which are waiting for the
response. The Consumer threads do not do too much of processing and hence 40
concurrent users do not cause a SOAP Consumer Thread contention.
Case 3 and Case 4: When the maxpoolsize value of even one of the Java Nodes is
set to 10, the throughput is 10 txn/sec. Hence the throughput gets limited by the
maxpoolsize of any of the Components, namely MaxThreadPool size of N2 in Case
3 and N3 in Case 4 being set to 10 in this case.
Case 5 and case 6: When the maxpoolsize value of the Java, SOAP and BWSE
Nodes is set to 32 whereas the Engine.threadcount is set to 16, the throughput is
16 tps. The number of Engine threads, here, controls the throughput.
The throughput is limited by the size of the Framework Thread Pool and the Engine
threads, and not by the MAX size of the HTTP Server Thread pool.
The throughput depends on the size of the framework Thread Pool and Engine Thread
Pool. Each thread from the framework Thread Pool creates a session with the EMS
Queue. For maximum possible performance, the sessioncount should be kept equal to the
maxpoolsize. If the sessioncount value is set less than the value of the maxpoolsize, the
throughput would be limited by the sessioncount.
References
BW Performance Analysis and Tuning Guide
http://insidearc2.tibco.com/kb_new/article2.asp?
BN=&arid=106146&mode=&comment=&rec=&wf=
https://newsalescentral.tibco.com/docs/DOC-9657
https://newsalescentral.tibco.com/docs/DOC-7834
https://newsalescentral.tibco.com/docs/DOC-9658
http://qa.tibco.com/EngDocLibrary/index.html
Thread Behavior
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ThreadPoolExecutor.html
EchoString 1
String 1 (string) (string)
EchoString String 2 (string)
String 3 (string)
……
EchoString 1
(string)
String 2 (string)
String 1 (string)
EchoString String 3 (string)
5 MB payload
5 MB payload
……
EchoString 1
String 1 (string) (string)
EchoString
……
Batch Service
EchoString 1
String 1 (string) (string)
EchoString (5 MB file is
read)
……
Machine 3
(AMSG Node Machine 1 (Load
1/BW Engine ) Generator)
Server Class Proliant HP Proliant DL-380
BL460c G1 (C- G4
Class)
Chip Vendor Intel Intel
Total Number 1 2
Of Processors
Cores Per 2 1
Processor
Threads Per 2 2
Processor
Clock 3 GHz 3 GHz
Memory 4 GB 4 GB
OS Windows 2003 Windows 2003
Server Server
L2 Cache 4 MB 2 MB
The default parameters in the Node .tra were used for the JVM.