Oracle Enterprise gateway load test

Last few weeks we have done a few exercises with OEG, develop a lot of filters, asynchronous delivery with JMS and much more. Last week we decided to do some load test, specially how many web service we can register in OEG. We develop jython script to register web services and assign policy in web services. However it's very tough to collect a big amount of web services online - we decided to register on web service in separate web service group, because you can register two similar web service in one group. The Jython script is as follows:

'''
Register WSDL group in Gateway
'''


from java.util import ArrayList
from deploy import DeployAPI
from esapi import EntityStoreAPI
from com.vordel.client.manager.actions.services import AddWebServiceServletProcessor
from vtrace import Tracer
import common
import datetime

t = Tracer(Tracer.INFO) # Set trace to info level
gw_deployURL = "http://localhost:8090/configuration/deployments/DeploymentService"
dep = DeployAPI(gw_deployURL, common.defUserName, common.defPassword)
es = dep.getStore('')

t0 = datetime.datetime.now()
print t0
wsdlURL = 'http://www.restfulwebservices.net/wcf/CurrencyService.svc?wsdl'
deployLocation = es.get('/[NetService]name=Service/[HTTP]name=Default Services')
defaultWsGroup = es.get('/[WebServiceRepository]name=Web Service Repository/[WebServiceGroup]name=Web Services')
t.info("---------------------------------")
# add new group on default webservice group
aws = AddWebServiceServletProcessor(es.es)
deploymentPks = ArrayList()
deploymentPks.add(deployLocation.getPK())
for i in range(1,100):
 try:
  t.info("Count Service :"+ str(i))
  wsGroup = es.addNewEntity(defaultWsGroup,"WebServiceGroup",{"name":"test"+str(i)})
  aws.addWebService(wsGroup, wsdlURL, deploymentPks, 60)
  # add policy Request and response
  #Add sample policy to user interception point.
  #Get generated entites for the web-service
  entities = ArrayList()
  entities = aws.getCreatedWebServiceEntities()
  en = entities[0]
  t.info("---------------------------------")
  t.info("Name: "+en.getStringValue("name")+"\n")
  #Looking for generated circuit for the web-service by service name
  policy = es.get('/[CircuitContainer]name=Generated Policies/[CircuitContainer]name=Web Services.'+'test'+str(i)+'.'+en.getStringValue("name")+'/[FilterCircuit]/[WSFilter]/[OperationCircuitChain]name=Request From Front End')
  t.info("---------------------------------")
  t.info("Name: "+policy.getStringValue("name")+"\n")
  #Attach sample policy to generated circuit
  #Modify the path to ur policy here
  pk = es.get('/[CircuitContainer]name=Policy Library/Modify the path to ur policy here');
  policy.setReferenceField("afterOperationHooks",pk.getPK())
  # -------- add Response handler ---------
  #Looking for generated circuit for the web-service by service name [Response handler]
  policyResponse = es.get('/[CircuitContainer]name=Generated Policies/[CircuitContainer]name=Web Services.'+'test'+str(i)+'.'+en.getStringValue("name")+'/[FilterCircuit]/[WSFilter]/[OperationCircuitChain]name=Response From Back End')
  t.info("---------------------------------")
  t.info("Name: "+policyResponse.getStringValue("name")+"\n")
  pkResponse = es.get('/[CircuitContainer]name=Policy Library/Modify the path to ur policy here');
  policyResponse.setReferenceField("afterOperationHooks",pkResponse.getPK())
  #Write updated policy to store
  t.info("---- update request policy-----")
  es.updateEntity(policy)
  t.info("---- update response policy-----")
  es.updateEntity(policyResponse) 
 except Exception, err:
  print "something wrong with registration!" 
  print "error:"+ str(err)

#Deploy new configuration
t.info("---- update entity -----")
res = dep.setStore(es)
t.info("---- Web services registered-----")
if res.getStatus() != True:
    t.error("Failed to deploy: " + res.getFailureReason())
    #t.error("Failures: "+ Integer.toString(res.getErrorCount())) 
# close entity Store  
es.close()
dep.logout()
delta_t = datetime.datetime.now() - t0 
print delta_t
t.info("end of commands...")

The above script will register 99 web services in one transaction. run the script from the directory %OEG_HOME%\samples\scripts\run.bat For load test we use Oracle Linux machine with 2 cpu, 6 Gb RAM. For better performance we increased heap size for the OEG to 4 gb in the jvm.xml as follows:

    <vmarg name="-Xmx4000m"/>
    <vmarg name="-Xms4000m"/>
    
    <vmarg name="-XX:PermSize=128m"/>
    <vmarg name="-XX:MaxPermSize=128m"/>
    <vmarg name="-XX:+UseConcMarkSweepGC"/>

Here is the Log of all step: Register 100 service - elapsed time 6 min - memory in use after registration 120 MB Register 200 service - elapsed time 9 min - memory in use after registration 280 MB Register 300 service - elapsed time 17 min - memory in use after registration 470 MB Restart OEG - memory in use 200 MB Register 400 service - elapsed time 27 min - memory in use after registration 900 MB Register 450 service - elapsed time 13 min - memory in use after registration 1400 MB Register 500 service - elapsed time 21 min - memory in use after registration 1700 MB Register 520 service - elapsed time 14 min - memory in use after registration 1900 MB Register 540 service - elapsed time 16 min - memory in use after registration 2100 MB After 540 service registration, OEG start crush with java.lang.OutOfMemoryError: Java heap space. Within registration OEG always replace old configuration file with new configuration file, but OEG could not release the allocated memory. After 540 service any deployment to OEG failed with crush. It indicate memory leak ((. In the time of deployment OEG always allocated double of it's uses memory - it means if OEG uses 500 MB memory - in the time of deployment it will need 1gb of memory. Here is the stack trace:

Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:3209)
        at java.lang.String.<init>(String.java:215)
        at java.lang.StringBuffer.toString(StringBuffer.java:585)
        at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeVa
lueString(DeferredDocumentImpl.java:1170)
        at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeVa
lueString(DeferredDocumentImpl.java:1120)
        at com.sun.org.apache.xerces.internal.dom.DeferredTextImpl.synchronizeDa
ta(DeferredTextImpl.java:93)
        at com.sun.org.apache.xerces.internal.dom.CharacterDataImpl.getNodeValue
(CharacterDataImpl.java:88)
        at com.vordel.es.impl.EntityCodec.getData(EntityCodec.java:263)
        at com.vordel.es.impl.EntityCodec.parse(EntityCodec.java:185)
        at com.vordel.es.impl.EntityCodec.parse(EntityCodec.java:228)
        at com.vordel.es.impl.EntityCodec.decode(EntityCodec.java:57)
        at com.vordel.es.impl.Importer.parse(Importer.java:152)
        at com.vordel.es.impl.Importer.parse(Importer.java:168)
        at com.vordel.es.impl.Importer.importData(Importer.java:97)
        at com.vordel.es.impl.AbstractEntityStore.importData(AbstractEntityStore

We also use JvisualVM to profilling the JVM. Here is an illustration from my local machine

The result of the load is that, OEG contain memory leak in it, also if you want to register a big amount web services you have to need much more memory to healthy your heap of OEG.