axis2 client memory leak
DESCRIPTION
TRANSCRIPT
![Page 1: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/1.jpg)
Axis2 client memory leak
![Page 2: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/2.jpg)
Outline
• Issue & diagnosis• Postmortem from technique point– Axis2 1.4/1.4.1/1.5.6 client– Implement & design issue
![Page 3: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/3.jpg)
Issue
CServer exhaust 8G memory -> memory leak
![Page 4: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/4.jpg)
diagnosis
jmap –histo pid
num #instances #bytes class name---------------------------------------------- 1: 1001744 246906600 [C //char array 2: 2855952 137085696 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment 3: 282338 115692192 [I // int array 4: 677362 102508648 [Ljava.util.HashMap$Entry; 5: 192117 99164392 [B // byte array 6: 2856122 91732120 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry; 7: 2855952 91390464 edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync 8: 1337498 53499920 java.lang.String 9: 433634 43130120 [Ljava.lang.Object; 10: 597908 38266112 java.util.HashMap
1) The main change in this release is using web service client (axis2 1.4.1)
From the histogram, the memory leak is caused by Axis2.
2) char[] / edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap
3) Google ‘axis2 1.4.1 concurrentHashMap memory leak‘
![Page 5: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/5.jpg)
diagnosis
• We got a 2.39G memory heap dump– Very fast, less than 30 seconds (because 8G
memory in production)• Problem : too big to open in dev box– Need enough physical memory– (May) need 64 bit OS– Try Jhat/Jprofile/YJP/Jmat eclipse plugin
(windows/2G)
jmap –dump:file=cdump pid
Open by Jmat standalone application
![Page 6: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/6.jpg)
diagnosis
AxisConfiguration retain 540.9m heap
![Page 7: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/7.jpg)
diagnosis
Hashtable allEndPoints holds more than 2k axis2 endpoint instances
![Page 8: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/8.jpg)
diagnosis
All of the endpoint instances are about FavoriteService
1) Confirm in the code, only FavoriteService related did not call cleanup
![Page 9: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/9.jpg)
Axis2 1.4 memory leak
• Reproduce
for( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request); }
![Page 10: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/10.jpg)
Axis2 1.4 memory leak
• Informal Model (OO design)
AxisConfiguration
Map allServices = new Hashtable();Map allEndpoints = new Hashtable();
![Page 11: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/11.jpg)
Axis2 1.4 memory leak
• Cause : Stub Initiation, add into AxisConfiguration
//Init, AddallServices.put(serviceName, axisService);..allEndpoints.put(serviceName + "." + endpointName, axisService);
![Page 12: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/12.jpg)
Axis2 1.4 memory leak
• Cause : Stub Finalize
//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName());}
//AsixConigurationpublic synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }
![Page 13: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/13.jpg)
Axis2 1.4 memory leak
• Cause : Client Finalize
//ServiceClientprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); …}
//AsixConigurationpublic AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName());
… } …}
![Page 14: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/14.jpg)
Axis2 1.4 memory leak
• Cause
• Memory Leak : allEndpoints
//Init, AddallServices.put(serviceName, axisService);..allEndpoints.put(serviceName + "." + endpointName, axisService);
//Cleanup(Finalize), RemoveallServices.put(serviceName, axisService);
![Page 15: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/15.jpg)
Axis2 1.4.1 fix
• Fix the bug AXIS2-3870
//AsixConiguration.removeServiceGroup
//removes the endpoints to this service String serviceName = axisService.getName(); String key = null; for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){ key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key); }
![Page 16: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/16.jpg)
Axis2 1.4.1 memory leak
• Reproduce
for( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request); stub.cleanup(); }
Programmer: 1.4 has memory leak issue, so call cleanup
![Page 17: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/17.jpg)
Axis2 1.4.1 memory leak
• Cause : Stub Finalize (no change)
//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName());}
//AsixConigurationpublic synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }
![Page 18: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/18.jpg)
Axis2 1.4.1 memory leak
• Cause: Client Finalize (no change)
//ServiceClientprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); …}
![Page 19: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/19.jpg)
Axis2 1.4.1 memory leak
• Cause: Client Finalize (no change)
//AsixConigurationpublic AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName());
… for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){ key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key); } …
}
…}
Servce are already removed from ServiceGroup in Stub cleanupThe while loop would never enter
![Page 20: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/20.jpg)
Axis2 1.4.1 memory leak
• Cause– Stub cleanup and ServiceClient cleanup have
dependency– Can not call in below order
• Two are two memory leak bugs in 1.4, 1.4.1 only fix 1 bug
Stub.cleanup ServiceClient.cleanup
![Page 21: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/21.jpg)
Axis2 1.5(.6) fix
• Fix the bug AXIS2-4007 AXIS2-4163
//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { // _service.getAxisConfiguration().removeService(_service.getName()); _serviceClient.cleanup();}
![Page 22: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/22.jpg)
Implement issue
• Forget to cleanup– Container object: add only, no remove– Resource object: apply only, no return/close
• Cleanup dependency– One object cleanup depend on other objects– Two object cleanup/two method has order
dependency
![Page 23: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/23.jpg)
Design Issue
• AxisConfiguration is a global shared object– Usually only 1 instance even in client side.
• Purpose for put service/endpoint map in this global object AxisConfiguration ?
![Page 24: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/24.jpg)
Design Issue
• Message Dispatch in server side // RequestURIBasedServiceDispatcher public AxisService findService(MessageContext messageContext) throws AxisFault {
AxisConfiguration registry = configurationContext.getAxisConfiguration();
AxisService axisService = registry.getService(values[0]);
// If the axisService is not null we get the binding that the request came to add // add it as a property to the messageContext if (axisService != null) { Map endpoints = axisService.getEndpoints(); if (endpoints != null) { if (endpoints.size() == 1) { messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(axisService.getEndpointName())); } else { String endpointName = values[0].substring(values[0].indexOf(".") + 1); messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(endpointName)); } } }
return axisService; }
![Page 25: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/25.jpg)
Design Issue
• Service is ‘singleton’ in server side // AxisConfiguration public AxisService getService(String name) throws AxisFault { AxisService axisService = (AxisService) allServices.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } else { axisService = (AxisService) allEndpoints.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } } return null; }
![Page 26: Axis2 client memory leak](https://reader033.vdocuments.mx/reader033/viewer/2022061210/54918714b479599d2d8b532b/html5/thumbnails/26.jpg)
Design Issue
• Client side– (Usually) New instance every method invocation– No need for message routing• Why still register in Axi2Configuration ?
axisConfig = configContext.getAxisConfiguration(); if (axisService == null) { axisService = createAnonymousService(); } this.axisService = axisService; if (axisConfig.getService(axisService.getName()) == null) { axisService.setClientSide(true); axisConfig.addService(axisService); } else { … }
…