Java Projects

Tools and techniques

Mike GodfreyInformation Technology Services / The University of Texas at Austin

Outline

Look at three java projects

• Different architectures adopted• Problems encountered and lessons learnt• Tools and other useful techniques

Who are we?

• Remains of centralized data processing• Part of the unit providing IT services across campus• Traditionally mainframe based applications with web

front-ends scripted in a simple in-house language• Changing mission as our administrative application

development role is done by departments• Project list that requires a much more powerful language

pushing adoption of Java• Changes are forcing developer to acquire new skills and

ways of working – java camp.

The Systems

• Clips – a web-based file system for distributing course materials

• Jwebagent – a java runtime for our in-house scripting language

• Tech lounge – a web-based developer documentation store

Clips – file system

Clips Requirements

• A simple to use, intuitive interface • Handle all file types• Accessible from any location• Date controlled display of materials• Easy transfer of materials from class to class• Work well with existing mainframe class

management system• Work within the UTDirect portal framework

Basic Design Choices

• File system metaphor for user interface• Browser access for both students and faculty• Use native XML database to model the file

system in a recursive schema• Store binary files in same XML database• JSP with scriptlets for display• clips interface

Use of XML for Meta Data

• File system metaphor seems ideally suited to the tree structure of XML

• Meta data structure based on two nodes types; directory-nodes and binary-nodes

• One XML document contains all meta information needed to build file structure for a given class

• Easy to re-build hierarchy of resources from XML tree• Database uses schema to specify stored document

structure

Meta-Data Details

• Directory nodes store information about a directory: name, creation time, description, etc

• Binary nodes store information about a binary file: name, foreign key, size, content type, etc

• Directory nodes can contain either binary nodes or other directory nodes (hierarchy)

• clips meta data

Clips Architecture

NaturalExisting class

management systemAdabas

Brokermainframe

XML database(binaries andMeta-data)

HelperClassesB

roker api

Authorization and class info

Tamino api

JSP

presentation layer

User

New Issues for us

• Handling and auditing file uploads from web pages (multipart encoded forms)

• Building and manipulating XML records• How to best make large scale use of JSPs in a

complex Java system• How to simplify communication with mainframe

Useful Tools

• Log4j – used for debugging and recording production exceptions (system written before Java 1.4)

• JDOM – very user friendly api for manipulating the XML

• cos.jar – used to parse the multipart encoded forms

• Entire-X broker ACI for Java• Java stub generator for mainframe modules• Tamino - native XML database

Lessons Learnt

• Be careful of scriplets in JSP; they expand to fill the available space!

• Better to use a controller servlet to organize navigation through JSPs

• XP and pair-programming works• A Java developer’s best friend is Goggle

Alternative JSP architecture

Viewjsp

Maintainjsp

Updatejsp

Copyjsp

CopyTask

FormParser

UpdateTask

XML store

ResourceList

mainframe

AuthTask

CopyTask

FormParser Update

Task

XML store

ResourceList

mainframe

AuthTask

Controllerservlet

ViewjspMaintain

jspUpdate

jspCopyjsp

Jwebagent

.WBXfile

Jwebagent HTMLfile

What is Webagent?

• A scripting language with a syntax like Natural (mainframe language)

• Scripts transformed into C++ objects forming a virtual machine which is serialized to a .WBX file

• Runtime reads .WBX file, rebuilds the C++ virtual machine and executes it.

• Pro: easy and productive for creating web front ends to mainframe applications.

• Con: limited functionality

Why Jwebagent?

• Same as Webagent but the rebuilding of the runtime is done in Java …plus

• Enable java classes to run existing Webagent scripts

• Enable Webagent scripts to use java classes• Improve execution speed – no process forking

and script buffering

Main Runtime Elements

• Servlet – handles all execution requests for a WBX file• Executor – coordinates loading file, running the VM, and

handling errors for an individual file request• File loader – re-builds the serialized script objects• Context objects – handle data specific to that particular

request• VmOpCodes – functional elements of the VM• Data objects – variables, literals

Runtime

WBXfile

Servlet

file request

Executor

vmOpvmOp

vmOpvmOp

vmOpvmOp

brokerbroker

broker

Serial code object

Script context

Request context

env

httpreq

httprespData buffer

Builds then executes

Useful Tools

• Junit – unit testing framework (junit.org)• JProbe – application analyzer (Quest Software) • JMeter – load testing suite (Apache.org)• HttpExplorer – gui http request and response

tool (in-house)• Jedit – text editor (Jedit.org)• IntelliJ IDEA – Java IDE (Jet Brains)• CVS (Tortoise, WinCVS, Putty for SSH)• Xvi32 – hex editor

Junit – unit testing

• Provides easy way to set and teardown test situations

• Automated running of many tests• Organize tests in suites and suites into meta-

suites• Text or gui UI for test reporting• Refactor with confidence! (very valuable)• Trick is to know what tests to write• Good with Jwebagent as it is not database

driven

JProbe – code analyser

• Threadalizer – finds deadlocks and conflicts (did not show any problems for Jwebagent)

• Memory Debugger – shows memory use by classes and can find hanging objects (useful – strings to byte arrays)

• Profiler – tracks objects and the call pattern, records method time and number of calls (most useful tool –identify prime objects to optimize)

JProbe Code Optimizations

• Strings to byte arrays – saved memory on string generation

• io to nio gave 25% increase in through-put• In-lining of frequently used methods to reduce

calling overhead• Combining of related classes when frequently

constructed• Un-doing OOP improved performance!

JMeter – load testing

• Build and store test plans• Control number of threads, ramp-up times and thread

delay to simulate different testing scenarios• Test http apps, SOAP and web services, LDAP, JDBC,

etc• Display test results in graphs, tables, files, etc• Build tests with logical flows to suit application• Apply pre and post requests during tests• Very valuable for performance tuning (with JProbe)

JVM Tuning

• Changed default of client mode to server mode (20% improvement)

• Other settings have given some improvements• Use incremental GC rather than ‘stop the world’

style• Uses the flags that control survivor ratio for the

GC generations – we have short lived objects• Increased default memory allocation

Did we make it?

• First tests showed consistently slower times than the C++ version ( up to 33* slower)

• JProbe analysis and JVM tuning reduced this to about 2* slower on average

• Through-put is slightly lower• JMeter shows a mean that is much faster than C++, but

and average that is much higher• Problem appears to be in the multi-treading of Broker• Tried JNI against C++ broker and see same results

Lessons learned

• Always build a test suite before refactoring• Never optimize until you know what to optimize• Do not assume GC defaults are optimal• Java needs lots of memory!• Allow for distractions in your project estimates

(estimated time 9 months, actual time start to finish 2.5 years, actual time spent on project 10 months)

Tech Lounge

Project Goals

• Nice, easy to use documentation store• Replace mainframe based system• Provide better editing and display options • Allow limited access to subsets of articles• Control the life-time of articles to ensure

relevancy• Provide good search capabilities and cross-

referencing options• Store multiple document formats

Design Choices

• Store articles in native XML database• Article content stored against a subset of XHTML strict

DTD – easy to display and construct• Each article has a meta-data XML file for owners, dates,

related articles, number of reads, etc• Article display via XSLT for HTML and pdf• Article maintenance via jsps and custom tag libraries• tech lounge

Article Maintenance Architecture

FormParser

XML store

Article Bean

mainframe

AuthTask

Controllerservlet

New Articlejsp

MaintainArticle

jsp

Useful Tools

• Many things from Apache• DHTML• Graphic designer

Created Tools/utilities• Tamino Wrappers – abstraction layer• UID (QUID-like foreign key generator)

Problems/Mistakes

• Not using CVS well during first half of project• Not writing unit tests • Not using logging for debug messages

Evolution

• Display of XML – moving towards all XSLT• JSP usage – moving away from interlinked

pages towards controller server style• JSP structure – scriplets bad, custom tags

good(?)• Building up institutional tools/utilities and

standards that are increasing productivity• Individual experts towards wide spread Java use

through in-house training courses

Review of our Java Experience

Pros:• More flexible and capable • Greater abstraction possible, increased

elegance• Many very powerful tools/utilities freely availableCons:• Requires higher competency of developers• Less productive for our typical applications• Needs larger machines

Other projects

• PdaParser – code generation to ease communication with mainframe modules

• Fatcookie – authentication• Virtual File System extension to JEdit to work

with our unix server script editing proceedure Current• Jar management database (entirely XML/XSLT)• Web service wrapper for mainframe modules• Dynamic pdf generation from mainframe data

Questions?

Useful Project Management Tip

Remember the Bullfighter!