a java file transfer framework (and more) using quartz jared lynem amway corporation...
TRANSCRIPT
Hello I’m Jared We’ll talk about:
File transfers: requirements, reasons, concepts A framework for file transfers A way to schedule them Other quick victories
Links to resources at the end of this document
Feel free to contact me [email protected]
Requirements The first business data “integrations”
Moving paper files from one office to another Ditto machines! Crazy. Offices like on Mad Men
Fast forward to early digital integrations Moving data files from one server to another
Bread and butter, reliable, easy
Requirements – What? What – select and move files based on
location, file name When – based on a schedule or as a
process step Where – multiple targets, multiple
sources, archiving, PCI/PII considerations
How – Protocols: FTP, SFTP, FTPS, CIFS, VPN; Encryption: PGP
Requirements – Why?
Why indeed. Not our problem, just do as the
customer requests.
Seriously though, fight for the best design.
Reasons
Why should we use file transfers? To communicate securely with external
entities To interact with legacy applications Because they’re simple and easy to
support When you don’t need data transformation When someone need to review the data
Beware of PCI/PII requirements!
Reasons
Why develop a framework? Oracle SOA Suite “FTP Adapter” doesn’t
do what you might think it does. Oracle Data Integrator FTP/SFTP setup
is clunky, not easily extensible Often times, many very similar
integrations are required
Concepts - Java Object oriented programming Native language for Weblogic
Application Server Relatively easy, JDeveloper is a great
IDE Much, much faster than SOA tools Widely used, plenty of knowledgeable
developers Oracle: “Resistance is futile”
Concepts – Transfer Protocols
FTP – Old ‘n reliable Credential based login, or anonymous Not encrypted Easy to implement, widely supported
SFTP – “FTP” over SSH Encrypted transfer using Secure Shell Allows for additional authentication
measures
Concepts – Transfer Protocols
FTPS – FTP with encryption over SSL Support for multiple authentication
methods Support for public key certificates (SSL)
CIFS (aka SMB) – Used by Windows Allows for Active Directory or Kerberos
authentication Handy when FTP isn’t set up on
Windows servers
Concepts – Security
Transport Level Security (See above) Data Level Security – PGP (Pretty
Good Privacy) Offers encryption in a wide variety of
algorithms and strengths Offers data compression, integrity
checking, and message authentication Unbreakable!
The Framework
Most file transfers follow a similar design:
1. Trigger condition (sometimes)2. Source(s) for files3. Encryption/decryption/renaming
(sometimes)4. Target(s) for files5. Archiving (sometimes)
The Framework - Goal
To make these common tasks configurable via XML elements
To allow custom tasks to work seamlessly with common tasks to minimize development time
The Framework – JAXB
The XML part is easy: Create an XML schema with elements
for each step’s configuration Use JAXB (built-in to JDeveloper) to
create matching Java objects Use these objects as “properties” for
your step implementations Create a mapping between XML element
types and Java classes to execute
The Framework – Trigger Condition A trigger condition decides whether
the transfer should occur Typically a “trigger file” is used,
placed in a directory near where the actual file will be
Useful for preventing accidental transfer during file write
Configurations: FTP location, authentication, file name
The Framework – Source config
FTP location, authentication, remote directory, file names or file patterns
Java supports regular expressions, so use them
Not all FTP servers support glob expressions
The Framework – local work
Encryption config Public key Whether to delete unencrypted file
afterwards (yes) Decryption config
Private key Passphrase
The Framework – Target Config
Multiple target elements, each with: FTP location, authentication Remote directory
The Framwork - Archiving
Considerations Local vs. remote Append timestamps, other clerical tasks Archiving sensitive data
The Framework – Custom task
Create your own Java class to do some complicated work (i.e., building a file from a database query)
XML configuration: Class name Name/Value properties used by the class
Putting it All Together – Task Interface
A “Task” needs: A definition: settings, configurations A context: what happened before? A result: something that can be
interpreted and acted upon
Task Interface – Task Definition
Say, one of the XML elements you just defined
The Task knows what to do with it
Task Interface - Context
A way to keep track of what happened in previous steps
Most obvious example is, what files are we working with?
Doesn’t need to be too complex
Task Interface – Result
Three values Success/Failure Whether to continue Message for grateful support person
One More Interface - Plan Just a way to build a list of tasks from
a given XML configuration Execute them in order, checking the
result after each task
Bonus points: have “Plan” implement “Task”
phew
Hard part is over Lunch soon!
I feel ya, bro
Quartz
Well known (among Java nerds) library for scheduling jobs
Free as in speech, beer Well documented, tons of features Runs in a clustered environment Free web-app UIs exist You can pay for support if you want
Quartz – Runs stuff
Use Quartz if: You want to run something every five
minutes You want to run something every day You want to run something on demand You want to run something every 15
minutes from noon to 2pm on the fourth day of each month and every second Tuesday
An Alternative to Quartz
Weblogic Job Scheduler Built-in to Weblogic =) Not portable =( Not widely used =( More complicated setup =( Documented by Oracle =(
Quartz – Concepts Job – interface for Java classes, with only
and “execute” method. Quartz can run any class that implements Job.
Trigger – Tells the schedule when to run Groups – Group Jobs for better control Listener – Your own optional class that gets
notified when things run (for logging purposes)
Job Data Map – Configurations or data passed in when the Job is run.
Putting the FTP Framework in Quartz
Create a Java class that implements Job (so Quartz can run it)
Schedule the Job, put your XML file in the Job Data Map so it is accessible by your new class.
Have the class set up a Plan of Tasks based on the XML file, and execute it.
Bonus - Quartz/SOA Interaction Quartz -> SOA
Use the javax.xml.soap libraries to create a simple web service Job
Now you can schedule your SOA composites! SOA -> Quartz
Create a simple class in your scheduler project that triggers a job on-demand in Quartz based on an input job name.
In JDeveloper, right-click and select “Create Web Service”
Now you can kick-off scheduler jobs from SOA!
You’ve been a great audience
Takeaways: Java is your best bet for file transfers if
you’re using Oracle products for integration
Abstract Source/Target/Encryption tasks since they’re all pretty much the same
XML is a good format for configurations since it’s extensible and easy to import to Java
Thanks OWMUG
Takeaways: Quartz is a powerful, free scheduling
tool used by many large organizations Quartz can interact with SOA via web
services, too (or anything Java can interact with- ODI, AS400, SQL, etc.)
select * from QUESTIONS
Libraries and Links (all free!) Quartz - http://quartz-scheduler.org/ Apache Java FTP/FTPS client library -
http://commons.apache.org/net/ JSCH Java SFTP client library –
http://www.jcraft.com/jsch/ CIFS/SMB client library -
http://www.samba.org/ Bouncy Castle Java PGP library -
http://www.bouncycastle.org/ JWatch Quartz Manager UI -
http://code.google.com/p/jwatch/