application engine etl
DESCRIPTION
Using the PeopleTools delivered with PeopleSoft, create a fully configurable ETL tool to connect PeopleSoft with external databases.TRANSCRIPT
![Page 1: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/1.jpg)
Application Engine ETL – Direct access to external databases from PeopleSoft made easy
Session #08714Ocotber 6, 2011
![Page 2: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/2.jpg)
Your Presenters David Sexton PeopleSoft ver. 5.01 Senior Systems Analyst “Application Engine ETL – Direct access to external databases from
PeopleSoft made easy”, presentation at HEUG Alliance ’11, March, 2011.
“Justifying a Data Governance Initiative” presentation at HEUG Alliance ’11, with Lumen Data, March, 2011.
“Justifying a Data Governance Initiative” presentation at Oracle Open World ’10, with Lumen Data, September, 2010.
“Putting the Service into SOA – Consuming WSDLs in PeopleSoft”, presentation at HEUG Alliance ‘09, Anaheim, March, 2009.
“Web Services for PSQuery”, presentation at Oracle Open World, September, 2008.
“Web Services for PSQuery”, presented at HEUG Alliance ‘08, March, 2008.
“Extending Applications with Java”, presented at HEUG Alliance ‘07, March, 2007.
“Real World Web Services Integration from PeopleSoft”, presented at HEUG Alliance ‘07, March, 2007.
![Page 3: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/3.jpg)
University of Utah
28,000 students 18,000 employees Joining PAC 12 2002 Olympic Opening and
Closing Ceremonies Tier 1 Research Instition Medical School and Hospital 30 minutes from….
+ =
![Page 4: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/4.jpg)
PeopleSoft Timeline Sep 97 : SA Campus Community / Admissions / Recruitment
Jul 98 : FS Purchasing / Accounts Payable / General Ledger
Oct 98 : Human Resources
Apr 99 : Go live HE Upgrade 7.0 to 7.5 / Time & Attendance (KRONOS)
May 99 : HR Base Benefits / Payroll
Apr 00 : HE HR/SA Upgrade 7.5 to 7.6 / SA Financial Aid
May 00 : SA Student Financials
Feb 01 : FS Upgrade 6.0 to 7.02
Oct 02 : HE HR/SA Upgrade 7.6 to 8.0 / Enterprise Portal 8.4
Dec 03 : Upgrade 8.4 to 8.8 Enterprise Portal
July 04 : FS Upgrade 7.02 to 8.4
Mar 06 : HE HR/SA Upgrade 8.9
Feb 10 : FS Upgrade 8.4 to 9.0
Oct 10 : HR/SA Upgrade 9.0
PeopleTools 8.50.12
We have been working with PeopleSoft for awhile
![Page 5: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/5.jpg)
!!!!!! Disclaimer !!!!!! This presentation is based on our
experience This presentation contains techniques
only endorsed by us, and not necessarily by our employers
Hopefully this presentation encourages you to share your experience at next year’s Openworld conference.
If you have questions, please ask!
![Page 6: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/6.jpg)
Overview
What is App Eng ETL ? Why we developed it
Why not SOA/ Web Services Advanced Use Case What is required DEMO Where you can get it
![Page 7: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/7.jpg)
App Eng ETL – what is it
We create an application engine program that leverages the PeopleSoft record object to dynamically create insert statements. These statements are executed against the external database through a JDBC connection which is instantiated in PeopleCode.
![Page 8: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/8.jpg)
App Eng ETL – what is it
Transform the data in the PeopleSoft record/view
Extract the data from PeopleSoft using the SQL and Record objects
Load the data using a JDBC connection to the external databse
![Page 9: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/9.jpg)
App Eng ETL Why ?
Started MDM/Data governance/ Integration project Nov 2009
Session Justifying a Data Governance Initiative: Get Approval the First Time #29312 Monday Mar 28 4:30 – 5:30
People want Data ! The more the better
We wanted something that was Easy to maintain/reusable Easy to consume
![Page 10: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/10.jpg)
App Eng ETL Why not Web Services?
SOA requires public services, data sets your organization has declared public
https://wiki.doit.wisc.edu/confluence/display/EBOS/Repository+Home
We’re not there yet Each data request must be approved individually
Each integration is point to point
Consuming web services (wsdl) is not a common skill set (on our campus)
People are excited about it, but can they use it today?
Thank You to Jim Phelps at Wisconsin
![Page 11: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/11.jpg)
App Eng ETL Why not Web Services?
Web Services are best suited where the data request is high frequency
And the response data is low bandwidth Good use of web services
![Page 12: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/12.jpg)
App Eng ETL Why ?
• Build an SQR• Creates a file• Get permissions on the destination server• SFTP the File
Destination does
We do
Custom process to load the file into their database
We do
• Create View• Configure – 1 page• Run a process
Destination doesNothing – data shows up in their data base
![Page 13: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/13.jpg)
App Eng ETL Configure
![Page 14: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/14.jpg)
App Eng ETL Run
![Page 15: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/15.jpg)
App Eng ETL Positives
All integration objects are managed/show on compare reports SQL is in a view – provides a layer or abstration, should
not be impacted by upgrades All integrations are easily visible Low cost per additional integration
No additional coding, configuration only Less moving parts
Customers LOVE it !
![Page 16: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/16.jpg)
App Eng ETL what is required ?
1) Server name where external DB is, DB sid
Jeff.Robbins.awesome testschema
2) Username/Password for the external DB
3) /PSHOME/Appserv/Classes directory contains
driver manager (“java.sql.DriverManager”) The MySql driver (“com.mysql.jdbc.Driver”)
![Page 17: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/17.jpg)
App Eng ETL what is required ?
4) Record/View in Peoplesoft containing the data to be integrated, all fields
will be sent
5) A record on the external database with the same structure as the PeopleSoft record/view
field names, field types, field lengths of the source and target table must match EXACTLY Generate the DDL from the PeopleSoft
record/view, send to external database
![Page 18: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/18.jpg)
Demo configure and run
![Page 19: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/19.jpg)
WHAT ABOUT SECURITY?Gee Dave, this sounds great, but is the
transmission of the data secure? Muliple options to secure the tranmission
Oracle Wallet Talk to your Sales Rep Encyrption options at the JDBC level Source and destination must have Oracle Wallet
Other Vendors supply JDBC drivers that have encrytpion capabilities
Customs options we have investigated Custom Java using ports and SSL SSH tunneling with JDBC*
http://www.ibm.com/developerworks/data/library/techarticle/dm-0312lurie/index.html
![Page 20: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/20.jpg)
SSH Tunnel
![Page 21: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/21.jpg)
Key Code PiecesSSH script
# Referenced in UU_DATA_PUSH application engine. Used for # integrating PeopleSoft data to remote systems. This script # sets up a secure tunnel to a remote host to connect to a # remote database securely. local_port=$1 remote_host=$2 remote_database_host=$3 remote_database_port=$4 ssh -f -g -L $local_port:$remote_database_host:
$remote_database_port $remote_host sleep 60
![Page 22: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/22.jpg)
Key Code Pieces Call SSH script from PeopleCode
&script_path = &path | "ssh_tunneling.sh"; &local_port = &properties.get(UU_DP_RQST_AET.URL | ".local_port"); &remote_host = &properties.get(UU_DP_RQST_AET.URL | ".remote_host"); &remote_database_host = &properties.get(UU_DP_RQST_AET.URL |
".remote_database_host"); &remote_database_port = &properties.get(UU_DP_RQST_AET.URL |
".remote_database_port"); &shell_command = "/usr/bin/ksh -c """ | &script_path | " " | &local_port | " "
| &remote_host | " " | &remote_database_host | " " | &remote_database_port | """";
&processId = Exec(&shell_command, %Exec_Asynchronous + %FilePath_Absolute);
![Page 23: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/23.jpg)
Questions?
![Page 24: Application Engine ETL](https://reader036.vdocuments.mx/reader036/viewer/2022062405/555e9ec2d8b42a0d738b5a10/html5/thumbnails/24.jpg)
Contacts David Sexton
Senior Systems Analyst
University Information Technology
University of Utah