experiences of soacs
TRANSCRIPT
Copyright © 2014, eProseed and/or its affiliates. All rights reserved. | Confidential
Implementing the UK’s 1st Production
SOA Cloud Service
Kiran Tailor
Global Lead DBA / BI Solutions Architect, CIMA
Simon Haslam
Technical Director, eProseed
1
© CIMA 2
Me and CIMA
• Kiran Tailor , Global Lead DBA/BI Solutions Architect
• Blog : http://blog.puredba.co.uk/
•Twitter : @KiranTailorUK
CIMA and the AICPA are joining forces to create a new
association to represent the entire breadth of the
accounting profession.*
600,000 members and students worldwide. *Offices and Staff across the world. *
Simon Haslam
Technical Director, eProseed UK
• Platform / Infrastructure Architect
• Using Oracle products since ~1995 (Oracle7)
• Formerly UKOUG App Server & Middleware SIG Chair
Copyright © 2016 eProseed and its affiliates. All rights reserved. © CIMA 4
• Our Previous and New Architectures
• Provisioning DBCS and SOACS
• Tailoring Cloud Services
• Experiences and Oracle Cloud Tips
• Monitoring in the Cloud and Reporting
Our Previous System – All Running On-Premises
Oracle Databases
CRM
Biztalk
GP
SQL Server
Hybrid Cloud
Public Cloud
Private Cloud
New System
SiebelCRM
Oracle Databases
Exalytics
Oracle ERPOracleDBCS
SOACS
Oracle Database Cloud Service
Oracle SOA Cloud Services
Financials
Our Integration
SOA Composites
Fusion Cloud Services
Connectivity S
erv
ices
SOA Infra
Integration Tables
Customer Processes Invoice Processes
Activity Services
On-Premises
Payment Processes
Connectivity S
erv
ices
SQL Server
© CIMA
Batch
UpdateNotifications Fault
Handling
Copyright © 2016 eProseed and its affiliates. All rights reserved. © CIMA 8
• Our Previous and New Architectures
• Provisioning DBCS and SOACS
• Tailoring Cloud Services
• Experiences and Oracle Cloud Tips
• Monitoring in the Cloud and Reporting
Copyright © 2016 eProseed and its affiliates. All rights reserved.
WHAT THIS PRESENTATION IS IS NOT
• This is not a step-by-step guide about provisioning SOACS or DBCS manually through console
• We focus on some of the decisions you have to make, lessons learnt and tips
• You will notice some of these are same as for on-prem… many of your old skills are still useful
Copyright © 2016 eProseed and its affiliates. All rights reserved.
TOPOLOGIES
• Before eProseed CIMA were in development as single instance SOACS:–WebLogic Managed Server directly open to internet
–No clustering / failover options (other than VM restart)
–Can’t easily add OTD later without re-provisioning
–Not really an ‘Oracle intended’ design for production
10
SOAWebLogic
DBCS
Copyright © 2016 eProseed and its affiliates. All rights reserved.
TOPOLOGIES
• So… we introduced OTD–OTD is only internet-facing component
• Hardened, good track record/few security patches
• Option for WAF features etc
–Option to add second OTD node• Note: actually independent – different to on-prem config.
–Allows future scale-out and rolling patching withoutchanging any end points• Scale out could even be done online
11
SOAWebLogic
DBCS
OracleTraffic
Director
Copyright © 2016 eProseed and its affiliates. All rights reserved.
TOPOLOGIES – ACROSS ENVIRONMENTS
• Hard to justify OTD cost on Dev but we want all envs to behave the same
• Wire Dev SOACS MS into Test OTD - unsupported but quite practical
12
SOAWebLogic
DBCS
OracleTraffic
Director
PROD
SOAWebLogic
DBCS
OracleTraffic
Director
TEST
SOAWebLogic
DBCS
DEV
Copyright © 2016 eProseed and its affiliates. All rights reserved.
STORAGE TIPS
• Both DBCS and SOACS need Oracle Cloud Storage for backups
– Note: SOACS runs on “full PaaS” JCS – no virtual image option
• Provisioning containers – how to subdivide & manage storage?
– We provision per instance – exactly same name for simplicity
– Drop storage with instance (if you don’t want it make sure you don’t pay for old backups)
• Initial Sizing
– Size bigger rather than smaller
– Can extend in DBCS later though but you do need a restart of the database
Copyright © 2016 eProseed and its affiliates. All rights reserved.
OPC USER TIPS
1. ONLY use lower case email addresses. If mixed case:
– Some displays, e.g. user listing, show it as lower case
– Some interfaces are happy with lower case, (probably authentication) insist on it mixed
2. Use a non-email name for the primary provisioning account
– All objects are created with this in their identifier, e.g.• Better to have to reference /Compute_exampleco/exampleco/…
• Than /Compute_exampleco/[email protected]/…
• (especially with multiple admins, plus admins leave)
3. Limit the use of the provisioning/owner account to the provisioning tools
– All admins should have their own logins
14
Copyright © 2016 eProseed and its affiliates. All rights reserved.
TLS (SSL)
• SOA CS (JCS) – WebLogic and OTD has SSL configured but only with demo certificates
=> not suitable for production usage, especially OTD since that’s public
• Inputs to configuring SSL:
– Signed certificate including any intermediates: either by public CA (simplest) or internal CA
– Private key: keep it safe, but keep it outside SOACS for simplicity (to save re-signing… at least for now)
– Any additional certificates to be trusted: typically the internal CA
• We currently switch from OPSS Key Store Service to Identity and Trust JKS on disk
– In domain Custom Identiy & Cutom Trust | JKS | <DOMAIN_HOME>/security/trust-v5.jks
– setUserEnv.sh -Djavax.net.ssl.trustStore=${DOMAIN_HOME}/security/trust-v5.jks
– (we may consider OPSS later since java can reference KSS too)
15
eProseed Accelerator for Oracle Cloudconfigures your SSL for OTD HTTPS listener,
Admin Server & Node Manager listening port
Copyright © 2016 eProseed and its affiliates. All rights reserved.
SSL TRUST TIP
16
DO NOT INSTALL CERTS FROM 3rd PARTY SERVICE PROVIDERS… YOU ARE PROBABLY MISSING SOMETHING!
Otherwise you will build a brittle integration that could break outside
of your control
Beware blogs and even MOS notes or SRs advising otherwise!!!
Cloud providers typically use wildcard certificates.Wildcards are not enabled out of the box
See next slide…
Copyright © 2016 eProseed and its affiliates. All rights reserved.
WEBLOGIC CONFIGURATION FOR WILDCARD SSL
• When checking SSL certificate, WebLogic looks at the CN
– CN = login.eproseed.com <= this works fine by default
– CN = *.eproseed.com <= this is a wildcard certificate
• By default, even in 12.2.x WebLogic will not trust wildcards or certificate SANs
– Use this weblogic.security.utils.SSLWLSWildcardHostnameVerifier in domain Custom Hostname Verifier for Admin Server and all Managed Servers.
– Don’t ask me why this is not default by now!
• This hostname verifier has been available since 10.3.6 (probably backports for earlier, otherwise roll your own)
• Same is true for Subject Alterative Names (SANs) which is what we’re supposed to use instead of wildcards
Copyright © 2016 eProseed and its affiliates. All rights reserved.
AUTOMATED PROVISIONING – REST API
• All operations (& more) that can be done through console can also be done through REST APIs
• Can call REST API via most modern tools, e.g. curl, Postman in Firefox etc
• Authentication
– Most APIs use header tokens
– Compute Service APIs need you to call authentication first & get a token
18
Copyright © 2016 eProseed and its affiliates. All rights reserved.
REST API
19
Copyright © 2016 eProseed and its affiliates. All rights reserved.
EXAMPLE PAYLOAD
{
"serviceName": "soacs-unit-test-1",
"level": "PAAS",
"topology": "soa",
"trial": false,
"subscriptionType": "MONTHLY",
"description": "soacs-unit-test-1",
"provisionOTD": true,
"cloudStorageContainer":"Storage-orclnnsoa/soabackup",
"cloudStorageUser":"soacs.Storageadmin",
"cloudStoragePassword":"welcome1",
"parameters": [
<see next slide>
]
}
Copyright © 2016 eProseed and its affiliates. All rights reserved.
EXAMPLE PAYLOAD (CONTD.){
"version": "12.1.3",
"edition": "SUITE",
"managedServerCount": "1",
"templates": "full",
"adminUserName": "weblogic",
"adminPassword": "welcome1",
"connectString": "example.com:1535:orcl12c",
"dbaName": "sys",
"dbaPassword": "fmwpwd1",
"shape": "oc3m",
"VMsPublicKey": "ssh-rsa ...",
"type": "weblogic"
},{
"listenerPortsEnabled": true,
"loadBalancingPolicy": "LEAST_CONNECTION_COUNT",
"otdAdminUserName": "otdadmin",
"otdAdminPassword": "welcome1",
"shape": "oc3",
"type": "OTD"
}
Copyright © 2016 eProseed and its affiliates. All rights reserved.
IaaS VM
EPROSEED ACCELERATOR FOR ORACLE CLOUD
22
SOA
CS
Oracle Cloud Service Managers
JCS
DB
CS
Sto
rage
Net
wo
rk
Co
mp
ute
Identity
eProseedAccelerator
for Oracle Cloud
Creation
Vir
tual
M
ach
ine
Vir
tual
M
ach
ine
Vir
tual
M
ach
ine
Config. Report
Copyright © 2016 eProseed and its affiliates. All rights reserved.
AUTOMATION & PUTTING IT ALL TOGETHER
23
Env masterJSON
BlueprintJSON
Oracle Cloud configJSON
Cookbook(Recipes)
PayloadJSON
Env type
Env no
Blueprint type
DBCS override
Instances
Runlist
Sizing
etc
OPC DC
API URLs
Customer DetailsJSON
Domain, Auth, DCRESTcalls
© CIMA 24
• Our Previous and New Architectures
• Provisioning DBCS and SOACS
• Tailoring Cloud Services
• Experiences and Oracle Cloud Tips
• Monitoring in the Cloud and Reporting
Copyright © 2016 eProseed and its affiliates. All rights reserved.
TAILORING CLOUD SERVICES
• Additional SSH keys
• User accounts (Linux & WebLogic)
• Firewall / VPN
• (Backup schedule)
25
Connecting from Windows
User and Key Management
# useradd ukoug
# mkdir /home/ukoug/.ssh
The next step we copy and paste in the public key we have generated for
the new user:
# echo "<key here ssh-rsa…" > /home/ukoug/.ssh/authorized_keys
To check# cat /home/ukoug/.ssh/authorized_keys
Compute Cloud Service
Compute Cloud Service
Compute Cloud Service
Tunneling
SQL Connection
Copyright © 2016 eProseed and its affiliates. All rights reserved.
SSH TUNNELLING IS 1ST STEP BUT SEE…
Tomorrow, 11:35, Hall 6B
Copyright © 2016 eProseed and its affiliates. All rights reserved. © CIMA 33
• Our Previous and New Architectures
• Provisioning DBCS and SOACS
• Tailoring Cloud Services
• Experiences and Oracle Cloud Tips
• Monitoring in the Cloud and Reporting
Copyright © 2016 eProseed and its affiliates. All rights reserved.
FAILING BACKUPS
1. Domain locks
2. Admin user password change
34
Copyright © 2016 eProseed and its affiliates. All rights reserved.
1. FAILING BACKUPS IF ADMIN HAS LOCK ON WEBLOGIC CONFIGURATION
• Backup tool takes a domain lock at start, releases at end
– Primarily (presumably) to stop you changing domain during backup
– Also if an administrator has started a session there could be changes in configuration that have not yet been activated
• If an Admin has the lock then the backup tool can’t get it
A better approach might be just to force discard of changes & release of lock? Debatable.
35
Backups – Domain Configuration Lock
Backups – Domain Configuration Lock
Backups – Domain Configuration Lock
Copyright © 2016 eProseed and its affiliates. All rights reserved.
2. FAILING BACKUPS AFTER PASSWORD CHANGES
• OPC Passwords (i.e. those in an identity domain) expire monthly
• You can change the password when logging into the cloud console
• But…
39
DON’T IGNORETHESE EMAILS!
Backups – Password Changed
Backups – Password Changed
• Same change in DBCS
Backups – Password Changed
• Update the Wallet
- sudo –s
- /var/opt/oracle/bkup_api update_wallet --password=new-password
• Validate in oss.cfg
Copyright © 2016 eProseed and its affiliates. All rights reserved. © CIMA 43
• Our Previous and New Architectures
• Provisioning DBCS and SOACS
• Tailoring Compute Cloud Service
• Experiences and Oracle Cloud Tips
• Monitoring in the Cloud and Reporting
Copyright © 2016 eProseed and its affiliates. All rights reserved.
EM CC MONITORING
44 DB CS instance
SOA CS instance
44
OTD
SOA
Oracle Financials
Cloud (ERP)
DB EE
CIMA Data Centre
SQL Server
Paypaletc
Siebel
EM 13cCloud Control
Hybrid CloudGateway
EMAgent
EMAgent
EMAgent
• Prod• Test• Dev
Copyright © 2016 eProseed and its affiliates. All rights reserved.
MONITORING & NOTIFICATIONS
• Target types
– SOA
– WLS
– OTD
– Database
– Hosts
– Exalytics
– TimesTen
45
Copyright © 2016 eProseed and its affiliates. All rights reserved.
ALERTING
46
Copyright © 2016 eProseed and its affiliates. All rights reserved.
REPORTING
• Daily Report about integrations
• Scheduled emails to IT Team/Managers
• Automatically raises incidents (check?)
47
Copyright © 2016 eProseed and its affiliates. All rights reserved.
EXPERIENCE & LEARNING POINTS
• Create a Hybrid Cloud Agent EM user and private key (ssh-keygen -b 2048 -t rsa)
• Add the EM Server (IP Hostname) to host
• Bug 23013302 in EM OPC Agent - 'fproxy forwarder' process is utilizing 95 to 100% CPU
• According to Oracle certifications SOACS 12.1.3 is not supported with EM 13.1
• Error Hospital Fix – Copy Jar files manually
• Weblogic Domain Refresh
• OTD SNMP Port for availability status?/ Firewall
• Downtime (Oracle Patching and Changes)
48
Now certifiedFixed in 13.2
Copyright © 2016 eProseed and its affiliates. All rights reserved. © CIMA 49
• Our Previous and New Architectures
• Provisioning DBCS and SOACS
• Tailoring Compute Cloud Service
• Experiences and Oracle Cloud Tips
• Monitoring in the Cloud and Reporting
© CIMA 50
Benefits
• Provisioning – 50% Faster
• Enhanced monitoring – 20% Reduction in Man-hours
• 50% Reduction in TX completion time
• Security
• Backups
• Scale Up – Scale Down
• Reduce Head Count ??
© CIMA 51
Oracle Excellence Award 2016 – Cloud Innovation
Copyright © 2016 eProseed and its affiliates. All rights reserved.
HOW TO CONTACT US
53
@simon_haslam@kirantailoruk