EASY ORACLE AND WEBLOGIC PROVISIONING & DEPLOYMENT

TODAY

ENTERPRISE DEVOPS. CAN IT BE DONE?

AUTOMATION IS REQUIRED THE VALUE OF COMMUNICATION

ORACLE AND WEBLOGIC CONFIGURATION

HOW DOES IT WORK?

WAT IS THE ISSUE? HOW TE GET STARTED?

INTRODUCTION• More then 20 years experience in IT • As project manager & development

manager at large IT companies and government

• Founder of Enterprise Modules (twitter: @enterprisemodul)

• (Co)developer of Puppet modules for Oracle RDBMS en WebLogic

• bert.hajee@enterprisemodules.com • twitter: @bhajee

MISSION STATEMENTProvide our customers with high quality Puppet modules for all the Enterprise products. Thereby allowing them to exploit all advantages of Puppet not only for their base OS and open source products, but for their entire Enterprise Infrastructure

https://www.enterprisemodules.com twitter: @enterprisemodul

FINALLY...

John looked at the ceiling. Yesterday, after more than one and a half years of preparation, the latest version of the application went live. But it took a lot of effort and sparked a lot of negative energy.

The application was a registration application that used Oracle WebLogic, and stored all data in an Oracle database.

Before getting the application into production, the development team and the operations team where coastally at each other’s throat. Every time an issue was reported, during the tests, they were pointed fingers at each other and calling names.

The operations guys blamed the developers. Saying the only delivered bad code. And that in the end the operations guys would be called in the middle of the night to clean up the mess. The developers, on the other hand, said that the operations team was just splitting hairs and fretting about non important issues. They were just stalling the project. If the operations team hadn’t been so fearfull of changes, the would already be running in production for a long time.

When there was a performance problem, the DBA-er told the developer that he was not using the indexes correct. But the developers blamed the operations team for not providing a correctly configured operating system.

Does anybody recognize these stories? These are stories we often hear from customers. Although the word DevOps has been buzzing in the world of startups, it has been tough to get it to work in larger companies.

ENTERPRISE DEVOPS??

Wouldn’t it be great if we are able to do things in the devops-way?

Wouldn’t it be great if John did not have to wait for one and a half years to get this new release into production? But actually do a new release every week. Or even do a release every day.

Wouldn’t it be great if it didn’t matter if the change was in the application or in the underlying middleware database or operating system? The change would be done quick, consistent and painless.

Wouldn’t it be great if the development team and the operations teams worked together in close cooperation towards attaining these goals.

Or is it just impossible to get devops working in large companies……

Well…. I can tell you it is possible. In this presentation I will take you down the road that John and his team traveled. The steps they took to transform the current status que into a smooth operation where they were able to do multiple deployments per week and make controlled changes to the applications, middleware and operating systems.

This story is based on actual experiences we had with multiple customers, both in the Netherlands and in other parts of Europe.

FAST RELIABLE & CONSISTENT

As I mentioned before, John and his team needed one and a half years to put the new version of the application into production. This time was divided over and his team had needed a half year to put the new version of the application into production. This time of course was divided on the various aspects and stages of such development process. What I did notice was that it proved difficult to put down a fast reliable and consistent environment.....a lot of hand work ...Inconsistent …Sensitive to errorsSmall annoying errorslabor-intensiveTakes a long time

AUTOMATION REQUIRED

For the team things where clear. Automation of the work flow was required.

Application

Problem??WHAT IS THE PROBLEM???

But before we continue. Let’s see what the problem is. How can installing an application be a problem?

Application

Middleware

Problem?WHAT IS THE PROBLEM??

Well, one of the reasons is, you normally need a little bit more then just an application. Most enterprise applications need some sort of middleware.

Application

Middleware

Database

ProblemWHAT IS THE PROBLEM?

And since data is important for large companies. Most of them need a database too.

Application

Middleware

Database

OS

Problem!WHAT IS THE PROBLEM

And you would almost always need an operating system.

Problem!!

Application

Middleware

Database

OS

Network

WHAT IS THE PROBLEM!

And stuff needs to be connected to a network. And as you can see slowly but surely the number of components that need to be installed and configured, increases.

Problem!!!

Application

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

WHAT IS THE PROBLEM!!

Large companies most of the time, cannot afford downtime. So one of the risk mitigation counter measures used a lot it make the infrastructure high available by duplicating everything. And as you can see, the tasks of installing and configuring you IT infrastructure increasingly becomes a challenge.

Problem!!!!

Application

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

WHAT IS THE PROBLEM!!!

But even this is manageable. But when we start to look at the the number of individual settings that need to be controlled, we start to see the real problem. The amount of stuff to manage is huge.

Problem!!!!!

Application

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

WHAT IS THE PROBLEM!!!!

It is not only huge, it is also complicated. Some of the configuration points are interdepended. So when you change the size of the SGA, you also need to change the amount of memory reserved. When you change the port number of a weblogic server, you also need to change the firewall rules to allow communication to the specified IP port.

And we start to see why it has been incredible difficult to get all of the systems in our chain of systems configured correctly.

A SOLUTION

Like the figured out before, they needed a tool. A tool that would help them manage these infrastructure layers and the huge amount of configuration points.

Like the figured out before, they needed a tool. A tool that would help them manage these infrastructure layers and the huge amount of configuration points.

In the team the chose Puppet. Reasons for that where:• Communication• Large and stable player in the market• Big partners including Cisco, VMWare, Microsoft and EMC• Multi-platform including windows• Also for configuration of network and storage components• Support for configuration Oracle databases and WebLogic

Your infrastructure on a blueprintYOUR INFRASTRUCTUE IN A BLUEPRINT

For the people who don’t know, let me explain how puppet works. When you are using using Puppet you describe your infrastructure in sort of a blue print. This blueprint contains a description of all the required components in your infrastructure, all the components on your servers. All the settings you need, are described in your blueprint. For example…

What’s reality?

REALITY?

After Puppet has read your blueprint, it goes to the system. It starts to make pictures of how the components are really configured and if they are installed at all. It makes an inventory of reality of the current system.

Spot the differencesSPOT THE DIFFERENCES

By comparing the blueprint and the pictures, it can make a list of differences. And based on the list of differences, it can make a list of actions that need to be executed. For example: when your blueprint says it needs a datasource in domain ‘production’ in a WebLogic 12.2.1 environment, Puppet will look at its pictures and see no weblogic software is installed and obviously no datasource is configured. The puppet modules will have the knowledge of what the required actions are to install the software, create the domain and create the dataasource.

If however an existing datasource is available, but just the url of the database has changed, puppet will see everything is ok and just change the url property of the datasource.

What is really important, is tha puppet describes the end result and not the procedural steps needed to get there. Because depending on the begin situation, they might be different.

For new install’s and updatesNEW INSTALLS & UPDATES

This is a really handy mechanism. You can use this mechanism to provision a totally new system. Puppet will see a huge difference between the blue print and the pictures and will make a long list of actions to execute.

But if there is just a small change, this mechanism also works. Just the list will be much shorter.

Blueprint for Oracle Table space..ooora_tablespace {'my_app_ts@sid': ensure => present, datafile => 'my_app_ts.dbf', size => 5G, logging => yes, autoextend => on, next => 100M, max_size => 20G, extent_management => local, segment_space_management => auto},}

NEW INSTALLS & UPDATES

Here is an example of a part of a blueprint. This is a definition of an oracle tablespace. Key here is that you don’t have to be a database specialist to understand what is defined here. Later in the presentation I will come back to this.

ABSTRACTION

But how does this solve the issue of the many many connected configuration points in our infrastructure? and the answer is: Abstraction

Abstraction Application

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

In essence Puppet provides a consistent abstraction layer over the real infrastructure components. It allows you to make your infrastructure components into building blocks with a defined interface. Some of the building blocks are small and provide just a small configuration element. An example of this, is the ora_tablespace definition we saw before.

But you can also put those small building blocks into a little bit bigger building blocks and define an interface for your standard configured weblogic domain, or your standard application setup on your database.

Separation of ConcernsApplication

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

DEVMYAPP

10.10.10.100Queue1APPDEV

ns1.dev.comusernamepassword

mw_homedb_home

wls_serverdb_sizeem_size

webserverdb_server

db_server

Another part of the way Puppet helps you is by separating concerns. Separating the what from the how.

Most of the times we need multiple kind of setups. We need a setup for development, we need a setup of production and in between we need multiple setups for regular testing and acceptance testing. For the most parts, these setups are the same. But porbaby you need different IP setting in testing then in production. And maybe we can do with a little bit less space in the database.

Puppet has a mechanism for allowing you to separate those settings from the rest. It is called hiera. Using hiera you can connect certain settings in your infrastructure to values in your hiera key-> value lookup.

Here you see a setup where we have “wired" our setup for a development environment.

For different EnvironmentsApplication

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

TESTMYAPP

10.10.10.100Queue1APPTEST

ns1.dev.comusernamepassword

mw_homedb_home

wls_serverdb_sizeem_size

webserverdb_server

db_server

Because we have extracted the information, it becomes really easy to transform this blueprint into a blueprint for a testing environment. We just have to adjust the specified settings and we can use the same blueprint with a different layover to manage our testing environment.

Through AcceptanceApplication

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

ACCMYAPP

10.10.10.100Queue1APPACC

ns1.dev.comusernamepassword

mw_homedb_home

wls_serverdb_sizeem_size

webserverdb_server

db_server

And acceptance environment….

Finally in Production Application

Middleware

Database

OS

Network

Application

Middleware

Database

OS

Network

PRODMYAPP

10.10.10.100Queue1APPPROD

ns1.dev.comusernamesecret

mw_homedb_home

wls_serverdb_sizeem_size

webserverdb_server

db_server

And of course also our production environment.

DIFFERENT KIND OF ENVIRONMENTS

When John and his team started with puppet and the environment, things worked well. Finally they where able to consistently provision and deploy systems and applications. The started with a Puppet blueprint for production, then the extracted the properties that needed to be different for acceptance and put those into hiera. They gained a lot. But something was missing….

They noticed that a lot of issue still where only discovered when the application was deployed into the acceptance setup. There was still to big of a gap between what the developers used on there development workstations and what was running in production.

What they wanted was to provide all developers with a minified version of the production environment on there own workstation. A setup that didn’t need to be high available and high performance, but when looked at from the application, provide the same interface as a production environment. The same queue’s, the same dataseource, the same tablespaces and the same tables. Just the clustering (for HA) and the sizing would be different.

PORTABLE PLATFORM

An they managed to get that working. It is called the portable setup. Now they can use the same puppet blueprint to:- install a minified version of the setup on a PC. This contains both the database, the weblogic domain and loadbalancers.- Install a test setup where the database is on one machine and the weblogic domain is on an other machine- And a HA setup where they use a RAC oracle database and a 10 node WebLogic cluster to run the application on.

John and his team noticed that a lot of the late issues disappeared. Because the developers had access to the same setup as used in production, a lot of setup issues like missing tablespaces, differently named queue’s and datasources, would be found much earlier in the development process.

PLATFORM CI

Platform Code

Application Code

Because of this easy portable setup, they were able to get two interconnected CI environments.

PROCES CHANGES

Because of this interconnected CI environments, they had a consistent view of the quality of the platform and application. So they became pretty confident. At any moment they were sure they could provision and deploy the latest combination of working platform with a working application.

Based on this confidence, they started to change their processes. Every week they would release a version of their platform puppet code and there application. Depending on the customer’s needs, they would just use this in their development chain, or deploy this to production.

With this process they achieved agility while retaining stability. With this mechanism and process they were able to deploy new versions of the application, provision changes in the WebLogic and Oracle setup and provision patches, both on the OS as well as in the middleware layer.

COMMUNICATION AND COOPERATION

Automating their setups sure helped. But John and his team where certain tooling was not enough to overcome the conflicts and heated discussions between the developers and the operations guys. Something had to be done to improve the collaboration.

When you look at the literature, one of the key enablers of collaboration, is communication. Many development teams use a Kanban board to enable communication about the status of the project. The scrum methodology uses standups for communication and mandates that a team has all required skills and knowledge to do everything needed from idea to delivery.

A BIG TEAM

Based on the scrum methodology, they thought about having one big team. A team that would contain all required skills and be responsible and accountable for the whole process.

But they noticed it didn’t work. The amount of different IT skills is large in large companies. If you would put all those skills in one team, the team would become huge.

Also because of corporate policies it is hard to get all authorizations into that one team. Large companies often require separation of duties and authorities.

So… Although in theory very nice. Not practical for large companies.

COMMON LANGUAGE

But while John and his team where looking at solutions, they noticed the amount of conflicts was decreasing. There were still heated discussion, but they seemed to be targeted at some specific setting.

The use of Puppet seemed to have a huge positive impact on communication as well.

FACILITATES TARGETED DISCUSSIONS

The simplicity of the Puppet language provided a common language between the development team and the operations team. No longer were they fighting over some unseen settngs somewhere, but they were discussing the pro’s and con’s of certain settings.

SMALL DEMO

OUR PUPPET MODULES

OUR MODULES

ORA_CONFIG module

• ora_asm_diskgroup • ora_asm_volume • ora_database • ora_exec • ora_init_param • ora_listener • ora_object_grant • ora_record • ora_role • ora_schema_definition • ora_service • ora_tablespace • ora_user • AND MORE

ORA_INSTALL module

• db_control • db_listener • db_rcu.rb • db_opatch • installdb • installem • install_emagent • installs • opatchupgrade • tnsnames • net • goldengate • client • autostartdatabase

FOR ORACLE WE HAVE GOT

WLS_CONFIG module

• wls_authentication_provider • wls_cluster • wls_datasource • wls_deployment • wls_domain • wls_jms_queue • wls_jms_topic • wls_messaging_bridge • wls_role • wls_saf_imported_destination_object • wls_saf_remote_context • wls_server • wls_workmanager_constraint • AND MUCH MUCH MORE

WLS_INSTALL module

• wls_install::bsu • wls_install::cluster_node • wls_install::domain • wls_install::fmw • wls_install::managed_server • wls_install::nodemanager • wls_install::opatch • wls_install::packdomain • wls_install::software • wls_install::storeuserconfig • wls_install::utils::fmwcluster • wls_install::utils::fmwclusterjrf • wls_install::utils::oimconfig • wls_install::utils::webtier • … AND MORE

FOR WEBLOGIC/FUSION WE HAVE GOT

HOW TO GET STARTED?

IN RETROSPECTIVE

BUSINESS AS USUAL

John was staring at the ceiling. He was just thinking back about a couple of months ago. Then they needed months, or even more than a year for a new version in production. And not only that, but these times would be filled with conflicts and over heated arguments,

But not anymore. They were able to put a release into production any time. It doesn’t matter if it is a functional change to the application, of changes in middleware, database or OS.

Although in case of an emergency, they were able to a release any time, the customers most of the time didn’t need this. So they adjusted their process to a release cycle of a small changes every week and once a month they did a larger release. For the larger release they had more personnel standby for problems, but until now the larger release had been just as stable as the smaller releases.