manage your odi development cycle – odtug webinar

[email protected] www.rittmanmead.com @rittmanmead

Jerome Francoisse | ODTUG Webinar | January 2016

Manage Your ODI Development Cycle

1


Jérôme Françoisse

• Consultant for Rittman Mead• ODI Trainer

• Providing ODI support on OTN Forums• ODI 12c Beta Program Member• Blogger at http://www.rittmanmead.com/blog/• Email : [email protected]• Twitter : @JeromeFr

3

http://www.rittmanmead.com/blog/

mailto:[email protected]


About Rittman Mead

4

•World’s leading specialist partner for technical excellence, solutions delivery and innovation in Oracle Data Integration, Business Intelligence, Analytics and Big Data

•Providing our customers targeted expertise; we are a company that doesn’t try to do everything… only what we excel at

• 70+ consultants worldwide including 1 Oracle ACE Director and 2 Oracle ACEs

•Founded on the values of collaboration, learning, integrity and getting things done

Optimizing your investment in Oracle Data Integration

•Comprehensive service portfolio designed to support the full lifecycle of any analytics solution

5

20% discount on rittmanmead.com/save-on-training/


Successful ODI project

6



6

• Having good developers is great



6

• Having good developers is great• … but it’s not enough!



6


• We also need :



6


• We also need :- Assurance that the code deployed in production is correct



6


• We also need :- Assurance that the code deployed in production is correct- Possibility to restore any previous state



6


• We also need :- Assurance that the code deployed in production is correct- Possibility to restore any previous state- A way to manage concurrent development


Agenda

7

• Lifecycle Management- Architecture - Promotion - Versioning


ODI 12c Key Components

8

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository (Development)

Execution

Execution Repository (Production)

• ODI Studio (the main development IDE)

- Designer Navigator - Operator Navigator - Topology Navigator - Security Navigator • ODI Agents (executing mappings and packages)

• ODI Console (web-based interface)

• ODI Repositories (contain definitions of models, mappings and other objects)


Physical and Logical Topology

9

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical Contexts Logical



9

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

HR

Contexts Logical

SH



9

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

HR

Contexts Logical

DEV

TEST

PROD

SH


Architecture

10

Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution



How many repositories do I need?

11

Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution


Execution

Execution Repository (Test)



12

Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository



13

Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository

Execution

Execution Repository (Pre-Production)

Security

Topology

Versioning

Master Repository



14

Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository (Hotfix)

Execution


Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository



15

Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution


Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository

Execution

Execution Repository (Smoke Test)



16

Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository

Execution

Execution Repository (Smoke Test)

Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository


Problems

17


Problems

17

• As a developer, how do I create a consistent set of objects for a release?


Problems

17


Execution

Execution Repository


Problems

17


Models

Projects

Execution

Work Repository

Execution



Problems

17


Models

Projects

Execution

Work Repository

Execution


• As a release manager, how do I easily


Problems

17


Models

Projects

Execution

Work Repository

Execution


• As a release manager, how do I easily- promote code from one environment to another?


Problems

17


Models

Projects

Execution

Work Repository

Execution


• As a release manager, how do I easily- promote code from one environment to another?- keep track of these sets of objects?


Problems

17


Models

Projects

Execution

Work Repository

Execution


• As a release manager, how do I easily- promote code from one environment to another?- keep track of these sets of objects?- roll back to a previous release?


The Solution – Deployment Archive (DA)

18



• Archive file containing a set of consistent ODI objects

18



• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)

18



• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)• 3 types :

18



• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)• 3 types :- Initial

18



• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)• 3 types :- Initial- Patch

18



• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)• 3 types :- Initial- Patch- Rollback

18



• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)• 3 types :- Initial- Patch- Rollback• Development or Execution

18


Example

19

Development environment

QA environment (Deployed repository)

Developer

Release Manager


Example

19



Developer

Release Manager

Creates Initial Release 1.0.0.0


Example

19



Developer

Release Manager


Initialize repository with Release 1.0.0.0


Example

19



Developer

Release Manager



Creates Patch Release 1.1.0.0

to add a features


Example

19



Developer

Release Manager




to add a features

Apply Patch Release 1.1.0.0

and create rollback DA


Example

19



Developer

Release Manager

Bug




to add a features




Example

19



Developer

Release Manager

Bug




to add a features



Apply Rollback Release 1.1.0.0


Example

19



Developer

Release Manager

Bug




to add a features





to fix the previous bug


Developer / Administrator

20

• Create a deployment archive



21

• Create a deployment archivefrom VCS label

• Best Practice :Link documentation and DDL statements to be applied



22

• Create a deployment archive from selected objects

• Best Practice :Mention the bug number in the description



23

• No need to think about development or execution-only environment


Release Manager

24

• Apply Deployment Archives


Release Manager

25

• Initial DA• Can choose from

development or execution-only archive


Release Manager

26

• Patch DA• Can choose from

development or execution-only archive

• Best Practice :Always create a Rollback DA!


Release Manager

27

• See applied DAs – Inventory


Release Manager

27

• See applied DAs – Inventory

• See available DAs – View Details


Release Manager

28

• Rollback DA


Automation

29

• Available through ODI SDK

• Scenarios :- Import and be sure you always generate a rollback DA


Automation

29


• Scenarios :- Import and be sure you always generate a rollback DA- Import daily into a smoke test environment


Automation

29


• Scenarios :- Import and be sure you always generate a rollback DA- Import daily into a smoke test environment- Import the same release in Hotfix and Production at the same

time


Other Problems

30


Other Problems

30

• How to rollback on the Development environment?


Other Problems

30

• How to rollback on the Development environment?• How to restore delete objects in the Development

environment?


Other Problems

30

• How to rollback on the Development environment?• How to restore delete objects in the Development

environment?

• How to develop concurrently?


The Solution - VCS

31


The Solution - VCS

31

• Integration with Subversion (SVN)


The Solution - VCS

31

• Integration with Subversion (SVN)- Everything is done through ODI Studio


The Solution - VCS

31

• Integration with Subversion (SVN)- Everything is done through ODI Studio- Git support in the next release


The Solution - VCS

31

• Integration with Subversion (SVN)- Everything is done through ODI Studio- Git support in the next release- Tagging


The Solution - VCS

31

• Integration with Subversion (SVN)- Everything is done through ODI Studio- Git support in the next release- Tagging- Branching / Merging


VCS - Features

32

• Add an object to VCS


VCS - Features

33

• Create a new version of an object


VCS - Features

34

• View and compare versions


VCS - Features

35

• Restore a version


VCS - Features

36

• Tags


VCS - Features

36

• Tags- Snapshot of consistent set of versioned objects


VCS - Features

36

• Tags- Snapshot of consistent set of versioned objects- Ready for promotion – Deployment Archive


VCS - Features

36

• Tags- Snapshot of consistent set of versioned objects- Ready for promotion – Deployment Archive- … or for branching


VCS - Features

36

• Tags- Snapshot of consistent set of versioned objects- Ready for promotion – Deployment Archive- … or for branching- Created from


VCS - Features

36

• Tags- Snapshot of consistent set of versioned objects- Ready for promotion – Deployment Archive- … or for branching- Created from‣ Whole repository (Full Tag)


VCS - Features

36

• Tags- Snapshot of consistent set of versioned objects- Ready for promotion – Deployment Archive- … or for branching- Created from‣ Whole repository (Full Tag)‣ Selected Objects and

dependencies (Partial Tag)


VCS - Features

37

• Branches


VCS - Features

37

• Branches- Created from a Tag


VCS - Features

37

• Branches- Created from a Tag- Populate a new environment – Master and Work repository


VCS - Features

37

• Branches- Created from a Tag- Populate a new environment – Master and Work repository- For a new feature or a new team


VCS - Features

37

• Branches- Created from a Tag- Populate a new environment – Master and Work repository- For a new feature or a new team- Can be merged back in the trunk


VCS - Features

38


VCS - Features

39


VCS - Features

40


VCS - Working on multiple releases

41

Trunk



41

Trunk

New features for Release 1



41

Trunk 1




41

Trunk 1 2




41

Trunk 1 2 3




41

Trunk 1 2 3

Bug Fixes for Release 1New features for Release 1



41

Trunk

Branch 1

1 2 3




41

Trunk

Branch 1

Branching

1 2 3

4




41

Trunk

Branch 1

Branching

1 2 3

4

Bug Fixes for Release 1





41

Trunk

Branch 1

Branching

1 2 3 5

4






41

Trunk

Branch 1

Branching

1 2 3 5

4 6






41

Trunk

Branch 1

Branching

1 2 3 5

4 6 7






41

Trunk

Branch 1

Branching

Release 11 2 3 5

4 6 7




8



41

Trunk

Branch 1

Branching

Release 11 2 3 5

4 6 7 9




8



41

Trunk

Branch 1

Branching

Release 1101 2 3 5

4 6 7 9

Merging




8



41

Trunk

Branch 1

Branching

Release 1101 2 3 5

4 6 7 9

Merging



New features for Release 2New features for Release 1

8



41

Trunk

Branch 1

Branching

Release 1101 2 3 5

4 6 7 9

Merging

11



New features for Release 2New features for Release 1

8


VCS - Working with different cycles

42

Trunk



42

Trunk 42



42

Trunk

Branch 1

42



42

Trunk

Branch 1

Branching

42

43



42

Trunk

Branch 1

Branching

42

43 44



42

Trunk

Branch 1

Branching

42

43 44

Branch 2



42

Trunk

Branch 1

Branching

42

43 44

Branch 2 45

Branching



42

Trunk

Branch 1

Branching

42

43 44

Branch 2 45

Branching

46



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46

Merging

48



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46

Merging

48 49



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46

Merging

48 49

50



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46

Merging

48 49 51

50



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46

Merging

48 49

52

51

Merging

50



42

Trunk

Branch 1

Branching

42

43 44 53

Merging

47

Branch 2 45

Branching

46

Merging

48 49

52

51

Merging

Merging

50



42

Trunk

Branch 1

Branching

42

43 44 53

Merging

47

Branch 2 45

Branching

46

Merging

48 49

52

51

Merging

Merging

54

50



42

Trunk

Branch 1

Branching

42

43 44 53

Merging

47

Branch 2 45

Branching

46

Merging

48 49

52

51

Merging

Merging

54

5550



42

Trunk

Branch 1

Branching

42

43 44 53

Merging

47

Branch 2 45

Branching

46

48 49

52

51

Merging

54

5550



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46

48 49

52

51

Merging

Branch 3 50



42

Trunk

Branch 1

Branching

42

43 44

Merging

47

Branch 2 45

Branching

46

48 49

52

51

Merging

Branch 3

Branching

50

[email protected] www.rittmanmead.com @rittmanmead 43

Summary

• ODI - Deployment Archives

- Promote, Rollback and keep track of your code - For development or execution-only environment - Easy for Developers and Release Managers - Can be automated

• ODI - VCS

- Be able to get back to any version to object - Branching for Team development - Cannot be automated


Further Reading / Testing

44

• ODI Lifecycle- Managing versions in SVN

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_21 - Parallel Development using SVN Branches

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_22 - ODI Getting Started VM 12.2.1

http://www.oracle.com/technetwork/middleware/data-integrator/odi-demo-2032565.html

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_21

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_22

http://www.oracle.com/technetwork/middleware/data-integrator/odi-demo-2032565.html


Questions?

45

manage your odi development cycle – odtug webinar

Data & Analytics