Faster Sorting, Manipulation, Reporting & Test Data - Speed High-Volume File Processing In/Outside Natural

www.cosort.comCopyright 2006, IRI, Inc.

ISV - HQ in Melbourne, FLFounded 1978 – Manhasset, NY

World’s 1st commercial sorts for: CP/M, DOS, UNIX, Windows, Linux

Experts in sorting and data manipulation

Recommended by all UNIX H/W vendors

Embedded by leading ISVs like:Cincom, Clerity, EDIWatch, Experian, Fiserv, Kalido,Mereo, Sabre, SPSS, ViPS

30+ international support offices

Who We AreInnovative Routines International

(IRI), Inc.

www.cosor t.com

1-800-333-SORT

What We Sell

CoSORTCoSORTHigh Performance Sorting, Reporting, ETL

RowGenRowGenSafe Test Data in Custom File Formats

FAst extraCT (FACT)FAst extraCT (FACT)High Performance Oracle Unload

netCONVERTnetCONVERT Legacy File Migration using Copybook layouts

Logon SecurityLogon SecurityGranular UNIX Account Access Control and Audit

x-PRESSx-PRESSComprehensive Cross-Platform Compression Suite

Permitas Permitas Software Licensing Libraries and Management

CoSORT Platforms

ALL 32 & 64-bit UNIX (AIX, HP-UX, Solaris, Tru64, IRIX, MP-RAS, DG/UX, SINIX, ptx, etc.)

ALL 32 & 64-bit LINUX(RHEL, SLES, Debian, Fedora,

Debian, Mandrake, Gentoo, Turbo, WOW, AsianUX, Ubuntu, etc.)

Windows XP, NT, 2K/3, DC

IBM i, p, x & z Series

When You’d Want:

CoSORTCoSORT

Fast, Large File TransformationFast, Large File TransformationSelect, Sort, Join, Convert, Aggregate, Reformat

Legacy Sort MigrationLegacy Sort MigrationFrom: VS/VSE JCL, SS Unix, Natural Sort, et al

Data and File Format ConversionData and File Format Conversione.g. EBCDIC to ASCII, MF-ISAM to CSV

Custom Reporting & Hand-offCustom Reporting & Hand-offMulti-target/format, segmented detail/summary BI

DB Loader AccelerationDB Loader AccelerationPre-sort flat files on primary index key

ETL Tool & Application AccelerationETL Tool & Application AccelerationSort & Metadata Hooks for DataStage, Informatica

RowGenRowGen

Test Data & Safe File Synthesis Test Data & Safe File Synthesis

FAst extraCT (FACT)FAst extraCT (FACT)

Oracle Unload, Reorg & ETLOracle Unload, Reorg & ETL

CoSORT Functionality

High-Performance Record ProcessingHigh-Performance Record Processing – Sort, Copy, Merge, Join, Check– Input / Output Conditional Filter/Select, De-Dupe– Aggregate (Sum, Min, Max, Count, Average), Sequence– Cross-Calculate (Expression Logic)– Segment, Re-map, Re-format, Report (Custom Layouts)

File Integration & TransformationFile Integration & Transformation– Sequential files (Line, Record, Variable)– Unisys VB & Blocked format files– MF-ISAM & ACUCOBOL Vision (Index) files– Named & Un-named Pipes– Records-in-memory– Custom input, compare & output procedures (User Exits)– Coming soon: Huge LDIF & Flat XML Sources

CoSORT Functionality

Data Type Collation & ConversionData Type Collation & Conversion

MiscellaneousMiscellaneous– Replacement and Conversion of 3rd-Party Sorts– LOCALE (operating system defined) collation– Thread-safe APIs – Granular resource tuning and monitoring

ASCII

Binary integerEBCDICEdited numeric

Bit Embedded signCharacter FloatDate/timestamps Packed decimalIP Address Whole numberCurrency UnicodeDouble Unsigned decimal

Sort Replacements– Seamless drop-in services for many third-party sorts

Sort-I (Sort Interactive)– Command line prompt/batch program for novices

SortCL (Sort Control Language)– mainframe-familiar DDL/DML for JCL sort migration,

data warehouse integration/staging (ETL), + reporting

– CLI, API, Java GUI for cross-platform design/launch

SortCL Conversion Tools – for third-party metadata and legacy sort parms

Application Programming Interfaces (APIs)– 3 callable libraries for third-party software integration

CoSORT User Interfaces

Sort Interface Support for:

– Amdocs Ensemble– Ascential (IBM WebSphere) DataStage – Cincom Supra SQL– IBM DB2 UDB Loader – Informatica PowerMart/Center– MF COBOL (Workbench, Srv/Net Express)– SAS System – Software AG Natural– Sun Mainframe Rehosting MTP/MBM– SyncSort UNIX (via script conversion)– Unix (/bin/sort)

Metadata Re-Use Support for:

– MVS and VSE JCL sort parms– COBOL Copybooks– Common & Extended Web Log formats– Microsoft CSV files– ETL, BI, XML and RDB file formats via MIMB

Leveraging What’s There

“The maturing IT industry can no longer propagate the notion of scrapping previous investments to

adopt new technologies. Billions of dollars have already

been invested in hardware, operating systems and applications.

Our solutions integrate withwhat is already there, and thus, can deliver exceptional ROI.”

Norman Praed, CEOProgeni Corporation

CoSO

RT’s

Natur

al

SORT

Repla

cem

ent

1) Copy $COSORT_HOME/etc/Makefile.nat2cs

into your ~sag/nat/vxxx/bin/build directory

2) Install the replacement with:

cd ~sag/nat/vxxx/bin/build

mv Makefile Makefile.orig

cp Makefile.nat2cs Makefile

3) Uncomment Makefile LIB_COSORT entry for your O/S

4) Link with:

make natural cosort=yes

5) Run with:

setenv PATH $PATH:$COSORT_HOME/bin

natural [...]

6) Enable debugging by setting the environment variable:

NAT2SCL_DEBUG=1.

Coroutine SORT Architecture + Parallel CPU Exploitation

19 MB in 2 seconds on P200/2 w/ NT, 2 keys

1.0 GB in 12 seconds on IBM p690/4

1.8 GB in 67 seconds on Compaq GS140/8

2.4 GB in 39 seconds on SunFire 15K/6

5.2 GB in 20 minutes on Sun UE3000, 23 keys

272 GB in <2 hours on IBM Numa-Q 2000/4 setting a 2000 TPC-H DSS benchmark record.

TDWI’s Customer Intelligence LifecycleCoSORT’s SortCL …

is used for very fast

data integration and staging:

extract –transform –load (ETL) operations

on multiple,

large external (flat file)

data sources.

Sort Control Language (SortCL)

SortCL: Single PassSingle PassData ManipulationsData Manipulations

… through many, large, differently-formatted inputs:

Sort/Mergeon any number of keys in any position

Join matching 1-1, many-1, inner/outer, left/right

Select via record filters or conditional include/omit

Convert translate input field data types to new types

Aggregate min, max, average, sum, count (sub and grand)

Calculate across rows to perform math (+ sci functions)

Re-map change field positions, sizes, and values

Report to highly-formatted, multi-level output targets

User Exits for custom input, compare and output criteria

SortCL Also Re-hosts JCL Sorts.

Consider Tetrad’s OPX:Operational Processing for UNIX

Tools and architecture to manage jobs on UNIX

Separates job definitions from job processing

Easy integration with 3rd party tools like:Natural, CoSORT, MF COBOL

Developed in Perl for multi-platform use

Not dependent on Software AG products

How OPX Rehosts via “Job Wrappers”

OPX provides job access to programs through the use of “Wrappers”

A wrapper is a script or program that facilitates the interface between the OPX job and an external program or process

A sample set of wrappers are provided with OPX (including Natural, SORT, IEFBR14, GENER, FTP)

SORT WrapperCreates pseudo JCL

for CoSORT

Final CoSORT.scl statements to

perform sort

SORT WrapperPre-processes the

.scl statements

pseudo-JCLfor CoSORTTranslation

ConvertedSort Control

Language (.scl)

CoSORT mvs2sclconversion utility

SORT WrapperCalls CoSORTSortCL utility

CoSORT SortCLperforms the .scl job as requested

Done!

OPX SORT WrapperIntegration with CoSORT

OPX SORT Wrapper Example

//SUMS JOB 1// EXEC PGM=SORT 2//STEPLIB DD DSN=SORT.RESI.DENCE,DISP=SHR 3//SYSOUT DD SYSOUT=A 4//SORTIN DD DSN=chiefs30.votes, 5// UNIT=2400-3,VOL=SER=887766,// DISP=(OLD,KEEP)//SORTOUT DD DSN=termsums, 6// UNIT=2400-3,VOL=SER=554433,// DD DISP=(NEW,KEEP)//SORTWK01 DD UNIT=SYSDA,SPACE=(CYL,20) 7//SYSIN DD * 8 SORT FIELDS=(40,3,CH,A,45,2,CH,A) 9 SUM FIELDS=(23,3,CH) 10/* 11

Can we run this on Unix?

...with OPX and CoSORT - Sure!

OPX SORT Wrapper Example

What does the CoSORT SortCL Version Look Like?

Same job, easier to expand …

/INFILES=chiefs30.votes /FIELD=(field_0, POSITION=40, SIZE=3, EBCDIC) /FIELD=(field_1, POSITION=45, SIZE=2, EBCDIC) /FIELD=(field_2, POSITION=23, SIZE=3, EBCDIC) /CONDITION=(cond_0, TEST=(field_0 OR field_1))

/SORT /KEY=(field_0, ASCENDING) /KEY=(field_1, ASCENDING)

/OUTFILE=termsums /FIELD=(field_0, POSITION=40, SIZE=3, EBCDIC) /FIELD=(field_1, POSITION=45, SIZE=2, EBCDIC) /FIELD=(field_2_sum, POSITION=23, SIZE=3, EBCDIC) /SUM field_2_sum FROM field_2 BREAK cond_0

Sort Control Language (SortCL)CLI/Batch Mode Example (Join)

################################################# CoSORT/SortCL Job Spec file csumtv_join_1.spec# gets run with $sortcl /spec=csumtv_join_1.spec# Created 09/18/2006 15:20EST by DW_team_alpha# Conditional indexed summary join report + calc################################################

/STATISTICS=/warehouse/stats/csumptv_join.sta # runtime performance log/MEMORY-WORK="$COSORT_HOME/etc/cosortrc“ # job-specific tuning file/SPEC=/warehouse/views/csumtv # metadata, condition references

/INFILE=${SQL_DATA}csumtv_temp # first input/join file (EV)/ALIAS=left/INFILE=${SQL_DATA}csumtv_ICN_temp # second input/join file (EV)/ALIAS=right

/JOIN LEFT_OUTER left right WHERE left.TID==right.TID AND left.SSN==right.SSN

/OUTFILE=${SQL_DATA}csumtv.report # new record layout, data types

/HEADREC=“>>>>> STARS Report >>>> %D, DATE, %S, USER ****”

/FIELD=(left.YDATE, SEPARATOR='~', POSITION=1, SIZE=4.2, NUMERIC)/FIELD=(left.TID, SEPARATOR=‘\t', POSITION=2, SIZE=10, EBCDIC) /DATA="~*“

/FIELD=(left.SSN, SEPARATOR=‘,', POSITION=3, SIZE=22, ASCII)/FIELD=(left.sum_DAYS_SUPPLY, SEPARATOR=‘|', POSITION=4, SIZE=12, INTEGER)/FIELD=(left.sum_A_AMT, SEPARATOR='~', POSITION=5, SIZE=15.2, NUMERIC)/DATA= IF Cond1 THEN "~ 0.0“ THEN IF Cond2 ELSE left.YDATE+1900

/SUM RUNNING FROM left.PERIOD WHERE COND3 # accumulating aggregate

/FIELD=(SEQUENCER+50, SEPARATOR=‘|', POSITION=6, SIZE=5) # DB reindexer

CoSORT’s Java GUI–to–SortCL (gui2scl) Client/Server Sort/ETL Application(gui2scl) Client/Server Sort/ETL Application

CoSORT’s Java GUI–to–SortCL (gui2scl) Client/Server Sort/ETL Application(gui2scl) Client/Server Sort/ETL Application

CALL shcmd sortcl /spec=file.scl

/INFILE=Natural WORK FILE

/OUTFILE=Natural WORK FILE

Benefits:– 1-pass, multi-file integration, staging, and reporting– No limit on # of in/output files– No limits on file sizes or layouts– Same metadata as RowGen –

so …. The same SortCL job script can be used in RowGen to generate safe test data in exactly the same file format.

Using CoSORT SortCL with Natural for Unix

Affordable ($1K - $25K)

Perpetual use (not a lease)

Volume license discounts

ISV / OEM runtime pricing

GSA schedule, state bidder

Global support (30+ offices)

Product bundling discounts:– FAst extraCT (DB unload)– RowGen (safe test data)– netCONVERT (file porting)– x-PRESS (compression)– Logon Security (access)– Permitas (app licensing)

CoSORT Value

New New Product!Product!

Prototype ApplicationsPrototype Applications– Create data and file formats your projects needCreate data and file formats your projects need

Share Files with OutsourcersShare Files with Outsourcers– Provide accurate layouts, not real dataProvide accurate layouts, not real data

Specify Value RangesSpecify Value Ranges– Use selection and set files: better than real dataUse selection and set files: better than real data

Simulate DB OpsSimulate DB Ops– Quickly test table loading and query scenariosQuickly test table loading and query scenarios

Benchmark Testing Benchmark Testing – Gen big files for hardware and software PoCsGen big files for hardware and software PoCs

CoSORT’s CoSORT’s RowGenRowGen

Data SynthesizerData Synthesizer

Create Custom Files with Safe DataCreate Custom Files with Safe Data(Using SortCL Metadata!)

UsingOracle?

Use FACT to SpeedExtractsand write

metadata for SortCL and

SQL*Loader.

Single-pass entire E-T-L operations through a

pipe!

Now You Know.

CoSORTCoSORT is the innovator in UNIX and Windows sort software, and a key infrastructure tool for the staging, integrating, manipulating and presenting of large data volumes. Since 1978, IT installations have chosen CoSORTCoSORT to meet their project and performance objectives in:

Natural & JCL sort migrations

VLDB reorg (unload, sort, reload)

Data warehouse staging (ETL)

Detail and summary reporting

3rd-party sort replacements

Batch jobs and new products