Incremental Load

using qvd files

Incremental Load

Is sometimes called…Incremental Load

Differential Load

Delta Load

Incremental Load

Goal:

Load only the new or the changed records from the database. The rest should already be available, one way or another.

Comments on Buffer Load

• Buffer (Incremental) Load … is a solution only for Log files (text files), but not for DBs.

• Buffer (Stale after 7 days) Select … is not a good solution. It makes a full Load after 7 days. And nothing in between…

Incremental Load

• Load new data from Database table (slow, but few records)

• Load old data from QVD file (many records, but fast)

• Create new QVD file • Procedure must be

repeated for each table

Different DB-changes

If source allows …1) Append only. (Logfiles)

2) Insert only. (No Update or Delete)

3) Insert and Update. (No Delete)

4) Insert, Update and Delete.

1) Append only

• Must be Log file• Loads records

added in the end of the file

1) Append only

Buffer (Incremental) Load * From LogFile.txt(ansi, txt, delimiter is '\t', embedded labels);

Done!But it should be renamed to „Buffer (Append) Load…“

2) Insert only

• Can be any DB• Loads INSERTed

records• Needs the field ModificationDate

2) Insert only

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;

2) Insert only

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;

ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;

2) Insert only

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;

ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;

STORE QV_Table INTO File.QVD;

Almost doneBut there is a small chancethat a record gets loaded twice

2) Insert only

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)# AND ModificationTime < #$(BeginningThisExecTime)#;

ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;

STORE QV_Table INTO File.QVD;

Done!

3) Insert and Update

• Can be any DB• Loads INSERTed and

UPDATEd records• Needs the fields ModificationDate and PrimaryKey

3) Insert and Update

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;

ConcatenateLOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT Exists(PrimaryKey);

STORE QV_Table INTO File.QVD;

Done!

4) Insert, Update and Delete

• Can be any DB• Loads INSERTed and

UPDATEd records• Removes DELETEd

records• Needs the fields ModificationDate and PrimaryKey

• Tricky to implement

4) Insert, Update and Delete

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;

ConcatenateLOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT EXISTS(PrimaryKey);

Inner JoinSQL SELECT PrimaryKey FROM DB_TABLE;

STORE QV_Table INTO File.QVD;OK, but slow…

4) Insert, Update and DeleteListOfDeletedEntries:SQL SELECT PrimaryKey AS Deleted FROM DB_TABLEWHERE DeletionFlag = 1

and ModificationTime >= #$(LastExecTime)#;

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT Exists(PrimaryKey) AND NOT Exists(Deleted,PrimaryKey);

Drop Table ListOfDeletedEntries;

STORE QV_Table INTO File.QVD;

OK, but needs a DeletionFlag

LastExecutionTime & Error handling

Let ThisExecTime = Now();

{ Load sequence }

If ScriptErrorCount = 0 then Let LastExecTime = ThisExecTime;

End If

Final Script

Let ThisExecTime = Now();

QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)# AND ModificationTime < #$(ThisExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT EXISTS(PrimaryKey);

Inner Join SQL SELECT PrimaryKey FROM DB_TABLE;

If ScriptErrorCount = 0 then STORE QV_Table INTO File.QVD;Let LastExecTime = ThisExecTime;

End If

Summary 1

• Incremental Load possible for…• Append only. (Logfiles) Yes!• Insert only. (No Update or Delete) Yes!• Insert and Update. (No Delete) Yes!• Insert, Update and Delete. Slow,

or demands

DeletionFlag

Summary 2

• „Incremental Load“ normally not equivalent to Buffer (Incremental) Load …