Download - Incremental Load
Incremental Load
Goal:
Load only the new or the changed records from the database. The rest should already be available, one way or another.
Comments on Buffer Load
• Buffer (Incremental) Load … is a solution only for Log files (text files), but not for DBs.
• Buffer (Stale after 7 days) Select … is not a good solution. It makes a full Load after 7 days. And nothing in between…
Incremental Load
• Load new data from Database table (slow, but few records)
• Load old data from QVD file (many records, but fast)
• Create new QVD file • Procedure must be
repeated for each table
Different DB-changes
If source allows …1) Append only. (Logfiles)
2) Insert only. (No Update or Delete)
3) Insert and Update. (No Delete)
4) Insert, Update and Delete.
1) Append only
Buffer (Incremental) Load * From LogFile.txt(ansi, txt, delimiter is '\t', embedded labels);
Done!But it should be renamed to „Buffer (Append) Load…“
2) Insert only
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;
2) Insert only
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;
ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;
2) Insert only
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;
ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;
STORE QV_Table INTO File.QVD;
Almost doneBut there is a small chancethat a record gets loaded twice
2) Insert only
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)# AND ModificationTime < #$(BeginningThisExecTime)#;
ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;
STORE QV_Table INTO File.QVD;
Done!
3) Insert and Update
• Can be any DB• Loads INSERTed and
UPDATEd records• Needs the fields ModificationDate and PrimaryKey
3) Insert and Update
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;
ConcatenateLOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT Exists(PrimaryKey);
STORE QV_Table INTO File.QVD;
Done!
4) Insert, Update and Delete
• Can be any DB• Loads INSERTed and
UPDATEd records• Removes DELETEd
records• Needs the fields ModificationDate and PrimaryKey
• Tricky to implement
4) Insert, Update and Delete
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;
ConcatenateLOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT EXISTS(PrimaryKey);
Inner JoinSQL SELECT PrimaryKey FROM DB_TABLE;
STORE QV_Table INTO File.QVD;OK, but slow…
4) Insert, Update and DeleteListOfDeletedEntries:SQL SELECT PrimaryKey AS Deleted FROM DB_TABLEWHERE DeletionFlag = 1
and ModificationTime >= #$(LastExecTime)#;
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)#;
Concatenate LOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT Exists(PrimaryKey) AND NOT Exists(Deleted,PrimaryKey);
Drop Table ListOfDeletedEntries;
STORE QV_Table INTO File.QVD;
OK, but needs a DeletionFlag
LastExecutionTime & Error handling
Let ThisExecTime = Now();
{ Load sequence }
If ScriptErrorCount = 0 then Let LastExecTime = ThisExecTime;
End If
Final Script
Let ThisExecTime = Now();
QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime >= #$(LastExecTime)# AND ModificationTime < #$(ThisExecTime)#;
Concatenate LOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT EXISTS(PrimaryKey);
Inner Join SQL SELECT PrimaryKey FROM DB_TABLE;
If ScriptErrorCount = 0 then STORE QV_Table INTO File.QVD;Let LastExecTime = ThisExecTime;
End If
Summary 1
• Incremental Load possible for…• Append only. (Logfiles) Yes!• Insert only. (No Update or Delete) Yes!• Insert and Update. (No Delete) Yes!• Insert, Update and Delete. Slow,
or demands
DeletionFlag