Incremental Load

Download Incremental Load

Post on 28-Oct-2015

13 views

Category:

Documents

6 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>Incremental Loadusing qvd files</p></li><li><p>Incremental LoadIs sometimes calledIncremental Load Differential LoadDelta Load</p></li><li><p>Incremental LoadGoal:Load only the new or the changed records from the database. The rest should already be available, one way or another.</p></li><li><p>Comments on Buffer LoadBuffer (Incremental) Load is a solution only for Log files (text files), but not for DBs.Buffer (Stale after 7 days) Select is not a good solution. It makes a full Load after 7 days. And nothing in between</p></li><li><p>Incremental LoadLoad new data from Database table (slow, but few records)Load old data from QVD file (many records, but fast)Create new QVD file Procedure must be repeated for each table</p></li><li><p>Different DB-changesIf source allows Append only. (Logfiles)Insert only. (No Update or Delete)Insert and Update. (No Delete)Insert, Update and Delete.</p></li><li><p>1) Append onlyMust be Log fileLoads records added in the end of the file</p></li><li><p>1) Append only</p><p>Buffer (Incremental) Load * From LogFile.txt(ansi, txt, delimiter is '\t', embedded labels);</p><p>Done! But it should be renamed to Buffer (Append) Load</p></li><li><p>2) Insert onlyCan be any DBLoads INSERTed recordsNeeds the field ModificationDate</p></li><li><p>2) Insert only</p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)#;</p></li><li><p>2) Insert only</p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)#;</p><p>ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;</p></li><li><p>2) Insert only</p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)#;</p><p>ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;</p><p>STORE QV_Table INTO File.QVD;</p><p>Almost done But there is a small chance that a record gets loaded twice</p></li><li><p>2) Insert only</p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)# AND ModificationTime &lt; #$(BeginningThisExecTime)#;</p><p>ConcatenateLOAD PrimaryKey, X, Y FROM File.QVD;</p><p>STORE QV_Table INTO File.QVD;</p><p>Done!</p></li><li><p>3) Insert and UpdateCan be any DBLoads INSERTed and UPDATEd recordsNeeds the fields ModificationDate and PrimaryKey</p></li><li><p>3) Insert and Update</p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)#;</p><p>ConcatenateLOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT Exists(PrimaryKey);</p><p>STORE QV_Table INTO File.QVD;</p><p>Done!</p></li><li><p>4) Insert, Update and DeleteCan be any DBLoads INSERTed and UPDATEd recordsRemoves DELETEd recordsNeeds the fields ModificationDate and PrimaryKeyTricky to implement</p></li><li><p>4) Insert, Update and Delete</p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)#;</p><p>ConcatenateLOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT EXISTS(PrimaryKey);</p><p>Inner JoinSQL SELECT PrimaryKey FROM DB_TABLE;</p><p>STORE QV_Table INTO File.QVD;</p><p>OK, but slow</p></li><li><p>4) Insert, Update and DeleteListOfDeletedEntries:SQL SELECT PrimaryKey AS Deleted FROM DB_TABLEWHERE DeletionFlag = 1 and ModificationTime &gt;= #$(LastExecTime)#;</p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)#;</p><p>Concatenate LOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT Exists(PrimaryKey) AND NOT Exists(Deleted,PrimaryKey);</p><p>Drop Table ListOfDeletedEntries;</p><p>STORE QV_Table INTO File.QVD;</p><p>OK, but needs a DeletionFlag</p></li><li><p>LastExecutionTime &amp; Error handlingLet ThisExecTime = Now(); </p><p>{ Load sequence }</p><p>If ScriptErrorCount = 0 then Let LastExecTime = ThisExecTime; End If</p></li><li><p>Final ScriptLet ThisExecTime = Now(); </p><p>QV_Table:SQL SELECT PrimaryKey, X, Y FROM DB_TABLEWHERE ModificationTime &gt;= #$(LastExecTime)# AND ModificationTime &lt; #$(ThisExecTime)#;</p><p>Concatenate LOAD PrimaryKey, X, Y FROM File.QVDWHERE NOT EXISTS(PrimaryKey);</p><p>Inner Join SQL SELECT PrimaryKey FROM DB_TABLE;</p><p>If ScriptErrorCount = 0 then STORE QV_Table INTO File.QVD;Let LastExecTime = ThisExecTime; End If</p></li><li><p>Summary 1Incremental Load possible forAppend only. (Logfiles)Yes!Insert only. (No Update or Delete)Yes!Insert and Update. (No Delete)Yes!Insert, Update and Delete.Slow, or demands DeletionFlag</p></li><li><p>Summary 2Incremental Load normally not equivalent to Buffer (Incremental) Load </p></li><li><p>CommentThe solutions above are (alone) probably not robust enough. In addition, the complete table should probably be reloaded regularly, perhaps once a month.</p></li><li><p>*************************</p></li></ul>