qlikview tips tricks

Download Qlikview Tips Tricks

Post on 14-Oct-2015

1.818 views

Category:

Documents

31 download

Embed Size (px)

DESCRIPTION

A collection of QlikView tips and tricks from the Qlik Community. The required attribution and source has been provided in the document. This document has about 90+ tips, tricks and solutions for common qlikview challenges.

TRANSCRIPT

  • TekAcademyLabs.com

    QlikView Tips and Tricks

    June 9

    2014QlikView tips and tricks is the document in which content is aggregated from the qlik community. The appropriate source of the content is also mentioned to avoid plagiarism.

    http://tekacademylabs.com/

  • Contents 1. Load XML field from database (Oracle) ............................................................................................. 3

    2. Hash functions and Collisions ............................................................................................................. 4

    Hash functions.................................................................................................................................... 4

    Hash functions in QlikView ................................................................................................................. 5

    Detecting changed records ................................................................................................................... 5

    Hash collisions ................................................................................................................................... 6

    The birthday problem .......................................................................................................................... 7

    Calculating the probability of a hash collision .......................................................................................... 7

    3. Autonumber Vs AutonumberHash Vs Autonumberhash128 ............................................................. 9

    4. Loosely coupled tables ...................................................................................................................... 10

    5. Circular References ........................................................................................................................... 11

    6. Incremental Load .............................................................................................................................. 14

    7. Three types of Incremental Load ...................................................................................................... 17

    8. Qlikview Associative data model ...................................................................................................... 20

    9. The magic of variables ...................................................................................................................... 21

    10. The QlikView Cache......................................................................................................................... 25

    11. Null handling in QlikView ................................................................................................................ 27

    12. Text searches .................................................................................................................................. 27

    13. Automatic Number interpretation .................................................................................................. 30

    15. Colors in Chart ................................................................................................................................. 35

    16. Aggregations and Function Classes ................................................................................................ 38

    17. Its all Aggregations ........................................................................................................................ 40

    18. Dimensions and Measures ............................................................................................................. 42

    19. Qlikview Quoteology ...................................................................................................................... 45

    20. The Crosstable Load ....................................................................................................................... 47

    21. On Boolean fields and functions ..................................................................................................... 49

    22. The Dual() function ........................................................................................................................ 52

    23. A primer on Section Access ............................................................................................................. 56

    24. Data reduction using multiple fields .............................................................................................. 58

    25. Color, state and vectors .................................................................................................................. 61

    26. Handling multiple languages .......................................................................................................... 62

    27. Dynamically selecting timeframes ................................................................................................. 65

    28. The Only() function ......................................................................................................................... 67

  • 29. AND and OR .................................................................................................................................... 69

    30. To JOIN or not to JOIN..................................................................................................................... 71

    31. Canonical Date ................................................................................................................................ 73

    32. Linking to two or more dates .......................................................................................................... 77

    33. IntervalMatch and Slowly Changing Dimension ............................................................................ 78

    34. The Calculation engine ................................................................................................................... 78

    35. Symbol Tables and Bit stuffed pointers .......................................................................................... 80

    36. Basics for Complex authorization .................................................................................................. 82

    37. Generic Keys .................................................................................................................................... 83

    38. Generate missing data in QlikView ................................................................................................ 84

    39. Strategies for creating key tables ................................................................................................... 85

    40. Recipe for a Gantt chart .................................................................................................................. 86

    41. Relative Calendar Fields ................................................................................................................. 87

    42. Master Calendar .............................................................................................................................. 90

    43. Year Over Year Comparisons........................................................................................................... 91

    44. Redefining the week numbers ........................................................................................................ 94

    45. Preceding Load ................................................................................................................................ 97

    46. Macros are BAD .............................................................................................................................. 99

    47. Recipe for Pareto Analysis ............................................................................................................ 101

    48. Monte Carlo Methods .................................................................................................................. 104

    49. A myth about COUNT distinct ....................................................................................................... 107

    50. Unbalanced n level hierarchies ................................................................................................... 110

    51. Hierarchies .................................................................................................................................... 113

    52. Loops in the Script ......................................................................................................................... 114

    53. IntervalMatch ................................................................................................................................ 117

    54. Counters in the Load ..................................................................................................................... 119

    55. Synthetic Keys ............................................................................................................................... 121

    56. Data types in QlikView .................................................................................................................. 124

    57. The nature of Dual flags ................................................................................................................ 126

    58. Dont Join use ApplyMap instead ................................................................................................ 128

    59. Slowly Changing Dimension .......................................................................................................... 130

    60. Search, but what shall you find? .................................................................................................. 132

    71. Cyclic or Sequential ....................................................................................................................... 134

    72. The magic of Dollar Expansion ...................................................................................................... 137

  • 73. When should the Aggr function not be used ................................................................................ 139

    74. Recipe for memory statistics analysis ........................................................................................... 142

    75. The Table Viewer.......................................................................................................................... 145

    76. Creating a Scatter Chart ................................................................................................................ 146

    77. Fiscal Year ...................................................................................................................................... 147

    78. The Master Time Table ................................................................................................................ 150

    79. Create reference dates for intervals ............................................................................................. 152

    80. Fact table with mixed granularity ................................................................................................. 154

    81. How to populate a sparsely populated field ................................................................................. 156

    82. Calculated Dimensions .................................................................................................................. 158

    83. Finding Null .................................................................................................................................. 158

    84. Creating intervals from a single date ........................................................................................... 160

    85. Why dont my dates work ............................................................................................................. 162

    86. Master table with multiple roles................................................................................................... 165

    87. Rounding errors ........................................................................................................................... 168

    88. Generic Load ................................................................................................................................. 170

    89. Clarity Vs. Speed ........................................................................................................................... 173

    90. Green is the Colour ....................................................................................................................... 176

    91. Joins ............................................................................................................................................... 178

    92. On format codes for numbers and date ....................................................................................... 182

    93. The Aggregation Scope ................................................................................................................. 185

    QlikView Tips and Tricks

    1. Load XML field from database (Oracle)

    Source: http://community.qlik.com/thread/8453

    SQL SELECT extract(XML_DOCUMENTO,'/','xmlns="http://www.portalfiscal.inf.br/cte"').getClobVal() AS

    XML_DOCUMENTO

    FROM XML_DOCUMENTO_FISCAL

  • This way the files were extracted and can work with the data to make the load of the QVD created.

    2. Hash functions and Collisions

    Source: http://www.qlikfix.com/2014/03/11/hash-functions-collisions/

    Im currently updating my materials for the upcoming Masters

    Summit for QlikView in Chicago, and thought Id share a little bit with you. In my session on

    data modeling, I explain how you can deal with various types of Slowly Changing

    Dimensions in QlikView. One of the techniques I explain is using hash functions to detect

    changes in (historical) records. During the previous events, this always lead to two questions

    from the audience:

    What exactly are hash functions and hashes?

    And, from those who already know the answer to the first question: Arent you worried

    about hash collisions?

    Today I will answer both questions and hopefully give you some insight into hash functions,

    their usefulness in QlikView and the risks of hash collisions.

    Hash functions

    A hash function is an algorithm that maps data of arbitrary length to data of a fixed length.

    The value returned by a hash function is like a fingerprint of the input value, and is called a

    hash value or simply hash. For example, all of the text above can be translated into the

    following MD5 hash: 357799131ceffdd43cc0fe9f52b36eeb.

    You will notice that this hash is much shorter than the original string used to generate

    it.Besides that, if only a single character in the text is changed this will lead to a completely

    different hash. This property makes hash functions very useful to compare things, for

    example files, but also historical versions of a record.

  • A hash function is deterministic, meaning that the same input value should always lead to

    the same hash value. Typically, a hash function is a one-way function, you cannot decode

    the original input value based on the hash value alone. Besides that, a good hash function is

    also uniform, which means that each hash value should have the same probability of being

    picked. The image at the top of this post illustrates a very simple hash function. Each of the

    four input values is mapped to a unique output value.

    Hash functions in QlikView

    In QlikView, the following hash functions are available:

    Hash128(): a 128 bit hash function that returns a 22 character string.

    Hash160(): a 160 bit hash function that returns 27 character string.

    Hash256(): a 256 bit hash function that returns a 43 character string.

    The number of bits determines the output range of the function. A 128 bit hash can store

    2^128 (or, 340.282.366.920.938.000.000.000.000.000.000.000.000) different combinations.

    160 and 256 bit can store even more combinations (2^160 and 2^256 respectively).

    Besides these functions, QlikView also has

    the AutoNumberHash128() and AutoNumberHash256()functions. These functions

    basically take the output of the Hash128() and Hash256() function and passes it through

    the AutoNumber() function. While I think they have a nicer syntax than the

    regularAutoNumber(), you can supply a comma-separated list of fields instead of a

    concatenated string, the usefulness of these functions eludes me.

    Detecting changed records

    Consider a QlikView application containing the following Employee table:

    Now, assume we get some new, changed data and want to quickly determine which rows

    have changed:

  • As you can see, Jim has moved to another office. How can we detect that this row has

    changed? We could compare each field in the table to each previous version of the field, but

    as we are only interested in detecting if the row has changed, using a hash function is a more

    elegant solution. UsingHash128(Name, Position, Office) we can calculate a hash value for

    each row:

    The hash value for Dwights record hasnt changed, because the record hasnt changed either.

    Jims changed record however does have another hash value than the previous one. Once

    weve detected this we can do further processing on these records. This will be the topic of a

    future blog post. Or, if you dont want to wait for that, my data modeling session at

    the Masters Summit for QlikView.

    Hash collisions

    As noted before, a hash function is an algorithm that maps data of arbitrary length to data of a

    fixed length. When different input values lead to the same output hash value, this is known as

    a hash collision. Consider the following, simplified hash function:

    In this example, both Michael and Toby get the same hash value of 2. Its easy to see what

    the problem is here, there are 5 input values and only 4 possible hash values. The input

    domain is greater than the output range.

    Now, you may think this isnt a problem for me, the number of input values I deal with is

    much less than 2^128, let alone 2^256. Its a simple assumption to make, but also a wrong

  • one as hash collisions can occur long before the number of input values reaches the range of

    the hash function.

    The birthday problem

    Imagine youre in a room with a group of people. How many people do you think need to be

    in that room before the probability of two people sharing the same birthday reaches 50%?

    There are (excluding leap years) 365 days in a year, so maybe 185? 200?

    The answer is 23. Surprising, isnt it? If we raise the number of people to 75, the probability

    of at least two people sharing a birthday raises to 99,95%. This is known as the birthday

    problem.

    As this is a QlikView blog and not a math blog, I wont go through the complete solution and

    proof. Basically, instead of calculating the probability that two people in a group share a

    birthday, the trick is to calculate the probability that no one in the group shares a birthday.

    This is much easier to calculate. The result is then subtracted from 1, which gives the

    probability that at least two people in the group share a birthday.

    Calculating the probability of a hash collision

    If you looked closely at the previous example, you may see that the people can be considered

    input values and that their birthdays can be considered hash values. When two people share

    the same birthday its a hash collision! If we understand this, then we can apply the same

    logic to determine the probability of a hash collision in our data sets. To calculate the

    approximate probability of a hash collision we can use the following formula:

    I created a small Excel workbook to calculate the probability of a hash collision. Now, its

    good to realize that Excel only uses 30 significant digits. As these probabilities are very

    small, this means that Excel is unable to calculate probabilities for very small input values.

    So, in the example below, I calculated the probability that 1 quadrillion (thats a 1 with 15

    zeroes) input values could lead to a hash collision when using a 128 bit hash.

  • The probability of this happening are around 1 in 680 million. Or, to put it in perspective:

    Now, there is a small caveat with this calculation. It assumes the hash functions used in

    QlikView leads to a uniform output, meaning each value has the same probability. This may

    not be the case.

    On the other hand, we are not comparing a quadrillion records, we are only comparing two.

    When calculating the probability of a hash collision with just 2 records and a 128 bit hash

    using an online high precision calculator, the result is 2.938735877055718769922E-39 (1 in

    2.9 Duodecillion). Or, to put it in perspective again, this is less likely than a single person

    winning the lottery, getting hit by a meteorite, getting attacked by a shark -and- becoming

    president of the USA in their lifetime.

    Switch to a 160 bit hash and the likelihood of a collision becomes lower than the combined

    probability of all events in the chart above. Now, just because it is very unlikely doesnt mean

    that it cant happen (see:Law of large numbers), but I like those odds!

  • 3. Autonumber Vs AutonumberHash Vs Autonumberhash128

    I read it that autonumber stores the expression value and gives it a unique integer value

    whereas autonumberhash128 stores just the hash value (in 128 bits) of the corresponding

    expression value. Therefore, autonumberhash128 should be more efficient in data storage

    (particularily when the expression value is larger) and so the document size reduced.

    Other notes:

    Having our new baby (AKA the mini QlikView addict) around has meant very little time for

    anything, let alone blogging. So in order to ensure I at least manage the odd post or 2 I

    thought it would be good to start a new series of short posts on different qlikview functions

    and their uses. To kick things off I have decided to take a look at the autonumber() function

    and the closely related autonumberhash128() and autonumberhash256(). All 3 functions do a

    very similar thing so let's look at autonumber() first and then consider how the other 2

    functions differ.

    Autonumber() can be considered a lookup function. It takes a passed expression and looks up

    the value in a lookup table. If the expression value isn't found then it is added to the table and

    assigned an integer value which is returned. If the expression value is found then it returns

    the integer value that is assigned against it. Simply put, autonumber() converts each unique

    expression value into a unique integer value.

    Autonumber() is only useful within the QlikView script and has the following syntax:

    autonumber(expression [, index])

    The passed expression can be any string, numeric value or most commonly a field within a

    loaded table. The passed index is optional and can again be any string or numeric value. For

    each distinct value within the passed index, QlikView will create a separate lookup table and

    so the same passed expression values will result in a different returned integer if a different

    index is specified.

    So how exactly are the 3 autonumber functions different? Autonumber() stores the expression

    value in its lookup table whereas autonumberhash128() stores just the 128bit hash value of

    the expression value. I'm sure you can guess therefore, autonumberhash256() stores the

    256bit hash value expression value.

    Why on earth would I want to use any of these functions? Well the answer is quite simply for

    efficiency. Key fields between two or more tables in QlikView are most efficient if they

    contain only consecutive integer values starting from 0. All 3 of the autonumber functions

    allow you to convert any data value and type into a unique integer value and so using it for

    key fields allow you to maintain optimum efficiency within your data model.

    A final word of warning. All 3 of the autonumber functions have one pitfall, the lookup

    table(s) exist only whilst the current script execution is active. After the script completes, the

    lookup table is destroyed and so the same expression value may be assigned different integer

  • values in different script executions. This means that the autonumber functions can't be used

    for key fields within incremental loads.

    2 comments:

    1. Mike Taylor8 March 2014 11:13

    Nice, simple explanation. Can you clarify how autonumber function deal with null values. I

    had some issues recently and resorted back to using the original values where I had one table

    with nulls (which were assigned a autonumber of 0) and I was trying to join to another table

    that had no null values.

    Reply

    2. Matthew Fryer24 April 2014 15:00

    Hi Mike

    First of all, how autonumber() will behave depends on if it is a true null or a zero length

    string. For true nulls, the result of the autonumber() will also be null. You can prove this by

    running the following script:

    LOAD autonumber(null()) AS field AUTOGENERATE 1;

    By adding "field" to a list box you will see no records. Being a true null and the fact that null

    values don't really exist in QlikView (they are simply the absence of a value) means that

    when using it for a key field, QlikView will not associate a null in one table to a null in the

    other.

    For a zero length string, autonumber() will assign it a value as it would any other passed

    value. The number assigned to the zero length string will depend on the order it appears in the

    values that are passed to autonumber(). You can see this by running the following script:

    LOAD autonumber('') AS field AUTOGENERATE 1;

    The result will be a single value in "field" of "1". Autonumber() is 1 indexed and so I'm not

    sure where you are getting your 0 value back.

    4. Loosely coupled tables http://community.qlik.com/thread/104608

    NOTES:

    Loosely coupled tables are created automatically when the data model(3 or More Tables) that

    includes circular references is loaded into QlikView,to avoid them the circular references create a loop

    in the QlikView internal logic. These loosely coupled tables need to be handled in order to visualize

    data in a way that is expected and understandable.

    See Article Circular References

    Any table can also be made loosely coupled interactively from this dialog or via macros. Additionally,

    it is possible to declare loosely coupled tables explicitly in the script via the Loosen Table statement.

  • The normal QlikView associative logic is disconnected internally for loosely coupled tables. This

    means that selections in one field do not Associate through to the other fields in the table. its very

    useful to avoid circular references in the data structure in various scenarios. For more examples of

    how this feature can be used, please refer to the QlikView Reference Manual - Intentionally Creating

    Loosely Coupled Tables

    One or more QlikView internal data tables can be explicitly declared loosely coupled during script

    execution by using a Loosen Table statement.

    The use of one or more Loosen Table statements in the script will make QlikView disregard any setting

    of tables as loosely coupled made before the script execution.

    The syntax is:

    Loosen Table[s] tablename [ , tablename2 ...]

    Either syntax: Loosen Table and Loosen Tables can be used.

    Example:

    Table1:

    Select * from Trans;

    Loosen table Table1;

    Note!

    Should QlikView find circular references in the data structure which cannot be broken by tables

    declared loosely coupled interactively or explicitly in the script, one or more additional tables will be

    forced loosely coupled until no circular references remain. When this happens, the Loop

    Warning dialog, gives a warning.

    5. Circular References http://community.qlik.com/blogs/qlikviewdesignblog/2013/06/25/circular-references

    There are two Swedish car brands, Volvo and SAAB. Or, at least, there used to be... SAAB was made

    in Trollhttan and Volvo was and still is made in Gothenburg.

    Two fictive friends Albert and Herbert live in Trollhttan and Gothenburg, respectively. Albert

    drives a Volvo and Herbert drives a SAAB.

    If the above information is stored in a tabular form, you get the following three tables:

    Logically, these tables

    form a circular reference: The first two tables are linked through City; the next two through Person; the

    last and the first through Car. Further, the data forms an anomaly: Volvo implies Gothenburg;

    Gothenburg implies Herbert; and Herbert implies SAAB. Hence, Volvo implies SAAB which doesnt

    make sense. This means that you have ambiguous results from the logical inference - different results

    depending on whether you evaluate clockwise or counterclockwise.

  • If you load these tables into QlikView, the circular reference will be identified and you will get the

    following data model:

    To avoid ambiguous results, QlikView marks one of the tables as loosely coupled, which means that

    the logical inference cannot propagate through this table. In the document properties you can decide

    which table to use as the loosely coupled table. You will get different results from the logical inference

    depending on which you choose.

    So what did I do wrong? Why did I get a circular reference?

    It is not always obvious why they occur, but when I encounter circular references I always look for

    fields that are used in several different roles at the same time. One obvious example is if you have a

    table listing external organizations and this table is used in several roles: as Customers, as Suppliers

    and as Shippers. If you load the table only once and link to all three foreign keys, you will most likely

    get a circular reference. You need to break the circular reference and the solution is of course to load

    the table several times, once for each role.

    In the above data model you have a similar case. You can think of Car as Car produced in the city or

    Car that our friend drives. And you can think of City as City where car is produced or City where

    our friend lives. Again, you should break the circular reference by loading a table twice. One possible

    solution is the following:

    In real life circular references are not as obvious as this one. I once encountered a data model with

    many tables where I at first could not figure out what to do, but after some analyzing, the problem

    boiled down to the interaction between three fields: Customers, Machines and Devices. A customer

    had bought one or several machines; a device could be connected to some of the machine types

    but not to all; and a customer had bought some devices. Hence, the device field could have two roles:

    Devices that the customer actually had bought; and devices that would fit the machine that the customer

    had bought, i.e. devices that the customer potentially could buy. Two roles. The solution was to load

    the device table twice using different names.

    Bottom line: Avoid circular references. But you probably already knew that

  • The post assumes that the reader knows what the Logical Inference engine does.

    The Logical Inference engine is the core of QlikView. It evaluates which field values are possible,

    given the selection. Basically it first evaluates which records are possible, and then the result of the

    evaluation "propagates" into the next table via the possible values of the key field, and then the next

    table is evaluated. It is this propagation that is disabled by the loosely coupled table.

    Read more about Logical Inference

    under http://community.qlik.com/blogs/qlikviewdesignblog/2013/07/15/logical-inference-and-aggregations

    5. Logical Inference and Aggregations

    Every time you click, QlikView recalculates everything.

    Everything.

    A new selection implies a new situation: Other field values than before are possible; other

    summations need to be made; the charts and the KPIs get other values than before. The state vectors

    and the objects are invalidated.Everything needs to be recalculated since this is what the user demands.

    Well, there is of course a cache also so that QlikView doesnt have to recalculate something which

    has been calculated before. So it isnt quite true that everything is recalculated: If a calculation has

    been made before, the result is simply fetched from the cache. But it is true that nothing is pre-

    calculated. There is no need for that. Everything canbe done in real-time.

    QlikView is an on-demand calculation engine.

    From a principal point, there are two steps in the recalculation of data: The logical inference in the data

    model, and the calculations of all objects, including sheet labels and alerts.

    The logical inference is done first. The goal is to figure out which field values in the symbol tables are

    possible and which records in the data tables are possible, given the new selection. There is no

    number crunching involved - it is a purely logical process. The result is stored in the state vectors.

    Think of it as if the selection propagates from one table in the data model to all other tables. Table by

    table is evaluated and QlikView figures out which values and records are possible, and which are

    excluded.

  • When the logical inference is done, QlikView starts to evaluate all exposed objects. List boxes and

    dimensions in charts must be populated and sorted. All expressions in charts, in text boxes, in

    labels, in alerts must be calculated. Objects that are on other sheets, minimized or hidden, are

    however not calculated.

    The calculations are always aggregations based on the data records that have been marked as

    possible by the logical inference engine. I.e., the objects do not persist any data on their own.

    The calculation phase is usually the phase that takes time often over 90% of the response time is

    due to calculations. The calculations are asynchronous and multi-threaded on several levels: First of

    all, every object is calculated in its own thread. Secondly, in the 64-bit version, many aggregations

    e.g. Sum() are calculated using several threads, so that a sum in one single object can be calculated

    quickly using several CPUs.

    Finally, when an object has been calculated, it is rendered. Since the calculation is asynchronous and

    multi-threaded, some objects are rendered long before other objects are ready.

    And when an object has been rendered, you can click again. And everything is repeated.

    HIC

    If you want to read more about QlikView internals, see Symbol Tables and Bit-Stuffed

    Pointers and Colors, states and state vectors.

    6. Incremental Load Souce: http://www.resultdata.com/qlikview-incremental-loads-and-qvds/

    QlikView, by design, includes many new and innovative technologies such as the associative data

    model and highly effective data compression algorithms which make possible its state-of-the-art in-

    memory technology. QlikView allows us to load and keep all the data in memory for evaluation,

  • analysis and reporting. If youve worked with QlikView you understand the value of this approach,

    but it sometimes comes with a price. Very large data sets can often take a long time to load bogging

    down the performance of your QlikView documents over time. This month we will take a look at a

    way to minimize the load time of very large data sets and increase your performance using

    incremental data loads.

    What is an Increment Load?

    Incremental load is term that describes loading only new or changed records from the database. It is

    a common task with databases and can greatly reduce the time needed to load data into your

    QlikView application. The bulk of the data needed will already be available within your application

    and only the new or changed data will be necessary to complete the picture. Incremental loads are

    possible through the use of .QVD files.

    What is a QVD file?

    A QVD file is a native QlikView file format optimized and compacted for speed when reading data

    from within a QlikView load script. Reading data from a QVD file can be 10-100 times faster than

    reading records directly from other data sources. A QVD file contains a single table of data from a

    QlikView application. While that may seem somewhat restricting remember that table can be the

    result of a concatenation or a join so the structure you create in the application can greatly increase

    its use. You can also include all calculations and manipulations in the script that creates you QVD file

    further increases load performance at run time.

    How could you use a QVD file?

    There are several uses for a QVD file and in many cases more than one will be applicable at the same

    time.

    Decreasing Load Time

    Decreasing Database Server Loads

    Joining Data from Different QlikView Applications

    Incremental Data Loading

    Decreasing Load Time

    By saving data to and loading large amounts of data from a QVD file you eliminate most of the time

    used during load by using an optimized and compressed data file. By scripting all of your

    concatenation, joining, calculations and data manipulations in the file used to create the QVD you

    will increase your performance even more.

    Decreasing Database Server Loads

    By isolating your large data volumes and loading them from QVD files you will reduce the processing

    on the database server at load time and dramatically increase the load time of your scripts as well.

  • You only need to provide data since the last load of your QVD to your QlikView document during

    refresh. The fresher the data in your QVD the less data needed from the database server.

    Joining Data from Different QlikView Applications

    Once youve formatted and manipulated your data and get it working just the way you want, you

    can save that table to a QVD and use the same vetted structure in other QlikView documents. While

    it is true that you could copy and paste your load script into another QlikView document, by using a

    QVD file instead you have the added advantage of dramatically faster loading. As your scripts

    become more and more complex based on the business questions asked by the users you can

    standardized your scripts and maintain the logic in one place. This increases our ability to create a

    single version of the truth.

    Incremental Data Loading

    By adding business logic to the creation of you QVD files you can extend that logic to all of the

    QlikView Applications that use that data; to create a more dynamic loading scenario. Lets say you

    schedule your QVD loads monthly, after the close of business for the previous month. You

    application now only needs to load data for the current period directly from the database and then

    load all previous periods from your QVD file.

    Incremental Load Scenarios

    The structure of your data, available evaluation fields and how you choose to store your data in

    QVDs will determine your particular scenario but here are a few examples to get you started

    thinking.

    Daily Incremental Reloads of a QVD

    This scenario requires a date field in data that identifies the creation or update time of all records.

    We can retrieve the last modified/created date from the existing QVD, use that date to retrieve new

    records from the database and then concatenate the previously saved records from the QVD file to

    our current data and save the QVD file again.

    1. Load the latest (max) modified date from you previously saved QVD. If you have not yet

    created the QVD then use the current date.

    2. Select only the records from the database where the last modified date is between the last

    modified date you retrieved in step on and right now.

    3. Concatenate all the data from the QVD file where there is no match in new data table on the

    unique ID field. This allows QlikView to only add the records that do not exist and accounts

    for updated records as well as new records.

    4. Save the resulting data set by overwriting the QVD file with all of the records in the new data

    set.

  • This scenario will force QlikView into Fast mode instead of Super-Fast mode but will still be

    significantly faster than loading all data from the database. You may also need to extend this logic to

    your production QlikView Application if it needs to retrieve data since the last daily load.

    Daily/Monthly/Yearly Stacked QVDs

    At close of each Day, Month or Year (Month and/or Year being the most popular) you will create a

    QVD containing that periods data. Each QVD will be named so that the data in them is clearly

    identified by the name (I.E.: 3-1-2010.qvd or 3-2010.qvd or 2010.qvd). You may wish to use a

    combination approach such as saving data from previous year in a yearly QVD and data within the

    current year in a monthly QVD. This will give you the option of loading only the appropriate data into

    your QlikView Applications. Depending on the target audience for your application you may need

    different combinations of data. One application might require all available data while other may only

    require a specific number of years past. A more analytic application may only require yearly and/or

    monthly data while others will require up-to-the-minute data. This approach will give you flexibility

    for all of those scenarios.

    Another advantage of this approach is that the daily, monthly or yearly data can be loaded in Super-

    Fast mode since no date evaluation is needed. Only the data needed to supplement the application

    since the last saved QVD file, if any, will be read directly from the database.

    7. Three types of Incremental Load Source: http://www.resultdata.com/three-types-of-qlikview-incremental-loads/

    Large transactional tables can be significantly time consuming in a reload. Taking advantage of

    Qlikviews incremental load logic can reduce that time considerably. An incremental load is a process

    of simply appending new records or updating specific records in an existing QVD. There are three key

    scripting options available for incremental loads.

    Insert Only

    Insert & Update

    Insert, Update, & Delete

    For a detailed review of QVDs and the concept of incremental loads, please review the following

    article

    Incremental Loads and QVDs

    SET UP

    Each of these three scenarios is designed to run once an INITIAL LOAD has occurred. An initial load is

    a task that creates the source QVDs. These QVDs from then on can be optimized to reload with one

    of the following incremental load scripts. Since an incremental load is designed to pull only new or

  • altered data, a source QVD is needed to hold all non-modified information and must exist before an

    incremental load can run.

    INSERT ONLY

    For an INSERT ONLY scenario, there is the assumption that new data will not create duplicate

    records. There is no set way to determine NEW data, so this must be reviewed case by case. Once a

    method for finding new records is determined, the reload process is a simple three step process.

    1. Load all NEW data from the data source

    2. Concatenate this data with a load of all data from the QVD file

    3. Store the entire table out to the QVD file

    As long as the QVD is named the same, this will overwrite the previous QVD so the process can

    repeat for the next reload.

    INSERT & UPDATE

    The INSERT & UPDATE scenario also takes new data from the source but it also pulls in updated

    records. Additional precautions need to be taken in order to avoid duplicate records. During the load

    from the QVD, exclude records where there is a match on the primary key. This will ensure that the

    updated records will not be duplicated.

    1. Load all NEW and UPDATED data from the data source

  • 2. Concatenate this data with a load of only the missing records from the QVD file

    3. Store the entire table out to the QVD file

    Example of Script

    Data:

    SQL SELECT

    PrimaryKey,

    A,

    B,

    C

    FROM DB_Table

    WHERE ModifyDate >= $(vDate);

    CONCATENATE

    LOAD

    PrimaryKey,

    A,

    B,

    C

    FROM Data.qvd

    WHERE NOT exists (PrimaryKey);

    STORE Data into Data.qvd;

    Using the Exists() function keeps the QVD from loading the obsolete records since the UPDATED

    version is currently in memory.

    INSERT, UPDATE, & DELETE

    An INSERT, UPDATE, & DELETE script is very similar to the load process of the INSERT & UPDATE,

    however there is an additional step needed to remove deleted records. The most effective method is

    to load all the PrimaryKeys from the source and then apply an inner join. This will achieve the delete

    process.

    1. Load all NEW and UPDATED data from the data source

    2. Concatenate this data with a load of only the missing records from the QVD file

    3. Inner join all PrimaryKeys from the data source

    4. Store the entire table out to the QVD file

    Example of Script

  • Data:

    SQL SELECT

    PrimaryKey,

    A,

    B,

    C

    FROM DB_Table

    WHERE ModifyDate >= $(vDate);

    CONCATENATE

    LOAD

    PrimaryKey,

    A,

    B,

    C

    FROM Data.qvd

    WHERE NOT exists (PrimaryKey);

    INNER JOIN

    SQL SELECT

    PrimaryKey,

    FROM DB_Table;

    STORE Data into Data.qvd;

    Very large data sets can take a long time to load and greatly effect the performance of your QlikView

    documents over time. By implementing QVD optimization with incremental loads, this technique can

    be employed to perform faster loads in less time, utilizing less system resources.

    8. Qlikview Associative data model Source: http://community.qlik.com/blogs/theqlikviewblog/2010/08/16/qlikview-is-associative-to-

    its-very-core

    One thing we're trying to do a better job of at QlikTech is communicating the associative nature of

    QlikView. I've seen lots of conversations taking place online (for example on the QlikCommunity

    site as well as Donald Farmer's blog andCurt Monash's blog). So I tapped into the brains of Dan

    English, our Global Product Manager for OEM and Integration for his explanation, and I'm sharing it

    with you here.

    First and foremost we should clear up the semantics. If one uses the Wikipedia definition of an

    associative model of data then it is correct to say that QlikView does not store data in an associative

    format. However, QlikTech uses the word associative in an entirely different sense. When we say

    that QlikView is associative we mean that at a data engine level QlikView creates and maintains real-

    time associations among all result sets, creating a cohesive and intuitive view of business

    information.

  • We describe QlikView's architecture as associative to differentiate it from query-based business

    intelligence tools. With all query-based BI tools (whether ROLAP, MOLAP, or HOLAP) each individual

    result set is returned from the underlying data engine without any inherent association back to the

    data repository as a whole, or to any other query result set (see figure below).

    When we say QlikView is associative, we aren't talking just about QlikView's intuitive user

    interface?the UI that utilizes green for selected data, white for associated data, and gray for

    unassociated data to illustrate relationships hidden in business information. (See this QlikView blog

    post.) We're talking about a revolution in data engine architecture, in that:

    Every data point in a QlikView document shares a common selection state. With QlikView's

    data engine, each and every discrete data point in a given QlikView document?whether it is

    part of an aggregated result set (e.g., straight table, pivot table, chart, etc.) or unaggregated

    data (e.g., data in a list box)?shares a common selection state (e.g., universe of included and

    excluded data).

    All data points are constantly updated based on the selection state. All the data points in a

    QlikView document are continually and instantaneously updated based on changes the user

    makes to the selection state. The associations among result sets are maintained 100% by the

    underlying data engine, which is built on a column-store, in-memory architecture.

    QlikView's associative architecture delivers unprecedented flexibility

    Why is QlikView's associative engine so important? One might argue that a real-time query tool gives

    you the capability to answer any question you want. After all, within the limits of the tool's user

    interface, you can define any result set you want, right? We maintain that the answers to real-world

    business questions are almost never exposed in the result set of a single query. Almost always the

    answer can only be extracted by examining the relationships of two or more associated result sets,

    often aggregated along completely different dimensionality.

    The bottom line: QlikView represents a fundamentally different class of analytic engine. All

    associations are based on the data model set up when the QlikView document is developed. Those

    associations are used to update every single result set in real time each and every time the user

    changes the selection state. This is the source of QlikView's associative magic.

    9. The magic of variables Source: http://community.qlik.com/blogs/qlikviewdesignblog/2013/11/04/the-magic-of-variables

    Variables can be used in many ways in QlikView. They can have static values or they can be

    calculated. But when are they calculated? At script run-time or when the user clicks? And how

    should they be called? With or without dollar expansion?

  • One basic way to assign a value to a variable is to use a Let statement in the script:

    Let vToday = Num(Today()) ;

    This will calculate the expression and assign it to the variable when the script is run. This is exactly

    what you want if you want to use a variable as a numeric parameter in your expressions.

    But if you want the expression to be evaluated at a later stage, e.g. every time the user clicks, what

    should you do then? One way is to store the expression as a string in the variable, using either the

    Set or the Let statement or by defining it in the Document Properties -> Variables:

    Set vSales = Sum(Sales) ;

    Let vSales = 'Sum(Sales)' ;

    In neither case, the expression will be calculated. The variable will contain the string Sum(Sales),

    which subsequently can be used in an expression using a dollar expansion: $(vSales).

    With a dollar expansion, QlikView will substitute the $(vSales) with Sum(Sales) before the

    expression with the dollar expansion is evaluated. Some of you will recognize this as an old style

    assembler macro expansion. The subsequent calculation will be made based on the evaluation of the

    resulting expression. Note the two steps: (1) Variable expansion; and (2) Expression evaluation.

  • In the chart above, you can see the result of using a normal variable reference (the first expression)

    or using a dollar expansion (the second expression). In the second expression, the variable is

    expanded and the numbers are calculated correctly.

    But this is just the beginning

    It is also possible to calculate the variable value, i.e. determine how it should be expanded, by using

    an initial equal sign in the variable definition.

    Let vSales2 = '=Sum(Sales)';

    In this case, the variable value is calculated after each click, whereupon the dollar expansion in the

    chart expression is made, and finally the expression is evaluated. This means that the evaluation of

    Sum(Sales) is done before the variable expansion. Note the three steps: (1) Variable calculation; (2)

    Variable expansion; and (3) Expression evaluation.

    The table below summarizes the three methods.

  • With the above, you can do almost magical things. You can for instance make conditional

    calculations that depend on e.g. selections, client platform or user.

    Example:

    Create a field [Field to Aggregate] containing the names of two other numeric fields:

    'Quantity' and 'Sales'

    Create a variable vConditionalAggregationField = '=Only([Field to Aggregate])'

    Create a chart with an expression = Sum($(vConditionalAggregationField))

    The calculation in a chart will now toggle between Sum(Quantity) and Sum(Sales) depending on your

    selection.

  • The use of variables is an extremely powerful tool that you can use to create flexible applications.

    Use it but with caution. Too much magic behind the curtains can be confusing.

    10. The QlikView Cache Source: http://community.qlik.com/blogs/qlikviewdesignblog/2014/04/14/the-qlikview-cache

    QlikView has a very efficient, patented caching algorithm that effectively eliminates the calculation

    time for calculations that have been made before. In other words, if you use the back button in the

    toolbar, or if you happen to make a selection that you have made before, you usually get the result

    immediately. No calculation is necessary.

    But how does it work? What is used as lookup ID?

    For each combination of data set and selection or data sub-set and expression QlikView calculates a

    digital fingerprint that identifies the context. This is used as lookup ID and stored in the cache

    together with the result of the calculation.

  • Here "calculation" means both the Logical Inference and Chart calculation - or in fact, any expression

    anywhere. This means that both intermediate and final results of a selection are stored.

    There are some peculiarities you need to know about the cache

    The cache is global. It is used for all users and all documents. A cache entry does not belong

    to one specific document or one user only. So, if a user makes a selection that another user

    already has made, the cache is used. And if you have the same data in two different apps,

    one single cache entry can be used for both documents.

    Memory is not returned, when the document is unloaded. Cache entries will usually not be

    purged until the RAM usage is close to or has reached the lower working set limit. QlikView

    will then purge some entries and re-use the memory for other cache entries. This behavior

    sometimes makes people believe there is a memory leak in the product. But have no fear it

    should be this way. So, you do not need to restart the service to clear the cache.

    The oldest cache entries are not purged first. Instead several factors are used to calculate a

    priority for each cache entry; factors like RAM usage, cost to calculate it again and time since

    the most recent usage. Entries with a combined low priority will be purged when needed.

    Hence, an entry that is cheap to calculate again will easily be purged, also if it recently was

    used. And another value that is expensive to recalculate or just uses a small amount of RAM

    will be kept for a much longer time.

    The cache is not cleared when running macros which I have seen some people claim.

  • You need to write your expression exactly right. If the same expression is used in several

    places, it should be written exactly the same way Capitalization, same number of spaces,

    etc. otherwise it will not be considered to be the same expression. If you do, there should

    be no big performance difference between repeating the formula, referring to a different

    expression using the label of the expression or using the Column() function.

    The cache efficiently speeds up QlikView. Basically it is a way to trade memory against CPU-time: If

    you put more memory in your server, you will be able to re-use more calculations and thus use less

    CPU-time.

    11. Null handling in QlikView Source: http://community.qlik.com/docs/DOC-3155

    Refer: Null and Nothing.pdf

    12. Text searches Source: http://community.qlik.com/blogs/qlikviewdesignblog/2013/10/16/text-searches

    One of the strengths of QlikView is its search engine. With it, you can find pieces of information in a

    fraction of a second and select the found field values. The response is immediate, which is necessary

    for the user experience. Without it, you would easily get an empty result set without understanding

    why.

    Search strings can be made in many different ways, and QlikView will respond differently depending

    on how the search string is defined. Normally you just enter a text, and QlikView will match this

    against the beginning of the words in the field values. If several strings are entered, QlikView will

    return the union of the matches of each of the strings.

  • But if you instead use a wildcard in your search string, the evaluation of the search string will be

    made in a different way: the entire search string with the wild card will be matched against the

    entire field value, sometimes yielding more matches, sometimes fewer.

    If you want to create more complex search strings (and e.g. store them in actions or bookmarks) you

    can do this too. Just use (, |, & and double quotes to define the syntax.

  • In all the above cases, the search and the selection are made in one and the same field. But

    sometimes you want to make the selection in one field, but make the search in another. This can be

    done using the associated search, which is an indirect search method. Start with the field where you

    want to make the selection, enter the search string, and click on the small chevron to the right. You

    will then get a list of other fields containing this search string. By clicking the desired match, you will

    narrow down the number of matches in the primary list to show just the relevant values. You can

    then make your selection by hitting Enter.

    Further, did you know that

    In the user preferences and in the list box properties, you can define how a default search

    string should be created, but this does not affect how it is evaluated only how it is created.

    Once created, you can add or remove wild cards as you please.

    When you make a search and save the resulting selection in a bookmark, the bookmark will

    contain the search string and not the list of selected values. When the bookmark is applied,

    it will perform the search and select the found values. If data has changed, this may imply a

    different search result than before.

  • You can use the same search string in many places: In list boxes, in Set analysis, in the

    Advanced search dialog, in actions and in bookmarks.

    Bottom line: The search string is a powerful tool that helps you find the values you want. Use it.

    13. Automatic Number interpretation Source: http://community.qlik.com/blogs/qlikviewdesignblog/2013/07/07/automatic-number-

    interpretation

    I have in several previous blog posts written about the importance to interpret dates and numbers

    correctly e.g. in Why dont my dates work?. These posts have emphasized the use of interpretation

    functions in the script, e.g. Date#().

    But most of the time, you dont need any interpretation functions, since there is an automatic

    interpretation that kicks in before that.

    So, how does that work?

    In most cases when QlikView encounters a string, it tries to interpret the string as a number. It

    happens in the script when field values are loaded; it happens when strings are used in where-

    clauses, or in formulae in GUI objects, or as function parameters. This is a good thing QlikView

    would otherwise not be able to interpret dates or decimal numbers in these situations.

    QlikView needs an interpretation algorithm since it can mix data from different sources, some typed,

    some not. For example, when you load a date from a text file, it is always a string: there are no data

    types in text files it is all text. But when you want to link this field to date from a database, which

    usually is a typed field, you would run into problems unless you have a good interpretation

    algorithm.

  • For loaded fields, QlikView uses the automatic interpretation when appropriate (See table: In a text

    file, all fields are text - also the ones with dates and timestamps.) QlikView does not use any

    automatic interpretation for QVD or QVX files, since the interpretation already is done. It was done

    when these files were created.

    The logic for the interpretation is straightforward: QlikView compares the encountered string with

    the information defined in the environment variables for numbers and dates in the beginning of the

    script. In addition, QlikView will also test for a number with decimal point and for a date with the ISO

    date format.

    If a match is found, the field value is stored in a dual format (see Data Types in QlikView) using the

    string as format. If no match is found, the field value is stored as text.

    An example: A where-clause in the script:

    Where Date > '2013-01-01' will make a correct comparison

    The field Date is a dual that is compared to a string. QlikView automatically interprets the string on

    the right hand side and makes a correct numeric date comparison. QlikView does not (at this stage)

    interpret the content of the field on the left hand side of the comparison. The interpretation should

    already have been done.

    A second example: The IsNum() function

    IsNum('2013-01-01') will evaluate as True

    IsNum('2013-01-32') will evaluate as False

    In both cases, strings are used as parameters. The first will be considered a number, since it can be

    interpreted as a date, but the second will not.

  • A third example: String concatenation

    Month(Year & '-' & Month & '-' & Day) will recognize correct dates and return the dual month

    value.

    Here the fields Year, Month and Day are concatenated with delimiters to form a valid date format.

    Since the Month() function expects a number (a date), the automatic number interpretation kicks in

    before the Month() function is evaluated, and the date is recognized.

    A final example: The Dual() function

    Dual('Googol - A large number', '1E100') will evaluate to a very large number

    Here the second parameter of Dual() is a string, but QlikView expects a number. Hence: automatic

    interpretation. Here, you can see that scientific notation is automatically interpreted. This

    sometimes causes problems, since strings that really are strings in some cases get interpreted as

    numbers. In such cases you need to wrap the field in a text function.

    With this, I hope that the QlikView number handling is a little clearer.

    14. Why dont my dates work?

    Source: http://community.qlik.com/blogs/qlikviewdesignblog/2013/02/19/why-don-t-my-dates-

    work

    A common recurring question on the QlikCommunity forum is around dates that dont work. Here

    follows a help on fixing the three most common causes. If you encounter such a question on the

    forum, just link to this post in your answer.

    1. Incorrect Date Interpretation

    When data is loaded into QlikView, dates are often read as strings. QlikView then tries to recognize a

    pattern in the string that looks like the date format specified in the DateFormat environment

    variable. This sometimes fails and then you need to use the Date#() function to help QlikView

    understand that it is a date.

  • How do I know that a date is correctly interpreted? Thats easy. Just format it as a number and see

    what you get. (List box properties Number Integer)

    The question is now what your list box looks like. If you have a number which is roughly 40000

    (usually right-aligned), then you are all set. But if you still have a date stamp (usually left-aligned),

    then you need to use the Date#() function in the script. See QlikView Date fields.

    2. Linking integer dates with fractional dates

    You have a date in two different tables, and you want to use this date as a key, but it doesnt seem

    to work. Then you should suspect that you have true dates (integers) in one table and timestamps

    (fractional numbers) in the other, but the formatting of the dates hides this fact.

    How do I know whether this is the case? Thats easy. Just format it as a timestamp and see what you

    get. (List box properties Number TimeStamp)

  • The question is now what your list box looks like. If you have timestamps where hours, minutes and

    seconds are all zero, then you are all set. But if you have numbers in these places, then you need to

    use the Floor() function in the script to get integer dates. See QlikView Date fields.

    3. Incorrect date comparisons

    The most subtle error is however the one with timestamps in comparisons, e.g.

    Where Date = '2011-12-31';

  • Will this work? Yes, provided that the date format inside the string is recognized by QlikView, i.e.

    that it corresponds to the date format specified in the environment variable DateFormat in the

    beginning of the script.

    It becomes even more complex if you use variables. Then it is important to use quotes correctly. The

    following will work:

    Let vToday = Today();

    Where Date = '$(vToday)';

    but the following will not:

    Where Date = $(vToday);

    The reason is that the $(vToday) will expand to the date, and then the comparison will be e.g.

    Where Date = 2/19/2013;

    So the date (which is approximately 40000) will be compared to 2 divided by 19 divided by 2013,

    which of course is not what you want.

    My recommendation is to always use numeric variables for dates. They always work - quotes or no

    quotes:

    Let vToday = Num(Today());

    Where Date = $(vToday);

    15. Colors in Chart Source: http://community.qlik.com/blogs/qlikviewdesignblog/2012/12/04/colors-in-charts

    It is not uncommon that users want specific products or customers to be displayed in specific colors.

    The most obvious way to do this is to change the colors in the chart properties. This is in fact quite

    easy if you use the copy and paste functions found when you right-click a color button. Just copy one

    button and paste on another, and you have moved the color.

  • This way you can assign which color is to be used for the different values of the field. However, a

    prerequisite for this to work is that the order of the field values doesnt change.

    A more robust way is to use color functions. Usually, you want to set the color of a bar, line or

    bubble and this is done by using the Background Color on the Expression tab:

    By the way dont use Visual cues. This feature is old and not very versatile. Use color functions as

    described here instead.

    In the picture above, both the product ID and the color are hard-coded in the expression. However,

    if you want to define colors for many products, the if-function will not be manageable. Then it is

    better to store this information in a table either in the database or in an Excel sheet or as an inline

    statement in a scriptlet that is included in the script. Hence,

  • 1. Create your color definition table and store it in an appropriate place. The Red, Green and

    Blue columns hold the different color components and define the color uniquely.

    2. Load the color definitions into a mapping table:

    ProductColors:

    Mapping Load ProductID, Rgb(Red,Green,Blue) as ProductColor From ProductColors

    3. Use this mapping table when loading the products table, creating a new field for the product

    color:

    Applymap('ProductColors', ProductID , lightgray()) as ProductColor

    The third parameter, here lightgray(), defines which color the unlisted products should get. If

    you instead use null(), the unlisted products will be multicolored according to the color

    settings in the chart properties.

    4. Finally, use this field as product color in the charts:

    This way it is easy to define which color specific products, customers, or other dimensions should

    get.

    Which colors to use? Oh, that is a completely different topic:

    Stephen Few has a number of good general recommendations.

    Adam Bellerby has some recommendations on how to avoid problems for color blind users.

    Shima Auzins suggests using colors as warning signals.

  • HIC

    16. Aggregations and Function Classes Source: http://community.qlik.com/blogs/qlikviewdesignblog/2014/05/19/function-classes

    A typical QlikView application may have one million records in the data, one hundred rows in a pivot

    table and a single number, a KPI, in a gauge or text box. Although different in magnitudes, all three

    numbers may still represent all data. The numbers are just different aggregation levels.

    There are many functions in QlikView that can help you write the necessary formulae to calculate

    aggregated KPI:s. Some will collapse many records into one value, others will not. Today I will write

    about the different function classes, and how you can combine them.

    The Scalar Functions constitute the first class. Typical for these is that they are

    one-to-one functions, i.e. they take a single value as parameter and return a single value (of

    the dual data type). Examples: Left(), If(), Num(), Date(), Year(), Subfield(), etc.

    The Aggregation Functions constitute the second class. These are many-to-one

    functions, i.e. they use the values from many records as input and collapse these into one

    single value that summarizes all records. Examples: Sum(), Count(), Avg(), Min(), Only(),

    Concat(), etc.

  • Aggregation functions are special: You must use one to collapse several records into one number

    which means that you need them in pretty much any formula in QlikView: In Chart expressions, in

    Text boxes, in Labels, etc. If you dont write an aggregation function in your expression, QlikView will

    assign one for you: It will use the Only() function.

    Scalar functions can be used both inside and outside the aggregation function:

    Date( Min( Date )

    Money( Sum( If( Group='A', Amount ) ) )

    There is one restriction: You can normally not use an aggregation function inside another

    aggregation function. Hence, you usually need every field reference to be wrapped in exactly

    one aggregation function.

    The next function class has only one member: The Aggr Function. It is in spite

    of its name not an aggregation function. It is a many-to-many function, rather like a tensor

    or a matrix in mathematics. It converts an ntuple (table) with N records to an ntuple with M

    records. In other words: It returns an array of values. Regard it as a virtual straight table with

    one expression and one or several dimensions.

    Most places in QlikView demand that you write your expression so that it returns one single value.

    This means that you must wrap the Aggr function in an aggregation function to get a meaningful

    result. The only exception is if you use the Aggr function to define a calculated dimension or field.

    This means that you have two aggregation steps; one nested in the other:

    Avg( Aggr( Sum( Amount ), Month ) )

  • Charts complicate the matters slightly: A chart is like a For-Next loop where the number of distinct

    dimension values determines the number of loops. In each loop, the expression must return one

    value only, and this is the value used for the bar/slice/pivot table row.

    However, sometimes you need values from other rows in the chart, and it could even be that you

    need values from several rows. To solve this, there are two additional classes of functions that

    should be used together:

    The Chart Inter-record Functions return values fetched from other rows in the

    chart. Some of these can return several values, i.e. an array of values. These functions are

    only meaningful inside a chart or Aggr() function. Examples: Above(), Below(), Top(), etc.

    The Range Functions are functions that can collapse a chart inter-record array

    into one single value. Examples: RangeSum(), RangeMin(), RangeMax(), etc.

    Example:

    RangeSum( Above( Sum( Amount ), 0, 12 ) )

    Bottom line: Know your functions. It will help you write correct expressions.

    17. Its all Aggregations Source: http://community.qlik.com/blogs/qlikviewdesignblog/2013/08/06/it-s-all-aggregations

    I often see incorrect expressions being used in the QlikCommunity forum. Expressions that seem to

    work correctly but really dont

    So, let me make this clear: Calculations in QlikView are aggregations.

  • It doesnt matter if it is a measure in a chart, or a calculated object label, or a show condition for an

    object, or a calculated color, or an advanced search all expressions in the user interface are

    evaluated as aggregations. (Except calculated dimensions.)

    This means that it is correct to use the Sum() function in an expression, since this is an aggregation

    function - a function that uses several records as input. But if you omit the aggregation function or

    use a scalar function only, e.g. RangeSum(), you can get an unexpected behavior.

    Basically, all field references should be wrapped in an aggregation function. The Aggr() function and

    some constructions using the total qualifier can even have several layers of aggregations.

    But if the created expression does not contain an aggregation function, the expression is ill-formed

    and potentially incorrect.

    Examples:

    =Sum(Amount)

    =Count(OrderID)

    These are both correct aggregations. Amount is wrapped in the Sum() function which will sum

    several records of the field Amount. OrderID is wrapped in the Count() function, which will count the

    records where OrderID has a value.

    =Only(OrderID)

    This is also a correct aggregation. OrderID is wrapped in the Only() function, which will return the

    OrderID if there is only one value, otherwise NULL.

  • =OrderID

    A single field reference is not an aggregation, so this is an ill-formed expression. But QlikView will

    not throw an error. Instead it will use the Only() function to interpret the field reference. I.e., if there

    is only one value, this value will be used. But if there are several possible values, NULL will be used.

    So, it depends on the circumstances whether an expression without aggregation function is correct

    or not.

    =If(Year=Year(Today()), Sum(Amount1), Sum(Amount2))

    Here, both the amounts are correctly wrapped in the Sum() function. But the first parameter of the

    if() function, the condition, is not. Hence, this is an ill-formed expression. If it is used in a place where

    there are several possible Years, the field reference will evaluate to NULL and the condition will be

    evaluated as FALSE, which is not what you want. Instead, you probably want to wrap the Year in the

    Min() or Max() function.

    =ProductGroup= 'Shoes'

    =IsNull(ProductGroup)

    These expressions can both be used as show conditions or as advanced searches. However, since

    there are no aggregation functions, the expressions are ill-formed. If you want to test whether there

    exists Shoes or NULL values among the field values, you probably want to use the following instead:

    =Count(If(ProductGroup= 'Shoes', ProductGroup))>0

    =NullCount(ProductGroup)>0

    Conclusions:

    An aggregation function is a function that returns a single value describing some property

    of several records in the data.

    All UI expressions, except calculated dimensions, are evaluated as aggregations.

    All field references in expressions must be wrapped in an aggregation function. If they

    arent, QlikView will use the Only() function.

    18. Dimensions and Measures Source: http://community.qlik.com/blogs/qlikviewdesignblog/2013/03/25/dimensions-and-

    measures

    To make a chart in QlikView or in any Business Intelligence tool, for that matter you need to know

    what Dimensions and Measures are. But not all people have a clear picture of the difference between

    the two. So this weeks post will try to straighten out whats what.

  • When you make a chart, you should start by asking yourself What do I want to show? The answer

    is usually Sales, Quantity or some other number. This is your Measure. In QlikView we have

    traditionally called this an Expression, but "Measure" is really the correct word. (There are

    expressions that are not measures, e.g. expressions used as labels, or as sort order definitions).

    The second question you should ask yourself is How many times should this be calculated? Per

    what do I want to show this measure? The answer could be once per Month, per Customer, per

    Supplier or something similar. This is your Dimension.

    In the bar chart below, you have one bar per month, and a general rule is that you always have one

    data point per distinct dimensional value: But depending on which visualization form you have

    chosen, it can be a row (in a table), a point (in a scatter chart) or a slice (in a pie chart).

    Measures

    A database or a QlikView app can consist of thousands or millions of records that each contains a

    small piece of information. A Measure is simply a calculation that can be made over multiple records

    in this data set. The calculation always returns one single value that summarizes all relevant records.

    This type of calculation is called an aggregation. There are several aggregation functions: Sum(),

    Count(), Min(), Max(), etc.

  • Examples:

    Each record contains a sales number. Then Sum(Sales) is a relevant measure that calculates

    the total sales value.

    Each record represents an order and OrderID is the key. Then Count(OrderID) is a relevant

    measure that calculates the number of orders.

    A Measure can be used almost anywhere in QlikView: In charts, in text boxes, as label for objects, in

    gauges, etc. Typical measures are all KPI:s, Revenue, Number of orders, Performance, Cost, Quantity,

    Gross Margin, etc.

    Once again: A Measure is always based on an aggregation. Always!

    Dimensions

    Contrary to Measures, dimensions are descriptive attributes typically textual fields or discrete

    numbers. A dimension is always an array of distinct values and the measure will be calculated once

    per element in the array.

    Example:

    The field Customer is used as dimension. The individual customers will then be listed and

    the measure will be calculated once per customer.

    Typical dimensions are Customer, Product, Location, Supplier, Activity, Time, Color, Size, etc.

    Like a For-Next loop

    You can regard a chart like a For-Next loop: The Dimension is the loop variable; the calculations will

    be made once per dimensional value. So the Dimension determines how many

    rows/bars/points/slices the chart will have. The Measure is what is calculated in each loop.

    Several Dimensions

  • If you have two or three dimensions in a chart, the dimensional values no longer form an array, but

    instead a matrix or a cube, where the measures are calculated once per cell in the cube.

    SQL

    You can also compare a chart with an SQL SELECT statement. The GROUP BY symbols are the

    dimensions and the aggregations are the Measures.

    With this, I hope that the difference between Dimensions and Measures is a little clearer.

    19. Qlikview Quoteology Source: http://community.qlik.com/blogs/qlikviewdesignblog/2013/04/09/quoteology

  • In all programming environments there is a need for quotation marks, and QlikView is no exception.

    But which symbol should you use? " ", [ ], ` ` or ' '? This post will try to explain the differences

    between the different quotation marks.

    When creating the script or an expression in QlikView, you need to reference fields, explicit values

    and variables. To do this correctly, you sometimes need to write the string inside a pair of quotation

    marks. One common case is when a field name contains a symbol that prevents QlikView from

    parsing it correctly, like a space or a minus sign.

    For example, if you have a field called Unit Cost, then

    Load Unit Cost

    will cause a syntax error since QlikView expects an "as" or a comma after the word "Unit". If you

    instead write

    Load [Unit Cost]

    QlikView will load the field Unit Cost. Finally, if you write

    Load 'Unit Cost'

    QlikView will load the text string "Unit Cost" as field value. Hence, it is important that you choose

    the correct quotation mark.

    So, what are the rules? Which quote should I use? Single? Double? Square brackets?

    There are three basic rules:

    1. Single quotes are used to quote literals, e.g. strings that should be used as field values.

    2. Inside a Load statement, to the left of the as, double quotes are used to quote source field

    references, i.e. names of fields.

    3. Double quotes can always be substituted by square brackets or by grave accents.

    With these three rules, most cases are covered. However, they dont cover everything, so I'll

    continue:

  • In the script, but outside a Load statement, double quotes denote a variable reference and

    not a field reference. If double quotes are used, the enclosed string will be interpreted as a

    variable and the value of the variable will be used.

    A general rule in QlikView is that field references inside a Load must refer to the fields in the input

    table the source of the Load statement. They are source field references or in-context field

    references. Aliases and fields that are created in the Load cannot be referred since they do not exist

    in the source. There are however a couple of exceptions: the functions Peek() and Exists(). The first

    parameters of these functions refer to fields that either have already been created or are in

    the output of the Load. These are out-of-context field references.

    Out-of-context field references and table references, e.g. the parameters in NoOfRows() and

    Peek(), should be regarded as literals and therefore need single quotes.

    Finally, in many places you are free to use any of the four quotation methods, e.g.

    o Inside a Set statement, to the right of the =

    o Inside a Load statement, to the right of the as

    o In places where QlikView expects a file name, a URL or a table name

    o Defining the beginning and end of an inline table

    o For the first parameter of Peek() or Exists() when used inside a Load

    I have deliberately chosen not to say anything about SELECT statements. The reason is that the rules

    depend on which database and which ODBC/OLEDB you have. But usually, rules 1-3 apply there also.

    With this, I hope that the QlikView quoteology is a little clearer.

    20. The Crosstable Load Source: http://community.qlik.com/blogs/qlikviewdesignblog/2014/03/24/crosstable

    There are a number of prefixes in QlikView, that help you load and transform data. One of them is

    the Crosstable transformation.

    Whenever you have a crosstable of data, the Crosstable prefix can be used to transform the data

    and create the desired fields. A crosstable is basically a matrix where one of the fields is displayed

    vertically and another is displayed horizontally. In the input table below you have one column per

    month and one row per product.

  • But if you want to analyze this data, it is much easier to have all numbers in one field and all months

    in another, i.e. in a three-column table. It is not very practical to have one column per month, since

    you want to use Month as dimension and Sum(Sales) as measure.

    Enter the Crosstable prefix.

    It converts the data to a table with one column for Month and another for Sales. Another way to

    express it is to say that it takes field names and converts these to field values. If you compare it

    to the Generic prefix, you will find that they in principle are each others inverses.

    The syntax is

    Crosstable (Month, Sales) Load Product, [Jan 2014], [Feb 2014], [Mar 2014], From ;

    There are however a couple of things worth noting:

    Usually the input data has only one column as qualifier field; as internal key (Product in the

    above example). But you can have several. If so, all qualifying fields must be listed before the

    attribute fields, and the third parameter to the Crosstable prefix must be used to define the

    number of qualifying fields.

  • It is not possible to have a preceding Load or a prefix in front of the Crosstable keyword.

    Auto-concatenate will however work.