sap iq performance and tuning series: basics

Administration Guide | PUBLICSAP IQ 16.1 SP 05Document Version: 1.0.0 – 2022-09-22

SAP IQ Performance and Tuning Series: Basics

© 2

022

SAP

SE o

r an

SAP affi

liate

com

pany

. All r

ight

s re

serv

ed.

THE BEST RUN

Content

1 SAP IQ Performance and Tuning Series: Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Performance Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Configuring Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73.1 Setting the Number of CPUs Available. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 The Process Threading Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.3 Network Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Configuring the Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.1 Understanding Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Server Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Required Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Cache Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Large Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13IQ Page Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14Wired Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Tuning Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Optimizing for Typical Usage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Optimizing for Large Numbers of Users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Restricting Concurrent Queries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Limiting Query Temp Space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Limiting Queries by Rows Returned. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Forcing Cursors to be Non-Scrolling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Limiting the Number of Cursors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Limiting the Number of Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23Prefetching Cache Pages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Controlling the Number of Prefetched Rows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Controlling File System Buffering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Optimizing the Cache Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.3 Balancing Input/Output. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Raw Devices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Disk Striping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Internal Striping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Random and Sequential File Access. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Transaction and Message Logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.4 Monitoring Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Database Profiling Procedures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

2 PUBLICSAP IQ Performance and Tuning Series: Basics

Content

Event Profiling Procedures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Key Performance Indicators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Buffer Cache Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5 Tuning your Multiplex. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .405.1 Managing Multiplex Disk Space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405.2 Managing Logical Server Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.3 Solving Bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .415.4 Balancing Query Loads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6 Designing your Schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436.1 Indexing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Indexing Tips. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45Index Storage Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Indexing Small Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46Indexing Large Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48HG Index Loads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Multi-Column Indexes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

6.2 Join Column. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .506.3 Primary Keys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.4 Foreign Keys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.5 Proper Data Type Sizing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526.6 Null Values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536.7 Unsigned Data Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546.8 LONG VARCHAR and LONG VARBINARY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546.9 Large Object Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.10 Temporary Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566.11 Denormalizing for Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576.12 UNION ALL Views for Faster Loads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Queries Referencing UNION ALL Views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59UNION ALL View Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6.13 Hash Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647.1 Monitoring SAP IQ Statement Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647.2 Isolating Performance Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667.3 Diagnostic Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677.4 Common Performance Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Paging and Disk Swapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68Index and Row Fragmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69Catalog File Growth. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70Thrashing and Query Execution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

SAP IQ Performance and Tuning Series: BasicsContent PUBLIC 3

Views and Tables Using Wide Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

8 Structuring Queries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738.1 Structuring Queries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Enhancing ORDER BY Query Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74Improved Subquery Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74Using Caching Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Zone Maps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

8.2 Distributed Query Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Influencing Parallelism and Scalability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77Controlling Distributed Queries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Queries Likely to Benefit from DQP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Queries Unlikely to Benefit from DQP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

8.3 Generating Query Plans. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Query Evaluation Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81Using Query Plans. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83Viewing DQP in a Query Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84Timing Details in a Query Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85Thread Details in a Query Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

8.4 Controlling Query Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87Setting Query Time Limits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87Setting Query Priority. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88Setting Query Optimization Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Setting User-Supplied Condition Hints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90Monitoring Workloads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

8.5 Optimizing Delete Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91HG Delete Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91WD Delete Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93TEXT Delete Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94


Content

1 SAP IQ Performance and Tuning Series: Basics

Tuning databases for improved performance.

SAP IQ Performance and Tuning Series: BasicsSAP IQ Performance and Tuning Series: Basics PUBLIC 5

2 Performance Considerations

Performance is usually measured in response time and throughput. A good design and indexing strategy leads to the largest performance gains.

Response time is the time it takes for a single task to complete. Several factors affect response time:

• Reducing contention and wait times, particularly disk I/O wait times• Using faster components• Reducing the amount of time the resources are needed (increasing concurrency)

Throughput refers to the volume of work completed in a fixed time period. Throughput is commonly measured in transactions per second (tps), but can be measured per minute, per hour, per day, and so on.

To realize the largest performance gains run SAP IQ on a correctly configured system, establish a good design,and choose the correct indexing strategy.

Other considerations, such as hardware and network analysis, can locate bottlenecks in your installation.

Related Information

Solving Bottlenecks [page 41]


Performance Considerations

3 Configuring Hardware

Hardware issues that impact SAP IQ performance.

In this section:

Setting the Number of CPUs Available [page 7]Set the -iqnumbercpus startup switch to specify the number of CPUs available. This parameter overrides the physical number of CPUs for resource planning purposes.

The Process Threading Model [page 7]SAP IQ uses operating system kernel threads for best performance. By default, SAP IQ allocates the number of threads based on the number of CPUs on the system.

Network Performance [page 8]Minor changes in your environment can solve some network performance issues.

3.1 Setting the Number of CPUs Available

Set the -iqnumbercpus startup switch to specify the number of CPUs available. This parameter overrides the physical number of CPUs for resource planning purposes.

Using the -iqnumbercpus switch is recommended only:

• On machines with Intel CPUs and hyperthreading enabled, set -iqnumbercpus to the actual number of cores

• On machines where an operating system utility has been used to restrict SAP IQ to a subset of the CPUs within the machine

3.2 The Process Threading Model

SAP IQ uses operating system kernel threads for best performance. By default, SAP IQ allocates the number of threads based on the number of CPUs on the system.

Lightweight processes are underlying threads of control that are supported by the kernel. The operating system decides which lightweight processes (LWPs) should run on which processor and when. It has no knowledge about what the user threads are, but does know if they are waiting or able to run.

The operating system kernel schedules LWPs onto CPU resources. It uses their scheduling classes and priorities. Each LWP is independently dispatched by the kernel, performs independent system calls, incurs independent page faults, and runs in parallel on a multiprocessor system.

SAP IQ Performance and Tuning Series: BasicsConfiguring Hardware PUBLIC 7

A single, highly threaded process serves all SAP IQ users. The database server assigns varying numbers of kernel threads to each user connection, based on the type of processing being done by that connection, the total number of threads available, and the various option settings.

Insufficient Threads Error

If there are insufficient threads for a query, SAP IQ generates this error:Not enough server threads available for this queryThis condition may well be temporary. When some other query finishes, threads are made available and the query may succeed the next time. If the condition persists, you may need to restart the server and specify more SAP IQ threads. It is also possible that -iqmt is set too low for the number of connections.

SAP IQ Options for Managing Thread Usage

• Use the server start-up option -iqmt to set the maximum number of threads. The default value is calculated from the number of connections and the number of CPUs and is usually adequate.

• Use the server start-up option -iqtss to set the stack size of the internal execution threads. The default value is generally sufficient, but may be increased if complex queries return an error indicating that the depth of the stack exceeded this limit.

• Use the SET OPTION MAX_IQ_THREADS_PER_CONNECTION command to set the maximum number of threads for a single user. The SET OPTION MAX_IQ_THREADS_PER_TEAM command sets the number of threads available to a team of threads, enabling you to constrain the number of threads (and thereby the amount of system resources) allocated to a single operation.

• Use these options to control the amount of resources an operation consumes. For example, you can set this option before issuing an INSERT, LOAD, BACKUP DATABASE, or RESTORE DATABASE command.

3.3 Network Performance

Minor changes in your environment can solve some network performance issues.

To improve network throughput, provide multiple network adaptors. Classes of users can be assigned to different networks depending on service level agreements.

In case A (see the figure below) clients accessing two different database servers use one network card. That means that clients accessing Servers A and B must compete over the network and past the network card. In the case B, clients accessing Server A use a different network card than clients accessing Server B.

It would be even better to put your database servers on different machines. You may also want to put heavy users of different databases on different machines.


Configuring Hardware

Figure 1: Isolating heavy network users

Put Small Amounts of Data in Small Packets

If you send small amounts of data over the network, keep the default network packet size small (default is 512 bytes). The -p server start-up option lets you specify a maximum packet size. Your client application may also let you set the packet size.

Put Large Amounts of Data in Large Packets

If most of your applications send and receive large amounts of data, increase default network packet size. This will result in fewer (but larger) transfers.

Process at the Server Level

Filter as much data as possible at the server level.

SAP IQ Performance and Tuning Series: BasicsConfiguring Hardware PUBLIC 9

4 Configuring the Server

Tune SAP IQ for faster query execution.

In this section:

Understanding Memory [page 10]Understanding how SAP IQ allocates memory can help you get the best performance from your system.

Tuning Options [page 16]Tuning options that provide faster query execution.

Balancing Input/Output [page 26]Use disk striping, random and sequential file disk access to balance Input/Output (I/O).

Monitoring Performance [page 31]Tools you can use to determine whether your system is making optimal use of available resources.

4.1 Understanding Memory

Understanding how SAP IQ allocates memory can help you get the best performance from your system.

In this section:

Server Memory [page 11]SAP IQ allocates heap memory for buffers, transactions, databases, and servers. Shared memory may also be used, but in much smaller quantities.

Required Memory [page 11]After you determine how much physical memory the operating system and other applications require, calculate how much of the remaining memory is required by SAP IQ.

Cache Memory [page 13]Allocate as much memory as possible to the IQ main and temporary buffer caches for optimal performance. Change the defaults to accommodate loads, queries, and applications.

Large Memory [page 13]The -iqlm startup parameter specifies the maximum amount of large memory that SAP IQ can dynamically request from the operating system.

IQ Page Size [page 14]IQ page and buffer cache size affect memory use and disk I/O throughput for the database.

Wired Memory [page 15]On HP and Solaris platforms, you can designate a specified amount of memory as wired memory. Wired memory is shared memory that is locked into physical memory. The kernel cannot page this memory out of physical memory.


Configuring the Server

4.1.1 Server Memory

SAP IQ allocates heap memory for buffers, transactions, databases, and servers. Shared memory may also be used, but in much smaller quantities.

At the operating system level, SAP IQ server memory consists of heap memory. For the most part, you do not need to be concerned with whether memory used by SAP IQ is heap memory or shared memory. All memory allocation is handled automatically. Make sure that your operating system kernel is correctly configured to use shared memory before you run SAP IQ

Most operating systems use a large percent of available memory for file system buffering. Understand the buffering policies for your operating system to avoid over-allocating memory.

The total memory used for SAP IQ main and temporary buffer caches, plus SAP IQ memory overhead, and memory used for the operating system and other applications, must not exceed the physical memory on your system.

Related Information

Required Memory [page 11]Cache Memory [page 13]Large Memory [page 13]IQ Page Size [page 14]Wired Memory [page 15]

4.1.2 Required Memory

After you determine how much physical memory the operating system and other applications require, calculate how much of the remaining memory is required by SAP IQ.

Raw Partitions Versus File Systems

For UNIX-like operating systems, databases using file systems rather than raw partitions may require another 30% of the remaining memory to handle file buffering by the operating system. On Windows, file system caching should be disabled by setting OS_FILE_CACHE_BUFFERING = ‘OFF’ (the default for new databases).

Multiuser Database Access

For multiuser queries of a database, SAP IQ needs about 10 MB per “active” user. Active users are defined as users who simultaneously access or query the database. For example, 30 users may be connected to SAP IQ, but only 10 or so may be actively using a database at any one time.

SAP IQ Performance and Tuning Series: BasicsConfiguring the Server PUBLIC 11

Memory for Thread Stacks

Processing threads require a small amount of memory. The more processing threads you use, the more memory needed. The -iqmt server switch controls the number of threads. The -iqtss and -gss server switches control the amount of stack memory allocated for each thread. The total memory allocated for IQ stacks is roughly equal to: (-gn * (-gss + -iqtss)) + (-iqmt * -iqtss ).

If you have a large number of users, the memory needed for processing threads increases. The -gn switch controls the number of tasks (both user and system requests) that the database server can execute concurrently. The -gss switch controls — in part — the stack size for server execution threads that execute these tasks. SAP IQ calculates the stack size of these worker threads using the following formula: (-gss + -iqtss).

The total number of threads (-iqmt plus -gn) must not exceed the number allowed for your platform.

Other Memory Use

All commands and transactions use some memory. The following operations are the most significant memory users in addition to those discussed previously:

• Backup. The amount of virtual memory used for backup is a function of the IQ PAGE SIZE specified when the database was created. It is approximately 2 * number of CPUs * 20 * (IQ PAGE SIZE/16). On some platforms, you may be able to improve backup performance by adjusting BLOCK FACTOR in the BACKUP DATABASE command, but increasing BLOCK FACTOR also increases the amount of memory used.

• Database validation and repair. When you check an entire database, the sp_iqcheckdb procedure opens all tables, their respective fields, and indexes before initiating any processing. Depending on the number of tables and the cumulative number of columns and indexes in those tables, sp_iqcheckdb may require very little or a large amount of virtual memory. To limit the amount of memory needed, use the sp_iqcheckdb options to check or repair a single index or table.

• Dropping leaked blocks. The dropleaks operation also needs to open all tables, files, and indexes, so it uses as much virtual memory as sp_iqcheckdb uses when checking an entire database. It uses the temp buffer cache to keep track of blocks used.

Related Information

Server Memory [page 11]Cache Memory [page 13]Large Memory [page 13]IQ Page Size [page 14]Wired Memory [page 15]



4.1.3 Cache Memory

Allocate as much memory as possible to the IQ main and temporary buffer caches for optimal performance. Change the defaults to accommodate loads, queries, and applications.

Default cache size for the main and temporary buffer caches is 64 MB each. Cache size is controlled with the -iqmc (main buffer cache) and -iqtc (temporary cache) server startup options. These startup options only remain in effect while the server is running, so you need to include them every time you restart the server.

Large memory requirements are one third of the total available physical memory allocated to SAP IQ. To ensure adequate memory for the main and temporary IQ stores, set the -iqlm, -iqmc, and -iqtc startup parameters so that each parameter receives one third of all available physical memory.

Related Information

Server Memory [page 11]Required Memory [page 11]Large Memory [page 13]IQ Page Size [page 14]Wired Memory [page 15]

4.1.4 Large Memory

The -iqlm startup parameter specifies the maximum amount of large memory that SAP IQ can dynamically request from the operating system.

Some load operations may require more large memory than the 2 GB default provides. If memory requirements exceed the default, use the -iqlm startup option to increase the memory that SAP IQ can dynamically request from the OS. Set -iqlm as a switch as part of the command or configuration file that starts the server.

Large Memory Allocation

As a general rule, large memory requirements represent one third of the total available physical memory allocated to SAP IQ. To ensure adequate memory for the main and temporary IQ stores, set the -iqlm, -iqtc, and -iqmc startup parameters so that each parameter receives one third of all available physical memory allocated to SAP IQ.

In most cases, you should allocate 80 percent of total physical memory to SAP IQ to prevent SAP IQ processes from being swapped out. Adjust actual memory allocation to accommodate other processes running on the same system. For example, on a machine with 32 cores and 128 GB of total available physical memory, you would allocate 100 GB (approximately 80 percent of the 128 GB total) to SAP IQ processes. Following the general rule, you would set the -iqlm, -iqtc, and -iqmc parameters to 33 GB each.


Related Information

Server Memory [page 11]Required Memory [page 11]Cache Memory [page 13]IQ Page Size [page 14]Wired Memory [page 15]

4.1.5 IQ Page Size

IQ page and buffer cache size affect memory use and disk I/O throughput for the database.

SAP IQ performs I/O in units of page size. When you create a database, you specify a separate page size for the catalog store and the IQ store. The temporary store has the same page size as the IQ store.

Page size for the catalog store has no real impact on performance. The default value of 4096 bytes should be adequate. The IQ page size determines two other performance factors, the default I/O transfer block size, and the maximum data compression for your database. SAP IQ compresses all data. The amount of compression is determined by the IQ page size.

Saving Memory

If your machine does not have enough memory, increase the memory and decrease the buffer cache sizes. Decreasing the buffer caches too much, however, can make your data loads or queries inefficient or incomplete due to insufficient buffers.

NoteThe page size cannot be changed and determines the upper size limit on some database objects and whether LOB features can be used.

Related Information

Server Memory [page 11]Required Memory [page 11]Cache Memory [page 13]Large Memory [page 13]Wired Memory [page 15]



4.1.6 Wired Memory

On HP and Solaris platforms, you can designate a specified amount of memory as wired memory. Wired memory is shared memory that is locked into physical memory. The kernel cannot page this memory out of physical memory.

Wired Memory Pool

Wired memory may improve SAP IQ performance when other applications are running on the same machine at the same time. Dedicating wired memory to SAP IQ, however, makes it unavailable to other applications on the machine.

To create a pool of wired memory on these UNIX platforms only, specify the -iqwmem command line switch, indicating the number of MB of wired memory (you must be user root to set -iqwmem, except on Solaris). On 64-bit platforms, the only upper limit on -iqwmem is the physical memory on the machine.

For example, on a machine with 14 GB of memory, you may be able to set aside 10 GB of wired memory. To do so, you specify:

-iqwmem 10000

NoteUse -iqwmem only if you have enough memory to dedicate the amount you specify for this purpose. Otherwise, you can cause serious performance degradation.

• On Solaris – -iqwmem always provides wired memory.• On HP – -iqwmem provides wired memory if you start the server as root. It provides unwired memory if

you are not root when you start the server. This behavior may change in a future version.

Impact of Other Applications and Databases

Server memory comes out of a pool of memory used by all applications and databases. If you try to run multiple servers or multiple databases on the same machine at the same time, or if you have other applications running, you may need to reduce the amount of memory your server requests.

You can also issue the UNIX command ipcs -mb to see the actual number of segments.

Troubleshooting HP Memory Issues

On HP-UX, check the value of the maxdsiz_64bit kernel parameter. This parameter restricts the amount of virtual memory available to SAP IQ on 64-bit HP processors. See the SAP IQ Installation and Update Guide for your platform for the recommended value.


Related Information

Server Memory [page 11]Required Memory [page 11]Cache Memory [page 13]Large Memory [page 13]IQ Page Size [page 14]

4.2 Tuning Options

Tuning options that provide faster query execution.

In this section:

Optimizing for Typical Usage [page 17]Set the USER_RESOURCE_RESERVATION option to adjust memory use for the number of current users.

Optimizing for Large Numbers of Users [page 18]Adjust the startup parameters for large numbers of users.

Restricting Concurrent Queries [page 19]Set the -iqgovern switch to specify the number of concurrent queries on a particular server. This is not the same as the number of connections, which is controlled by your license.

Limiting Query Temp Space [page 20]Set the QUERY_TEMP_SPACE_LIMIT to specify the maximum estimated amount of temp space before a query is rejected.

Limiting Queries by Rows Returned [page 21]Set the value of the QUERY_ROWS_RETURNED_LIMIT option to prevent the optimizer from rejecting queries with large result sets.

Forcing Cursors to be Non-Scrolling [page 21]Eliminate the temporary store node in queries that return a very large result set to improve performance.

Limiting the Number of Cursors [page 22]Set the MAX_CURSOR_COUNT option to prevent a single connection from taking too much available memory or CPU resources.

Limiting the Number of Statements [page 23]Set the MAX_STATEMENT_COUNT option to limit the number of prepared statements for a connection can make.

Prefetching Cache Pages [page 23]Set the BT_PREFETCH_MAX_MISS option to control prefetch memory behavior.

Controlling the Number of Prefetched Rows [page 24]Set the PrefetchRows and PrefetchBuffer parameters to improve performance on cursors under certain conditions. This is a client option that you can set on the ODBC connection dialog, or in the .odbc.ini file.



Controlling File System Buffering [page 25]On some file systems, you can turn file system buffering on or off. Turning file system buffering off usually reduces paging and improves performance.

Optimizing the Cache Partitions [page 25]Changing the CACHE_PARTITIONS value may improve load or query performance in a multi-CPU configuration.

4.2.1 Optimizing for Typical Usage

Set the USER_RESOURCE_RESERVATION option to adjust memory use for the number of current users.

SAP IQ tracks the number of open cursors and allocates memory accordingly. In certain circumstances, USER_RESOURCE_RESERVATION option can be set to adjust the minimum number of current cursors that SAP IQ thinks is currently using the product and hence allocate memory from the temporary cache more sparingly.

Set this option only if required. Contact Technical Support with details if you need to set this option.

Related Information

Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]


4.2.2 Optimizing for Large Numbers of UsersAdjust the startup parameters for large numbers of users.

Parameter Description

-iqgovern Places a ceiling on the maximum number of queries to execute at once. If more users than the -iqgovern limit have submitted queries, new queries will be queued until one of the active queries is finished. Usage:

-iqgovern <#_ ACTIVE_ queries_to_support>

The optimal value for -iqgovern depends on the nature of your queries, number of CPUs, and size of the buffer cache. The default value is 2*<numCPU> + 10. With a large number of connected users, you may find that setting this option to 2*<numCPU> + 4 provides better throughput.

-gn Sets the number of execution threads for the catalog store and connectivity while running with multiple users. Usage:

-gn <number of tasks (both userand system requests) that the database server can execute concurrently>

The correct value for -gn depends on the value of -gm. The start_iq utility calculates -gn and sets it appropriately. Setting -gn too low can prevent the server from operating correctly.

-c Sets the catalog store cache size. Usage:

-c <catalog_store_cache_size>

The catalog cache size is highly dependent on schema size and the number of objects. The catalog store buffer cache is also the general memory pool for the catalog store. To specify in MB, use the form -c <nM>, for example, -c <64M>.

For up to 1000 users, set -c to 16MB or higher. For up to 2000 users, set -c to 48MB (default).

-cl and -ch Set upper (-ch) and lower (-cl) limits for the catalog store cache size. <-cl> <minimum cache size> -ch <maximum cache size>. If the standard catalog cache size is too small, set -cl and -ch parameters.

Do not use -c in the same configuration file or command line with -ch or -cl. For related information, see the -ch cache-size option.

-iqmt Sets the number of processing threads. If -iqmt is set too low for the -gm setting, then thread starvation can occur.

NoteTo control catalog store cache size explicitly, you must do either of the following, but not both, in your configuration file (.cfg) or on the command line for server startup:



• Set the -c parameter• Set specific upper and lower limits for the catalog store cache size using the -ch and -cl parameters

Related Information

Optimizing for Typical Usage [page 17]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]

4.2.3 Restricting Concurrent Queries

Set the -iqgovern switch to specify the number of concurrent queries on a particular server. This is not the same as the number of connections, which is controlled by your license.

There is an optimal value for -iqgovern that will provide the correct number of concurrent query access to provide optimal throughput. If -iqgovern is set over this threshold, contention or resource starvation occurs, slowing down all requests.

By specifying the -iqgovern switch, you can help SAP IQ optimize paging of buffer data out to disk, and avoid over committing memory. The default value of -iqgovern is (2 x the number of CPUs) + 10. You may need to experiment to find an ideal value. For sites with large numbers of active connections, try setting -iqgovernslightly lower.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]


Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]

4.2.4 Limiting Query Temp Space

Set the QUERY_TEMP_SPACE_LIMIT to specify the maximum estimated amount of temp space before a query is rejected.

The QUERY_TEMP_SPACE_LIMIT option causes queries to be rejected if their estimated temp space usage exceeds the specified size. By default, there is no limit on temporary store usage by queries.

SAP IQ estimates the temporary space needed to resolve the query. If the estimate exceeds the current QUERY_TEMP_SPACE_LIMIT setting, SAP IQ returns an error:

Query rejected because it exceeds total space resource limitIf this option is set to 0 (the default), there is no limit, and no queries are rejected based on their temporary space requirements.

To limit the actual temporary store usage per connection, set the MAX_TEMP_SPACE_PER_CONNECTION option for all DML statements, including queries. MAX_TEMP_SPACE_PER_CONNECTION monitors and limits the actual run time temporary store usage by the statement. If the connection exceeds the quota set by the MAX_TEMP_SPACE_PER_CONNECTION option, an error is returned and the current statement rolls back.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]



4.2.5 Limiting Queries by Rows Returned

Set the value of the QUERY_ROWS_RETURNED_LIMIT option to prevent the optimizer from rejecting queries with large result sets.

The QUERY_ROWS_RETURNED_LIMIT option tells the query optimizer to reject queries that might otherwise consume too many resources. If the query optimizer estimates that the result set from a query will exceed the value of this option, it rejects the query with the message:

Query rejected because it exceed resource: Query_Rows_Returned_Limit

Set this option only to reject queries that consume vast resources.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]

4.2.6 Forcing Cursors to be Non-Scrolling

Eliminate the temporary store node in queries that return a very large result set to improve performance.

When you use scrolling cursors with no host variable declared, SAP IQ creates a temporary store node where query results are buffered. This storage is separate from the temporary store buffer cache. The temporary store node enables efficient forward and backward scrolling when your application searches through a result set.

However, if the query returns very large numbers (such as millions) of rows of output, and if your application performs mostly forward-scrolling operations, the memory requirements of the temporary store node may degrade query performance. To improve performance, eliminate the temporary store node by issuing the following command:

SET TEMPORARY OPTION FORCE_NO_SCROLL_CURSORS = 'ON'


NoteIf your application performs frequent backward-scrolling, setting the FORCE_NO_SCROLL_CURSORS option to ON may actually degrade query performance, as the absence of the temporary cache forces SAP IQ to re-execute the query for each backward scroll.

If your application rarely performs backward-scrolling, make FORCE_NO_SCROLL_CURSORS = 'ON' a permanent PUBLIC option. It will use less memory and improve query performance.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]

4.2.7 Limiting the Number of Cursors

Set the MAX_CURSOR_COUNT option to prevent a single connection from taking too much available memory or CPU resources.

The MAX_CURSOR_COUNT option limits the maximum number of cursors that a connection can use at once. The default is 50. Setting this option to 0 allows an unlimited number of cursors.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]



Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]

4.2.8 Limiting the Number of Statements

Set the MAX_STATEMENT_COUNT option to limit the number of prepared statements for a connection can make.

The MAX_STATEMENT_COUNT option limits the maximum number of prepared statements that a connection can use at once. If a server needs to support more than the default number (50) of prepared statements at any one time for any one connection, then you can set the MAX_STATEMENT_COUNT option to a higher value.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]

4.2.9 Prefetching Cache Pages

Set the BT_PREFETCH_MAX_MISS option to control prefetch memory behavior.

The BT_PREFETCH_MAX_MISS option determines whether to continue prefetching pages for a given query. If queries using HG indexes run more slowly than expected, try gradually increasing the value of this option.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]


Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]

4.2.10 Controlling the Number of Prefetched Rows

Set the PrefetchRows and PrefetchBuffer parameters to improve performance on cursors under certain conditions. This is a client option that you can set on the ODBC connection dialog, or in the .odbc.ini file.

Prefetching improves performance on cursors that only fetch relative 1 or relative 0. Two connection parameters let you change cursor prefetch defaults. PrefetchRows (PROWS) sets the number of rows prefetched; PrefetchBuffer (PBUF) sets the memory available to this connection for storing prefetched rows. Increasing the number of rows you prefetch may improve performance under certain conditions:

• The application fetches many rows (several hundred or more) with very few absolute fetches.• The application fetches rows at a high rate, and the client and server are on the same machine or

connected by a fast network.• Client/server communication is over a slow network, such as a dial-up link or wide area network.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling File System Buffering [page 25]Optimizing the Cache Partitions [page 25]



4.2.11 Controlling File System Buffering

On some file systems, you can turn file system buffering on or off. Turning file system buffering off usually reduces paging and improves performance.

On Windows you can disable file system buffering for IQ Main dbspaces of existing databases by issuing the following statement:

SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING = 'OFF'

On Windows you can disable file system buffering for IQ Temporary dbspaces of existing databases by issuing the following statement:

SET OPTION "PUBLIC".OS_FILE_CACHE_BUFFERING_TEMPDB = 'OFF'

You can only set this option on Windows for the PUBLIC. Shut down the database and restart it for the change to take effect.

NoteWindows can bias the paging algorithms to favor applications at the expense of the file system. This bias is recommended for SAP IQ performance.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Optimizing the Cache Partitions [page 25]

4.2.12 Optimizing the Cache Partitions

Changing the CACHE_PARTITIONS value may improve load or query performance in a multi-CPU configuration.

SAP IQ automatically calculates the number of cache partitions for the buffer cache according to the number of CPUs on your system. If load or query performance in a multi-CPU configuration is slower than expected, you may be able to improve it by changing the value of the CACHE_PARTITIONS database option.


As buffers approach the Least Recently Used (LRU) end of the cache, they pass a wash marker. SAP IQ writes the oldest pages — those past the wash marker — out to disk so that the cache space they occupy can be reused. A team of SAP IQ processing threads, called sweeper threads, sweeps (writes) out the oldest buffers.

When SAP IQ needs to read a page of data into the cache, it grabs the LRU buffer. If the buffer is still “dirty” (modified) it must first be written to disk. The Gdirty column in the monitor -cache report shows the number of times the LRU buffer was grabbed dirty and SAP IQ had to write it out before using it.

Usually SAP IQ can keep the Gdirty value at 0. If this value is greater than 0 for more than brief periods, you may need to adjust one of the database options that control the number of sweeper threads and the wash marker.

Related Information

Optimizing for Typical Usage [page 17]Optimizing for Large Numbers of Users [page 18]Restricting Concurrent Queries [page 19]Limiting Query Temp Space [page 20]Limiting Queries by Rows Returned [page 21]Forcing Cursors to be Non-Scrolling [page 21]Limiting the Number of Cursors [page 22]Limiting the Number of Statements [page 23]Prefetching Cache Pages [page 23]Controlling the Number of Prefetched Rows [page 24]Controlling File System Buffering [page 25]CACHE_PARTITIONS OptionSWEEPER_THREADS_PERCENT OptionWASH_AREA_BUFFERS_PERCENT Option

4.3 Balancing Input/Output

Use disk striping, random and sequential file disk access to balance Input/Output (I/O).

In this section:

Raw Devices [page 27]You can create a database or dbspace on a raw device or a file system file.

Disk Striping [page 28]Striping data across multiple disks is an essential technique for good performance.

Internal Striping [page 28]Disk striping takes advantage of multiple disk spindles and provides the speed of parallel disk writes.

Random and Sequential File Access [page 29]



https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a62ef8de84f2101599bb9ae14767ff72.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a65a401984f21015921f928c403787e3.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a667a9e684f21015a87fa925d061b800.html

Performance related to randomly accessed files can be improved by increasing the number of disk drives devoted to those files, and therefore, the number of operations per second performed against those files.

Transaction and Message Logs [page 30]Manage the size of the transaction and message logs to conserve disk space.

4.3.1 Raw Devices

You can create a database or dbspace on a raw device or a file system file.

Disk partitions are typically accessed in two modes: file system mode (for example through the UFS file system) or raw mode. Raw mode does unbuffered I/O, generally making a data transfer to or from the device with every read or write system call. UFS is the default UNIX file system, and is a buffered I/O system, which collects data in a buffer until it can transfer an entire buffer at a time.

SAP IQ uses the path name you specify to determine whether the partition is a raw partition or a file system file. Raw partitions can be any size.

When you allocate file system files for dbspaces (System, IQ main, or IQ temporary), do not place the files on a file system that is shared over a local area network. Doing so can lead to poor I/O performance and other problems, including overloading the local area network. Any storage type can be used for SAP IQ so long as it is configured in a way that SAP IQ supports (raw devices on any platform, local file systems for databases on any platform, NFS on Linux, and GPFS on Linux) and follows proper sizing guidelines.

For Solaris, use the mount options forcedirectio and noforcedirectio. See https://docs.oracle.com/cd/E53394_01/html/E54789/gntim.html#SVNFSgntio .

forcedirectio improves performance of large sequential data transfers. Data is copied directly to a user buffer. No caching is performed in the kernel on the client. This option is OFF by default (noforcedirectio).

For an example of how to use this option, refer to your Solaris documentation.

To avoid conflicts, restrict the management of dbspaces to a single database administrator on a single connection.

Related Information

Disk Striping [page 28]Internal Striping [page 28]Random and Sequential File Access [page 29]Transaction and Message Logs [page 30]


http://help.sap.com/disclaimer?site=https%3A%2F%2Fdocs.oracle.com%2Fcd%2FE53394_01%2Fhtml%2FE54789%2Fgntim.html%23SVNFSgntio

http://help.sap.com/disclaimer?site=https%3A%2F%2Fdocs.oracle.com%2Fcd%2FE53394_01%2Fhtml%2FE54789%2Fgntim.html%23SVNFSgntio

4.3.2 Disk Striping

Striping data across multiple disks is an essential technique for good performance.

Disk striping can be performed at different places in a system, often as part of RAID hardware or software, for example:

• At the device layer, such as on a disk array or controller.• In the operating system or dedicated device management software, such as Veritas.• In the application.

By default, SAP IQ internally stripes pages across all files within a dbspace, so additional striping at the software or hardware level are not needed for performance. Of course, additional striping may be necessary as part of implementing storage redundancy for the database, for example if RAID-5 is used.

Best performance in SAP IQ with storage redundancy is achieved with simple mirroring or “RAID-1”. As stated above, SAP IQ will distribute the data across all of the 2-disk mirror sets within a dbspace.

Due to cost, most SAP IQ databases will not use mirroring, and will be implemented with RAID-5 or a similar RAID level to achieve redundancy. With RAID-5, choosing an appropriate chunk size (how much data is written to one disk before moving on to the next disk) will have a significant performance impact on the system, since RAID-5 has a significant write overhead. If your application does frequent or time-sensitive loads, updates, or deletes, or if queries often do temp dbspace I/O, a smaller chunk size in the range of 25-50% of the size of an SAP IQ database page will likely give best performance. If your application is mostly reads, with little write activity, a larger chunk size 75-100% of an SAP IQ page size will likely provide best performance

Since SAP IQ normally attempts to prefetch multiple reads or flush multiple writes in parallel, even with only a single active query, using a very small chunk size to spread each page read or write across many disks will have little benefit, and will usually hurt performance.

When using RAID, best performance is usually achieved using hardware (such as controller or array) based RAID. Software based RAID tools will work well, but may add a modest additional performance load on the server’s CPUs.

Related Information

Raw Devices [page 27]Internal Striping [page 28]Random and Sequential File Access [page 29]Transaction and Message Logs [page 30]

4.3.3 Internal Striping

Disk striping takes advantage of multiple disk spindles and provides the speed of parallel disk writes.

SAP IQ provides disk striping, options without using third-party software. If you already have a disk striping solution through third-party software and hardware, use that method instead. Disk striping can be enabled by specifying the STRIPING ON option to the CREATE DBSPACE command.



To change the default striping when creating a dbspace:

SET OPTION "PUBLIC".DEFAULT_DISK_STRIPING = { ON | OFF }

The default for the DEFAULT_DISK_STRIPING option is ON for all platforms. When disk striping is ON, incoming data is spread across all dbspaces with space available. When disk striping is OFF, dbspaces (disk segments) are filled up from the front on the logical file, filling one disk segment at a time.

Changing the value of DEFAULT_DISK_STRIPING affects all subsequent CREATE DBSPACE operations that do not specify a striping preference.

You can remove a file from a dbspace using the ALTER DBSPACE DROP command when disk striping is on. Before dropping the dbspace, however, you must relocate all of the data in the dbspace using the sp_iqemptyfile stored procedure. Because disk striping spreads data across multiple files, the sp_iqemptyfile process may require the relocation of many tables and indexes. Use the sp_iqdbspaceinfo and sp_iqdbspace stored procedures to determine which tables and indexes reside on a dbspace.

Related Information

Raw Devices [page 27]Disk Striping [page 28]Random and Sequential File Access [page 29]Transaction and Message Logs [page 30]

4.3.4 Random and Sequential File Access

Performance related to randomly accessed files can be improved by increasing the number of disk drives devoted to those files, and therefore, the number of operations per second performed against those files.

Random files include those for the IQ store, the temporary store, the catalog store, programs (including the SAP IQ executables, user and stored procedures, and applications), and operating system files.

Conversely, performance related to sequentially accessed files can be improved by locating these files on dedicated disk drives, thereby eliminating contention from other processes. Sequential files include the transaction log and message log files.

To avoid disk bottlenecks:

• Keep random disk I/O away from sequential disk I/O. Also for best performance, use only one partition from a physical device (disk or HW RAID set) per dbspace.

• Isolate SAP IQ database I/O from I/O in other databases or other I/O intensive application.• Place the database file, temporary dbspace, and transaction log file on the same physical machine as the

database server.


Related Information

Raw Devices [page 27]Disk Striping [page 28]Internal Striping [page 28]Transaction and Message Logs [page 30]

4.3.5 Transaction and Message Logs

Manage the size of the transaction and message logs to conserve disk space.

The transaction log file contains recovery and auditing information. Place the transaction log on a separate device or partition from the database itself to avoid database file fragmentation and to protect against media failure.

The transaction log can consume a large amount of disk space over time. Truncate the transaction log periodically to conserve disk space.

To truncate the log:

1. Shut down the server.2. Start the server with the –m parameter as part of the start_iq command or .cfg file.3. Shut down and restart the server without the –m parameter.

Do not leave the –m switch permanently set. When –m is set, there is no protection against media failure on the device that contains the database files. Remove the –m from the .cfg after you restart the server. To move or rename the transaction log file, use the transaction log utility (dblog).

CautionThe SAP IQ transaction log file is different from most relational database transaction log files. If for some reason you lose your database files, then you lose your database (unless it is the log file that is lost). However, if you have an appropriate backup, then you can reload the database.

Message Log

SAP IQ logs all messages in the message log file, including error, status, and insert notification messages. Limit the size of this file to conserve disk space.

At some sites the message log file tends to grow rapidly. To limit the size of this file:

• Set a maximum file size and archive the log files when the active message log is full• Increase NOTIFY_MODULUS database option setting• Use the NOTIFY parameter to turn off notification messages in LOAD TABLE. INSERT, and CREATE INDEX

statements• Use -iqmsgsz switch to limit the size of the message log



Related Information

Raw Devices [page 27]Disk Striping [page 28]Internal Striping [page 28]Random and Sequential File Access [page 29]-iqmsgsz Database Server Option-m Database Server OptionNOTIFY_MODULUS OptionCREATE INDEX StatementINSERT StatementLOAD TABLE Statement

4.4 Monitoring Performance

Tools you can use to determine whether your system is making optimal use of available resources.

In this section:

Database Profiling Procedures [page 31]Stored procedures that return database usage statistics.

Event Profiling Procedures [page 33]Event profiling procedures return performance statistics for stored procedures, functions, events, and triggers.

Key Performance Indicators [page 33]Set up a Statistics Collection in SAP IQ Cockpit to monitor key performance indicators (KPI) on the server. KPI values are grouped into collections and appear in SAP IQ Cockpit monitors.

Buffer Cache Performance [page 35]Buffer cache performance is a key factor in overall performance. The IQ UTILITIES Statement starts a cache monitor that collects buffer caches statistics. Use output from the cache monitor to fine-tune main and temp buffer cache memory allocation.

4.4.1 Database Profiling Procedures

Stored procedures that return database usage statistics.

For complete reference information on these stored procedures, see SAP IQ SQL Reference.


https://help.sap.com/viewer/a893062984f21015b9e8b03f96ed0cbb/16.1.5.0/en-US/a2fafae984f2101596ba8ee24c4fc00f.html

https://help.sap.com/viewer/a893062984f21015b9e8b03f96ed0cbb/16.1.5.0/en-US/3bcbee8b6c5f1014be23ed6f194835d6.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a644aafb84f21015ad5fdadb6a18cd53.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a617ca4484f21015b2cdfdebbf4a5eee.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a61fdeff84f21015aa66b9add387d7f9.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/7ca3f60902f3473296cb309533d89210.html

Name Description

sp_iqconnection Shows information about connections and versions, including which users are using temporary dbspace, which users are keeping versions alive, what the connections are doing inside SAP IQ, connection status, database version status, and so on. Usage:

sp_iqconnection [ connhandle ]

sp_iqcontext Tracks and displays, by connection, information about statements that are currently executing. Usage:

sp_iqcontext [ connhandle ]

sp_iqcheckdb Checks validity of the current database. Optionally corrects allocation problems for dbspaces or databases. Usage:

sp_iqcheckdb 'mode target [ … ] [ resources resource-percent ]'

sp_iqdbstatistics

Reports results of the most recent sp_iqcheckdb. Usage:

sp_iqdbstatistics

sp_iqdbsize Displays the size of the current database. Usage:

sp_iqdbsize([ main ] )

sp_iqspaceinfo Displays space usage by each object in the database. Usage:

sp_iqspaceinfo [‘main | [table table-name | index index-name] [...] ‘]

sp_iqstatus Displays miscellaneous status information about the database. Usage:

sp_iqstatus

sp_iqtablesize Displays the number of blocks used by each object in the current database and the name of the dbspace in which the object is located. Usage:

sp_iqtablesize ( table_owner.table_name )

Related Information

Event Profiling Procedures [page 33]Key Performance Indicators [page 33]Buffer Cache Performance [page 35]



4.4.2 Event Profiling Procedures

Event profiling procedures return performance statistics for stored procedures, functions, events, and triggers.

Procedure Name Description

sa_server_option Sets database profiling options in Interactive SQL. Usage:

CALL sa_server_option ( 'procedureprofiling', 'ON')

sa_procedure_profile

Returns execution times for each line in database procedures, functions, events, or triggers. Usage:

sa_procedure_profile [ filename [, save_to_file ] ] )

sa_procedure_profile_summary

Summarizes execution times for all procedures, functions, events, or triggers. Usage:

sa_procedure_profile_summary [ filename [, save_to_file ] ] )

Related Information

Database Profiling Procedures [page 31]Key Performance Indicators [page 33]Buffer Cache Performance [page 35]sa_server_option System Proceduresa_procedure_profile System Proceduresa_procedure_profile_summary System Procedure

4.4.3 Key Performance Indicators

Set up a Statistics Collection in SAP IQ Cockpit to monitor key performance indicators (KPI) on the server. KPI values are grouped into collections and appear in SAP IQ Cockpit monitors.

Key performance areas include SAP IQ servers, multiplex servers, and logical servers

SAP IQ Servers Statistics

Various server status and usage statistics.


https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a22d1ccb84f210159e13ba00048f5e61.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a483c04c84f21015badef3211e1aa75f.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a48384f784f21015b5c8ca512cfde02f.html

Key Performance Area Usage Statistics

SAP IQ Availability Statistics Resource state, CPU usage, memory allocation, cache use, and active connections.

Overview Statistics Server status, CPU usage, memory allocation, and current active connections.

Connection Statistics Active, available, resumed, rolled back, and suspended user and internode connections. Also displays average number of connections and disconnections per minute.

DBSpace and DBSpace File Statistics Dbspace and dbspace file size and available percentage.

Store Input and Output Statistics Store Input and Output Statistics identify store reads and writes per second.

Cache Statistics Main buffer, catalog, and temporary cache statistics.

Operations and Request Statistics Average, minimum, maximum, and total waiting, active, and total operations.

Network Statistics Network statistics display network activity.

Transaction Statistics Transaction details currently on the server.

Multiplex and Node-Related Statistics

Multiplex and node-related statistics for multiplex servers.

Key Performance Area Description

Multiplex Availability Availability statistics for each multiplex node.

Multiplex Status Status of the multiplex.

Multiplex Node Properties Role, status, and failover state of each multiplex node.

Multiplex Link Availability Internode communication status between a secondary node and the coordinator.

Logical Server Statistics

Logical server and node-related statistics for logical servers.


Logical Server Availability Logical server status.

Logical Server Engine Statistics CPU usage, connection, and connection statistics.

Logical Server Connection Average, minimum, maximum, and total number of available connections.




Logical Server Transaction Average, minimum, maximum, and total transactions. Also displays average and minimum number of load transactions.

Logical Server Cache Statistics Average, minimum, and maximum cache use statistics for the catalog, temporary, and main buffer cache.

Logical Server Operations and Requests Statistics Average, minimum, maximum, and total waiting and active operations.

Related Information

Database Profiling Procedures [page 31]Event Profiling Procedures [page 33]Buffer Cache Performance [page 35]

4.4.4 Buffer Cache Performance

Buffer cache performance is a key factor in overall performance. The IQ UTILITIES Statement starts a cache monitor that collects buffer caches statistics. Use output from the cache monitor to fine-tune main and temp buffer cache memory allocation.

Review this checklist to isolate cache behavior that falls outside the normal range.

Statistic Normal Behavior Behavior That Needs Adjusting Recommended Action

HR% (Cache hit rate)

Above 90%.

For individual internal data structures like garray, barray, bitmap (bm), hash object, sort object, variable-length btree (btreev), fixed-length btree (btreef), bit vector (bv), dbext, dbid, vdo, store, checkpoint block (ckpt), the hit rate should be above 90% while a query runs. It may be below 90% at first. Once prefetch starts working (PF or PrefetchReqs > 0), the hit rate should gradually grow to above 90%.

Hit rate below 90% after prefetch is working.

NoteSome objects do not do prefetching, so their hit rate may be low normally.

Try rebalancing the cache sizes of main versus temp by adjusting -iqmc and -iqtc.

Also try increasing the number of prefetch threads by adjusting PREFETCH_THREADS_PERCENT option.



Gdirty (Grabbed Dirty)

0 in a system with a modest cache size (< 10 GB).

GDirty > 0

NoteSweeper threads are activated only when the number of dirty pages reaches a certain percentage of the wash area. If GDirty/GrabbedDirty is above 0 and the I/O rate (Writes) is low, the system may simply be lightly loaded, and no action is necessary.

Adjust SWEEPER_THREADS_PERCENT option (default 10%) or WASH_AREA_ BUFFERS_PERCENT option (default 20%) to increase the size of the wash area.

BWaits (Buffer Busy Waits)

0 Persistently > 0, indicating that multiple jobs are colliding over the same buffers.

If the I/O rate (Writes) is high, Busy Waits may be caused by cache thrashing. Check Hit Rate in the cache report to see if you need to rebalance main versus temp cache.

Allocate more memory to the main or temp cache, whichever has BWaits consistently > 0.

If a batch job is starting a number of nearly identical queries at the same time, try staggering the start times. Nearly identical queries would include any that touch data and use buffers, such as INSERT, UPDATE, DELETE, and SELECT.

LRU Waits (LRUNum TimeOuts percentage in debug report)

20% or less > 20%, which indicates a serious contention problem.

Check the operating system patch level and other environment settings. This problem tends to be an O.S. issue.

IOWait (IONumWaits)

10% or lower > 10% Check for disk errors or I/O retries




FLWait (FLMutexWaits)

20% or lower > 20% Check the dbspace config-uration:

Is the database almost out of space?

Is DISK_STRIPING ON?

Does sp_iqcheckdb report fragmentation greater than 15%?

HTWait (BmapHTNumWaits)

MemWts (MemNtimesWaited)

(PFMgrCondVarWaits)

10% or lower > 10% Contact SAP Technical Support.

CPU time (CPU Sys Seconds, CPU Total Seconds, in debug report)

CPU Sys Seconds < 20% CPU Sys Seconds > 20%

If CPU Total Seconds also reports LOW utilization, and there are enough jobs that the system is busy, the cache may be thrashing or parallelism may be lost.

Adjust -iqgovern to reduce allowed total number of concurrent queries.

Check Hit Rate and I/O Rates in the cache report for cache thrashing. Also check if hash object is thrashing by looking at the hit rate of the has object in cache_by_type (or debug) report: is it <90% while the I/O rate (Writes) is high?

Check query plans for attempted parallelism. Were enough threads available?

Does the system have a very large number of CPUs? Strategies such as multiplex configuration may be necessary.



InUse% (Buffers in use)

At or near 100% except during startup

Less than about 100% The buffer cache may be too large.

Try rebalancing the cache sizes of main versus temp by adjusting -iqmc and -iqtc.

Pin% (Pinned buffers)

< 90% > 90 to 95%, indicating system is dangerously close to an Out of Buffers condition, which would cause transactions to roll back

Try rebalancing the cache sizes of main versus temp.

If rebalancing buffer cache sizes is not possible, try reducing -iqgovern to limit the number of jobs running concurrently.

Free threads (ThrNumFree)

Free > Resrvd If the number of free threads drops to the reserved count, the system may be thread starved.

Try one of the following:

Increase the number of threads by setting -iqmt.

Reduce thread-related options: MAX_IQ_THREADS_ PER_CONNECTION, MAX_IQ_THREADS_ PER_TEAM.

Restrict query engine resource allocations by setting USER_RESOURCE_ RESERVATION.

Limit the number of jobs by setting -iqgovern.

FlOutOfSpace (debug only)

0, indicating that the free list for this store is not full; unallocated pages are available

1, indicating that this store (main or temporary) is fully allocated

Add more dbspace to that store

NoteIf one cache performs significantly more I/O than the other, reallocate some of the memory in small amounts, such as 10 percent of the cache allocation on an iterative basis. After reallocating, rerun the workload and monitor the performance changes.



Related Information

Database Profiling Procedures [page 31]Event Profiling Procedures [page 33]Key Performance Indicators [page 33]IQ UTILITIES Statement


https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a6203ded84f21015bc44c8462220d329.html

5 Tuning your Multiplex

Adjust your multiplex system for maximum performance or better use of disk space.

Each server in the multiplex can be on its own host or share a host with other servers. Two or more servers on the same system consume no more CPU time than a single combined server handling the same workload, but separate servers might need more physical memory than a single combined server, because the memory used by each server is not shared by any other server.

In this section:

Managing Multiplex Disk Space [page 40]Get users to commit their current transactions periodically, and allow the write server to drop old table versions to free disk blocks. Specifying the auto_commit option helps minimize space due to minimize version buildup.

Managing Logical Server Resources [page 41]Logical servers let you to manage the use of multiplex resources most effectively. Use logical servers to assign different sets of multiplex servers to different applications to meet their individual performance requirements.

Solving Bottlenecks [page 41]Tuning your system means resolving the most common bottlenecks: compute-bound and I/O bound workloads.

Balancing Query Loads [page 41]SAP IQ provides partial load balancing through distributed query processing, or full load balancing through a network client (which requires third party software).

5.1 Managing Multiplex Disk Space

Get users to commit their current transactions periodically, and allow the write server to drop old table versions to free disk blocks. Specifying the auto_commit option helps minimize space due to minimize version buildup.

SAP IQ cannot drop old versions of tables while any user on any server might be in a transaction that might need the old versions. SAP IQ may therefore consume a very large amount of disk space when table updates and queries occur simultaneously in a multiplex database. The amount of space consumed depends on the nature of the data and indexes and the update rate.

You can free disk blocks by allowing the write server to drop obsolete versions no longer required by queries. All users on all servers should commit their current transactions periodically to allow recovery of old table versions. The servers may stay up and are fully available. The sp_iqversionuse stored procedure can be used to display version usage for remote servers.


Tuning your Multiplex

5.2 Managing Logical Server Resources

Logical servers let you to manage the use of multiplex resources most effectively. Use logical servers to assign different sets of multiplex servers to different applications to meet their individual performance requirements.

In a multiplex, each connection operates under a single logical server context. When you submit a query to a multiplex server, its execution may be distributed to one or more multiplex servers, depending upon the configuration of the connection's logical server. To dynamically adjust the resources assigned to a logical server, add or remove multiplex servers from the logical server to meet the changing needs of the applications that it serves.

5.3 Solving Bottlenecks

Tuning your system means resolving the most common bottlenecks: compute-bound and I/O bound workloads.

Distributed query processing improves compute-bound workloads. Queries might be parallel, but using maximum CPUs on a single machine. Adding hardware or implementing a cluster solves bottlenecks by extending compute power, but at the expense of more licenses and more complex systems.

Related Information

Performance Considerations [page 6]Distributed Query Performance [page 76]

5.4 Balancing Query Loads

SAP IQ provides partial load balancing through distributed query processing, or full load balancing through a network client (which requires third party software).

Distributed Query Processing

DQP (distributed query processing) occurs automatically for qualifying queries on multiplex servers that meet requirements.

Multiplex servers must have established MIPC (multiplex interprocess communication connections). The logical server of the current connection must have at least one other available member node. The shared temporary dbspace must have available writable files.

SAP IQ Performance and Tuning Series: BasicsTuning your Multiplex PUBLIC 41

Load Balancing with a Network Client

Using a network client to balance the query load among multiplex query servers requires an intermediate system that can dispatch the client connection to a machine in a pool.

To use this method, create a special ODBC DSN on the client system, with the IP address and port number of this intermediate load balancing system, a generic server name, and the VerifyServerName connection parameter set to NO. When a client connects using this DSN, the load balancer establishes the connection to the machine it determines is least loaded.

NoteThird-party software is required. VerifyServerName simply allows this method to work.

Related Information

VerifyServerName (VERIFY) Communication ParameterDistributed Query Processing


Tuning your Multiplex

https://help.sap.com/viewer/8e6989ea146b41d78f671443295cd7a0/16.1.5.0/en-US/a6f0804684f210159857964e78e57c22.html

https://help.sap.com/viewer/a8986af484f21015b55bed80129dc962/16.1.5.0/en-US/a230362b84f21015b0c4978dd1653301.html

6 Designing your Schema

Good database performance begins with good database design. Take the time to incorporate design features into your schema during development for better response time and faster query results.

In this section:

Indexing [page 44]Indexing selection and solutions for SAP IQ.

Join Column [page 50]For joins, keep the data types as narrow as possible to reduce disk I/O and memory requirements.

Primary Keys [page 51]Multi-column primary keys should have an additional HG index placed on each column specified in the primary key. This must be done manually as SAP IQ only creates an HG index on the composite columns.

Foreign Keys [page 51]As with primary keys, use foreign keys to improve query join performance. This gives SAP IQ one more piece of information on how tables are joined and the statistics behind those joins.

Proper Data Type Sizing [page 52]Size all data types as accurately as possible, especially character-based data types.

Null Values [page 53]Defining columns as NULL or NOT NULL helps the optimizer work more efficiently.

Unsigned Data Types [page 54]In some cases, using unsigned data types can eliminate sign comparisons and create faster queries.

LONG VARCHAR and LONG VARBINARY [page 54]Use VARCHAR() and VARBINARY() to increase column storage without using large object storage mechanisms.

Large Object Storage [page 55]Use Large Object data types for data that requires more than 32K in storage.

Temporary Tables [page 56]If you want the data to persist through transaction commits, use the ON COMMIT PRESERVE ROWS option when you create global temporary tables or declare local temporary tables.

Denormalizing for Performance [page 57]Denormalizing your database can improve performance, but there are risks and disadvantages.

UNION ALL Views for Faster Loads [page 58]UNION ALL views can improve load performance when it is too expensive to maintain secondary indexes for all rows in a table.

Hash Partitioning [page 61]Hash table partitioning distributes data to logical partitions for parallel execution, which can enhance join performance on large tables and distributed queries (PlexQ).

SAP IQ Performance and Tuning Series: BasicsDesigning your Schema PUBLIC 43

6.1 Indexing

Indexing selection and solutions for SAP IQ.

Leverage NBit storage where possible and selectively add indexes where required.

In this section:

Indexing Tips [page 45]Choose the correct column index type to make your queries run faster.

Index Storage Considerations [page 46]Index location affects query performance.

Indexing Small Tables [page 46]Follow these recommendations when indexing small or infrequently updated tables.

Indexing Large Tables [page 48]For large tables, add indexes for primary keys or to accelerate point queries.

HG Index Loads [page 48]Relative to other indexes, the HG indexes are more expense to maintain during data loads and deletions. A main contributor to the performance of the HG index is the location of the data within the HG index structure: the sparsity or density of the operation.

Multi-Column Indexes [page 49]Currently, only HG, UNIQUE HG, UNIQUE CONSTRAINT, and PRIMARY KEY indexes support multiple columns in index creation, but multi-column indexes are also useful for GROUP BY and ORDER BY statements.

Parent topic: Designing your Schema [page 43]

Related Information

Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]


Designing your Schema

6.1.1 Indexing Tips

Choose the correct column index type to make your queries run faster.

SAP IQ provides some indexes automatically — an index on all columns that optimizes projections, and an HG index for UNIQUE and PRIMARY KEYS and FOREIGN KEYS. While these indexes are useful for some purposes, you may need other indexes to process certain queries as quickly as possible.

Foreign Keys

Use discretion when declaring foreign keys; indexes will be added to both tables.

INDEX_ADVISOR

INDEX_ADVISOR generates messages when the optimizer would benefit from an additional index on one or more columns in your query.

To activate the index advisor, set the INDEX_ADVISOR option ON. Messages print as part of a query plan or as a separate message in the message log (.iqmsg) if query plans are not enabled, and output is in OWNER.TABLE.COLUMN format.

HG Indexes

Consider creating an HG index on grouping columns referenced by the WHERE clause in a join query if the columns are not using NBit storage. The optimizer may need metadata from the enumerated FP or HG index to produce an optimal query plan. Non-aggregated columns referenced in the HAVING clause may also benefit from a HG index to help with query optimization. For example:

SELECT c.name, SUM(l.price * (1 - l.discount)) FROM customer c, orders o, lineitem lWHERE c.custkey = o.custkey AND o.orderkey = l.orderkey AND o.orderdate >= "1994-01-01" AND o.orderdate < "1995-01-01"GROUP by c.nameHAVING c.name NOT LIKE "I%" AND SUM(l.price * (1 - l.discount)) > 0.50 ORDER BY 2 desc

Adding indexes increases storage requirements and load time. Add indexes only if there is a net benefit to query performance.

Parent topic: Indexing [page 44]


Related Information

Index Storage Considerations [page 46]Indexing Small Tables [page 46]Indexing Large Tables [page 48]HG Index Loads [page 48]Multi-Column Indexes [page 49]INDEX_ADVISOR Option

6.1.2 Index Storage Considerations

Index location affects query performance.


Related Information

Indexing Tips [page 45]Indexing Small Tables [page 46]Indexing Large Tables [page 48]HG Index Loads [page 48]Multi-Column Indexes [page 49]

6.1.3 Indexing Small Tables

Follow these recommendations when indexing small or infrequently updated tables.

When and Where to Index

Indexes are a major tuning mechanism inside SAP IQ. Knowing when and where to use indexes can make your queries on small tables run faster.

General indexing recommendations:

• Index join columns (HG index regardless of cardinality)• Index columns frequently used with search conditions (HG index based on cardinality)• Index DATE, TIME, and DATETIME/TIMESTAMP columns (DATE, TIME, DTTM). These columns should also

have an HG index depending on data cardinality.



https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63943db84f210158981e9ab54220322.html

• If you are uncertain whether the column will be used heavily, place an HG index on the column. Workload Management can subsequently be enabled to monitor the use of indexes.

• Use PRIMARY KEY, UNIQUE CONSTRAINT, or UNIQUE HG indexes where appropriate, as they provide SAP IQ with additional information about the unique data in the indexed column(s).

• Additional indexes are not needed on columns whose data is only returned to the client (projected). Typically, that means only used in the SELECT list, and not in such places as a WHERE, ON, or GROUP BY clause.

Simple Index Selection Criteria

To determine the best indexes for your data model without regard for queries, ask yourself these simple questions about each column:

Question Action if Answer is Yes

Does the column contain DATE, TIME, DATETIME, or TIMESTAMP data?

Place a DATE, TIME, or DTTM index on this column. You should also place an HG on the column.

Will the column be used in range searches or aggregations? Place an HG index on the column. This does not apply to DATE, TIME, or DATETIME types.

Will this column be used for word searching? Place a WD index on the column. An HG index is not necessary and would consume significant space.

Will this column be used for full text searching? Place a TEXT index on the column. An HG is not necessary and would consume significant space.

Will two columns in the same table be compared to each other (A = B, A < B, A > B, A <= B, A >= B)?

Place a CMP index on the two columns.

Will this column, or set of columns, be used in GROUP BY or ORDER BY statements?

Place an HG index on the column, or columns in the GROUP BY or ORDER BY statement. Each column should also have a corresponding HG index.

Is this column part of a multicolumn primary key, constraint, or index?

Place an HG index on each column in the multicolumn index.


Related Information

Indexing Tips [page 45]Index Storage Considerations [page 46]Indexing Large Tables [page 48]HG Index Loads [page 48]Multi-Column Indexes [page 49]


6.1.4 Indexing Large Tables

For large tables, add indexes for primary keys or to accelerate point queries.

Point queries include equality, narrow range, or small IN list queries.


Related Information

Indexing Tips [page 45]Index Storage Considerations [page 46]Indexing Small Tables [page 46]HG Index Loads [page 48]Multi-Column Indexes [page 49]

6.1.5 HG Index Loads

Relative to other indexes, the HG indexes are more expense to maintain during data loads and deletions. A main contributor to the performance of the HG index is the location of the data within the HG index structure: the sparsity or density of the operation.

Dense HG operations are those in which the affected rows are tightly grouped around certain keys. Sparse operations are those where there may be just a few rows per key that must be affected. For instance, dates on data are typically grouped around the time the operation was logged, data modified, etc. This means that new data will be placed at the end of the HG index structure. When deleting data in the date HG index, said data would typically come off in chunks of days, weeks, months, etc. and thus be removed from the beginning of the HG btree or be tightly grouped around a few keys for deletion. These operations are very fast, relatively speaking, as SAP IQ will operate on a few pages and affect a tremendous number of rows.

Data that is rather sparse, like Prices, Customer IDs, City, Country, etc., are very different. As “pricing" data, for instance, is loaded each value will vary widely across all data already in the index. If the column is tracking stock prices the numeric field to store that data will be densely updated because the data being changed will be across the nearly the entire range of values already loaded. These operations are slower due to the amount of index pages that must be maintained for each row being affected. A worst case scenario is that SAP IQ is forced to read and write 1 page for EACH ROW being loaded or deleted. While this can be less than optimal, SAP IQ has been design to parallel process phase 2 of the HG index loads and the deletes, so that the impact is greatly reduced.

All of this is well and good, but how does it affect the data model design and indexing? Typical tuning and optimization within SAP IQ generally boils down to indexes or the lack thereof. Knowing how the indexes can be affected by the data and loading is an important aspect when deciding which indexes to put in place and which to leave off. Because HG indexes take, relatively, more time to load than other indexes they are often the subject of focus when it comes to use and design. Certainly, HG indexes can help with query performance. There are times, though, where adding an index may have a slight positive impact on queries but have more of an impact



to data loads. In these situations, it is important to understand why the load or delete took longer and what can be done about it.

The sparsity or density of new data with respect to currently loaded data plays a critical role in this. If a relatively random column of a Customer ID must be indexed for fast query performance and an index must be on that column. Suppose, though, that a primary key exists on the table and it is the Customer ID and a Date field storing a transaction datetime. If the ordering were left as (customer_id, transaction_date) the data would be sparsely loaded or deleted from the table in most case. Data being loaded will be done so by transaction date. Since the Customer ID column is first in the multicolumn index, though, it will force SAP IQ to touch data throughout the entire HG index structure.

A simple change in order to (transaction_date, customer_id) changes this behavior. The index is still in place to control referential integrity for the primary key. The ordering of the columns is immaterial for primary key enforcement. As such, we can change the column order without causing any downstream ill effects. This simple change will now force all new data being loaded by transaction date to be inserted at the end of the HG index structure in a very dense manner. Over time the loads will perform consistently as the data is, generally, always going to the end of the HG structure.

Simply changing the column ordering in a multicolumn index can have drastic impacts on performance. The size of the HG index shouldn't change much as the data is still the same width regardless of order. What will change is how fast the data is loaded or deleted from the table.


Related Information

Indexing Tips [page 45]Index Storage Considerations [page 46]Indexing Small Tables [page 46]Indexing Large Tables [page 48]Multi-Column Indexes [page 49]

6.1.6 Multi-Column Indexes

Currently, only HG, UNIQUE HG, UNIQUE CONSTRAINT, and PRIMARY KEY indexes support multiple columns in index creation, but multi-column indexes are also useful for GROUP BY and ORDER BY statements.

From a statistics point of view, multi-column indexes provide enough information in multi-column table joins to let the optimizer know the exact statistics of the join and whether or not it is a many-to-many or one-to-many join. The optimizer is also smart enough to use the statistics for optimization, but use individual HG indexes for the actual work. The optimizer costs out all join and sort scenarios and decides which index(es) is best for that operation. The statistics help it get to that point.

Some items to keep in mind about the HG indexes:

• HG inserts are the most expensive


• Try to guarantee that inserts will happen at the end of the index

Place generally incrementing data, like a transaction date or batch number (sequential data), at the beginning of the index list. Something that will try to guarantee a sequential key.


Related Information

Indexing Tips [page 45]Index Storage Considerations [page 46]Indexing Small Tables [page 46]Indexing Large Tables [page 48]HG Index Loads [page 48]

6.2 Join Column

For joins, keep the data types as narrow as possible to reduce disk I/O and memory requirements.

Because integer comparisons are quicker than character comparisons, use integer data types (unsigned if possible) in joins. Keeping the data types as narrow as possible improves join performance by reducing disk I/O and memory requirements. Because the HG index has more capability from a join perspective, use an HG index on join columns.


Related Information

Indexing [page 44]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]



6.3 Primary Keys

Multi-column primary keys should have an additional HG index placed on each column specified in the primary key. This must be done manually as SAP IQ only creates an HG index on the composite columns.

UNIQUE constraint, UNIQUE HG, and primary key share an identical structure. That structure uses an HG index with no G-Array to store the row ids. When possible, use primary keys on tables. This helps the optimizer make more informed query path decisions even if the index is not used. The index structure provides detailed statistics to help the optimizer make better choices as well as providing a structure to traverse the data.


Related Information

Indexing [page 44]Join Column [page 50]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.4 Foreign Keys

As with primary keys, use foreign keys to improve query join performance. This gives SAP IQ one more piece of information on how tables are joined and the statistics behind those joins.

SAP IQ automatically creates an HG Index on the foreign key column, so no additional HG index is necessary. A foreign key requires that a primary key exists on referenced table.



Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.5 Proper Data Type Sizing

Size all data types as accurately as possible, especially character-based data types.

To decide which data type to use for a column, consider these factors:

• SAP IQ includes a large number of data types. Using the correct data types for your application leads to optimal performance gains.

• If HOUR, MINUTE, and SECOND information are not necessary, use DATE instead of DATETIME• If the data will fit within a TINYINT or SMALLINT datatype use that rather than INTEGER or BIGINT• Do not over allocate storage when defining NUMERIC() or DECIMAL() as it can be costly for data that does

not need all that level of precision• CHAR() and VARCHAR() types are fixed width in the default Flat FP index. The only difference is the

addition of 1 byte to each VARCHAR() row that represents the number of bytes in use.

SAP IQ includes compression algorithms that compress large repeating patterns often seen in BINARY(), CHAR(), VARCHAR(), and VARBINARY() data types.


Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]



Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.6 Null Values

Defining columns as NULL or NOT NULL helps the optimizer work more efficiently.

Specifying NULL or NOT NULL allows the optimizer a more educated guess at joins and search criteria by having one more piece of information about the characteristics of the data. NULL data does not save space on the database page, as it would in other databases. NULL data will, however, be compressed out when stored on disk due to the SAP IQ compression algorithms and optimized indexes.

• Always specify NULL or NOT NULL• Open Client and ODBC connections have different default behavior when table is created• Give the optimizer an additional piece of information about the characteristics of the data for joins and

search arguments


Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]


6.7 Unsigned Data Types

In some cases, using unsigned data types can eliminate sign comparisons and create faster queries.

Use unsigned data types when the sign of the data does not matter as all data will always be greater than or equal to zero. The lack of sign storage results in column comparisons that no longer have to perform sign comparison. This increases performance and eliminates a step in the joining and searching of data, particularly for key columns.


Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.8 LONG VARCHAR and LONG VARBINARY

Use VARCHAR() and VARBINARY() to increase column storage without using large object storage mechanisms.

Typically, developers and DBAs think of VARBINARY() and VARCHAR() data as being limited to 255 bytes. SAP IQ supports VARCHAR() and VARBINARY() widths of up to 32 K (also known as LONG VARCHAR or LONG VARBINARY). This allows for much larger storage of text or binary data without needing to move into the highly specialized large objects storage mechanism of BLOB/CLOB or IMAGE/TEXT data types.

• Can be used to store moderate amounts of text or binary data• Maximum width is 32 K (64K ASCII hex for VARBINARY())• The WORD and TEXT index is the only index allowed on VARCHAR() data wider than 255 bytes• Storage will be allocated in 256 byte chunks



• A 257-byte string will require 512 bytes of storage• A 511-byte string will also require 512 bytes of storage


Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.9 Large Object Storage

Use Large Object data types for data that requires more than 32K in storage.

• Large object data types store ASCII (TEXT/CLOB) and binary (IMAGE/BLOB) data. Each BLOB/CLOB cell of data is stored on one or more pages:• Assuming the page size is 128K• If the data is 129K, it will require 2 pages to store the information• If the data is 1K, it will require 1 page to store the data• In either case, the page(s) are compressed on disk into multiples of the block size

• Can be used to store binary or text based objects• Extends the long binary data type from a maximum size of 6K to an unlimited size• Supported by index types WD and TEXT• Can be fully searched with the TEXT index and its search capabilities• Special function to return the size of an object (byte_length64)• Special function to return portions of the object, not the entire contents (byte_substr64)• Can extract contents of a binary object cell to an individual file with the BFILE() function



Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.10 Temporary Tables

If you want the data to persist through transaction commits, use the ON COMMIT PRESERVE ROWS option when you create global temporary tables or declare local temporary tables.

There are three types of Temporary Tables:

• # tables

CREATE TABLE #temp_table( col1 int )

• Local Temporary Tables

DECLARE LOCAL TEMPORARY TABLE temp_table ( col1 int )

Local Temporary Tables behave just like # tables• Global Temporary Tables

CREATE GLOBAL TEMPORARY TABLE temp_table ( col1 int )

Global Temporary Table structure is static across connections and reboots

Normal hash (#) tables do not need the ON COMMIT PRESERVE ROWS option because the data in a hash table will always persist through transaction commits.


Related Information

Indexing [page 44]



Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.11 Denormalizing for Performance

Denormalizing your database can improve performance, but there are risks and disadvantages.

Denormalization can be successfully performed only with thorough knowledge of the application and should be performed only if performance issues indicate that it is needed. Consider the effort required to keep your data up-to-date.

This is a good example of the differences between decision support applications, which frequently need summaries of large amounts of data, and transaction processing needs, which perform discrete data modifications. Denormalization usually favors some processing, at a cost to others.

Denormalization has the potential for data integrity problems, which must be carefully documented and addressed in application design.

Deciding to Denormalize

Analyze the data access requirements of the applications in your environment and their actual performance characteristics, including:

• What are the critical queries, and what is the expected response time?• What tables or columns do they use? How many rows per access?• What is the usual sort order?• What are concurrency expectations?• How big are the most frequently accessed tables?• Do any processes compute summaries?



Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]UNION ALL Views for Faster Loads [page 58]Hash Partitioning [page 61]

6.12 UNION ALL Views for Faster Loads

UNION ALL views can improve load performance when it is too expensive to maintain secondary indexes for all rows in a table.

SAP IQ lets you split the data into several separate base tables (for example, by date). You load data into these smaller tables. You then join the tables back together into a logical whole by means of a UNION ALL view, which you can then query.

This strategy can improve load performance, but may negatively impact the performance of some types of queries. Most types of queries have roughly similar performance against a single base table or against a UNION ALL view over smaller base tables, as long as the view definition satisfies all constraints. However, some types of queries, especially those involving DISTINCT or involving joins with multiple join columns, may perform significantly slower against a UNION ALL view than against a single large base table. Before choosing to use this strategy, determine whether the improvements in load performance are worth the degradation in query performance for your application.

To create a UNION ALL view, choose a logical means of dividing a base table into separate physical tables. The most common division is by month. For example, to create a view including all months for the first quarter, enter:

CREATE VIEW SELECT * JANUARYUNION ALLSELECT * FEBRUARYUNION ALLSELECT * MARCH UNION ALL

Each month, you can load data into a single base table—JANUARY, FEBRUARY, or MARCH in this example. Next month, load data into a new table with the same columns, and the same index types.



NoteYou cannot perform an INSERT...SELECT into a UNION ALL view.

In this section:

Queries Referencing UNION ALL Views [page 59]SAP IQ includes optimizations for UNION ALL views.

UNION ALL View Performance [page 60]Structure queries to evaluate the DISTINCT operator before the ORDER BY, where the sort order is ASC.


Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]Hash Partitioning [page 61]

6.12.1 Queries Referencing UNION ALL Views

SAP IQ includes optimizations for UNION ALL views.

Optimizations include:

• Split GROUP BY over UNION ALL view• Push-down join into UNION ALL view

A UNION can be treated as a partitioned table only if it satisfies all of the following constraints:

• It contains only one or more UNION ALL.• Each arm of the UNION has only one table in its FROM clause, and that table is a physical base table.


• No arm of the UNION has a DISTINCT, a RANK, an aggregate function, or a GROUP BY clause.• Each item in the SELECT clause within each arm of the UNION is a column.• The sequence of data types for the columns in the SELECT list of the first UNION arm is identical to the

sequence in each subsequent arm of the UNION.

Parent topic: UNION ALL Views for Faster Loads [page 58]

Related Information

UNION ALL View Performance [page 60]

6.12.2 UNION ALL View Performance

Structure queries to evaluate the DISTINCT operator before the ORDER BY, where the sort order is ASC.

Certain optimizations, such as pushing a DISTINCT operator into a UNION ALL view, are not applied when the ORDER BY is DESC because the optimization that evaluates DISTINCT below a UNION does not apply to DESC order. For example, the following query would impact performance:

SELECT DISTINCT state FROM testVU ORDER BY state DESC;

To work around this performance issue, queries should have the DISTINCT operator evaluated before the ORDER BY, where the sort order is ASC and the optimization can be applied:

SELECT c.state FROM (SELECT DISTINCT state FROM testVUA) c ORDER BY c.state DESC;

Parent topic: UNION ALL Views for Faster Loads [page 58]

Related Information

Queries Referencing UNION ALL Views [page 59]



6.13 Hash Partitioning

Hash table partitioning distributes data to logical partitions for parallel execution, which can enhance join performance on large tables and distributed queries (PlexQ).

New join algorithms and aggregation algorithms can take advantage of hash partitioning by reducing the amount of intermediate storage and network transfer required as well as providing increased parallelism by leveraging the semantic division of the table rows into partitions. The improved cache behavior provided by data affinity and affinity based work allocation provide further scalability in the PlexQ environment.

Using Hash Partitioned Join or Hash Partitioned Aggregation Algorithms

To use the hash partitioned join or hash partitioned aggregation algorithms, it is critical that all the columns of the hash partitioning key for the table or tables, be used in the equi-join conditions or grouping expressions for those tables. Additional join conditions or grouping expressions may be used, but if a column that is a component of the hash partitioning key is not used in the query, the partitioned algorithms will not be eligible. In such a case, other non-partitioned algorithms remain eligible and would be chosen as for non-partitioned tables. Hash partitioning of a set of tables should cover the smallest common set of columns used by the application queries on those tables. Frequently a single column is a sufficient partitioning basis. It is essential that the data types of columns in joins between partitioned tables be identical.

Small tables benefit less from hash partitioning than large tables. Consider hash partitioning for tables of at least 1 billion rows.

Improving Equality Condition Performance

SAP IQ leverages information available in a hash-partitioned or a hash-range-partitioned table to improve the performance of some equality conditions on the hash-partitioning basis column.

The optimizer is applied when all of the following are true:

1. The SQL statement contains a condition of the form:

<column> = <constant-expression>

The expression cannot apply to <>, <, <=, >, >=, IN, NOT IN, or BETWEEN conditions.2. That <column> is a base column in a real table, not from a view or a derived table.3. The base column is the sole basis for the hash partitioning of the table containing that <column>.4. That <constant-expression> is either a literal constant, a host variable, or a simple scalar expression

containing only constants and host variables, not a scalar subquery or an aggregate expression.5. The equality condition is in the SQL statement at a location, which allows it to be applied at the leaf node in

the query plan, not in some other query block, and not in an ON clause of FULL OUTER JOIN in the same query block.

6. The data type of the <constant-expression> is highly compatible with the data type of the base column.


7. There is no HG index on that base column. When all these conditions are true, then the optimizer will infer an additional condition of the form, where that integer-constant is the hash partition number for the <constant-expression> in the original condition:

HashPartitionNumber( <column> ) = integer-constant

This condition will appear in the query plan like any other inferred condition.

By executing this inferred condition before the original equality condition that original equality condition only needs to scan the FP cell values for rows contained within that one hash partition.

Large Memory

Some load operations may require more large memory than the 2GB default provides. If the memory requirements exceed the default, use the -iqlm startup option to increase the memory that SAP IQ can dynamically request from the OS.

As a general rule, large memory requirements represent one third of the total available physical memory allocated to SAP IQ. To ensure adequate memory for the main and temporary IQ stores, set the -iqlm, -iqtc, and -iqmc startup parameters so that each parameter receives one third of all available physical memory allocated to SAP IQ.

In most cases, you should allocate 80% of total physical memory to SAP IQ to prevent SAP IQ processes from being swapped out. Adjust actual memory allocation to accommodate other processes running on the same system. For example, on a machine with 32 cores and 128 GB of total available physical memory, you would allocate 100 GB (approximately 80% of the 128 GB total) to SAP IQ processes. Following the general rule, you would set the -iqlm, -iqtc, and -iqmc parameters to 33 GB each.


Related Information

Indexing [page 44]Join Column [page 50]Primary Keys [page 51]Foreign Keys [page 51]Proper Data Type Sizing [page 52]Null Values [page 53]Unsigned Data Types [page 54]LONG VARCHAR and LONG VARBINARY [page 54]Large Object Storage [page 55]Temporary Tables [page 56]Denormalizing for Performance [page 57]UNION ALL Views for Faster Loads [page 58]JOIN_PREFERENCE Option



https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63bb2e884f210158490d70844b57a75.html

AGGREGATION_PREFERENCE Option-iqlm Database Server Option-iqmc Database Server Option-iqtc Database Server Option


https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a62bf97e84f21015843682381c2172d5.html

https://help.sap.com/viewer/a893062984f21015b9e8b03f96ed0cbb/16.1.5.0/en-US/a873465e84f21015a242ee479167f51d.html

https://help.sap.com/viewer/a893062984f21015b9e8b03f96ed0cbb/16.1.5.0/en-US/a2f7d19584f210159dc2bf226e3b718e.html

https://help.sap.com/viewer/a893062984f21015b9e8b03f96ed0cbb/16.1.5.0/en-US/a2fe164584f21015b1e8d41ed50f1af2.html

7 Troubleshooting

Identify and solve common performance issues.

In this section:

Monitoring SAP IQ Statement Performance [page 64]The statement performance monitoring feature is not an exhaustive, complete audit of slow SQL statements (queries), but it is a useful tool for providing an approximation, or high-level summary, of query workload. Statement performance monitoring flags certain outlier statements with execution times exceeding an established baseline.

Isolating Performance Problems [page 66]Determine whether the problem is external or internal to SAP IQ.

Diagnostic Tools [page 67]Utilities on your operating system that monitor system activities.

Common Performance Issues [page 68]Solutions to common performance problems.

7.1 Monitoring SAP IQ Statement Performance

The statement performance monitoring feature is not an exhaustive, complete audit of slow SQL statements (queries), but it is a useful tool for providing an approximation, or high-level summary, of query workload. Statement performance monitoring flags certain outlier statements with execution times exceeding an established baseline.

Context

Statement performance metrics can help you narrow-down the cause of performance problems. Use statement performance monitoring to answer questions like:

• In general, my queries appear to be taking longer – can I view some performance data to support this assumption?

• What are some frequently-run statements that are significantly slower today than they were previously?• What is the standard deviation, in seconds, for one of these slow outlier statements compared to its

average execution time?

Statement performance monitoring is expensive, meaning exhaustive reporting can affect performance. To minimize performance degradation, XML plan and SQL query text information is only recorded once the algorithm determines the query is expensive enough to warrant recording. There may be a processing lag between the time the algorithm selects a query, and the time its XML plan and SQL query text information appears.


Troubleshooting

In addition, statement performance monitoring only maintains information for up to 10,000 statements. If workload is high, this buffer may be full and some slow statements may not be monitored – even if they are slow statements of interest.

The QUERY_PLAN_MIN_TIME threshold helps the algorithm decide whether or not to collect query plan information for a statement, but it is not a monitoring guarantee. Not all statements whose execution times exceed the QUERY_PLAN_MIN_TIME threshold are considered outlier statements by the algorithm. Statement performance monitoring is not a means of auditing all queries with execution times exceeding the QUERY_PLAN_MIN_TIME threshold.

Diagnostic data is not saved to disk. When the SAP IQ server shuts down, all diagnostic data is removed. Diagnostic collection starts afresh with each SAP IQ server start-up.

Procedure

1. Set option COLLECT_IQ_PERFORMANCE_STATS to ON.2. Set option QUERY_PLAN_MIN_TIME in milliseconds. This influences the statement performance

algorithm, helping determine which SQL statements with long execution times get their query plans recorded.

3. Use the sp_top_k_statements and sp_find_top_statements system procedures to report some of the statement and plan combinations that take the longest time to run:

• sp_top_k_statements provides timing metrics including the standard deviation (stddev_seconds) from average statement execution, and the maximum runtime (max_seconds) for certain statements.

• sp_find_top_statements provides resource metrics, including CPU usage (max_cpu_usage_perc), temp space usage (temp_space_used_mb), number of times the statement was executed by the current plan (num_exec), and number of threads used (max_thread_count).

4. Analyze statement performance data in the following system views:

• GTSYSPERFCACHESTMT system view – provides the SQL for a set of slow statements chosen by the algorithm. Note that this view removes all constants from the SQL.

• GTSYSPERFCACHEPLAN system view – provides the XML plan for a set of slow statements chosen by the algorithm. Note that this plan is in XML format, not the usual HTML query plan format.

NoteNot all statements recorded in GTSYSPERFCACHESTMT system view show a corresponding plan in GTSYSPERFCACHEPLAN system view. Writing to GTSYSPERFCACHEPLAN is more expensive than writing to GTSYSPERFCACHESTMT, and therefore the algorithm may decide to record the SQL, but not the XML plan. Moreover, some statements will have multiple plan entries recorded in GTSYSPERFCACHEPLAN.

5. If a statement's standard deviation (standdev_seconds) in sp_top_k_statements indicates a slow statement, consider further investigation by turning on query plan generation. See Generating a Query Plan in the SAP IQ Interactive SQL Guide.

Task overview: Troubleshooting [page 64]

SAP IQ Performance and Tuning Series: BasicsTroubleshooting PUBLIC 65

Related Information

Isolating Performance Problems [page 66]Diagnostic Tools [page 67]Common Performance Issues [page 68]COLLECT_IQ_PERFORMANCE_STATS OptionQUERY_PLAN_MIN_TIME Optionsp_find_top_statements System Proceduresp_top_k_statements System ProcedureGTSYSPERFCACHEPLAN System ViewGTSYSPERFCACHESTMT System ViewGenerating a Query Plan

7.2 Isolating Performance Problems

Determine whether the problem is external or internal to SAP IQ.

External to SAP IQ

• Monitor the OS, hardware, and storage for any bottlenecks or issues• Look for high CPU use, high CPU system time, low CPU user time, high wait time• Look for I/O service times that are more than 10ms

Internal to SAP IQ

• Enable INDEX_ADVISOR and look for missing indexes• Enable QUERY_PLAN_AS_HTML and the INDEX_ADVISOR if the issue is with a single query

Parent topic: Troubleshooting [page 64]

Related Information

Monitoring SAP IQ Statement Performance [page 64]Diagnostic Tools [page 67]Common Performance Issues [page 68]


Troubleshooting

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a920c77d22574357b8426dd3b69ff32f.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a31267e384f210158cfeeef7b6d1a826.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/c95656706e1044c095af69e8a407420a.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/6a511317920b4552a61efffd7f50540b.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/6df8e7a30fda4bca914f4c1f9b5b574e.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/7c163a0aa89641c59d30b07ffecd9bd3.html

https://help.sap.com/viewer/a895964984f210158925ce02750eb580/16.1.5.0/en-US/a47d6a4484f21015a348e30a0fe4cb18.html

7.3 Diagnostic Tools

Utilities on your operating system that monitor system activities.

Use these utilities to get statistics on the number of running processes, including and the number of page-outs and swaps. Use this information to find out if the system is paging excessively, then make any necessary adjustments. You may want to put your swap files on special fast disks.

OS Utility Description

UNIX top, topas Provides an ongoing look at processor activity in real time.

top is available on Solaris, Linux, and HP-UX platforms. topas is available on AIX.

ps Reports process status.

vmstat Displays information about system processes, memory, paging, block IQ, traps, and CPU activity.

iostat -x Displays disk subsystem information.

sar Writes selected OS activity results to standard output.

Windows Task Manager, Resource Monitor Provide detailed information about computer performance and running applications, processes, CPU usage, and other system services.

NoteTo access System Monitor, select the object Logical Disk, the instance of the disk containing the file PAGEFILE.SYS, and the counter Disk Transfers/Sec.

Put the Windows page files on different disks than your database dbspace devices. You can also monitor the Object Memory and the counter Pages/Sec. However, this value is the sum of all memory faults, which includes both soft and hard faults.


Related Information

Monitoring SAP IQ Statement Performance [page 64]Isolating Performance Problems [page 66]Common Performance Issues [page 68]


7.4 Common Performance Issues

Solutions to common performance problems.

In this section:

Paging and Disk Swapping [page 68]Good memory management avoids page swapping. To minimize use of operating system files, increase or reallocate physical memory.

Index and Row Fragmentation [page 69]Internal index fragmentation occurs when index pages are not being used to their maximum volume. Row fragmentation occurs when rows are deleted. Deleting an entire page of rows frees the page, but if some rows on a page are unused, the unused space remains on the disk. DML operations (INSERT, UPDATE, DELETE) on tables can cause index fragmentation.

Catalog File Growth [page 70]Growth of the catalog files is normal and varies depending on the application and catalog content. The size of the .db file does not affect performance, and free pages within the .db file are reused as needed.

Thrashing and Query Execution [page 70]Adjust the HASH_THRASHING_PERCENT option to limit thrashing during queries that involve hash algorithms.

Views and Tables Using Wide Tables [page 71]CREATE TABLE statements for wide tables and CREATE VIEW statements containing wide tables may run more slowly than in prior releases.


Related Information

Monitoring SAP IQ Statement Performance [page 64]Isolating Performance Problems [page 66]Diagnostic Tools [page 67]

7.4.1 Paging and Disk Swapping

Good memory management avoids page swapping. To minimize use of operating system files, increase or reallocate physical memory.

Insufficient memory severely degrades performance. If this is the case, you need to find a way to make more memory available. The more memory you can allocate to SAP IQ, the better.

Because there is always a fixed limit to the amount of memory, the operating system may sometimes keep part of the data in memory and the rest on disk. Paging or swapping occurs when the operating system must go out


Troubleshooting

to disk and retrieve any data before a memory request can be satisfied. Good memory management avoids or minimizes paging or swapping.

The most frequently used operating system files are swap files. When memory is exhausted, the operating system swaps pages of memory to disk to make room for new data. When the pages that were swapped are called again, other pages are swapped, and the required memory pages are brought back. This is very time-consuming for users with high disk usage rates. Try to organize memory to avoid swapping and, thus, to minimize use of operating system files.

To make the maximum use of your physical memory, SAP IQ uses buffer caches for all reads and writes to your databases.

NoteYour swap space on disk must be at least large enough to accommodate all of your physical memory. Having swap/paging space striped across fast disks is essential.

Parent topic: Common Performance Issues [page 68]

Related Information

Index and Row Fragmentation [page 69]Catalog File Growth [page 70]Thrashing and Query Execution [page 70]Views and Tables Using Wide Tables [page 71]

7.4.2 Index and Row Fragmentation

Internal index fragmentation occurs when index pages are not being used to their maximum volume. Row fragmentation occurs when rows are deleted. Deleting an entire page of rows frees the page, but if some rows on a page are unused, the unused space remains on the disk. DML operations (INSERT, UPDATE, DELETE) on tables can cause index fragmentation.

Run these stored procedures for information about fragmentation issues:

• sp_iqrowdensity reports row fragmentation at the FP index level.• sp_iqindexfragmentation reports internal fragmentation within supplemental indexes.

Review the output and decide whether you want to recreate, reorganize, or rebuild the indexes. You can create other indexes to supplement the FP index.



Related Information

Paging and Disk Swapping [page 68]Catalog File Growth [page 70]Thrashing and Query Execution [page 70]Views and Tables Using Wide Tables [page 71]sp_iqrowdensity Proceduresp_iqindexfragmentation Procedure

7.4.3 Catalog File Growth

Growth of the catalog files is normal and varies depending on the application and catalog content. The size of the .db file does not affect performance, and free pages within the .db file are reused as needed.

To minimize catalog file growth:

• Avoid using IN SYSTEM on CREATE TABLE statements• Issue COMMIT statements after running system stored procedures• Issue COMMIT statements during long-running transactions


Related Information

Paging and Disk Swapping [page 68]Index and Row Fragmentation [page 69]Thrashing and Query Execution [page 70]Views and Tables Using Wide Tables [page 71]

7.4.4 Thrashing and Query Execution

Adjust the HASH_THRASHING_PERCENT option to limit thrashing during queries that involve hash algorithms.

Adjusting the HASH_THRASHING_PERCENT database option controls the percentage of hard disk I/Os allowed before the statement is rolled back and an error is returned. The default value of HASH_THRASHING_PERCENT is 10%. Increasing HASH_THRASHING_PERCENT permits more paging to disk before a rollback and decreasing HASH_THRASHING_PERCENT permits less paging before a rollback.

Queries involving hash algorithms that executed in earlier versions of SAP IQ may now be rolled back when the default HASH_THRASHING_PERCENT limit is reached. SAP IQ reports one of the following error messages:

Hash insert thrashing detected


Troubleshooting

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a5b5cb9b84f2101585b8e3b5e25893af.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a5ac10a084f210158b30b3fa5c350b40.html

Hash find thrashing detectedTake one or more of the following actions to provide the query with the resources required for execution:

• Relax the paging restriction by increasing the value of HASH_THRASHING_PERCENT.• Increase the size of the temporary cache. Keep in mind that increasing the size of the temporary cache

requires an equal size reduction in main buffer cache allocation to prevent the possibility of system thrashing.

• Attempt to identify and alleviate why SAP IQ is not estimating one or more hash sizes for this statement correctly. For example, check that all columns that need an HG index have one. Also consider if a multicolumn index is appropriate.

• Decrease the value of the database option HASH_PINNABLE_CACHE_PERCENT.

To identify possible problems with a query, generate a query plan by running the query with the temporary database options QUERY_PLAN='ON' and QUERY_DETAIL ='ON'. Examine the estimates in the query plan.


Related Information

Paging and Disk Swapping [page 68]Index and Row Fragmentation [page 69]Catalog File Growth [page 70]Views and Tables Using Wide Tables [page 71]

7.4.5 Views and Tables Using Wide Tables

CREATE TABLE statements for wide tables and CREATE VIEW statements containing wide tables may run more slowly than in prior releases.

If you experience slow performance for CREATE TABLE or CREATE VIEW statements involving wide tables, reset the startup command line options -ch (max_cache_size) and -c (initial cache size). Set the maximum cache size at least 10% larger than the initial cache size.


Related Information

Paging and Disk Swapping [page 68]Index and Row Fragmentation [page 69]Catalog File Growth [page 70]Thrashing and Query Execution [page 70]-c Database Server Option


https://help.sap.com/viewer/a893062984f21015b9e8b03f96ed0cbb/16.1.5.0/en-US/a2f3b4ac84f21015a676b6345f403535.html

-ch Database Server Option


Troubleshooting

https://help.sap.com/viewer/a893062984f21015b9e8b03f96ed0cbb/16.1.5.0/en-US/a2f43a5a84f210159d52b361942719a6.html

8 Structuring Queries

Recommendations to help you plan, structure, and control your queries, and information on optimizing delete operations.

In this section:

Structuring Queries [page 73]Improving query structures can make your queries run faster.

Distributed Query Performance [page 76]In general, the more nodes and resources that are available, the better the potential query performance.

Generating Query Plans [page 80]Generating a query plan can help you understand the execution plan developed by the optimizer.

Controlling Query Processing [page 87]Any user can set limits on the amount of time spent processing a particular query. Users with the SET ANY PUBLIC OPTION system privilege can give certain user's queries priority over others, or change processing algorithms to influence the speed of query processing.

Optimizing Delete Operations [page 91]SAP IQ chooses the best algorithm to process delete operations on columns with HG and WD indexes.

8.1 Structuring Queries

Improving query structures can make your queries run faster.

• In some cases, command statements that include subqueries can also be formulated as joins and may run faster.

• If you group on multiple columns in a GROUP BY clause, list the columns by descending order by number of unique values if you can. This will give you the best query performance.

• You can improve performance by using an additional column to store frequently calculated results.

In this section:

Enhancing ORDER BY Query Performance [page 74]Using multicolumn HG indexes can enhance the performance of ORDER BY queries.

Improved Subquery Performance [page 74]Use SUBQUERY_FLATTENING_PREFERENCE and SUBQUERY_FLATTENING_PERCENT to control subquery flattening.

Using Caching Methods [page 75]Set the SUBQUERY_CACHING_PREFERENCE option to choose caching methods for a correlated subquery.

SAP IQ Performance and Tuning Series: BasicsStructuring Queries PUBLIC 73

Zone Maps [page 75]A zone map can improve performance when executing queries.

8.1.1 Enhancing ORDER BY Query Performance

Using multicolumn HG indexes can enhance the performance of ORDER BY queries.

You can use multicolumn HG indexes to enhance the performance of ORDER BY queries with reference to multiple columns in a single table query. This change is transparent to users, but improves query performance.

Queries with multiple columns in the ORDER BY clause may run faster using multicolumn HG indexes. For example, if the user has multicolumn index HG(x,y,z) on table T, then this index is used for ordered projection:

SELECT abs (x) FROM T ORDER BY x, y

In the above example, the HG index vertically projects <x> and <y> in sorted order.

If the ROWID() function is in the SELECT list expressions, multicolumn HG indexes are also used. For example:

SELECT rowid()+x, z FROM T ORDER BY x,y,z

If ROWID() is present at the end of an ORDER BY list, and if the columns of that list—except for ROWID()— exist within the index, and the ordering keys match the leading HG columns in order, multicolumn indexes are used for the query. For example:

SELECT z,y FROM T ORDER BY x,y,z,ROWID()

Parent topic: Structuring Queries [page 73]

Related Information

Improved Subquery Performance [page 74]Using Caching Methods [page 75]Zone Maps [page 75]

8.1.2 Improved Subquery Performance

Use SUBQUERY_FLATTENING_PREFERENCE and SUBQUERY_FLATTENING_PERCENT to control subquery flattening.

Subquery flattening is an optimization technique in which the optimizer rewrites a query containing a subquery into a query that uses a join. SAP IQ flattens many but not all subqueries. Use


Structuring Queries

SUBQUERY_FLATTENING_PREFERENCE and SUBQUERY_FLATTENING_PERCENT to control when the optimizer chooses to use this optimization.


Related Information

Enhancing ORDER BY Query Performance [page 74]Using Caching Methods [page 75]Zone Maps [page 75]

8.1.3 Using Caching Methods

Set the SUBQUERY_CACHING_PREFERENCE option to choose caching methods for a correlated subquery.

A correlated subquery contains references to one or more tables outside of the subquery and is re-executed each time the value in the referenced column changes. Use the SUBQUERY_CACHING_PREFERENCE option to choose caching methods for executing the correlated subquery.


Related Information

Enhancing ORDER BY Query Performance [page 74]Improved Subquery Performance [page 74]Zone Maps [page 75]

8.1.4 Zone Maps

A zone map can improve performance when executing queries.

Also known as a storage index, data skipping, or constraint exclusion, it optimizes scan processing by ignoring complete pages of values if the server can guarantee that there is no value of interest to the predicate(s) being executed on that page. A zone map is a data structure containing the min/max values of each page of column data, and is stored as part of the FP index.

A zone map is most effective if no other index (besides an FP index) exists on a column or the values in a column are arranged in a sequential order (for example, by transaction date). A zone map is less effective for tables with few rows, or for columns that have a uniform distribution of values across all pages.


When executing a query, the optimizer checks for existing indexes. If multiple indexes are found, it evaluates and uses whichever index is optimal for the query, which may or may not be a zone map. Zone maps require the values in the predicate(s) be fixed; not derived from a function, and supports four classes of predicates: equality (=), inequality (<, >, !=), between, and not between.

The zone map feature requires SAP IQ 16.1 SP 03 or later. A zone map is automatically created on a column when you add a column to a new or existing table. For databases created prior to SP 03, after upgrading to SAP IQ 16.1 SP 03 or later, to add a zone map to an existing column, rebuild its FP index. Adding a zone map on a column with multiple indexes does not impede performance.

An FP index with a RidMap version of zero means the FP index has no zone map. Run sp_iqrebuildindex to rebuild the FP index on the column with a zone map.

To determine the version of the FP index on a column, execute sp_iqzonemapenable.

The zone map feature is enabled by default, but can be disabled by setting the DML_OPTIONS90 option to 1.


Related Information

Enhancing ORDER BY Query Performance [page 74]Improved Subquery Performance [page 74]Using Caching Methods [page 75]sp_iqrebuildindex Proceduresp_iqzonemapenable ProcedureSetting Query Optimization Options [page 89]DML_OPTIONS90 Option

8.2 Distributed Query Performance

In general, the more nodes and resources that are available, the better the potential query performance.

Distributed query processing uses the available memory and CPU resources of all nodes of the logical server.

The amount of improvement benefit depends on the type of query, the size of the query, and the current workload of the nodes in the logical server.

NoteIf you change the properties of multiplex server, including the server name, hostname, and port, then you must wait at least two minutes after restarting the multiplex server for it to participate in a DQP eligible query. In the first two minutes after restarting the server, if a DQP eligible query is executed, then the server may not participate.

It is unlikely that any two runs of the same query result in the same work distribution — as load levels change in the cluster, so does the load distribution. Distributed query performance is determined by the overall workload


Structuring Queries

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a5b342e484f21015b652c2f5c7685bca.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/620016d0cba8496b9ae75bdcd7533083.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/57bc33b5a1124caa8b117e73bf8b1988.html

of the logical server at any given time. Similarly, in a single run of a query with a long processing time, the work distribution changes over the course of query execution as the load balance changes across worker nodes.

NoteThe -iqmc and -iqtc switches allow different cache sizes for each node in a multiplex, but this may have adverse affects. For example, if a node worker is configured with a much smaller cache than the leader, hash joins on the leader will operate in a paging mode that disallows parallelism.

See Queries Likely to Benefit from DQP in SAP IQ Performance and Tuning Series: Basics.

A high-speed private interconnect is preferred for best distributed query performance, but not required. See Network Options in the SAP IQ Installation and Update Guide for your platform.

NoteDo not use the NOEXEC option to examine DQP performance. NOEXEC is not useful for troubleshooting DQP.

In this section:

Influencing Parallelism and Scalability [page 77]Operators, options and query behavior affect query parallelism.

Controlling Distributed Queries [page 79]Certain database options are specifically designed to manage distributed query processing.

Queries Likely to Benefit from DQP [page 79]Certain types of queries scale better than others.

Queries Unlikely to Benefit from DQP [page 79]Certain types of queries inherently do not scale well, and the optimizer may choose not to distribute them because they will probably perform best on a single node.

Related Information

Queries Likely to Benefit from DQP [page 79]

8.2.1 Influencing Parallelism and Scalability

Operators, options and query behavior affect query parallelism.

Query Operators

In order for a query operator to be distributed, it must be able to be executed in parallel. When an operator is executed in parallel, multiple threads can be applied to execute the processing in parallel. Most query operators can be parallelized but not all are distributed. The following query operators are distributed.


• JOIN• Nested Loop/Nested Loop Pushdown• Hash/Hash Pushdown• Sort Merge/Sort Merge PushDown

• GROUP BY• GROUP BY SINGLE• GROUP BY (HASH)• GROUP BY (SORT)

• DISTINCT• DISTINCT (HASH)DISTINCT (SORT)

• SORT• ORDER BY• ORDER BY (N)• SORTED IN

• SUBQUERY – uncorrelated• PREDICATES – Condition Execution (using FP/LF/HG indexes)• OLAP – OLAP RANK and WINDOW with PARTITION• SELECT component of INSERT operations

• INSERT…SELECT• INSERT…LOCATION

Server and Database Options

The following DQP specific options affect query parallelism and performance.

• MAX_QUERY_PARALLELISM – sets an upper bound that limits how parallel the optimizer will permit query operators, such as joins, GROUP BY, and ORDER BY. The default value is 64. Systems with more than 64 CPU cores often benefit from a large value – up to the total number of CPU cores on the system, to a maximum of 512. This option is DQP-specific.

• FP_NBIT_AUTOSIZE_LIMIT – limits the number of distinct values that implicitly load as NBit FP if there is no IQ_UNIQUE constraint.

• FP_NBIT_LOOKUP_MB –sets a threshold for the total NBit dictionary size if there is no IQ_UNIQUE constraint.

• FP_NBIT_ROLLOVER_MAX_MB – sets the dictionary size for implicit NBit rollovers from NBit to Flat FP.• FP_NBIT_ENFORCE_LIMITS – enforces NBit dictionary sizing limits.

For more database and server options that affect parallelism, see the related topics.

Behaviors That Prevent Distribution

Query fragments that have the following behavior are never distributed:


Structuring Queries

• Write to the database (including DDL, INSERT, LOAD, UPDAT,E and DELETE)• Reference temporary tables• Reference tables that reside in the SYSTEM DBSpace• Reference proxy tables• Utilize non-deterministic functions, such as NEWID

8.2.2 Controlling Distributed Queries

Certain database options are specifically designed to manage distributed query processing.

When a worker node does not finish processing its query fragment within the MPX_WORK_UNIT_TIMEOUT value, the work is passed back to the leader to retry. If time-outs adversely affect DQP performance, you can increase the time-out value up to 3600 seconds to allow a worker to complete. In general, however, most time-out issues result from some other underlying problem. If DQP occurs without visible benefits, you can turn off the temporary database option DQP_ENABLED for your database connection.

8.2.3 Queries Likely to Benefit from DQP

Certain types of queries scale better than others.

Queries likely to distribute well include:

• Compute-intensive column scans, such as LIKE conditions.• Complex queries involving aggregation, expensive expressions, and numeric data types.• Queries comprised of query fragments that reduce the size of intermediate or final results. An example of

this is a chain of hash joins with a “group by hash” at the top.• Queries with low cardinality data that uses hash-based processing. This occurs with star schemas, which

are characterized by a large fact table with low-cardinality dimension tables.• Queries with medium cardinality data, provided that you tune database options, and allocate more

memory to temp cache, to bias the query optimizer to choose more hash based algorithms.

8.2.4 Queries Unlikely to Benefit from DQP

Certain types of queries inherently do not scale well, and the optimizer may choose not to distribute them because they will probably perform best on a single node.

Examples include:

• Queries that return many rows, so that returning rows is a large part of the query execution time. Note that producing rows out of the “top” of a query is a serial operation that cannot be distributed.

• Small queries. Queries that finish in less than 2 seconds are unlikely to benefit from DQP; those between 2 and 10 seconds are somewhat more likely to benefit; and those greater than 10 seconds are generally more likely to benefit.


• Queries with many fragments. If there are many fragments, sorts are often involved. Sorts may reduce scalability, because sorting large amounts of data uses disk storage in the IQ_SHARED_TEMP DBSpace. This is another reason to place the shared temporary DBSpace on the fastest storage possible.

Joining large, high-cardinality tables leads to merge joins, which do not scale as well as hash joins.

8.3 Generating Query Plans

Generating a query plan can help you understand the execution plan developed by the optimizer.

Before it executes any query, the query optimizer creates a query execution plan. A query execution plan represents the set of steps the database server uses to access information in the database related to a statement. The execution plan for a statement can be saved and reviewed, regardless of whether it was just optimized, whether it bypassed the optimizer, or whether its plan was cached from previous executions. Although a query execution plan may not correspond exactly to the syntax used in the original statement, operations described in the execution plan are semantically equivalent to the original query.

Query plans generate an execution tree that consists of a series of nodes that represent a processing stage. The lowest nodes on the tree are leaf nodes. Each leaf node represents a table in the query. At the top of the plan is the root of the operator tree. Information flows up from the tables and through any operators representing joins, sorts, filters, stores, aggregation, and subqueries.

Load Execution Plans

Load execution plans detail the steps that the database engine uses to insert data into a table. Load plans use the same database and output options as query execution plans. The Data Flow Object (DFO) tree identifies the number of rows processed at each stage of the load. Different SQL statements may generate different DFO trees and the same statement may generate different trees for different kind of tables (un-partitioned, range partitioned, hash partitioned, hash-range partitioned, etc.).

Generating Query Plans

To generate a query plan, set the appropriate evaluation options, then execute the query. Text versions of the plan are written to the .iqmsg file. HTML versions can be displayed in the Interactive SQL Plan Viewer or in most Web browsers.

NoteUse query plans only to evaluate the efficiency of a particular query or load. Running SAP IQ with the QUERY_PLAN option set to ON can significantly impact performance, particularly as the volume of INSERT...VALUE statements increases.

In this section:


Structuring Queries

Query Evaluation Options [page 81]Setting the appropriate options helps you evaluate the query plan.

Using Query Plans [page 83]Set the QUERY_PLAN_AS_HTML option to generate an HTML version of the query plan that you can view in a Web browser.

Viewing DQP in a Query Plan [page 84]SAP query plans measure how work is distributed, show timing details, and show which servers participated in processing.

Timing Details in a Query Plan [page 85]Each query plan includes a timing diagram that shows how work for a particular node was distributed to servers and threads.

Thread Details in a Query Plan [page 86]The threads display in a query plan shows which threads on which servers perform work at a particular time.

8.3.1 Query Evaluation Options

Setting the appropriate options helps you evaluate the query plan.

Option Description

INDEX_ADVISOR When set to ON, the index advisor prints index recommendations as part of the SAP IQ query plan or as a separate message in the SAP IQ message log file if query plans are not enabled. These messages begin with the string “Index Advisor:” and you can use that string to search and filter them from an SAP IQ message file. This option outputs messages in OWNER.TABLE.COLUMN format and is OFF by default.

See also the sp_iqindexadvice procedure in System Procedures in the SAP IQ SQL Reference

INDEX_ADVISOR_MAX_ROWS

Used to limit the number of messages stored by the index advisor. Once the specified limit is reached, the INDEX_ADVISOR does not store new advice. It does, however, continue to update count and timestamps for existing advice.

NOEXEC When set to ON, SAP IQ produces a query plan but does not execute the entire query. When the EARLY_PREDICATE_EXECUTION option is ON, some portions of a query are still executed.

You should not set EARLY_PREDICATE_EXECUTION to OFF, because the query plan may greatly differ from when the query is run normally.

QUERY_DETAIL When this option and either QUERY_PLAN or QUERY_PLAN_AS_HTML are both ON, SAP IQ displays additional information about the query when producing its query plan. When QUERY_PLAN and QUERY_PLAN_AS_HTML are OFF, this option is ignored.

QUERY_PLAN When set ON (the default), SAP IQ produces messages about queries.


Option Description

QUERY_PLAN_TEXT_ACCESS

When this option is turned ON, you can view, save, and print IQ query plans from the Interactive SQL client. When QUERY_PLAN_ACCESS_FROM_CLIENT is turned OFF, query plans are not cached, and other query plan-related database options have no effect on the query plan display from the Interactive SQL client. This option is OFF by default.

See GRAPHICAL_PLAN function [String] and HTML_PLAN function [String] in SAP IQ SQL Reference.

QUERY_PLAN_AFTER_RUN

When set to ON, the query plan is printed after the query has finished running. This allows the plan to include additional information, such as the actual number of rows passed on from each node of the query. This option does not work if QUERY_PLAN is set to OFF, its default setting.

QUERY_PLAN_AS_HTML Produces a graphical query plan in HTML format for viewing in a Web browser. Hyperlinks between nodes make the HTML format much easier to use than the text format in the .iqmsg file. Use the QUERY_NAME option to include the query name in the file name for the query plan. This option is OFF by default.

QUERY_PLAN_AS_HTML_DIRECTORY – When QUERY_PLAN_AS_HTML is ON and a directory is specified with QUERY_PLAN_AS_HTML_DIRECTORY, SAP IQ writes the HTML query plans in the specified directory.

QUERY_PLAN_TEXT_CACHING

Gives users a mechanism to control resources for caching plans. With this option OFF (the default), the query plan is not cached for that user connection.

If the QUERY_PLAN_TEXT_ACCESS option is turned OFF for a user, the query plan is not cached for the connections from that user, regardless of the setting for QUERY_PLAN_TEXT_CACHING.

See also GRAPHICAL_PLAN function [String] and HTML_PLAN function [String] in SAP IQ SQL Reference.

QUERY_TIMING Controls the collection of timing statistics on subqueries and some other repetitive functions in the query engine. Normally, set this to OFF (the default); for very short correlated subqueries, the cost of timing every subquery execution can be very expensive in terms of performance.

QUERY_PLAN_MIN_TIME Specifies a threshold for query execution. The query plan is generated only if query execution time exceeds the threshold in microseconds. This can improve system performance by turning off query plan generation for queries with very short execution times.

NoteQuery plans can add a lot of text to your .iqmsg file. When QUERY_PLAN is ON, and especially if QUERY_DETAIL is ON, you might want to enable message log wrapping or message log archiving to avoid filling up your message log file.

Parent topic: Generating Query Plans [page 80]


Structuring Queries

Related Information

Using Query Plans [page 83]Viewing DQP in a Query Plan [page 84]Timing Details in a Query Plan [page 85]Thread Details in a Query Plan [page 86]sp_iqindexadvice ProcedureGRAPHICAL_PLAN Function [String]HTML_PLAN Function [String]

8.3.2 Using Query Plans

Set the QUERY_PLAN_AS_HTML option to generate an HTML version of the query plan that you can view in a Web browser.

HTML versions can be displayed in the Interactive SQL Plan Viewer. HTML query plan, each node in the tree is hyper-linked to processing details. You can click on any node to navigate quickly through the plan.

SQL functions GRAPHICAL_PLAN and HTML_PLAN return IQ query plans in XML and HTML format, respectively, as a string result set. Database options QUERY_PLAN_TEXT_ACCESS and QUERY_PLAN_TEXT_CACHING control the behavior of the new functions.

View graphical query plans in the Interactive SQL Plan Viewer. Text plans are not supported in the Plan Viewer return the error message, Plan type is not supported. Use the SQL functions, GRAPHICAL_PLAN and HTML_PLAN, to return the query plan as a string result.


Related Information

Query Evaluation Options [page 81]Viewing DQP in a Query Plan [page 84]Timing Details in a Query Plan [page 85]Thread Details in a Query Plan [page 86]GRAPHICAL_PLAN Function [String]HTML_PLAN Function [String]QUERY_PLAN_TEXT_ACCESS OptionQUERY_PLAN_TEXT_CACHING Option


https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a5ab8bc984f2101593388431a4a60c82.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a553c53a84f21015b601a68bb24731e3.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a557623e84f21015afe7f3fd90fa41ad.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a553c53a84f21015b601a68bb24731e3.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a557623e84f21015afe7f3fd90fa41ad.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a64f443284f21015acbbd8e919d9c49b.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a64fc89984f210159c879335f5b3713d.html

8.3.3 Viewing DQP in a Query Plan

SAP query plans measure how work is distributed, show timing details, and show which servers participated in processing.

The query plan is a graphical representation of the query tree. If part of a query is distributed, the plan shows three black vertical bars between the distributed nodes. To show the number of distributed rows, hover the mouse cursor over the row count beside the bars. The thickness of the rightmost bar reflects the number of distributed rows.

Figure 2: Sample Query Plan Showing Distributed Query Fragments


Related Information

Query Evaluation Options [page 81]Using Query Plans [page 83]Timing Details in a Query Plan [page 85]Thread Details in a Query Plan [page 86]


Structuring Queries

8.3.4 Timing Details in a Query Plan

Each query plan includes a timing diagram that shows how work for a particular node was distributed to servers and threads.

The timing diagram appears below the query tree. The top of the diagram shows timings for each phase of query execution for each node in the tree, across all servers in the multiplex. The CPU utilization part of the timing diagram shows aggregated times across all servers.

Below the timing diagram are node-specific details. In the fragment shown, SAP IQ assigned 25 work units to server_123 and 2, 6, 4, 4, 3, 2, 3, and 1 of those work units respectively to 8 different threads. The second table shows the amount of private and shared temporary space used to execute query fragment 3. Parallel Sink Work Units are the total number of work units for the entire fragment. First Worker Work Unit shows the number of the first work unit assigned to a worker. If the number exceeds 1, the leader executed some work before the worker started processing, perhaps due to a delay getting the worker what it needs to start working.

#39 Parallel Combiner

Parent Node #09

Child Node 1 #06

Generated Result Rows 40

Estimated Result Rows 1

Work Units – server_123 25 (2, 6, 4, 4, 4, 3, 2, 3, 1)

Work Units – server_456 24 (4, 3, 4, 3, 3, 3, 3, 1)

Max. Possible Parallel Arms 16

Max. Active Parallel Threads 16

Parallel Sink Work Units 49

First Worker Work Unit 2

Act. temp space used for this node (MB) 0.00000000

Act. shared temp space for this node (Mb) 101 812 50000

Fragment ID 3


Related Information

Query Evaluation Options [page 81]Using Query Plans [page 83]Viewing DQP in a Query Plan [page 84]Thread Details in a Query Plan [page 86]


8.3.5 Thread Details in a Query Plan

The threads display in a query plan shows which threads on which servers perform work at a particular time.

Thread assignments appear as a stacked bar graph.

If you mouse over a thread block, you will see various statistics, such as:

• #53: The number of the node at the root of the query fragment that is executing• S:2: Server ID (2) that owns the threads doing the processing• T: 0–3: Range of threads doing the processing• A:2: Average number of threads (2) doing the processing during that slice of time• N:3: Number of samples taken (3) to calculate thread statistics during that slice of time• 23:25:13… - 23:25:14: Start and end times of the time slice

If a query fragment executes on multiple servers at the same time, thread blocks for the same root node of the fragment appear stacked above each other.


Related Information

Query Evaluation Options [page 81]Using Query Plans [page 83]


Structuring Queries

Viewing DQP in a Query Plan [page 84]Timing Details in a Query Plan [page 85]

8.4 Controlling Query Processing

Any user can set limits on the amount of time spent processing a particular query. Users with the SET ANY PUBLIC OPTION system privilege can give certain user's queries priority over others, or change processing algorithms to influence the speed of query processing.

In this section:

Setting Query Time Limits [page 87]Set the MAX_QUERY_TIME option to limit the time a query can run. If a query takes longer to execute than the MAX_QUERY_TIME , SAP IQ stops the query with an appropriate error.

Setting Query Priority [page 88]Setting query priority options assigns query processing priorities by user.

Setting Query Optimization Options [page 89]Optimization options affect query processing speed.

Setting User-Supplied Condition Hints [page 90]Selectivity hints help the optimizer choose an appropriate query strategy.

Monitoring Workloads [page 90]Use the stored procedures that monitor table, column, and index usage for better query performance.

8.4.1 Setting Query Time Limits

Set the MAX_QUERY_TIME option to limit the time a query can run. If a query takes longer to execute than the MAX_QUERY_TIME , SAP IQ stops the query with an appropriate error.

NoteSAP IQ truncates all decimal <option-value> settings to integer values. For example, the value 3.8 is truncated to 3.

See Database Options in SAP IQ SQL Reference for details on setting these database options.

Parent topic: Controlling Query Processing [page 87]

Related Information

Setting Query Priority [page 88]


Setting Query Optimization Options [page 89]Setting User-Supplied Condition Hints [page 90]Monitoring Workloads [page 90]Database Options

8.4.2 Setting Query Priority

Setting query priority options assigns query processing priorities by user.

Queries waiting in queue for processing are queued to run in order of the priority of the user who submitted the query, followed by the order in which the query was submitted. No queries are run from a lower priority queue until higher priority queries have all been executed.

The following options assign queries a processing priority by user.

• IQGOVERN_PRIORITY – assigns a numeric priority (1, 2, or 3, with 1 being the highest) to queries waiting in the processing queue.

• IQGOVERN_MAX_PRIORITY – allows users with the SET ANY SYSTEM OPTION system privilege to set an upper boundary on IQGOVERN_PRIORITY for a user or a role.

• IQ_GOVERN_PRIORITY_TIME – allows high priority users to start if a high priority (priority 1) query has been waiting in the -iqgovern queue for more than a designated amount of time.

To check the priority of a query, check the IQGovernPriority attribute returned by the sp_iqcontext stored procedure.

See SAP IQ SQL Reference for details on setting these database options.


Related Information

Setting Query Time Limits [page 87]Setting Query Optimization Options [page 89]Setting User-Supplied Condition Hints [page 90]Monitoring Workloads [page 90]IQGOVERN_PRIORITY OptionIQGOVERN_MAX_PRIORITY OptionIQGOVERN_PRIORITY_TIME Option


Structuring Queries

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a629349a84f210158efed53aa6889155.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63a661e84f210158ee2c8e1602d1eb4.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63a36dd84f21015b5a7cce84258e338.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63a967f84f21015ba7fc9407ef812d1.html

8.4.3 Setting Query Optimization Options

Optimization options affect query processing speed.

• AGGREGATION_PREFERENCE – controls the choice of algorithms for processing an aggregate (GROUP BY, DISTINCT, SET functions). This option is designed primarily for internal use; use it only if you are an experienced database administrator.

• JOIN_PREFERENCE – controls the choice of algorithms when processing joins.• DEFAULT_HAVING_SELECTIVITY_PPM – Sets the selectivity for all HAVING predicates in a query,

overriding optimizer estimates for the number of rows that will be filtered by the HAVING clause.• DEFAULT_LIKE_MATCH_SELECTIVITY_PPM – sets the default selectivity for generic LIKE predicates, for

example, LIKE '<string%string>' where % is a wildcard character. The optimizer relies on this option when other selectivity information is not available and the match string does not start with a set of constant characters followed by a single wildcard.

• DEFAULT_LIKE_RANGE_SELECTIVITY_PPM – sets the default selectivity for leading constant LIKE predicates, of the form LIKE '<string%>' where the match string is a set of constant characters followed by a single wildcard character (%). The optimizer relies on this option when other selectivity information is not available.

• MAX_JOIN_ENUMERATION – sets the maximum number of tables to be optimized for join order after optimizer simplifications have been applied. Normally you should not need to set this option.

• DML_Options90 – requires SAP IQ 16.1 SP 03 or later. Controls the use of zone maps to potentially improve query performance. See Zone Maps [page 75].

See SAP IQ SQL Reference for details on setting these database options.


Related Information

Setting Query Time Limits [page 87]Setting Query Priority [page 88]Setting User-Supplied Condition Hints [page 90]Monitoring Workloads [page 90]AGGREGATION_PREFERENCE OptionJOIN_PREFERENCE OptionDEFAULT_HAVING_SELECTIVITY_PPM OptionDEFAULT_LIKE_MATCH_SELECTIVITY_PPM OptionDEFAULT_LIKE_RANGE_SELECTIVITY_PPM OptionMAX_JOIN_ENUMERATION OptionDML_OPTIONS90 Option


https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a62bf97e84f21015843682381c2172d5.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63bb2e884f210158490d70844b57a75.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a633d7fb84f21015bfdcb8e97b696f63.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a634664c84f2101583ecdbfe31ed14ef.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63494cd84f210159d72d59e38117625.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a63eeb7884f210158f16b0a316e7e5e1.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/57bc33b5a1124caa8b117e73bf8b1988.html

8.4.4 Setting User-Supplied Condition Hints

Selectivity hints help the optimizer choose an appropriate query strategy.

The SAP IQ query optimizer uses information from available indexes to select an appropriate strategy for executing a query. For each condition in the query, the optimizer decides whether the condition can be executed using indexes, and if so, the optimizer chooses which index and in what order with respect to the other conditions on that table. The most important factor in these decisions is the selectivity of the condition; that is, the fraction of the table’s rows that satisfy that condition.

The optimizer normally decides without user intervention, and it generally makes optimal decisions. In some situations, however, the optimizer might not be able to accurately determine the selectivity of a condition before it has been executed. These situations normally occur only where either the condition is on a column with no appropriate index available, or where the condition involves some arithmetic or function expression and is, therefore, too complex for the optimizer to accurately estimate.


Related Information

Setting Query Time Limits [page 87]Setting Query Priority [page 88]Setting Query Optimization Options [page 89]Monitoring Workloads [page 90]User-Supplied Condition Hints

8.4.5 Monitoring Workloads

Use the stored procedures that monitor table, column, and index usage for better query performance.

Indexes are often created to provide optimization metadata and to enforce uniqueness and primary/foreign key relationships. Once an index is created, however, DBAs face the challenge of quantifying benefits that the index provides.

Tables are often created in the IQ Main Store for the temporary storage of data that must be accessed by multiple connections or over a long period. These tables might be forgotten while they continue to use valuable disk space. Moreover, the number of tables in a data warehouse is too large and the workloads are too complex to manually analyze usage.

Thus, unused indexes and tables waste disk space, increase backup time, and degrade DML performance.

SAP IQ offers tools for collecting and analyzing statistics for a defined workload. DBAs can quickly determine which database objects are being referenced by queries and should be kept. Unused tables/columns/indexes can be dropped to reduce wasted space, improve DML performance, and decrease backup time.

Workload monitoring is implemented using stored procedures, which control the collection and report detailed usage of table, column, and, index information. These procedures complement INDEX_ADVISOR functionality,


Structuring Queries

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a5023f4284f21015abc5e5c08be1034b.html

which generates messages suggesting additional column indexes that may improve performance of one or more queries. Once recommended indexes have been added, their usage can be tracked to determine if they are worth keeping.


Related Information

Setting Query Time Limits [page 87]Setting Query Priority [page 88]Setting Query Optimization Options [page 89]Setting User-Supplied Condition Hints [page 90]

8.5 Optimizing Delete Operations

SAP IQ chooses the best algorithm to process delete operations on columns with HG and WD indexes.

In this section:

HG Delete Operations [page 91]SAP IQ chooses one of three algorithms to process delete operations on columns with an HG (High_Group) index.

WD Delete Operations [page 93]SAP IQ chooses one of three algorithms to process delete operations on columns with a WD (Word) index.

TEXT Delete Operations [page 94]SAP IQ chooses one of two algorithms to process delete operations on columns with a TEXT index.

8.5.1 HG Delete Operations

SAP IQ chooses one of three algorithms to process delete operations on columns with an HG (High_Group) index.

• Small delete provides optimal performance when rows are deleted from very few groups. It is typically selected when the delete is only 1 row or the delete has an equality predicate on the columns with an HG index. The small delete algorithm can randomly access the HG. Worst case I/O is proportional to the number of groups visited.

• Mid delete provides optimal performance when rows are deleted from several groups, but the groups are sparse enough or few enough that not many HG pages are visited. The mid delete algorithm provides ordered access to the HG. Worst case I/O is bounded by the number of index pages. Mid delete has the added cost of sorting the records to delete.


• Large delete provides optimal performance when rows are deleted from a large number of groups. The large delete scans the HG in order until all rows are deleted. Worst case I/O is bounded by the number of index pages. Large delete is parallel, but parallelism is limited by internal structure of the index and the distribution of group to deleted from. Range predicates on HG columns can be used to reduce the scan range of the large delete.

HG Delete Costing

The delete cost model considers many factors including I/O costs, CPU costs, available resources, index metadata, parallelism, and predicates available from the query.

Specifying predicates on columns that have HG or enumerated FP indexes greatly improves costing. In order for the HG costing to pick an algorithm other than large delete, it must be able to determine the number of distinct values (groups) affected by deletions. Distinct count is initially assumed to be lesser of the number of index groups and the number of rows deleted. Predicates can provide an improved or even exact estimate of the distinct count.

Costing currently does not consider the effect of range predicates on the large delete. This can cause mid delete to be chosen in cases where large delete would be faster. You can force the large delete algorithm if needed in these cases, as described in the next section.

Using HG Delete Performance Option

You can use the HG_DELETE_METHOD option to control HG delete performance.

The value of the parameter specified with the HG_DELETE_METHOD option forces the use of the specified delete algorithm as follows:

• 1 = Small delete• 2 = Large delete• 3 = Mid delete• DML_OPTIONS5 = 4 (Disable Push Delete Predicates) Default 0 – Disables pushing range predicates to the

HG large delete.

Parent topic: Optimizing Delete Operations [page 91]

Related Information

WD Delete Operations [page 93]TEXT Delete Operations [page 94]HG_DELETE_METHOD Option


Structuring Queries

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a638554984f210159bbf86926740c08b.html

8.5.2 WD Delete Operations

SAP IQ chooses one of three algorithms to process delete operations on columns with a WD (Word) index.

• Small delete provides optimal performance when the rows deleted contain few distinct words, so that not many WD pages need to be visited. The WD small delete algorithm performs an ordered access to the WD. Worst case I/O is bounded by the number of index pages. Small delete incorporates the cost of sorting the words and record IDs in the records to delete.

• Mid delete for WD is a variation of WD small delete, and is useful under the same conditions as small delete, that is, when the rows deleted contain few distinct words. Mid delete for WD sorts only words in the records to delete. This sort is parallel, with parallelism limited by the number of words and CPU threads available. For Word index, the mid delete method is generally faster than small delete.

• Large delete provides optimal performance when the rows deleted contain a large number of distinct words, and therefore need to visit a large number of “groups” in the index. The large delete scans the WD in order, until all rows are deleted. Worst case I/O is bounded by the number of index pages. Large delete is parallel, but parallelism is limited by the internal structure of the index and the distribution of groups from which to delete.

WD Delete Costing

The WD delete cost model considers many factors including I/O costs, CPU costs, available resources, index metadata, and parallelism.

You can use the WD_DELETE_METHOD database option to control WD delete performance.

Using WD Delete Performance Option

The value of the parameter specified with the WD_DELETE_METHOD option forces the use of the specified delete algorithm as follows:

• 0 = Mid or large delete as selected by the cost model• 1 = Small delete• 2 = Large delete• 3 = Mid delete


Related Information

HG Delete Operations [page 91]TEXT Delete Operations [page 94]


WD_DELETE_METHOD Option

8.5.3 TEXT Delete Operations

SAP IQ chooses one of two algorithms to process delete operations on columns with a TEXT index.

• Small delete provides optimal performance when the rows deleted contain few distinct words, so that not many TEXT pages need to be visited. The TEXT small delete algorithm performs an ordered access to the TEXT. Worst case I/O is bounded by the number of index pages. Small delete incorporates the cost of sorting the words and record IDs in the records to delete.

• Large delete provides optimal performance when the rows deleted contain a large number of distinct words, and therefore need to visit a large number of “groups” in the index. The large delete scans the TEXT in order, until all rows are deleted. Worst case I/O is bounded by the number of index pages. Large delete is parallel, but parallelism is limited by the internal structure of the index and the distribution of groups from which to delete.

TEXT Delete Costing

The TEXT delete cost model considers many factors including I/O costs, CPU costs, available resources, index metadata, and parallelism.

You can use the TEXT_DELETE_METHOD database option to control TEXT delete performance.

Using TEXT Delete Performance Option

The value of the parameter specified with the TEXT_DELETE_METHOD option forces the use of the specified delete algorithm as follows:

• 0 – Mid or large delete as selected by the cost model• 1 – Small delete• 2 – Large delete


Related Information

HG Delete Operations [page 91]WD Delete Operations [page 93]TEXT_DELETE_METHOD Option


Structuring Queries

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a668acaf84f21015bb88a6eb8a856991.html

https://help.sap.com/viewer/a898e08b84f21015969fa437e89860c8/16.1.5.0/en-US/a663868b84f21015a23cbc124262f0e9.html

Important Disclaimers and Legal Information

HyperlinksSome links are classified by an icon and/or a mouseover text. These links provide additional information.About the icons:

• Links with the icon : You are entering a Web site that is not hosted by SAP. By using such links, you agree (unless expressly stated otherwise in your agreements with SAP) to this:

• The content of the linked-to site is not SAP documentation. You may not infer any product claims against SAP based on this information.

• SAP does not agree or disagree with the content on the linked-to site, nor does SAP warrant the availability and correctness. SAP shall not be liable for any damages caused by the use of such content unless damages have been caused by SAP's gross negligence or willful misconduct.

• Links with the icon : You are leaving the documentation for that particular SAP product or service and are entering a SAP-hosted Web site. By using such links, you agree that (unless expressly stated otherwise in your agreements with SAP) you may not infer any product claims against SAP based on this information.

Videos Hosted on External PlatformsSome videos may point to third-party video hosting platforms. SAP cannot guarantee the future availability of videos stored on these platforms. Furthermore, any advertisements or other content hosted on these platforms (for example, suggested videos or by navigating to other videos hosted on the same site), are not within the control or responsibility of SAP.

Beta and Other Experimental FeaturesExperimental features are not part of the officially delivered scope that SAP guarantees for future releases. This means that experimental features may be changed by SAP at any time for any reason without notice. Experimental features are not for productive use. You may not demonstrate, test, examine, evaluate or otherwise use the experimental features in a live operating environment or with data that has not been sufficiently backed up.The purpose of experimental features is to get feedback early on, allowing customers and partners to influence the future product accordingly. By providing your feedback (e.g. in the SAP Community), you accept that intellectual property rights of the contributions or derivative works shall remain the exclusive property of SAP.

Example CodeAny software coding and/or code snippets are examples. They are not for productive use. The example code is only intended to better explain and visualize the syntax and phrasing rules. SAP does not warrant the correctness and completeness of the example code. SAP shall not be liable for errors or damages caused by the use of example code unless damages have been caused by SAP's gross negligence or willful misconduct.

Bias-Free LanguageSAP supports a culture of diversity and inclusion. Whenever possible, we use unbiased language in our documentation to refer to people of all cultures, ethnicities, genders, and abilities.

SAP IQ Performance and Tuning Series: BasicsImportant Disclaimers and Legal Information PUBLIC 95

www.sap.com/contactsap

© 2022 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice.

Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies.

Please see https://www.sap.com/about/legal/trademark.html for additional trademark information and notices.

THE BEST RUN

https://www.sap.com/about/legal/trademark.html

sap iq performance and tuning series: basics

Documents