performance tuning through iterations

Performance Tuning a CFML script

Gert FranzRasia GmbH

WHO AM I?

• Gert Franz

– Involved with Lucee and Railo since day 1

– Into CFML for 10 11 12 13 14 15 16 17 years

– DBA, System architect

– CTO Rasia Ltd, CIO Rasia CH

• Rasia is founding Member of the Lucee association

MY EVOLUTION

Assembler Basic Clipper Pascal Delphi Java CFML Lucee

WHAT IS THIS ABOUT

• Having some fun with CFML

• Discrediting long known approaches

• Looking for new ways of solving a given problem

TASK AT HAND

• The client reported that there was

– An import job that imports an .xlsx file

– Into a database table

– With some special rules

INITIAL CODE

• The code is around 2 years old

• Takes around 20 minutes to execute

• Simply imports a total of 25000 recordsinto a database with some specific checks

• So nothing really fancy

STARTING POINT

• Let's have a look at the source code that was used to fulfil the task

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

MAIN POINTS VERSION 1

• Uses a default datasource• Uses lists for lookups• Creates an array to hold fieldnames• Creates a query and imports all into the

query• Uses two QoQ to delete unnecessary

records• Uses a QoQ to add only needed records to

an array• Insert the array into the database table

IDEAS? PROPOSALS?

FIRST ITERATION

• Convert everything to CFScript

WHY???

ITERATION 1

• Use a function to load the XLSX content

• Optimize filling the fields

• Exit early in the cleanUpNumber() function

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript, reduced the field building effort

ITERATION 2

• Replace the main QoQ with a nested struct

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

QUERY OF QUERY EVIL

QUERY OF QUERY

• Lucee is no database server

• So nothing of the following

– Query analyzer

– Indexes

– Statistics

– Query Optimizations

– etc!

QUERY OF QUERY

• Avoid if possible at all times!

ITERATION 3

• Eliminated the evaluate approach

EVALUATE – SMALLER EVIL

EVALUATE

• Evaluate sometimes does a lot of parsing

• I mostly use it only on serialized complex objects

• VERY OFTEN SEEN:

<cfset b = evaluate("form.#fieldName#")>

Instead use:

<cfset b = form[fieldname]>

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

ITERATION 4

• Eliminated the SQL statements from getMarketID()

• Avoid too many unnecessary SQL statements

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

ITERATION 5

• Eliminated all QoQ

• Only added stuff when model > 0

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

ITERATION 6

• Why create a query in the first place?

• Why not create the resulting struct right away?

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminated creating a query and filling the struct

ITERATION 7

• What is always the thing that consumes the most time?

• Queries

• Let's get rid of many of them

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

ITERATION 8

• We only fill the struct in cases where it makes sense

• So why filter afterwards, if we can avoid filling these values in, in the first place

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Exiting early from filling the struct

ITERATION 9

• Create a bulk insert, instead of lots of individual inserts

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine

9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File

ITERATION 10

• Instead of nested structs, use composite keys

• Nested structs use lots of recursions

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine

9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File

10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key

ITERATION 11

• Replaced lists with arrays

• Generally speaking: Lists are to be avoided if possible

• Used len() instead of isEmpty() and added an exit early strategy

INTERMEDIATE RESULT

# T (s) Δ (s) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine

9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File

10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key

11 0.32 0.09 21.90% 0.05% 2050.31 4'035 B Replaced isEmpty() with len() and eq 0, eliminated Trim

FINAL RESULT CHART

0

100

200

300

400

500

600

700

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

100

200

300

400

500

600

700

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

0.5

1

1.5

2

2.5

3

3.5

4

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4 5 6 7 8 9 10 11 12

FINAL RESULT CHART

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

1 2 3 4 5 6 7 8 9 10 11 12

AS A PIE CHART

3.6%

91.7%

0.9%1.5% 0.4%0.2%1.0%0.1%0.5%0.0%0.0%

3.6%

0.9%

1.5%

0.4%

0.2%

1.0%

0.1% 0.5%

0.0%0.0%

CONCLUSIONS

• DON'T USE QueryOfQuery

• Don't use heavily nested structs

• Avoid lists, use structs or arrays instead

• Avoid too many database calls

CONCLUSIONS

• If you know the data you're handling, performance tuning is a lot easier

• First generalize in order to understand the code

• Then specialize in order to tune it