edit scripting language specification'...01.20 25/04/2013 added xtnet/edit domain related...

Commission européenne, B-1049 Bruxelles / Europese Commissie, B-1049 Brussel - Belgium. Telephone: (32-2) 299 11 11.

Commission européenne, L-2920 Luxembourg. Telephone: (352) 43 01-1.

EUROPEAN COMMISSION DIRECTORATE-GENERAL INFORMATICS Information systems Directorate

European Commission

Specific Contract Fixed Price 008154

"EDIT Scripting Language Specification" under framework contract DI/06770

Date: 23/09/2015

Version: 1.120

Authors: Gheorghe MORARU, Bogdan

TUDOR

Revised by: Gheorghe MORARU, Bogdan

TUDOR

Approved by: Theodoros MIGLAKIS

Public: No

Reference Number: D1.7.1

"EDIT Scripting Language Specification" / 86 Document Version 1.120 dated 23/09/2015

TABLE OF CONTENTS

1. TERMINOLOGY ..................................................................................................................................... 7

1.1. Abbreviations and Acronyms ................................................................................................................... 7

1.2. Syntax Definition ..................................................................................................................................... 7

2. OVERVIEW OF THE SCRIPTING LANGUAGE ............................................................................... 8

3. FUNCTIONALITY OVERVIEW ........................................................................................................... 8

4. DETAILED FUNCTIONALITY DESCRIPTION ................................................................................. 9

4.1. Format Definition ..................................................................................................................................... 9

4.2. Program Definition ................................................................................................................................ 11

4.3. Steps Definition...................................................................................................................................... 16

4.4. Auxiliary Datasets .................................................................................................................................. 18

4.4.1. Simplified Lookup .............................................................................................................................. 19

4.4.2. Simplified Transcode .......................................................................................................................... 19

4.5. External Data Sources ............................................................................................................................ 20

4.6. Pre-Editing Programs ............................................................................................................................. 22

4.7. Temporary Outputs ................................................................................................................................ 24

4.8. Validation Output Datasets .................................................................................................................... 24

4.9. Dataset Operation Definition ................................................................................................................. 26

4.9.1. Aggregation ......................................................................................................................................... 26

4.9.2. Alter .................................................................................................................................................... 27

4.9.3. Append ................................................................................................................................................ 29

4.9.4. Copy .................................................................................................................................................... 30

4.9.5. Keysets Template ................................................................................................................................ 31

4.9.6. Merge .................................................................................................................................................. 32

4.9.7. Missing Values .................................................................................................................................... 33

4.9.8. Hierarchical Merge ............................................................................................................................. 34

4.9.9. Transpose ............................................................................................................................................ 35

4.9.10. Reorder ............................................................................................................................................. 36

4.9.11. Select ................................................................................................................................................. 37

4.9.12. Sort .................................................................................................................................................... 38

4.9.13. Split ................................................................................................................................................... 39

4.9.14. Statistics ............................................................................................................................................ 39

4.9.15. TYPE ................................................................................................................................................ 40

4.9.15.1. CopyMetadataToAllOutputs .......................................................................................................... 41

4.9.15.2. SetOutputType ............................................................................................................................... 41

4.9.15.1. GenerateMoveToRefHeaders ......................................................................................................... 41

4.9.15.2. GenerateTot ................................................................................................................................... 42

4.9.15.3. GenerateVal ................................................................................................................................... 42


4.9.15.4. GenerateErrReport ......................................................................................................................... 42

4.9.15.5. GenerateChiffrierErr ...................................................................................................................... 43

4.9.15.6. GenerateCamouflage ...................................................................................................................... 43

4.9.15.7. GeneratePrevision .......................................................................................................................... 43

4.9.15.8. GenerateReference ......................................................................................................................... 44

4.9.15.9. GeneratePub ................................................................................................................................... 44

4.9.15.10. DetailedStatisticsExport ............................................................................................................... 45

4.9.16. Update ............................................................................................................................................... 45

4.9.17. Import................................................................................................................................................ 46

4.9.18. EVE Dataset Operation ..................................................................................................................... 47

4.9.19. Export ............................................................................................................................................... 48

4.9.20. Comext .............................................................................................................................................. 50

4.9.21. Food Safety ....................................................................................................................................... 52

4.9.22. Growth Rate ...................................................................................................................................... 53

4.9.23. HBHistorical (Hidiroglou Berthelot) ................................................................................................ 54

4.9.24. LFS.................................................................................................................................................... 55

4.9.25. Sigma Gap ......................................................................................................................................... 57

4.9.26. Urban Method 3 ................................................................................................................................ 58

4.9.27. Time Series ....................................................................................................................................... 60

4.9.27.1. Input Format ................................................................................................................................... 60

4.9.27.2. Output Format ................................................................................................................................ 60

4.9.27.3. Timeseries operation ...................................................................................................................... 60

4.10. Validation Rules Definition ................................................................................................................. 61

4.10.1. No Record Rule................................................................................................................................. 61

4.10.2. Horizontal Rule ................................................................................................................................. 62

4.10.3. Vertical Rule ..................................................................................................................................... 62

4.10.4. Hierarchical Rule .............................................................................................................................. 64

4.11. Scripting Language Expressions Definition ......................................................................................... 65

4.11.1. Expression Fields .............................................................................................................................. 65

4.11.2. Data Types ........................................................................................................................................ 65

4.11.3. Operators ........................................................................................................................................... 66

4.11.4. Functions ........................................................................................................................................... 67

4.11.5. Variables ........................................................................................................................................... 72

4.12. Comments ............................................................................................................................................ 73

4.13. Multi-line Strings ................................................................................................................................. 73

4.14. New Line Characters ............................................................................................................................ 73

4.15. Keyword Abbreviations ....................................................................................................................... 74

5. ANNEX I – RESERVED KEYWORDS ................................................................................................ 74

5.1. Scripting Language Reserved keywords ................................................................................................ 74

5.2. Oracle 10G Reserved Words.................................................................................................................. 76


5.3. PostgreSQL v8 Reserved Words ............................................................................................................ 83


Document History

Version Date Comment Modified

Pages

00.10 07/06/2012 EBB2011 version updated with new dataset

operations and functions

-

01.00 08/06/2012 Update to final version -

01.10 25/07/2012 Corrected errors in examples 9,12

01.20 25/04/2013 Added XTNET/Edit domain related

functions supportability

27-30

01.30 05/07/2013 Added format details to AGGREGATION 13

01.40 08/07/2013 Added aux_datasets 15-17

01.50 10/07/2013 Added foreach, transcodeDate 56-57, 60, 63

01.60 26/08/2013 Fixed EDAMIS publish example 40

01.70 07/10/2013 Added aggregateToVariable,

excludeValues, findTranspositionValue,

getTranspositionValues, lpad, rpad,

transcodeStatistics

59-62

01.80 12/17/2013 Added strings in sql condition for select,

order values in getTranspositionValues,

bulk export

30, 39-42, 59

01.90 05/02/2014 Corrective update -

01.95 24/02/2014 Added new features, External Data Sources,

Pre-Editing Programs, Temporary outputs,

Validation Output Datasets. Added 4.9.1.1.

GenerateMoveToRefHeaders, Updated

4.11.4 Functions

17-22, 38,

64-67

01.100 30/05/2014 Added Missing Values in Format definition 7,8

01.101 25/07/2014 Modified the inLookupDate function

definition

65

01.102 28/07/2014 Add the admitted Boolean values.

Modify the Precision definition in the

format definition.

7

01.110 05/08/2014 Added Detailed Statistics Export, added

transposed variables, added

getFileRepoFolder function

42, 70, 66

01.111 08/08/2014 Fixed minor issues in the Vertical rules

definition

60


01.112 05/11/2014 Added reserved keywords for Oracle 10g

and PostgreSQL v8 for EDIT 2013, moved

all reserved section as an ANNEX

74

01.113 26/01/2015 Add dynamic specification of table name

and where clause

20,21

01.114 03/02/2015 Add getDatasetInstanceName,

getEngineVersion and getJobName

functions

68

01.115 26/06/2015 Modified detailedStatisticsExport for QTM

011054 – Task 9

45

01.120 23/09/2015 Added the reformatDate function

description

70


1. TERMINOLOGY

1.1. Abbreviations and Acronyms

Terms/Abbreviation Description

CLI Command Line Interface

PDL Program Definition Language

FDL Format Definition Language

ESLE EDIT Scripting Language Expression

1.2. Syntax Definition

The following rules will be used during the definition of the syntax of the Program Definition

Language:

Definition Description

ItalicText Keyword

[...] Optional element

[...]* Optional repeatable element

[...]+ Repeatable element with at least one occurrence

[ItalicText] Optional Keyword

[ItalicText ...]+ Repeatable keywords, see also repeatable parameters.

Elements in the enumeration will not be separated by any

special character.

[ItalicText1… |

ItalicText2… |… ]

Choice between two or more elements.

[ItalicText1… |

ItalicText2… |… ]?

Optional choice between two or more elements.

<Text> Parameter

[<Text>] Optional Parameter

[<Text>]+ Repeatable parameters. The elements in the enumeration

will be separated by comma. Ex: Key1, Key2, Key3

[<ESLE>]+ Block of EDIT Scripting Language Expressions. The

expressions in the block will be separated by semicolon.

Ex:

product := “testProd”; value := value + 1;


2. OVERVIEW OF THE SCRIPTING LANGUAGE

The Edit System offers the possibility to store the definition of formats or programs in text

files. These definition files can be imported into the System through the Command Line

Interface or the Web Application. The language used to create format definition files and

program definition files is called the Edit Scripting Language.

The language used to define a format in a definition file is called the Format Definition

Language. The FDL has similar functional capabilities as the GUI, allowing the user to define a

format or a file format which contains various types of fields.

For consistency purposes the users are advised to use “.FDL” for the extension of the files in

which they store format or file format definitions.

An FDL file can contain definitions for multiple formats.

The language used to define a program in a definition file is called the Program Definition

Language. The PDL has similar functional capabilities as the GUI, allowing the user to define a

program which contains various types of steps and rules.

For consistency purposes the users are advised to use “.PDL” for the extension of the files in

which they store program definitions.

A PDL file can contain definitions for multiple programs.

3. FUNCTIONALITY OVERVIEW

Format Definition Language:

1. Format Definition

Program Definition Language:

2. Program Definition

3. Steps Definition

4. Dataset Operations Definition

5. Rules Definition

Auxiliary Functions:

6. Comments

7. Multiline Strings

8. Multi-line Commands

9. New Line Characters

10. Keyword Abbreviations


4. DETAILED FUNCTIONALITY DESCRIPTION

4.1. Format Definition

The following syntax is used to define a format:

SYNTAX :

FORMAT <FORMAT_NAME> {

[DESCRIPTION <FORMAT_DESCRIPTION>;]

FIELDS {

<FIELD_NAME> {

[DESCRIPTION <FIELD_DESCRIPTION>;]

CAPTION <CAPTION>;

[MVALUES <MISSING_VALUE>+;]

[DEFAULTVALUE <VALUE>;]

TYPE [STRING | NUMBER | DOUBLE | BOOLEAN];

LENGTH <LENGTH> [.<PRECISION>] [MANDATORY];

}

}

}

Note: The BOOLEAN type allows the following values:

for TRUE: “1”, “true”, “t”, “y”

for FALSE: “0”, “false”, “f”, “n”;

All the values are case-insensitive. In the dataset details page the displayed value for True will

be 1 and for False will be 0.

The maximum allowed length is for each type as follows:

STRING – 4000

NUMBER – 38

DOUBLE - 38

Boolean – N/A

PARAMETERS:

<FORMAT_NAME> The name of the new format

<FORMAT_DESCRIPTION> The description of the new format

<FIELD_NAME> The name of the field

<FIELD_DESCRIPTION> The description of the field

<CAPTION> Caption to be used for the field


<MISSING_VALUE> Value to be considered as NULL during import

<VALUE> String or numeric value

<LENGTH> Total length of the field

<PRECISION> Precision to be used by the DOUBLE type. A DOUBLE type declared

with 0 as the precision or without precision will be identical to a NUMBER type.

Ex:

DValue {

CAPTION Value;

TYPE DOUBLE;

LENGTH 6.0;

}

NValue {

CAPTION Value;

TYPE NUMBER;

LENGTH 6.0;

}

DValue and NValue will be treated as Number types.

EX:

FORMAT MyFormat {

DESCRIPTION “This is my format“;

FIELDS {

Year {

CAPTION “Year”;

TYPE NUMBER;

LENGTH 4;

}

Value {

CAPTION Value;

MVALS ":","-";

TYPE DOUBLE;

LENGTH 6.2 MANDATORY;

}

}


}

4.2. Program Definition

The following syntax is used to define a program:

SYNTAX:

PROGRAM <PROGRAM_NAME> {

[DESCRIPTION <PROGRAM_DESCRIPTION>;]

[[INPUT <DATASET_FORMAT_NAME>

<DATASET_REFERENCE_VARIABLE_NAME>

[<CONSTANT_DATASET_INSTANCE>];]

| [EXTERNAL <DATASET_FORMAT_NAME>

<DATASET_REFERENCE_VARIABLE_NAME> {

[ IMPORTPATH <EXTERNAL_PATH>;

[PROPERTIES{

[<PROPERTY_NAME> < PROPERTY_NAME >;]+

}]

| [ PROPERTIESTEMPLATE <PROPERTY_TEMPLATE_NAME> ;]]

FILETYPE <IMPORT_TYPE>;

[FIELDSETTEMPLATE <IMPORT_FIELDSET_TEMPLATE>;]

]

| [TYPE DB;

PROPERTIES{

[<PROPERTY_NAME> < PROPERTY_NAME >;]+

}

[FIELDSETTEMPLATE <IMPORT_FIELDSET_TEMPLATE>;]]]]+

[TEMPORARY <DATASET_REFERENCE_VARIABLE_NAME> [,

<DATASET_REFERENCE_VARIABLE_NAME>]]

[KEYSET <KEYSET_NAME> [<FIELD_NAME>]+]*

[PARAMETER <PARAMETER_NAME> <FORMAT_FIELD>;]*

[PARAMETERSET <PARAMETER_SET_NAME> {

[<PARAMETER_NAME>=<PARAMETER_VALUE>;]+

} ]

[CONSTANT [STRING | BOOLEAN | NUMBER | DOUBLE] <CONSTANT_NAME> :=

<CONSTANT_VALUE>; ]*


STEPS {

<STEPS_DEFINITION_SCRIPT>

}

}

PARAMETERS:

<PROGRAM_NAME> The name of the new program

<PROGRAM_DESCRIPTION> The description of the new program

<DATASET_FORMAT_NAME> Name of the dataset format

<DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset reference

variable

< CONSTANT_DATASET_INSTANCE> Name of an existing dataset instance to be

used as a constant input for this program

<EXTERNAL_PATH> The name for the external file. There are three cases:

It is a string denoting the real path of the file (in the server)

It is a string denoting a parameter of type string (the value of this parameter will

be the real path of the file in the server)

It is a string denoting a parameter of type file (the file is on the user computer

and it will be uploaded)

<PROPERTY_NAME> Name of the import property

< PROPERTY_NAME > Value of the import property

<PROPERTY_TEMPLATE_NAME> Name of the property template

<IMPORT_TYPE> Type of the imported file (CSV, FLR, ...)

<IMPORT_FIELDSET_TEMPLATE> Name of the fieldset template

<KEYSET_NAME> Name for a reusable keyset that groups fields from a format

<PARAMETER_NAME> Name of the parameter

<FORMAT_FIELD> Format field representing the type of the parameter

<PARAMETER_SET_NAME> Name of the parameter set

<PARAMETER_VALUE> Value of the parameter

<CONSTANT_NAME> Name of the constant

<CONSTANT_VALUE> Value of the constant

<STEPS_DEFINITION_SCRIPT> Block of script containing the definition of the

program steps

EX :

PROGRAM p01 {

DESCRIPTION “Monthly trades program”;


INPUT inputTrades MonthlyTradesFormat;

PARAMETER pContinent continent;

PARAMETER pYear year;

PARAMETERSET mypset {

pContinent=Asia;

pYear=1500;

}

CONSTANT STRING const1 := “test 23”;

STEPS {

DATAOPERATION copyDS {

COPY {

INPUT inputTrades;

OUTPUT tradesCopy;

}

}

VALIDATION {

INPUT tradesCopy;

ERROR errDs;

ERRORDATA errData;

RULES {

RECORD countryNotEmpty {

CONDITION country != "";

ERRMSG 'failed' SEVERITY Error;

}

RECORD checkChinaImports{

CONDITION (country = "CHINA" AND flow = “I”) ->

quantity = 0 AND value = 0;

ELSE {quantity := 0; value := 0;}

}

RECORD recalculateChina {;

CONDITION (country = "CHINA" AND flow=”E”) ->

value < 10000000;

THEN {

value := value * 100;

quantity:= quantity *100;


}

ELSE {

value := value * 10;

quantity:=quantity * 10;

}

}

VERTICAL v01 {

EXPRESSION {

KEYS country, flow, month;

TRKEYS country;

CONDITION value[‘GB ‘] > 100000;

}


}

}

}

DATAOPERATION aggregate {

AGGREGATION {

INPUT tradesCopy;

OUTPUT aggregatedTrades;

CREATE NEW FORMAT annualTrades;

KEYS country, flow;

SUM(value); SUM(quantity); DELETE month;

}

}

}

}

Program with external file input:

PROGRAM preedit1 {

EXTERNAL countryDsd input1 {

IMPORTPATH fileParam;

PROPERTIESTEMPLATE inTemplate;

FILETYPE CSV;

FIELDSETTEMPLATE fstmpl;

}


PARAMETER fileParam FILE;

steps

{

VALIDATION Validation {

INPUT input1;

ERROR errData;

RULES {

RECORD record1 {

condition false;

errmsg 'message' severity 'fatal';

}

}

}

}

}

Program with external database for input and output:

PROGRAM dbreadwrite {

EXTERNAL countryDsd dsin {

TYPE DB;

PROPERTIES {

CONNECTION conn1;

SELECT readTemplate;

INDEX row_number;

}

}

EXTERNAL countryDsd dsi1{

TYPE DB;

PROPERTIES{

CONNECTION conn1;

SELECT stmt1;

INSERT stmt2;

INDEX COUNTRY_ID;

}

}

steps


{


INPUT dsin;

OUTPUT dsi1 countryDsd;

RULES {

RECORD record1 {

condition true;

then {

dsi1.row_number := row_number;

dsi1.country := country;

dsi1.quantity := quantity;

write(dsi1);

}

}

}

}

}

}

4.3. Steps Definition

The steps script contains a series of definitions of steps that will be executed in the program.

The steps can be either a dataset operation or a validation step.

Step conditions must be defined before dataset operation scripts for dataset operations and

input definition for validation steps.

SYNTAX :

STEPS {

[DATAOPERATION <STEP_NAME> {

[CONDITION { <BOOLEAN_CONDITION>; }]

<DATASET_OPERATION_SCRIPT>

} |

VALIDATION <STEP_NAME> {

[CONDITION <BOOLEAN_CONDITION>;]

INPUT <DATASET_REFERENCE_VARIABLE_NAME>;

[LOOKUP <LOOKUP_DATASET_REFERENCE_VARIABLE_NAME>;]*


[AUX <AUXILIARY_DATASET_REFERENCE_VARIABLE_NAME>;]*

[ERROR <ERROR_ DATASET_REFERENCE_VARIABLE_NAME>;]

[ERRORDATA

<ERROR_DATA_DATASET_REFERENCE_VARIABLE_NAME>;]

[VALIDDATA

<VALID_DATA_DATASET_REFERENCE_VARIABLE_NAME>;]

[OUTPUT

<OUTPUT_DATA_DATASET_REFERENCE_VARIABLE_NAME>

<OUTPUT_DATA_DATASET_FORMAT_NAME>]+

RULES {

<VALIDATION_RULES_SCRIPT>

}

} ]

}

PARAMETERS : <STEP_NAME> Name of the step

<BOOLEAN_CONDITION> A BOOLEAN type condition (SL expression), if a single

boolean variable is used, then it must be passed as an argument to the isTrue function;

Ex: CONDITION isTrue(BOOL_VAR);

<DATASET_OPERATION_SCRIPT> Block of script containing the definition of a

dataset operation

<DATASET_REFERENCE_VARIABLE_NAME> Name of the input dataset

reference variable

<LOOKUP_DATASET_REFERENCE_VARIABLE_NAME> Name of a dataset

reference variable to be prepared for a lookup operation in the current step

<AUXILIARY_DATASET_REFERENCE_VARIABLE_NAME> Name of a dataset

reference variable to be prepared for use as an auxiliary dataset in the current step

<ERROR_ DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset

reference variable where the errors will be stored

<ERROR_DATA_DATASET_REFERENCE_VARIABLE_NAME> Name of the

dataset reference variable to be used for storing the rows that failed validation

<VALID_DATA_DATASET_REFERENCE_VARIABLE_NAME> Name of the

dataset reference variable to be used for storing the rows that passed validation

successfully

< OUTPUT_DATA_DATASET_FORMAT_NAME > Name of the dataset format used

for the output (validation outputs)

EX :

STEPS {

DATAOPERATION copyDS {

CONDITION “a” = “b”;


COPY {

INPUT inputTrades;

OUTPUT tradesCopy;

}

}

VALIDATION {

CONDITION NUM_VAR < 100;

INPUT tradesCopy;

ERROR errDs;

ERRORDATA errData;

OUTPUT validReference countryDsd;

OUTPUT invalidReference countryDsd;

RULES {

RECORD countryNotEmpty {



}

RECORD externalReference {


THEN {

validReference.country = country;

validReference.price = 201.45;

write(validReference);

} ELSE {

invalidReference.country = country;

invalidReference.price = 0;

write(invalidReference);

}

}

}

}

4.4. Auxiliary Datasets

Auxiliary datasets can be used for Simplified Lookup and Simplified Transcode operations

based on a set of keys defined as the keyset of the auxiliary dataset.

SYNTAX:


VALIDATION validationStep {

INPUT …;

AUX <REFERENCE_NAME> (<AUX_KEY>)+;

…

}

PARAMETERS:

< REFERENCE_NAME > Name of the reference to be loaded as an auxiliary dataset

< AUX_KEY > Field to be used as a key of the auxiliary dataset

EX:

VALIDATION validationStep {

INPUT workData;

AUX lookupData CODE, MONTH;

ERROR errorLog;

RULES {

…

}

}

4.4.1. Simplified Lookup

The Simplified Lookup checks the existence of a combination of keys in an auxiliary dataset.

SYNTAX:

isNull(<AUX_REFERENCE_NAME>([‘<AUX_KEY_VALUE>’])+)

PARAMETERS:

< AUX_REFERENCE_NAME > Name of the auxiliary dataset reference

< AUX_KEY_VALUE > Value of the auxiliary key; can be an explicit value or a field

from the current partition

EX:

isNull(lookupData[COUNTRY][‘JAN’])

4.4.2. Simplified Transcode

The Simplified transcode operation fetches a value from an auxiliary dataset corresponding to a

set of specified value keys.

SYNTAX:

<AUX_REFERENCE_NAME>.<VALUE_COLUMN>([‘<TRANSPOSITION_VALUE>’])+)

PARAMETERS:


< VALUE_COLUMN > Value column to be returned from the auxiliary dataset

< AUX_REFERENCE_NAME > Name of the auxiliary dataset reference

< TRANSPOSITION_VALUE > Value of the transposition or a field from the current

partition

EX:

lookupData.TARGET[COUNTRY][‘JAN’]

4.5. External Data Sources

The program can accept the following external sources as an input or output:

External file

External database

External data sources may be used as:

input for lookup reference,

input for aux data references

input references in validation steps executing only RECORD and NORECORD rules

output reference in a validation step

For external databases, it is required that the connection template is pre-defined as well as a

numeric index column to be used for paginated reads. If the data source will be used as an

output the insert template is required and if it used as input the select template is required. An

additional requirement is that for the select template the first field should be the index of the

selection (the records are selected and ordered based on this field).

External files may use three different types of input values for the file to be read:

string containing the path to the file

name of a parameter or variable to be extracted from the vector map at runtime

FILE type parameter; this will require for the file to be uploaded and passed on as a

parameter before the program is executed

For external databases the table name and where clause can be dynamically specified at runtime

by using tablename and where properties (in the example below the TABLE_NAME_V and

WHERE_CLAUSE are two variables that are assigned at runtime prior to using the data

source).

Examples:

EXTERNAL countryCodes dsIn {

TYPE DB;

PROPERTIES {

CONNECTION external_db;

SELECT listCountries;

INDEX row_number;

}


}

EXTERNAL countryCodes outputDatabase{

TYPE DB;

PROPERTIES{

CONNECTION local_external_db;

INSERT writeCountries;

INDEX row_number;

}

}

EXTERNAL countryDsd constantInput {

IMPORTPATH "c:/ebb2013/input/FailRows.csv";


FILETYPE CSV;

FIELDSETTEMPLATE allFields;

}

EXTERNAL countryDsd fileInput {



FILETYPE CSV;


}


EXTERNAL TEST_MD OLD_VISIBLE_DATA{

TYPE DB;

PROPERTIES{

CONNECTION template_old_visible_data;

SELECT select_old_visible_data;

INDEX MYINDEX;

TABLENAME TABLE_NAME_V;

WHERE WHERE_CLAUSE;

}


FIELDSETTEMPLATE TEST_MD_ALL;

}

4.6. Pre-Editing Programs

Pre-editing programs allow users to execute validation rules on data when it is imported,

allowing also for the data to be split into separate datasets.

In order for a program to be considered a “pre-editing” program, the following restrictions

apply:

The program must define exactly one external file datasource

The external datasource must use a FILE type parameter as IMPORTPATH

All dataset inputs must be constant

Only RECORD and NORECORD rules may be applied in the program

The program may define multiple outputs

On program import, the engine checks for these restrictions and if they are fulfilled marks the

program as pre-editing using the summary column from the Program entity.

In order to write data to a dataset instance, an output dataset must be declared in the validation

step (for more information regarding validation outputs please refer to section 4.7).

The following additional functions have been implemented to support the pre-editing

functionality:

partitionReadOK() – check if the row has been read successfully or not by the import

plugin

getInvalidPartitionData() – get the raw data that failed to be processed correctly by the

import plugin (Ex: the current row in a CSV file)

getPartitionReadError() – get the error that has been recorded by the import plugin

Example:

PROGRAM PreEditTestWrite {

EXTERNAL countryDsd fileInput {



FILETYPE CSV;


}


steps

{


INPUT fileInput;


ERROR errData;

OUTPUT outputDataset countryDsd;

RULES {

RECORD checkAbort {

CONDITION getErrorCount() < 100;

ELSE {

abort("Max errors exceeded");

}

}

RECORD initFlag {

VAR BOOLEAN FLAG_WRITE := true;

}

RECORD checkRead {

CONDITION partitionReadOK();

ELSE {

VAR RAW_DATA := getInvalidPartitionData();

VAR ERR_MESSAGE := "Error while reading row:" +

getPartitionReadError();

FLAG_WRITE := false;

}

ERRMSG "Error reading file" SEVERITY "FATAL" (RAW_DATA,

ERR_MESSAGE);

}

RECORD checkSkip {

CONDITION partitionReadOK();

ELSE {

skipPartition();

}

}

RECORD checkValue {

CONDITION NOT isNull(VALUE) AND VALUE > 7500;

ELSE {

FLAG_WRITE := false;

}

ERRMSG "VALUE must be > 7500" SEVERITY "ERROR" (VALUE);


}

RECORD writeToDataset {

CONDITION FLAG_WRITE;

THEN {

VAR init := fromVectorMap(outputDataset, ".", "");

VAR count := write(outputDataset);

}

}

}

}

}

}

4.7. Temporary Outputs

Temporary outputs – these are used internally by the program and will not be persisted in the

database. If an output it is not declared as temporary, it will be persisted in the database.

Syntax:

PROGRAM <Program Name> {

INPUT <input format and reference name>;

…

TEMPORARY <TEMPORARY_REFERENCE_LIST>;

…

}

Example:

PROGRAM ValidationProgram {

INPUT COUNTRY_DSD input1;

INPUT FlagCodes CFLAG CFLAG;

TEMPORARY output1, output2;

…

}

4.8. Validation Output Datasets

Starting with EDIT2013 validation steps have been enhanced to support the generation of

outputs. Multiple outputs can be declared at the step level and referenced in the validation

rules.

Output declaration Syntax:

VALIDATION Validation_Step {


INPUT <input reference>;

ERROR <error reference>;

[OUTPUT <OUTPUT_REFERENCE_NAME> <OUTPUT_REFERENCE_FORMAT>;]*

…

}

Example:

VALIDATION VALIDITYCHECKS1 {

INPUT INPUT_DATA;

ERROR ERROR_LOG;

OUTPUT OUTPUT1 COUNTRY_DSD;

…

}

In order to access the field of an output reference, the following syntax must be used:

Output usage Syntax:

<REFERENCE_NAME>.<FIELD_NAME>

Example:

CONDITION FIELD1 > 0;

THEN {

OUTPUT_1.FIELD10 := FIELD1 + FIELD2;

OUTPUT_2.FIELD4 := FIELD3 – FIELD4;

}

The following additional functions have been implemented to support the usage of output

references

write(datasetReference) – write the current record of the output reference to the

database; a new record is written every time this function is called

reset(datasetReference) – reset all values in the specified output reference record;

records which have already been written are not affected

fromVectorMap(datasetReference, thousandsSeparator, decimalSeparator) – initialize

all possible values on the output reference record by extracting based on field names

information from the Vector Map; automatic conversions are performed by the engine if

the column types are different

fromTrValue(datasetReference, thousandsSeparator, decimalSeparator

transpositionPath…) – initialize all possible values on the output reference record by

extracting based on field names information from the Vector Map using the specified

transposition path; automatic conversions are performed by the engine if the column

types are different


4.9. Dataset Operation Definition

4.9.1. Aggregation

The Aggregation operation performs a series of aggregations on an input dataset based on

grouping functions and stores the data in the output dataset.

The format of the output for the Aggregation operation is defined as follows:

The CREATE NEW option means that the dataset definition for the

output will be created, and an error is given if the name is already used by a different

format.

The USE EXISTING option means that the format must exist in the

database with the specified name.

The name of the output fields will have the following format:

if the fields are aggregated by using a single function, the name of the

output fields will be the same as the name of the aggregated fields

if fields are aggregated using multiple functions, the name of the

output fields will follow the pattern “function_field”

in the case an aggregation accepts multiple arguments, the name of the

output fields will contain the names of the fields passed as arguments separated by “_”

EX :

Sum(price) -> price, sum(quantity) -> quantity (different fields are

aggregated using a single function)

sum(price) -> sum_price, max(price) -> max_price (the field price is

aggregated using two different functions)

wmean(price, quantity) -> price_quantity (an aggregation function

takes two fields as arguments)

SYNTAX :

AGGREGATION {


OUTPUT <OUT_DATASET_REFERENCE_VARIABLE_NAME>;

[CREATE NEW | USE EXISTING] FORMAT <FORMAT_NAME>;

KEYS [<FIELD_NAME>]+;

[<GROUPING_FUNCTION> (<ARGUMENTS>); ]+

[DELETE [<FIELD_NAME>]+ ;]

}

PARAMETERS :


reference variable


<OUT_DATASET_REFERENCE_VARIABLE_NAME> Name of the dataset

reference variable to be used for storing the output data

<FORMAT_NAME> Name of the output format to be created/used

<FIELD_NAME> Field to be used as a key for a grouping function

<GROUPING_FUNCTION> (<ARGUMENTS>) Grouping function to be used. The

supported grouping functions are:

Mean : mean(price) – accepts only number columns as parameter

Count : count(country) – accepts any column type as parameter

Sum : sum(value) – accepts only number columns as parameter

Max : max(price) – accepts only number columns as parameter

Min : min(price) – accepts only number columns as parameter

Variance : variance(value) – accepts only number columns as parameter

Standard deviation : stddev(quantity) – accepts only number columns as

parameter

Median : median(price) – accepts only number columns as parameter

Weighted mean : wmean(price,quantity) – accepts only number columns as

parameters

EX :

AGGREGATION

{

INPUT input1;

OUTPUT output1;

USE EXISTING FORMAT aggDD;

KEYS country;

sum(price);

count(product);

DELETE month,type,quantity,value;

}

4.9.2. Alter

The Alter operation alter the data from one dataset in order to drop or (and) create fields. The

output is another dataset. The format of this new dataset can be a new or an existing

(compatible) one. . The DEFAULTVALUE option gives a default value for the empty fields.

The RENAME option gives the possibility of renaming the fields. The CONDITION option

gives the possibility of filtering out the data with this sql condition.

SYNTAX :


ALTER{



[DROP FIELDS [<FIELD_NAME>]+;]

[CREATE FIELDS {

<FIELD_NAME> {

[DESC <FIELD_DESCRIPTION>;]

CAPTION <FIELD_CAPTION>;

MISSINGVALUES [<FIELD_MISSING_VALUE> ]+;

[DEFAULTVALUE <FIELD_DEFAULT_VALUE>;]

TYPE <FIELD_TYPE>;

LENGTH <FIELD_LENGTH>;

[MANDATORY;]

}

}]

[RENAME FIELDS {

[<FIELD_NAME> <NEW_FIELD_NAME>;]+

}]

[DEFAULTVALUE{

[<FIELD_NAME> <FIELD_VALUE>;]+

}]

[CONDITION <CONDITION_STRING>;]


}

PARAMETERS :


reference variable



<FIELD_NAME> Name of the field to be dropped / created

<NEW_FIELD_NAME> The new name of the field

<FIELD_VALUE> Value of the field


< FIELD_DESCRIPTION > Description of the field

< FIELD_CAPTION > Caption of the field


<FIELD_MISSING_VALUE> Possible missing value (forms a list for a field)

<FIELD_DEFAULT_VALUE>

<FIELD_TYPE> Type of the field (can be one of String, Number, Double or Boolean)

<FIELD_LENGTH> Length of the field

MANDATORY Specify if the field it is mandatory or not (if a field is mandatory it

cannot specify the default value)

CONDITION A valid sql condition

EX :

ALTER

{

INPUT input1;

OUTPUT output1;

DROP FIELDS field1;

RENAME FIELDS {

OUT_R1 R1;

}

DEFAULTVALUE {

R1 "ZZZZZZ";

}

CONDITION 'where OUT_R1 is not null';

USE EXISTING FORMAT format1;

}

4.9.3. Append

The Append operation appends the data from one dataset to another dataset. In order to do this,

the operation copies the first input dataset into the output dataset and then it appends the

second input dataset to the output dataset.

If there are some missing fields in the second dataset and CREATE MISSING option is not

present the operation will give an exception. If the CREATE MISSING option is present the

missing fields will be filled with a default value (not specified). The FORCE CONVERSION

option is being used in the case of column type mismatchThe format of the output for the

Append operation is defined as follows:



format.



SYNTAX :


APPEND {

INPUT <FIRST_DATASET_REFERENCE_VARIABLE_NAME>;

INPUT <SECOND_DATASET_REFERENCE_VARIABLE_NAME>;



[CREATE MISSING;] [FORCE CONVERSION;]

}

PARAMETERS :

<FIRST_DATASET_REFERENCE_VARIABLE_NAME> Name of the first input

dataset reference variable

<SECOND_DATASET_REFERENCE_VARIABLE_NAME> Name of the second

input dataset reference variable




EX :

APPEND

{

INPUT input1;

INPUT input2;

OUTPUT output1;

CREATE MISSING;

FORCE CONVERSION;

}

4.9.4. Copy

The Copy operation copies the input dataset into the output dataset.

The IF NOT EMPTY option implies that the actual copying is being performed if the input

dataset is not empty. If the output dataset is not empty, the data is being appended to the output.

The format of the output dataset must be a superset of the input format. The order of the

columns may be different.

SYNTAX :

COPY {



[IF NOT EMPTY;]

}


PARAMETERS :


reference variable

EX :

COPY

{

INPUT input1;

OUTPUT output1;

IF NOT EMPTY;

}

4.9.5. Keysets Template

The Keysets template operation adds a keyset template (together with a set of keysets) for an

input dataset. The keysets must present a hierarchical order.

SYNTAX :

KEYSETSTEMPLATE {


[KEYSET <KEYSET_NAME> KEYS [<KEY_COLUMN>]+]+;

KEYSETSTEMPLATE <KEYSET_TEMPLATE_NAME>;

}

PARAMETERS :


reference variable

<KEYSET_NAME> Name of the keyset

<KEY_COLUMN> Columns that forms the keyset (key columns)

<KEYSET_TEMPLATE_NAME> Name of the keyset template

EX :

KEYSETSTEMPLATE {

INPUT input1;

KEYSET keyset1 KEYS CONTINENT;

KEYSET keyset1 KEYS CONTINENT, COUNTRY;

KEYSETSTEMPLATE keysetTemplate1;

}


4.9.6. Merge

The Merge operation performs a join between two input datasets based on a set of keys. The operation can be performed using four different types of join: INNER, LEFT, RIGHT and OUTER. Fields from the first input dataset can be overwritten if the OVERWRITE option is specified. Columns from the second input dataset can also be renamed as specified by the RENAME option.

The format of the output for the Merge operation is defined as follows:



format.



SYNTAX :

MERGE {






[INNER | LEFT | RIGHT | OUTER] JOIN;

[ [OVERWRITE | DONT OVERWRITE] EXISTING FIELDS ;

| RENAME {

[<FIELD_NAME> TO <NEW_FIELD_NAME>]+

} ]

}

PARAMETERS :








EX :

MERGE

{

INPUT input1;

INPUT input2;


OUTPUT output1;

USE EXISTING FORMAT countryMerge;

KEYS row_number;

LEFT JOIN;

RENAME { country TO mycountry ; price TO myprice; }

}

4.9.7. Missing Values

The Missing values operation detects the missing dates from the input dataset. Missing rows

are inserted with the null values. This operation is partition-based as defined by the keys. The

date column can be of type w(week), m(month), q(quarterly) and s(semester).

SYNTAX :

MISSINGVALUES {



KEYS [<KEY_COLUMN>]+;

AUX <TIME_COLUMN>;

}

PARAMETERS :





< KEY_COLUMN > Name of the key column (part of the partition)

< TIME _COLUMN > Name of the date column (the filling is performed based on the

values of this column)

EX :

MISSINGVALUES

{

INPUT input1;

OUTPUT output1;

KEYS country, product;

AUX dateColumn;

}


4.9.8. Hierarchical Merge

The Hierarchical Merge operation performs a join between many input datasets based on a set of keysets (the join between the datasets is described in the JOIN clause). After the new dataset is created, a keysets template is associated with it, defining the hierarchical structure of the dataset.

SYNTAX :

HIERARCHICAL {

[INPUT <DATASET REFERENCE_NAME> KEYSET

<KEYSET_NAME> [<COLUMN_NAME>]+ ]+;

JOIN {

[<KEYSET_NAME>.<COLUMN_NAME>=<KEYSET_NAME>.<COLUMN_NAME>]+;

}

RENAME {

[<DATASET_REFERENCE_NAME> <COLUMN_NAME> <NEW_COLUMN_NAME>]+;

}

OUTPUT <OUTPUT_FORMAT><OUT_DATASET_REFERENCE >;

[KEYSETSTEMPLATE <KEYSET_TEMPLATE_NAME>];

}

PARAMETERS :

< DATASET REFERENCE_NAME > Name of the input dataset reference variable

< KEYSET_NAME > Name of the keyset

< COLUMN_NAME > Name of the column(s) that are composing the keyset

< NEW_COLUMN_NAME > The new name of the column

< OUTPUT_FORMAT > The name of the format of the output dataset reference format

< OUT_DATASET_REFERENCE > The name of the output dataset reference format

< KEYSET_TEMPLATE_NAME > The name of the keyset template – all the

described keysets are grouped under this name

EX :

HIERARCHICAL {

INPUT DATASET1 KEYSET ContinentalTrade CONTINENT;

INPUT DATASET2 KEYSET CountryTrade COUNTRY,

CONTINENTNAME;

INPUT DATASET3 KEYSET CityTrade CITY, COUNTRYNAME;

INPUT DATASET4 KEYSET GroupTrade NGROUP, CONTINENT, CITY;


INPUT DATASET5 KEYSET SubgroupTrade SUBGROUP, NGROUP,

NAME;

JOIN {

ContinentalTrade.CONTINENT = CountryTrade.CONTINENTNAME;

CountryTrade.COUNTRY = CityTrade.COUNTRYNAME;

ContinentalTrade.CONTINENT = GroupTrade.CONTINENT;

GroupTrade.NGROUP = SubgroupTrade.NGROUP,

GroupTrade.CITY = SubgroupTrade.INAME;

}

RENAME {

DATASET1 POLICY CONTINENTALPOLICY,

TVALUE CONTINENTALTVALUE;

DATASET2 POLICY COUNTRYPOLICY;

}

OUTPUT hMerge1 output1;

KEYSETSTEMPLATE kst1;

}

4.9.9. Transpose

The Transpose operation allows the possibility to transpose a given dataset from horizontal to

vertical and from vertical to horizontal against a given a partition key. In case of the horizontal

transposition the output format will determine the value columns to be transposed and the

transposition values to be taken into account. The format must be predefined by the user and it

must contain besides the partition columns, value columns with the following name

convention:

<COLUMN_NAME><SEPARATOR><TRANSPOSITION_VALUE>

Ex:

VALUE_EN

In the case of the vertical transposition, the value columns must be in accordance to the same

naming convention above, to allow the application to extract the name of the target value

column and the transposition value to be set on the resulting record.

SYNTAX :

TRANSPOSE {

TYPE ["toHorizontal"|”toVertical”]?;

INPUT <INPUT_DATASET>;

OUTPUT <OUTPUT_FORMAT> <OUTPUT_REFERENCE_NAME>;

KEYS [<PARTITION_KEY>]+;

TRKEYS <TRANSPOSITION_KEY>;


FIELDDELIMITER <FIELD_DELIMITER>;

}

PARAMETERS :

<INPUT_DATASET> the input dataset

<OUTPUT_FORMAT> the format of the output

<OUTPUT_REFERENCE_NAME> the name of the output dataset reference

<PARTITION_KEY> name of the partition key column

<TRANSPOSITION_KEY> name of the transposition key column

<FIELD_DELIMITER> delimiter to be used in the name of the transposed value

columns

EX :

DATAOPERATION tranposeDataset {

TRANSPOSE {

TYPE "toHorizontal";

INPUT INPUT1;

OUTPUT newFormat1 OUTPUT1;

KEYS col1, col2, col3;

TRKEYS trkey1;

FIELDDELIMITER "_";

}

}

4.9.10. Reorder

The Reorder operation reorders the rows (changes the ROW_NUMBER) from an input dataset

and stores them in the output dataset according to the FIELDS option. This is an equivalent to a

Sort operation in which the fields are sorted ascending.

The format of the output for the Reorder operation is defined as follows:



format.



SYNTAX :

REORDER {





FIELDS [<FIELD_NAME>]+;

}

PARAMETERS :


reference variable




<FIELD_NAME> Name of a field to be used for the ordering of the rows

EX :

REORDER

{

INPUT input1;

OUTPUT output1;

CREATE NEW FORMAT countryDsd1;

FIELDS QUANTITY;

}

4.9.11. Select

The Select operation selects rows from an input dataset and writes them to the output dataset

according to a specified where condition. The condition must be a valid SQL where clause.

The format of the output dataset is the same as the format of the input dataset.

SYNTAX:

SELECT {



CONDITION <SELECT_CONDITION>;

}

PARAMETERS:


reference variable

reference variable




<SELECT_CONDITION> SQL where clause to be applied for the operation, can be a

string (ex: “where country like ‘%RO%’ ” – the usage of wildcards ‘%’ is possible only

within the strings)

EX:

SELECT

{

INPUT input1;

OUTPUT output1;

CONDITION where price > 100 and country='RO';

}

4.9.12. Sort

The Sort operation reorders the rows (changes the ROW_NUMBER) from an input dataset and

stores them in the output dataset according to the FIELDS option and the specified sort

direction.


SYNTAX:

SORT {



ORDER [<FIELD_NAME> [ASC | DESC] ]+;

}

PARAMETERS:


reference variable



<FIELD_NAME> Name of a field to be used as a sorting key

EX:

SORT

{

INPUT input1;

OUTPUT input2;

ORDER COUNTRY ASC,VALUE DESC;

}


4.9.13. Split

The Split operation creates separate output datasets starting from an input dataset for a list of

partitions based on a set of keys that determine the partitions list. For each partition, an output

dataset will be created using the following naming convention:

<INPUT_DATASET_REFERENCE_VARIABLE_NAME>[_<KEY_VALUE>]+

Ex: input1_RO_100, input1_BG_100, input1_RO_200


SYNTAX:

SPLIT {



}

PARAMETERS:


reference variable



<FIELD_NAME> Name of a field to be used as a key in defining a partition for the

split operation

EX:

SPLIT

{

INPUT input1;

KEYS COUNTRY,PRICE ;

}

4.9.14. Statistics

The Statistics operation computes the statistics of the input dataset by partitions. The statistic

has the predefined format (Q1, Q3 ...).

SYNTAX:

STATISTICS {


STATISTICS <STATISTIC_REFERENCE_VARIABLE_NAME>;

FIELD <FIELD_NAME>;

[KEYS [<KEY_NAME>]+;]

[MISSINGVALUES (TRUE | FALSE);]


}

PARAMETERS:


reference variable

<STATISTIC_REFERENCE_VARIABLE_NAME> Name of the statistic dataset

reference variable

<FIELD_NAME> Name of the field on which the statistic is being computed

<KEY_NAME> Name of the column that is part of the partition

MISSINGVALUES If TRUE, the missing values (null ones) will be treated as 0.0.

Otherwise, the null values will remain the same and will not be part of the computations.

EX:

STATISTICS

{

INPUT input1;

STATISTICS stat1;

FIELD PRICE;

KEYS COUNTRY, PRODUCT ;

}

4.9.15. TYPE

The TYPE operation performs a custom dataset operation.

SYNTAX:

TYPE {

TYPE <METHOD_NAME>; {

[INPUT <INPUT_NAME>;]*

[OUTPUT <OUTPUT_NAME>;]*

[PARAMETER <PARAMETER_VALUE>;]*

}

}

PARAMETERS:

<METHOD_NAME> Name of custom operation

<INPUT_NAME> Name of the input (can be a dataset reference, a variable ...)


<OUTPUT_NAME> Name of the output

<PARAMETER_VALUE> value of the parameter (order is important)

EX :

TYPE {

TYPE generateChiffrierErr;

INPUT errorLog;

OUTPUT CHIFFRIERERR CHIFFRIER_ERR;

PARAM FORMAT_CODE;

}

4.9.15.1. CopyMetadataToAllOutputs

Copies the metadata from the input to all outputs

EX:

TYPE{

TYPE copyMetadataToAllOutputs;

INPUT inputData;

}

4.9.15.2. SetOutputType

Sets the input type to be the corresponding parameter.

EX:

TYPE{

TYPE setOutputType;

INPUT CORRECTEDDATA;

PARAM OUTPUTVALIDDATA;

}

4.9.15.1. GenerateMoveToRefHeaders

Implementation of a XTNET/Edit generateMoveToRefHeaders function which creates the

headers needed for the execution of the Move To Ref operation.

EX:

TYPE{

TYPE generateMoveToRefHeaders;

INPUT inputData;

INPUT V_AS_EU_PART_ISOEU_REPORTERS;

}


4.9.15.2. GenerateTot

Implementation of a XTNET/Edit generateTOT function which computes the sums of the trade

values by flow generating the CHIFFRIER_TOT dataset

EX:

TYPE {

TYPE generateTot;

INPUT correctedData;

OUTPUT CHIFFRIERTOT CHIFFRIER_TOT;

PARAM FORMAT_CODE;

}

4.9.15.3. GenerateVal

Implementation of a XTNET/Edit generateVal function which computes the sums of the trade

values by flow, period and confidentiality flag and generates the CHIFFRIER_VAL dataset

EX:

TYPE {

TYPE generateVal;


OUTPUT CHIFFRIERVAL CHIFFRIER_VAL;

PARAM TOTALVALUE;

PARAM TOTALQUANTITY;

PARAM TOTALCOUNT;

PARAM TRANSACTION_TYPE_TMP;

}

4.9.15.4. GenerateErrReport

Implementation of a function which generates the ERR_REPORT dataset

EX:

TYPE{

TYPE generateErrReport;


INPUT errorLog;

OUTPUT ERRREP ERR_REPORT;

PARAM FORMAT_CODE;

}


4.9.15.5. GenerateChiffrierErr

Implementation of a function which reports the distribution of errors and the associated values

in the CHIFFRIER_ERR dataset

EX:

TYPE{

TYPE generateChiffrierErr;

INPUT errorLog;

OUTPUT CHIFFRIERERR CHIFFRIER_ERR;

PARAM FORMAT_CODE;

}

4.9.15.6. GenerateCamouflage

Implementation of a XTNET/Edit function which produces the BRE and BRI files

EX:

TYPE{

TYPE generateCamouflage;


INPUT V_NC8_CTCI5R3;

INPUT AS_EU_PART_ISO;

INPUT V_TARIC;

INPUT V_NC8;

INPUT AS_VALDEF_DPS_VALDEF_CXT_E;

INPUT AS_VALDEF_DPS_VALDEF_CXT_I;


INPUT AV_NC_HS6_SU;

OUTPUT SSS SSS;

OUTPUT BRE26 BRE;

OUTPUT BRI26 BRI;

PARAM FORMAT_CODE;

}

4.9.15.7. GeneratePrevision

Implementation of a XTNET/Edit-specific function which presents the differences between the

validated trade values and the previsions of those values in the same period

EX:

TYPE{


TYPE generatePrevision;

INPUT BRI;

INPUT V_AS_V_EEC_PREV_CHAP_I;

INPUT V_AS_V_EEC_PREV_CHAP_E;


OUTPUT PREV PREV;

OUTPUT MAJPREV MAJPREV;

PARAM FORMAT_CODE;

}

4.9.15.8. GenerateReference

Implementation of a XTNET/Edit function which computes the sums of the trade values by NC

level, generating the CHIFFRIERREF dataset

EX:

TYPE{

TYPE generateReference;

INPUT BRE;

OUTPUT CHIFFRIERREF CHIFFRIERREF;

PARAM FORMAT_CODE;

}

4.9.15.9. GeneratePub

Implementation of the XTNET/Edit function which computes the validated trade values by

partner and SITC product code, generating the PUB dataset

EX:

TYPE{

TYPE generatePub;

INPUT BRE;

INPUT V_AV_ECU_ALL;

INPUT V_NC8_CTCI5R3;

INPUT CTCI5R3_AGRBEC;

INPUT CTCI1_R3_AGR_CTCI1;

INPUT CTCI3_PUBL;

INPUT V_EU_PARTNERS_ZONES_PUBL_DPS;

OUTPUT PUB PUB;

PARAM FORMAT_CODE;


}

4.9.15.10. DetailedStatisticsExport

Exports the specified input dataset to the path specified by PUBLISH_FILE in the detailed

statistics format.

SYNTAX:

TYPE {

TYPE detailedStatisticsExport;

INPUT <INPUT_DATASET>;

[PARAMETER <PUBLISH_TYPE>;]

PARAMETER <PUBLISH_DETAILS>;

}

PARAMETERS:

<INPUT_DATASET> Name of the input dataset reference; must be an error report in

order for the operation to perform the export correctly

<PUBLISH_TYPE> The type of publish to perform (EDAMIS/FILE); when no publish

type is specified, it is considered to be FILE by default

<PUBLISH_DETAILS> The file location or the transmission ID to be used

EX:

TYPE {

TYPE detailedStatisticsExport;

INPUT input1;

PARAMETER EDAMIS;

PARAMETER TRANSMISSION_ID;

}

4.9.16. Update

The Update operation creates an output dataset containing the rows from the first input dataset,

on which certain fields are updated with values from the second input dataset, by joining the

two input datasets on a set of specified keys.

The format of the output dataset is the same as the format of the first input dataset.

SYNTAX:

UPDATE {






}

PARAMETERS:







<FIELD_NAME> Name of a field to be used as a key for update

EX :

UPDATE

{

INPUT input1;

INPUT input2;

OUTPUT output1;

KEYS country, price;

}

4.9.17. Import

The import operation reads selected columns of data from an external data source and saves

them to a dataset. Depending on the parameter selected as the FILETYPE, the import operation

can read either from a local file (CSV, FLR, etc.), SDMX registry (with option

SDMX_REGISTRY) or from an external database (with option DB). For file imports, an

import path, where the data file is located, must be specified. For database imports, the

CONNECTION and STATEMENT properties must be specified. For SDMX imports, the

CODELIST and VERSION properties must be specified. When an existing name is specified,

users running the program will not have to enter a dataset instance for the output reference in

the creation of the job.

See also the datasetInstanceExpired function.

SYNTAX:

IMPORTDATASET

{

OUTPUT <DATASET_NAME> <DATASET_FORMAT>;

[IMPORTPATH <FILE_PATH>;]

(PROPERTIES { [<IMPORT_KEY> <IMPORT_VALUE>]+


}) | ( PROPERTIESTEMPLATE

<PROP_TEMPLATE_NAME>);

FILETYPE <IMPORTTYPE_NAME>;

[FIELDSETTEMPLATE <FIELDTEMPLATE_NAME>;]

(OVERWRITE (<EXISTING_NAME>)?;)?

}

PARAMETERS:

<DATASETINPUT_NAME> Name of the dataset to be exported

<IMPORT_KEY> The name of the property

<IMPORT_VALUE> The value of the property

<PROP_TEMPLATE_NAME> The template used to export the dataset

<IMPORTTYPE_NAME> The type used to export the dataset

<FIELDTEMPLATE_NAME> The field template used to export the dataset

< EXISTING_NAME > The name of the instance that will be overwritten

EX ;

IMPORTDATASET

{

OUTPUT dsImportFromDB1 testDatasetFormat;

PROPERTIES {

CONNECTION "externaldb";

STATEMENT "select_ext_DB";

}

FILETYPE db;

FIELDS {

TEST1;

TEST2;

TEST3;

TEST4;

}

}

4.9.18. EVE Dataset Operation

The EVE Dataset Operation reads the selected EVE xml file and executes the EVE validation

rules described in it against the input reference. The errors are written in the error reference

(EVE validation is composed from record validations with the ERRMSG option).

SYNTAX:


EVEDO

{

INPUT <INPUT_REFERENCE>;

ERROR <ERROR_REFERENCE>;

IMPORTPATH <FILE_PATH>;

}

PARAMETERS:

< INPUT_REFERENCE > Name of the input reference

< ERROR_REFERENCE > Name of the error reference

< FILE_PATH > The full path of the file (can be a parameter or a variable)

EX :

EVEDO

{

INPUT INPUT1;

ERROR OUTPUT1;

IMPORTPATH "C:/VALIDATION_RULES_1.xml";

}

4.9.19. Export

The export operation saves an output dataset containing the selected columns from an input

dataset. Depending on the parameter selected as the FILETYPE, the export operation can send

data either to a local file (CSV, FLR, etc.), EDAMIS backchannel (EDAMIS) or to an external

database (DB). The operation can be performed either in a synchronous or asynchronous

fashion. For database exports, the CONNECTION and STATEMENT properties must be

specified. If an email address is specified, a notification message will be sent to this address

when the export is performed.

SYNTAX:

EXPORT

{

[INPUT <DATASETINPUT_NAME>]+;

(PROPERTIES { [<EXPORT_KEY> <EXPORT_VALUE>]+

}) | ( PROPERTIES_TEMPLATE

<EXPORTTEMPLATE_NAME>);

FILETYPE <EXPORTTYPE_NAME>;

[FIELDSETTEMPLATE <FIELDTEMPLATE_NAME>;]+

[SYNCHRONOUS (TRUE | FALSE);]

[PUBLISH <PUBLISH_LOCATION>;]


[TO [<EMAIL_ADDRESS>]+;]?

[ORDER [<FIELD_NAME>] [,<FIELD_NAME>]+]*

}

PARAMETERS:

<DATASETINPUT_NAME> Name(s) of the dataset to be exported

<EXPORT_KEY> The name of the property

<EXPORT_VALUE> The value of the property

<EXPORTTEMPLATE_NAME> The template used to export the dataset

<EXPORTTYPE_NAME> The type used to export the dataset

<FIELDTEMPLATE_NAME> The field template used to export the dataset

<PUBLISH_ LOCATION > The location where the exported file will be published; this

location must be accessible to the application

<EMAIL_ADDRESS> The email address where a notification regarding the export

operation will be sent (must be enclosed in double quotes)

<FIELD_NAME> The name of the ordered field

EX:

DATAOPERATION fileExport

{

EXPORT

{

INPUT dataset1;

PROPERTIES_TEMPLATE “template1”;

FILETYPE “csv”;

PUBLISH “/outputs/data/exported_data.csv”;

}

}

DATAOPERATION databaseExport

{

EXPORT

{

INPUT INPUT1;

PROPERTIES {

CONNECTION "externalDB";

STATEMENT "insert_ext_DB";


}

FILETYPE db;

FIELDSETTEMPLATE fieldsetCountry;

SYNCHRONOUS FALSE;

}

}

DATAOPERATION edamisExport

{

EXPORT

{

INPUT INPUT1;

PROPERTIESTEMPLATE defaultTemplate;

FILETYPE “csv”;


PUBLISH EDAMIS edamisId;

TO “[email protected]”;

ORDER COUNTRY ASC, PRODUCT DESC;

}

}

DATAOPERATION fileExport {

EXPORT

{

INPUT dataset1;

INPUT dataset2;

PROPERTIESTEMPLATE "template1";

FILETYPE "csv";

PUBLISH "c:/jira/exported/exported_data.zip";

}

}

4.9.20. Comext

The Comext Method data operation writes in the designated column the result of the Comext

Method outlier algorithm.

SYNTAX :


COMEXT {


FIELD <VALUE_COLUMN>;

[KEYS [<KEY_COLUMN>]+;]

OUTLIER <OUTLIER_COLUMN>;

TRANSPOSE <TRANSPOSE_COLUMN>;

[STATISTICS <DATASET_REFERENCE_STATISTICS>;]

[CONDITION <VALUE_CONDITION>;]

CA <VALUE_THRESHOLD>;

[MISSINGVALUES TRUE|FALSE ]

SEVERITY <VALUE_SEVERITY>;

}

PARAMETERS :


reference variable

<VALUE_COLUMN> Name of the column that gives the input values to the algorithm

<KEY_COLUMN> Name of a field to be used as a key for computing the partitions

<OUTLIER_COLUMN> Name of a field to be used as a key

<TRANSPOSE_COLUMN> Name of a field to be used as a date / time (transposed

column)

<DATASET_REFERENCE_STATISTICS> Name of the statistical reference variable

<VALUE_CONDITION> Expression that defines which observations will be part of

the computation of the median

< VALUE_THRESHOLD> Value of the threshold



< VALUE_SEVERITY> Value of the outlier

EX :

DATAOPERATION comextDO {

COMEXT{

INPUT input1;

FIELD PRICE;

OUTLIER outlier_result;

TRANSPOSE calendar_column;

STATISTICS stat1;

CONDITION VALUE>90;


CA 100;

SEVERITY 'comext';

}

}

4.9.21. Food Safety

The Food Safety data operation writes in the designated column the result of the Food Safety

Domain outlier algorithm.

SYNTAX :

FOODSAFETY {








INCREASE <VALUE_INCREASE>;

DECREASE <VALUE_DECREASE>;


}

PARAMETERS :


reference variable





column)




<VALUE_INCREASE> Value of the maximum growth rate between periods

< VALUE_DECREASE> Value of the minimum growth rate between periods



EX :

DATAOPERATION foodSafety {

FOODSAFETY{

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

INCREASE 0.25;

DECREASE 0.995;

SEVERITY 'FOOD_SAFETY';

}

}

4.9.22. Growth Rate

The Growth Rate data operation writes in the designated column the result of the Growth Rate

between periods outlier algorithm.

SYNTAX :

GROWTHRATE {









DECREASE <VALUE_DECREASE>;


}

PARAMETERS :


reference variable






column)




<VALUE_INCREASE> Value of the maximum growth rate between periods

< VALUE_DECREASE> Value of the minimum growth rate between periods


EX :

DATAOPERATION growthRate {

GROWTHRATE{

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

INCREASE 0.25;

DECREASE 0.995;

SEVERITY 'GROWTH_RATE';

}

}

4.9.23. HBHistorical (Hidiroglou Berthelot)

The HB (Hidiroglou Berthelot) historical data operation writes in the designated column the

result of the HB historical bounds outlier algorithm.

SYNTAX :

HBHISTORICAL {







[CI <VALUE_CI>;]

[CE <VALUE_CE>;]

[CA <VALUE_CA>;]


[CP <VALUE_CP>;]

EXPONENT <VALUE_EXPONENT>;

}

PARAMETERS :


reference variable





column)


<VALUE_CI> Value of the imputation coefficient

< VALUE_CE> Value of the exclusion coefficient

< VALUE_CA> Value of the minimum distance multiplication coefficient

< VALUE_CP> The percent of the highest values that are excluded from calculation

< VALUE_EXPONENT> Value of the exponent to be used in the effect calculation

EX :

DATAOPERATION hbHistorical {

HBHISTORICAL {

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

CI 2;

CE 1;

EXPONENT 1;

}

}

4.9.24. LFS

The LFS1/2 data operation writes in the designated column the result of the Labour Force

Survey 1/2 outlier algorithm. The difference is that LFS1 uses rates and LFS2 uses ranks. In the

definition of the operation only one of the following methods can be specified to be used as

modality values: lookup or explicit values through the keyword RATES.


SYNTAX :

(LFS1 | LFS2) {





FIELDS {

[<COLUMN_NAME> -> <MODALITY>;]+

}

[LOOKUP <LOOKUP_NAME>]

[RATES {

<MODALITY> -> <MODALITY_VALUE>;

}]

}

PARAMETERS :


reference variable





column)

<COLUMN_NAME> Name of the column that has modality

<MODALITY> Modality value

<LOOKUP_NAME> The name of the lookup for rates (LFS1) / ranks (LFS2) – if this

is specified, the explicit rates are ignored.

< MODALITY_VALUE> Value of the modality (rates / ranks). This represents values

to be used as thresholds in the absence of the above lookup.

EX :

DATAOPERATION Lfs {

LFS1{

INPUT input1;



FIELDS {

Columns1 -> 4;


Columns2 -> 10;

Columns3 -> 20;

}

LOOKUP lookup1;

}

}

4.9.25. Sigma Gap

The Sigma Gap data operation writes in the designated column the result of the Sigma Gap

outlier algorithm.

SYNTAX :

SIGMAGAP {







APPROACH <VALUE_APPROACH>;

CM <VALUE_CM>;

[CI <VALUE_CI>;]

[CJ <VALUE_CJ>;]

[CP <VALUE_CP>;]

[UPPER <VALUE_UPPER>;]

[LOWER <VALUE_LOWER>;]



}

PARAMETERS :


reference variable





column)



<VALUE_APPROACH> Value of the approach (“stddev” for standard deviation and

other for difference between two percentiles CI and CJ)

<VALUE_CM> Value of the multiplier coefficient

<VALUE_CI> The first percentile (must be lower than 100)

<VALUE_CJ> The second percentile (must be greater than 0)

< VALUE_CP> Value of the predefined percentage of the excluded values

< VALUE_UPPER> Value of the upper bound

< VALUE_LOWER> Value of the lower bound




EX :

DATAOPERATION sigmaGap {

SIGMAGAP{

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

APPROACH 'stddev';

CM 0.5;

UPPER 120;

LOWER 15;

SEVERITY 'OUTLIER_GAP';

}

}

4.9.26. Urban Method 3

The Urban Method 3 data operation writes in the designated column the result of the Urban

Method 3 outlier algorithm.

SYNTAX :

URBANMETHOD3 {







(STATISTICS <DATASET_REFERENCE_STATISTICS>;)?




}

PARAMETERS :


reference variable





column)




<VALUE_INCREASE> Value of the maximum growth rate


EX :

DATAOPERATION urbanMethod3 {

URBANMETHOD3{

INPUT input1;

FIELD PRICE;



STATISTICS stat1;

INCREASE 0.25;

SEVERITY 'URBAN_METHOD_3';

}

}


4.9.27. Time Series

4.9.27.1. Input Format

For the time series the input format is fixed with name “time_series”. This format can be

loaded from the file time_series.fdl. The output is in the column named “outlier”. This

processing takes as input only series with time components (year, month, value) according to

the tramo documentation. The tramo program and documentation can be found at

http://www.bde.es/servicio/software/econome.htm.

4.9.27.2. Output Format

The only output format is the error one.

4.9.27.3. Timeseries operation

The Timeseries data operation writes in the designated column the result of the Tramo run.

SYNTAX :

TIMESERIES {




ERROR <VALUE_ERROR>;

PARAMETER <VALUE_PARAMETER>;

}

PARAMETERS :


reference variable



<VALUE_ERROR> Name of the error dataset reference variable

<VALUE_PARAMETER> Parameters for the “tramo” to run

The outlier column will contain the outlier value and the type (e.g. Additive (AO) 4.31).

EX :

DATAOPERATION timeSeries {

TIMESERIES{

INPUT timeseries1;

FIELD price;

ERROR error1;

PARAMETER lam=-1,iatip=1,aio=2,va=3.3,noadmiss=1,seats=2;

}

http://www.bde.es/servicio/software/econome.htm


}

4.10. Validation Rules Definition

The validation rules script contains a series of rules that will be applied to the same input

dataset. The rules can be horizontal, vertical or rules that do not apply to data coming from

inside the input dataset.

4.10.1. No Record Rule

No-Record rules are applied without retrieving data from the input dataset. . A no record rule

can either be a computation rule, when the THEN/ELSE keywords are present, or a validation

rule, when the ERRMSG keyword is used. If a validation fails, then an error is reported in the

error log. The main purpose of the no record rules is to define variables with a global scope, but

any function or parameter can be used as well inside the expressions.

SYNTAX:

NORECORD <RULE_NAME> {

[CONDITION < CONDITIONAL_ESLE>;

[ERRMSG <ERROR_MESSAGE> SEVERITY <SEVERITY>

[( [<VARIABLE>]+ )];]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+} ]

| [<ESLE>;]+]?

}

PARAMETERS:

<RULE_NAME> Name of the rule

<CONDITIONAL_ESLE> EDIT SL Expression that returns a Boolean value

<ELSE> EDIT SL Expression

<ERROR_MESSAGE> Message to be displayed in the validation fails

<SEVERITY> Severity of the rule

<VARIABLE> Name of the variable to be reported as part of the Auxiliary Data in the

error log

EX :

NORECORD checkNegativeValues

{

VAR STRING TRANSACTION_TYPE_VAR

:= upper(substring(trim(pTRANSACTYP), 1, 2));

VAR BOOLEAN INTRA := false;

}


4.10.2. Horizontal Rule

Horizontal rules are applied to each row in the input dataset. A horizontal rule can either be a

computation rule, when the THEN/ELSE keywords are present, or a validation rule, when the

ERRMSG keyword is used. If a validation fails, then an error is reported in the error log

indicating the ROW_NUMBER of the current row. Additional fields with their current values

can be reported as part of the Auxiliary Data field inside the error log. In the expression(s) can

appear only the dataset fields, variables, parameters, constants. No transposed values are

allowed in this step.

SYNTAX:

RECORD <RULE_NAME> {

[CONDITION < CONDITIONAL_ESLE>;


[( [<FIELD>]+ )];]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+} ]

| [<ESLE>;]+]?

}

PARAMETERS:




<ERROR_MESSAGE> Message to be displayed in the validation fails


<FIELD> Name of the field or variable to be reported as part of the Auxiliary Data in

the error log

EX :

RECORD checkNegativeValues

{

CONDITION price > 0 AND quantity > 0 AND value > 0;

ERRMSG 'Negative values' SEVERITY 'ERROR' (price,quantity,value);

}

4.10.3. Vertical Rule

Vertical rules are applied on blocks of data from the input dataset, which are called partitions.

The definition of the rule contains also the definition of the partition, indicating the key

columns and the transposition keys to be used. A vertical rule can either be a computation rule,

when the THEN/ELSE keywords are present, or a validation rule, when the ERRMSG keyword


is used. If a validation fails, then an error will be reported in the error log indicating the values

of the current partition. Additional fields with their current value for a specific transposition

value can be reported as part of the Auxiliary Data field inside the error log. The expression(s)

that can appear are the keys (excepting transposed keys), variables, parameters, constants. The

rest of the fields minus keys and transposed keys must be referenced only using the transposed

syntax. The transposed keys cannot be directly referenced.

SYNTAX :

VERTICAL <RULE_NAME> {

[EXPRESSION {

[KEYSET <KEYSET_NAME> | KEYS [<KEY_FIELD >]+ ]+ ;

TRKEYS [<TRANSPOSED_KEY_FIELD_NAME>]*;

CONDITION <CONDITIONAL_ESLE>;


[ ( [<FIELD >]+ ) ]; ]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+}]

]?

}

}

PARAMETERS :


<KEYSET_NAME> Name of a keyset defined at the program level

<KEY_FIELD > Name of a field to be used as a key for the partition

<TRANSPOSED_KEY_FIELD_NAME> Name of the field to be used as a transposed

key



<ERROR_MESSAGE> Message to be displayed in case the validation fails


<FIELD > For Vertical Rules a Field Identifier can be either one of the key fields minus

the transposed key, the value of a field for a specific transposition value or a variable.

EX :

vertical checkGBQ

{

expression

{

keyset keyset1;


trkeys PRODUCT;

condition country='GB' -> quantity['GAS'] > quantity['OIL'];

errmsg 'Incorrect quantity' severity 'WARNING' (country, quantity[‘OIL’]);

}

}

4.10.4. Hierarchical Rule

Hierarchical rules are applied on data cases, comprised of multiple data shapes grouped under a

root node. The definition of the rule contains also the definition of hierarchy, indicating the

keysets template defining the hierarchy and the keysets to be involved in the validation rule. A

hierarchical rule can either be a computation rule, when the THEN/ELSE keywords are present,

or a validation rule, when the ERRMSG keyword is used. If a validation fails, then an error will

be reported in the error log indicating the values of the current case. Additional fields with their

current value for a specific transposition value can be reported as part of the Auxiliary Data

field inside the error log.

SYNTAX :

HIERARCHICAL <RULE_NAME> {

KEYSETSTEMPLATE <TEMPLATE_NAME> [<KEYSET_NAME>]+;

CONDITION <CONDITIONAL_ESLE>;

[ ERRMSG <ERROR_MESSAGE> SEVERITY <SEVERITY>

[ ( [<FIELD_IDENTIFIER>]+ ) ]; ]

| [THEN {[<ESLE>;]+} ]

[ELSE {[<ESLE>;]+} ] ]?

}

PARAMETERS :


<TEMPLATE_NAME> Name of the keysets template defining the hierarchy

<KEYSET_NAME> Name of the keyset defining a hierarchical level

<FIELD_NAME> Name of a field to be used as a key for the partition


<ERROR_MESSAGE> Message to be displayed in case the validation fails


<FIELD_IDENTIFIER> For Hierarchical Rules a Field Identifier can be either one of

the root key fields or the value of a field for a specific transposition value.



EX :

hierarchical rule1

{

KEYSETSTEMPLATE kst1 Country1, City1;

condition NOT isNull(value['COUNTRY1']) ->

value['COUNTRY1'] = sum(value['CITY1'][*][*]);

errmsg 'the sum is not correct1'

severity 'severity' (Country);

}

4.11. Scripting Language Expressions Definition

The SL Expressions are used in order to evaluate validation conditions or to perform

computations inside a program. The values from the Vector Map of the current partition are

used as Expression Fields in order to evaluate an expression.

4.11.1. Expression Fields

A field can be accessed inside the expression script by using the name of the field as a

reference. All the notations are case insensitive (for example if a dataset field is named

“COUNTRY” it can be referenced as “country” or “COUNTRY”).

<VALUE_FIELD>

Ex: country, product

In case of a transposition, each value field will contain a list of values corresponding to the

values of the transposed column. The values corresponding to the keys of the partition can be

used like normal fields as described above. The full list corresponding to all transposition

values for a value field can be used inside the script with the following syntax:

<VALUE_FIELD>[*]

Ex: price[*], quantity[*]

Using the same syntax, the list of values in case of a transposition can be passed as an argument

to a function.

Ex: sum(price[*]), uniqueInList(product[*])

A single value from the list can be used inside the script with the following syntax:

<VALUE_FIELD>[<TRANSPOSITION_VALUE>]

Ex: price[‘OIL’] , quantity[‘GAS’]

4.11.2. Data Types

There are four types of data:

Boolean

Double

Number


String

4.11.3. Operators

There are three types of operators in the SL Expressions:

Boolean operators, used to evaluate expressions into a true/false result:

o OR

o AND

o NOT

o -> (implication)

o = (equals)

o <> (not equals)

o > (greater than)

o < (lower than)

o >= (greater than or equal to)

o <= (lower than or equal to)

Computation operators, used to produce a value result following evaluation:

o + (plus)

o – (minus)

o * (multiply)

o / (divide)

Assignment operator, used to assign a value to an operand

o := (supports assignment to NULL, value := NULL)

Listing operator, used to iterate over lists (list, strToList)

o foreach

EX:

1. numeric list

VERTICAL loopCheckAllValues {

EXPRESSION

{

KEYS COUNTRY, CTYPE, MONTH;

TRKEYS MONTH;

FOREACH cursorValue IN list(10,1000);

CONDITION VALUE[cursorValue] < mean(VALUE[*]) - 200;

ERRMSG "invalid values" SEVERITY "error" (cursorValue);

}

}

2. string list (with variables)

VERTICAL loopCheckAllValues {

EXPRESSION

{

KEYS COUNTRY, CTYPE, MONTH;

TRKEYS MONTH;


VAR TEST := "zxc";

FOREACH cursorValue IN list(TEST);

CONDITION VALUE[cursorValue] < mean(VALUE[*]) - 200;

ERRMSG "invalid values" SEVERITY "error" (cursorValue);

}

}

4.11.4. Functions

These following function calls are supported:

A

abs(Double) – absolute value

abort() or abort(String s) – abort the job execution, adding in the job summary the

information

aggregateToVariable(String reference, String function, String column) – the function

will allow the calculation of the value to be considered as the reference value at the

dataset level (ex: VAR DATASETVALUE := aggregateToVariable(inputData, "sum",

"VALUE"))

appendChar(String originalValue, String added, Integer numberOfAdditions) – add to

the originalValue the added string numberOfAdditions times

ascii(Char) – returns the ASCII code for a character

B

between(Double, Double, Double) – verifies if a number is inside a closed interval

between(Double, Double, Double, Boolean, Boolean) – verifies if a number is inside an

interval allowing the user to specify if the interval is closed or open at each end

between(String, String, String) – same as above

between(String, String, String, Boolean, Boolean) – same as above

C

ceiling(Double) – ceiling for number (Ex: ceiling(3.2) => 4)

chr(Integer) – returns the ASCII character for the ASCII code

chiffrerVal() – DPS specific implementation commonRoot(String s1, String s2) –

returns the number of the contiguous common characters between s1 and s2 counted

from left

commonRoot(String, String) – detects the common part of the arguments starting left to

right – returns the start index of the common string

concatenate(String…) – concatenate Strings

containsChars(String source, String s1) – returns true if the source contains any

character from the s1

containsChars(String source, String char1, String char2, ... ) – returns true if the source

contains any character from the list s1, s2 ...

containsStrings(String source, String s1, String s2, ...) – returns true if the source

contains one of the s1, s2 ...

countMissing(List) – returns the number of null values in the list


count(List) – returns the number of elements in the list

D

datasetExpired(String, Double) – checks if a dataset reference has expired according to

a number of hours parameter

datasetInstanceExpired(String, Double, String) – can be used on the CONDITION of

the import dataset operation, the function checks if a dataset instance has expired

according to a number of hours parameter and adds it to the execution context on the

specified reference

E

excludeValues(String list, args...) – exclude the values from a list

exp(Double)

F

findTranspositionValue(String field, String value, String particularTranspositions...) –

finds the transposition given the value. By default it searches over the entire

transpositions. In the case that particular transposition values are specified, the search is

performed only over these.

firstIndexOf(String toSearch, String searchIn, Double startingFrom) – first occurance of

the toSearch String in the searchIn String

floor(Double) – floor for number (Ex: ceiling(3.2) => 3)

fromVectorMap(datasetReference, thousandsSeparator, decimalSeparator) – initialize

all possible values on the output reference record by extracting based on field names

information from the Vector Map; automatic conversions are performed by the engine if

the column types are different

fromTrValue(datasetReference, thousandsSeparator, decimalSeparator

transpositionPath…) – initialize all possible values on the output reference record by

extracting based on field names information from the Vector Map using the specified

transposition path; automatic conversions are performed by the engine if the column

types are different

G

getErrorCount() – get the number of errors recorded by the current validation step

getFileRepoFolder() – get the root folder of the files repository

getInvalidPartitionData() – get the raw data that failed to be processed correctly by the

import plugin (Ex: the current row in a CSV file)

getPartitionReadError() – get the error that has been recorded by the import plugin

getRowCount(datasetReference) – returns the number of rows for the specified dataset

reference; the function call does not take into account changes that are pending or have

already occurred on the dataset reference in the current step

getReferencePeriods(String periodCode) – receives the reference period code month as

an input in MM format, returns a string containing the list of months included in the

reference period code


Example: Input:"62", Returns: "07;08;09;10;11;12"

getSysdateAsString() – returns the current date in the format “yyyyMMdd”

getReference(String) – returns a String that contains the valid periods (months) for the

code from the input (year + code) with the year prepended

getTranspositionValues(String key) – returns an ordered list of all transposition values

if the key is a valid transposition

getDatasetInstanceName(String datasetReference) – returns the name of the dataset

instance referenced by datasetReference name in this job

getEngineVersion() – returns the engine version (as it is displayed in the first page of

the web client)

getJobName() – returns the name of the current job

I

in(Boolean, Boolean List) – check to see if a value is inside a list

in(Double, Double List)

in(String, String List)

identicalInList(List) – checks that all elements in a list are identical

isIdentical(List, List, …) – returns a boolean indicating if each list contains identical

elements (elements are identical inside a single list) – Ex:

isIdentical(price[*],quantity[*],value[*])

isUnique(List, List, …) – returns a boolean indicating if the combination of elements

from each list for all the index positions are unique– Ex:

isUnique(price[*],quantity[*],value[*])

isTrue(Boolean) – checks if a boolean is true

isNull(Boolean/String/Double) – checks if a value is NULL

inLookup(value1, value2, …, datasetReference, “fieldName1”, “fieldName2”,… ) –

returns a boolean indicating whether or not the value or combination of values is

defined for the fieldname or combination of fieldnames in the specified dataset

inLookupDate(date_input, code_input, datasetReference, ”START_PERIOD_lookup”,

”END_PERIOD_lookup”, ”lookup_code”) - returns true if the code exists in the lookup

in the specified date. The input column date must have the MMYYYY format and the

lookup columns date interval must have the DDMONYYYYHHmmSS. For example:

01OCT2009213030.

L

lastIndexOf(String toSearch, String searchIn, Double startingFrom) – last occurance of

the toSearch String in the searchIn String

length(String) – returns the length of the String

like(String, String) – compares two strings in a SQL manner

list(value1, value2, ...) – creates a list from the individual values. The type of the list

will be given by the type of the first element if all the elements are of the same type, or

string (all the values will be converted to String)

log(Double)

ln(Double)

ltrim(String) – trim left side of String

lower(String) – switch to lower case

left(String S, Double N) – returns the first N characters from S


lpad(String S, Integer I[, String P]) – pads to the right the string S to the length defined

by I. If P is defined, the padding will be performed with P, otherwise with “ “.

M

max(Double List) – maximum value from a list

min(Double List) – minimum value from a list

mean(Double List) – computes the average value not counting null values

missingMean(Double List) – computes the average value counting null values as zero

mod(Double N, Double n) – N%n

N

nvl(Boolean, Boolean) – if first argument value is null return second argument

nvl(Double, Double)

nvl(String, String)

O

occurs(String S, String s) – returns the number of occurences of s in S

P

partitionReadOK() – check if the row has been read successfully or not by the import

plugin

printf(String, String/Double…) – offers the capabilities of the printf method

pow(Double N, Double n) – N**n

R

reset(datasetReference) – reset all values in the specified output reference record;

records which have already been written are not affected

right(String S, Double N) – returns the last N characters from S

round(Double N, Double n) – round N till n decimals (Ex: round(4.46, 1) => 4.5)

rpad(String param, Integer length[, String padding]) – pads to the left the param till the

length. The default padding character is “ “. If the padding is defined, then this is used

padding character.

rtrim(String) – trim right side of String

replaceChars(String input, String mask, String replacement) – returns a positional

replacement of the individual characters from the input with the replacement ruled by

the mask (‘0’ – means no replacement)

reformatDate(String input, format date input, format date output) – transforms the value

of “String input” from “format date input” into “format date output”. In case, the

“String input” does not match the “format date input”, the value returned is null.

Example: reformatDate(“20141026T125252+00”, “yyyyMMdd'T'HHmmssZ”,

“MM.dd.yyyy HH:mm:ss”)= "10.26.2014 14:52:52".

S

skipPartition() – skips the current partition from being processed


str(Double N1, Double N2, Double N3) – Ex: str(30.25, 7, 3)= “_BLANK_30.250“

stringCompliesToFormat(String value, String type, Number length, String

thousandsSeparator, String decimalSeparator, Number precision) – checks if the value

string complies to the required format; only the first 2 parameters are mandatory – Ex:

stringCompliesToFormat(“33”, “NUMBER”)

strToDouble(String value, Number maxLength, String thousandsSeparator, String

decimalsSeparator, Number precision) – convert a String into a Double – return null if

String cannot be converted

strToNumber(String value, Number maxLength, String thousandsSeparator) – convert a

String into a Number – return null if String cannot be converted

strToList(String) – create a list of Strings that contains the individual characters of the

original String

strToList(String input, String separator) – create a list of Strings that contains the result

of splitting the first argument by using the second argument as a separator

substring(String, Double N, Double n) – substring starting from N, counting n

characters

sum(Double List) – sum of elements from list

sqrt(Double) – returns the square root of the value

T

trim(String) – trim String

transcode (“targetField”, lookupReference, “lookupField”, lookupValue) – performs a

lookup based on the specified field and value and returns the value of the target field on

the matching row

transcodeDate(“date”, “code”, lookupReference, “startColumn”, “endColumn”,

“codeColumn”, “valueColumn”) – return the value if the code exists in the lookup in

the specified date with respect to startColumn and endColumn.

EX:

VAR STRING DECL_CODE := nvl(transcodeDate(R1,pDECL,

V_AS_EU_PART_ISOEU_REPORTERS, "START_VAL", "END_VAL", "CODE",

"TARGET") , "");

transcodeStatistics(String indicator, String input, String cols..., String vals...) – returns

the value of the statistic indicator in the specified input dataset with the given values for

the columns. Ex: transcodeStatistics("Q1", "INPUTL1", "PRODUCT", "MONTH",

"OIL", "JAN")

U

upper(String) – switch to upper case

uniqueInList(List) – checks if a list contains unique values

V

validPeriod(String key, String value) – returns true if the value is can be found between

the allowable values given by the key (year, month)

In all functions related to Strings, the first character in the string has the index value 1.

The behaviour of the isIdentical and isUnique functions is further explained by the following

example:

C1 C2 C3

A B X


A B X

A B X

isUnique(C1, C2, C3) will return FALSE

isIdentical(C1, C2, C3) will return TRUE

C1 C2 C3

A B X

A B Y

A B Z

isUnique(C1, C2, C3) will return TRUE

isIdentical(C1, C2, C3) will return FALSE

C1 C2 C3

A B X

A B X

A Z W

isUnique(C1, C2, C3) will return FALSE

isIdentical(C1, C2, C3) will return FALSE

W

write(datasetReference) – write the current record of the output reference to the

database; a new record is written every time this function is called

4.11.5. Variables

The Scripting Language allows the definition and usage of variables inside the expressions.

Variables can have different scopes, depending on where they are defined as follows:

Global Variables: Global variables are defined in NORECORD rules. They are

available throughout the execution of the entire program after their declaration;

Local Variables: Local variables are defined in any rule type other than NORECORD.

They are available only throughout the execution of the current Block.

The same data types as for fields apply also to variables.

A variable can be defined by as follows:

SYNTAX :

VAR <DATA_TYPE>? <VARIABLE_NAME> [:=<INITIAL_VALUE>];

PARAMETERS :

<DATA_TYPE> The data type of the variable

<VARIABLE_NAME> The name of the variable

<INITIAL_VALUE> The initial value of the variable

EX:

VAR STRING STR_VAR := “initial value”;

VAR NUMBER NUM_VAR;

When the INITIAL_VALUE is not specified, the variable takes the following default value, depending

on its type:


Boolean - false

Double – 0 (zero)

Number – 0 (zero)

String – null

When the data type can be deduced from the variable value, the type can be omitted and will be

inferred from the value.

EX:

VAR STR_VAR := “initial value”; - this value will be of type STRING, because there

is no other possibility of assigning this value to other variable types (Boolean, double, ...).

VAR STR_VAR := 30.7; - this will be interpreted as DOUBLE

VAR STR_VAR := 20; - this will be interpreted as DOUBLE (it is more general than

NUMBER)

When assigning values to a variable, it is possible to perform a transposition using the

following syntax:

<VARIABLE>[‘<TRANSPOSITION>’] := <VALUE>;

EX:

VAR STRING STR_VAR;

STR_VAR[“B”] := “B”;

4.12. Comments

The user can insert comments inside the Scripting Language. A line from the file will be

marked as a comment line by using sharp character (‘#’) in the beginning of the line.

EX: #This is a comment line

4.13. Multi-line Strings

The user can use multi-line strings inside the Scripting Language. The lines will be

automatically concatenated and interpreted as a single string.

EX:

DESCRIPTION “This is my description “

“and it spreads on multiple lines “

“because it is very long “;

4.14. New Line Characters

New Line characters can be inserted anywhere inside the script without affecting the behavior

of the language. These characters will be ignored by the Scripting Language.

EX :

FORMAT MyFormat {

DESCRIPTION

“This is my format“;

FIELDS {


Year {

CAPTION Year;

MISSINGVALUES 0,

“NONE”, “NULL”, “ZERO”

,”NOTHING”;

TYPE NUMBER;

LENGTH 4;

}

}

}

4.15. Keyword Abbreviations

The user can use the following abbreviations for the keywords in the syntax:

Keyword Abbreviation

DESCRIPTION DESC

MISSINGVALUES MVALS

MANDATORY MNDTR

PARAMETER PARAM

PARAMETERSET PSET

DATAOPERATION DATA

INPUT IN

OUTPUT OUT

PRECONDITION PCOND

CONDITION COND

VALIDATIONSTEP VSTEP

5. ANNEX I – RESERVED KEYWORDS

The following are reserved as Scripting Language keywords and cannot be used otherwise

inside the program definition:

5.1. Scripting Language Reserved keywords

A

AND, AGGREGATION, ALTER, APPEND, APPROACH,ASC, AUX

B

BETWEEN, BLOCKNAME, BOOLEAN


C

CAPTION, CODELIST, CONDITION, COND, CONSTANT, COPY, COMEXT, CREATE, CI, CE, CA, CM,

CJ, CP, CQU, CQL, CL, CH, CFU, CFL, CONVERSION

D

DATA, DATAOPERATION, DECIMAL_POINT_CHARACTER, DECREASE, DEFAULTVALUE,

DEFINITIONS, DESC, DELETE, DROP, DESCRIPTION, DONT, DOUBLE

E

ELSE, EMPTY, ERRMSG, ERROR, ERRORS, ERRORDATA, EXISTING, EXPONENT, EXPORT,

EXPRESSION, EXTRACT, EXTRACTION

F

FALSE, FIELD, FIELDS, FIELD_DELIMITER, FIELD_SET_TEMPLATE, FILE_TYPE, FOODSAFETY,

FORCE, FOREACH, FORMAT

G

GENERATE_HEADER_LINE, GROWTHRATE

H

HBHISTORICAL, HIERARCHICAL

I

IF, INNER, INCREASE, IN, IDENTICAL, INPUT, IMPORTDATASET, IMPORTPATH

J

JOIN

K

KEYSET, KEYS, KEY, KEYSETS_TEMPLATE

L

LEFT, LENGTH, LOOKUP, LOWER, LFS1, LFS2

M

MANDATORY, MNDTR, MISSING, MISSINGVALUES, MVALS, MERGE

N

NEW, NORECORD, NOT, NUMBER

O

OR, OUTPUT, OUT, OUTLIER, ORDER, OVERWRITE, OUTER

P

PARAMETER, PARAM, PARAMETERSET, PCOND, PRECONDITION, PRNAME, PROGRAM,

PROPERTIES, PROPERTIES_TEMPLATE, PSET, PUBLISH

Q

-

R

R, RANKS, RATES, RENAME, RULES, RECORD, REORDER, RIGHT

S

SELECT, SEVERITY, SIGMAGAP, SORT, SPLIT, SPLIT_INTO_QUARTILE, STATISTICS, STRING,

STEPS, SYNCHRONOUS


T

TEMPORARY, TEXT_QUALIFIER, THOUSAND_DELIMITER, TIMESERIES, TRKEYS, TYPE, THEN,

TO, TRANSPOSE, TRUE

U

UNIQUE, USE, UPDATE, URBANMETHOD3, UPPER, UPPERLOWER

V

VAR, VERSION, VERTICAL, VALIDATION, VALIDDATA, VALIDATIONSTEP, VSTEP, VALID

W

-

X

-

Y

-

Z

-

In addition the following reserved keywords are applicable for the program definition

depending on the deployment of the system.

EDIT 2013 versions are using Oracle 10g products for Server deployments and PostgreSQL

Standalone versions.

5.2. Oracle 10G Reserved Words

This appendix provides a complete list of Oracle reserved words1. In addition to the reserved

words in the table below, Oracle also uses system-generated names beginning with SYS_ for

implicitly generated schema objects and subobjects. Oracle discourages you from using this

prefix in the names you explicitly provide to your schema objects and subobjects to avoid

possible conflicts in name resolution.

Oracle Reserved Words and Keywords

ACCESS ACCOUNT ACTIVATE

ADD ADMIN ADVISE

AFTER ALL ALL_ROWS

ALLOCATE ALTER ANALYZE

AND ANY ARCHIVE

ARCHIVELOG ARRAY AS

1 http://docs.oracle.com/cd/B19306_01/em.102/b40103/app_oracle_reserved_words.htm

http://docs.oracle.com/cd/B19306_01/em.102/b40103/app_oracle_reserved_words.htm



ASC AT AUDIT

AUTHENTICATED AUTHORIZATION AUTOEXTEND

AUTOMATIC BACKUP BECOME

BEFORE BEGIN BETWEEN

BFILE BITMAP BLOB

BLOCK BODY BY

CACHE CACHE_INSTANCES CANCEL

CASCADE CAST CFILE

CHAINED CHANGE CHAR

CHAR_CS CHARACTER CHECK

CHECKPOINT CHOOSE CHUNK

CLEAR CLOB CLONE

CLOSE CLOSE_CACHED_OPEN_CURSORS CLUSTER

COALESCE COLUMN COLUMNS

COMMENT COMMIT COMMITTED

COMPATIBILITY COMPILE COMPLETE

COMPOSITE_LIMIT COMPRESS COMPUTE

CONNECT CONNECT_TIME CONSTRAINT

CONSTRAINTS CONTENTS CONTINUE

CONTROLFILE CONVERT COST

CPU_PER_CALL CPU_PER_SESSION CREATE

CURRENT CURRENT_SCHEMA CURREN_USER

CURSOR CYCLE

DANGLING DATABASE DATAFILE

DATAFILES DATAOBJNO DATE

DBA DBHIGH DBLOW



DBMAC DEALLOCATE DEBUG

DEC DECIMAL DECLARE

DEFAULT DEFERRABLE DEFERRED

DEGREE DELETE DEREF

DESC DIRECTORY DISABLE

DISCONNECT DISMOUNT DISTINCT

DISTRIBUTED DML DOUBLE

DROP DUMP EACH

ELSE ENABLE END

ENFORCE ENTRY ESCAPE

EXCEPT EXCEPTIONS EXCHANGE

EXCLUDING EXCLUSIVE EXECUTE

EXISTS EXPIRE EXPLAIN

EXTENT EXTENTS EXTERNALLY

FAILED_LOGIN_ATTEMPTS FALSE FAST

FILE FIRST_ROWS FLAGGER

FLOAT FLOB FLUSH

FOR FORCE FOREIGN

FREELIST FREELISTS FROM

FULL FUNCTION GLOBAL

GLOBALLY GLOBAL_NAME GRANT

GROUP GROUPS HASH

HASHKEYS HAVING HEADER

HEAP IDENTIFIED IDGENERATORS

IDLE_TIME IF IMMEDIATE

IN INCLUDING INCREMENT



INDEX INDEXED INDEXES

INDICATOR IND_PARTITION INITIAL

INITIALLY INITRANS INSERT

INSTANCE INSTANCES INSTEAD

INT INTEGER INTERMEDIATE

INTERSECT INTO IS

ISOLATION ISOLATION_LEVEL KEEP

KEY KILL LABEL

LAYER LESS LEVEL

LIBRARY LIKE LIMIT

LINK LIST LOB

LOCAL LOCK LOCKED

LOG LOGFILE LOGGING

LOGICAL_READS_PER_CALL LOGICAL_READS_PER_SESSION LONG

MANAGE MASTER MAX

MAXARCHLOGS MAXDATAFILES MAXEXTENTS

MAXINSTANCES MAXLOGFILES MAXLOGHISTORY

MAXLOGMEMBERS MAXSIZE MAXTRANS

MAXVALUE MIN MEMBER

MINIMUM MINEXTENTS MINUS

MINVALUE MLSLABEL MLS_LABEL_FORMAT

MODE MODIFY MOUNT

MOVE MTS_DISPATCHERS MULTISET

NATIONAL NCHAR NCHAR_CS

NCLOB NEEDED NESTED

NETWORK NEW NEXT



NOARCHIVELOG NOAUDIT NOCACHE

NOCOMPRESS NOCYCLE NOFORCE

NOLOGGING NOMAXVALUE NOMINVALUE

NONE NOORDER NOOVERRIDE

NOPARALLEL NOPARALLEL NOREVERSE

NORMAL NOSORT NOT

NOTHING NOWAIT NULL

NUMBER NUMERIC NVARCHAR2

OBJECT OBJNO OBJNO_REUSE

OF OFF OFFLINE

OID OIDINDEX OLD

ON ONLINE ONLY

OPCODE OPEN OPTIMAL

OPTIMIZER_GOAL OPTION OR

ORDER ORGANIZATION OSLABEL

OVERFLOW OWN PACKAGE

PARALLEL PARTITION PASSWORD

PASSWORD_GRACE_TIME PASSWORD_LIFE_TIME PASSWORD_LOCK_TIME

PASSWORD_REUSE_MAX PASSWORD_REUSE_TIME PASSWORD_VERIFY_FUNCTION

PCTFREE PCTINCREASE PCTTHRESHOLD

PCTUSED PCTVERSION PERCENT

PERMANENT PLAN PLSQL_DEBUG

POST_TRANSACTION PRECISION PRESERVE

PRIMARY PRIOR PRIVATE

PRIVATE_SGA PRIVILEGE PRIVILEGES

PROCEDURE PROFILE PUBLIC



PURGE QUEUE QUOTA

RANGE RAW RBA

READ READUP REAL

REBUILD RECOVER RECOVERABLE

RECOVERY REF REFERENCES

REFERENCING REFRESH RENAME

REPLACE RESET RESETLOGS

RESIZE RESOURCE RESTRICTED

RETURN RETURNING REUSE

REVERSE REVOKE ROLE

ROLES ROLLBACK ROW

ROWID ROWNUM ROWS

RULE SAMPLE SAVEPOINT

SB4 SCAN_INSTANCES SCHEMA

SCN SCOPE SD_ALL

SD_INHIBIT SD_SHOW SEGMENT

SEG_BLOCK SEG_FILE SELECT

SEQUENCE SERIALIZABLE SESSION

SESSION_CACHED_CURSORS SESSIONS_PER_USER SET

SHARE SHARED SHARED_POOL

SHRINK SIZE SKIP

SKIP_UNUSABLE_INDEXES SMALLINT SNAPSHOT

SOME SORT SPECIFICATION

SPLIT SQL_TRACE STANDBY

START STATEMENT_ID STATISTICS

STOP STORAGE STORE



STRUCTURE SUCCESSFUL SWITCH

SYS_OP_ENFORCE_NOT_NULL$ SYS_OP_NTCIMG$ SYNONYM

SYSDATE SYSDBA SYSOPER

SYSTEM TABLE TABLES

TABLESPACE TABLESPACE_NO TABNO

TEMPORARY THAN THE

THEN THREAD TIMESTAMP

TIME TO TOPLEVEL

TRACE TRACING TRANSACTION

TRANSITIONAL TRIGGER TRIGGERS

TRUE TRUNCATE TX

TYPE UB2 UBA

UID UNARCHIVED UNDO

UNION UNIQUE UNLIMITED

UNLOCK UNRECOVERABLE UNTIL

UNUSABLE UNUSED UPDATABLE

UPDATE USAGE USE

USER USING VALIDATE

VALIDATION VALUE VALUES

VARCHAR VARCHAR2 VARYING

VIEW WHEN WHENEVER

WHERE WITH WITHOUT

WORK WRITE WRITEDOWN

WRITEUP XID YEAR


5.3. PostgreSQL v8 Reserved Words

This appendix provides a complete list of PostgreSQL reserved words2. As a general rule, if

you get spurious parser errors for commands that contain any of the listed key words as an

identifier you should try to quote the identifier to see if the problem goes away. It is important

to understand before studying the following table that the fact that a key word is not reserved in

PostgreSQL does not mean that the feature related to the word is not implemented. Conversely,

the presence of a key word does not indicate the existence of a feature.

Key Word PostgreSQL

ALL reserved

ANALYSE reserved

ANALYZE reserved

AND reserved

ANY reserved

ARRAY reserved

AS reserved

ASC reserved

ASYMMETRIC reserved

AUTHORIZATION reserved (can be function or type)

BETWEEN reserved (can be function or type)

BIGINT non-reserved (cannot be function or type)

BINARY reserved (can be function or type)

BIT non-reserved (cannot be function or type)

BOOLEAN non-reserved (cannot be function or type)

BOTH reserved

CASE reserved

CAST reserved

CHAR non-reserved (cannot be function or type)

CHARACTER non-reserved (cannot be function or type)

CHECK reserved

COALESCE non-reserved (cannot be function or type)

COLLATE reserved

COLUMN reserved

CONSTRAINT reserved

CREATE reserved

CROSS reserved (can be function or type)

CURRENT_DATE reserved

CURRENT_ROLE reserved

2 http://www.postgresql.org/docs/8.3/static/sql-keywords-appendix.html

http://www.postgresql.org/docs/8.3/static/sql-keywords-appendix.html


CURRENT_TIME reserved

CURRENT_TIMESTAMP reserved

CURRENT_USER reserved

DEC non-reserved (cannot be function or type)

DECIMAL non-reserved (cannot be function or type)

DEFAULT reserved

DEFERRABLE reserved

DESC reserved

DISTINCT reserved

DO reserved

ELSE reserved

END reserved

EXCEPT reserved

EXISTS non-reserved (cannot be function or type)

EXTRACT non-reserved (cannot be function or type)

FALSE reserved

FLOAT non-reserved (cannot be function or type)

FOR reserved

FOREIGN reserved

FREEZE reserved (can be function or type)

FROM reserved

FULL reserved (can be function or type)

GRANT reserved

GREATEST non-reserved (cannot be function or type)

GROUP reserved

HAVING reserved

ILIKE reserved (can be function or type)

IN reserved

INITIALLY reserved

INNER reserved (can be function or type)

INOUT non-reserved (cannot be function or type)

INT non-reserved (cannot be function or type)

INTEGER non-reserved (cannot be function or type)

INTERSECT reserved

INTERVAL non-reserved (cannot be function or type)

INTO reserved

IS reserved (can be function or type)

ISNULL reserved (can be function or type)

JOIN reserved (can be function or type)

LEADING reserved

LEAST non-reserved (cannot be function or type)


LEFT reserved (can be function or type)

LIKE reserved (can be function or type)

LIMIT reserved

LOCALTIME reserved

LOCALTIMESTAMP reserved

NATIONAL non-reserved (cannot be function or type)

NATURAL reserved (can be function or type)

NCHAR non-reserved (cannot be function or type)

NEW reserved

NONE non-reserved (cannot be function or type)

NOT reserved

NOTNULL reserved (can be function or type)

NULL reserved

NULLIF non-reserved (cannot be function or type)

NUMERIC non-reserved (cannot be function or type)

OFF reserved

OFFSET reserved

OLD reserved

ON reserved

ONLY reserved

OR reserved

ORDER reserved

OUT non-reserved (cannot be function or type)

OUTER reserved (can be function or type)

OVERLAPS reserved (can be function or type)

OVERLAY non-reserved (cannot be function or type)

PLACING reserved

POSITION non-reserved (cannot be function or type)

PRECISION non-reserved (cannot be function or type)

PRIMARY reserved

REAL non-reserved (cannot be function or type)

REFERENCES reserved

RETURNING reserved

RIGHT reserved (can be function or type)

ROW non-reserved (cannot be function or type)

SELECT reserved

SESSION_USER reserved

SETOF non-reserved (cannot be function or type)

SIMILAR reserved (can be function or type)

SMALLINT non-reserved (cannot be function or type)

SOME reserved


SUBSTRING non-reserved (cannot be function or type)

SYMMETRIC reserved

TABLE reserved

THEN reserved

TIME non-reserved (cannot be function or type)

TIMESTAMP non-reserved (cannot be function or type)

TO reserved

TRAILING reserved

TREAT non-reserved (cannot be function or type)

TRIM non-reserved (cannot be function or type)

TRUE reserved

UNION reserved

UNIQUE reserved

USER reserved

USING reserved

VALUES non-reserved (cannot be function or type)

VARCHAR non-reserved (cannot be function or type)

VERBOSE reserved (can be function or type)

WHEN reserved

WHERE reserved

WITH reserved

XMLATTRIBUTES non-reserved (cannot be function or type)

XMLCONCAT non-reserved (cannot be function or type)

XMLELEMENT non-reserved (cannot be function or type)

XMLFOREST non-reserved (cannot be function or type)

XMLPARSE non-reserved (cannot be function or type)

XMLPI non-reserved (cannot be function or type)

XMLROOT non-reserved (cannot be function or type)

XMLSERIALIZE non-reserved (cannot be function or type)

edit scripting language specification'...01.20 25/04/2013 added xtnet/edit domain related...

Documents