PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 5.7.1

Pivotal Greenplum DatabaseDocumentationRev: A03

© 2018 Pivotal Software, Inc.

Copyright OpenTopic

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2018 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised June 2018 (5.7.1)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents OpenTopic

3

Contents

Chapter 2: Pivotal Greenplum 5.7.1 Release Notes............................... 14Welcome to Pivotal Greenplum 5.7.1............................................................................................... 15Experimental Features...................................................................................................................... 16Differences Compared to Open Source Greenplum Database.........................................................17Supported Platforms..........................................................................................................................18

Veritas NetBackup.................................................................................................................. 19Supported Platform Notes...................................................................................................... 19

Pivotal Greenplum Tools and Extensions Compatibility................................................................... 21Client Tools.............................................................................................................................21Extensions...............................................................................................................................22Pivotal Greenplum Data Connectors......................................................................................22Pivotal GPText Compatibility.................................................................................................. 23Pivotal Greenplum Command Center.................................................................................... 23

Hadoop Distribution Compatibility..................................................................................................... 24Upgrading to Greenplum Database 5.7.1......................................................................................... 25

Upgrading from 5.x to 5.7.1................................................................................................... 26Troubleshooting a Failed Upgrade.........................................................................................27

Migrating Data to Pivotal Greenplum 5.x..........................................................................................28Pivotal Greenplum on DCA Systems................................................................................................29

Installing the Pivotal Greenplum 5.7.1 Software Binaries on DCA Systems.......................... 29Upgrading from 5.x to 5.7.1 on DCA Systems.......................................................................29

Resolved Issues................................................................................................................................ 31Known Issues and Limitations.......................................................................................................... 33Update for gp_toolkit.gp_bloat_diag Issue........................................................................................40

Chapter 3: Pivotal Greenplum 5.7.0 Release Notes............................... 42Welcome to Pivotal Greenplum 5.7.0............................................................................................... 43New Feature...................................................................................................................................... 44

Query Optimization Enhancement..........................................................................................44Changed Features.............................................................................................................................45Experimental Features...................................................................................................................... 46Differences Compared to Open Source Greenplum Database.........................................................47Supported Platforms..........................................................................................................................48

Veritas NetBackup.................................................................................................................. 49Supported Platform Notes...................................................................................................... 49

Pivotal Greenplum Tools and Extensions Compatibility................................................................... 51Client Tools.............................................................................................................................51Extensions...............................................................................................................................52Pivotal Greenplum Data Connectors......................................................................................52Pivotal GPText Compatibility.................................................................................................. 53Pivotal Greenplum Command Center.................................................................................... 53

Hadoop Distribution Compatibility..................................................................................................... 54Upgrading to Greenplum Database 5.7.0......................................................................................... 55

Upgrading from 5.x to 5.7.0................................................................................................... 56Troubleshooting a Failed Upgrade.........................................................................................57

Migrating Data to Pivotal Greenplum 5.x..........................................................................................58Pivotal Greenplum on DCA Systems................................................................................................59Resolved Issues................................................................................................................................ 60

Contents OpenTopic

4

Known Issues and Limitations.......................................................................................................... 62Update for gp_toolkit.gp_bloat_diag Issue........................................................................................69

Chapter 5: Greenplum Database Installation Guide...............................71Introduction to Greenplum.................................................................................................................72

The Greenplum Master...........................................................................................................72The Segments........................................................................................................................ 73The Interconnect.....................................................................................................................74ETL Hosts for Data Loading.................................................................................................. 75Greenplum Performance Monitoring...................................................................................... 76

Estimating Storage Capacity............................................................................................................. 77Calculating Usable Disk Capacity.......................................................................................... 77Calculating User Data Size.................................................................................................... 77Calculating Space Requirements for Metadata and Logs......................................................78

Configuring Your Systems and Installing Greenplum....................................................................... 79System Requirements.............................................................................................................79Setting the Greenplum Recommended OS Parameters........................................................ 81Creating the Greenplum Database Administrative User Account.......................................... 86Installing the Greenplum Database Software.........................................................................86Installing and Configuring Greenplum on all Hosts................................................................88Installing Oracle Compatibility Functions............................................................................... 89Installing Optional Modules.................................................................................................... 90Installing Greenplum Database Extensions........................................................................... 90Installing and Configuring the Greenplum Platform Extension Framework (PXF)..................91Creating the Data Storage Areas...........................................................................................91Synchronizing System Clocks................................................................................................ 92Enabling iptables.................................................................................................................... 92Amazon EC2 Configuration (Amazon Web Services)............................................................95Next Steps............................................................................................................................ 100

Installing the Data Science Packages............................................................................................ 101Python Data Science Module Package................................................................................101R Data Science Library Package......................................................................................... 103

Validating Your Systems................................................................................................................. 106Validating OS Settings..........................................................................................................106Validating Hardware Performance........................................................................................106Validating Disk I/O and Memory Bandwidth.........................................................................107

Configuring Timezone and Localization Settings............................................................................109Configuring the Timezone.................................................................................................... 109About Locale Support in Greenplum Database................................................................... 109Character Set Support..........................................................................................................111Setting the Character Set.....................................................................................................113Character Set Conversion Between Server and Client........................................................114

Initializing a Greenplum Database System..................................................................................... 117Overview............................................................................................................................... 117Initializing Greenplum Database...........................................................................................117Setting Greenplum Environment Variables.......................................................................... 120Next Steps............................................................................................................................ 121

About Implicit Text Casting in Greenplum Database......................................................................122Workaround: Manually Creating Missing Operators.............................................................123

Installation Management Utilities.....................................................................................................126Greenplum Environment Variables................................................................................................. 127

Required Environment Variables..........................................................................................127Optional Environment Variables........................................................................................... 127

Contents OpenTopic

5

Chapter 7: Greenplum Database Administrator Guide........................ 129Greenplum Database Concepts...................................................................................................... 130

About the Greenplum Architecture....................................................................................... 130About Management and Monitoring Utilities........................................................................ 132About Concurrency Control in Greenplum Database...........................................................133About Parallel Data Loading................................................................................................ 141About Redundancy and Failover in Greenplum Database...................................................142About Database Statistics in Greenplum Database............................................................. 144

Managing a Greenplum System..................................................................................................... 151About the Greenplum Database Release Version Number................................................. 151Starting and Stopping Greenplum Database....................................................................... 151Accessing the Database.......................................................................................................154Configuring the Greenplum Database System.....................................................................164Enabling High Availability and Data Consistency Features................................................. 174Backing Up and Restoring Databases................................................................................. 193Expanding a Greenplum System..........................................................................................230Migrating Data with gptransfer............................................................................................. 245Monitoring a Greenplum System..........................................................................................251Routine System Maintenance Tasks....................................................................................271Recommended Monitoring and Maintenance Tasks............................................................ 275

Managing Greenplum Database Access.........................................................................................283Configuring Client Authentication......................................................................................... 283Managing Roles and Privileges............................................................................................309

Defining Database Objects..............................................................................................................316Creating and Managing Databases......................................................................................316Creating and Managing Tablespaces...................................................................................318Creating and Managing Schemas........................................................................................321Creating and Managing Tables............................................................................................ 322Choosing the Table Storage Model..................................................................................... 325Partitioning Large Tables......................................................................................................336Creating and Using Sequences........................................................................................... 348Using Indexes in Greenplum Database............................................................................... 351Creating and Managing Views............................................................................................. 354

Distribution and Skew..................................................................................................................... 355Local (Co-located) Joins.......................................................................................................355Data Skew............................................................................................................................ 355Processing Skew.................................................................................................................. 356

Inserting, Updating, and Deleting Data...........................................................................................359About Concurrency Control in Greenplum Database...........................................................359Inserting Rows...................................................................................................................... 360Updating Existing Rows........................................................................................................361Deleting Rows.......................................................................................................................361Working With Transactions...................................................................................................361Vacuuming the Database..................................................................................................... 363

Querying Data................................................................................................................................. 364About Greenplum Query Processing....................................................................................364About GPORCA....................................................................................................................367Defining Queries................................................................................................................... 380WITH Queries (Common Table Expressions)......................................................................389Using Functions and Operators............................................................................................392Working with JSON Data..................................................................................................... 401Working with XML Data........................................................................................................405Query Performance.............................................................................................................. 417Managing Spill Files Generated by Queries........................................................................ 417

Contents OpenTopic

6

Query Profiling...................................................................................................................... 417Working with External Data.............................................................................................................423

Defining External Tables...................................................................................................... 423Accessing External Data with PXF...................................................................................... 441Accessing HDFS Data with gphdfs...................................................................................... 441Using the Greenplum Parallel File Server (gpfdist)..............................................................464

Loading and Unloading Data.......................................................................................................... 468Loading Data Using an External Table................................................................................ 469Loading and Writing Non-HDFS Custom Data.................................................................... 469Handling Load Errors............................................................................................................472Loading Data with gpload.....................................................................................................474Transforming External Data with gpfdist and gpload........................................................... 475Loading Data with COPY..................................................................................................... 485Running COPY in Single Row Error Isolation Mode............................................................485Optimizing Data Load and Query Performance................................................................... 485Unloading Data from Greenplum Database......................................................................... 486Formatting Data Files........................................................................................................... 488Example Custom Data Access Protocol.............................................................................. 491

Managing Performance................................................................................................................... 498Defining Database Performance.......................................................................................... 498Common Causes of Performance Issues............................................................................ 499Greenplum Database Memory Overview............................................................................. 502Managing Resources............................................................................................................505Investigating a Performance Problem.................................................................................. 529

Chapter 9: Greenplum Database Security Configuration Guide......... 533Securing the Database....................................................................................................................534Greenplum Database Ports and Protocols..................................................................................... 535Configuring Client Authentication.................................................................................................... 539

Allowing Connections to Greenplum Database....................................................................539Editing the pg_hba.conf File.................................................................................................540Authentication Methods........................................................................................................ 541SSL Client Authentication.....................................................................................................544PAM Based Authentication...................................................................................................546Radius Authentication...........................................................................................................546Limiting Concurrent Connections......................................................................................... 546Encrypting Client/Server Connections..................................................................................547

Configuring Database Authorization................................................................................................549Access Permissions and Roles............................................................................................549Managing Object Privileges..................................................................................................549Using SSH-256 Encryption...................................................................................................550Restricting Access by Time..................................................................................................552Dropping a Time-based Restriction.................................................................................... 554

Greenplum Command Center Security........................................................................................... 555Auditing............................................................................................................................................ 558Encrypting Data and Database Connections.................................................................................. 563

Encrypting gpfdist Connections............................................................................................ 563Encrypting Data at Rest with pgcrypto.................................................................................564

Enabling gphdfs Authentication with a Kerberos-secured Hadoop Cluster.....................................572Prerequisites......................................................................................................................... 572Configuring the Greenplum Cluster......................................................................................572Creating and Installing Keytab Files.................................................................................... 573Configuring gphdfs for Kerberos.......................................................................................... 574Testing Greenplum Database Access to HDFS...................................................................575Troubleshooting HDFS with Kerberos..................................................................................576

Contents OpenTopic

7

Security Best Practices................................................................................................................... 578

Chapter 11: Greenplum Database Best Practices................................ 582Best Practices Summary.................................................................................................................583System Configuration...................................................................................................................... 589Schema Design............................................................................................................................... 594

Data Types........................................................................................................................... 594Storage Model...................................................................................................................... 594Compression......................................................................................................................... 595Distributions.......................................................................................................................... 596Partitioning............................................................................................................................ 599Indexes..................................................................................................................................601Column Sequence and Byte Alignment............................................................................... 601

Memory and Resource Management with Resource Queues........................................................ 603System Monitoring and Maintenance..............................................................................................607

Monitoring............................................................................................................................. 607Updating Statistics with ANALYZE.......................................................................................608Managing Bloat in the Database..........................................................................................609Monitoring Greenplum Database Log Files..........................................................................613

Loading Data................................................................................................................................... 615INSERT Statement with Column Values..............................................................................615COPY Statement.................................................................................................................. 615External Tables.....................................................................................................................615External Tables with Gpfdist................................................................................................ 615Gpload...................................................................................................................................616Best Practices.......................................................................................................................617

Migrating Data with Gptransfer....................................................................................................... 618Security............................................................................................................................................ 623Encrypting Data and Database Connections.................................................................................. 626Accessing a Kerberized Hadoop Cluster........................................................................................ 635

Prerequisites......................................................................................................................... 635Configuring the Greenplum Cluster......................................................................................635Creating and Installing Keytab Files.................................................................................... 636Configuring gphdfs for Kerberos.......................................................................................... 637Testing Greenplum Database Access to HDFS...................................................................638Troubleshooting HDFS with Kerberos..................................................................................639

Tuning SQL Queries....................................................................................................................... 641How to Generate Explain Plans........................................................................................... 641How to Read Explain Plans................................................................................................. 641Optimizing Greenplum Queries............................................................................................ 643

High Availability............................................................................................................................... 645Disk Storage......................................................................................................................... 645Master Mirroring....................................................................................................................645Segment Mirroring................................................................................................................ 646Dual Clusters........................................................................................................................ 647Backup and Restore.............................................................................................................647Detecting Failed Master and Segment Instances................................................................ 648Segment Mirroring Configuration..........................................................................................649

Chapter 13: Greenplum Database Utility Guide................................... 655Management Utility Reference........................................................................................................ 656

Backend Server Programs................................................................................................... 657analyzedb..............................................................................................................................658gpactivatestandby................................................................................................................. 662

Contents OpenTopic

8

gpaddmirrors......................................................................................................................... 664gpbackup...............................................................................................................................667gpcheck.................................................................................................................................671gpcheckcat............................................................................................................................ 673gpcheckperf...........................................................................................................................676gpconfig.................................................................................................................................679gpcrondump.......................................................................................................................... 682gpdbrestore........................................................................................................................... 696gpdeletesystem..................................................................................................................... 703gpexpand.............................................................................................................................. 704gpfdist....................................................................................................................................707gpfilespace............................................................................................................................ 710gpinitstandby......................................................................................................................... 713gpinitsystem.......................................................................................................................... 715gpload................................................................................................................................... 722gplogfilter...............................................................................................................................732gpmapreduce........................................................................................................................ 735gpmfr..................................................................................................................................... 737gpmovemirrors...................................................................................................................... 740gpperfmon_install..................................................................................................................741gppkg.................................................................................................................................... 746gprecoverseg........................................................................................................................ 747gpreload................................................................................................................................ 752gprestore............................................................................................................................... 754gpscp.....................................................................................................................................757gpseginstall........................................................................................................................... 759gpssh.....................................................................................................................................761gpssh-exkeys........................................................................................................................ 764gpstart................................................................................................................................... 766gpstate.................................................................................................................................. 768gpstop................................................................................................................................... 772gpsys1...................................................................................................................................774gptransfer.............................................................................................................................. 775pgbouncer............................................................................................................................. 787

Client Utility Reference....................................................................................................................807Client Utility Summary.......................................................................................................... 807

Additional Supplied Modules........................................................................................................... 857citext Data Type................................................................................................................... 857dblink Functions....................................................................................................................859hstore Functions................................................................................................................... 860Oracle Compatibility Functions.............................................................................................863passwordcheck..................................................................................................................... 884

Chapter 15: Greenplum Database Reference Guide............................ 886SQL Command Reference..............................................................................................................887

SQL Syntax Summary..........................................................................................................889ABORT..................................................................................................................................916ALTER AGGREGATE...........................................................................................................917ALTER CONVERSION......................................................................................................... 918ALTER DATABASE.............................................................................................................. 919ALTER DOMAIN...................................................................................................................920ALTER EXTENSION.............................................................................................................922ALTER EXTERNAL TABLE..................................................................................................925ALTER FILESPACE............................................................................................................. 926ALTER FUNCTION...............................................................................................................927

Contents OpenTopic

9

ALTER GROUP.................................................................................................................... 929ALTER INDEX...................................................................................................................... 930ALTER LANGUAGE............................................................................................................. 932ALTER OPERATOR............................................................................................................. 932ALTER OPERATOR CLASS................................................................................................ 933ALTER OPERATOR FAMILY...............................................................................................934ALTER PROTOCOL............................................................................................................. 936ALTER RESOURCE GROUP.............................................................................................. 937ALTER RESOURCE QUEUE...............................................................................................939ALTER ROLE....................................................................................................................... 941ALTER SCHEMA..................................................................................................................945ALTER SEQUENCE............................................................................................................. 946ALTER TABLE......................................................................................................................948ALTER TABLESPACE..........................................................................................................958ALTER TYPE........................................................................................................................959ALTER USER....................................................................................................................... 959ALTER VIEW........................................................................................................................ 960ANALYZE..............................................................................................................................961BEGIN................................................................................................................................... 964CHECKPOINT.......................................................................................................................966CLOSE.................................................................................................................................. 966CLUSTER............................................................................................................................. 967COMMENT............................................................................................................................968COMMIT................................................................................................................................970COPY.................................................................................................................................... 971CREATE AGGREGATE........................................................................................................981CREATE CAST.....................................................................................................................985CREATE CONVERSION...................................................................................................... 988CREATE DATABASE........................................................................................................... 989CREATE DOMAIN................................................................................................................990CREATE EXTENSION..........................................................................................................992CREATE EXTERNAL TABLE...............................................................................................993CREATE FUNCTION..........................................................................................................1002CREATE GROUP............................................................................................................... 1008CREATE INDEX................................................................................................................. 1009CREATE LANGUAGE........................................................................................................ 1012CREATE OPERATOR........................................................................................................ 1014CREATE OPERATOR CLASS........................................................................................... 1018CREATE OPERATOR FAMILY..........................................................................................1021CREATE PROTOCOL........................................................................................................ 1022CREATE RESOURCE GROUP......................................................................................... 1023CREATE RESOURCE QUEUE..........................................................................................1025CREATE ROLE.................................................................................................................. 1028CREATE RULE...................................................................................................................1033CREATE SCHEMA.............................................................................................................1034CREATE SEQUENCE........................................................................................................ 1036CREATE TABLE.................................................................................................................1039CREATE TABLE AS...........................................................................................................1050CREATE TABLESPACE.....................................................................................................1053CREATE TYPE...................................................................................................................1054CREATE USER.................................................................................................................. 1060CREATE VIEW................................................................................................................... 1060DEALLOCATE.................................................................................................................... 1062DECLARE........................................................................................................................... 1063DELETE.............................................................................................................................. 1065DISCARD............................................................................................................................ 1067

Contents OpenTopic

10

DO.......................................................................................................................................1068DROP AGGREGATE..........................................................................................................1070DROP CAST.......................................................................................................................1071DROP CONVERSION........................................................................................................ 1071DROP DATABASE............................................................................................................. 1072DROP DOMAIN.................................................................................................................. 1073DROP EXTENSION............................................................................................................1073DROP EXTERNAL TABLE.................................................................................................1074DROP FILESPACE.............................................................................................................1075DROP FUNCTION..............................................................................................................1076DROP GROUP................................................................................................................... 1077DROP INDEX..................................................................................................................... 1077DROP LANGUAGE.............................................................................................................1078DROP OPERATOR............................................................................................................ 1079DROP OPERATOR CLASS............................................................................................... 1080DROP OPERATOR FAMILY.............................................................................................. 1080DROP OWNED...................................................................................................................1081DROP PROTOCOL............................................................................................................ 1082DROP RESOURCE GROUP..............................................................................................1083DROP RESOURCE QUEUE.............................................................................................. 1084DROP ROLE.......................................................................................................................1085DROP RULE.......................................................................................................................1086DROP SCHEMA................................................................................................................. 1087DROP SEQUENCE............................................................................................................ 1087DROP TABLE..................................................................................................................... 1088DROP TABLESPACE.........................................................................................................1089DROP TYPE....................................................................................................................... 1089DROP USER...................................................................................................................... 1090DROP VIEW....................................................................................................................... 1091END.....................................................................................................................................1091EXECUTE........................................................................................................................... 1092EXPLAIN............................................................................................................................. 1093FETCH................................................................................................................................ 1095GRANT................................................................................................................................1098INSERT............................................................................................................................... 1102LOAD.................................................................................................................................. 1104LOCK.................................................................................................................................. 1104MOVE..................................................................................................................................1107PREPARE........................................................................................................................... 1108REASSIGN OWNED...........................................................................................................1110REINDEX............................................................................................................................ 1111RELEASE SAVEPOINT......................................................................................................1112RESET................................................................................................................................ 1113REVOKE............................................................................................................................. 1114ROLLBACK......................................................................................................................... 1116ROLLBACK TO SAVEPOINT.............................................................................................1116SAVEPOINT........................................................................................................................1117SELECT.............................................................................................................................. 1119SELECT INTO.................................................................................................................... 1133SET..................................................................................................................................... 1134SET ROLE.......................................................................................................................... 1136SET SESSION AUTHORIZATION..................................................................................... 1137SET TRANSACTION.......................................................................................................... 1138SHOW................................................................................................................................. 1140START TRANSACTION..................................................................................................... 1141TRUNCATE.........................................................................................................................1142

Contents OpenTopic

11

UPDATE..............................................................................................................................1143VACUUM.............................................................................................................................1146VALUES.............................................................................................................................. 1149

SQL 2008 Optional Feature Compliance......................................................................................1151Greenplum Environment Variables............................................................................................... 1180

Required Environment Variables........................................................................................1180Optional Environment Variables......................................................................................... 1180

System Catalog Reference........................................................................................................... 1182System Tables.................................................................................................................... 1182System Views..................................................................................................................... 1183System Catalogs Definitions...............................................................................................1184

The gp_toolkit Administrative Schema..........................................................................................1269Checking for Tables that Need Routine Maintenance........................................................1269Checking for Locks.............................................................................................................1270Checking Append-Optimized Tables.................................................................................. 1272Viewing Greenplum Database Server Log Files................................................................ 1276Checking Server Configuration Files..................................................................................1279Checking for Failed Segments........................................................................................... 1280Checking Resource Group Activity and Status.................................................................. 1281Checking Resource Queue Activity and Status................................................................. 1283Checking Query Disk Spill Space Usage...........................................................................1285Viewing Users and Groups (Roles)....................................................................................1287Checking Database Object Sizes and Disk Space............................................................ 1288Checking for Uneven Data Distribution.............................................................................. 1292

The gpperfmon Database..............................................................................................................1293database_*.........................................................................................................................1295diskspace_*....................................................................................................................... 1296interface_stats_*................................................................................................................ 1296log_alert_*..........................................................................................................................1298queries_*............................................................................................................................. 1299segment_*..........................................................................................................................1301socket_stats_*.....................................................................................................................1302system_*............................................................................................................................. 1303dynamic_memory_info........................................................................................................ 1305memory_info...................................................................................................................... 1305

Greenplum Database Data Types.................................................................................................1307Character Set Support...................................................................................................................1311

Setting the Character Set...................................................................................................1313Character Set Conversion Between Server and Client...................................................... 1313

Server Configuration Parameters..................................................................................................1316Parameter Types and Values.............................................................................................1316Setting Parameters............................................................................................................. 1316Parameter Categories.........................................................................................................1317Configuration Parameters...................................................................................................1327

Summary of Built-in Functions...................................................................................................... 1407Greenplum Database Function Types................................................................................1407Built-in Functions and Operators........................................................................................1408JSON Functions and Operators......................................................................................... 1411Window Functions.............................................................................................................. 1414Advanced Aggregate Functions......................................................................................... 1416

Greenplum MapReduce Specification...........................................................................................1418Greenplum MapReduce Document Format........................................................................1418Greenplum MapReduce Document Schema......................................................................1419Example Greenplum MapReduce Document..................................................................... 1426

Greenplum PL/pgSQL Procedural Language............................................................................... 1432About Greenplum Database PL/pgSQL............................................................................. 1432

Contents OpenTopic

12

PL/pgSQL Plan Caching.....................................................................................................1434PL/pgSQL Examples...........................................................................................................1434References..........................................................................................................................1438

Greenplum PostGIS Extension..................................................................................................... 1439About PostGIS.................................................................................................................... 1439Enabling and Removing PostGIS Support......................................................................... 1440Usage..................................................................................................................................1441PostGIS Extension Support and Limitations...................................................................... 1442PostGIS Support Scripts.....................................................................................................1443

Greenplum PL/R Language Extension..........................................................................................1446About Greenplum Database PL/R......................................................................................1446

Greenplum PL/Python Language Extension................................................................................. 1452About Greenplum PL/Python..............................................................................................1452Enabling and Removing PL/Python support...................................................................... 1452Developing Functions with PL/Python................................................................................1453Installing Python Modules...................................................................................................1456Examples............................................................................................................................ 1459References..........................................................................................................................1461

Greenplum PL/Container Language Extension.............................................................................1462About the PL/Container Language Extension.................................................................... 1462PL/Container Docker Images............................................................................................. 1463Prerequisites....................................................................................................................... 1463Installing the PL/Container Language Extension............................................................... 1463Installing PL/Container Docker Images.............................................................................. 1465Uninstalling PL/Container................................................................................................... 1466Using PL/Container.............................................................................................................1467About PL/Container Running PL/Python............................................................................ 1469About PL/Container Running PL/R.....................................................................................1470Configuring PL/Container....................................................................................................1471Installing Docker................................................................................................................. 1480References..........................................................................................................................1481

Greenplum PL/Java Language Extension.....................................................................................1483About PL/Java.................................................................................................................... 1483About Greenplum Database PL/Java.................................................................................1484Installing PL/Java................................................................................................................1485Uninstalling PL/Java........................................................................................................... 1486Enabling PL/Java and Installing JAR Files........................................................................ 1487Writing PL/Java functions................................................................................................... 1487Using JDBC........................................................................................................................ 1493Exception Handling.............................................................................................................1493Savepoints.......................................................................................................................... 1493Logging............................................................................................................................... 1494Security............................................................................................................................... 1494Some PL/Java Issues and Solutions..................................................................................1495Example.............................................................................................................................. 1496References..........................................................................................................................1497

Greenplum PL/Perl Language Extension......................................................................................1498About Greenplum PL/Perl...................................................................................................1498Greenplum Database PL/Perl Limitations.......................................................................... 1498Trusted/Untrusted Language.............................................................................................. 1498Enabling and Removing PL/Perl Support...........................................................................1499Developing Functions with PL/Perl.....................................................................................1499

Greenplum MADlib Extension for Analytics.................................................................................. 1503About MADlib......................................................................................................................1503Installing MADlib................................................................................................................. 1503Upgrading MADlib...............................................................................................................1504

Contents OpenTopic

13

Uninstalling MADlib.............................................................................................................1505Examples............................................................................................................................ 1505References..........................................................................................................................1511

Greenplum Fuzzy String Match Extension....................................................................................1513Soundex Functions............................................................................................................. 1513Levenshtein Functions........................................................................................................1514Metaphone Functions......................................................................................................... 1514Double Metaphone Functions.............................................................................................1515Installing and Uninstalling the Fuzzy String Match Functions............................................ 1515

Summary of Greenplum Features.................................................................................................1516Greenplum SQL Standard Conformance........................................................................... 1516Greenplum and PostgreSQL Compatibility.........................................................................1518

Chapter 17: Greenplum Database UNIX Client Documentation........ 1527Greenplum Database Client Tools for UNIX.................................................................................1528

Installing the Greenplum Client Tools................................................................................ 1528Client Tools Reference.......................................................................................................1531

Greenplum Database Load Tools for UNIX..................................................................................1532Installing the Greenplum Load Tools................................................................................. 1532Load Tools Reference........................................................................................................ 1533

Chapter 18: Greenplum Database Windows Client Documentation..1535Greenplum Database Client Tools for Windows...........................................................................1536

Installing the Greenplum Client Tools................................................................................ 1536Running the Greenplum Client Tools.................................................................................1539Client Tools Reference.......................................................................................................1540

Greenplum Database Load Tools for Windows............................................................................ 1542Installing Greenplum Loader.............................................................................................. 1542Running Greenplum Loader............................................................................................... 1544Running gpfdist as a Windows Service..............................................................................1548Loader Program Reference................................................................................................ 1549

Chapter 19: DataDirect ODBC Drivers for Pivotal Greenplum...........1550Prerequisites.................................................................................................................................. 1551Supported Client Platforms........................................................................................................... 1552Installing on Linux Systems.......................................................................................................... 1553

Configuring the Driver on Linux......................................................................................... 1554Testing the Driver Connection on Linux.............................................................................1555

Installing on Windows Systems.................................................................................................... 1556Verifying the Version on Windows..................................................................................... 1556Configuring and Testing the Driver on Windows................................................................1556

DataDirect Driver Documentation..................................................................................................1558

Chapter 20: DataDirect JDBC Driver for Pivotal Greenplum............. 1559Prerequisites.................................................................................................................................. 1560Downloading the DataDirect JDBC Driver.................................................................................... 1561Obtaining Version Details for the Driver....................................................................................... 1562Usage Information......................................................................................................................... 1563DataDirect Driver Documentation..................................................................................................1564

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

14

Chapter 2

Pivotal Greenplum 5.7.1 Release Notes

Updated: June, 2018

• Welcome to Pivotal Greenplum 5.7.1• Experimental Features• Differences Compared to Open Source Greenplum Database• Supported Platforms• Pivotal Greenplum Tools and Extensions Compatibility• Hadoop Distribution Compatibility• Upgrading to Greenplum Database 5.7.1• Migrating Data to Pivotal Greenplum 5.x• Pivotal Greenplum on DCA Systems• Resolved Issues• Known Issues and Limitations• Update for gp_toolkit.gp_bloat_diag Issue

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

15

Welcome to Pivotal Greenplum 5.7.1Pivotal Greenplum Database is a massively parallel processing (MPP) database server that supports nextgeneration data warehousing and large-scale analytics processing. By automatically partitioning dataand running parallel queries, it allows a cluster of servers to operate as a single database supercomputerperforming tens or hundreds times faster than a traditional database. It supports SQL, MapReduce parallelprocessing, and data volumes ranging from hundreds of gigabytes, to hundreds of terabytes.

This document contains pertinent release information about Pivotal Greenplum Database 5.7.1. Forprevious versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 5.x software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 5.x is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

Pivotal Greenplum 5.7.1 is a patch release that resolves issues.

Important: This patch release resolves a timezone issue, and the documentation clarifiesGreenplum Database behavior in selecting a timezone to use when no specific timezone isconfigured. Pivotal recommends that all customers configure a fixed timezone value that iscompatible with Greenplum Database and their host system. This prevents Greenplum Databasefrom recalculating a “best match” timezone each time the cluster is restarted. If you have notconfigured a timezone value, after you upgrade to this release please verify that the selectedtimezone is acceptable for your deployment. See Configuring Timezone and Localization Settingsfor more information.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

16

Experimental FeaturesBecause Pivotal Greenplum Database is based on the open source Greenplum Database project code,it includes several experimental features to allow interested developers to experiment with their use ondevelopment systems. Feedback will help drive development of these features, and they may becomesupported in future versions of the product.

Warning: Experimental features are not recommended or supported for production deployments.These features may change in or be removed from future versions of the product based on furthertesting and feedback. Moreover, any features that may be visible in the open source code butthat are not described in the product documentation should be considered experimental andunsupported for production use.

Greenplum Database 5.7.1 includes these experimental features:

• The --plugin option for gpbackup and gprestore utilities. This option is currently unused. Theoption will allow users to specify third-party applications that process backup files during a backupoperation and supply backup data during a restore operation.

• Integration with the Pivotal Greenplum-Informatica Connector (experimental). The connector supportshigh speed data transfer from an Informatica PowerCenter cluster to a Pivotal Greenplum Databasecluster for batch and streaming ETL operations. See the Pivotal Greenplum-Informatica Connector(Experimental) Documentation.

• Recursive WITH Queries (Common Table Expressions). See WITH Queries (Common TableExpressions) in the Pivotal Greenplum Database Documentation.

• Resource groups remain an experimental feature only on the SuSE 11 platform, due to limited cgroupsfunctionality in the kernel.

SuSE 12 resolves the Linux cgroup issues that caused the performance degradation when GreenplumDatabase resource groups are enabled.

http://greenplum.org/https://greenplum-informatica.docs.pivotal.iohttps://greenplum-informatica.docs.pivotal.iohttps://gpdb.docs.pivotal.io/

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

17

Differences Compared to Open Source GreenplumDatabase

Pivotal Greenplum 5.x includes all of the functionality in the open source Greenplum Database project andadds:

• Product packaging and installation script.• Support for QuickLZ compression. QuickLZ compression is not provided in the open source version of

Greenplum Database due to licensing restrictions.• Support for managing Greenplum Database using Pivotal Greenplum Command Center.• Support for full text search and text analysis using Pivotal GPText.• Spark Connector and Gemfire Connector• Data Direct ODBC/JDBC Drivers

http://greenplum.org/

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

18

Supported PlatformsPivotal Greenplum 5.7.1 runs on the following platforms:

• Red Hat Enterprise Linux 64-bit 7.x (See the following Note)• Red Hat Enterprise Linux 64-bit 6.x• SuSE Linux Enterprise Server 64-bit 12 SP2 and SP3 with kernel version greater than 4.4.73-5. (See

the following Note)• SuSE Linux Enterprise Server 64-bit 11 SP4 (See the following Note)• CentOS 64-bit 7.x• CentOS 64-bit 6.x

Note: For the supported Linux operating systems, Pivotal Greenplum Database is supportedon system hosts using either AMD or Intel CPUs based on the x86-64 architecture. Pivotalrecommends using a homogeneous set of hardware (system hosts) in a Greenplum Databasesystem.

Important: Significant Greenplum Database performance degradation has been observed whenenabling resource group-based workload management on Red Hat 6.x, CentOS 6.x, and SuSE11 systems. This issue is caused by a Linux cgroup kernel bug. This kernel bug has been fixed inCentOS 7.x and Red Hat 7.x systems.

If you use Red Hat 6 and the performance with resource groups is acceptable for your use case,upgrade your kernel to version 2.6.32-696 or higher to benefit from other fixes to the cgroupsimplementation.

SuSE 11 does not have a kernel version that resolves this issue; resource groups are stillconsidered to be an experimental feature on this platform. Resource groups are not supported onSuSE 11 for production use. See known issue 149789783.

Pivotal Greenplum on SuSE 12 supports resource groups for production use. SuSE 12 resolves theLinux cgroup kernel issues that caused the performance degradation when Greenplum Databaseresource groups are enabled.

Note: For Greenplum Database that is installed on Red Hat Enterprise Linux 7.x or CentOS 7.xprior to 7.3, an operating system issue might cause Greenplum Database that is running largeworkloads to hang in the workload. The Greenplum Database issue is caused by Linux kernel bugs.

RHEL 7.3 and CentOS 7.3 resolves the issue.

Note: Greenplum Database on SuSE Linux Enterprise systems does not support these features.

• The PL/Perl procedural language• The gpmapreduce tool• The PL/Container language extension• The Greenplum Platform Extension Framework (PXF)

Greenplum Database support on Dell EMC DCA.

• Pivotal Greenplum Database 5.7.1 is supported on DCA systems that are running DCA softwareversion 3.4 or greater.

• Only Pivotal Greenplum Database is supported on DCA systems. Open source versions of GreenplumDatabase are not supported.

• FIPS is supported on DCA software version 3.4 and greater with Pivotal Greenplum Database 5.2.0 andgreater.

Pivotal Greenplum 5.7.1 supports these Java versions:

• 8.xxx

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

19

• 7.xxx

Greenplum Database 5.7.1 software that runs on Linux systems uses OpenSSL 1.0.2l (with FIPS 2.0.16),cURL 7.54, OpenLDAP 2.4.44, and Python 2.7.12.

Greenplum Database client software that runs on Windows and AIX systems uses OpenSSL 0.9.8zg.

The Greenplum Database s3 external table protocol supports these data sources:

• Amazon Simple Storage Service (Amazon S3)• Dell EMC Elastic Cloud Storage (ECS), an Amazon S3 compatible service

Pivotal Greenplum 5.7.1 supports Data Domain Boost on Red Hat Enterprise Linux.

This table lists the versions of Data Domain Boost SDK and DDOS supported by Pivotal Greenplum 5.x.

Table 1: Data Domain Boost Compatibility

Pivotal Greenplum Data Domain Boost2 DDOS

5.7.1

5.7.0

5.5.0

5.4.0

5.2.0

5.1.0

5.0.0

3.3

3.0.0.316.1 (all versions)

6.0 (all versions)

Note: In addition to the DDOS versions listed in the previous table, Pivotal Greenplum 5.0.0 andlater supports all minor patch releases (fourth digit releases) later than the certified version.1Support for Data Domain Boost 3.0.0.3 is deprecated. The DELL EMC end of Primary Supportdate is December 31, 2017.2The Greenplum Database utilities gpbackup and gprestore support Data Domain DD Boost FileSystem Plug-In (BoostFS) v1.1 with DDOS 6.0 or greater. Data Domain Boost is not supported.

Note: Pivotal Greenplum 5.7.1 does not support the ODBC driver for Cognos Analytics V11.

Connecting to IBM Cognos software with an ODBC driver is not supported. Greenplum Databasesupports connecting to IBM Cognos software with the DataDirect JDBC driver for PivotalGreenplum. This driver is available as a download from Pivotal Network.

Veritas NetBackupPivotal Greenplum 5.7.1 supports backup with Veritas NetBackup version 7.7.3. See Backing UpDatabases with Veritas NetBackup.

Supported Platform NotesThe following notes describe platform support for Pivotal Greenplum. Please send any questions orcomments to Pivotal Support at https://support.pivotal.io.

• The only file system supported for running Greenplum Database is the XFS file system. All other filesystems are explicitly not supported by Pivotal.

• Greenplum Database is supported on all 1U and 2U commodity servers with local storage. Specialpurpose hardware that is not commodity may be supported at the full discretion of Pivotal ProductManagement based on the general similarity of the hardware to commodity servers.

https://www.emc.com/en-us/storage/ecs/index.htmhttps://network.pivotal.io/products/pivotal-gpdb../admin_guide/managing/backup-veritas.html../admin_guide/managing/backup-veritas.htmlhttps://support.pivotal.io

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

20

• Greenplum Database is supported on network or shared storage if the shared storage is presentedas a block device to the servers running Greenplum Database and the XFS file system is mountedon the block device. Network file systems are not supported. When using network or sharedstorage, Greenplum Database mirroring must be used in the same way as with local storage, and nomodifications may be made to the mirroring scheme or the recovery scheme of the segments. Otherfeatures of the shared storage such as de-duplication and/or replication are not directly supported byPivotal Greenplum Database, but may be used with support of the storage vendor as long as they donot interfere with the expected operation of Greenplum Database at the discretion of Pivotal.

• Greenplum Database is supported when running on virtualized systems, as long as the storageis presented as block devices and the XFS file system is mounted for the storage of the segmentdirectories.

• A minimum of 10-gigabit network is required for a system configuration to be supported by Pivotal.• Greenplum Database is supported on Amazon Web Services (AWS) servers using either Amazon

instance store (Amazon uses the volume names ephemeral[0-20]) or Amazon Elastic Block Store(Amazon EBS) storage. If using Amazon EBS storage the storage should be RAID of Amazon EBSvolumes and mounted with the XFS file system for it to be a supported configuration.

• For Red Hat Enterprise Linux 7.2 or CentOS 7.2, the default systemd setting RemoveIPC=yesremoves IPC connections when non-system users logout. This causes the Greenplum Databaseutility gpinitsystem to fail with semaphore errors. To avoid this issue, see "Setting the GreenplumRecommended OS Parameters" in the Greenplum Database Installation Guide.

Pivotal Greenplum 5.7.1 Release Notes OpenTopic

21

Pivotal Greenplum Tools and Extensions Compatibility• Client Tools• Extensions• Pivotal Greenplum Data