protecting mysql network traffic - percona · booking.com at a glance started in 1996; still based...
TRANSCRIPT
Protecting MySQL network traffic
Daniël van Eeden | 25 April 2017
Booking.com at a glance● Started in 1996; still based in Amsterdam
● Member of the Priceline Group since 2005 (stock: PCLN)● Amazing growth; continuous scaling challenges
● Online Hotel/Accommodation/Travel Agent (OTA):● Over 1.2 million active properties in 227 countries● Over 1.2 million room nights reserved daily● 40+ languages (website and customer service)● Over 13,000 people working in 187 offices in 70 countries
● We use a lot of MySQL and MariaDB:● Thousands (1000s) of servers, ~90% replicating● >150 masters: ~30 >50 slaves & ~10 >100 slaves
2
Why protect MySQL network traffic?
● Protect leaking of authentication data (passwords, etc)● Protect leaking of sensitive data (PII, credit card numbers,
medical records)● Ensure data is not tampered with.● Because of regulations● Because why not? Are you still using telnet to manage
servers?
How?● Use SSL!● Done!
SSL Support in MySQL● MySQL doesn't have SSL support● MySQL never had any SSL support● MySQL has TLS support.. this is what is called SSL but isn't● Supported since 4.0.0 (~2003)● For now just assume SSL and TLS are the same
What is NOT protected by TLS● Data-at-rest
○ InnoDB and MyISAM data files○ Binlogs, redo logs, slow query logs○ Backups
● Does not protect against a DoS○ e.g. corrupting traffic
● Might not protect the query text○ performance_schema etc.
● Does not hide the traffic pattern
First steps with TLS1. Get a certificate2. Restart MySQL3. Enable TLS on the client4. Check if the connection actually uses TLS
Generating the certificate● With 5.7 and up: Might already be done by your installation● If not use mysql_ssl_rsa_setup● For older versions: https://github.com/dveeden/mysslgen● Or use the openssl commandline utilities as described in the reference
manual on https://dev.mysql.com/doc/mysql/en/creating-ssl-files-using-openssl.html
● Did you know MySQL Workbench has a SSL Wizard?
Configuration● On 5.7+: Place the ca.pem , server-cert.pem and
server-key.pem in your datadir. (already the case if you use mysql_ssl_rsa_setup)
● Or set ssl-ca , ssl-key , ssl-cert in your my.cnf● Restart MySQL● Enable SSL in your application. You probably want to copy
your ca.pem file to your client
Checking your connection
● 'status' or \s● Look for 'Cipher in use'● Or check the 'Ssl_cipher' session
status.
What if it doesn't work?● Check your mysqld.log● Check the permissions on the pem files
○ Should be readable for the mysql user● Try to connect with --ssl-mode=REQUIRED● Use the OpenSSL commandline tools to see what's in the certificate.● Use Wireshark.
ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)ERROR 2026 (HY000): SSL connection error: unknown error number ERROR 2026 (HY000): SSL connection error: SSL certificate validation failureERROR 2026 (HY000): SSL connection error: SSL_CTX_set_default_verify_paths failedERROR 2026 (HY000): SSL connection error: protocol version mismatchERROR 2026 (HY000): --ssl-mode=REQUIRED option forbids non SSL connectionsERROR 2026 (HY000): SSL connection error: Failed to set ciphers to useERROR 2026 (HY000): SSL connection error: Unable to get certificateERROR 2026 (HY000): SSL connection error: Unable to get private key
Now let's make it more secure● Require the use of TLS on the server● Require the use of TLS on the client● Enable more security checks● Security updates
Make TLS a requirement● On a per user basis:
○ ALTER USER foo REQUIRE SSL○ Undo: ALTER USER foo REQUIRE NONE
● But what happens if you accidentally create a user?○ e.g. GRANT on a nonexistent user?○ Set: sql_mode=NO_AUTO_CREATE_USER,…
● On a server level:○ SET GLOBAL require_secure_transport=ON○ This still allows UNIX socket connections w/o TLS
Issues with full-on TLS● Is your monitoring capable of using TLS connections?● What about load balancer health checks?
On the client● Use --ssl-mode=REQUIRED
○ The default in 5.7 is PREFERRED○ Older releases default to DISABLED
● This only makes a TLS connection a requirement○ Does not check if issued by a trusted CA○ Does not check if the hostname matches the cert○ To do this use VERIFY_CA or VERIFY_IDENTITY
● On older versions:○ Use --ssl-ca to allow TLS and enable CA checks○ Use --ssl-verify-server-cert to do hostname checks.○ Often not possible to force the use of TLS: this is the
BACKRONYM vulnerability● Use --ssl-ca=/path/to/ca.pem to specify which CA(s) are
trusted.
Client checks● The client could do these checks:
○ Is the certificate signed by a trusted CA?○ Does the CommonName (CN) in the certificate match
the hostname we are connecting to?○ Is the certificate expired?
Certificate Authority validation● Validates that the server certificate is signed by one of the CA's
present in the specified CA file.● Note that a CA file can have multiple CA's● There is also a CA path option.● The auto generated certificates from mysql_ssl_rsa_setup all
have their own CA.
Hostname validation● mysql_ssl_rsa_setup generates certificates with
○ CN=MySQL_Server_5.7.18_Auto_Generated_Server_Certificate● So generate the certificates manually if want this to match your
hostname● A certificate can have a list of hostnames in SubjectAltName
○ MySQL doesn't check those... Bug #68052● So if you use a virtual-IP, cname, etc. it might be difficult to match
this.● What if your clients connect on a CNAME and your replicas connect
on the hostname? You can't have both!
Security updates● I reported a few issues to Oracle.● CVE-2017-3590 for Connector/Python● CVE-2017-3469 for MySQL Workbench● CVE-2017-3467 for libmysqlclient● Those are fixed. See the Oracle Critical Patch Update for
details.● But if you care about security you should follow the release
notes and Critical Patch Update anyways...
What library does MySQL use?● Community Edition: YaSSL
○ Because GPL and the OpenSSL license are not really compatible
○ This library is maintained by WolfSSL○ This not CyaSSL/WolfSSL○ WolfSSL made a patch to include WolfSSL in MySQL
5.6.30 (https://github.com/wolfSSL/mysql-patch)● Enterprise Edition: OpenSSL● If you build MySQL yourself: you can compile against either
of them.
Why not TLS?● Because it is SLOW!● Because we trust our network!● Because we encrypt with:
○ The application (store encrypted data)○ SSH (Also works great with Workbench)○ VPN
● Because we want to inspect our network traffic!○ Wireshark can decrypt it if you hand over your private
key. Some ciphers require you to somehow extract session keys.
How slow is slow?● Overhead in milliseconds for
setting up a TLS connection on localhost with TCP.
● Client: go● 5.7 is faster than 5.6● OpenSSL is faster than YaSSL● Using TLS tickets (OpenSSL only)
helps● Best case: 0.99ms (5.7 OpenSSL w/
tickets) vs. 0.60ms (no TLS)● TLS does need more roundtrips,
but this will change with TLS 1.3● OpenSSL performs better because
it uses AVX2 and AES-NI
Bulk transfer performance● Easy to test: mysqldump with and without TLS● Different ciphers do make a difference.
mysqldump performance with MySQL 5.6.35 (YaSSL)
No TLS 4.5s
TLS Default 10.4s
RC4-MD5 7.1s
DES-CBC3-SHA 23.2s
Monitoring● Monitor the Expiry of certificates● Not just the certificate on disk, also the one in memory.● Use TLS for your monitoring on 5.6 and earlier, otherwise
you might not see the status vars● Performance schema can show you the ciphers and TLS
versions in use by all connections● Using SYS is even easier:
○ SELECT * FROM session_ssl_status
Client certificates● This allows mutual authentication● Often used together with a password● You might want to use REQUIRE SUBJECT or REQUIRE
ISSUER on accounts.● At least use REQUIRE X509 instead of REQUIRE SSL
Replication● Use CHANGE MASTER TO MASTER_SSL=1, etc● Think about what happens if your certificate(s) expire● Does the hostname match the certificate?
Changing certificates● Needs restart● Moving slaves around might not work until you restart..● Same for a switchover.
CRL and OCSP● Only possible with OpenSSL● Does not auto download the CRL from the distribution
point● Does not use OCSP● Basically restart MySQL every time your CRL changes.. which
is not practical
Where to get your certificate?● Official CA?● Internal CA?● Self signed?
TLS handshake with MySQL● server helo with ssl flag set● 'empty' login packet with ssl flag set● Start SSL handshake● Basically STARTTLS-ish
○ SSL and non-SSL on the same port
Protection of authentication data● native password with nonce● sha256 password with RSA keys or TLS● cleartext plugin
TLS ciphers● Possible to set restriction on Server and Client● How are you going manage and maintain that?● 'REQUIRE cipher' also requires client certificates● One practical use case would be to use a faster cipher for
mysqldump● Might help with compliance● 5.7.10 already places more strict requirements on the list of
ciphers
TLS versions● Can be limited on the server and client● Note that YaSSL only has TLS 1.0 and TLS 1.1 support● Minimum is TLS 1.0
What about MariaDB?● Doesn't use --ssl-mode ● Does have good TLS support● MariaDB Connector/C has support for
○ fingerprint verification○ password protected private keys
● 19 Open MDEV's tagged with SSL
Connector support● Works for C, C++, Python (multiple), Perl, Java, ODBC, Go, etc● The Go MySQL driver lets you specify a TLS Config, which is
really flexible.● Do update your Connector.. Many connectors did have
security updates related to TLS.
Don't forget these● MySQL Cluster (NDB) communication within the cluster● Galera communication● Sending backups to a central location (xbstream etc)● Network traffic for iSCSI, FCP, NFS
Future● TLSv1.3 with 0-RTT● WolfSSL?
Oh, and Booking.com is hiring!● Almost any role:
● MySQL Engineer / DBA● System Administrator● System Engineer● Site Reliability Engineer● Developer● Designer● Technical Team Lead● Product Owner● Data Scientist● And many more…
● https://workingatbooking.com/39
Thank you!
All references to “Booking.com", including any mention of “us”, “we” and “our” refer to Booking.com BV, the company behind Booking.com™
Title● Item● Item