optimizing your magento store for heavy api access · optimizing your magento store for heavy api...
TRANSCRIPT
Optimizing your Magento store for heavy API access 1.0 June 2013
About This Guide
This guide is for anyone who is looking to optimise the API link speed between a Magento installation and Brightpearl. In this guide we will offer practical advice and settings for MySQL, PHP and Magento itself.
In general most VPS/Cloud or dedicated hosting companies offer both managed and unmanaged hosting services for clients. If you're managing your server directly then you will have access to change and update settings as suggested. If your server is fully managed then you may wish to request these changes are made by your hosting provider.
In general we are assuming that you are running cPanel on a Linux distribution and that you are running your Magento installation as an account on this cPanel/Linux environment. If you're hosting environment is non standard or distributed then settings suggested in this guide may not apply or be appropriate.
Suggestions in this guide are not to be taken as definitive and due consideration should be given the repercussions of changes on other installed software or services.
Background
Magento is a highly configurable shopping cart built on PHP and MySQL. It is known to offer a great deal of configurability allowing a high level of customisation. It is also known however to suffer from performance issues if poorly configured which can affect not only front end user performance but also affect internal services such as the API interface.
Brightpearl utilises the Magento API to perform its communication to Magento. As this is the case it is important to consider API performance to ensure that the connection between the two systems remains stable and trouble free.
Using the Magento API means that the servers Apache service, PHP services along with mySQL services are all put under load. For that reason optimisation of these 3 critical components is vital.
How to update your Magento system
If you are not a technical expert, please do not attempt to make the changes to the server yourself. You may take your store offline, and it may be hard to recover, even by professionals. Brightpearl cannot restore your Magento installation or Magento server to a previous state; your Magento store is your own property and access by Brightpearl is not required at any stage of the integration.
Hosting Recommendations
Due consideration should be given to the specification of the server (dedicated or virtualised) that you intend to run Magento on. In general the higher the specification a server is will ultimately result in performance improvements. Of course a business decision needs to be made to decide at what price point is appropriate and then obtain the best package possible within that price range.
For small to medium size Magento stores of product catalogue size of up to around 20,000 items an entry dedicated server or a similarly specified cloud/vps hosting package is recommended.
An example entry level dedicated server specification is as follows:
Processor Quad Core Xeon 3.1GHz
Cores 4 Core
Memory 8GB RAM
Hard Drives 2 x 250GB SATA-III RAID1
Hot Swappable Drives
SSD Available
Bandwidth UNLIMITED
Network Uptime Guarantee 100%
Control Panel Options Plesk / cPanel
Servers below this specification are not guaranteed to cope with the demands placed on it by a Brightpearl integration. Typically we have found that hosting of the above specification costs from £100 / $150 per month upwards.
Wherever possible the use of SSD local disks is highly recommended
Server Loading
How many sites run on your server?
Logically the more sites & other services that are being run on a single server the more load that server is under at any given point in time and therefore the slower your Magento store and API may run. Wherever possible we highly recommend the use of one dedicated server for one business critical Magento site/installation. This approach minimises the risk of timeouts and allows for easier debugging of problems, it is also appropriate if you are required to run a PCI DSS validated server.
Apache Optimization
1. Enable module mod_fcgid for Apache
What does this do? mod_fcgid is a high performance alternative to mod_cgi or mod_cgid, which starts a sufficient number instances of the CGI program to handle concurrent requests, and these programs remain running to handle further incoming requests. It is favored by the PHP developers, for example, as a preferred alternative to running mod_php in-process, delivering very similar performance.
(standard module for Apache version >= 2.2 ) This module can be enabled via EasyApache in the exhaustive options list as below.
Build, compile and restart Apache
Additional configuration settings for mod_fcgid
Include the following configuration lines in configuration file http.conf (accessed from the command line or from Cpanel -> Service Configuration -> Apache Configuration -> Include Editor -> Pre Main Include -> I wish to edit the Pre Main configuration include file for: All Versions)
FcgidMaxRequestLen 20971520
FcgidIOTimeout 300
FcgidMaxRequestsPerProcess 5000
FcgidMaxProcesses 20
FcgidMinProcessesPerClass 0
FcgidIdleTimeout 20
FcgidIdleScanInterval 10
FcgidBusyTimeout 20
FcgidBusyScanInterval 10
FcgidErrorScanInterval 5
FcgidZombieScanInterval 5
FcgidProcessLifeTime 10
Example:
Note: for cPanel do not edit the http.conf file directly as this will be overwritten on any cPanel upgrade or Apache rebuild.
2. Enable mod_deflate and mod_headers via command line or EasyApache
What does this do? The mod_deflate module provides the DEFLATE output filter that allows output from your server to be compressed before being sent to the client over the network. This reduces bandwidth load on your server.
and
Build, compile and restart Apache
Go to file .htaccess on your Magento site and change deflate section to following:
<IfModule mod_deflate.c>
############################################
## enable apache served files compression
## http://developer.yahoo.com/performance/rules.html#gzip
# Insert filter on all content
SetOutputFilter DEFLATE
# Insert filter on selected content types only
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css
text/javascript
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don’t compress images
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dontvary vary
# Make sure proxies don’t deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>
Your Magento site should now be passing content in gzipped compressed form. Test here: http://www.whatsmyip.org/http-compression-test/
3. Enable module mod_expires via command line or EasyApache
What does this do? This module controls the setting of the Expires HTTP header and the max-age directive of the Cache-Control HTTP header in server responses. The expiration date can set to be relative to either the time the source file was last modified, or to the time of the client access. These HTTP headers are an instruction to the client about the document's validity and persistence. If cached, the document may be fetched from the cache rather than from the source until this time has passed. After that, the cache copy is considered "expired" and invalid, and a new copy must be obtained from the source.
Build, compile and restart Apache
Go to file .htaccess on your Magento site and change deflate section to following:
############################################
## Add default Expires header
## http://developer.yahoo.com/performance/rules.html#expires
ExpiresActive On
ExpiresByType image/gif "access plus 30 days"
ExpiresByType image/jpg "access plus 30 days"
ExpiresByType image/jpeg "access plus 30 days"
ExpiresByType image/png "access plus 30 days"
ExpiresByType image/x-icon "access plus 30 days"
ExpiresByType text/css "access plus 30 days"
ExpiresByType application/x-javascript "access plus 30 days"
</IfModule>
Test here https://developers.google.com/speed/pagespeed/
4. Additional Apache optimisations.optimizing in configuration file http.conf ( from command line or from Cpanel -> Service Configuration -> Apache Configuration -> Global Configuration
Turn Directive `KeepAlive` to 'on'
Change the following values
StartServers 50
MinSpareServers 15
MaxSpareServers 30
MaxClients 225
MaxRequestsPerChild 4000
PHP optimization
1. Install a PHP accelerator/cache
Install APC, Eaccelerator, Zend Accelerator or Xcache on your server or contact with your hosting provider to install these PHP Cache Systems.
2. Update PHP configuration either in php.ini (usually in directory /etc) or via cPanel GUI where applicable.
realpath_cache_size=1M
realpath_cache_ttl=86400
memory_limit = 128M
post_max_size = 20M
upload_max_filesize = 20M
max_execution_time = 300 ***Very important can help reduce timeouts
max_input_time = 300
output_buffering = 4096
disable_functions = exec, passthru, system, proc_open, proc_get_status,
proc_nice, proc_terminate, proc_close, curl_multi_exec, show_source, dl,
shell, system_exec, virtual
error_reporting = E_COMPILE_ERROR|E_ERROR|E_CORE_ERROR
Example:
MySQL Magento database optimization
An optimised well performing Magento database is vital for overall system performance and speed.
1. Go to phpMyadmin 2. Choose your magento database 3. Select checkboxes for all tables 4. Choose Repair on select menu 5. Select with checkboxes all tables 6. Choose Optimize on select menu 7. Your database is ready
It is always recommended to backup your database before performing any optimisation.
MySql Server Optimization (Advanced)
1. If you have subtable RAM (no less 8GB) - you can disable the swap file in the OS. This can be done via SSH command line >swap off
2. Update key_buffer_size MyISAM engine.
go to mysql with superuser permissions (usually - root account on Mysql)
enter two commands
mysql > SHOW VARIABLES LIKE '%key_buffer%'; mysql > SET GLOBAL key_buffer_size = 536870912;
3. Changes of configuration file. Go to my.cnf (/etc or /etc/mysql directory)
open file and create section [mysqld]
[mysqld]
key_buffer = 512M
max_allowed_packet = 64M
thread_stack = 192K
thread_cache_size = 32
table_cache = 512
query_cache_type = 1
query_cache_size = 52428800
tmp_table_size = 128M
expire_logs_days = 10
max_binlog_size = 100M
sort_buffer_size = 4M
read_buffer_size = 4M
read_rnd_buffer_size = 2M
myisam_sort_buffer_size = 64M
wait_timeout = 300
max_connections = 400
4. Restart Mysql
Magento Optimization
1. Turn on Cache Control on Magento admin
Admin Panel -> System -> Cache Management
Enable each of Caches
2. Index database data
Admin Panel -> System -> Index Management
If you see "Update Required" - check index on left and "reindex data" on top right corner.
Caching and indexing can be automated via a cron job to run at appropriate intervals.
3. Merge JavaScript files
Admin Panel -> System -> Configuration
on this page -> left column -> Developer (bottom of column) -> JavaScript Settings > Choose "Yes" and "Save config"
Caution: Please test your store after enabling this option as incorrectly written or placed js code may not work when merged.
4. Merge CSS Files
Admin Panel -> System -> Configuration
on this page -> left column -> Developer (bottom of column) -> CSS Settings
Choose "Yes" and "Save config"
Caution: Please test your store after enabling this option.
5. Turn off debug
Admin Panel -> System -> Configuration
on this page -> left column -> Developer (bottom of column) -> Debug
Choose "No" on "Profiler" row and "Save config"
6. Turn off logs
Admin Panel -> System -> Configuration
on this page -> left column -> Developer (bottom of column) -> Log Settings
Choose "No" on "Enabled" row and "Save config"
7. Attributes optimization
Admin Panel -> Catalog -> Attributes -> Manage Attributes
enter 'Yes' only for used attributes. For non-used attributes use 'No'.
8. Run Compilation
Admin Panel -> System > Tools -> Compilation
and click on “Run Compilation Process” (top right corner)
10. Delete or uninstall non-used extensions/modules
Example
Admin Panel -> System -> Configuration -> Advanced -> Disable Modules Output and turn off Mage_poll (as example).
11. Delete non used Java-scripts from Theme page.xml file
12. Use custom Jquery scripts on page footer.
Using of APC / Memcached as cache backend (Advanced)
1. Magento uses cache system on files, its possible to change this so that Magento uses to an APC or Memcached system.
Installation, APC (root permissions)
apt-get install php5-apc
or
pecl install APC
or
contact your hosting provider
Add cache block to local.xml file
<?xml version="1.0"?>
<config>
<global>
<install>
<date><![CDATA[Sat, 26 Jun 2010 11:55:18 +0000]]></date>
</install>
<cache>
<backend>apc</backend>
<prefix>alphanumeric</prefix>
</cache>
<crypt>
<key><![CDATA[870f60e1ba58fd34dbf730bfa8c9c152]]></key>
</crypt>
<disable_local_modules>false</disable_local_modules>
<resources>
<db>
<table_prefix><![CDATA[]]></table_prefix>
</db>
<default_setup>
<connection>
<host><![CDATA[localhost]]></host>
<username><![CDATA[root]]></username>
<password><![CDATA[f]]></password>
<dbname><![CDATA[magento]]></dbname>
<active>1</active>
</connection>
</default_setup>
</resources>
<session_save><![CDATA[files]]></session_save>
</global>
<admin>
<routers>
<adminhtml>
<args>
<frontName><![CDATA[backend]]></frontName>
</args>
</adminhtml>
</routers>
</admin>
</config>
Installation, Memcached (root permissions)
apt-get install php5-memcached
or
contact with your hosting provider
Add cache block to local.xml file
<?xml version="1.0"?>
<config>
<global>
<install>
<date><![CDATA[Sat, 26 Jun 2010 11:55:18 +0000]]></date>
</install>
<crypt>
<key><![CDATA[870f60e1ba58fd34dbf730bfa8c9c152]]></key>
</crypt>
<disable_local_modules>false</disable_local_modules>
<resources>
<db>
<table_prefix><![CDATA[]]></table_prefix>
</db>
<default_setup>
<connection>
<host><![CDATA[localhost]]></host>
<username><![CDATA[root]]></username>
<password><![CDATA[f]]></password>
<dbname><![CDATA[magento]]></dbname>
<active>1</active>
</connection>
</default_setup>
</resources>
<session_save><![CDATA[files]]></session_save>
[b]<cache>
<backend>memcached</backend> apc / memcached / xcache /
empty=file
<slow_backend>file</slow_backend> database / file (default)
- used for 2 levels cache setup, necessary for all shared memory
storages
<memcached> memcached cache backend related config
<servers> any number of server nodes can be included
<server>
<host><![CDATA[127.0.0.1]]></host>
<port><![CDATA[11211]]></port>
<persistent><![CDATA[1]]></persistent>
<weight><![CDATA[2]]></weight>
<timeout><![CDATA[10]]></timeout>
<retry_interval><![CDATA[10]]></retry_interval>
<status><![CDATA[1]]></status>
</server>
</servers>
<compression><![CDATA[0]]></compression>
<cache_dir><![CDATA[]]></cache_dir>
<hashed_directory_level><![CDATA[]]>
</hashed_directory_level>
<hashed_directory_umask><![CDATA[]]>
</hashed_directory_umask>
<file_name_prefix><![CDATA[]]></file_name_prefix>
</memcached>
</cache>[/b]
</global>
<admin>
<routers>
<adminhtml>
<args>
<frontName><![CDATA[backend]]></frontName>
</args>
</adminhtml>
</routers>
</admin>
</config>
Save local.xml file. Clear directory var/cache/. Restart Magento from Admin Panel.
Mount var/cache TMPFS (command line, root permissions)
mount tmpfs /path/to/your/magento/var/cache -t tmpfs -o size=64m