The challenge
A couple of years ago, I accepted the challenge of setting up the digital presence of a Mobile Operator Startup. We started working early November and were challenged to be online by December 15th.
When you are put in front of impossible deadlines you need shortcuts. I met with my new colleagues and they told what we had: a LAMP infrastructure and PHP development skills. We agreed on employing WordPress with WooCommerce, to buy a commercial template and to start working right away.
I had some experience with WordPress. Previously I had developed a few websites for friends, my Parish, and my personal blog.
A series of delays due to other organizational issues made us start three months after the deadline. We were able to deliver a first working website by the first deadline anyway and could dedicate the remaining time to refine, optimize and test. To say the truth this life-cycle has been going on since and won’t end anytime soon.
The first infrastructure
The first infrastructure was far from professional grade, we had Pf-sense as a software firewall and used LVS as a software load balancer. Four virtual machines were running WordPress, a fifth was used to do editing in wp-admin, and we had a single MySql Server.
The first time we fired up systems and tested some load, the servers saturated at no more than ten concurrent sessions.
There was need quick wins. I identified four, caching, increasing the server pool, upgrading php and deploying a CDN.
So we installed a caching plugin, after some testing I chose WP-Rocket. After doing this our load capacity improved five fold.
Our IT increased the number of servers from four to twelve. We would have preferred giving more resources to fewer servers but our virtual infrastructure had limited resources and it was necessary to scale horizontally.
We upgraded PHP from version 5.6 to 7.2, just this gave us four times better performance. On a sideline I tested HHVM but it gave similar results so we have stuck with PHP.
Deploying a CDN was a bit more complicated because we started with the wrong foot. We first used a service which we could deploy quickly (and that was less expensive) but which demonstrated itself unsatisfactory. The main problems were that it had a single border gateway in Italy (Milan) and that it made our site unreachable to part of our customer base. We then reverted to that which would have been the first choice, AKAMAI Web Accelerator, and since then our servers are offloaded 90% of all traffic.
With these four quick changes our website was able to handle significant traffic (more than 500 hits per second) with no major impact on server load.
Continuing to improve
Since you never finish improving, we have now migrated to a more powerful infrastructure to be able to add more resources to our servers if necessary.
Deploying a powerful hardware firewall cluster and a load balancing appliance cluster (F5 BIG IP) has been another significant step. These are a big help for servers because they cut network latency, allowing the web server to close connections more quickly.
Our IT team also upgraded our MySQL server to the Enterprise version in a three server cluster (two read-only and one read/write). This has opened the possibility of employing HyperDB which allows us to distribute read-only queries to the two read-only servers giving us an extra performance boost (we could actually disable caching and still handle the load).
What next?
While employing a commercial template gave us a head start, it now is a problem. Commercial templates are not the quickest, lightest and most efficient templates. We must develop out own template built to reduce drastically the number of objects loaded in a page (CSS, JavaScript, and images). This will help our customer user experience significantly (my objective is to have pages rendered in less than half a second) and will reduce server load at least ten fold.
Another next step is improving performance of the web server, either employing NGINX instead of Apache or we could move to LiteSpeed Web Server Enterprise. Some tests I have done with OpenLiteSpeed are very promising and the enterprise version is a plug-in replacement for Apache with a its own cache service which works with a custom WordPress caching plugin (LSCache).