Migrating to Digital Ocean: Sink or Swim!

This article is the first of a 2-part follow up to a talk recently given by Andrew Armitage at the Manchester Craft CMS Meetup.

Part 1 explains our decision making process, and points out a few problems we identified, while Part 2 (to follow) will show some of the benefits we've seen since making the change.

Six months ago, I wrote a post about about how we managed our domains and hosting. With our existing dedicated and managed hardware due for retirement, we recently made the decision to go it alone, migrating our sites to DigitalOcean.

This was an important decision for us. Many agencies see hosting as a distraction — perhaps even something of a poison chalice, but in all honesty, it's always been a fundamental (and valuable) part of our business. Perhaps we'd been shielded from the challenges of managing servers, as for the last few years we've enjoyed the benefits of full server management with Manchester based Melbourne hosting.

By comparison, DigitalOcean provide Infrastructure as a Service, which loosely translated means, you choose your kit, and from that point on, you're responsible for what you do with it.

The risk with DigitalOcean was that we became sysadmins which we didn’t want. We were happy managing our servers through the Plesk control panel, but poking around with command line tools is very different. Even though we weren't strangers to the command line, it was always reassuring to know that Melbourne would have our back if we needed them. So it was for this reason that we'd largely overlooked DigitalOcean and felt more comfortable with a fully managed service.

However, following a discussion with fellow developer Steve Rowling who pointed out ServerPilot not only offered a control panel, but would deploy and maintain our stack to the server, DigitalOcean was suddenly a much more practical option. With both platforms offering excellent documentation, and reassured by the knowledge within the CraftCMS and ExpressionEngine developer communities, we felt DigitalOcean was worthy of closer consideration.

@lealea @nystudio107 That’s the point of ServerPilot/Forge. They do it for you so you don’t need to.

— Steve Rowling (@steverowling) March 4, 2017

Out with the old

The key components of our hosting was as follows:

• 2 Dell PowerEdge R200 series servers
• Plesk control panels
• Automated off-site backups
• Shared firewall
• External DNS management
• 24/7 Server management

As mentioned above, it was ease of use of having a control panel and stack deployment that would be critical if DigitalOcean was going to work for us. So faced with a choice of 2 possible control panels, we needed to decide on either ServerPilot or Forge.

ServerPilot or Forge?

When alls said and done, there's little to choose from between these 2 apps, but the final decision was based on ServerPilot offering more freedom to use older versions of PHP.

Running a number of legacy sites, the bleeding edge stack installed by Forge wouldn’t work for us. However, Forge does offer a more attractive pricing model with unlimited servers for the a flat fee, compared to ServerPilot pricing per server. Ultimately, there’s no reason why we couldn’t use ServerPilot on some droplets, and Forge on others, but then there’s the simplicity and consistency of having all droplets using the same control panel.

What have we learned?

Inevitably when moving to a new hosting environment, there's a learning curve. While it wasn't as steep as I'd feared, there were still some key points that we learnt through our early trials.

The Droplet Name Matters

As we created some new droplets to have a play with, we'd initially just been giving them our own imaginative names, such as 'test', or 'craft3'. On the face of it, these were working out just fine, however we quickly realised that these hostnames wouldn't allow us to create a reverse DNS lookup which would be essential for our sites to send outbound email.

Fast forward to the first emails being sent from the droplet, and initially we weren't seeing anything come through, but the mail logs told their own story.

status=bounced (host said: 550 The email sent from 178.62.1.46
was rejected by the receiving server as the sending server did
not pass the required reverse DNS lookup. Please got to - 
http://www.emailbouncing.info/#reversedns for more information.
(in reply to RCPT TO command))

Immediately we realised that to enable a reverse DNS lookup, we'd need to give our droplet names fully qualified domain names (FQDN).

Not only this, but to complete the loop we'd also have to create new forward looking DNS entries that matched the FQDN and pointed to the server IP address.

Getting the Message with a Fix to Postfix

Postfix is the default mailserver installed by ServerPilot. Even though you won't be hosting email accounts, of course it's more than likely that you'll want your app or site to send outbound email, perahps in the form of password resets or order notifications.

Unfortunately, out of the box from ServerPilot, Postfix needs a quick 'fix' to reduce the likelihood of your outbound mail becoming trapped in spam filters.

Through plenty of trial and error, we the email source from our tests showed us this:

with SMTP;
Thu, 13 Apr 2017 14:29:50 +0100
Received: from ubuntu-512mb-lon1-01.example.com ([138.68.135.39]) by
messagestream.com with the MessageStream SpamAWARE Engine - inbound
scan; Thu, 13 Apr 2017 14:29:48 +0100
Received: by ubuntu-512mb-lon1-01.example.com (Postfix, from userid 1001)
id 92FC042C1E; Thu, 13 Apr 2017 13:29:47 +0000 (UTC)

This suggested that somewhere in the system, .example.com was being appended to our hostname. When this was being checked against a DNS lookup, it failed, and the spam filter cast doubt on the validity of the message. After some scratching around, we figured we needed to find out where postfix was getting the hostname of the droplet from. Time to roll our sleeves up and fire up the Terminal window!

Connecting as root user, we looked in the main Postfix config file.

root@ubuntu-512mb-lon1-01:~# cd /etc/postfix

root@ubuntu-512mb-lon1-01:/etc/postfix# ls
dynamicmaps.cf	master.cf	postfix-script	sasl
main.cf	postix-files	post-install

root@ubuntu-512mb-lon1-01:/etc/postfix# nano main.cf

For those unfamiliar with using the command line, it's not as frightening as it might first appear! However, do tread carefully, as you can cause problems here, especially as root user.

Essentially, we've changing directory (cd) to where Postfix is installed, then using the text editor nano to view the current configuration.

In the main.cf file, there were 2 particular lines that were relevant:

myhostname = ubuntu-512mb-lon1-01.localdomain

smtp_helo_name = $myhostname.example.com

The hostname (apart from it not being a fully qualified domain name) is fine, but if you look at the smtp_helo_name, it uses the $hostname variable, but is suffixed with example.com. That was where the added example.com extension was being added to outbound messages. Without stating the obvious, this would never pass a DNS lookup.

ubuntu-512mb-lon1-01.example.com

So, even if you've named your droplet with a FQDN and created your DNS records, your reverse DNS or PTR record will be broken. So with your hostname set correctly with a FQDN, simply knock off the .example.com from the smtp_helo_name.

myhostname = wansfell.adigital.hosting.localdomain

smtp_helo_name = $myhostname

Do you even need to fix PostFix?

In a perfect world, you could avoid the headaches of even digging into PostFix altogether, simply by disabling the service and using an alternative SMTP mail delivery platform such as SendGrid, MailGun or Amazon SES.

However, this all depends on the ease of adding SMTP credentials to your platform - easy done for ExpressionEngine and Craft, but perhaps less so without recoding on other platforms.

Getting closer to the command line in our servers was one thing, but configuring a mail server is a completely different undertaking that we don't want to get involved with!

Setting up the server for installing CraftCMS

The current version of MySQL installed by ServerPilot is 5.7 which follows a 'strict' declaration. Craft 2.x sites don't run with this, so MySQL's strict mode needs to be disabled. This has been well documented in the community, so it's a simple command to change this. It's also something we would have likely found on any other updated hosting environment, so not really fair to highlight this as a DigitalOcean issue.

The same issue also caused some of our legacy ExpressionEngine sites to break as well, although this is no longer an issue in ExpressionEngine 3 or Craft 3.

/etc/mysql/conf.d/disable_strict_mode.cnf

[mysqld] sql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,
ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

sudo service mysql restart

Set a Default Domain

By default, ServerPilot will set a default domain, based on the alphabetical names of your apps on each server. Initially, we weren't too concerned by this, until we realised that anyone pointing their domain at one of our IP addresses, could get a surprise when someone else's site appeared under their domain!

So we decided to create a default domain on each droplet where we were hosting more than 1 site, and publish a simple holding page to avoid any confusion.

So by this point, we're very happily up and running with most of our sites on DigitalOcean.

Next week, we'll share how we've setup our automated backups, some new tricks that might not even have been possible with our existing servers, and talk about some of the improvements in performance we've seen — which have been pretty impressive.