Eliminating 404 error pages with Craft CMS after removing content

If you've recently deleted or perhaps intentionally removed content on your website, there's a risk that you might see an increase in 404 errors (or a page not found error to most people). This can have a negative impact on your overall search engine visibility because the likes of Google don't want to direct visitors using their search results to a broken link.

So let's have a look at the 410 HTTP Status Code and why you should be using it.

Confession time; until last week, I'd never heard of the 410 HTTP status code. But now that I have, I realise that I have been misusing the 301 (and maybe even the 404) status for years! If your initial thought is, "Why are you speaking about a bunch of numbers?", you would be forgiven for not understanding their importance.

So, why are HTTP status codes important? Well:

HTTP response status codes indicate whether a specific HTTP request has been successfully completed.

MDN web docs

I bet you feel enlightened, right? OK, so let's explain that in english.

For the purposes for today, HTTP requests are essentially the sending back and forth of information over the internet. Information can be anything from images, text, video, music, code, and probably lots of other things I can't think of. The HTTP response status codes tell us if a request has been successful or has failed; and can give us some information as to where or how it went wrong.

HTTP response status codes come in 5 main categories and there are a whole bunch of them should you need some bedtime reading. There are a few interesting ones in there, not least 418 - I'll let you check that one out for yourself.

Why are they important?

All this is very interesting, honestly. But why should you care? As I said, HTTP status codes tell us useful information about if and why something has failed. Or not. There is the very lovely 200 OK - we like this one because everything has worked as planned.

You may have come across Page Not Found or the Error 404 during your travels around the internet. This status code has lead to some creativity with fun and interesting 404 pages popping up. There are even blog posts dedicated to sharing the most interesting ones people have found.

But did you know that dealing with error codes is important for maintaining your search engine rankings? That is, resolving those 404 errors so that users don't come to a dead-end but find the information they are looking for. One way you might achieve this is by redirecting the user from an old page to a new page. So, let's say you recently redesigned and built your company website, and during that update you changed Blog to News. Well, any old links to Blog or the Blog articles, now need to go to News and the corresponding News articles. The status code to use when doing this is 301 Moved Permanently; and it tells search engines that the Blog content now lives under News and to update its search results accordingly.

But what if during that rebuild you decided some of that Blog content was so old and out-of-date that it wasn't worth keeping? There is no new page to direct users to so we shouldn't use 301. And 404 also isn't correct as the page used to exist but no longer does. We also know that the page won't exist in the future. This is when we use 410 Gone.

[Status Code 410] Indicates that the resource requested is no longer available and will not be available again. This should be used when a resource has been intentionally removed and the resource should be purged. Upon receiving a 410 status code, the client should not request the resource in the future. Clients such as search engines should remove the resource from their indices.

Wikipedia

How to use Error 410 with Retour?

One of our 'must have' plugins that we install with every Craft CMS project is Retour, which "allows you to intel­li­gent­ly redi­rect lega­cy URLs, so that you don’t lose SEO val­ue when rebuild­ing & restruc­tur­ing a website." Pretty useful, eh?

Retour is very simple to use and does a lot of the heavy-lifting for you. You can see in the screen-shot below that Retour records all the 404s encountered on your site and gives you nice (sometimes scary) graphs of those hits. More importantly, it generates a list of 404s and gives you some information about each, such as the referrer, the number of hits it has had, and whether or not it has been handled - whether a suitable redirect has been implemented.

Clicking on the blue cross on the right of a result enables you to add some settings to each 404 entry, including setting the redirect type and destination url.

You can see that the redirect type options include a few statuses that you should now be familiar with, 301 Permanent and 410 Gone, as while as 302 Temporary.

For an old Blog or News article which we have deleted, that is, that has no new version or location to link to, we want to use the 410 Gone option. Click save, and we are done!

Well, almost.

Create a 410 Page

Like 404s, we need to create a 410 page. This should provide the user with useful information, namely that the page that they are looking for no longer exists. You may even like to explain that it has been deleted because the information it contained is old and out-of-date, or whatever your reason is for removing it.

Making it clever

We decided to keep our 410 page very simple by using the same layout as our 404 page - just with different text to explain the error the user is getting. But one thing we did add to this page is a dynamically generated suggested URL based on the URL that got you to the 410 page in the first place. This is to try and reduce the likelihood of a dead-end for the user, as I mentioned above.

You can see in my screenshot that the URL the user visited is adigital.agency/services/expressionengine-development and in the text the suggested URL is adigital.agency/services. I'm sure you can spot the pattern, that we can simply dropped off the last segment of the URL, that is /expressionengine-development is removed from the suggested URL.

This is really easy to achieve in Craft CMS with a small bit of Twig code in the template:

{% set legacyUrl = craft.app.request.absoluteUrl %}
{% set lastSeg = "/" ~ craft.app.request.segments|last %}
{% set redirectUrl = legacyUrl|replace({ (lastSeg): '' }) %}
<a href="{{ redirectUrl }}">{{ redirectUrl }}</a>

This simple bit of code first gets the full URL, then it works out what the last segment is, and we have included the last forward-slash too with "/" ~, and then we replace the last segment with an empty string (or nothing) using |replace({ (lastSeg): '' }) to generate our suggested URL, which is redirectUrl in the code block above.

There is a chance that the suggested URL is a page that also does not exist, but using Retour, we can handle that too.

Finally, you might like to add a search field, to give your user multiple ways to navigate away from the dead-end they have found themselves in. And that's it!