Refactoring a slow search page with ajax to load faster

Explaining the problem

Before we kick things off, I want to lay out a few variables which contributed to our slow search experience. Craftcms is a fast and capable tool, but in this particular scenario we don’t have our product data stored within Crafts database, instead it is kept within a large enterprise system. We cannot import this data because it is frequently updated, so we have instead built a custom plugin to go through the enterprise softwares api and return the data based on a number of parameters. As a result of this we were seeing some very slow load times when fetching the data on our search pages.

The site has a product search which utilises keywords, category types, sort orders, and some other features. The search page can also be loaded without any parameters to show the entire product range. Visitors to the site won’t see any prices but they can browse the selection. Only logged in users get to see pricing information which is unique to each user, they also have the ability to favourite the products to be viewed in a separate account area.

The site needs to always show the most up to date stock information so that logged in users can add to their basket directly from the search pages. We were unable to implement a traditional cache because of this as the stock values would then be incorrect. We are also using imager to generate image transforms as the images are stored within the enterprise system and we have no control over their size, format, etc. We wouldn’t want to just output them onto the page as they could be very poorly optimised if we did this.

When you put together all of this functionality, it results in a number of queries which have been added into our results loop from the enterprise system. The enterprise system was also a little restrictive with our queries, so this limited our ability to optimise these any further and we had to store some data (such as favourites) within Craft itself.

Finding a solution

We were unable to refactor the queries or employ any caching as this would make the data quickly become out of date, so we needed to find a different solution. Whilst all of the data was relevant and correct, it was just taking too long to load and causing frustration for the users of the site. Enter the hero of our tale... Ajax! To be more specific, Axios paired with VueJs is the exact approach we used but this could be replicated in any js framework using an ajax approach.

We were already using ajax to load in our pagination for the search results. We did this with a “scroll to the bottom to load more” technique using the infinite js library with waypoints. The initial page load was still very slow though when getting the initial result set. It was also slow whenever any subsequent products were loaded in as part of the pagination. This made us rethink the fundamentals beneath our templates and made us strip out our twig and plugin tags.

Instead we decided that the initial page load should always be empty of products to ensure it was fast. We could then display a loading graphic to the user instead of having them stuck with their default browser loading experience, they were actually getting some feedback from the page now to show it was doing something. This also meant that they could play around with the search parameters whilst they waited. Any changes to the search at this point would cancel the original ajax call and start a new one to ensure they got the correct results.

Next up we created a new template as a json file which is where our plugin variables would be included now. We also added in the necessary parameters from the request so that the search criteria would still be honoured. VueJs was then implemented into our initial template along with Axios to fetch this json files response and populate our data attributes with it. At this point all of our template logic around favourites, image transforms, etc had now been moved into our plugin and VueJs was handling the templating. Twig (the default templating language of Craft) was no longer being used.

Sadly we were still getting some rather slow load times with this approach, but we had gained a small performance boost. Further investigation was clearly going to be needed to crack this, but we were on the correct path.

Delving Deeper

How do we improve on this further then? Database query refactoring isn’t an option, caching isn’t an option, we are loading our data via ajax, what else can we do here? Well you may recall I mentioned the imager plugin earlier which we are using to generate our image transforms. The transform tags were no longer in our templates and had been moved into our plugin tags. This meant that we were generating our image transforms when looping through our search results, before returning them to the json file. So lets move that logic again.

On this site we have a placeholder image which is used on products where no image has been added into the enterprise system, let's make that our default image. Every single product now has this placeholder and all imager logic in our plugin has been removed. This has worked wonders and we are getting much faster load speeds for our search results now. We still want to have our product images though, otherwise everything looks the same and that’s not a very good user experience. Let’s ajax load them.

{% set image = craft.app.request.getParam('image') %}
{% set imageFormat = null %}
{% if craft.imager.serverSupportsWebp and craft.imager.clientSupportsWebp %}
    {% set imageFormat = "webp" %}
{% endif %}
{% set transformedImage1 = craft.imager.transformImage(image, {'width': 245, 'height': 300, 'mode': 'fit', 'format': imageFormat}) %}
{% set transformedImage2 = craft.imager.transformImage(image, {'width': 490, 'height': 600, 'mode': 'fit', 'format': imageFormat}) %}

{"transform1": "{{ transformedImage1 }}","transform2": "{{ transformedImage2 }}"}

Using VueJs we can use v-if conditions in our templates to load a simple <img> tag with our placeholder image. We can then hide it and show a more complicated <img> with srcset attributes using our product images once they have been loaded. Whenever we refresh the product array with some new json results, we loop through each product within Vue and make an ajax call to a second json file. We pass through our product image url and it gets transformed by some imager tags and returns an array of transforms, this request can also be cached. The response array is then saved into a custom attribute on our products data model, when this attribute is populated it is reflected in the template thanks to our v-if conditionals.

<img v-if="!product.transformedImage1x.length" v-bind:src="[product.loadingImage.length ? product.loadingImage : '']" v-bind:alt="product.title">
<img v-if="product.transformedImage1x.length" v-bind:src="product.transformedImage1x" v-bind:srcset="product.transformedImage1x+' 1x, '+product.transformedImage2x+' 2x" v-bind:alt="product.title">

The end result for the user is a fast loading initial page with no data, which is quickly replaced by a page of results with placeholder images. Each of these images is then replaced with the relevant product images. Scrolling down the page will then load in another batch of products due to our lazy loaded pagination, and subsequently load the transformed images for those products to replace their placeholders. The user can also change the search parameters and kick off the process all over again without reloading the page. We’ve added a 10 second delay to the keywords text input field so that we aren’t querying on multiple keypresses, the query only happens once they’ve stopped typing. This has helped to prevent any additional slowdown which may be caused by multiple queries running at the same time within the same browser session.

Other areas such as saved favourites and stock availability could also be moved into json files to speed them up. We would do this in the same way that we’ve done the image transforms and request them via ajax, but we don’t need to talk through that process as it’s exactly the same as what we’ve detailed above.

Final Thoughts

Just because a situation has unique complexities which prevent traditional methods, it doesn’t mean you should ever settle for a slow loading experience. As developers we need to rethink the underlying structures of our pages sometimes and just try something new.

Try a number of different approaches and don’t be afraid of throwing it all away and starting again. Cut it right down to a simpler output, remove the complex functionality, and then layer it all back up step by step. If you notice one piece of functionality is causing the majority of the speed issues, then this feature needs refactoring. You can change the queries or add caching, or in our case offload it all to sit outside of the results loop and be handled by a separate request.

Don’t settle for slow!