Bulk redirect mapping with Retour from a crawl

—

The client-friendly version of this post is What happens to your SEO when you redesign your website?. This one is the working: how we get from "the old site has a few hundred URLs" to a verified set of redirects in Craft, using a crawl, two small scripts and a plugin called Retour.

The short version of the method:

  1. Crawl the old site and the new site.
  2. Match them automatically where it's safe, by hand where it isn't.
  3. Import the result into Retour as a CSV.
  4. Check every redirect, in a script, before and after launch.

Crawl both sites

We use Screaming Frog, but any crawler that exports a list of URLs will do.

  • Old site: crawl the live site, then export the HTML pages that return a status code of 200 (in Screaming Frog, the Internal tab filtered to HTML). Ignore images, CSS and JS: they don't carry rankings and the new build will likely have new asset URLs anyway.
  • New site: crawl the development site (or export the URLs from Craft). This is the list of places an old URL can legitimately point to.

A crawl only finds pages that are linked. Pages that are orphaned but still ranking, or still linked from other sites, won't appear. Two cheap additions: import the old sitemap into the crawl, and pull the pages that have had impressions or clicks in Search Console over the last year.

Reduce each export to one URL per line (the Address column), so the next step doesn't depend on a particular crawler's columns.

Match old to new

Most of a redesigns URL changes follow a pattern: the same page with a different section name, or the same slug under a new parent. A script can find those safely. The rules it follows are:

  • If the old path exists unchanged on the new site, do nothing.
  • If exactly one new URL has the same final slug, that's a match.
  • If several new URLs share the slug, or none do, it is not guessed. It goes to a review file for a person to decide.
// node match-redirects.mjs old-urls.txt new-urls.txt
import fs from 'node:fs';

const [oldFile, newFile] = process.argv.slice(2);
const lines = (f) => fs.readFileSync(f, 'utf8').split('\n').map((l) => l.trim().replace(/^"|"$/g, '')).filter((l) => l.startsWith('http'));
const pathOf = (u) => new URL(u).pathname.replace(/\/+$/, '') || '/';
const slugOf = (p) => p.split('/').filter(Boolean).pop() ?? '';

const newPaths = new Set(lines(newFile).map(pathOf));
const bySlug = new Map();
for (const p of newPaths) {
        const s = slugOf(p);
        bySlug.set(s, [...(bySlug.get(s) ?? []), p]);
}

const rows = ['redirectSrcUrl,redirectMatchType,redirectDestUrl,redirectHttpCode,priority'];
const review = ['old path,reason,candidates'];

for (const oldPath of new Set(lines(oldFile).map(pathOf))) {
        if (newPaths.has(oldPath)) continue; // same URL on both sites: nothing to do
        const candidates = bySlug.get(slugOf(oldPath)) ?? [];
        if (candidates.length === 1) rows.push(`${oldPath},exactmatch,${candidates[0]},301,99`);
        else review.push(`${oldPath},${candidates.length ? 'ambiguous' : 'no match'},${candidates.join(' | ')}`);
}

fs.writeFileSync('retour-redirects.csv', rows.join('\n') + '\n');
fs.writeFileSync('needs-review.csv', review.join('\n') + '\n');
console.log(`${rows.length - 1} matched, ${review.length - 1} need a human`);

This script writes its output into two files. retour-redirects.csv is ready to import. needs-review.csv is the part that takes the time, and it should: it contains every page where a decision is needed. Typical outcomes are a page that was merged into another, a page that has genuinely been retired (send it to the nearest parent section, not the homepage), or a renamed page whose slug also changed.

Two cautions:

  • Slug matching is a shortcut, not a guarantee. /news/summer-offer and /offers/summer-offer can share a slug and mean different things. Skim the matched list as well as the review list.
  • Don't send everything to the homepage. It's tempting for the leftovers, but Google treats a redirect to an unrelated page much like a missing one, and visitors are confused by it. A relevant parent page is better, and a clean 404 is better than a misleading redirect.

Pattern rules for groups of pages

When a whole section moved (every /news/... is now /blog/...), 300 one-to-one rules is the wrong tool. One pattern rule does the same job and also covers pages the crawl missed:

redirectSrcUrl,redirectMatchType,redirectDestUrl,redirectHttpCode,priority
/news/(.*),regexmatch,/blog/$1,301,99

On a July 2026 redesign the final file had 23 rules: 17 exact matches and six pattern rules. The pattern rules came from section moves like the one above, plus a couple of catch-alls for old sections that were merged into one page. Use them where the old and new structures genuinely correspond, and keep a specific exact rule for anything that is an exception.

Import into Retour

Retour accepts redirects from a CSV, with these columns:

redirectSrcUrl,redirectMatchType,redirectDestUrl,redirectHttpCode,priority
  • redirectMatchType is exactmatch for a single URL or regexmatch for a pattern.
  • redirectHttpCode is 301 for permanent moves.
  • priority controls ordering between rules.

Two housekeeping points worth noting from experience:

  • Keep the CSV in the repository. It is the record of what was redirected and why, it can be re-imported to a staging copy, and it's the input for the verify step below.
  • Turn off automatic URI-change redirects if you manage the map yourself. Retour can create a redirect whenever an entry's URI changes (createUriChangeRedirects). That's useful in everyday editing, but during a migration it can add rules you didn't write and can't easily explain. We recommend setting it to false in config/retour.php on projects if the map is built by hand, it can always be enabled later after a redesign has gone live and everything has settled down.

Verify, before and after launch

A redirect map that hasn't been tested is a list of guesses. The usual failures are all easy to detect with a script:

  • the rule doesn't fire at all (the old URL returns 404),
  • it goes to the wrong place,
  • it goes through a chain of two or three redirects,
  • it points at a destination that itself errors or redirects back (a loop).
// node verify-redirects.mjs retour-redirects.csv https://www.example.com
import fs from 'node:fs';

const [csvFile, base] = process.argv.slice(2);
const rows = fs.readFileSync(csvFile, 'utf8').trim().split('\n').slice(1).map((l) => l.split(','));
const hop = (url) => fetch(url, { redirect: 'manual' });

let failures = 0;
for (const [src, type, dest, code] of rows) {
        if (type !== 'exactmatch') continue; // pattern rules need a sample URL, test those by hand
        const first = await hop(base + src);
        const location = first.headers.get('location');
        const landed = location ? new URL(location, base).pathname : null;
        let problem = null;
        if (first.status !== Number(code)) problem = `expected ${code}, got ${first.status}`;
        else if (landed !== dest) problem = `went to ${landed}, expected ${dest}`;
        else if ((await hop(base + dest)).status !== 200) problem = `destination does not return 200`; // catches chains and loops
        if (problem) { failures++; console.log(`FAIL ${src}: ${problem}`); }
}
console.log(failures ? `${failures} problem(s)` : `all ${rows.length} checked`);
process.exit(failures ? 1 : 0);

This script asks for each old URL without following the redirect, checks the status code and the Location header, then checks that the destination return a status code of 200 on its own. That last step is what exposes chains and loops. It exits non-zero on any failure, so it can be dropped into a launch checklist (we'll come back to that in more detail in a future post about our launches).

Run it twice:

  1. Against the development site before launch, to catch typos and wrong destinations while they're cheap to fix.
  2. Against the live site on launch day, because live and dev differ (domain, HTTPS, www, trailing slashes) in ways that only show up there.

For pattern rules, pick two or three real old URLs per rule and test them by hand. The script skips them on purpose.

After launch, crawl the old URL list again in list mode, to confirm the final status for every one reports a status code of 200, and watch the Pages report in Search Console for "Not found (404)" and "Page with redirect" for the first few weeks. Anything that shows up there is a URL the crawl didn't know about, and now you have a place to add it.

What we'd do differently

  • Start with Search Console, not the crawl. The crawl tells you what's linked; Search Console tells you what Google actually cares about. We now look at both before deciding what's worth protecting.
  • Never skip the review file. The temptation with a few hundred URLs is to accept the automatic matches and send the rest to the homepage. The review file is the whole value of the exercise.
  • Test on live as well as dev. A map that passes on dev can still fail on live because of the domain, www, or a rule in .htaccess that fires before Craft does.
  • Keep the redirects. Don't prune the map after a few months because it is long. The cost of a redirect is close to nothing; the cost of a lost ranking isn't.

Summary

Step: Crawl both sites, Tool: Screaming Frog, Search Console, What it prevents: Missing URLs

Step: Match, Tool: match-redirects.mjs script, What it prevents: Hours of manual spreadsheet work

Step: Review, Tool: A person, What it prevents: Wrong guesses and redirects to the homepage

Step: Import, Tool: Retour CSV, What it prevents: Rules living only in the control panel

Step: Verify, Tool: verify-redirects.mjs script, re-crawl, What it prevents: Dead, chained and looping redirects

If you want the plain-English, less technical version, you can find it here.