Discussion
  • Agency

Hiring junior SEOs: what task has best predicted who becomes good, now that AI can handle the textbook parts?

Published 1 reply

Discussion starter: an agency question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    My answer is a pair rather than a single task: a messy diagnostic work sample, done with whatever tools the job allows, AI included, followed by a structured debrief in which every candidate answers the same questions against a written scoring guide. The work sample shows how someone handles imperfect evidence. The debrief shows whether they understand what they produced, and it is the part the research supports most strongly. In a 2022 re-analysis of selection research by Sackett and colleagues, structured interviews came out as the strongest single predictor of job performance, with work sample tests also among the stronger predictors.

    No SEO-specific validation study exists that I can point to, so the design below applies general selection research to SEO work. Treat it as a reasoned starting point, and use the last section to check whether it predicts anything in your own team.

    What AI changed about the signal

    A take-home audit used to test whether a candidate knew the checklist: status codes, canonicals, titles, internal links, sitemaps. An AI assistant now produces that checklist in seconds, so a polished audit says less about the person than it did. What I would look for instead is harder to automate:

    • noticing when the data does not support the obvious story
    • checking claims against primary sources and their dates before repeating them
    • asking what decision the work is meant to inform
    • prioritising under real constraints such as developer time and revenue
    • explaining uncertainty to someone who is not an SEO
    • improving quickly after feedback

    I would weight the last one most heavily for juniors, because you are hiring a trajectory rather than current skill. The junior worth hiring is the one whose second attempt is clearly better than their first.

    What the selection research supports

    Method Revised validity estimate What it means for an SEO hire
    Structured interview .42 Same questions, scoring anchors and trained interviewers: use this format for the debrief
    Job knowledge test .40 For juniors, test knowledge that AI tools can get wrong or out of date, not definitions
    Empirically keyed biodata .38 Needs large hiring samples to build, so impractical at low hiring volumes
    Work sample test .33 The diagnostic task below
    Cognitive ability test .31 Still predictive, but lower than the older estimates that made it the headline predictor

    The figures are Sackett and colleagues’ revised operational validity estimates, as summarised by SIOP. In the same re-analysis, unstructured interviews (.19) and years of job experience (.07) ranked below everything in the table, which is worth remembering when a CV with two years of “SEO experience” looks like a shortcut.

    Two caveats keep these numbers honest. They are average correlations with rated job performance across many jobs, not SEO specifically, and the SIOP summary notes a relatively wide spread around the structured interview average. And even.42 is a modest correlation: squared, it corresponds to about 18% of the variance in rated performance. No single method predicts well on its own, which is the practical case for combining a work sample with a structured debrief, since they measure different things.

    A work sample that tests judgement rather than recall

    Build the task around a real decision with imperfect data and a few planted traps. Use an anonymised export you have permission to share, or synthetic data built to behave like one, not live client work.

    Part 1: the diagnostic (about two hours, take-home)

    Give the candidate a brief from a fictional head of marketing (“Organic is down and the CEO thinks the redesign did it”), 16 months of Search Console data by query, page, device and date, a crawl sample, and the development team’s change log. Plant three things:

    • A reporting artefact. Include September 2025, when Google stopped supporting the &num=100 results parameter. Many sites saw desktop impressions fall and average position improve sharply in Search Console, which the data suggested was the end of a reporting distortion rather than a real ranking change. A candidate who reads that as lost visibility or a ranking gain has not checked clicks and device before concluding.
    • A real cause in one directory. A canonical change in the change log that lines up with a drop on one template only. It rewards segmenting before explaining.
    • Seasonality. The same dip in the same month last year. It rewards a year-on-year comparison before anyone blames the redesign.

    Ask four questions: what happened, how confident are you, what would you check next, and what would you tell the CEO in five sentences? Allow AI, and ask for a short note on where they used it and what they checked themselves. Banning it would test a way of working your team has probably already left behind.

    Part 2: the AI-output critique (30 minutes, live)

    Hand over eight AI-generated recommendations for the same site, three of them flawed in ways that reward checking over recall:

    • “Add FAQPage markup to win FAQ rich results”: those results stopped appearing in May 2026.
    • “Publish an llms.txt file to appear in AI Overviews”: Google’s guide to its generative AI features says sites do not need llms.txt or other special AI text files.
    • “Block /search/ in robots.txt and add noindex to those pages”: the block stops Google crawling the pages, so it may never see the noindex.

    Ask which they would ship, which they would verify first and how, and which they would drop. Some candidates will know all three from memory, which is fine. The signal is what they do with the items they do not know: do they check the source and its date, or restate the AI’s confidence as their own?

    Part 3: the structured debrief (45 minutes)

    Two interviewers ask every candidate the same questions and score each answer independently against written anchors before anyone discusses the candidate:

    1. What did you check first, and why that?
    2. What would make your conclusion wrong?
    3. Where did you use AI, and what did you verify yourself?
    4. What did you choose not to investigate?
    5. After one piece of specific feedback (for example, “you did not split by device”), how does your conclusion change?

    The last question gives you a rough learning-rate measure, which is hard to observe any other way during hiring. It is my reasoning rather than something the research above measured, but it targets what I would most want to know about a junior: whether feedback changes what they do.

    How to score it

    Dimension Weak signal Strong signal
    Evidence Accepts the headline trend Segments by device, directory and query type before explaining
    Confounds Attributes every movement to rankings Separates reporting changes, seasonality and deployments, and says which ones it can rule out
    Verification Repeats tool or AI output Checks claims against primary documentation and dates, and says what was checked
    Uncertainty Certain about everything, or hedges everything States a confidence level and names the evidence that would change it
    Communication Findings without a decision Leads with what the CEO should decide, in plain language
    Response to feedback Defends the first answer, or abandons all of it Updates what the feedback touched and explains why

    Hypothetical example:

    Candidate A submits a polished, clearly AI-assisted report: twelve findings, clean formatting, and a confident conclusion that the redesign caused the visibility loss. Candidate B submits two rough pages: clicks are flat, the impressions drop is desktop-only and starts in September 2025, one directory fell after a canonical change, and last year shows the same seasonal dip. B asks for crawl data on that directory before recommending a fix. On a checklist rubric, A wins. On the rubric above, B wins on every dimension except polish, and B is the hire I would expect to be running diagnoses without supervision within a year.

    Practicalities that protect the signal

    • Keep the take-home to about two hours and pay for anything longer. Unpaid multi-day tasks can put off strong candidates who have other options.
    • Give everyone the same materials, instructions and rubric, and score before discussing candidates.
    • State the AI rules up front: allowed, with a short note on how it was used.
    • Offer adjustments, such as extra time or a different format, to candidates who need them.
    • Weight the debrief above the document. An AI-assisted document can look senior; the debrief shows whether the thinking belongs to the candidate.

    Checking whether your task predicts anything

    Record rubric scores by dimension for every hire, then compare them with outcomes you can observe:

    Indicator When Why it helps
    Review corrections per deliverable First 90 days Shows how much senior time the hire consumes
    Time to first unsupervised client-facing work By six months A practical measure of trajectory
    Quality of diagnoses on real incidents, rated by a senior Six and twelve months Mirrors what the work sample tried to predict
    Manager rating on the same six dimensions Twelve months Shows which rubric dimensions carried signal

    Keep expectations realistic. An agency hiring a handful of juniors a year will not reach statistical confidence, and you only observe the people you hired. That is the restriction-of-range problem selection researchers correct for, and the 2022 re-analysis argued earlier corrections for it had been too large, so treat your internal numbers as rough. Use them to find rubric dimensions that never separate candidates, and hires where the score and later performance disagree sharply. If the task’s scores do not separate the juniors you later rate strong from the ones you rate weak, change the task rather than defending it.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.