How to run an AI screening pilot for one staffing vacancy

An AI screening pilot is a limited test of how a screening tool fits your recruiting process. For a staffing agency, start with one client vacancy, an agreed scorecard and a recruiter who owns every submission decision. Measure the effort needed to produce a useful candidate submission, including the checking and correction work.

The purpose is to find out whether the workflow helps your desk qualify candidates with evidence the client can use. Keep the scope small enough that you can inspect every result and understand why the recruiter and the tool disagree.

Choose a vacancy with a stable brief

Pick a role whose requirements the client can explain clearly, such as a backend developer or a B2B sales development representative. Avoid starting with a vacancy that changes daily or mixes several unrelated skill sets. Changing the brief halfway through makes the results difficult to interpret.

Name the recruiter responsible for the pilot and the client contact who will review submissions. Agree the application window and the date you will review the results. Keep the usual human route available when a candidate needs help or cannot complete the standard assessment.

Agree the evidence the client wants to see

Turn the brief into a short scorecard before inviting candidates. Ask the client to separate essential skills from preferences and explain what would make them accept a submission for interview. Treat notice period, location and expected compensation as separate fit checks.

For a sales role, the scorecard might ask for evidence of prospect research, a relevant opening message and a considered response to an objection. For a developer, it might examine problem diagnosis and the reasoning behind a technical choice. These are starting examples, not a ready-made assessment for every vacancy.

• The skill or requirement being checked.
• The question or work example that could provide relevant evidence.
• What a strong, partial or insufficient answer would contain.
• What needs clarification rather than a negative judgement.
• Who will review the evidence before a candidate is submitted.

Record a baseline without misleading yourself

Record how the desk currently qualifies candidates for comparable work. Track hands-on recruiter time separately from elapsed time waiting for candidates or the client. Note differences in sourcing channel, seniority and brief quality before comparing a previous vacancy with the pilot.

During the pilot, have a recruiter review the candidate evidence against the agreed criteria before seeing the AI assessment. Then compare the two judgements. Use the same interview evidence where practical so candidates do not have to repeat an assessment solely for your experiment. This checks review disagreements, but it does not prove that the AI interview collects the same evidence as a recruiter-led conversation.

Test the setup before sending invitations

• Run a test application and interview with a colleague. Use invented test details, not an unrelated real candidate record.
• Check that questions relate to the vacancy and the scoring criteria match the client-approved brief.
• Review the invitation so candidates know the format, expected duration and how to contact the recruiter.
• Confirm what candidate information the agency and client need to see, and check access before sharing it.
• Test interruption recovery and the alternative process for someone who cannot complete the standard format.

Review the whole pilot cohort

Review candidates across the range of results, including incomplete interviews and lower assessments. Looking only at the top-ranked people can hide missed evidence. Check whether a low result reflects the answer, an unsuitable question, a transcription issue or a technical interruption.

Record the recruiter decision and the reason for any disagreement. Where evidence is missing, contact the candidate or arrange a relevant human follow-up. Keep the client submission tied to the approved criteria. A concise submission should explain the skills demonstrated, any unresolved question and the next conversation the client should have.

Measure effort and submission usefulness separately

• Recruiter effort: minutes spent on setup, candidate support, evidence review, corrections and submission preparation.
• Completion: invited, started and completed candidates, with known reasons for non-completion recorded separately.
• Review disagreements: cases where the recruiter changed the assessment and the evidence behind that change.
• Client response: submissions accepted for interview, requests for clarification and reasons for declining.
• Candidate feedback: confusing instructions, technical difficulties and requests for another assessment format.

Report the underlying counts beside any percentage. Separate candidates still awaiting a response from confirmed declines. Do not describe every non-completion as lack of interest. A short pilot may help assess the workflow, but placement results and later job performance need longer observation.

Decide whether to expand, change or stop

Review the results with the client contact and recruiter. Expand only if the evidence is usable, the work required is acceptable and unresolved candidate issues have been addressed. Change the questions or review process when disagreements reveal a fixable problem. Stop if the workflow does not support defensible submissions.

Write down the decision, its reason and what the next test must establish. Keep the results scoped to this vacancy. One successful pilot does not show that the same setup will work across every client or job family.

Use Tarkflo to test the screening and review steps

Tarkflo supports CV screening, conversational AI interviews and candidate reports with scores and transcripts. Its staffing workflow is designed around qualification and recruiter-reviewed client submissions.

Explore two-way AI interviews against your vacancy brief, and confirm the required features and usage costs on the pricing page. Keep the pilot scorecard and review log alongside the candidate evidence so your expansion decision reflects what actually happened.