A team ships an agent that works in every demo. Then a worker restarts halfway through a job, a login expires overnight, and a message to a chat tool goes out twice. The agent was fine. The environment around it was not. The engineer hired to build that environment had a strong resume and no scoring sheet, so nobody could say in advance what good looked like. That is an illustration, not a case study, but it shows why a rubric matters.
This guide explains how to build a hiring rubric for AI agent infrastructure roles. It covers what these roles involve, why reliability needs its own scoring, what a rubric should include, how to run a mini acceptance test, a four-step plan, and a table of warning signs.
We found no study showing that a rubric picks better infrastructure engineers, and “AI agent infrastructure role” is our label, not a standard job title. The advice rests on a vendor guide, an agent benchmark, selection research, and our reasoning.
What These Roles Cover
In our usage, an AI agent infrastructure engineer builds and runs the environment an agent works in: browsers or sandboxes, files, saved logins, running processes, progress tracking, and recovery when something fails. That definition is ours. The model and the prompts are someone else’s job. Titles for this work may vary, so write the rubric around the work and not the title.
In small teams, one person covers all of it, and in larger ones it splits across platform, reliability, and application engineers. Our guide to hiring DevOps engineers covers the operations side, and our piece on a technical talent pipeline covers sourcing engineers in tight markets.
Why Agent Reliability Needs Its Own Scoring
A capable agent is not the same as a dependable one. The tau-bench paper by Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan, from Sierra, tested agents on retail and airline customer service tasks. It reports that even strong function calling agents such as GPT-4o succeeded on fewer than half of tasks overall, and that agents were “quite inconsistent”: in retail, they succeeded on all eight repeated tries of the same task less than 25% of the time.
The limits matter. It is a June 2024 preprint; models have changed since, and the tasks used simulated users in two customer service settings. The authors also note that a passing check is necessary but not sufficient for a good run. Our reasoning is that if repeated runs vary this much, the environment must be able to pause, resume, and reconcile, and the people building it need scoring for exactly that. That is an inference, not a finding.
What a Rubric Should Score
Steel, a browser infrastructure vendor, lists what must survive when an agent’s environment fails. We use it as a starting point and treat it as one vendor’s view. The article’s author discloses that Steel has a commercial stake in its recommendation, so we use its testing ideas and not its buying advice.
Score candidates on these five areas, which follow Steel’s list:
- Files. Do snapshots, downloads, and finished work remain available after a restart?
- Browser identity. Can the agent use the right account, or can a person restore access?
- Running processes. Do necessary tools resume or restart from saved inputs?
- Agent progress. Does the next run know what finished and what remains?
- External actions. Can the system tell which changes already reached another service?
Add two areas of our own: whether the candidate can name who owns each repair, and whether they can say when not to build something themselves. Steel says to build only for a hard requirement no provider meets or an infrastructure advantage worth owning. Weights are your call. These areas are our adaptation, not a validated instrument.
Score Candidates on Failure Recovery With a Mini Acceptance Test
A paper rubric only works if candidates face real failures. Steel’s guide on how to test an agent’s computer describes four acceptance tests: replace the worker after some work finishes, expire the login during a run, lose the response to a Slack post, and disconnect the person during a handoff.
Each has a pass condition and an owner. For example, an expired login passes if the agent pauses the affected actions, keeps its progress, and resumes after a person signs in again.
That is the vendor’s guidance, and we adapt it. Give every candidate the same one-page system description, such as an agent that reads reports from a portal and posts a summary to a chat channel. Ask them to explain, in writing or live, what the system should do in each of the four failures, what evidence they would check, and who owns the fix.
Score each answer on three things: whether the candidate names the failure correctly, whether the fix keeps progress, and whether it avoids repeating an action that may already have happened. For a hands-on version, give a small repository with a simulated restart and an expired token and ask for a working fix within a time limit such as three hours.
Steel adds a caution we repeat: a clean pause and resume does not prove recovery from an abrupt host failure. Our reasoning is that a candidate who says so is thinking about the real problem.
Structure the Scoring
Write behavior anchors before you read any answers. For each area, define what a weak, acceptable, and strong answer looks like. A strong answer to the lost chat message names the outcome as uncertain, checks the receiving system before retrying and records the result. A weak answer retries blindly. These anchors are our suggestions.
Selection research supports a shared, scored task, with a limit. A HumRRO summary of Sackett, Zhang, Berry and Lievens (2022) reports revised validity estimates in which structured interviews were highest at .42, and work sample tests were .33. The authors argue that earlier estimates were inflated by range restriction corrections, and they note that structured interview validity varies considerably.
These figures concern job performance in general, not infrastructure engineers. They support a modest point: the same realistic task, scored the same way, beats improvising. Our interview guide covers how to structure the conversation, and our HR scorecard guide can help you set up the scoring sheet.
Pay for the time on any take-home, since unpaid tests ask people to work for nothing and may put off experienced candidates. That is our view, not a finding. Have two people score independently and compare before they discuss. Before the first real candidate, score one invented answer together so the team agrees on what a strong answer sounds like, and revise any anchor that two scorers read differently. Include an engineer who will work with the hire, since they will notice gaps a hiring manager might miss.
A Four-Step Plan to Build the Rubric
Use this four-step plan for every infrastructure hire. It is our suggested process, informed by the sources above.
- Define the state. List what must survive a failure in your system.
- Write the tests. Turn each failure into a scenario with a pass condition.
- Score with anchors. Write weak, acceptable, and strong answers first.
- Start with one system. Begin with one small agent system before more.
The last step matters because a test shows how someone handles planned failures, and a first real system shows how they handle the ones nobody predicted.
Measure Whether the Hire Works
Track how many runs finish without a person stepping in, how long it takes to find the cause of a failed run, how often an action is repeated or lost, and whether each failure has a named owner. Judge the engineer over several weeks and several incidents, since one clean run proves little, and be careful about crediting changes caused by a provider or model update. These measures are our suggestions, not findings. Our guide to AI process automation covers the business side of running these systems.
Warning Signs Your Agent Infrastructure Hire Is Risky, and What to Do About Each
Check this table before you hire and after each trial system. One sign means fix one thing. Three or more mean pause before you commit.
| Warning sign | What it may mean | What to do |
|---|---|---|
| Talks tools, not recovery | Failure depth may be thin | Ask what happens on a mid-run crash |
| Candidate skips the paid test | Skill may go untested | Use one paid test for all |
| Retries without checking results | Duplicate actions may follow | Ask how they reconcile first |
| Assumes logins never expire | Handoffs may break runs | Add an expired login case |
| Wants to build before scoping | Cost may outrun need | Ask for a named requirement |
| No owner for each fix | Repairs may stall | Assign an owner per failure |
Score for Recovery, Then Test for It
A good agent infrastructure engineer builds an environment that can fail without losing work or repeating actions, and can say who fixes what. Define the state that must survive, turn each failure into a scenario, score answers against anchors written in advance, and start with one small system.
We found no study proving that this method picks better engineers, but each step replaces a guess with evidence. If the first system recovers cleanly from a restart and an expired login, and every failure has an owner, you have a better reason to keep the person than any demo.
