How to Hire AI Developers: Roles, Skills, a Paid Test Task and Red Flags

Three roles that get called "AI developer"

Most failed AI hires start with the wrong job title. A founder who needs a support assistant over their documentation hires a PhD who wants to train models, or a team that needs a demand forecast hires a prompt engineer. Pin down the work first.

If you are not sure, read RAG vs fine-tuning first: most teams that think they need a fine-tuned model need retrieval and a good evaluation set. For agent work in particular, our AI agent developers are LLM application developers who specialise in tool use, workflows and guardrails.

Skills checklist for an LLM application developer

• Evaluation: can build a test set of 50 to 200 real questions with expected answers, score outputs automatically and with an LLM judge, and track regressions per prompt or model change.
• Retrieval: chunking strategies, embeddings, hybrid keyword and vector search, reranking, and how to measure retrieval separately from generation.
• Structured output: JSON schemas, tool calling, validation and retries when the model returns something invalid.
• Cost and latency: token budgets, caching, streaming, picking a smaller model where it scores the same, per-request cost logging.
• Safety: prompt injection through retrieved documents, data leakage between tenants, refusing out-of-scope requests, human review for high-risk actions.
• Ordinary engineering: an API in FastAPI or Node, a queue for slow jobs, tests, logging and deployment. AI features fail more often from missing basics than from bad prompts.

A paid test task that separates demos from production

Pay for it: four hours at the candidate's rate is cheap compared with a bad hire, and good engineers decline unpaid take-homes. Ours, used in our own vetting, is close to this:

1. Give the candidate 30 to 50 documents (policies, product docs) and 20 questions with reference answers.
2. Ask for a small service that answers questions with citations to the source document.
3. Require an evaluation script that scores answers against the references and reports accuracy, citation correctness and cost per question.
4. Ask for a one-page note: what they tried, what failed, what they would do with another week.

Interview questions worth asking

• "Walk me through how you knew an AI feature you shipped was good enough to release." Listen for an eval set and a threshold, not "we tested it".
• "A customer says the assistant gave a wrong answer. What do you do in the next hour?" Listen for traces, retrieval inspection and adding the case to the eval set.
• "How would you stop a document in the knowledge base from instructing the model to do something harmful?"
• "When would you not use an LLM for this?" Good candidates suggest rules, search or a classic model when they are cheaper and more reliable.
• "What did your last AI feature cost per user per month?"

Red flags

• Portfolio of chat demos with no production users or numbers.
• Talks about fine-tuning before asking what data you have.
• No opinion on evaluation, or "we checked it manually".
• Framework enthusiasm without understanding: can describe LangChain classes but not what the prompt actually sent was.
• Ignores tenant isolation, PII in prompts or logging.

What AI developers cost

These are our published rates for AI and ML engineers, all-inclusive with equipment, HR and a two-week replacement guarantee. For a fixed scope instead of a monthly engineer, a single AI feature in an existing product typically costs $8k to $60k and an AI MVP $25k to $60k. The AI agent cost calculator prices agent builds and their running costs. If your AI service will be in Python, a FastAPI developer is often the right second hire.

What our AI teams have shipped

• GovDoc AI: a RAG pipeline on AWS Bedrock with Claude for structured extraction of 50+ data points from government contracts, with a self-correcting three-pass analysis.
• Chat Your History: chat, SMS and phone interviews turned into memoir chapters with streaming GPT-4 generation.
• Cuts Like A Knife: a Python FastAPI machine-learning search service with OpenL3 audio embeddings and OpenAI.
• Treasure Island AI: a React Native app with an AI counsellor persona and a brainstorming engine generating up to 30 questions per idea.

More on how we scope this work is on our AI agent development page, and how we use Claude Code on client projects explains how AI coding tools fit into our own delivery.

Freelancer, marketplace, agency or in-house?

The first 30 days with a new AI developer

1. Week 1: access to data and systems, a written definition of the first feature, and 50 real examples of inputs with the outputs you would accept.
2. Week 2: a baseline: the simplest version that works, scored against those examples, with cost per request logged.
3. Week 3: improvements driven by the scores (better retrieval, a different model, structured output), each one measured.
4. Week 4: guardrails, monitoring and a release to a small group of real users, with a way for them to flag bad answers.

If, after a month, you cannot see a score that moved and a cost per request, something is wrong with the hire or the brief. Both are fixable, but only if you notice early.

Sources

• OWASP Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/ (checked October 2026)
• NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework (checked October 2026)
• AI and ML engineer rates