15 Questions to Ask Before You Hire an AI Developer

A demo proves very little. Almost anyone can make a model produce an impressive answer once; the skill you are paying for is making it produce acceptable answers thousands of times, at a known cost, without leaking data or doing something irreversible. These fifteen questions are grouped into five areas. For each one I describe what a good answer sounds like and the red flag to listen for.

I sit on the other side of these conversations, so treat this as the list I would want a careful client to ask me.

Questions about quality and evaluation

Evaluation is the clearest dividing line between people who ship AI features and people who build demos. Hamel Husain, who teaches AI evaluation, argues in Your AI Product Needs Evals that unsuccessful AI products "almost always share a common root cause: a failure to create robust evaluation systems". I cover what those evals look like in why AI agent projects fail.

1. How will we know the AI feature works? A good answer describes a test set of real examples, a way to score results, and a target agreed with you before launch. Red flag: "We will test it thoroughly" with no mention of examples or a score.

2. What did you measure on your last AI project, and how did it change? You want a number that moved, and what moved it: a retrieval change, a prompt change, a different model. Red flag: they cannot name anything they measured.

3. What happens when the model is wrong? Listen for validation in code, fallbacks, "I don't know" answers when sources are missing, and human review for risky outputs. Red flag: "The model is very accurate" as the whole answer.

Questions about data and privacy

The AI only knows what you give it, and everything you give it is a privacy decision.

4. What data will the AI see, and how do you limit it? Good answers reuse your existing permission checks, so the AI never sees more than the user could, and keep personal data out of prompts unless the feature needs it. Red flag: a plan to send whole databases or inboxes to the model "for context".

5. Which model provider will hold our data, and under what terms? A good candidate knows the provider's retention terms. For example, OpenAI documents that API data is not used for training by default and that abuse-monitoring logs are kept for up to 30 days, while Anthropic documents its retention policies and zero data retention for eligible commercial organisations. Red flag: they have never checked.

6. Do we need to train or fine-tune a model? For most business features the honest answer is "probably not; we will start with a hosted model and retrieval over your data". Red flag: a proposal that starts with weeks of training before anyone has tried the simple approach. My post on RAG vs fine-tuning explains why.

Questions about cost

There are two bills, building and running, and good developers talk about both without being asked.

7. What will this cost to run per month at our volume? A good answer is an estimate per request, multiplied by your expected usage, with the assumptions written down. Red flag: they have no idea what tokens cost or say it is "negligible" without checking.

8. How will you keep running costs under control? Look for caching of repeated prompts, batch processing for work that is not urgent, smaller models for simple steps and spend limits per key or project. Red flag: one large model for every call, with no caps.

9. Which model and provider would you use, and why? A good answer weighs quality on your task, cost per task and limits, and keeps the option to switch. I compare the two biggest providers in Claude API vs OpenAI API. Red flag: loyalty to one model for reasons they cannot explain.

Questions about safety and security

AI features fail in new ways, and the developer should already know the common ones.

10. How do you stop the AI from doing something harmful? OWASP's guidance on excessive agency names "excessive functionality", "excessive permissions" and "excessive autonomy" as root causes, and recommends minimal tools, least-privilege access and human approval for high-impact actions. A good candidate describes those controls in their own words. Red flag: giving the AI full write access "to keep it flexible".

11. How do you handle prompt injection? A good answer treats text from users, emails, documents and web pages as untrusted, separates it from instructions, and limits what a manipulated model could actually do. Red flag: "Our system prompt tells it to ignore malicious instructions."

12. What do you log, and who can see it? Good answers cover inputs, outputs, cost and errors, with personal data handled carefully and access restricted. Red flag: no logging, which means nobody can investigate a complaint.

Questions about ownership and handover

The project ends; your dependence on one person should not.

13. Who owns the code, prompts and evaluation data? The answer should be you, in writing. If the developer is outside your country, check how assignment works there: under Indian copyright law, for example, an assignment must be in writing and signed, and if the duration or territory is not specified it is deemed to be five years and to extend only within India. Red flag: vagueness about prompts or evaluation sets, which are part of the product.

14. What will you hand over at the end? Look for a repository in your account, documentation, the evaluation set, a running-cost summary and a walkthrough with your team. Red flag: the system only runs on their accounts.

15. What happens after launch? Models are updated and retired, prices change and user behaviour drifts. A good answer includes monitoring, a plan for model upgrades tested against the evaluation set, and clear support terms. Red flag: "It will just keep working."

Red flags at a glance

If you only have time for a quick screen, these patterns are the ones I would act on.

The last row matters as much as any answer. A good AI developer asks you a lot of questions before proposing anything, because the right design depends on your data, your users and your risk tolerance. For the wider decision of who to hire, see AI developer vs AI engineer and freelance AI engineer vs agency.

Work with me

I am happy to be asked all fifteen. I build AI features with evaluation, cost controls and a clean handover from the start. See AI integration and AI agent development, read how much it costs to hire an AI developer, or browse my work and book a scoping call.