LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. You will analyze AI‑generated content for factual accuracy, coherence, safety, bias, and alignment with our guidelines, helping to make our models more reliable and trustworthy.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple responses to the same prompt, select the preferred output, and justify your choice.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Design adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines and resolve ambiguous cases.
• Participate in cross‑checking sessions to ensure inter‑rater reliability.
• Investigate underperforming outputs, hypothesise root causes, and flag novel model behaviors.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Proven experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
• Ability to analyse evaluation metrics (e.g., IAA scores) and conduct calibration sessions.
• Experience sourcing, cleaning, and annotating datasets for LLM fine‑tuning or evaluation.

Required skills

• Prompt engineering
• RLHF data collection
• Dataset sourcing, cleaning, and annotation
• A/B testing concepts
• Experiment design for model comparison
• Analysis of evaluation metrics (IAA scores)