Senior AI Engineer - LLM Evaluation

Autodesk — Bengaluru, IND

Job Requisition ID # 26WD98447 Position Overview If you love building real Artificial Intelligence (AI) systems that solve real customer problems and are passionate about measuring, improving, and scaling AI quality, this role is for you. The Applied AI team within Autodesk's Architecture, Engineering, and Construction (AEC) organization builds cloud-native AI solutions that make industry workflows smarter, faster, and more connected. This role is primarily focused on AI evaluation and quality engineering for Generative AI systems. You will establish evaluation standards, build automated validation frameworks, and drive observability practices that ensure AI systems are reliable, trustworthy, and production-ready. In addition to AI evaluation, you will contribute to the design and development of AI-powered applications, including Retrieval-Augmented Generation (RAG) systems, agentic workflows, and Large Language Model (LLM)-based solutions deployed at scale. Responsibilities Lead AI evaluation and quality strategy for Generative AI and Machine Learning (ML)-powered systems Design and implement scalable evaluation frameworks for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, agentic workflows, and traditional Machine Learning (ML) models Fine-tune, evaluate, benchmark, and deploy Large Language Models (LLMs) in production environments Implement evaluation and observability standards, including regression testing, monitoring, tracing, feedback loops, and quality measurement for AI systems Define and track AI quality metrics, including accuracy, robustness, bias, drift, grounding, latency, safety, and hallucination rates Build automated evaluation pipelines and integrate them into Continuous Integration and Continuous Deployment (CI/CD) workflows for continuous model validation Evaluate and adopt AI testing frameworks, benchmark methodologies, and emerging evaluation tools Build scalable Retrieval-Augmented Generation (RAG) pipelines using

View on HireSetu