Staff Engineer - SDET, Pytest, Agentic Systems

NextGen Healthcare — Work From Anywhere-India

Job Description: We are seeking a highly skilled and performance-driven AI SDET (Software Development Engineer in Test) to join our engineering team. In this role, you will be responsible for defining and executing the technical quality assurance strategy for our Agentic AI solutions. You will focus on testing complex orchestrators and sub-agent architectures built on Python using open-source frameworks (e.g. Strands/LangChain ) and deployed on a managed execution infrastructure. Your primary mission is to ensure the reliability, accuracy, and efficiency of multi-agent systems while specifically optimizing for operational cost (token management) and system performance (latency and throughput). Design and implement comprehensive test suites for multi-agent architectures, focusing on the seamless interaction between the central orchestrator and specialized sub-agents. Utilize and test implementations within Agentic execution frameworks (e.g. ADK, Bedrock Agent core) and agents, ensuring robust tool-calling, state management, and memory retention. Develop automated monitors and test cases to track token consumption; identify and mitigate "looping" behaviors or redundant calls that drive up API costs. Conduct rigorous performance testing to measure and optimize end-to-end latency, specifically focusing on agent reasoning time versus API response time. Build and maintain automated evaluation pipelines using metrics such as faithfulness, relevancy, and correctness (e.g., FAVES) to validate LLM outputs. Test the decision-making capabilities of the orchestrator, ensuring it correctly routes tasks to sub-agents and handles edge cases or "agentic failures" gracefully. Build from-scratch Python-based automation frameworks tailored for non-deterministic AI outputs, moving beyond standard assertion-based testing. Integrate AI-specific testing gates into DevOps pipelines to ensure every deployment meets performance and cost benchmarks. Partner with AI Research Sci

View on HireSetu