Job Description

Job Description: AI Engineer
Location: Santa Clara, CA (Onsite)
Duration: 6-12+ months (Contract)

Overview
We are seeking an innovative AI Engineer to design, develop, and deploy agentic AI systems that automate engineering workflows. This role will focus on building evaluation frameworks, telemetry pipelines, and observability standards to ensure AI agents operate efficiently, reliably, and transparently. The ideal candidate will combine strong engineering skills with expertise in AI observability, anomaly detection, and performance optimization.
Key Responsibilities

• Design, develop, and deploy AI agents that automate engineering workflows, including experiment configuration, results analysis, regression detection, and recommendations.
• Build and maintain evaluation harnesses for agentic systems (regression suites, golden sets, scoring rubrics) integrated into release pipelines.
• Architect ingestion and storage pipelines for agent telemetry at scale, ensuring fast and actionable queries during live incidents.
• Instrument cost, latency, and token consumption per agent, tool, and routing path, surfacing efficiency signals for platform decisions.
• Advance trace and event models for agent execution (tool calls, routing decisions, retries, outcomes) to enable full reconstruction of production runs.
• Collaborate with data science teams on anomaly detection, drift detection, and failure clustering across agent execution history.
• Deliver instrumentation as composable libraries, making observability standards easy to adopt for agent authors.
• Lead design reviews and establish best practices for observability and evaluation across the agentic ecosystem.

Job Tags

Similar Jobs