Software Development Engineer - Content Quality and AI Evaluation, Technical Content Experience

Build the systems that define what "good" means for AWS documentation — and that measure whether AI agents are actually producing it. You'll work on an automated quality and governance engine that scores documentation across dimensions like readability, accessibility, link integrity, security, and discoverability, and on the evaluation platform that grades the AI agents now authoring and reviewing that content. You'll also own the shared metadata service that answers "what do we actually know about this page?" — one API where many producing systems write typed, versioned metadata about pages and the sections inside them, and every consumer reads the complete cross-producer picture in a single call instead of integrating with each producer separately. And you'll build the front door to our AI-first content platform: a home page and AI chat assistant, available both in the browser and in chat, that answers documentation questions for thousands of internal builders using retrieval over a knowledge graph and large language models. This is applied AI with a hard measurement problem at its center. Millions of AWS customers read the content your systems evaluate, and the engineers and writers who produce it rely on your quality signals every day. Key job responsibilities - Build and extend automated quality checks that score documentation across measurable dimensions, exposed through APIs and agent tooling so problems are caught while content is being written - Build and operate the AI evaluation platform — golden datasets, model-as-judge scoring, and regression gates — that teams use to prove an AI agent is good enough to ship and to catch regressions when models change - Extend the rules layer that turns quality scores into pass, warn, and block decisions, with configurable thresholds, automated ticketing, and self-healing remediation - Build and evolve the shared metadata service for pages and sections — namespaced multi-producer writes, typed and versioned records, single-call cross-producer reads, change events for downstream consumers, and the search and discovery layer on top of it - Build features on the platform home page and its AI chat assistant across web and chat surfaces, including response streaming, conversational context, and feedback collection - Own operational excellence for production services: on-call, availability and latency targets, and cost - Partner with content strategy, product management, and applied science teams to turn quality standards into rules a machine can measure and enforce A day in the life You'll start your day looking at quality signals from overnight runs — did a new check fire on more pages than expected, is a model-generated classification drifting, did the chat assistant's answer quality dip after a model upgrade? You might dig into a check that's reporting false positives, then tune its thresholds and remediation guidance so writers get advice they can act on rather than noise. Much of your week is building. You'll add a new quality dimension end to end, from the analysis logic through the API and agent-facing tooling, wire a team's AI agent into the evaluation platform so their agent gets scored against a curated dataset on every change, or onboard a new producer onto the metadata service so its data reaches every consumer without anyone building another integration. When an evaluation says an agent got worse, you'll help figure out whether the agent, the judge, or the dataset is wrong — that ambiguity is the interesting part of the job. You'll work directly with content strategists who define quality standards, product managers who own quality and governance outcomes, and engineers on peer teams whose AI agents your evaluations grade. Your customers are the writers, service teams, and AI agents producing AWS documentation — and ultimately the millions of AWS customers who read it. About the team The Content Platform team builds the quality, governance, and evaluation infrastructure for AWS's documentation ecosystem. Our quality service scores content against measurable dimensions and returns results to authors and AI agents in seconds. Our rules layer turns those scores into decisions about what ships. Our evaluation platform is how the wider organization proves an AI system is good enough to release — a shared capability rather than something every team rebuilds. Our metadata service is the single place to ask what any system knows about a documentation page or section, and the substrate that quality, search, and analytics systems build on. We also own the home page and AI chat assistant for our content platform, the entry point where builders start authoring sessions, publish documentation, and ask questions in the browser or in chat. We're part of the organization responsible for the AWS Documentation website and the AI-first content platform behind it. The team values pragmatic engineering, operational excellence, and making quality the default rather than an afterthought.