ML Operations Engineer (AI/LLM)

The AI / LLM Team’s mission is focused on three core pillars, “product”, "enablement" and "research", delivering new AI-driven features and user experiences to maximize product-facing impact for Mercari's business.

We do this both through independent initiatives owned by our team, as well as by horizontally collaborating with product, engineering, and research teams across the entire organization.

As an MLOps engineer on the AI/LLM team, you will own how our machine learning and LLM models reach production and stay healthy there in our cloud-native environment. Your focus is the production serving, deployment, and operations that turn models into reliable, cost-efficient services, seamlessly integrating with our machine learning operations to serve tens of millions of users.

Data and Model Orchestration: Own the end-to-end orchestration of model inference, including integrating with DataServ for retrieval (BigQuery, BigTable, Valkey) and managing the Model Inference Gateway and Console.

Model Serving and Deployment: Own production model serving on the cloud-native NVIDIA and TPU stacks (Triton Inference Server, TensorRT-LLM, JAX/TPU Gateways). Manage model repositories, dynamic batching, and concurrent model execution. Build CI/CD, rollout, and rollback paths for safe, scalable model deployment, and automate provisioning and lifecycle management (Terraform, Kubernetes) so the platform scales seamlessly across teams.

Inference Performance Optimization: Profile and optimize deployments across LLM and non-LLM workloads, including model compilation, quantization, and batching strategies to hit latency and throughput targets while managing cost. Maintain performance baselines and regression detection.

Monitoring and Reliability: Build robust monitoring and alerting for model health and latency, including service-level metrics for the data retrieval and inference gateway layers. Define SLOs and own on-call and incident response for the serving layer.

Model Quality and Evaluation: Build automated evaluation and quality monitoring into the deployment path, regression and drift detection, offline/online evaluation, and LLM output quality checks, so models stay healthy long after launch.

Research-to-Production Enablement: Partner with ML engineers and researchers to turn experimental models into production-ready services, providing self-service workflows and abstractions that let teams deploy safely and quickly without deep infrastructure expertise.

Build and operate the production ML serving and Data Orchestration platform behind Mercari Group's AI and LLM features, serving tens of millions of users.

Drive the strategy for model inference at scale, bridging the gap between complex data retrieval and fast-moving ML model inference to ensure high-performance, cost-effective service delivery.

Shape Mercari's next-generation LLM serving stack, from inference optimization (quantization, dynamic batching, KV caching) to the evaluation and execution infrastructure needed for emerging agentic AI workloads.