Google Stax: 자체 기준에 따라 모델 및 프롬프트 테스트

Google Stax: Testing Models and Prompts Against Your Own Criteria Learn how Google Stax tests AI models and prompts against your own criteria. Compare Gemini vs GPT with custom evaluators. Step-by-step guide for beginners Image by Author # Introduction If you're building applications with large language models (LLMs), you've probably experienced this scenario where you change a prompt, run it a few times, and the output feels better. But is it actually better? Without objective metrics, you are stuck in what the industry now calls "vibe testing," which means making decisions based on intuition rather than data. The challenge comes from a fundamental characteristic of AI models: uncertainty. Unlike traditional software, where the same input always produces the same output, LLMs can generate different responses to similar prompts. This makes conventional unit testing ineffective and leaves developers guessing whether their changes truly improved performance. Then came Google Stax, a new experimental toolkit from Google DeepMind and Google Labs designed to bring accuracy to AI evaluation. In this article, we take a look at how Stax enables developers and data scientists to test models and prompts against their own custom criteria, replacing subjective judgments with repeatable, data-driven decisions. # Understanding Google Stax Stax is a developer tool that simplifies the evaluation of generative AI models and applications. Think of it as a testing framework specifically built for the unique challenges of working with LLMs. At its core, Stax solves a simple but critical problem: how do you know if one model or prompt is better than another for your specific use case? Rather than relying on general criteria that may not reflect your application's needs, Stax lets you define what "good" means for your project and measure against those standards. // Exploring Key Capabilities - It helps define your own success criteria beyond generic metrics like fluency and safety - You can test different prompts across various models side-by-side - You can make data-driven decisions by visualizing gathered performance metrics, including quality, latency, and token usage - It can run assessments at scale using your own datasets Stax is flexible, supporting not only Google's Gemini models but also OpenAI's GPT, Anthropic's Claude, Mistral, and others through API integrations. # Moving Beyond Standard Benchmarks General AI benchmarks serve an important purpose, like helping track model progress at a high level. However, they often fail to reflect domain-specific requirements. A model that excels at open-domain reasoning might perform poorly on specialized tasks like: - Compliance-focused summarization - Legal document analysis - Enterprise-specific Q&A - Brand-voice adherence The gap between general benchmarks and real-world applications is where Stax provides value. It enables you to evaluate AI systems based on your data and your criteria, not abstract global scores. # Getting Started With Stax // Step 1: Adding An API Key To generate model outputs and run evaluations, you'll need to add an API key. Stax recommends starting with a Gemini API key, as the built-in evaluators use it by default, though you can configure them to use other models. You can add your first key during onboarding or later in Settings. For comparing multiple providers, add keys for each model you want to test; this enables parallel comparison without switching tools. Getting an API key // Step 2: Creating An Evaluation Project Projects are the central workspace in Stax. Each project corresponds to a single evaluation experiment, for example, testing a new system prompt or comparing two models. You'll choose between two project types: | Project Type | Best For | |---|---| | Single Model | Baselining performance or testing an iteration of a model or system prompt | | Side-by-Side | Directly comparing two different models or prompts head-to-head on the same dataset | Figure 1: A side-by-side comparison flowchart showing two models receiving the same input prompts and their outputs flowing into an evaluator that produces comparison metrics // Step 3: Building Your Dataset A solid evaluation starts with data that is accurate and reflects your real-world use cases. Stax offers two primary methods to achieve this: Option A: Adding Data Manually in the Prompt Playground If you don't have an existing dataset, build one from scratch: - Select the model(s) you want to test - Set a system prompt (optional) to define the AI's role - Add user prompts that represent real user inputs - Provide human ratings (optional) to create baseline quality scores Each input, output, and rating automatically saves as a test case. Option B: Uploading an Existing Dataset For teams with production data, upload CSV files directly. If your dataset doesn't include model outputs, click "Generate Outputs" and select a model to generate them. Best practice: Include the edge cases and conflictin