ESProfiler Handbook
LLM Ops

Evaluating Agents

How to evaluate LLM agents with Langfuse — datasets, evaluators, and experiments.

We use Langfuse to evaluate LLM agents in the ESProfiler ecosystem.

Evaluations follow three steps:

  1. Create datasets — test cases (input / expectedOutput / metadata)
  2. Create evaluators — scoring definitions (mostly LLM-as-judge)
  3. Run experiments — prompt + dataset + evaluators
PieceRole
DatasetReusable test cases
EvaluatorScores one quality dimension of an agent output
ExperimentRuns a prompt against a dataset and applies evaluators

Existing resources in ESP Development:

A walkthrough video of the full Langfuse UI will be recorded soon and added to this page later.

1. Creating Datasets

An evaluation dataset is a collection of test cases. Each item typically has:

FieldPurpose
inputWhat the agent/prompt receives at runtime
expectedOutputGolden answer or evaluation guidance
metadataFiltering, debugging, and audit context (not fed to the agent)

1.1 Create your first dataset (5 steps)

  1. Pick one agent — see Agents.
  2. Decide expected-output style — golden answer for deterministic tasks; judge guidance for open-ended ones.
  3. Write 3–5 items — input keys must match that agent’s prompt variables; add expectedOutput and optional metadata.
  4. Anonymize if data came from production — never upload raw tenant/PII.
  5. Create and upload — UI for tiny flat cases; Python SDK for nested JSON.

Then spot-check 2–3 items in Langfuse before attaching evaluators.

1.2 Dataset naming

{agentName}/{datasetRole}
DatasetRoleSize guidance
{agent}/smoke_testQuick sanity checks after prompt or wiring changes~5–15 items
{agent}/gate_testBroader frozen set for prompt comparison and release decisions~20–50 items

Examples: conversation-namer/smoke_test, findings/gate_test.

Keep both sets reviewed and anonymized. Update them deliberately — do not treat one as a staging queue for the other.

1.3 Upload overview

  • UI — Datasets → New dataset → add items (or CSV for flat strings). Best for small / simple cases.
  • Python SDK — preferred for nested JSON (Findings transcripts, structured guidance). Keys: Langfuse UI → Settings → API Keys.
  • Reference: Langfuse Datasets

1.4 Dataset reference (expand as needed)

2. Creating Evaluators

Evaluators are the scoring definitions you attach when you run prompt experiments.

Today we mostly set up LLM-as-judge evaluators in Langfuse.

Code-based checks still matter for deterministic rules, but we typically implement them in the application and/or unit/integration tests in platform-api, rather than as the primary Langfuse experiment evaluators. Langfuse also supports Code Evaluators if needed later.

Rule of thumb:

  • Needs reading comprehension / judgment → LLM-as-judge (Langfuse)
  • A junior engineer could assert it in a test → code (app or unit tests)

2.1 Create your first LLM-as-judge (5 steps)

  1. Pick one dimension — e.g. “grounded in transcript” or “aligns with directive”.
  2. Choose score shape — boolean / categorical / numeric.
  3. Write the judge prompt — explicit pass/fail or category rules; one job only.
  4. Create it in Langfuse — map {{input}} / {{output}} (and {{expected_output}} only if needed).
  5. Verify mapping — use Prompt Preview; spot-check that variables populate as expected.

Start with 2–3 focused judges per agent. Add more only when debugging a specific failure class.

Findings starter pack:

  1. findings_agent.grounding — Boolean
  2. findings_agent.directive_alignment — Categorical (fail / partial / pass)

2.2 Naming

{agentName}.{dimension}

Examples: conversation_namer.title_quality, findings_agent.grounding, findings_agent.directive_alignment.

2.3 Create in Langfuse (UI)

  1. Ensure an LLM Connection exists (Settings → LLM Connections). The judge model must support structured output.
  2. Open Evaluators → + Set up Evaluator.
  3. Pick a managed template, or Custom and paste your judge prompt with {{variables}}.
  4. Choose score type (boolean / categorical / numeric). For categorical, define labels and numeric mapping.
  5. Map variables to Input / Output / Expected output (add JSONPath if needed).
  6. Save. Attach these evaluators when running experiments.

Official guide: LLM-as-a-Judge.

2.4 Evaluator reference (expand as needed)

3. Running Experiments

Use a dataset and evaluators together in a Langfuse Prompt Experiment to compare prompt versions and decide whether to ship a change.

Example completed runs: findings/gate_test experiments.

Official docs: Experiments via UI · Experiments data model

3.1 What a Langfuse experiment is

ConceptMeaning
DatasetFrozen test cases (input, optional expectedOutput, metadata)
PromptVersioned prompt from Prompt Management
Experiment (Prompt Experiment)Runs the selected prompt on each dataset item
EvaluatorScores each experiment item output (LLM-as-judge and/or code)
Experiment comparisonSide-by-side aggregate + item-level score comparison across runs
Dataset item input
        │
        ▼
Prompt version (variables filled from input)
        │
        ▼
Model output
        │
        ▼
Evaluators attach scores
        │
        ▼
Compare runs → promote or reject prompt

Important: one experiment can attach multiple evaluators. Do not create one experiment per score.

3.2 Prerequisites

Before running an experiment, confirm:

  1. Prompt in Prompt Management with {{variables}} matching dataset input keys
  2. Dataset uploaded (smoke_test or gate_test) — see §1
  3. LLM connection configured; default evaluation model supports structured output for judges
  4. Evaluators created and able to target Experiments — see §2

3.3 Run a Prompt Experiment (UI)

  1. Go to Datasets → open the dataset (e.g. findings/gate_test) → spot-check 1–2 items
  2. Click Start Experiment / Run Experiment → Prompt Experiment → Create
  3. Configure:
    • Experiment name (see naming below)
    • Prompt + prompt version
    • LLM connection / model settings
    • Dataset (usually already selected)
    • Optional: structured output schema (recommended for Findings: summary, report, findings)
    • Evaluators to attach (all gate evaluators)
  4. Click Create

Langfuse runs the prompt per item, stores outputs, runs evaluators asynchronously, and shows aggregate scores. Runtime depends on dataset size, prompt length, and judge count.

3.4 Experiment naming

{agent}-{role}-{promptVersion}-{yyyymmdd}

Examples:

  • findings-baseline-v12-20260729
  • findings-candidate-v13-20260729

For prompt gates: baseline = current production prompt; candidate = proposed version; same dataset + same evaluators for both.

3.5 Compare experiments

After runs complete:

  1. Open Experiments (or the dataset’s Experiments tab)
  2. Select baseline and candidate runs
  3. Compare aggregate scores, item-level regressions (especially boolean fails), and judge comments on failures

Always spot-check a few failed items manually before promoting.

3.6 Experiment reference (expand as needed)

4. Related docs

ESPi Architecture & Query Flow

This document provides a high-level overview, a component breakdown, and detailed diagrams of how ESPi (ESPROFILER Intelligence), the AI co-pilot and agent orchestrator built into ESPROFILER, handles and processes user queries.

Message Queues

Copyright © 2026