Evaluate an existing application
Keep your application code and existing checks. Wrap the function you want to test in an experiment task, run it on a fixed dataset, and use Langfuse to compare outputs and review failures. This works for a single model call, a retrieval pipeline, or an agent that calls tools.
This example tests a small refund-policy function. The candidate fixes one case and breaks another, so the average score stays the same. Both SDK examples run without a model provider or an LLM judge. Replace the sample function with your application to evaluate fresh outputs from your own system.
Set up Langfuse
Use Langfuse v4 with Python SDK v4 or JS/TS SDK v5. Create a Cloud project or use your self-hosted instance, then get API keys from Settings → API Keys.
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"Use the URL for your data region or self-hosted instance. These scripts run the application and graders in your process and send test inputs, outputs, and scores to Langfuse.
Already have a local suite? Keep your JSON cases and grader in Pytest or Vitest. Already have outputs and scores? Publish those saved results without running the application or checks again.
Run two versions with the same checks
The scripts create a new dataset, add two reviewed cases, and pin the latest server-assigned creation timestamp returned by those writes. Both runs use that exact version, avoiding dependence on your computer's clock. In an existing project, reuse your dataset and store its approved version timestamp with your test configuration.
pip install langfuseSave as evaluate.py and run python evaluate.py:
from uuid import uuid4
from langfuse import Evaluation, get_client
def answer_question(question, version):
# Replace this sample function with your application's entry point.
# Its returned output, not a separately rewritten prompt, is evaluated.
if version == "baseline":
return {"refund_days": 30}
return {"refund_days": 14}
def grade(output, expected_output):
# Keep your existing business-rule checks here.
return output["refund_days"] == expected_output["refund_days"]
def exact_refund_window(*, output, expected_output, **kwargs):
return Evaluation(
name="refund_window",
value=int(grade(output, expected_output)),
)
langfuse = get_client()
dataset_name = f"refund-regression-{uuid4().hex[:8]}"
try:
langfuse.create_dataset(name=dataset_name)
created_versions = []
for case_id, question, days in [
("standard", "What is the standard refund window?", 30),
("sale", "What is the sale-item refund window?", 14),
]:
created_item = langfuse.create_dataset_item(
dataset_name=dataset_name,
input={"question": question},
expected_output={"refund_days": days},
metadata={"case_id": case_id},
)
created_versions.append(created_item.created_at)
version = max(created_versions)
dataset = langfuse.get_dataset(dataset_name, version=version)
print(f"Dataset: {dataset_name}; version: {version.isoformat()}")
for application_version in ["baseline", "candidate"]:
def task(*, item, **kwargs):
return answer_question(item.input["question"], application_version)
result = dataset.run_experiment(
name="Refund policy",
run_name=application_version,
task=task,
evaluators=[exact_refund_window],
metadata={
"application_version": application_version,
"evaluator_version": "refund-window-v1",
},
)
print(result.format())
finally:
langfuse.flush()npm install @langfuse/client @langfuse/otel @opentelemetry/sdk-node
npm install --save-dev tsx typescriptSave as evaluate.ts and run npx tsx evaluate.ts:
import { randomUUID } from "node:crypto";
import { LangfuseClient } from "@langfuse/client";
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { NodeSDK } from "@opentelemetry/sdk-node";
type Answer = { refund_days: number };
async function answerQuestion(
question: string,
version: string,
): Promise<Answer> {
// Replace this sample function with your application's entry point.
return { refund_days: version === "baseline" ? 30 : 14 };
}
function grade(output: Answer, expectedOutput: Answer): boolean {
// Keep your existing business-rule checks here.
return output.refund_days === expectedOutput.refund_days;
}
async function main() {
const otel = new NodeSDK({ spanProcessors: [new LangfuseSpanProcessor()] });
otel.start();
const langfuse = new LangfuseClient();
const datasetName = `refund-regression-${randomUUID().slice(0, 8)}`;
try {
await langfuse.api.datasets.create({ name: datasetName });
const createdVersions: string[] = [];
for (const [caseId, question, days] of [
["standard", "What is the standard refund window?", 30],
["sale", "What is the sale-item refund window?", 14],
] as const) {
const createdItem = await langfuse.api.datasetItems.create({
datasetName,
input: { question },
expectedOutput: { refund_days: days },
metadata: { case_id: caseId },
});
createdVersions.push(createdItem.createdAt);
}
const version = createdVersions.reduce((latest, current) =>
Date.parse(current) > Date.parse(latest) ? current : latest,
);
const dataset = await langfuse.dataset.get(datasetName, { version });
console.log(`Dataset: ${datasetName}; version: ${version}`);
for (const applicationVersion of ["baseline", "candidate"]) {
const result = await dataset.runExperiment({
name: "Refund policy",
runName: applicationVersion,
task: async (item) =>
answerQuestion(item.input.question, applicationVersion),
evaluators: [
async ({ output, expectedOutput }) => ({
name: "refund_window",
value: Number(grade(output, expectedOutput)),
}),
],
metadata: {
application_version: applicationVersion,
evaluator_version: "refund-window-v1",
},
});
console.log(await result.format());
}
} finally {
await otel.shutdown();
}
}
main().catch((error) => {
console.error(error);
process.exitCode = 1;
});Open Experiments in Langfuse and select the two runs. The standard-refund case changes from pass to fail; the sale-item case changes from fail to pass. Both runs score 50%, which is why reviewing individual cases matters. Follow Compare experiments to inspect the differences.
Connect your application
Replace answer_question or answerQuestion with an import from your application. Pass the same inputs and configuration that the deployed application receives, including retrieved context or tool results when applicable. For a multi-stage application, return the fields your checks need and trace the stages so you can inspect where a failure originated.
Keep grade as ordinary application code. You can call it from your existing test runner and from the SDK evaluator. Add semantic evaluators only for requirements those checks cannot decide, such as whether a paraphrase is supported by retrieved context. See RAG faithfulness evaluation and hallucination detection.
Evaluate saved outputs
To evaluate recorded answers without calling your application again, replace the task body with a lookup in your saved outputs, keyed by dataset item ID or case_id. Let missing outputs raise an error; do not substitute an empty answer. Record the original application version and mark the experiment metadata as execution_mode: replay.
For example, save this as saved-outputs.json:
{
"standard": { "refund_days": 30 },
"sale": { "refund_days": 30 }
}Use the following task in place of the task in the example above. Keep the same dataset and evaluators, set the run name to identify the recorded version, and add execution_mode: replay to the run metadata.
import json
from pathlib import Path
saved_outputs = json.loads(Path("saved-outputs.json").read_text())
def replay_task(*, item, **kwargs):
return saved_outputs[item.metadata["case_id"]]Pass task=replay_task to dataset.run_experiment.
import { readFileSync } from "node:fs";
import type { ExperimentTask } from "@langfuse/client";
const savedOutputs = JSON.parse(readFileSync("saved-outputs.json", "utf8"));
const replayTask: ExperimentTask = async (item) => {
const metadata = item.metadata as { case_id: string };
if (!Object.hasOwn(savedOutputs, metadata.case_id)) {
throw new Error(`Missing saved output: ${metadata.case_id}`);
}
return savedOutputs[metadata.case_id];
};Pass task: replayTask to dataset.runExperiment.
This measures the quality of recorded outputs under the current graders. It does not validate a new prompt, model, or code change. For that, run the changed application on the same inputs. To score existing traces directly, use Scores via API/SDK.
Publish saved check results
If your test runner already saved outputs and grader results, report them as an experiment on local data. The adapter below returns the saved output as the task result and the saved check as an evaluation. It calls neither your application nor your grader, and does not require a hosted dataset.
Export your existing results into this format. Keep stable case IDs and the original application and evaluator versions. Include the evidence reviewers need in the input or metadata. Only include data you intend to upload to your Langfuse project.
{
"application_version": "release-42",
"evaluator_version": "refund-window-v1",
"cases": [
{
"id": "standard",
"input": { "question": "What is the standard refund window?" },
"expected_output": { "refund_days": 30 },
"output": { "refund_days": 30 },
"passed": true,
"reason": "The refund window matches the reviewed policy."
},
{
"id": "sale",
"input": { "question": "What is the sale-item refund window?" },
"expected_output": { "refund_days": 14 },
"output": { "refund_days": 30 },
"passed": false,
"reason": "Sale items have a 14-day window."
}
]
}Use the credentials and dependencies above. The scripts validate the boolean check results before uploading and hash the complete artifact to identify its contents. Export task or grader errors separately and resolve them before publishing a complete scored run; never turn an error into a passing check.
import hashlib
import json
from pathlib import Path
from langfuse import Evaluation, get_client
artifact = Path("results.json").read_bytes()
saved = json.loads(artifact)
cases = saved["cases"]
ids = [case["id"] for case in cases]
if not ids or not all(ids) or len(set(ids)) != len(ids):
raise ValueError("Expected nonempty, unique case IDs")
for case in cases:
if type(case["passed"]) is not bool or case["output"] is None:
raise ValueError(f"Missing output or boolean check: {case['id']}")
by_id = {case["id"]: case for case in cases}
def saved_task(*, item, **kwargs):
return by_id[item["metadata"]["case_id"]]["output"]
def saved_check(*, metadata, **kwargs):
case = by_id[metadata["case_id"]]
return Evaluation(
name="existing_checks", value=int(case["passed"]), comment=case["reason"]
)
langfuse = get_client()
try:
result = langfuse.run_experiment(
name="Application checks",
data=[{
"input": case["input"],
"expected_output": case["expected_output"],
"metadata": {"case_id": case["id"]},
} for case in cases],
task=saved_task,
evaluators=[saved_check],
metadata={
"application_version": saved["application_version"],
"evaluator_version": saved["evaluator_version"],
"execution_mode": "imported_results",
"artifact_sha256": hashlib.sha256(artifact).hexdigest(),
},
)
print(result.format()) # Includes the experiment link.
finally:
langfuse.flush()Run python publish_results.py.
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { LangfuseClient } from "@langfuse/client";
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { NodeSDK } from "@opentelemetry/sdk-node";
type SavedCase = {
id: string;
input: unknown;
expected_output: unknown;
output: unknown;
passed: boolean;
reason: string;
};
async function main() {
const artifact = readFileSync("results.json");
const saved = JSON.parse(artifact.toString("utf8"));
const cases: SavedCase[] = saved.cases;
const ids = cases.map((item) => item.id);
if (!ids.length || !ids.every(Boolean) || new Set(ids).size !== ids.length) {
throw new Error("Expected nonempty, unique case IDs");
}
for (const item of cases) {
if (typeof item.passed !== "boolean" || item.output == null) {
throw new Error(`Missing output or boolean check: ${item.id}`);
}
}
const byId = new Map(cases.map((item) => [item.id, item]));
const otel = new NodeSDK({ spanProcessors: [new LangfuseSpanProcessor()] });
otel.start();
try {
const langfuse = new LangfuseClient();
const result = await langfuse.experiment.run({
name: "Application checks",
data: cases.map((item) => ({
input: item.input,
expectedOutput: item.expected_output,
metadata: { case_id: item.id },
})),
task: async (item) => {
const metadata = item.metadata as { case_id: string };
return byId.get(metadata.case_id)!.output;
},
evaluators: [async ({ metadata }) => {
const item = byId.get(metadata!.case_id)!;
return { name: "existing_checks", value: Number(item.passed), comment: item.reason };
}],
metadata: {
application_version: saved.application_version,
evaluator_version: saved.evaluator_version,
execution_mode: "imported_results",
artifact_sha256: createHash("sha256").update(artifact).digest("hex"),
},
});
console.log(await result.format()); // Includes the experiment link.
} finally {
await otel.shutdown();
}
}
main().catch((error) => {
console.error(error);
process.exitCode = 1;
});Run npx tsx publish-results.ts.
These are new experiment observations containing the recorded outputs and scores. Their timing describes the import, not the original application's latency or cost, and they do not reconstruct its intermediate tool calls. Retain the original run or trace identifiers in item metadata when available. If the original traces are already in Langfuse, attach scores to those traces instead of creating duplicate observations.
Open the experiment link printed by the script, then share the comparison or hand answers to a QA team. A completed import reports existing results; it does not establish that the application passes your release policy.
Add a regression gate
Choose a reviewed baseline, keep the dataset and evaluator versions fixed, and fail CI when a previously passing critical case fails. See Compare against an approved baseline for a gate that checks individual cases as well as aggregate scores.
Hosted datasets are useful for sharing and versioning cases. You can also run experiments on local data. In Langfuse v4, these experiments appear in the same experiment list without requiring a hosted dataset.