Build realistic tasks
We collect agent tasks that require planning, tool use, information synthesis, and artifact delivery across realistic desktop and web environments.
Agent Evaluation Suite
Evaluating agents where real work gets done.
AIPERT builds benchmark tasks, scoring rubrics, and inspection tools for AI agents that operate across files, browsers, code, documents, data, and long-horizon workflows.
Evidence-firstEvery score is connected to artifacts, logs, traces, or human-review checkpoints.
Real workflowsTasks are modeled after professional work rather than short prompt puzzles.
Reproducible runsEvaluation cases are designed for deterministic replay, versioning, and trace analysis.
Reviewable scoringRubrics separate objective checks from expert judgement, risks, and ambiguity.
Benchmark Design
From task construction to final judgement, the benchmark keeps evidence in the loop.
Define observable success
Each task includes expected outputs, constraints, validation hooks, and review notes so performance can be inspected instead of guessed.
Run, score, and audit
Agents are evaluated with execution traces, intermediate artifacts, final deliverables, and expert review where pure automation is not reliable enough.
Evaluation Philosophy
Harder than a chat benchmark, clearer than a vibes test.
AIPERT focuses on whether an agent can finish work under constraints: read source material, operate tools, revise artifacts, recover from errors, and leave an auditable record of what happened.
Task fidelity
Scenarios preserve the messy structure of real work while keeping evaluation boundaries explicit.
Rubric clarity
Scoring distinguishes exact checks, partial credit, unsafe behavior, and unresolved review items.
Trace literacy
Evaluation includes the path an agent took, not only the final answer it produced.
Release discipline
Public claims will be tied to documented task suites, versioned cases, and reproducible evidence.
Task Domains
Designed for agents that need to act, not just answer.
Software engineering
Repository navigation, issue fixing, regression testing, documentation, and code review.
Data and research
Spreadsheet analysis, web research, citation-aware synthesis, and structured extraction.
Document workflows
Slides, reports, PDFs, spreadsheet artifacts, formatting QA, and revision tasks.
Browser operations
Multi-step browsing, form interaction, page inspection, comparison, and evidence capture.
Tool orchestration
Command-line work, local files, service checks, deployment routines, and recovery from errors.
Reliability and safety
Instruction following, boundary handling, risk awareness, and refusal quality under pressure.
Partners & Contributors
Building verifiable agent evaluation with universities, researchers, and domain experts.
AIPERT's collaboration network will include university labs, workflow experts, evaluation engineers, and student contributors. Confirmed institutions and collaborators will be published here.
Partner institutionsUpdating
University LabsComputer Science SchoolsAI InstitutesSoftware SchoolsData Science CentersInterdisciplinary AI CentersIndustry Intelligence LabsEvaluation Research Groups
Research collaborators
Agent evaluation methodology, task construction, and result analysis.
Domain experts
Real workflows, industry constraints, and expert review standards.
Evaluation engineers
Environment packaging, automated checks, trace capture, and regression validation.
Student contributors
Task curation, annotation review, documentation, and case maintenance.
Roadmap
What will be published
Preview
Benchmark overview, evaluation principles, and representative task categories.
Task suite
Versioned evaluation cases with rubrics, environment notes, and validation artifacts.
Reports
Model and agent results with trace-backed analysis, limitations, and review notes.
Agent evaluation should make progress visible: what was attempted, what worked, what failed, and what still needs human review.
Resources
Research materials and benchmark artifacts will land here as they are released.
Contact
Building reliable agent benchmarks is a team sport.
AIPERT welcomes conversations around task design, evaluation methodology, trace analysis, and domain-specific benchmark construction.
contact@aipert.top