Airecmark Logo
info DEMO DATA: scores shown are template sample values pending independent evaluation · as of 2026-09-05 · Methodology
arrow_back EVAL_DIR / TERMINAL_AGENTS / SPEC-ID: ANTH-CLD-CODE-0229 ACTIVE EVALUATION
BENCHMARK RUN: 2026.Q1.SWE-V
Terminal Native Agent Research Preview v0.2.29 SOC 2 Type II Certified MCP 1.0 Compliant
terminal

Claude Code

verified

By Anthropic · Powered by Claude 3.7 Sonnet (Extended Thinking Engine)

Autonomous command-line software engineering harness operating directly inside developer subshells. Bridges deep AST-level project context via MCP protocol and dynamic reasoning budgets (1k–64k reasoning tokens) with native git graph orchestration and shell telemetry.

terminal Deploy via CLI npm i -g @anthropic-ai/claude-code
Production Evaluated
Airecmark Composite Index Alpha Rank #1
93.8 / 100
arrow_upward +42.4%
Velocity Percentile: 99.5th
Baseline: 50.0 Median: 78.4 Max: 93.8
SWE-bench Verified
84.8% (Pass@1 Loop)
Cache Hit Rate
88.2% (Epistemic)
Latency P95 TTFT
110ms (Cached)
Data Retention
0-Days (ZDR Ready)
terminal CLI subshell: node v18+ | check_circle Git integration: Native
Empirical Index Metrics

Macro Intelligence & Performance Benchmarks

Strict regression results derived from 4,500 continuous eval iterations across SWE-bench Verified, multi-turn AST refactor suites, and enterprise repo graph resolutions.

Baseline: SWE-Bench (Verified) Sampling: Temperature 0.2
SWE-bench Verified code_blocks
84.8% +14.5% vs Aider

Single-attempt autonomous resolution with agentic bash loop & dynamic extended thinking enabled.

Prompt Cache Hit Efficiency bolt
88.2% 90% Discount

Epistemic repository prefix caching preserving full AST memory across repeated terminal loops.

Subshell Autonomy smart_toy
99.4% Zero Escape

Deterministic tool calling compliance with strict sandboxing and user confirmation gateways.

Cached P95 TTFT speed
110ms 200k context

Time-to-first-token on deep repo sweeps with prompt caching hot slots activated in Anthropic infra.

table_chart Direct Agent Peer Matrix
Ranked by Composite SWE & Autonomy Index
Agent / Tool Runtime Target SWE-bench Verified Reasoning Budget Context Engine Airecmark Score Status
terminal Claude Code (Anthropic) Leader Terminal CLI (Native Shell) 84.8% Dynamic 1k–64k 200k + Hot Cache 93.8 Active (Eval)
code Cursor (Anysphere) Custom VS Code Fork 74.2% Fixed o1/Sonnet Local Vector Embeddings 94.2 GA
alt_route Aider Python CLI / Terminal 72.9% BYO API Key Repo Map + Tree-sitter 92.4 Open Source
air Windsurf (Codeium) Standalone IDE / Cascade 70.8% Cascade Flows Multi-file Context Engine 91.8 GA
terminal Copilot Workspace / Edits IDE Extension + Web 64.5% Preset OpenAI Blackbox RAG 91.0 Enterprise
Deep Dive & Analysis

Autonomous Subshell Execution Architecture

claude — agent session [swe_bench_repo]
THINKING_BUDGET: 32,768 TOKENS
$ claude "Resolve issue #1428: race condition in connection pooling teardown"
sync [Thinking] Analyzing connection pool lifecycle in src/pool.ts, tracing async socket close events...
✓ Read 18 project files (prompt cache hit 92.4%)
❯ Bash tool proposed: git checkout -b fix/pool-teardown-race
❯ Applying unified diff to src/pool.ts (+14 lines, -6 lines)
✓ Executed `npm test -- test/pool.test.ts` (14 passing in 420ms)
Session cost: $0.142 (Input: 184k cached, 2k new | Output: 1.4k thinking)
psychology

Hybrid Reasoning & Extended Thinking Budget

Operates atop Claude 3.7 Sonnet's hybrid reasoning engine. Unlike fixed reasoning models (e.g. OpenAI o1 series), Claude Code allows programmatic or manual thinking budgets from 1,000 to 64,000 tokens per subtask, enabling rapid single-shot edits or deep architectural multi-file proofs before bash operations are proposed.

hub

Model Context Protocol (MCP) Integration

Native first-class client implementation of Anthropic's open standard MCP. Seamlessly binds local SQLite schemas, external PostgreSQL databases, GitHub PR review agents, and internal API documentation servers directly into the agent's scratchpad without custom glue code.

security

Human-in-the-Loop Shell Guardrails & Security

Automated subshell execution operates with configurable safety tiers: default interactive confirmation for all bash writes/network calls, or auto-approved sandboxed read operations. Enterprise configurations feature Zero Data Retention (ZDR) guarantees where model completions and repo buffers are purged immediately post-inference.

gavel Analyst Consensus Verdict
Consensus: OVERWEIGHT
"Claude Code represents the definitive shift from IDE editor forks to autonomous terminal-native agent loops. By combining Claude 3.7 Sonnet’s hybrid reasoning with deep prompt caching economics, Anthropic has eliminated editor lock-in while setting the new industry gold standard for real-world SWE task resolution."
  • • Unrivaled SWE-bench Verified score (84.8% pass@1 loop)
  • • Native prompt cache reduces token run-rate by up to 90%
  • • Zero IDE lock-in: works across Neovim, VS Code, tmux, remote SSH
  • • Fine-grained control over reasoning token depth
  • • High CLI comfort required; no inline visual UI gutter diffs
  • • Complex unattended loops can exhaust token quotas rapidly
  • • Requires active Node 18+ runtime and Anthropic API provisioning

Commercial Tiers

Pay-as-you-go
Direct Anthropic API Usage Based
Input: $3.00 / MTok ($0.30 cached)
Output: $15.00 / MTok (incl. reasoning)
90% discount on cache hits
Claude Pro / Team $20 – $30 / mo
Native CLI auth via existing browser session with rate limits mapped to subscription tier.
Enterprise Tier Custom Contract
Zero Data Retention (ZDR), dedicated rate limits, custom MCP protocol relays, and SAML SSO.

Technical Specifications

Package Name
@anthropic-ai/claude-code
Runtime Base
Node.js ≥ 18.0.0
Default Model
claude-3-7-sonnet-latest
Extended Thinking
Supported (1k–64k)
Context Window
200,000 Tokens
Memory System
CLAUDE.md local indexing
Shell Support
zsh, bash, fish, PowerShell
Telemetry / Tracking
Opt-out via --no-telemetry
Vector Breakdown 6 Dimensions
Code Generation & Reasoning 99.1 / 100
Tool Calling & Subshell Autonomy 98.4 / 100
Prompt Caching Economics 96.5 / 100
Context Preservation (200k) 95.0 / 100
Enterprise Governance & Privacy 93.2 / 100
Setup & Developer DX 92.8 / 100
Ready to execute Claude Code in your repository?

Requires Node.js 18+ and an Anthropic API key with Claude 3.7 Sonnet access enabled.

Analyst Trade-off Summary

Strengths & Limitations

thumb_upStrengths
  • check_circleHeadless 重构与测试修复
  • check_circle终端原生工作流
report_problemLimitations
  • error_outline无图形界面