AstroResearch/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Build, Lint & Test Commands

# Build (debug)
cargo build

# Build (release with optimizations)
cargo build --release

# Build (release-min profile: size-optimized LTO)
cargo build --profile release-min

# Run (debug, starts server on http://localhost:8000)
cargo run

# Run with Obscura in-process browser (no external binaries needed)
cargo run --features obscura-inprocess

# Run CLI binary
cargo run --bin astroresearch_cli

# Run health check tool
cargo run --bin health_check          # read-only scan
cargo run --bin health_check -- --fix # auto-repair

# Lint
cargo clippy

# Format
cargo fmt

# All tests
cargo test

# Unit tests only
cargo test --lib

# Run a specific test
cargo test test_name

# Frontend (cd dashboard first)
npm run dev       # HMR dev server on :5173, proxies /api to :8000
npm run build     # TypeScript check + Vite production build → dashboard/dist/
npm run lint      # ESLint

Architecture Overview

Stack: Rust Axum backend (port 8000) + React/Vite/TypeScript frontend (port 5173 in dev). In production, the Rust binary serves the pre-built dashboard/dist/ via ServeDir and ServeFile fallback, so there is a single process.

Source Layer Map

src/
├── main.rs          # Axum server entry: logging, DB pool, migrations, vec0 table,
│                    # client/service construction, route registration, AppState assembly
├── lib.rs           # Config struct + from_env() loading from .env
├── api/             # HTTP handlers, AppState, StandardPaper type
│   ├── mod.rs       # AppState (shared state), StandardPaper, handlers re-exports
│   ├── agent.rs     # SSE chat_agent endpoint, session CRUD, metrics, audit log
│   ├── papers.rs    # Search, download, parse, translate, embed, citations, library, export
│   ├── notes.rs     # Highlight/note CRUD
│   ├── sync.rs      # Meta-sync and asset-batch endpoints
│   ├── targets.rs   # Target query/associate/extract, RAG chat, figure chat
│   └── helpers.rs   # Shared DB helpers, format conversion, path validation
├── agent/           # ReAct-based research agent (LLM-driven tool-use loop)
│   ├── tools/       # AgentTool trait, ToolRegistry, 25+ tool implementations per domain file
│   ├── runtime/     # ReAct loop, streaming, session/context, token budget, error recovery,
│   │                #   permission checker, file cache, system prompt assembly, circuit breaker
│   ├── compact/     # Context compression (micro/auto/manual layers + collapse)
│   ├── memory/      # Persistent memory manager: extraction, dedup, decay, age, guardrails
│   ├── hooks.rs     # Lifecycle events (PreToolUse/PostToolUse/Stop/etc.)
│   ├── skills.rs    # SkillRegistry: hot-loads SKILL.md files from skills/ directory
│   ├── subagent.rs  # Context-isolated sub-agent runner (subagent tool)
│   ├── team/        # Multi-agent team: file-based inbox, lead/teammate coordination
│   ├── background.rs# BgNotificationQueue for async slow-task (download/parse) notifications
│   ├── task_board.rs# Persistent task board (agent_tasks table)
│   ├── trajectory.rs# Session trajectory recording for audit/debug
│   └── terminal.rs  # Escape sequence filter for ANSI-heavy tool outputs
├── clients/         # External API wrappers
│   ├── llm.rs       # LlmClient (OpenAI-compatible chat + streaming), EmbeddingClient
│   ├── ads.rs       # NASA ADS API
│   ├── arxiv.rs     # arXiv Atom XML API
│   └── qiniu.rs     # Qiniu cloud storage
├── services/        # Business logic
│   ├── search.rs    # Unified cross-source search (ADS + arXiv dedup)
│   ├── download.rs  # PDF/HTML download with anti-bot measures and fallback chain
│   ├── parser/      # HTML/PDF → Markdown parsers (A&A, IOP, ar5iv, generic, PDF via MinerU)
│   ├── translation.rs# LLM bilingual translation with Trie-based astronomy glossary
│   ├── rag.rs       # Embedding ingest + vector similarity retrieval + LLM answer generation
│   ├── target.rs    # Celestial target extraction (IAU name regex) + CDS Sesame lookup
│   ├── chunker.rs   # Markdown text chunking for embedding
│   ├── batch/       # Meta-sync (ADS bulk harvest) and asset-batch processing engines
│   ├── query_parser.rs# Advanced search query syntax parser
│   └── logging.rs   # Pretty console + rolling file logger
└── bin/
    ├── health_check.rs # Library consistency checker and auto-repair
    ├── cli.rs          # CLI interface
    └── reparse.rs      # Re-parse existing library items

AppState — Central Shared State

All handlers access state via Arc<AppState>. Key fields:

• db: SqlitePool — SQLite connection pool (5 max connections, foreign keys enforced)
• llm: LlmClient / embedding: EmbeddingClient — OpenAI-compatible LLM clients
• ads: AdsClient / arxiv: ArxivClient — academic search clients
• skill_registry: Arc<RwLock<SkillRegistry>> — hot-reloaded agent skills
• memory_manager: Arc<MemoryManager> — persistent agent memory (MEMORY.md + decay)
• cancelled_runs: Arc<Mutex<HashSet<String>>> — agent cancellation tokens
• harvest_status / batch_status — async batch operation status tracking

Agent System Design

The agent (src/agent/) implements a ReAct (Thought → Action → Observation) loop:

1. AgentRuntime (runtime/mod.rs) orchestrates the loop: session create/resume → context build → ReAct loop → finalize
2. Streaming: LLM response is streamed via SSE (AgentStreamEvent) — thought, tool_call (with id), tool_result (with tool_call_id), text_delta, usage, error, done. Tool calls execute in parallel.
3. Tools: Each tool implements AgentTool trait (name, description, JSON Schema parameters, execute). Core tools: read_file, grep_files, glob_files, run_bash, file_write, file_edit, search_papers, download_paper, parse_paper, get_paper_content, rag_search, query_target, save_note, todo_write, compress_context, load_skill, subagent, ask_user, save_memory. Plus background tools (bg_task_run, bg_task_check) and team tools (spawn_teammate, send_teammate_message, team_broadcast, check_team_inbox).
4. Thinking mode: enable_thinking flag propagates from AgentChatRequest → AgentConfig → ToolContext → LlmClient::chat_stream. Only enabled for Qwen/DashScope backends; frontend-controlled via the thinking request field.
5. Tool call ID tracking: LLM may not return tool_call IDs — LlmClient generates UUID fallbacks. ToolCall and ToolResult SSE events carry matching IDs for precise frontend pairing.
6. ToolContext (tools/mod.rs): Injected into every tool execution — holds app_state, session_id, sse_tx (for intermediate events), enable_thinking, read_file_state (file cache for dedup), silent (sub-agents skip permission prompts).
7. Context compression: Four layers — micro (placeholder replacement), snip (old-message truncation), auto (LLM summarization), aggro_micro (aggressive placeholder). Protected by CompactionCircuitBreaker. Transcripts persisted in agent_messages table, not filesystem snapshots.
8. Skills (skills.rs): Two-layer loading — system-reminder lists names (~20 tokens each), LLM calls load_skill to inject full SKILL.md content
9. Sub-agents (subagent.rs): subagent tool spawns a context-isolated sub-agent with its own ReAct loop. Sub-agent messages (system/user/assistant/tool) are persisted to agent_messages with agent_name identifier. Returns final summary + activity log. SSE progress forwarded to parent via ToolContext.
10. Memory (memory/): File-based persistent memory (MEMORY.md). MemoryManager handles extraction from conversation, dedup, recency decay, age-based pruning, and guardrails. Tools: save_memory, load_memory (auto-injected in system prompt).
11. Teams (team/): File-based inbox directory per session for lead/teammate message passing
12. Background tasks (background.rs): Slow ops (download, parse) can run async; results inject via BgNotificationQueue before next LLM call

Environment variables for agent tuning: AGENT_MAX_STEPS (default 8), AGENT_TOOL_TIMEOUT_SECS (default 120), AGENT_MAX_TOOL_OUTPUT_CHARS (default 4000), AGENT_CONTEXT_CHAR_LIMIT (default 16000), AGENT_TOKEN_SOFT_LIMIT / AGENT_TOKEN_HARD_LIMIT.

Database

SQLite via sqlx::sqlite. Migrations in migrations/ are auto-run on startup (sqlx::migrate!("./migrations")). Key tables: papers, citations_references, notes, agent_sessions, agent_messages, agent_tasks, agent_audit_log, paper_chunks_content. Vector embeddings use sqlite-vec (vec_paper_chunks virtual table, auto-registered before any DB connection).

The embedding dimension is controlled by EMBEDDING_DIM env var (default 1536). On dimension mismatch, the vec table and chunk content are dropped and recreated.

Frontend (dashboard/)

React 19 + TypeScript + Vite + Tailwind CSS 4. Features are organized by domain:

• features/search/ — Cross-source paper search panel
• features/library/ — Local library management
• features/reader/ — Bilingual reader with highlight annotations (KaTeX for math)
• features/citation/ — Canvas-based force-directed citation graph
• features/sync/ — Batch sync control panel
• features/agent/ — Agent chat: ResearchAgentPanel (timeline view with thought/tool_call/answer/subagent), AgentMetricsPanel (tool stats), AskUserQuestionCard (interactive Q&A), AuditLogViewer
• features/settings/ — System configuration

Dependencies: react-markdown + rehype-katex + remark-math for Markdown/LaTeX rendering, framer-motion for animations, lucide-react for icons.

Obscura In-Process Browser

The obscura-inprocess feature compiles obscura-browser and obscura-net directly into the binary, eliminating the need for external browser binaries. This is used for bypassing Cloudflare/WAF on PDF download. Enabled via --features obscura-inprocess.

Astronomy Glossary (dictionary.txt)

A 1MB+ bilingual astronomy terminology file loaded at startup into a Trie tree for longest-match glossary construction, used by the translation service to guide LLM translations with domain-accurate term mappings.

Code Conventions

• Use anyhow for application errors, thiserror for library-style typed errors
• Environment variables via dotenvy + std::env::var, with defaults in Config::from_env()
• SQL queries use parameterized bindings (sqlx::query("...").bind(...)) — never string interpolation
• API handlers take State(Arc<AppState>) and return Axum-compatible responses
• Agent tools implement AgentTool trait; new tools register in ToolRegistry::new()
• Front-end build is triggered by build.rs (auto npm install + build when dashboard/src/ changes)