feat: LAMOST DR12-14 与子版本体系接入、观测层安全加固与并发异步化

- LAMOST 新增 DR12/13/14 及子版本(v0/v1.0/v1.1/v2.0)维度,Internal 发布标记需登录认证并前端灰显,release×subtype 交叉约束下沉至 capabilities 统一声明
- ObservationFetcher trait 扩展版本/认证/交叉约束能力声明,version 参数贯穿 client→service→API→Agent tool→前端全链路
- 安全:observation cache SQL 全参数绑定 + LIKE 转义、cone_cache_hash 加长度前缀防碰撞、DESI survey/program 白名单防穿越
- 异步化:persist/cached_files_total_size/maybe_persist_tool_result迁移到 tokio::fs;cancelled_runs 与 session_permission_checkers改用 DashMap;auth 读锁优先 + 60s 节流
- Gaia 去 native-tls 改禁用连接池规避 UnexpectedEof,reqwest 移除 native-tls feature
- 重构:Source/ProductType from_str 集中解析、download 模块拆分为 try_download_pdf/html、AgentRuntime::init 抽取共享逻辑
- 部署:新增 deploy.sh 一键打包推送脚本、catch-panic 启用
This commit is contained in:
fmq
2026-07-07 21:16:46 +08:00
parent 2f1fd19d74
commit 2c8d0b8f8b
75 changed files with 2482 additions and 1370 deletions
+117 -30
View File
@@ -42,6 +42,9 @@ cargo test --lib
# Run a specific test
cargo test test_name
# Coverage (requires cargo-llvm-cov)
cargo llvm-cov --fail-under-lines 80
# Frontend (cd dashboard first)
npm run dev # HMR dev server on :5173, proxies /api to :8000
npm run build # TypeScript check + Vite production build → dashboard/dist/
@@ -62,46 +65,103 @@ src/
├── main.rs # Axum server entry: logging, DB pool, migrations, vec0 table,
│ # client/service construction, route registration, AppState assembly
├── lib.rs # Config struct + from_env() loading from .env
├── api/ # HTTP handlers, AppState, StandardPaper type
│ ├── mod.rs # AppState (shared state), StandardPaper, handlers re-exports
├── api/ # HTTP handlers, AppState, re-exports via handlers namespace
│ ├── mod.rs # AppState (shared state), handler re-exports, permission/event types
│ ├── agent.rs # SSE chat_agent endpoint, session CRUD, metrics, audit log
│ ├── papers.rs # Search, download, parse, translate, embed, citations, library, export
│ ├── catalog.rs # VizieR TAP catalog queries + cone search endpoints
│ ├── observation.rs # Observation data search/download (LAMOST/Gaia/SDSS/DESI)
│ ├── notes.rs # Highlight/note CRUD
│ ├── sync.rs # Meta-sync and asset-batch endpoints
│ ├── targets.rs # Target query/associate/extract, RAG chat, figure chat
── helpers.rs # Shared DB helpers, format conversion, path validation
── auth.rs # Login/logout/auth-check (token-based, session memory)
│ ├── permissions.rs # Agent tool permission rule management
│ ├── search.rs # Search history endpoint
│ ├── error.rs # Unified API error type (status code + user-safe message)
│ └── helpers.rs # Shared DB helpers, format conversion, path validation (private)
├── agent/ # ReAct-based research agent (LLM-driven tool-use loop)
│ ├── tools/ # AgentTool trait, ToolRegistry, 25+ tool implementations per domain file
│ ├── runtime/ # ReAct loop core: session, context, streaming, executor, token_budget,
│ │ # permission, permission_profile, checkpoint, circuit_breaker, error_recovery,
│ │ # hardline, partitioner, file_cache, system_prompt, finalize, denial_tracker
│ ├── tools/ # AgentTool trait, ToolRegistry, tool implementations per domain
│ ├── mod.rs # AgentTool trait, ToolContext (session_id, sse_tx, silent, etc.)
│ │ ├── filesystem/ # read_file, grep_files, write, edit
│ │ ├── astro/ # search_papers, download_paper, parse_paper, rag_search,
│ │ │ # query_target, save_note, analyze_image, research/*
│ │ ├── ask_user.rs # Permission-gated interactive user question
│ │ ├── background.rs# bg_task_run, bg_task_check (async slow tasks)
│ │ ├── compress.rs # Context compression trigger
│ │ ├── memory.rs # save_memory, load_memory
│ │ ├── persist.rs # Data persistence (used by memory/compaction)
│ │ ├── skill.rs # load_skill
│ │ ├── subagent.rs # Context-isolated sub-agent spawner
│ │ ├── team.rs # spawn_teammate, send_teammate_message, etc.
│ │ └── todo.rs # todo_write (task tracking)
│ ├── runtime/ # ReAct loop core
│ │ ├── mod.rs # AgentRuntime: session create/resume → context → ReAct → finalize
│ │ ├── executor/ # Step executor + helpers (tool dispatch, parallel tool calls)
│ │ ├── streaming.rs # SSE AgentStreamEvent types
│ │ ├── streaming_executor.rs # SSE-emitting executor wrapper
│ │ ├── session.rs # Session CRUD, DB persist, agent_config
│ │ ├── token_budget.rs # Soft/hard token limits, tiktoken counting
│ │ ├── permission.rs # Deny > Allow > Ask rule chain, pattern matching
│ │ ├── permission_profile.rs # Named profiles (research, readonly)
│ │ ├── checkpoint.rs # Rewind/retry state snapshots
│ │ ├── circuit_breaker.rs# CompactionCircuitBreaker (cooldown rate limiting)
│ │ ├── error_recovery.rs # Error classification + retry strategies
│ │ ├── hardline.rs # Terminal conditions (too many denials, loops, etc.)
│ │ ├── partitioner.rs # Message window partitioning for compaction
│ │ ├── file_cache.rs # ReadFileState — file content dedup cache
│ │ ├── system_prompt.rs # System prompt assembly (mode, skills, memory, tools)
│ │ ├── finalize.rs # Post-loop cleanup: memory extraction, audit log
│ │ └── denial_tracker.rs # Consecutive denial detection
│ ├── compact/ # Context compression (micro/auto/manual layers + collapse)
│ ├── memory/ # Persistent memory manager: extraction, dedup, decay, age, guardrails
│ ├── hooks/ # Lifecycle hooks: registry, dispatch, builtins, matcher, traits (PreToolUse/PostToolUse/Stop/etc.)
│ ├── modes/ # Agent session modes: default, deep-research, literature-reader (identity/tools/config presets)
│ ├── hooks/ # Lifecycle hooks: registry, dispatch, builtins, matcher, traits
│ ├── modes/ # Agent session modes (see Agent Session Modes section below)
│ ├── skills.rs # SkillRegistry: hot-loads SKILL.md files from skills/ directory
│ ├── subagent.rs # Context-isolated sub-agent runner (subagent tool)
│ ├── subagent.rs # Context-isolated sub-agent runner
│ ├── team/ # Multi-agent team: file-based inbox, lead/teammate coordination
│ ├── background.rs# BgNotificationQueue for async slow-task (download/parse) notifications
│ ├── background.rs# BgNotificationQueue for async slow-task notifications
│ ├── task_board.rs# Persistent task board (agent_tasks table)
│ ├── trajectory.rs# Session trajectory recording for audit/debug
│ └── terminal.rs # Escape sequence filter for ANSI-heavy tool outputs
├── clients/ # External API wrappers
│ ├── llm.rs # LlmClient (OpenAI-compatible chat + streaming), EmbeddingClient
├── clients/ # External API wrappers (pure communication, no business logic)
│ ├── llm/ # LlmClient (OpenAI-compatible chat + streaming), EmbeddingClient
│ ├── ads.rs # NASA ADS API
│ ├── arxiv.rs # arXiv Atom XML API
│ ├── cds/ # CDS services (sesame.rs — Sesame name resolver; vizier.rs — TAP queries)
│ ├── gaia/ # Gaia archive (TAP + DataLink)
│ ├── lamost/ # LAMOST archive (ConeSearch + FITS.gz download)
│ ├── sdss/ # SDSS archive (Data Lab TAP + SAS bulk download)
│ ├── desi/ # DESI archive (Data Lab TAP + HEALPix coadd SAS)
│ ├── vo/ # Virtual Observatory generic client
│ └── qiniu.rs # Qiniu cloud storage
├── services/ # Business logic
│ ├── search.rs # Unified cross-source search (ADS + arXiv dedup)
│ ├── download.rs # PDF/HTML download with anti-bot measures and fallback chain
│ ├── parser/ # HTML/PDF → Markdown parsers (A&A, IOP, ar5iv, generic, PDF via MinerU)
│ ├── translation.rs# LLM bilingual translation with Trie-based astronomy glossary
│ ├── rag.rs # Embedding ingest + vector similarity retrieval + LLM answer generation
│ ├── target.rs # Celestial target extraction (IAU name regex) + CDS Sesame lookup
│ ├── chunker.rs # Markdown text chunking for embedding
│ ├── batch/ # Meta-sync (ADS bulk harvest) and asset-batch processing engines
│ ├── query_parser.rs# Advanced search query syntax parser
└── logging.rs # Pretty console + rolling file logger
│ ├── search.rs # Unified cross-source search (ADS + arXiv dedup)
│ ├── download.rs # PDF/HTML download with anti-bot measures and fallback chain
│ ├── parser/ # HTML/PDF → Markdown parsers (A&A, IOP, ar5iv, generic, PDF via MinerU)
│ ├── paper/ # Paper lifecycle: model, db (CRUD), ingest (ADS/arXiv→DB), reader (markdown)
│ ├── translation.rs # LLM bilingual translation with Trie-based astronomy glossary
│ ├── rag.rs # Embedding ingest + vector similarity retrieval + LLM answer generation
│ ├── cds/ # CDS business layer: target.rs (Sesame name resolver), vizier.rs (TAP + cache)
│ ├── observation/ # Cross-source observation data (spectra, light curves, photometry, images)
│ ├── mod.rs # 双轴正交架构: Source × ProductType → ObservationFetcher → Registry
│ ├── types.rs # Source/ProductType/ProductSpec/Artifact/ObservationProduct/Candidate
│ │ ├── fetcher.rs # ObservationFetcher trait + cone search cache template methods
│ │ ├── registry.rs # ObservationRegistry (fetcher registration, capability discovery)
│ │ ├── dispatch.rs # search_observation + download_observation (unified entry points)
│ │ ├── cache.rs # observation_cache table read/write + file staging
│ │ ├── lamost.rs # LamostSpectrumFetcher
│ │ ├── gaia.rs # GaiaSpectrumFetcher + GaiaLightCurveFetcher
│ │ ├── sdss.rs # SdssSpectrumFetcher + SdssApogeeFetcher
│ │ └── desi.rs # DesiSpectrumFetcher
│ ├── batch/ # Meta-sync (ADS bulk harvest) and asset-batch processing engines
│ ├── citation.rs # Citation network graph data (references + citations)
│ ├── session.rs # Agent session listing, metrics, audit log (for API consumption)
│ ├── vision.rs # Image analysis via dedicated vision model
│ ├── note.rs # Highlight/note storage and retrieval
│ ├── pipeline.rs # Synchronous paper processing pipeline (download→parse→translate→embed)
│ ├── chunker.rs # Markdown text chunking for embedding
│ ├── query_parser.rs # Advanced search query syntax parser
│ └── logging.rs # Pretty console + rolling file logger
└── bin/
├── health_check.rs # Library consistency checker and auto-repair
├── cli.rs # CLI interface
@@ -113,11 +173,18 @@ src/
All handlers access state via `Arc<AppState>`. Key fields:
- `db: SqlitePool` — SQLite connection pool (5 max connections, foreign keys enforced)
- `llm: LlmClient` / `embedding: EmbeddingClient` — OpenAI-compatible LLM clients
- `llm: LlmClient` / `medium_llm` / `fast_llm` — OpenAI-compatible LLM clients (multi-tier)
- `vision_llm: Option<LlmClient>` — dedicated vision model client
- `embedding: EmbeddingClient` — embedding model client
- `ads: AdsClient` / `arxiv: ArxivClient` — academic search clients
- `vizier: VizierClient` — CDS VizieR TAP catalog query client
- `lamost` / `gaia` / `sdss` / `desi` — observation data source clients
- `observation_registry: Arc<ObservationRegistry>` — cross-source observation fetcher registry
- `skill_registry: Arc<RwLock<SkillRegistry>>` — hot-reloaded agent skills
- `memory_manager: Arc<MemoryManager>` — persistent agent memory (MEMORY.md + decay)
- `cancelled_runs: Arc<Mutex<HashSet<String>>>` — agent cancellation tokens
- `cancelled_runs: Arc<DashMap<String, ()>>` — agent cancellation tokens
- `session_permission_checkers: Arc<DashMap<String, PermissionChecker>>` — per-session permission rules
- `pending_questions` / `pending_permissions` — interactive agent-user handshake channels
- `harvest_status` / `batch_status` — async batch operation status tracking
### Agent System Design
@@ -131,7 +198,7 @@ The agent (`src/agent/`) implements a **ReAct** (Thought → Action → Observat
5. **Tool call ID tracking**: LLM may not return tool_call IDs — `LlmClient` generates UUID fallbacks. `ToolCall` and `ToolResult` SSE events carry matching IDs for precise frontend pairing.
6. **ToolContext** (`tools/mod.rs`): Injected into every tool execution — holds `app_state`, `session_id`, `sse_tx` (for intermediate events), `enable_thinking`, `read_file_state` (file cache for dedup), `silent` (sub-agents skip permission prompts).
7. **Context compression**: Four layers — micro (placeholder replacement), snip (old-message truncation), auto (LLM summarization), aggro_micro (aggressive placeholder). Protected by `CompactionCircuitBreaker`. Transcripts persisted in `agent_messages` table, not filesystem snapshots.
8. **Skills** (`skills.rs`): Two-layer loading — system-reminder lists names (~20 tokens each), LLM calls `load_skill` to inject full SKILL.md content
8. **Skills** (`skills.rs`): Two-layer loading — system-reminder lists names (~20 tokens each), LLM calls `load_skill` to inject full SKILL.md content. Skills are organized into subdirectories: `methodology/`, `plotting/`, `presentation/`.
9. **Sub-agents** (`subagent.rs`): `subagent` tool spawns a context-isolated sub-agent with its own ReAct loop. Sub-agent messages (system/user/assistant/tool) are persisted to `agent_messages` with `agent_name` identifier. Returns final summary + activity log. SSE progress forwarded to parent via ToolContext.
10. **Memory** (`memory/`): File-based persistent memory (MEMORY.md). `MemoryManager` handles extraction from conversation, dedup, recency decay, age-based pruning, and guardrails. Tools: `save_memory`, `load_memory` (auto-injected in system prompt).
11. **Teams** (`team/`): File-based inbox directory per session for lead/teammate message passing
@@ -141,17 +208,29 @@ Environment variables for agent tuning: `AGENT_TOKEN_SOFT_LIMIT` (default 80000)
### Database
SQLite via `sqlx::sqlite`. Migrations in `migrations/` are auto-run on startup (`sqlx::migrate!("./migrations")`). Key tables: `papers`, `citations_references`, `notes`, `agent_sessions`, `agent_messages`, `agent_tasks`, `agent_audit_log`, `paper_chunks_content`. Vector embeddings use `sqlite-vec` (`vec_paper_chunks` virtual table, auto-registered before any DB connection).
SQLite via `sqlx::sqlite`. Migrations in `migrations/` are auto-run on startup (`sqlx::migrate!("./migrations")`). Key tables: `papers`, `citations_references`, `notes`, `agent_sessions`, `agent_messages`, `agent_tasks`, `agent_audit_log`, `paper_chunks_content`, `vizier_cache`, `observation_cache`. Vector embeddings use `sqlite-vec` (`vec_paper_chunks` virtual table, auto-registered before any DB connection).
The embedding dimension is controlled by `EMBEDDING_DIM` env var (default 1536). On dimension mismatch, the vec table and chunk content are dropped and recreated.
### Observation Data System (`src/services/observation/`)
A cross-source astronomical observation data subsystem with a **dual-axis orthogonal** architecture: **Source** (LAMOST/Gaia/SDSS/DESI) × **ProductType** (Spectrum/LightCurve/Photometry/Image). Each valid combination implements an `ObservationFetcher`, registered in `ObservationRegistry`.
New fetchers follow OCP (pure addition, no modification to existing code):
1. Create `{source}.rs`, implement `ObservationFetcher` trait
2. Add `register!(YourFetcher)` in `registry.rs::default()`
3. Add new variants to `Source`/`ProductType` enums in `types.rs` if needed
External callers use `dispatch::search_observation()` (metadata only) and `dispatch::download_observation()` (metadata + file download) via `ObservationRequest`. Results are cached in `observation_cache` table with file staging to `library/`.
### Frontend (dashboard/)
React 19 + TypeScript + Vite + Tailwind CSS 4. Organized by technical layer (Type-Based):
- `pages/` — Page-level panel view components (SearchPanel, LibraryPanel, ReaderPanel, CitationPanel, SyncPanel, ResearchAgentPanel, SettingsPanel)
- `pages/` — Page-level panel view components (SearchPanel, LibraryPanel, ReaderPanel, CitationPanel, SyncPanel, ResearchAgentPanel, ObservationPanel, SettingsPanel)
- `components/` — Reusable and layout components (sub-folders: agent, reader, sync, layout, dialogs)
- `hooks/` — Global and feature-specific custom stateful React Hooks (e.g., useLibrary, useSearch, useNotes)
- `hooks/` — Global and feature-specific custom stateful React Hooks (e.g., useLibrary, useSearch, useNotes, useObservation)
- `types/` — Global TypeScript type definitions (types/index.ts)
- `utils/` — Common utility helper functions
- `assets/` — Static assets and global stylesheet styles
@@ -160,7 +239,7 @@ Dependencies: `react-markdown` + `rehype-katex` + `remark-math` for Markdown/LaT
### Obscura In-Process Browser
The `obscura-inprocess` feature compiles `obscura-browser` and `obscura-net` directly into the binary, eliminating the need for external browser binaries. This is used for bypassing Cloudflare/WAF on PDF download. Enabled via `--features obscura-inprocess`.
The `obscura-inprocess` feature compiles `obscura-browser` and `obscura-net` from `libs/obscura/` directly into the binary, eliminating the need for external browser binaries. This is used for bypassing Cloudflare/WAF on PDF download. Enabled via `--features obscura-inprocess`.
### Astronomy Glossary (dictionary.txt)
@@ -235,6 +314,10 @@ Controlled by env vars:
- `EXTRACT_MEMORY_THROTTLE_TURNS` (default `3`) — extract every N turns
- `EXTRACT_MEMORY_MAX_STEPS` (default `3`) — sub-agent max steps for extraction
### Deployment (`deploy.sh`)
The `deploy.sh` script builds the project locally (release-min profile, Composer-based) and deploys to a remote server via rsync + SSH. It handles: source compilation, frontend build, remote directory sync, service restart via docker compose, and health check verification.
## Code Conventions
- Use `anyhow` for application errors, `thiserror` for library-style typed errors
@@ -243,3 +326,7 @@ Controlled by env vars:
- API handlers take `State(Arc<AppState>)` and return Axum-compatible responses
- Agent tools implement `AgentTool` trait; new tools register in `ToolRegistry::new()`
- Front-end build is triggered by `build.rs` (auto npm install + build when `dashboard/src/` changes)
- New observation fetchers follow OCP: add module + register — don't modify existing fetchers or dispatch
- External API clients (`src/clients/`) are pure communication; business logic lives in `src/services/`
- Route registration happens in `src/main.rs`; handler re-exports are maintained in `src/api/mod.rs``handlers` namespace
- Agent hooks (`src/agent/hooks/`) implement lifecycles: PreToolUse, PostToolUse, Stop, etc. — register in `builtins.rs`