feat: LAMOST DR12-14 与子版本体系接入、观测层安全加固与并发异步化
- LAMOST 新增 DR12/13/14 及子版本(v0/v1.0/v1.1/v2.0)维度,Internal 发布标记需登录认证并前端灰显,release×subtype 交叉约束下沉至 capabilities 统一声明 - ObservationFetcher trait 扩展版本/认证/交叉约束能力声明,version 参数贯穿 client→service→API→Agent tool→前端全链路 - 安全:observation cache SQL 全参数绑定 + LIKE 转义、cone_cache_hash 加长度前缀防碰撞、DESI survey/program 白名单防穿越 - 异步化:persist/cached_files_total_size/maybe_persist_tool_result迁移到 tokio::fs;cancelled_runs 与 session_permission_checkers改用 DashMap;auth 读锁优先 + 60s 节流 - Gaia 去 native-tls 改禁用连接池规避 UnexpectedEof,reqwest 移除 native-tls feature - 重构:Source/ProductType from_str 集中解析、download 模块拆分为 try_download_pdf/html、AgentRuntime::init 抽取共享逻辑 - 部署:新增 deploy.sh 一键打包推送脚本、catch-panic 启用
This commit is contained in:
@@ -42,6 +42,9 @@ cargo test --lib
|
||||
# Run a specific test
|
||||
cargo test test_name
|
||||
|
||||
# Coverage (requires cargo-llvm-cov)
|
||||
cargo llvm-cov --fail-under-lines 80
|
||||
|
||||
# Frontend (cd dashboard first)
|
||||
npm run dev # HMR dev server on :5173, proxies /api to :8000
|
||||
npm run build # TypeScript check + Vite production build → dashboard/dist/
|
||||
@@ -62,46 +65,103 @@ src/
|
||||
├── main.rs # Axum server entry: logging, DB pool, migrations, vec0 table,
|
||||
│ # client/service construction, route registration, AppState assembly
|
||||
├── lib.rs # Config struct + from_env() loading from .env
|
||||
├── api/ # HTTP handlers, AppState, StandardPaper type
|
||||
│ ├── mod.rs # AppState (shared state), StandardPaper, handlers re-exports
|
||||
├── api/ # HTTP handlers, AppState, re-exports via handlers namespace
|
||||
│ ├── mod.rs # AppState (shared state), handler re-exports, permission/event types
|
||||
│ ├── agent.rs # SSE chat_agent endpoint, session CRUD, metrics, audit log
|
||||
│ ├── papers.rs # Search, download, parse, translate, embed, citations, library, export
|
||||
│ ├── catalog.rs # VizieR TAP catalog queries + cone search endpoints
|
||||
│ ├── observation.rs # Observation data search/download (LAMOST/Gaia/SDSS/DESI)
|
||||
│ ├── notes.rs # Highlight/note CRUD
|
||||
│ ├── sync.rs # Meta-sync and asset-batch endpoints
|
||||
│ ├── targets.rs # Target query/associate/extract, RAG chat, figure chat
|
||||
│ └── helpers.rs # Shared DB helpers, format conversion, path validation
|
||||
│ ├── auth.rs # Login/logout/auth-check (token-based, session memory)
|
||||
│ ├── permissions.rs # Agent tool permission rule management
|
||||
│ ├── search.rs # Search history endpoint
|
||||
│ ├── error.rs # Unified API error type (status code + user-safe message)
|
||||
│ └── helpers.rs # Shared DB helpers, format conversion, path validation (private)
|
||||
├── agent/ # ReAct-based research agent (LLM-driven tool-use loop)
|
||||
│ ├── tools/ # AgentTool trait, ToolRegistry, 25+ tool implementations per domain file
|
||||
│ ├── runtime/ # ReAct loop core: session, context, streaming, executor, token_budget,
|
||||
│ │ # permission, permission_profile, checkpoint, circuit_breaker, error_recovery,
|
||||
│ │ # hardline, partitioner, file_cache, system_prompt, finalize, denial_tracker
|
||||
│ ├── tools/ # AgentTool trait, ToolRegistry, tool implementations per domain
|
||||
│ │ ├── mod.rs # AgentTool trait, ToolContext (session_id, sse_tx, silent, etc.)
|
||||
│ │ ├── filesystem/ # read_file, grep_files, write, edit
|
||||
│ │ ├── astro/ # search_papers, download_paper, parse_paper, rag_search,
|
||||
│ │ │ # query_target, save_note, analyze_image, research/*
|
||||
│ │ ├── ask_user.rs # Permission-gated interactive user question
|
||||
│ │ ├── background.rs# bg_task_run, bg_task_check (async slow tasks)
|
||||
│ │ ├── compress.rs # Context compression trigger
|
||||
│ │ ├── memory.rs # save_memory, load_memory
|
||||
│ │ ├── persist.rs # Data persistence (used by memory/compaction)
|
||||
│ │ ├── skill.rs # load_skill
|
||||
│ │ ├── subagent.rs # Context-isolated sub-agent spawner
|
||||
│ │ ├── team.rs # spawn_teammate, send_teammate_message, etc.
|
||||
│ │ └── todo.rs # todo_write (task tracking)
|
||||
│ ├── runtime/ # ReAct loop core
|
||||
│ │ ├── mod.rs # AgentRuntime: session create/resume → context → ReAct → finalize
|
||||
│ │ ├── executor/ # Step executor + helpers (tool dispatch, parallel tool calls)
|
||||
│ │ ├── streaming.rs # SSE AgentStreamEvent types
|
||||
│ │ ├── streaming_executor.rs # SSE-emitting executor wrapper
|
||||
│ │ ├── session.rs # Session CRUD, DB persist, agent_config
|
||||
│ │ ├── token_budget.rs # Soft/hard token limits, tiktoken counting
|
||||
│ │ ├── permission.rs # Deny > Allow > Ask rule chain, pattern matching
|
||||
│ │ ├── permission_profile.rs # Named profiles (research, readonly)
|
||||
│ │ ├── checkpoint.rs # Rewind/retry state snapshots
|
||||
│ │ ├── circuit_breaker.rs# CompactionCircuitBreaker (cooldown rate limiting)
|
||||
│ │ ├── error_recovery.rs # Error classification + retry strategies
|
||||
│ │ ├── hardline.rs # Terminal conditions (too many denials, loops, etc.)
|
||||
│ │ ├── partitioner.rs # Message window partitioning for compaction
|
||||
│ │ ├── file_cache.rs # ReadFileState — file content dedup cache
|
||||
│ │ ├── system_prompt.rs # System prompt assembly (mode, skills, memory, tools)
|
||||
│ │ ├── finalize.rs # Post-loop cleanup: memory extraction, audit log
|
||||
│ │ └── denial_tracker.rs # Consecutive denial detection
|
||||
│ ├── compact/ # Context compression (micro/auto/manual layers + collapse)
|
||||
│ ├── memory/ # Persistent memory manager: extraction, dedup, decay, age, guardrails
|
||||
│ ├── hooks/ # Lifecycle hooks: registry, dispatch, builtins, matcher, traits (PreToolUse/PostToolUse/Stop/etc.)
|
||||
│ ├── modes/ # Agent session modes: default, deep-research, literature-reader (identity/tools/config presets)
|
||||
│ ├── hooks/ # Lifecycle hooks: registry, dispatch, builtins, matcher, traits
|
||||
│ ├── modes/ # Agent session modes (see Agent Session Modes section below)
|
||||
│ ├── skills.rs # SkillRegistry: hot-loads SKILL.md files from skills/ directory
|
||||
│ ├── subagent.rs # Context-isolated sub-agent runner (subagent tool)
|
||||
│ ├── subagent.rs # Context-isolated sub-agent runner
|
||||
│ ├── team/ # Multi-agent team: file-based inbox, lead/teammate coordination
|
||||
│ ├── background.rs# BgNotificationQueue for async slow-task (download/parse) notifications
|
||||
│ ├── background.rs# BgNotificationQueue for async slow-task notifications
|
||||
│ ├── task_board.rs# Persistent task board (agent_tasks table)
|
||||
│ ├── trajectory.rs# Session trajectory recording for audit/debug
|
||||
│ └── terminal.rs # Escape sequence filter for ANSI-heavy tool outputs
|
||||
├── clients/ # External API wrappers
|
||||
│ ├── llm.rs # LlmClient (OpenAI-compatible chat + streaming), EmbeddingClient
|
||||
├── clients/ # External API wrappers (pure communication, no business logic)
|
||||
│ ├── llm/ # LlmClient (OpenAI-compatible chat + streaming), EmbeddingClient
|
||||
│ ├── ads.rs # NASA ADS API
|
||||
│ ├── arxiv.rs # arXiv Atom XML API
|
||||
│ ├── cds/ # CDS services (sesame.rs — Sesame name resolver; vizier.rs — TAP queries)
|
||||
│ ├── gaia/ # Gaia archive (TAP + DataLink)
|
||||
│ ├── lamost/ # LAMOST archive (ConeSearch + FITS.gz download)
|
||||
│ ├── sdss/ # SDSS archive (Data Lab TAP + SAS bulk download)
|
||||
│ ├── desi/ # DESI archive (Data Lab TAP + HEALPix coadd SAS)
|
||||
│ ├── vo/ # Virtual Observatory generic client
|
||||
│ └── qiniu.rs # Qiniu cloud storage
|
||||
├── services/ # Business logic
|
||||
│ ├── search.rs # Unified cross-source search (ADS + arXiv dedup)
|
||||
│ ├── download.rs # PDF/HTML download with anti-bot measures and fallback chain
|
||||
│ ├── parser/ # HTML/PDF → Markdown parsers (A&A, IOP, ar5iv, generic, PDF via MinerU)
|
||||
│ ├── translation.rs# LLM bilingual translation with Trie-based astronomy glossary
|
||||
│ ├── rag.rs # Embedding ingest + vector similarity retrieval + LLM answer generation
|
||||
│ ├── target.rs # Celestial target extraction (IAU name regex) + CDS Sesame lookup
|
||||
│ ├── chunker.rs # Markdown text chunking for embedding
|
||||
│ ├── batch/ # Meta-sync (ADS bulk harvest) and asset-batch processing engines
|
||||
│ ├── query_parser.rs# Advanced search query syntax parser
|
||||
│ └── logging.rs # Pretty console + rolling file logger
|
||||
│ ├── search.rs # Unified cross-source search (ADS + arXiv dedup)
|
||||
│ ├── download.rs # PDF/HTML download with anti-bot measures and fallback chain
|
||||
│ ├── parser/ # HTML/PDF → Markdown parsers (A&A, IOP, ar5iv, generic, PDF via MinerU)
|
||||
│ ├── paper/ # Paper lifecycle: model, db (CRUD), ingest (ADS/arXiv→DB), reader (markdown)
|
||||
│ ├── translation.rs # LLM bilingual translation with Trie-based astronomy glossary
|
||||
│ ├── rag.rs # Embedding ingest + vector similarity retrieval + LLM answer generation
|
||||
│ ├── cds/ # CDS business layer: target.rs (Sesame name resolver), vizier.rs (TAP + cache)
|
||||
│ ├── observation/ # Cross-source observation data (spectra, light curves, photometry, images)
|
||||
│ │ ├── mod.rs # 双轴正交架构: Source × ProductType → ObservationFetcher → Registry
|
||||
│ │ ├── types.rs # Source/ProductType/ProductSpec/Artifact/ObservationProduct/Candidate
|
||||
│ │ ├── fetcher.rs # ObservationFetcher trait + cone search cache template methods
|
||||
│ │ ├── registry.rs # ObservationRegistry (fetcher registration, capability discovery)
|
||||
│ │ ├── dispatch.rs # search_observation + download_observation (unified entry points)
|
||||
│ │ ├── cache.rs # observation_cache table read/write + file staging
|
||||
│ │ ├── lamost.rs # LamostSpectrumFetcher
|
||||
│ │ ├── gaia.rs # GaiaSpectrumFetcher + GaiaLightCurveFetcher
|
||||
│ │ ├── sdss.rs # SdssSpectrumFetcher + SdssApogeeFetcher
|
||||
│ │ └── desi.rs # DesiSpectrumFetcher
|
||||
│ ├── batch/ # Meta-sync (ADS bulk harvest) and asset-batch processing engines
|
||||
│ ├── citation.rs # Citation network graph data (references + citations)
|
||||
│ ├── session.rs # Agent session listing, metrics, audit log (for API consumption)
|
||||
│ ├── vision.rs # Image analysis via dedicated vision model
|
||||
│ ├── note.rs # Highlight/note storage and retrieval
|
||||
│ ├── pipeline.rs # Synchronous paper processing pipeline (download→parse→translate→embed)
|
||||
│ ├── chunker.rs # Markdown text chunking for embedding
|
||||
│ ├── query_parser.rs # Advanced search query syntax parser
|
||||
│ └── logging.rs # Pretty console + rolling file logger
|
||||
└── bin/
|
||||
├── health_check.rs # Library consistency checker and auto-repair
|
||||
├── cli.rs # CLI interface
|
||||
@@ -113,11 +173,18 @@ src/
|
||||
All handlers access state via `Arc<AppState>`. Key fields:
|
||||
|
||||
- `db: SqlitePool` — SQLite connection pool (5 max connections, foreign keys enforced)
|
||||
- `llm: LlmClient` / `embedding: EmbeddingClient` — OpenAI-compatible LLM clients
|
||||
- `llm: LlmClient` / `medium_llm` / `fast_llm` — OpenAI-compatible LLM clients (multi-tier)
|
||||
- `vision_llm: Option<LlmClient>` — dedicated vision model client
|
||||
- `embedding: EmbeddingClient` — embedding model client
|
||||
- `ads: AdsClient` / `arxiv: ArxivClient` — academic search clients
|
||||
- `vizier: VizierClient` — CDS VizieR TAP catalog query client
|
||||
- `lamost` / `gaia` / `sdss` / `desi` — observation data source clients
|
||||
- `observation_registry: Arc<ObservationRegistry>` — cross-source observation fetcher registry
|
||||
- `skill_registry: Arc<RwLock<SkillRegistry>>` — hot-reloaded agent skills
|
||||
- `memory_manager: Arc<MemoryManager>` — persistent agent memory (MEMORY.md + decay)
|
||||
- `cancelled_runs: Arc<Mutex<HashSet<String>>>` — agent cancellation tokens
|
||||
- `cancelled_runs: Arc<DashMap<String, ()>>` — agent cancellation tokens
|
||||
- `session_permission_checkers: Arc<DashMap<String, PermissionChecker>>` — per-session permission rules
|
||||
- `pending_questions` / `pending_permissions` — interactive agent-user handshake channels
|
||||
- `harvest_status` / `batch_status` — async batch operation status tracking
|
||||
|
||||
### Agent System Design
|
||||
@@ -131,7 +198,7 @@ The agent (`src/agent/`) implements a **ReAct** (Thought → Action → Observat
|
||||
5. **Tool call ID tracking**: LLM may not return tool_call IDs — `LlmClient` generates UUID fallbacks. `ToolCall` and `ToolResult` SSE events carry matching IDs for precise frontend pairing.
|
||||
6. **ToolContext** (`tools/mod.rs`): Injected into every tool execution — holds `app_state`, `session_id`, `sse_tx` (for intermediate events), `enable_thinking`, `read_file_state` (file cache for dedup), `silent` (sub-agents skip permission prompts).
|
||||
7. **Context compression**: Four layers — micro (placeholder replacement), snip (old-message truncation), auto (LLM summarization), aggro_micro (aggressive placeholder). Protected by `CompactionCircuitBreaker`. Transcripts persisted in `agent_messages` table, not filesystem snapshots.
|
||||
8. **Skills** (`skills.rs`): Two-layer loading — system-reminder lists names (~20 tokens each), LLM calls `load_skill` to inject full SKILL.md content
|
||||
8. **Skills** (`skills.rs`): Two-layer loading — system-reminder lists names (~20 tokens each), LLM calls `load_skill` to inject full SKILL.md content. Skills are organized into subdirectories: `methodology/`, `plotting/`, `presentation/`.
|
||||
9. **Sub-agents** (`subagent.rs`): `subagent` tool spawns a context-isolated sub-agent with its own ReAct loop. Sub-agent messages (system/user/assistant/tool) are persisted to `agent_messages` with `agent_name` identifier. Returns final summary + activity log. SSE progress forwarded to parent via ToolContext.
|
||||
10. **Memory** (`memory/`): File-based persistent memory (MEMORY.md). `MemoryManager` handles extraction from conversation, dedup, recency decay, age-based pruning, and guardrails. Tools: `save_memory`, `load_memory` (auto-injected in system prompt).
|
||||
11. **Teams** (`team/`): File-based inbox directory per session for lead/teammate message passing
|
||||
@@ -141,17 +208,29 @@ Environment variables for agent tuning: `AGENT_TOKEN_SOFT_LIMIT` (default 80000)
|
||||
|
||||
### Database
|
||||
|
||||
SQLite via `sqlx::sqlite`. Migrations in `migrations/` are auto-run on startup (`sqlx::migrate!("./migrations")`). Key tables: `papers`, `citations_references`, `notes`, `agent_sessions`, `agent_messages`, `agent_tasks`, `agent_audit_log`, `paper_chunks_content`. Vector embeddings use `sqlite-vec` (`vec_paper_chunks` virtual table, auto-registered before any DB connection).
|
||||
SQLite via `sqlx::sqlite`. Migrations in `migrations/` are auto-run on startup (`sqlx::migrate!("./migrations")`). Key tables: `papers`, `citations_references`, `notes`, `agent_sessions`, `agent_messages`, `agent_tasks`, `agent_audit_log`, `paper_chunks_content`, `vizier_cache`, `observation_cache`. Vector embeddings use `sqlite-vec` (`vec_paper_chunks` virtual table, auto-registered before any DB connection).
|
||||
|
||||
The embedding dimension is controlled by `EMBEDDING_DIM` env var (default 1536). On dimension mismatch, the vec table and chunk content are dropped and recreated.
|
||||
|
||||
### Observation Data System (`src/services/observation/`)
|
||||
|
||||
A cross-source astronomical observation data subsystem with a **dual-axis orthogonal** architecture: **Source** (LAMOST/Gaia/SDSS/DESI) × **ProductType** (Spectrum/LightCurve/Photometry/Image). Each valid combination implements an `ObservationFetcher`, registered in `ObservationRegistry`.
|
||||
|
||||
New fetchers follow OCP (pure addition, no modification to existing code):
|
||||
|
||||
1. Create `{source}.rs`, implement `ObservationFetcher` trait
|
||||
2. Add `register!(YourFetcher)` in `registry.rs::default()`
|
||||
3. Add new variants to `Source`/`ProductType` enums in `types.rs` if needed
|
||||
|
||||
External callers use `dispatch::search_observation()` (metadata only) and `dispatch::download_observation()` (metadata + file download) via `ObservationRequest`. Results are cached in `observation_cache` table with file staging to `library/`.
|
||||
|
||||
### Frontend (dashboard/)
|
||||
|
||||
React 19 + TypeScript + Vite + Tailwind CSS 4. Organized by technical layer (Type-Based):
|
||||
|
||||
- `pages/` — Page-level panel view components (SearchPanel, LibraryPanel, ReaderPanel, CitationPanel, SyncPanel, ResearchAgentPanel, SettingsPanel)
|
||||
- `pages/` — Page-level panel view components (SearchPanel, LibraryPanel, ReaderPanel, CitationPanel, SyncPanel, ResearchAgentPanel, ObservationPanel, SettingsPanel)
|
||||
- `components/` — Reusable and layout components (sub-folders: agent, reader, sync, layout, dialogs)
|
||||
- `hooks/` — Global and feature-specific custom stateful React Hooks (e.g., useLibrary, useSearch, useNotes)
|
||||
- `hooks/` — Global and feature-specific custom stateful React Hooks (e.g., useLibrary, useSearch, useNotes, useObservation)
|
||||
- `types/` — Global TypeScript type definitions (types/index.ts)
|
||||
- `utils/` — Common utility helper functions
|
||||
- `assets/` — Static assets and global stylesheet styles
|
||||
@@ -160,7 +239,7 @@ Dependencies: `react-markdown` + `rehype-katex` + `remark-math` for Markdown/LaT
|
||||
|
||||
### Obscura In-Process Browser
|
||||
|
||||
The `obscura-inprocess` feature compiles `obscura-browser` and `obscura-net` directly into the binary, eliminating the need for external browser binaries. This is used for bypassing Cloudflare/WAF on PDF download. Enabled via `--features obscura-inprocess`.
|
||||
The `obscura-inprocess` feature compiles `obscura-browser` and `obscura-net` from `libs/obscura/` directly into the binary, eliminating the need for external browser binaries. This is used for bypassing Cloudflare/WAF on PDF download. Enabled via `--features obscura-inprocess`.
|
||||
|
||||
### Astronomy Glossary (dictionary.txt)
|
||||
|
||||
@@ -235,6 +314,10 @@ Controlled by env vars:
|
||||
- `EXTRACT_MEMORY_THROTTLE_TURNS` (default `3`) — extract every N turns
|
||||
- `EXTRACT_MEMORY_MAX_STEPS` (default `3`) — sub-agent max steps for extraction
|
||||
|
||||
### Deployment (`deploy.sh`)
|
||||
|
||||
The `deploy.sh` script builds the project locally (release-min profile, Composer-based) and deploys to a remote server via rsync + SSH. It handles: source compilation, frontend build, remote directory sync, service restart via docker compose, and health check verification.
|
||||
|
||||
## Code Conventions
|
||||
|
||||
- Use `anyhow` for application errors, `thiserror` for library-style typed errors
|
||||
@@ -243,3 +326,7 @@ Controlled by env vars:
|
||||
- API handlers take `State(Arc<AppState>)` and return Axum-compatible responses
|
||||
- Agent tools implement `AgentTool` trait; new tools register in `ToolRegistry::new()`
|
||||
- Front-end build is triggered by `build.rs` (auto npm install + build when `dashboard/src/` changes)
|
||||
- New observation fetchers follow OCP: add module + register — don't modify existing fetchers or dispatch
|
||||
- External API clients (`src/clients/`) are pure communication; business logic lives in `src/services/`
|
||||
- Route registration happens in `src/main.rs`; handler re-exports are maintained in `src/api/mod.rs` → `handlers` namespace
|
||||
- Agent hooks (`src/agent/hooks/`) implement lifecycles: PreToolUse, PostToolUse, Stop, etc. — register in `builtins.rs`
|
||||
|
||||
Reference in New Issue
Block a user