P1: Small terminal pages with hasMore=false were incorrectly treated as
empty based on file size alone, causing valid messages to be silently
dropped. Now checks actual message content via fileContainsMessages().
P1: Retry import results were not validated, so import failures (including
needFullResync) were masked as success. Now properly handles all failure
cases and tracks retryHasMore to exit the retry loop correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
50 pages (50k messages) was too low for groups with 100k+ messages.
Raise to 5000 pages (5M messages) as a safety valve while allowing
full-sync to complete in a single pull cycle for most conversations.
After pull-sync imports new messages, generate session index incrementally
instead of full regeneration. This preserves existing sessions and their
summaries, only processing newly imported (unindexed) messages.
Adds generateIncremental method across the full stack:
- core: already had generateIncrementalSessionIndex
- node-runtime: expose generateIncrementalIndex
- CLI: add POST /_web/sessions/:id/generate-incremental-index route
- Electron: add session:generateIncremental IPC handler
- Frontend: add generateIncremental to SessionIndexAdapter interface
- Show confirmation dialog with two choices: keep data or delete imported chats
- Backend supports deleteData param across CLI routes and Electron IPC
- DataSourceManager.removeSession now returns the removed session info
- Add i18n strings for the delete confirmation dialog (zh-CN/en-US/ja-JP/zh-TW)
- Add multi-retry (3 attempts, 2s/3s/5s delays) for cold-start empty responses
- Remove unused offset/end parameters from fetch URL construction
- Fix Accept header to application/json only (avoids content negotiation issues)
- Add PullProgress type and progress tracking in PullEngine
- Export progress via getProgress() for frontend polling
When a pullAllSessions was already running, the reloadTimer triggered
by adding sessions would be skipped by the source-level concurrency
lock, causing newly added sessions to wait until the next scheduled
interval (up to 60 minutes) before syncing.
Fix: trigger individual per-session pulls via triggerPull(sourceId,
sessionId) which bypasses the source-level lock, ensuring immediate
sync for newly added sessions regardless of concurrent pulls.
Also includes: connection pre-check on data source add, removal of
duplicate "browse remote sessions" button, hiding "other" session
types, and full resync fix for deleted local sessions.
Co-authored-by: Cursor <cursoragent@cursor.com>
Extract shared business logic from monolithic web.ts into reusable
services in @openchatlab/node-runtime (session, member, index, summary,
export), using SessionRuntimeAdapter to abstract platform differences.
Split web.ts into 12 domain-specific route modules under routes/web/.
Co-authored-by: Cursor <cursoragent@cursor.com>
MiniMax models embed thinking content as <think> tags in the content
field rather than using a separate reasoning field. Add a streaming
think-tag parser that detects these tags in real-time during streaming
and routes them to the thinking UI instead of displaying raw tags.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add directory import support for multi-file formats (e.g. shuakami chunked-jsonl).
Users can select or drag-and-drop a directory via the "Import Folder" checkbox.
- Add `findEntryFileInDirectory()` to parser layer for format-agnostic entry file discovery
- Add `importDirectory` method to ImportAdapter interface
- Implement FetchImportAdapter.importDirectory with multi-file FormData upload
- Add `POST /_web/import-directory` CLI route with directory reconstruction
- Add `chat:importDirectory` IPC handler in Electron main process
- Enhance FileDropZone with `directory` prop and drag-and-drop directory support
- Refactor FetchImportAdapter to extract reusable `consumeSseStream` helper
Co-authored-by: Cursor <cursoragent@cursor.com>
- Expand MCP_TOOL_REGISTRY from 8 to 33 tools by including all
AGENT_TOOL_REGISTRY tools (analytics, deep search, conversation, etc.)
- Add format=text|json parameter to all MCP tools (default: text for
most, json for SQL tools) for token-efficient LLM consumption
- Create format.ts: message filtering, consecutive-sender merging,
content truncation, and compact text output
- Inspired by community chatlab-mcp-server project
Co-authored-by: Cursor <cursoragent@cursor.com>
Move MCP Server logic (tool registration, resource registration, schema
conversion) from apps/cli/src/mcp.ts to packages/mcp-server so both CLI
and Desktop can share the same MCP implementation.
- New package: chatlab-mcp (packages/mcp-server) with McpDatabaseManager
interface abstraction
- CLI mcp.ts slimmed to thin entry (~37 lines) calling startMcpServer()
- Desktop helper prototype at apps/desktop/helper/mcp.ts for dev-time
MCP without GUI
- All 8 MCP tools and 3 resources unchanged
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add backendPersist Pinia plugin for declarative preferences.json persistence
- Move Preferences types to @openchatlab/shared-types (single source of truth)
- Migrate sessionGapThreshold from localStorage to config.toml [ui]
- Migrate chatlab_filter_history from localStorage to preferences.json
- Fix stale localStorage locale read in i18n/index.ts
- Simplify usePreferencesSync (172 → 75 lines) by delegating to plugin
Adding a new backend-synced preference now only requires 3 changes:
1. shared-types interface 2. node-runtime DEFAULTS 3. store backendPersist.pick
When a desktop app user installs the CLI, automatically detect existing
Electron data directory (including custom paths via storage.json),
repoint config.toml, and migrate system data (ai, settings, cache, etc.)
to ~/.chatlab/. Block CLI execution with actionable instructions when
Electron was used but databases cannot be located.
Move the CLI application from packages/server/ to apps/cli/ to align
with the monorepo convention (apps/ for deployable applications,
packages/ for shared libraries). Rename package to @openchatlab/chatlab.
Also fix a bug in ensureToken() that used raw string manipulation to
write config.toml, causing duplicate token fields that corrupted the
TOML file. This led to config parse failures and user_data_dir being
overwritten with defaults. Now uses writeConfigField() for safe writes.
Move Electron-specific code (main, preload, shared) and configs
(electron.vite, electron-builder) into apps/desktop/ to prepare
for multi-platform support (Web Browser standalone).
- Create apps/desktop/ with dedicated package.json and tsconfig
- Split Electron deps (electron, electron-updater, better-sqlite3,
etc.) from shared frontend deps in root package.json
- Declare better-sqlite3 and @node-rs/jieba in packages/node-runtime
- Add stream-json to packages/parser where it belongs
- Update all path aliases (@electron/* → apps/desktop/*) in tsconfig,
vite configs, and relative imports in preload/main
- Keep slim root tsconfig.node.json as shared base for packages
- Update CI/CD release workflow for new directory layout
- Fix pre-existing type errors (pi-ai Model generic, SQLLabTab
LLMChatStreamChunk, filter.test.ts empty methods)
These are frontend-only Vue/ECharts components that depend on app internals (@/),
not cross-platform shared packages. Moving them to src/features/charts-* keeps
packages/ purely for reusable Node.js libraries.
The query barrel file was missing re-exports for getSessionsByTimeRange,
getRecentChatSessions, loadSessionMessages, getPrivateChatMemberAvatar,
getExportSessionData, and searchMessagesWithFtsAsync.
- Extract getPrivateChatMemberAvatar to @openchatlab/core
- Extract getExportSessionData (meta/members/messages SQL) to @openchatlab/core
- Extract loadSessionMessages for summary generation to @openchatlab/core
- Replace inline SQL in summary buildDeps with shared core query functions
- Extract createCompressionLlmAdapter factory to @openchatlab/node-runtime
- Simplify server and electron compression adapter creation via shared factory
- Extract formatAIError to @openchatlab/node-runtime, eliminate duplicate in Electron and Server
- Extract stripAvatarFields to @openchatlab/core
- Extract FTS message search SQL to core (searchMessagesWithFtsAsync)
- Remove redundant sync searchMessagesWithLike from Electron worker
- Remove dead code: electron/main/ai/agent/content-parser.ts (duplicate of @openchatlab/core)
- Extract getSessionsByTimeRange and getRecentChatSessions to @openchatlab/core
- Route IPC session:getByTimeRange and session:getRecent through Worker instead of inline SQL
- Eliminate duplicate getSessionSummaries in workerManager.ts, delegate to core via Worker
- Extract runSimpleLlmStream to @openchatlab/node-runtime for shared
Electron/Server streaming with thinking_start/delta/end event handling
- Simplify Electron IPC ai.ts by delegating to shared stream function
- Add thinking/thinkingDone fields to LlmStreamChunk and preload types
- Add collapsible thinking process display in SQL LAB with auto-scroll
- Implement Web chatStream via SSE with proper error propagation
- Add /_web/cache/save-to-downloads and show-in-folder server routes
- Add cacheApi shim in Web registry for server-side file operations
- Increase SQL LAB maxTokens to 4096 to accommodate reasoning models
- Detect finishReason:length and show token limit error message
- Add i18n keys for thinking/thinkingProcess/errorTokenLimit (4 locales)
- Replace mapCoreEventToStream with shared AgentEventHandler for rich
status events (contextTokens, totalUsage, updatedAt)
- Add formatAIError for user-friendly error messages (429/503/quota)
- Compute maxToolResultTokens dynamically from model contextWindow
- Add steerMessage (answerWithoutTools) and onConvertToLlm callback
- Forward ownerInfo/mentionedMembers from web frontend to server
Move AI-related translations (tools, agent prompts, LLM, summary) from
Electron-specific locale files into @openchatlab/node-runtime, enabling
single-source maintenance across Electron and Server.
- Add packages/node-runtime/src/ai/i18n/ with createAiTranslate() and
bundled locale data (zh-CN, en-US, ja-JP, zh-TW)
- Electron locale files now compose: Electron-specific keys + shared AI
translations from node-runtime
- Server agent.ts switches to shared buildSystemPrompt with evidence
policy, data snapshot injection, ownerInfo and mentionedMembers support
- Server route fetches dataSnapshot from getChatOverview before agent run
- Extract JSON cache CRUD + overview/members compute to node-runtime/cache
- Functions now accept DatabaseAdapter instead of better-sqlite3 Database
- Electron sessionCache.ts becomes pure re-export
- Updated all Electron callers to wrap DB with BetterSqliteAdapter
- Extract migration definitions (v1-v4) to shared getChatDbMigrations() with DI for tokenizer
- DatabaseManager.openWritable() auto-runs pending migrations
- Electron migrations.ts becomes thin wrapper delegating to shared definitions
After streaming or incremental import, automatically generate the
session index (chat_session table) so the frontend does not show
the "generate index" prompt when entering a conversation.
- Replace buffered import with high-performance streaming pipeline
- Add POST /_web/sessions/:id/import/incremental (SSE)
- Add POST /_web/sessions/:id/import/incremental/analyze
- Add POST /_web/import/analyze (dry-run)
- CLI import now reports diagnostics from StreamingImporter
Platform-agnostic temp DB management for merge operations using
DatabaseAdapter. Includes TempDbWriter (streaming write), TempDbReader
(streaming read + MergerDataSource adapter), exportSessionToJson,
and cleanup helpers.
Platform-agnostic incremental import with dual-path dedup
(platformMessageId + content hash), member upsert, meta patching,
FTS sync, and injectable dependencies (IncrementalImportDeps).
Platform-agnostic streaming import with batched transactions, deferred
indexes, nickname history tracking, FTS indexing, and format fallback.
All Electron-specific dependencies replaced with injectable interfaces
(StreamImportDeps, ImportLogger, ImportProgressCallback).
Move the entire RAG module (~2458 lines) from electron/main/ai/rag/ to
@openchatlab/node-runtime. Includes chunking, embedding, vector store,
and semantic pipeline.
Key changes:
- Config operations use injected aiDataDir instead of getAiDataDir()
- Logger uses injectable RagLogger interface
- LLM config provided via getLLMConfig/getAssistantConfig callbacks
- Deduplicated cosineSimilarity into shared utils.ts
- Electron rag/index.ts is now a thin wrapper (~128 lines)
Replace ~120 lines of inline SQL in server stream-import.ts and ~60 lines
in importer.ts fullImport with calls to shared writeParseResultToDb from
@openchatlab/node-runtime. Server imports now gain name history tracking
that was previously Electron-only.
Move performance logging from electron/main/worker/core/perfLogger.ts to
@openchatlab/node-runtime. Log directory is now caller-provided instead
of derived from Electron paths. Electron wrapper provides the path.
Reduces Electron perfLogger from ~181 lines to ~28 lines.
Move core FTS operations (create, build, rebuild, insert, search) to
@openchatlab/node-runtime working with DatabaseAdapter. Electron fts.ts
becomes a thin adapter that adds session-level DB connection management.
Reduces Electron fts.ts from ~207 lines to ~90 lines of pure wrappers.
Move pure SQL session CRUD operations (owner_id update, rename) to the
shared core package as DatabaseAdapter functions. Electron wrappers now
delegate to core, keeping only connection management and error handling.
Move the core data writing logic (meta/member/message insertion with
member_name_history tracking) from electron/main/database/core.ts to
@openchatlab/node-runtime. Electron importData() now delegates to the
shared function via BetterSqliteAdapter, reducing ~130 lines to ~10.
This enables both Electron and Server to share the same import logic
including name history tracking.
The generateMessageKey function used crypto.createHash (Node.js only),
causing Vite browser externalization error. Replaced with pure-JS FNV-1a
64-bit hash that works in both Node.js and browser environments.
Co-authored-by: Cursor <cursoragent@cursor.com>
Platform-agnostic skill CRUD manager with dependency injection for file
system ops, builtin skill resources, and content hashing. Supports init,
CRUD, builtin import/reimport, cloud markdown import, and AI skill menu
generation.
Co-authored-by: Cursor <cursoragent@cursor.com>
Platform-agnostic AssistantManager with dependency injection for file
system ops, builtin configs, and ID generation. Supports init, CRUD,
builtin import/reimport/reset, and cloud markdown import.
Co-authored-by: Cursor <cursoragent@cursor.com>
diguaandClaude Sonnet 4.6
CEQ151