getJieba() invalidated its cache whenever no dict file existed on disk,
so users without a custom dict rebuilt a Jieba instance per tokenize call.
FTS indexing tokenizes every message, making imports seconds slower
(43MB sample: 9.2s -> 3.5s after the fix). Cache now only invalidates
when the on-disk dict state actually changes (added or removed).
Co-authored-by: Cursor <cursoragent@cursor.com>
Switching the word count (80/100/150/200/300) in Insights > Topics >
Wordcloud re-ran a full-corpus jieba segmentation on the Electron main
thread, since topN only affects the final slice while the segmentation
cost is independent of it. This blocked the main process and froze the UI.
- Frontend: fetch the full word list once (max tier) and slice locally on
word-count change, recomputing percentages over the displayed subset.
Word-count switching no longer hits the backend or triggers segmentation.
- Backend: for Chinese meaningful/custom mode, produce word frequency and
POS-tag stats in a single jieba.tag pass instead of two, halving CPU for
genuine recomputes (filter/time/member changes). Behavior is preserved
(verified by an equivalence regression test).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add "type": "module" to all workspace package.json files
- Replace __dirname with import.meta.url-based resolveCliPath() helper
- Convert stream-json/stream-chain CJS named imports to default + destructure
for ESM compatibility
- Use createRequire() for @node-rs/jieba native module loading in ESM context
Co-authored-by: Cursor <cursoragent@cursor.com>
Move platform-agnostic NLP types, stopwords, POS tag definitions, and
text utilities into @openchatlab/core. Move segmenter engine, word
frequency computation, and dict manager into @openchatlab/node-runtime.
Electron nlp/ becomes a thin re-export layer; server nlp.ts shrinks
from ~450 to ~85 lines retaining only HTTP route definitions. Both
platforms now share the same NLP implementation via node-runtime.
Also fixes createJiebaNlpProvider to use shared getJieba() with disk
dict loading (was using empty Jieba instance causing degraded POS tags
in language preference analysis), adds ensureDefaultDict at server
startup, and adds /_web/nlp/* HTTP routes with window.nlpApi shim.
digua
