mirror of
https://github.com/zhayujie/chatgpt-on-wechat.git
synced 2026-10-25 17:36:29 +08:00
alanyz106
0b052ec779
Replaces the plain char-count chunker for .md files with a structure-aware chunker (chunker.chunk_markdown) so a semantic unit is not split mid-section. Strategy (per file): - A file <= 1500 chars is ONE chunk (whole file = strong semantic unit). Most real memory/knowledge files take this path and stay untouched. - Larger files are parsed with markdown-it into per-heading "own body" candidates (heading line up to its first direct child heading; parents do not swallow child bodies), then greedily merged across headings up to the 1500-char target, then a single tail fold merges a trailing fragment (<= 400 chars) into the previous chunk. Calibrated against the retrieval eval (MRR 0.907 vs 0.818 fixed-char) and hand-checked on real memory files (services.md 5 chunks / geetest.md 3 / SKILL.md 14, contiguous 1-based line coverage, code-block fake headings correctly ignored by markdown-it). Call-site change (manager.sync): .md files now dispatch to chunk_markdown; non-.md content keeps the existing chunk_text. add_memory is unchanged (single unstructured entries, no file structure). NOTE for adopters: chunk boundaries change only when a file is re-indexed. Existing indexes keep their old chunks until `/memory rebuild-index` is run. Dependency: markdown-it-py added to requirements.txt (lazy import with fallback to chunk_text if the parser is unavailable).
46 lines
1.2 KiB
Plaintext
46 lines
1.2 KiB
Plaintext
numpy>=1.21
|
|
# markdown structure-aware chunking for memory/knowledge md files (chunker.chunk_markdown)
|
|
markdown-it-py>=3.0.0
|
|
# aiohttp<3.10 has no prebuilt wheels for Python 3.13+, so pip would compile from
|
|
# source (needs MSVC on Windows). Use >=3.10 on 3.13+ where cp313 wheels exist.
|
|
aiohttp>=3.8.6,<3.10; python_version < "3.13"
|
|
aiohttp>=3.10; python_version >= "3.13"
|
|
requests>=2.28.2
|
|
chardet>=5.1.0
|
|
Pillow
|
|
python-dotenv>=1.0.0
|
|
PyYAML>=6.0
|
|
croniter>=2.0.0
|
|
click>=8.0
|
|
qrcode
|
|
json-repair
|
|
# search_files tool: regex.search() supports a real per-call timeout, stdlib re does not
|
|
regex>=2019.3.12
|
|
|
|
# wechatcom & wechatmp
|
|
wechatpy
|
|
|
|
# zhipuai (>=0.2.3 required for GLM-5.3 reasoning_effort support)
|
|
zai-sdk>=0.2.3
|
|
# tongyi qwen sdk
|
|
dashscope
|
|
|
|
# feishu
|
|
lark-oapi>=1.5.5
|
|
# dingtalk
|
|
dingtalk_stream
|
|
# wecom bot websocket mode
|
|
websocket-client>=1.4.0
|
|
pycryptodome
|
|
# telegram bot
|
|
python-telegram-bot
|
|
# slack bot
|
|
slack_bolt
|
|
# discord bot
|
|
discord.py
|
|
|
|
# web.py: PyPI 0.62 fails to build on Python 3.13+ (cgi module removed), use GitHub fix instead
|
|
web.py; python_version < "3.13"
|
|
web.py @ git+https://github.com/webpy/webpy.git ; python_version >= "3.13"
|
|
legacy-cgi; python_version >= "3.13"
|