Merge pull request #1837 from dividduang/master

blackroom

README.md

@@ -1,18 +1,15 @@
 # 简介
 
-> ChatGPT近期以强大的对话和信息整合能力风靡全网，可以写代码、改论文、讲故事，几乎无所不能，这让人不禁有个大胆的想法，能否用他的对话模型把我们的微信打造成一个智能机器人，可以在与好友对话中给出意想不到的回应，而且再也不用担心女朋友影响我们 ~~打游戏~~ 工作了。
+> 本项目是基于大模型的智能对话机器人，支持微信、企业微信、公众号、飞书、钉钉接入，可选择GPT3.5/GPT4.0/Claude/文心一言/讯飞星火/通义千问/Gemini/LinkAI/ZhipuAI，能处理文本、语音和图片，通过插件访问操作系统和互联网等外部资源，支持基于自有知识库定制企业AI应用。
 
 最新版本支持的功能如下：
 
-- [x] **多端部署：** 有多种部署方式可选择且功能完备，目前已支持个人微信、微信公众号和、企业微信、飞书等部署方式
-- [x] **基础对话：** 私聊及群聊的消息智能回复，支持多轮会话上下文记忆，支持 GPT-3.5, GPT-4, claude, Gemini, 文心一言, 讯飞星火, 通义千问
+- [x] **多端部署：** 有多种部署方式可选择且功能完备，目前已支持个人微信、微信公众号和、企业微信、飞书、钉钉等部署方式
+- [x] **基础对话：** 私聊及群聊的消息智能回复，支持多轮会话上下文记忆，支持 GPT-3.5, GPT-4, Claude-3, Gemini, 文心一言, 讯飞星火, 通义千问，ChatGLM-4
 - [x] **语音能力：** 可识别语音消息，通过文字或语音回复，支持 azure, baidu, google, openai(whisper/tts) 等多种语音模型
-- [x] **图像能力：** 支持图片生成、图片识别、图生图（如照片修复），可选择 Dall-E-3, stable diffusion, replicate, midjourney, vision模型
-- [x] **丰富插件：** 支持个性化插件扩展，已实现多角色切换、文字冒险、敏感词过滤、聊天记录总结、文档总结和对话等插件
-- [X] **Tool工具：** 与操作系统和互联网交互，支持最新信息搜索、数学计算、天气和资讯查询、网页总结，基于 [chatgpt-tool-hub](https://github.com/goldfishh/chatgpt-tool-hub) 实现
-- [x] **知识库：** 通过上传知识库文件自定义专属机器人，可作为数字分身、领域知识库、智能客服使用，基于 [LinkAI](https://link-ai.tech/console) 实现
-
-> 欢迎接入更多应用，参考 [Terminal代码](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/channel/terminal/terminal_channel.py)实现接收和发送消息逻辑即可接入。 同时欢迎增加新的插件，参考 [插件说明文档](https://github.com/zhayujie/chatgpt-on-wechat/tree/master/plugins)。
+- [x] **图像能力：** 支持图片生成、图片识别、图生图（如照片修复），可选择 Dall-E-3, stable diffusion, replicate, midjourney, CogView-3, vision模型
+- [x] **丰富插件：** 支持个性化插件扩展，已实现多角色切换、文字冒险、敏感词过滤、聊天记录总结、文档总结和对话、联网搜索等插件
+- [x] **知识库：** 通过上传知识库文件自定义专属机器人，可作为数字分身、智能客服、私域助手使用，基于 [LinkAI](https://link-ai.tech) 实现
 
 # 演示
 
@@ -20,15 +17,29 @@ https://github.com/zhayujie/chatgpt-on-wechat/assets/26161723/d5154020-36e3-41db
 
 Demo made by [Visionn](https://www.wangpc.cc/)
 
-# 交流群
+# 商业支持
 
-添加小助手微信进群，请备注 "wechat"：
+> 我们还提供企业级的 **AI应用平台**，包含知识库、Agent插件、应用管理等能力，支持多平台聚合的应用接入、客户端管理、对话管理，以及提供
+SaaS服务、私有化部署、稳定托管接入 等多种模式。
+>
+> 目前已在私域运营、智能客服、企业效率助手等场景积累了丰富的 AI 解决方案， 在电商、文教、健康、新消费等各行业沉淀了 AI 落地的最佳实践，致力于打造助力中小企业拥抱 AI 的一站式平台。
+企业服务和商用咨询可联系产品顾问：
+
+<img width="240" src="https://img-1317903499.cos.ap-guangzhou.myqcloud.com/docs/product-manager-qrcode.jpg">
+
+# 开源社区
+
+添加小助手微信加入开源项目交流群：
 
 <img width="240" src="./docs/images/contact.jpg">
 
 # 更新日志
 
->**2023.11.11：** [1.5.3版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.3) 和 [1.5.4版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.4)，新增Google Gemini、通义千问模型
+>**2024.03.26：** [1.5.8版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.8) 和 [1.5.7版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.7)，新增 GLM-4、Claude-3 模型，edge-tts 语音支持
+
+>**2024.01.26：** [1.5.6版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.6) 和 [1.5.5版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.5)，钉钉接入，tool插件升级，4-turbo模型更新
+
+>**2023.11.11：** [1.5.3版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.3) 和 [1.5.4版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.4)，新增通义千问模型、Google Gemini
 
 >**2023.11.10：** [1.5.2版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.5.2)，新增飞书通道、图像识别对话、黑名单配置
 
@@ -42,17 +53,7 @@ Demo made by [Visionn](https://www.wangpc.cc/)
 
 >**2023.06.12：** 接入 [LinkAI](https://link-ai.tech/console) 平台，可在线创建领域知识库，并接入微信、公众号及企业微信中，打造专属客服机器人。使用参考 [接入文档](https://link-ai.tech/platform/link-app/wechat)。
 
->**2023.04.26：** 支持企业微信应用号部署，兼容插件，并支持语音图片交互，私人助理理想选择，[使用文档](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/channel/wechatcom/README.md)。(contributed by [@lanvent](https://github.com/lanvent) in [#944](https://github.com/zhayujie/chatgpt-on-wechat/pull/944))
-
->**2023.04.05：** 支持微信公众号部署，兼容插件，并支持语音图片交互，[使用文档](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/channel/wechatmp/README.md)。(contributed by [@JS00000](https://github.com/JS00000) in [#686](https://github.com/zhayujie/chatgpt-on-wechat/pull/686))
-
->**2023.04.05：** 增加能让ChatGPT使用工具的`tool`插件，[使用文档](https://github.com/goldfishh/chatgpt-on-wechat/blob/master/plugins/tool/README.md)。工具相关issue可反馈至[chatgpt-tool-hub](https://github.com/goldfishh/chatgpt-tool-hub)。(contributed by [@goldfishh](https://github.com/goldfishh) in [#663](https://github.com/zhayujie/chatgpt-on-wechat/pull/663))
-
->**2023.03.25：** 支持插件化开发，目前已实现 多角色切换、文字冒险游戏、管理员指令、Stable Diffusion等插件，使用参考 [#578](https://github.com/zhayujie/chatgpt-on-wechat/issues/578)。(contributed by [@lanvent](https://github.com/lanvent) in [#565](https://github.com/zhayujie/chatgpt-on-wechat/pull/565))
-
->**2023.03.09：** 基于 `whisper API`(后续已接入更多的语音`API`服务) 实现对微信语音消息的解析和回复，添加配置项 `"speech_recognition":true` 即可启用，使用参考 [#415](https://github.com/zhayujie/chatgpt-on-wechat/issues/415)。(contributed by [wanggang1987](https://github.com/wanggang1987) in [#385](https://github.com/zhayujie/chatgpt-on-wechat/pull/385))
-
->**2023.02.09：** 扫码登录存在账号限制风险，请谨慎使用，参考[#58](https://github.com/AutumnWhj/ChatGPT-wechat-bot/issues/158)
+更早更新日志查看: [归档日志](/docs/version/old-version.md)
 
 # 快速开始
 
@@ -82,6 +83,8 @@ git clone https://github.com/zhayujie/chatgpt-on-wechat
 cd chatgpt-on-wechat/
 ```
 
+注: 如遇到网络问题可选择国内镜像 https://gitee.com/zhayujie/chatgpt-on-wechat
+
 **(2) 安装核心依赖 (必选)：**
 > 能够使用`itchat`创建机器人，并具有文字交流功能所需的最小依赖集合。
 ```bash
@@ -93,23 +96,7 @@ pip3 install -r requirements.txt
 ```bash
 pip3 install -r requirements-optional.txt
 ```
-> 如果某项依赖安装失败请注释掉对应的行再继续。
-
-其中`tiktoken`要求`python`版本在3.8以上，它用于精确计算会话使用的tokens数量，强烈建议安装。
-
-
-使用`google`或`baidu`语音识别需安装`ffmpeg`，
-
-默认的`openai`语音识别不需要安装`ffmpeg`。
-
-参考[#415](https://github.com/zhayujie/chatgpt-on-wechat/issues/415)
-
-使用`azure`语音功能需安装依赖，并参考[文档](https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/quickstarts/setup-platform?pivots=programming-language-python&tabs=linux%2Cubuntu%2Cdotnet%2Cjre%2Cmaven%2Cnodejs%2Cmac%2Cpypi)的环境要求。
-:
-
-```bash
-pip3 install azure-cognitiveservices-speech
-```
+> 如果某项依赖安装失败可注释掉对应的行再继续
 
 ## 配置
 
@@ -125,7 +112,8 @@ pip3 install azure-cognitiveservices-speech
 # config.json文件内容示例
 {
   "open_ai_api_key": "YOUR API KEY",                          # 填入上面创建的 OpenAI API KEY
-  "model": "gpt-3.5-turbo",                                   # 模型名称, 支持 gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-4, wenxin, xunfei
+  "model": "gpt-3.5-turbo",                                   # 模型名称, 支持 gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-4, wenxin, xunfei, claude-3-opus-20240229
+  "claude_api_key":"YOUR API KEY"                             # 如果选用claude3模型的话，配置这个key，同时如想使用生图，语音等功能，仍需配置open_ai_api_key
   "proxy": "",                                                # 代理客户端的ip和端口，国内环境开启代理的需要填写该项，如 "127.0.0.1:7890"
   "single_chat_prefix": ["bot", "@bot"],                      # 私聊时文本需要包含该前缀才能触发机器人回复
   "single_chat_reply_prefix": "[bot] ",                       # 私聊时自动回复的前缀，用于区分真人
@@ -206,7 +194,6 @@ python3 app.py                                    # windows环境下该命令通
 使用nohup命令在后台运行程序：
 
 ```bash
-touch nohup.out                                   # 首次运行需要新建日志文件  
 nohup python3 app.py & tail -f nohup.out          # 在后台运行程序并通过日志输出二维码
 ```
 扫码登录后程序即可运行于服务器后台，此时可通过 `ctrl+c` 关闭日志，不会影响后台程序的运行。使用 `ps -ef | grep app.py | grep -v grep` 命令可查看运行于后台的进程，如果想要重新启动程序可以先 `kill` 掉对应的进程。日志关闭后如果想要再次打开只需输入 `tail -f nohup.out`。此外，`scripts` 目录下有一键运行、关闭程序的脚本供使用。
@@ -279,6 +266,10 @@ FAQs： <https://github.com/zhayujie/chatgpt-on-wechat/wiki/FAQs>
 
 或直接在线咨询 [项目小助手](https://link-ai.tech/app/Kv2fXJcH)  (beta版本，语料完善中，回复仅供参考)
 
+## 开发
+
+欢迎接入更多应用，参考 [Terminal代码](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/channel/terminal/terminal_channel.py) 实现接收和发送消息逻辑即可接入。 同时欢迎增加新的插件，参考 [插件说明文档](https://github.com/zhayujie/chatgpt-on-wechat/tree/master/plugins)。
+
 ## 联系
 
-欢迎提交PR、Issues，以及Star支持一下。程序运行遇到问题可以查看 [常见问题列表](https://github.com/zhayujie/chatgpt-on-wechat/wiki/FAQs) ，其次前往 [Issues](https://github.com/zhayujie/chatgpt-on-wechat/issues) 中搜索。参与更多讨论可加入技术交流群。
+欢迎提交PR、Issues，以及Star支持一下。程序运行遇到问题可以查看 [常见问题列表](https://github.com/zhayujie/chatgpt-on-wechat/wiki/FAQs) ，其次前往 [Issues](https://github.com/zhayujie/chatgpt-on-wechat/issues) 中搜索。个人开发者可加入开源交流群参与更多讨论，企业用户可联系[产品顾问](https://img-1317903499.cos.ap-guangzhou.myqcloud.com/docs/product-manager-qrcode.jpg)咨询。

app.py

@@ -3,6 +3,7 @@
 import os
 import signal
 import sys
+import time
 
 from channel import channel_factory
 from common import const
@@ -24,6 +25,21 @@ def sigterm_handler_wrap(_signo):
     signal.signal(_signo, func)
 
 
+def start_channel(channel_name: str):
+    channel = channel_factory.create_channel(channel_name)
+    if channel_name in ["wx", "wxy", "terminal", "wechatmp", "wechatmp_service", "wechatcom_app", "wework",
+                        const.FEISHU, const.DINGTALK]:
+        PluginManager().load_plugins()
+
+    if conf().get("use_linkai"):
+        try:
+            from common import linkai_client
+            threading.Thread(target=linkai_client.start, args=(channel,)).start()
+        except Exception as e:
+            pass
+    channel.startup()
+
+
 def run():
     try:
         # load config
@@ -41,22 +57,11 @@ def run():
 
         if channel_name == "wxy":
             os.environ["WECHATY_LOG"] = "warn"
-            # os.environ['WECHATY_PUPPET_SERVICE_ENDPOINT'] = '127.0.0.1:9001'
 
-        channel = channel_factory.create_channel(channel_name)
-        if channel_name in ["wx", "wxy", "terminal", "wechatmp", "wechatmp_service", "wechatcom_app", "wework", const.FEISHU,const.DINGTALK]:
-            PluginManager().load_plugins()
-
-        if conf().get("use_linkai"):
-            try:
-                from common import linkai_client
-                threading.Thread(target=linkai_client.start, args=(channel, )).start()
-            except Exception as e:
-                pass
-
-        # startup channel
-        channel.startup()
+        start_channel(channel_name)
 
+        while True:
+            time.sleep(1)
     except Exception as e:
         logger.error("App startup failed!")
         logger.exception(e)

bot/bot_factory.py

@@ -2,6 +2,7 @@
 channel factory
 """
 from common import const
+from common.log import logger
 
 
 def create_bot(bot_type):
@@ -43,7 +44,9 @@ def create_bot(bot_type):
     elif bot_type == const.CLAUDEAI:
         from bot.claude.claude_ai_bot import ClaudeAIBot
         return ClaudeAIBot()
-
+    elif bot_type == const.CLAUDEAPI:
+        from bot.claudeapi.claude_api_bot import ClaudeAPIBot
+        return ClaudeAPIBot()
     elif bot_type == const.QWEN:
         from bot.ali.ali_qwen_bot import AliQwenBot
         return AliQwenBot()
@@ -52,4 +55,9 @@ def create_bot(bot_type):
         from bot.gemini.google_gemini_bot import GoogleGeminiBot
         return GoogleGeminiBot()
 
+    elif bot_type == const.ZHIPU_AI:
+        from bot.zhipuai.zhipuai_bot import ZHIPUAIBot
+        return ZHIPUAIBot()
+
+
     raise RuntimeError

bot/chatgpt/chat_gpt_session.py

@@ -62,12 +62,14 @@ def num_tokens_from_messages(messages, model):
 
     import tiktoken
 
-    if model in ["gpt-3.5-turbo-0301", "gpt-35-turbo", "gpt-3.5-turbo-1106"]:
+    if model in ["gpt-3.5-turbo-0301", "gpt-35-turbo", "gpt-3.5-turbo-1106", "moonshot"]:
         return num_tokens_from_messages(messages, model="gpt-3.5-turbo")
     elif model in ["gpt-4-0314", "gpt-4-0613", "gpt-4-32k", "gpt-4-32k-0613", "gpt-3.5-turbo-0613",
-                   "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613", "gpt-35-turbo-16k", const.GPT4_TURBO_PREVIEW, const.GPT4_VISION_PREVIEW]:
+                   "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613", "gpt-35-turbo-16k", "gpt-4-turbo-preview",
+                   "gpt-4-1106-preview", const.GPT4_TURBO_PREVIEW, const.GPT4_VISION_PREVIEW]:
         return num_tokens_from_messages(messages, model="gpt-4")
-
+    elif model.startswith("claude-3"):
+        return num_tokens_from_messages(messages, model="gpt-3.5-turbo")
     try:
         encoding = tiktoken.encoding_for_model(model)
     except KeyError:

bot/claudeapi/claude_api_bot.py

@@ -0,0 +1,133 @@
+# encoding:utf-8
+
+import time
+
+import openai
+import openai.error
+import anthropic
+
+from bot.bot import Bot
+from bot.openai.open_ai_image import OpenAIImage
+from bot.chatgpt.chat_gpt_session import ChatGPTSession
+from bot.gemini.google_gemini_bot import GoogleGeminiBot
+from bot.session_manager import SessionManager
+from bridge.context import ContextType
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from config import conf
+
+user_session = dict()
+
+
+# OpenAI对话模型API (可用)
+class ClaudeAPIBot(Bot, OpenAIImage):
+    def __init__(self):
+        super().__init__()
+        self.claudeClient = anthropic.Anthropic(
+            api_key=conf().get("claude_api_key")
+        )
+        openai.api_key = conf().get("open_ai_api_key")
+        if conf().get("open_ai_api_base"):
+            openai.api_base = conf().get("open_ai_api_base")
+        proxy = conf().get("proxy")
+        if proxy:
+            openai.proxy = proxy
+
+        self.sessions = SessionManager(ChatGPTSession, model=conf().get("model") or "text-davinci-003")
+
+    def reply(self, query, context=None):
+        # acquire reply content
+        if context and context.type:
+            if context.type == ContextType.TEXT:
+                logger.info("[CLAUDE_API] query={}".format(query))
+                session_id = context["session_id"]
+                reply = None
+                if query == "#清除记忆":
+                    self.sessions.clear_session(session_id)
+                    reply = Reply(ReplyType.INFO, "记忆已清除")
+                elif query == "#清除所有":
+                    self.sessions.clear_all_session()
+                    reply = Reply(ReplyType.INFO, "所有人记忆已清除")
+                else:
+                    session = self.sessions.session_query(query, session_id)
+                    result = self.reply_text(session)
+                    logger.info(result)
+                    total_tokens, completion_tokens, reply_content = (
+                        result["total_tokens"],
+                        result["completion_tokens"],
+                        result["content"],
+                    )
+                    logger.debug(
+                        "[CLAUDE_API] new_query={}, session_id={}, reply_cont={}, completion_tokens={}".format(str(session), session_id, reply_content, completion_tokens)
+                    )
+
+                    if total_tokens == 0:
+                        reply = Reply(ReplyType.ERROR, reply_content)
+                    else:
+                        self.sessions.session_reply(reply_content, session_id, total_tokens)
+                        reply = Reply(ReplyType.TEXT, reply_content)
+                return reply
+            elif context.type == ContextType.IMAGE_CREATE:
+                ok, retstring = self.create_img(query, 0)
+                reply = None
+                if ok:
+                    reply = Reply(ReplyType.IMAGE_URL, retstring)
+                else:
+                    reply = Reply(ReplyType.ERROR, retstring)
+                return reply
+
+    def reply_text(self, session: ChatGPTSession, retry_count=0):
+        try:
+            actual_model = self._model_mapping(conf().get("model"))
+            response = self.claudeClient.messages.create(
+                model=actual_model,
+                max_tokens=1024,
+                # system=conf().get("system"),
+                messages=GoogleGeminiBot.filter_messages(session.messages)
+            )
+            # response = openai.Completion.create(prompt=str(session), **self.args)
+            res_content = response.content[0].text.strip().replace("<|endoftext|>", "")
+            total_tokens = response.usage.input_tokens+response.usage.output_tokens
+            completion_tokens = response.usage.output_tokens
+            logger.info("[CLAUDE_API] reply={}".format(res_content))
+            return {
+                "total_tokens": total_tokens,
+                "completion_tokens": completion_tokens,
+                "content": res_content,
+            }
+        except Exception as e:
+            need_retry = retry_count < 2
+            result = {"completion_tokens": 0, "content": "我现在有点累了，等会再来吧"}
+            if isinstance(e, openai.error.RateLimitError):
+                logger.warn("[CLAUDE_API] RateLimitError: {}".format(e))
+                result["content"] = "提问太快啦，请休息一下再问我吧"
+                if need_retry:
+                    time.sleep(20)
+            elif isinstance(e, openai.error.Timeout):
+                logger.warn("[CLAUDE_API] Timeout: {}".format(e))
+                result["content"] = "我没有收到你的消息"
+                if need_retry:
+                    time.sleep(5)
+            elif isinstance(e, openai.error.APIConnectionError):
+                logger.warn("[CLAUDE_API] APIConnectionError: {}".format(e))
+                need_retry = False
+                result["content"] = "我连接不到你的网络"
+            else:
+                logger.warn("[CLAUDE_API] Exception: {}".format(e))
+                need_retry = False
+                self.sessions.clear_session(session.session_id)
+
+            if need_retry:
+                logger.warn("[CLAUDE_API] 第{}次重试".format(retry_count + 1))
+                return self.reply_text(session, retry_count + 1)
+            else:
+                return result
+
+    def _model_mapping(self, model) -> str:
+        if model == "claude-3-opus":
+            return "claude-3-opus-20240229"
+        elif model == "claude-3-sonnet":
+            return "claude-3-sonnet-20240229"
+        elif model == "claude-3-haiku":
+            return "claude-3-haiku-20240307"
+        return model

bot/gemini/google_gemini_bot.py

@@ -33,7 +33,7 @@ class GoogleGeminiBot(Bot):
             logger.info(f"[Gemini] query={query}")
             session_id = context["session_id"]
             session = self.sessions.session_query(query, session_id)
-            gemini_messages = self._convert_to_gemini_messages(self._filter_messages(session.messages))
+            gemini_messages = self._convert_to_gemini_messages(self.filter_messages(session.messages))
             genai.configure(api_key=self.api_key)
             model = genai.GenerativeModel('gemini-pro')
             response = model.generate_content(gemini_messages)
@@ -44,6 +44,7 @@ class GoogleGeminiBot(Bot):
         except Exception as e:
             logger.error("[Gemini] fetch reply error, may contain unsafe content")
             logger.error(e)
+            return Reply(ReplyType.ERROR, "invoke [Gemini] api failed!")
 
     def _convert_to_gemini_messages(self, messages: list):
         res = []
@@ -60,9 +61,12 @@ class GoogleGeminiBot(Bot):
             })
         return res
 
-    def _filter_messages(self, messages: list):
+    @staticmethod
+    def filter_messages(messages: list):
         res = []
         turn = "user"
+        if not messages:
+            return res
         for i in range(len(messages) - 1, -1, -1):
             message = messages[i]
             if message.get("role") != turn:

bot/linkai/link_ai_bot.py

@@ -7,6 +7,7 @@ import requests
 import config
 from bot.bot import Bot
 from bot.chatgpt.chat_gpt_session import ChatGPTSession
+from bot.gemini.google_gemini_bot import GoogleGeminiBot
 from bot.session_manager import SessionManager
 from bridge.context import Context, ContextType
 from bridge.reply import Reply, ReplyType
@@ -15,6 +16,7 @@ from config import conf, pconf
 import threading
 from common import memory, utils
 import base64
+import os
 
 class LinkAIBot(Bot):
     # authentication failed
@@ -91,7 +93,8 @@ class LinkAIBot(Bot):
                 "frequency_penalty": conf().get("frequency_penalty", 0.0),  # [-2,2]之间，该值越大则更倾向于产生不同的内容
                 "presence_penalty": conf().get("presence_penalty", 0.0),  # [-2,2]之间，该值越大则更倾向于产生不同的内容
                 "session_id": session_id,
-                "channel_type": conf().get("channel_type")
+                "sender_id": session_id,
+                "channel_type": conf().get("channel_type", "wx")
             }
             try:
                 from linkai import LinkAIClient
@@ -106,7 +109,11 @@ class LinkAIBot(Bot):
                             body["group_name"] = context.kwargs.get("msg").from_user_nickname
                             body["sender_name"] = context.kwargs.get("msg").actual_user_nickname
                         else:
-                            body["sender_name"] = context.kwargs.get("msg").from_user_nickname
+                            if body.get("channel_type") in ["wechatcom_app"]:
+                                body["sender_name"] = context.kwargs.get("msg").from_user_id
+                            else:
+                                body["sender_name"] = context.kwargs.get("msg").from_user_nickname
+
             except Exception as e:
                 pass
             file_id = context.kwargs.get("file_id")
@@ -124,9 +131,12 @@ class LinkAIBot(Bot):
                 response = res.json()
                 reply_content = response["choices"][0]["message"]["content"]
                 total_tokens = response["usage"]["total_tokens"]
-                logger.info(f"[LINKAI] reply={reply_content}, total_tokens={total_tokens}")
-                self.sessions.session_reply(reply_content, session_id, total_tokens, query=query)
-    
+                res_code = response.get('code')
+                logger.info(f"[LINKAI] reply={reply_content}, total_tokens={total_tokens}, res_code={res_code}")
+                if res_code == 429:
+                    logger.warn(f"[LINKAI] 用户访问超出限流配置，sender_id={body.get('sender_id')}")
+                else:
+                    self.sessions.session_reply(reply_content, session_id, total_tokens, query=query)
                 agent_suffix = self._fetch_agent_suffix(response)
                 if agent_suffix:
                     reply_content += agent_suffix
@@ -155,7 +165,10 @@ class LinkAIBot(Bot):
                     logger.warn(f"[LINKAI] do retry, times={retry_count}")
                     return self._chat(query, context, retry_count + 1)
 
-                return Reply(ReplyType.TEXT, "提问太快啦，请休息一下再问我吧")
+                error_reply = "提问太快啦，请休息一下再问我吧"
+                if res.status_code == 409:
+                    error_reply = "这个问题我还没有学会，请问我其它问题吧"
+                return Reply(ReplyType.TEXT, error_reply)
 
         except Exception as e:
             logger.exception(e)
@@ -360,7 +373,9 @@ class LinkAIBot(Bot):
                         suffix += f"{turn.get('thought')}\n"
                     if plugin_name:
                         plugin_list.append(turn.get('plugin_name'))
-                        suffix += f"{turn.get('plugin_icon')} {turn.get('plugin_name')}"
+                        if turn.get('plugin_icon'):
+                            suffix += f"{turn.get('plugin_icon')} "
+                        suffix += f"{turn.get('plugin_name')}"
                         if turn.get('plugin_input'):
                             suffix += f"：{turn.get('plugin_input')}"
                     if i < len(chain) - 1:
@@ -383,14 +398,46 @@ class LinkAIBot(Bot):
     def _send_image(self, channel, context, image_urls):
         if not image_urls:
             return
+        max_send_num = conf().get("max_media_send_count")
+        send_interval = conf().get("media_send_interval")
         try:
+            i = 0
             for url in image_urls:
-                reply = Reply(ReplyType.IMAGE_URL, url)
+                if max_send_num and i >= max_send_num:
+                    continue
+                i += 1
+                if url.endswith(".mp4"):
+                    reply_type = ReplyType.VIDEO_URL
+                elif url.endswith(".pdf") or url.endswith(".doc") or url.endswith(".docx") or url.endswith(".csv"):
+                    reply_type = ReplyType.FILE
+                    url = _download_file(url)
+                    if not url:
+                        continue
+                else:
+                    reply_type = ReplyType.IMAGE_URL
+                reply = Reply(reply_type, url)
                 channel.send(reply, context)
+                if send_interval:
+                    time.sleep(send_interval)
         except Exception as e:
             logger.error(e)
 
 
+def _download_file(url: str):
+    try:
+        file_path = "tmp"
+        if not os.path.exists(file_path):
+            os.makedirs(file_path)
+        file_name = url.split("/")[-1]  # 获取文件名
+        file_path = os.path.join(file_path, file_name)
+        response = requests.get(url)
+        with open(file_path, "wb") as f:
+            f.write(response.content)
+        return file_path
+    except Exception as e:
+        logger.warn(e)
+
+
 class LinkAISessionManager(SessionManager):
     def session_msg_query(self, query, session_id):
         session = self.build_session(session_id)

bot/xunfei/xunfei_spark_bot.py

@@ -47,7 +47,8 @@ class XunFeiBot(Bot):
         # 默认使用v2.0版本: "ws://spark-api.xf-yun.com/v2.1/chat"
         # v1.5版本为: "ws://spark-api.xf-yun.com/v1.1/chat"
         # v3.0版本为: "ws://spark-api.xf-yun.com/v3.1/chat"
-        self.spark_url = "ws://spark-api.xf-yun.com/v3.1/chat"
+        # v3.5版本为: "wss://spark-api.xf-yun.com/v3.5/chat"
+        self.spark_url = "wss://spark-api.xf-yun.com/v3.5/chat"
         self.host = urlparse(self.spark_url).netloc
         self.path = urlparse(self.spark_url).path
         # 和wenxin使用相同的session机制

bot/zhipuai/zhipu_ai_image.py

@@ -0,0 +1,29 @@
+from common.log import logger
+from config import conf
+
+
+# ZhipuAI提供的画图接口
+
+class ZhipuAIImage(object):
+    def __init__(self):
+        from zhipuai import ZhipuAI
+        self.client = ZhipuAI(api_key=conf().get("zhipu_ai_api_key"))
+
+    def create_img(self, query, retry_count=0, api_key=None, api_base=None):
+        try:
+            if conf().get("rate_limit_dalle"):
+                return False, "请求太快了，请休息一下再问我吧"
+            logger.info("[ZHIPU_AI] image_query={}".format(query))
+            response = self.client.images.generations(
+                prompt=query,
+                n=1,  # 每次生成图片的数量
+                model=conf().get("text_to_image") or "cogview-3",
+                size=conf().get("image_create_size", "1024x1024"),  # 图片大小,可选有 256x256, 512x512, 1024x1024
+                quality="standard",
+            )
+            image_url = response.data[0].url
+            logger.info("[ZHIPU_AI] image_url={}".format(image_url))
+            return True, image_url
+        except Exception as e:
+            logger.exception(e)
+            return False, "画图出现问题，请休息一下再问我吧"

bot/zhipuai/zhipu_ai_session.py

@@ -0,0 +1,51 @@
+from bot.session_manager import Session
+from common.log import logger
+
+
+class ZhipuAISession(Session):
+    def __init__(self, session_id, system_prompt=None, model="glm-4"):
+        super().__init__(session_id, system_prompt)
+        self.model = model
+        self.reset()
+
+    def discard_exceeding(self, max_tokens, cur_tokens=None):
+        precise = True
+        try:
+            cur_tokens = self.calc_tokens()
+        except Exception as e:
+            precise = False
+            if cur_tokens is None:
+                raise e
+            logger.debug("Exception when counting tokens precisely for query: {}".format(e))
+        while cur_tokens > max_tokens:
+            if len(self.messages) > 2:
+                self.messages.pop(1)
+            elif len(self.messages) == 2 and self.messages[1]["role"] == "assistant":
+                self.messages.pop(1)
+                if precise:
+                    cur_tokens = self.calc_tokens()
+                else:
+                    cur_tokens = cur_tokens - max_tokens
+                break
+            elif len(self.messages) == 2 and self.messages[1]["role"] == "user":
+                logger.warn("user message exceed max_tokens. total_tokens={}".format(cur_tokens))
+                break
+            else:
+                logger.debug("max_tokens={}, total_tokens={}, len(messages)={}".format(max_tokens, cur_tokens,
+                                                                                       len(self.messages)))
+                break
+            if precise:
+                cur_tokens = self.calc_tokens()
+            else:
+                cur_tokens = cur_tokens - max_tokens
+        return cur_tokens
+
+    def calc_tokens(self):
+        return num_tokens_from_messages(self.messages, self.model)
+
+
+def num_tokens_from_messages(messages, model):
+    tokens = 0
+    for msg in messages:
+        tokens += len(msg["content"])
+    return tokens

bot/zhipuai/zhipuai_bot.py

@@ -0,0 +1,149 @@
+# encoding:utf-8
+
+import time
+
+import openai
+import openai.error
+from bot.bot import Bot
+from bot.zhipuai.zhipu_ai_session import ZhipuAISession
+from bot.zhipuai.zhipu_ai_image import ZhipuAIImage
+from bot.session_manager import SessionManager
+from bridge.context import ContextType
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from config import conf, load_config
+from zhipuai import ZhipuAI
+
+
+# ZhipuAI对话模型API
+class ZHIPUAIBot(Bot, ZhipuAIImage):
+    def __init__(self):
+        super().__init__()
+        self.sessions = SessionManager(ZhipuAISession, model=conf().get("model") or "ZHIPU_AI")
+        self.args = {
+            "model": conf().get("model") or "glm-4",  # 对话模型的名称
+            "temperature": conf().get("temperature", 0.9),  # 值在(0,1)之间(智谱AI 的温度不能取 0 或者 1)
+            "top_p": conf().get("top_p", 0.7),  # 值在(0,1)之间(智谱AI 的 top_p 不能取 0 或者 1)
+        }
+        self.client = ZhipuAI(api_key=conf().get("zhipu_ai_api_key"))
+
+    def reply(self, query, context=None):
+        # acquire reply content
+        if context.type == ContextType.TEXT:
+            logger.info("[ZHIPU_AI] query={}".format(query))
+
+            session_id = context["session_id"]
+            reply = None
+            clear_memory_commands = conf().get("clear_memory_commands", ["#清除记忆"])
+            if query in clear_memory_commands:
+                self.sessions.clear_session(session_id)
+                reply = Reply(ReplyType.INFO, "记忆已清除")
+            elif query == "#清除所有":
+                self.sessions.clear_all_session()
+                reply = Reply(ReplyType.INFO, "所有人记忆已清除")
+            elif query == "#更新配置":
+                load_config()
+                reply = Reply(ReplyType.INFO, "配置已更新")
+            if reply:
+                return reply
+            session = self.sessions.session_query(query, session_id)
+            logger.debug("[ZHIPU_AI] session query={}".format(session.messages))
+
+            api_key = context.get("openai_api_key") or openai.api_key
+            model = context.get("gpt_model")
+            new_args = None
+            if model:
+                new_args = self.args.copy()
+                new_args["model"] = model
+            # if context.get('stream'):
+            #     # reply in stream
+            #     return self.reply_text_stream(query, new_query, session_id)
+
+            reply_content = self.reply_text(session, api_key, args=new_args)
+            logger.debug(
+                "[ZHIPU_AI] new_query={}, session_id={}, reply_cont={}, completion_tokens={}".format(
+                    session.messages,
+                    session_id,
+                    reply_content["content"],
+                    reply_content["completion_tokens"],
+                )
+            )
+            if reply_content["completion_tokens"] == 0 and len(reply_content["content"]) > 0:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+            elif reply_content["completion_tokens"] > 0:
+                self.sessions.session_reply(reply_content["content"], session_id, reply_content["total_tokens"])
+                reply = Reply(ReplyType.TEXT, reply_content["content"])
+            else:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+                logger.debug("[ZHIPU_AI] reply {} used 0 tokens.".format(reply_content))
+            return reply
+        elif context.type == ContextType.IMAGE_CREATE:
+            ok, retstring = self.create_img(query, 0)
+            reply = None
+            if ok:
+                reply = Reply(ReplyType.IMAGE_URL, retstring)
+            else:
+                reply = Reply(ReplyType.ERROR, retstring)
+            return reply
+
+        else:
+            reply = Reply(ReplyType.ERROR, "Bot不支持处理{}类型的消息".format(context.type))
+            return reply
+
+    def reply_text(self, session: ZhipuAISession, api_key=None, args=None, retry_count=0) -> dict:
+        """
+        call openai's ChatCompletion to get the answer
+        :param session: a conversation session
+        :param session_id: session id
+        :param retry_count: retry count
+        :return: {}
+        """
+        try:
+            # if conf().get("rate_limit_chatgpt") and not self.tb4chatgpt.get_token():
+            #     raise openai.error.RateLimitError("RateLimitError: rate limit exceeded")
+            # if api_key == None, the default openai.api_key will be used
+            if args is None:
+                args = self.args
+            # response = openai.ChatCompletion.create(api_key=api_key, messages=session.messages, **args)
+            response = self.client.chat.completions.create(messages=session.messages, **args)
+            # logger.debug("[ZHIPU_AI] response={}".format(response))
+            # logger.info("[ZHIPU_AI] reply={}, total_tokens={}".format(response.choices[0]['message']['content'], response["usage"]["total_tokens"]))
+
+            return {
+                "total_tokens": response.usage.total_tokens,
+                "completion_tokens": response.usage.completion_tokens,
+                "content": response.choices[0].message.content,
+            }
+        except Exception as e:
+            need_retry = retry_count < 2
+            result = {"completion_tokens": 0, "content": "我现在有点累了，等会再来吧"}
+            if isinstance(e, openai.error.RateLimitError):
+                logger.warn("[ZHIPU_AI] RateLimitError: {}".format(e))
+                result["content"] = "提问太快啦，请休息一下再问我吧"
+                if need_retry:
+                    time.sleep(20)
+            elif isinstance(e, openai.error.Timeout):
+                logger.warn("[ZHIPU_AI] Timeout: {}".format(e))
+                result["content"] = "我没有收到你的消息"
+                if need_retry:
+                    time.sleep(5)
+            elif isinstance(e, openai.error.APIError):
+                logger.warn("[ZHIPU_AI] Bad Gateway: {}".format(e))
+                result["content"] = "请再问我一次"
+                if need_retry:
+                    time.sleep(10)
+            elif isinstance(e, openai.error.APIConnectionError):
+                logger.warn("[ZHIPU_AI] APIConnectionError: {}".format(e))
+                result["content"] = "我连接不到你的网络"
+                if need_retry:
+                    time.sleep(5)
+            else:
+                logger.exception("[ZHIPU_AI] Exception: {}".format(e), e)
+                need_retry = False
+                self.sessions.clear_session(session.session_id)
+
+            if need_retry:
+                logger.warn("[ZHIPU_AI] 第{}次重试".format(retry_count + 1))
+                return self.reply_text(session, api_key, args, retry_count + 1)
+            else:
+                return result

bridge/bridge.py

@@ -18,6 +18,7 @@ class Bridge(object):
             "text_to_voice": conf().get("text_to_voice", "google"),
             "translate": conf().get("translate", "baidu"),
         }
+        # 这边取配置的模型
         model_type = conf().get("model") or const.GPT35
         if model_type in ["text-davinci-003"]:
             self.btype["chat"] = const.OPEN_AI
@@ -31,6 +32,10 @@ class Bridge(object):
             self.btype["chat"] = const.QWEN
         if model_type in [const.GEMINI]:
             self.btype["chat"] = const.GEMINI
+        if model_type in [const.ZHIPU_AI]:
+            self.btype["chat"] = const.ZHIPU_AI
+        if model_type and model_type.startswith("claude-3"):
+            self.btype["chat"] = const.CLAUDEAPI
 
         if conf().get("use_linkai") and conf().get("linkai_api_key"):
             self.btype["chat"] = const.LINKAI
@@ -38,12 +43,12 @@ class Bridge(object):
                 self.btype["voice_to_text"] = const.LINKAI
             if not conf().get("text_to_voice") or conf().get("text_to_voice") in ["openai", const.TTS_1, const.TTS_1_HD]:
                 self.btype["text_to_voice"] = const.LINKAI
-
         if model_type in ["claude"]:
             self.btype["chat"] = const.CLAUDEAI
+
         self.bots = {}
         self.chat_bots = {}
-
+    # 模型对应的接口
     def get_bot(self, typename):
         if self.bots.get(typename) is None:
             logger.info("create bot {} for {}".format(self.btype[typename], typename))

bridge/reply.py

@@ -11,7 +11,7 @@ class ReplyType(Enum):
     VIDEO_URL = 5  # 视频URL
     FILE = 6  # 文件
     CARD = 7  # 微信名片，仅支持ntchat
-    InviteRoom = 8  # 邀请好友进群
+    INVITE_ROOM = 8  # 邀请好友进群
     INFO = 9
     ERROR = 10
     TEXT_ = 11  # 强制文本

channel/chat_channel.py

@@ -4,6 +4,7 @@ import threading
 import time
 from asyncio import CancelledError
 from concurrent.futures import Future, ThreadPoolExecutor
+from concurrent import futures
 
 from bridge.context import *
 from bridge.reply import *
@@ -17,6 +18,8 @@ try:
 except Exception as e:
     pass
 
+handler_pool = ThreadPoolExecutor(max_workers=8)  # 处理消息的线程池
+
 
 # 抽象类, 它包含了与消息通道无关的通用处理逻辑
 class ChatChannel(Channel):
@@ -25,7 +28,6 @@ class ChatChannel(Channel):
     futures = {}  # 记录每个session_id提交到线程池的future对象, 用于重置会话时把没执行的future取消掉，正在执行的不会被取消
     sessions = {}  # 用于控制并发，每个session_id同时只能有一个context在处理
     lock = threading.Lock()  # 用于控制对sessions的访问
-    handler_pool = ThreadPoolExecutor(max_workers=8)  # 处理消息的线程池
 
     def __init__(self):
         _thread = threading.Thread(target=self.consume)
@@ -168,11 +170,13 @@ class ChatChannel(Channel):
         reply = self._generate_reply(context)
 
         logger.debug("[WX] ready to decorate reply: {}".format(reply))
-        # reply的包装步骤
-        reply = self._decorate_reply(context, reply)
 
-        # reply的发送步骤
-        self._send_reply(context, reply)
+        # reply的包装步骤
+        if reply and reply.content:
+            reply = self._decorate_reply(context, reply)
+
+            # reply的发送步骤
+            self._send_reply(context, reply)
 
     def _generate_reply(self, context: Context, reply: Reply = Reply()) -> Reply:
         e_context = PluginManager().emit_event(
@@ -339,7 +343,7 @@ class ChatChannel(Channel):
                         if not context_queue.empty():
                             context = context_queue.get()
                             logger.debug("[WX] consume context: {}".format(context))
-                            future: Future = self.handler_pool.submit(self._handle, context)
+                            future: Future = handler_pool.submit(self._handle, context)
                             future.add_done_callback(self._thread_pool_callback(session_id, context=context))
                             if session_id not in self.futures:
                                 self.futures[session_id] = []

channel/wechat/wechat_channel.py

@@ -15,6 +15,7 @@ import requests
 from bridge.context import *
 from bridge.reply import *
 from channel.chat_channel import ChatChannel
+from channel import chat_channel
 from channel.wechat.wechat_message import *
 from common.expired_dict import ExpiredDict
 from common.log import logger
@@ -95,7 +96,7 @@ def qrCallback(uuid, status, qrcode):
         print(qr_api4)
         print(qr_api2)
         print(qr_api1)
-
+        _send_qr_code([qr_api1, qr_api2, qr_api3, qr_api4])
         qr = qrcode.QRCode(border=1)
         qr.add_data(url)
         qr.make(fit=True)
@@ -112,31 +113,43 @@ class WechatChannel(ChatChannel):
         self.auto_login_times = 0
 
     def startup(self):
-        itchat.instance.receivingRetryCount = 600  # 修改断线超时时间
-        # login by scan QRCode
-        hotReload = conf().get("hot_reload", False)
-        status_path = os.path.join(get_appdata_dir(), "itchat.pkl")
-        itchat.auto_login(
-            enableCmdQR=2,
-            hotReload=hotReload,
-            statusStorageDir=status_path,
-            qrCallback=qrCallback,
-            exitCallback=self.exitCallback,
-            loginCallback=self.loginCallback
-        )
-        self.user_id = itchat.instance.storageClass.userName
-        self.name = itchat.instance.storageClass.nickName
-        logger.info("Wechat login success, user_id: {}, nickname: {}".format(self.user_id, self.name))
-        # start message listener
-        itchat.run()
+        try:
+            itchat.instance.receivingRetryCount = 600  # 修改断线超时时间
+            # login by scan QRCode
+            hotReload = conf().get("hot_reload", False)
+            status_path = os.path.join(get_appdata_dir(), "itchat.pkl")
+            itchat.auto_login(
+                enableCmdQR=2,
+                hotReload=hotReload,
+                statusStorageDir=status_path,
+                qrCallback=qrCallback,
+                exitCallback=self.exitCallback,
+                loginCallback=self.loginCallback
+            )
+            self.user_id = itchat.instance.storageClass.userName
+            self.name = itchat.instance.storageClass.nickName
+            logger.info("Wechat login success, user_id: {}, nickname: {}".format(self.user_id, self.name))
+            # start message listener
+            itchat.run()
+        except Exception as e:
+            logger.error(e)
 
     def exitCallback(self):
-        self.auto_login_times += 1
-        if self.auto_login_times < 100:
-            self.startup()
+        try:
+            from common.linkai_client import chat_client
+            if chat_client.client_id and conf().get("use_linkai"):
+                _send_logout()
+                time.sleep(2)
+                self.auto_login_times += 1
+                if self.auto_login_times < 100:
+                    chat_channel.handler_pool._shutdown = False
+                    self.startup()
+        except Exception as e:
+            pass
 
     def loginCallback(self):
-        pass
+        logger.debug("Login success")
+        _send_login_success()
 
     # handle_* 系列函数处理收到的消息后构造Context，然后传入produce函数中处理Context和发送回复
     # Context包含了消息的所有信息，包括以下属性
@@ -149,7 +162,6 @@ class WechatChannel(ChatChannel):
     #        msg: ChatMessage消息对象
     #        origin_ctype: 原始消息类型，语音转文字后，私聊时如果匹配前缀失败，会根据初始消息是否是语音来放宽触发规则
     #        desire_rtype: 希望回复类型，默认是文本回复，设置为ReplyType.VOICE是语音回复
-
     @time_checker
     @_check
     def handle_single(self, cmsg: ChatMessage):
@@ -221,7 +233,6 @@ class WechatChannel(ChatChannel):
             logger.info("[WX] sendImage url={}, receiver={}".format(img_url, receiver))
         elif reply.type == ReplyType.IMAGE:  # 从文件读取图片
             image_storage = reply.content
-            image_storage.seek(0)
             itchat.send_image(image_storage, toUserName=receiver)
             logger.info("[WX] sendImage, receiver={}".format(receiver))
         elif reply.type == ReplyType.FILE:  # 新增文件回复类型
@@ -245,3 +256,27 @@ class WechatChannel(ChatChannel):
             video_storage.seek(0)
             itchat.send_video(video_storage, toUserName=receiver)
             logger.info("[WX] sendVideo url={}, receiver={}".format(video_url, receiver))
+
+def _send_login_success():
+    try:
+        from common.linkai_client import chat_client
+        if chat_client.client_id:
+            chat_client.send_login_success()
+    except Exception as e:
+        pass
+
+def _send_logout():
+    try:
+        from common.linkai_client import chat_client
+        if chat_client.client_id:
+            chat_client.send_logout()
+    except Exception as e:
+        pass
+
+def _send_qr_code(qrcode_list: list):
+    try:
+        from common.linkai_client import chat_client
+        if chat_client.client_id:
+            chat_client.send_qrcode(qrcode_list)
+    except Exception as e:
+        pass

channel/wework/wework_message.py

@@ -64,7 +64,10 @@ def cdn_download(wework, message, file_name):
         }
         result = wework._WeWork__send_sync(send_type.MT_WXCDN_DOWNLOAD_MSG, data)  # 直接用wx_cdn_download的接口内部实现来调用
     elif "file_id" in data["cdn"].keys():
-        file_type = 2
+        if message["type"] == 11042:
+            file_type = 2
+        elif message["type"] == 11045:
+            file_type = 5
         file_id = data["cdn"]["file_id"]
         result = wework.c2c_cdn_download(file_id, aes_key, file_size, file_type, save_path)
     else:

common/const.py

@@ -6,19 +6,25 @@ XUNFEI = "xunfei"
 CHATGPTONAZURE = "chatGPTOnAzure"
 LINKAI = "linkai"
 CLAUDEAI = "claude"
+CLAUDEAPI= "claudeAPI"
 QWEN = "qwen"
 GEMINI = "gemini"
+ZHIPU_AI = "glm-4"
+MOONSHOT = "moonshot"
+
 
 # model
+CLAUDE3 = "claude-3-opus-20240229"
 GPT35 = "gpt-3.5-turbo"
 GPT4 = "gpt-4"
-GPT4_TURBO_PREVIEW = "gpt-4-1106-preview"
+GPT4_TURBO_PREVIEW = "gpt-4-0125-preview"
 GPT4_VISION_PREVIEW = "gpt-4-vision-preview"
 WHISPER_1 = "whisper-1"
 TTS_1 = "tts-1"
 TTS_1_HD = "tts-1-hd"
 
-MODEL_LIST = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4", "wenxin", "wenxin-4", "xunfei", "claude", "gpt-4-turbo", GPT4_TURBO_PREVIEW, QWEN, GEMINI]
+MODEL_LIST = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4", "wenxin", "wenxin-4", "xunfei", "claude","claude-3-opus-20240229", "gpt-4-turbo",
+              "gpt-4-turbo-preview", "gpt-4-1106-preview", GPT4_TURBO_PREVIEW, QWEN, GEMINI, ZHIPU_AI, MOONSHOT]
 
 # channel
 FEISHU = "feishu"

common/linkai_client.py

@@ -2,9 +2,12 @@ from bridge.context import Context, ContextType
 from bridge.reply import Reply, ReplyType
 from common.log import logger
 from linkai import LinkAIClient, PushMsg
-from config import conf
+from config import conf, pconf, plugin_config
+from plugins import PluginManager
 
 
+chat_client: LinkAIClient
+
 class ChatClient(LinkAIClient):
     def __init__(self, api_key, host, channel):
         super().__init__(api_key, host)
@@ -21,8 +24,32 @@ class ChatClient(LinkAIClient):
         context["isgroup"] = push_msg.is_group
         self.channel.send(Reply(ReplyType.TEXT, content=msg_content), context)
 
+    def on_config(self, config: dict):
+        if not self.client_id:
+            return
+        logger.info(f"从控制台加载配置: {config}")
+        local_config = conf()
+        for key in local_config.keys():
+            if config.get(key) is not None:
+                local_config[key] = config.get(key)
+        if config.get("reply_voice_mode"):
+            if config.get("reply_voice_mode") == "voice_reply_voice":
+                local_config["voice_reply_voice"] = True
+            elif config.get("reply_voice_mode") == "always_reply_voice":
+                local_config["always_reply_voice"] = True
+        # if config.get("admin_password") and plugin_config["Godcmd"]:
+        #     plugin_config["Godcmd"]["password"] = config.get("admin_password")
+        #     PluginManager().instances["Godcmd"].reload()
+        # if config.get("group_app_map") and pconf("linkai"):
+        #     local_group_map = {}
+        #     for mapping in config.get("group_app_map"):
+        #         local_group_map[mapping.get("group_name")] = mapping.get("app_code")
+        #     pconf("linkai")["group_app_map"] = local_group_map
+        #     PluginManager().instances["linkai"].reload()
+
 
 def start(channel):
-    client = ChatClient(api_key=conf().get("linkai_api_key"),
+    global chat_client
+    chat_client = ChatClient(api_key=conf().get("linkai_api_key"),
                         host="link-ai.chat", channel=channel)
-    client.start()
+    chat_client.start()

config-template.json

@@ -2,6 +2,7 @@
   "channel_type": "wx",
   "model": "",
   "open_ai_api_key": "YOUR API KEY",
+  "claude_api_key": "YOUR API KEY",
   "text_to_image": "dall-e-2",
   "voice_to_text": "openai",
   "text_to_voice": "openai",

config.py

@@ -67,6 +67,8 @@ available_setting = {
     # claude 配置
     "claude_api_cookie": "",
     "claude_uuid": "",
+    # claude api key
+    "claude_api_key":"",
     # 通义千问API, 获取方式查看文档 https://help.aliyun.com/document_detail/2587494.html
     "qwen_access_key_id": "",
     "qwen_access_key_secret": "",
@@ -83,7 +85,7 @@ available_setting = {
     "voice_reply_voice": False,  # 是否使用语音回复语音，需要设置对应语音合成引擎的api key
     "always_reply_voice": False,  # 是否一直使用语音回复
     "voice_to_text": "openai",  # 语音识别引擎，支持openai,baidu,google,azure
-    "text_to_voice": "openai",  # 语音合成引擎，支持openai,baidu,google,pytts(offline),azure,elevenlabs
+    "text_to_voice": "openai",  # 语音合成引擎，支持openai,baidu,google,pytts(offline),azure,elevenlabs,edge(online)
     "text_to_voice_model": "tts-1",
     "tts_voice_id": "alloy",
     # baidu 语音api配置， 使用百度语音识别和语音合成时需要
@@ -148,7 +150,12 @@ available_setting = {
     "plugin_trigger_prefix": "$",  # 规范插件提供聊天相关指令的前缀，建议不要和管理员指令前缀"#"冲突
     # 是否使用全局插件配置
     "use_global_plugin_config": False,
-    # 知识库平台配置
+    "max_media_send_count": 3,     # 单次最大发送媒体资源的个数
+    "media_send_interval": 1,  # 发送图片的事件间隔，单位秒
+    # 智谱AI 平台配置
+    "zhipu_ai_api_key": "",
+    "zhipu_ai_api_base": "https://open.bigmodel.cn/api/paas/v4",
+    # LinkAI平台配置
     "use_linkai": False,
     "linkai_api_key": "",
     "linkai_app_code": "",

docs/version/old-version.md

@@ -0,0 +1,13 @@
+## 归档更新日志
+
+2023.04.26： 支持企业微信应用号部署，兼容插件，并支持语音图片交互，私人助理理想选择，使用文档。(contributed by @lanvent in #944)
+
+2023.04.05： 支持微信公众号部署，兼容插件，并支持语音图片交互，使用文档。(contributed by @JS00000 in #686)
+
+2023.04.05： 增加能让ChatGPT使用工具的tool插件，使用文档。工具相关issue可反馈至chatgpt-tool-hub。(contributed by @goldfishh in #663)
+
+2023.03.25： 支持插件化开发，目前已实现 多角色切换、文字冒险游戏、管理员指令、Stable Diffusion等插件，使用参考 #578。(contributed by @lanvent in #565)
+
+2023.03.09： 基于 whisper API(后续已接入更多的语音API服务) 实现对微信语音消息的解析和回复，添加配置项 "speech_recognition":true 即可启用，使用参考 #415。(contributed by wanggang1987 in #385)
+
+2023.02.09： 扫码登录存在账号限制风险，请谨慎使用，参考#58

plugins/godcmd/godcmd.py

@@ -313,7 +313,7 @@ class Godcmd(Plugin):
                     except Exception as e:
                         ok, result = False, "你没有设置私有GPT模型"
                 elif cmd == "reset":
-                    if bottype in [const.OPEN_AI, const.CHATGPT, const.CHATGPTONAZURE, const.LINKAI, const.BAIDU, const.XUNFEI, const.QWEN, const.GEMINI]:
+                    if bottype in [const.OPEN_AI, const.CHATGPT, const.CHATGPTONAZURE, const.LINKAI, const.BAIDU, const.XUNFEI, const.QWEN, const.GEMINI, const.ZHIPU_AI]:
                         bot.sessions.clear_session(session_id)
                         if Bridge().chat_bots.get(bottype):
                             Bridge().chat_bots.get(bottype).sessions.clear_session(session_id)
@@ -339,7 +339,7 @@ class Godcmd(Plugin):
                             ok, result = True, "配置已重载"
                         elif cmd == "resetall":
                             if bottype in [const.OPEN_AI, const.CHATGPT, const.CHATGPTONAZURE, const.LINKAI,
-                                           const.BAIDU, const.XUNFEI, const.QWEN, const.GEMINI]:
+                                           const.BAIDU, const.XUNFEI, const.QWEN, const.GEMINI, const.ZHIPU_AI, const.MOONSHOT]:
                                 channel.cancel_all_session()
                                 bot.sessions.clear_all_session()
                                 ok, result = True, "重置所有会话成功"
@@ -475,3 +475,11 @@ class Godcmd(Plugin):
         if model == "gpt-4-turbo":
             return const.GPT4_TURBO_PREVIEW
         return model
+
+    def reload(self):
+        gconf = plugin_config[self.name]
+        if gconf:
+            if gconf.get("password"):
+                self.password = gconf["password"]
+            if gconf.get("admin_users"):
+                self.admin_users = gconf["admin_users"]

plugins/plugin.py

@@ -46,3 +46,6 @@ class Plugin:
 
     def get_help_text(self, **kwargs):
         return "暂无帮助信息"
+
+    def reload(self):
+        pass

plugins/plugin_manager.py

@@ -99,7 +99,7 @@ class PluginManager:
                     try:
                         self.current_plugin_path = plugin_path
                         if plugin_path in self.loaded:
-                            if self.loaded[plugin_path] == None:
+                            if plugin_name.upper() != 'GODCMD':
                                 logger.info("reload module %s" % plugin_name)
                                 self.loaded[plugin_path] = importlib.reload(sys.modules[import_path])
                                 dependent_module_names = [name for name in sys.modules.keys() if name.startswith(import_path + ".")]
@@ -141,19 +141,21 @@ class PluginManager:
         failed_plugins = []
         for name, plugincls in self.plugins.items():
             if plugincls.enabled:
-                if name not in self.instances:
-                    try:
-                        instance = plugincls()
-                    except Exception as e:
-                        logger.warn("Failed to init %s, diabled. %s" % (name, e))
-                        self.disable_plugin(name)
-                        failed_plugins.append(name)
-                        continue
-                    self.instances[name] = instance
-                    for event in instance.handlers:
-                        if event not in self.listening_plugins:
-                            self.listening_plugins[event] = []
-                        self.listening_plugins[event].append(name)
+                if 'GODCMD' in self.instances and name == 'GODCMD':
+                    continue
+                # if name not in self.instances:
+                try:
+                    instance = plugincls()
+                except Exception as e:
+                    logger.warn("Failed to init %s, diabled. %s" % (name, e))
+                    self.disable_plugin(name)
+                    failed_plugins.append(name)
+                    continue
+                self.instances[name] = instance
+                for event in instance.handlers:
+                    if event not in self.listening_plugins:
+                        self.listening_plugins[event] = []
+                    self.listening_plugins[event].append(name)
         self.refresh_order()
         return failed_plugins
 

plugins/source.json

@@ -19,6 +19,14 @@
     "Apilot": {
       "url": "https://github.com/6vision/Apilot.git",
       "desc": "通过api直接查询早报、热榜、快递、天气等实用信息的插件"
+    },
+    "pictureChange": {
+      "url": "https://github.com/Yanyutin753/pictureChange.git",
+      "desc": "利用stable-diffusion和百度Ai进行图生图或者画图的插件"
+    },
+    "Blackroom": {
+      "url": "https://github.com/dividduang/blackroom.git",
+      "desc": "小黑屋插件，被拉进小黑屋的人将不能使用@bot的功能的插件"
     }
   }
 }

plugins/tool/README.md

@@ -3,11 +3,19 @@
 使用说明(默认trigger_prefix为$)：  
 ```text
 #help tool: 查看tool帮助信息，可查看已加载工具列表  
-$tool 命令: 根据给出的{命令}使用一些可用工具尽力为你得到结果。  
+$tool 工具名 命令: （pure模式）根据给出的{命令}使用指定 一个 可用工具尽力为你得到结果。
+$tool 命令: （多工具模式）根据给出的{命令}使用 一些 可用工具尽力为你得到结果。  
 $tool reset: 重置工具。  
 ```
 ### 本插件所有工具同步存放至专用仓库：[chatgpt-tool-hub](https://github.com/goldfishh/chatgpt-tool-hub)
 
+2024.01.16更新
+1. 新增工具pure模式，支持单个工具调用
+2. 新增消息转发工具：email, sms, wechat, 可以根据规则向其他平台发送消息
+3. 替换visual-dl（更名为visual）实现，目前识别图片链接效果较好。
+4. 修复了0.4版本大部分工具返回结果不可靠问题
+
+新版本工具名共19个，不一一列举，相应工具需要的环境参数见`tool.py`里的`_build_tool_kwargs`函数
 
 ## 使用说明
 使用该插件后将默认使用4个工具, 无需额外配置长期生效：
@@ -24,7 +32,7 @@ $tool reset: 重置工具。
 
 > 注1：url-get默认配置、browser需额外配置，browser依赖google-chrome，你需要提前安装好
 
-> 注2：当检测到长文本时会进入summary tool总结长文本，tokens可能会大量消耗！
+> 注2：（可通过`browser_use_summary`或 `url_get_use_summary`开关）当检测到长文本时会进入summary tool总结长文本，tokens可能会大量消耗！
 
 这是debian端安装google-chrome教程，其他系统请自行查找
 > https://www.linuxjournal.com/content/how-can-you-install-google-browser-debian
@@ -34,9 +42,10 @@ $tool reset: 重置工具。
 
 > terminal调优记录：https://github.com/zhayujie/chatgpt-on-wechat/issues/776#issue-1659347640
 
-### 4. meteo-weather
+### 4. meteo
 ###### 回答你有关天气的询问, 需要获取时间、地点上下文信息，本工具使用了[meteo open api](https://open-meteo.com/)
 注：该工具需要较高的对话技巧，不保证你问的任何问题均能得到满意的回复
+注2：当前版本可只使用这个工具，返回结果较可控。
 
 > meteo调优记录：https://github.com/zhayujie/chatgpt-on-wechat/issues/776#issuecomment-1500771334
 
@@ -65,18 +74,12 @@ $tool reset: 重置工具。
 #### 6.2. morning-news *
 ###### 每日60秒早报，每天凌晨一点更新，本工具使用了[alapi-每日60秒早报](https://alapi.cn/api/view/93)
 
-```text
-可配置参数：
-1. morning_news_use_llm: 是否使用LLM润色结果，默认false（可能会慢）
-```
-
 > 该tool每天返回内容相同
 
 #### 6.3. finance-news
 ###### 获取实时的金融财政新闻
 
-> 该工具需要解决browser tool 的google-chrome依赖安装
-
+> 该工具需要用到browser工具解决反爬问题
 
 
 ### 7. bing-search *
@@ -99,18 +102,33 @@ $tool reset: 重置工具。
 > 0.4.2更新，例子：帮我找一篇吴恩达写的论文
 
 ### 11. summary
-###### 总结工具，该工具必须输入一个本地文件的绝对路径
+###### 总结工具，该工具可以支持输入url
 
 > 该工具目前是和其他工具配合使用，暂未测试单独使用效果
 
-### 12. image2text
-###### 将图片转换成文字，底层调用imageCaption模型，该工具必须输入一个本地文件的绝对路径
+### 12. visual
+###### 将图片转换成文字，底层调用ali dashscope `qwen-vl-plus`模型
 
 ### 13. searxng-search *
 ###### 一个私有化的搜索引擎工具
 
 > 安装教程：https://docs.searxng.org/admin/installation.html
 
+### 14. email *
+###### 发送邮件
+
+### 15. sms *
+###### 发送短信
+
+### 16. stt *
+###### speak to text 语音识别
+
+### 17. tts *
+###### text to speak 文生语音
+
+### 18. wechat *
+###### 向好友、群组发送微信
+
 ---
 
 ###### 注1：带*工具需要获取api-key才能使用(在config.json内的kwargs添加项)，部分工具需要外网支持  
@@ -120,7 +138,7 @@ $tool reset: 重置工具。
 ###### 默认工具无需配置，其它工具需手动配置，以增加morning-news和bing-search两个工具为例：
 ```json
 {
-  "tools": ["bing-search", "news", "你想要添加的其他工具"],  // 填入你想用到的额外工具名，这里加入了工具"bing-search"和工具"news"(news工具会自动加载morning-news、finance-news等子工具)
+  "tools": ["bing-search", "morning-news", "你想要添加的其他工具"],  // 填入你想用到的额外工具名，这里加入了工具"bing-search"和工具"morning-news"
   "kwargs": {
       "debug": true, // 当你遇到问题求助时，需要配置
       "request_timeout": 120,  // openai接口超时时间
@@ -137,7 +155,6 @@ $tool reset: 重置工具。
   - `debug`: 输出chatgpt-tool-hub额外信息用于调试
   - `request_timeout`: 访问openai接口的超时时间，默认与wechat-on-chatgpt配置一致，可单独配置
   - `no_default`: 用于配置默认加载4个工具的行为，如果为true则仅使用tools列表工具，不加载默认工具
-  - `top_k_results`: 控制所有有关搜索的工具返回条目数，数字越高则参考信息越多，但无用信息可能干扰判断，该值一般为2
   - `model_name`: 用于控制tool插件底层使用的llm模型，目前暂未测试3.5以外的模型，一般保持默认
 
 ---

plugins/tool/config.json.template

@@ -3,10 +3,10 @@
     "python",
     "url-get",
     "terminal",
-    "meteo-weather"
+    "meteo"
   ],
   "kwargs": {
-    "top_k_results": 2,
+    "debug": false,
     "no_default": false,
     "model_name": "gpt-3.5-turbo"
   }

plugins/tool/tool.py

@@ -1,23 +1,20 @@
-import json
-import os
-
 from chatgpt_tool_hub.apps import AppFactory
 from chatgpt_tool_hub.apps.app import App
-from chatgpt_tool_hub.tools.all_tool_list import get_all_tool_names
+from chatgpt_tool_hub.tools.tool_register import main_tool_register
 
 import plugins
 from bridge.bridge import Bridge
 from bridge.context import ContextType
 from bridge.reply import Reply, ReplyType
 from common import const
-from config import conf
+from config import conf, get_appdata_dir
 from plugins import *
 
 
 @plugins.register(
     name="tool",
     desc="Arming your ChatGPT bot with various tools",
-    version="0.4",
+    version="0.5",
     author="goldfishh",
     desire_priority=0,
 )
@@ -36,10 +33,12 @@ class Tool(Plugin):
         if not verbose:
             return help_text
         help_text += "\n使用说明：\n"
-        help_text += f"{trigger_prefix}tool " + "命令: 根据给出的{命令}使用一些可用工具尽力为你得到结果。\n"
+        help_text += f"{trigger_prefix}tool " + "命令: 根据给出的{命令}模型来选择使用哪些工具尽力为你得到结果。\n"
+        help_text += f"{trigger_prefix}tool 工具名 " + "命令: 根据给出的{命令}使用指定工具尽力为你得到结果。\n"
         help_text += f"{trigger_prefix}tool reset: 重置工具。\n\n"
+
         help_text += f"已加载工具列表: \n"
-        for idx, tool in enumerate(self.app.get_tool_list()):
+        for idx, tool in enumerate(main_tool_register.get_registered_tool_names()):
             if idx != 0:
                 help_text += ", "
             help_text += f"{tool}"
@@ -91,17 +90,28 @@ class Tool(Plugin):
 
                     e_context.action = EventAction.BREAK
                     return
-
                 query = content_list[1].strip()
+                
+                use_one_tool = False
+                for tool_name in main_tool_register.get_registered_tool_names():
+                    if query.startswith(tool_name):
+                        use_one_tool = True
+                        query = query[len(tool_name):]
+                        break
 
                 # Don't modify bot name
                 all_sessions = Bridge().get_bot("chat").sessions
                 user_session = all_sessions.session_query(query, e_context["context"]["session_id"]).messages
 
-                # chatgpt-tool-hub will reply you with many tools
                 logger.debug("[tool]: just-go")
                 try:
-                    _reply = self.app.ask(query, user_session)
+                    if use_one_tool:
+                        _func, _ = main_tool_register.get_registered_tool()[tool_name]
+                        tool = _func(**self.app_kwargs)
+                        _reply = tool.run(query)
+                    else:
+                        # chatgpt-tool-hub will reply you with many tools
+                        _reply = self.app.ask(query, user_session)
                     e_context.action = EventAction.BREAK_PASS
                     all_sessions.session_reply(_reply, e_context["context"]["session_id"])
                 except Exception as e:
@@ -126,53 +136,111 @@ class Tool(Plugin):
         request_timeout = kwargs.get("request_timeout")
 
         return {
-            "debug": kwargs.get("debug", False),
-            "openai_api_key": conf().get("open_ai_api_key", ""),
-            "open_ai_api_base": conf().get("open_ai_api_base", "https://api.openai.com/v1"),
-            "deployment_id": conf().get("azure_deployment_id", ""),
-            "proxy": conf().get("proxy", ""),
+            # 全局配置相关
+            "log": False,  # tool 日志开关
+            "debug": kwargs.get("debug", False),  # 输出更多日志
+            "no_default": kwargs.get("no_default", False),  # 不要默认的工具，只加载自己导入的工具
+            "think_depth": kwargs.get("think_depth", 2),  # 一个问题最多使用多少次工具
+            "proxy": conf().get("proxy", ""),  # 科学上网
             "request_timeout": request_timeout if request_timeout else conf().get("request_timeout", 120),
+            "temperature": kwargs.get("temperature", 0),  # llm 温度，建议设置0
+            # LLM配置相关
+            "llm_api_key": conf().get("open_ai_api_key", ""),  # 如果llm api用key鉴权，传入这里
+            "llm_api_base_url": conf().get("open_ai_api_base", "https://api.openai.com/v1"),  # 支持openai接口的llm服务地址前缀
+            "deployment_id": conf().get("azure_deployment_id", ""),  # azure openai会用到
             # note: 目前tool暂未对其他模型测试，但这里仍对配置来源做了优先级区分，一般插件配置可覆盖全局配置
-            "model_name": tool_model_name if tool_model_name else conf().get("model", "gpt-3.5-turbo"),
-            "no_default": kwargs.get("no_default", False),
-            "top_k_results": kwargs.get("top_k_results", 3),
-            # for news tool
-            "news_api_key": kwargs.get("news_api_key", ""),
+            "model_name": tool_model_name if tool_model_name else conf().get("model", const.GPT35),
+            # 工具配置相关
+            # for arxiv tool
+            "arxiv_simple": kwargs.get("arxiv_simple", True),  # 返回内容更精简
+            "arxiv_top_k_results": kwargs.get("arxiv_top_k_results", 2),  # 只返回前k个搜索结果
+            "arxiv_sort_by": kwargs.get("arxiv_sort_by", "relevance"),  # 搜索排序方式 ["relevance","lastUpdatedDate","submittedDate"]
+            "arxiv_sort_order": kwargs.get("arxiv_sort_order", "descending"),  # 搜索排序方式 ["ascending", "descending"]
+            "arxiv_output_type": kwargs.get("arxiv_output_type", "text"),  # 搜索结果类型 ["text", "pdf", "all"]
             # for bing-search tool
             "bing_subscription_key": kwargs.get("bing_subscription_key", ""),
+            "bing_search_url": kwargs.get("bing_search_url", "https://api.bing.microsoft.com/v7.0/search"),  # 必应搜索的endpoint地址，无需修改
+            "bing_search_top_k_results": kwargs.get("bing_search_top_k_results", 2),  # 只返回前k个搜索结果
+            "bing_search_simple": kwargs.get("bing_search_simple", True),  # 返回内容更精简
+            "bing_search_output_type": kwargs.get("bing_search_output_type", "text"),  # 搜索结果类型 ["text", "json"]
+            # for email tool
+            "email_nickname_mapping": kwargs.get("email_nickname_mapping", "{}"),  # 关于人的代号对应的邮箱地址，可以不输入邮箱地址发送邮件。键为代号值为邮箱地址
+            "email_smtp_host": kwargs.get("email_smtp_host", ""),  # 例如 'smtp.qq.com'
+            "email_smtp_port": kwargs.get("email_smtp_port", ""),  # 例如 587
+            "email_sender": kwargs.get("email_sender", ""),  # 发送者的邮件地址
+            "email_authorization_code": kwargs.get("email_authorization_code", ""),  # 发送者验证秘钥（可能不是登录密码）
             # for google-search tool
             "google_api_key": kwargs.get("google_api_key", ""),
             "google_cse_id": kwargs.get("google_cse_id", ""),
+            "google_simple": kwargs.get("google_simple", True),   # 返回内容更精简
+            "google_output_type": kwargs.get("google_output_type", "text"),  # 搜索结果类型 ["text", "json"]
+            # for finance-news tool
+            "finance_news_filter": kwargs.get("finance_news_filter", False),  # 是否开启过滤
+            "finance_news_filter_list": kwargs.get("finance_news_filter_list", []),  # 过滤词列表
+            "finance_news_simple": kwargs.get("finance_news_simple", True),   # 返回内容更精简
+            "finance_news_repeat_news": kwargs.get("finance_news_repeat_news", False),  # 是否过滤不返回。该tool每次返回约50条新闻，可能有重复新闻
+            # for morning-news tool
+            "morning_news_api_key": kwargs.get("morning_news_api_key", ""),   # api-key
+            "morning_news_simple": kwargs.get("morning_news_simple", True),   # 返回内容更精简
+            "morning_news_output_type": kwargs.get("morning_news_output_type", "text"),  # 搜索结果类型 ["text", "image"]
+            # for news-api tool
+            "news_api_key": kwargs.get("news_api_key", ""),
             # for searxng-search tool
-            "searx_search_host": kwargs.get("searx_search_host", ""),
+            "searxng_search_host": kwargs.get("searxng_search_host", ""),
+            "searxng_search_top_k_results": kwargs.get("searxng_search_top_k_results", 2),  # 只返回前k个搜索结果
+            "searxng_search_output_type": kwargs.get("searxng_search_output_type", "text"),  # 搜索结果类型 ["text", "json"]
+            # for sms tool
+            "sms_nickname_mapping": kwargs.get("sms_nickname_mapping", "{}"),  # 关于人的代号对应的手机号，可以不输入手机号发送sms。键为代号值为手机号
+            "sms_username": kwargs.get("sms_username", ""),  # smsbao用户名
+            "sms_apikey": kwargs.get("sms_apikey", ""),  # smsbao
+            # for stt tool
+            "stt_api_key": kwargs.get("stt_api_key", ""),  # azure
+            "stt_api_region": kwargs.get("stt_api_region", ""),  # azure
+            "stt_recognition_language": kwargs.get("stt_recognition_language", "zh-CN"),  # 识别的语言类型 部分：en-US ja-JP ko-KR yue-CN zh-CN
+            # for tts tool
+            "tts_api_key": kwargs.get("tts_api_key", ""),  # azure
+            "tts_api_region": kwargs.get("tts_api_region", ""),  # azure
+            "tts_auto_detect": kwargs.get("tts_auto_detect", True),  # 是否自动检测语音的语言
+            "tts_speech_id": kwargs.get("tts_speech_id", "zh-CN-XiaozhenNeural"),  # 输出语音ID
+            # for summary tool
+            "summary_max_segment_length": kwargs.get("summary_max_segment_length", 2500),  # 每2500tokens分段，多段触发总结tool
+            # for terminal tool
+            "terminal_nsfc_filter": kwargs.get("terminal_nsfc_filter", True),  # 是否过滤llm输出的危险命令
+            "terminal_return_err_output": kwargs.get("terminal_return_err_output", True),  # 是否输出错误信息
+            "terminal_timeout": kwargs.get("terminal_timeout", 20),  # 允许命令最长执行时间
+            # for visual tool
+            "caption_api_key": kwargs.get("caption_api_key", ""),  # ali dashscope apikey
+            # for browser tool
+            "browser_use_summary": kwargs.get("browser_use_summary", True),  # 是否对返回结果使用tool功能
+            # for url-get tool
+            "url_get_use_summary": kwargs.get("url_get_use_summary", True),  # 是否对返回结果使用tool功能
+            # for wechat tool
+            "wechat_hot_reload": kwargs.get("wechat_hot_reload", True),  # 是否使用热重载的方式发送wechat
+            "wechat_cpt_path": kwargs.get("wechat_cpt_path", os.path.join(get_appdata_dir(), "itchat.pkl")),  # wechat 配置文件（`itchat.pkl`）
+            "wechat_send_group": kwargs.get("wechat_send_group", False),  # 是否向群组发送消息
+            "wechat_nickname_mapping": kwargs.get("wechat_nickname_mapping", "{}"),  # 关于人的代号映射关系。键为代号值为微信名（昵称、备注名均可）
+            # for wikipedia tool
+            "wikipedia_top_k_results": kwargs.get("wikipedia_top_k_results", 2),  # 只返回前k个搜索结果
             # for wolfram-alpha tool
             "wolfram_alpha_appid": kwargs.get("wolfram_alpha_appid", ""),
-            # for morning-news tool
-            "morning_news_api_key": kwargs.get("morning_news_api_key", ""),
-            # for visual_dl tool
-            "cuda_device": kwargs.get("cuda_device", "cpu"),
-            "think_depth": kwargs.get("think_depth", 3),
-            "arxiv_summary": kwargs.get("arxiv_summary", True),
-            "morning_news_use_llm": kwargs.get("morning_news_use_llm", False),
         }
 
     def _filter_tool_list(self, tool_list: list):
         valid_list = []
         for tool in tool_list:
-            if tool in get_all_tool_names():
+            if tool in main_tool_register.get_registered_tool_names():
                 valid_list.append(tool)
             else:
                 logger.warning("[tool] filter invalid tool: " + repr(tool))
         return valid_list
 
     def _reset_app(self) -> App:
-        tool_config = self._read_json()
-        app_kwargs = self._build_tool_kwargs(tool_config.get("kwargs", {}))
+        self.tool_config = self._read_json()
+        self.app_kwargs = self._build_tool_kwargs(self.tool_config.get("kwargs", {}))
 
         app = AppFactory()
-        app.init_env(**app_kwargs)
-
+        app.init_env(**self.app_kwargs)
         # filter not support tool
-        tool_list = self._filter_tool_list(tool_config.get("tools", []))
+        tool_list = self._filter_tool_list(self.tool_config.get("tools", []))
 
-        return app.create_app(tools_list=tool_list, **app_kwargs)
+        return app.create_app(tools_list=tool_list, **self.app_kwargs)

requirements-optional.txt

@@ -7,6 +7,7 @@ gTTS>=2.3.1 # google text to speech
 pyttsx3>=2.90 # pytsx text to speech
 baidu_aip>=4.16.10 # baidu voice
 azure-cognitiveservices-speech # azure voice
+edge-tts # edge-tts
 numpy<=1.24.2
 langid # language detect
 
@@ -18,13 +19,15 @@ web.py
 wechatpy
 
 # chatgpt-tool-hub plugin
-chatgpt_tool_hub==0.4.6
+chatgpt_tool_hub==0.5.0
 
 # xunfei spark
 websocket-client==1.2.0
 
 # claude bot
 curl_cffi
+# claude API
+anthropic
 
 # tongyi qwen
 broadscope_bailian
@@ -33,7 +36,10 @@ broadscope_bailian
 google-generativeai
 
 # linkai
-linkai
+linkai>=0.0.3.5
 
 # dingtalk
 dingtalk_stream
+
+# zhipuai
+zhipuai>=2.0.1

requirements.txt

@@ -6,4 +6,4 @@ requests>=2.28.2
 chardet>=5.1.0
 Pillow
 pre-commit
-web.py
+web.py

voice/audio_convert.py

@@ -64,7 +64,9 @@ def any_to_wav(any_path, wav_path):
     if any_path.endswith(".sil") or any_path.endswith(".silk") or any_path.endswith(".slk"):
         return sil_to_wav(any_path, wav_path)
     audio = AudioSegment.from_file(any_path)
-    audio.export(wav_path, format="wav")
+    audio.set_frame_rate(8000)    # 百度语音转写支持8000采样率, pcm_s16le, 单通道语音识别
+    audio.set_channels(1)
+    audio.export(wav_path, format="wav", codec='pcm_s16le')
 
 
 def any_to_sil(any_path, sil_path):

voice/baidu/baidu_voice.py

@@ -62,7 +62,7 @@ class BaiduVoice(Voice):
         # 识别本地文件
         logger.debug("[Baidu] voice file name={}".format(voice_file))
         pcm = get_pcm_from_wav(voice_file)
-        res = self.client.asr(pcm, "pcm", 16000, {"dev_pid": self.dev_id})
+        res = self.client.asr(pcm, "pcm", 8000, {"dev_pid": self.dev_id})
         if res["err_no"] == 0:
             logger.info("百度语音识别到了：{}".format(res["result"]))
             text = "".join(res["result"])

voice/edge/edge_voice.py

@@ -0,0 +1,50 @@
+import time
+
+import edge_tts
+import asyncio
+
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from common.tmp_dir import TmpDir
+from voice.voice import Voice
+
+
+class EdgeVoice(Voice):
+
+    def __init__(self):
+        '''
+        # 普通话
+        zh-CN-XiaoxiaoNeural
+        zh-CN-XiaoyiNeural
+        zh-CN-YunjianNeural
+        zh-CN-YunxiNeural
+        zh-CN-YunxiaNeural
+        zh-CN-YunyangNeural
+        # 地方口音
+        zh-CN-liaoning-XiaobeiNeural
+        zh-CN-shaanxi-XiaoniNeural
+        # 粤语
+        zh-HK-HiuGaaiNeural
+        zh-HK-HiuMaanNeural
+        zh-HK-WanLungNeural
+        # 湾湾腔
+        zh-TW-HsiaoChenNeural
+        zh-TW-HsiaoYuNeural
+        zh-TW-YunJheNeural
+        '''
+        self.voice = "zh-CN-YunjianNeural"
+
+    def voiceToText(self, voice_file):
+        pass
+
+    async def gen_voice(self, text, fileName):
+        communicate = edge_tts.Communicate(text, self.voice)
+        await communicate.save(fileName)
+
+    def textToVoice(self, text):
+        fileName = TmpDir().path() + "reply-" + str(int(time.time())) + "-" + str(hash(text) & 0x7FFFFFFF) + ".mp3"
+
+        asyncio.run(self.gen_voice(text, fileName))
+
+        logger.info("[EdgeTTS] textToVoice text={} voice file name={}".format(text, fileName))
+        return Reply(ReplyType.VOICE, fileName)

voice/factory.py

@@ -42,4 +42,8 @@ def create_voice(voice_type):
         from voice.ali.ali_voice import AliVoice
 
         return AliVoice()
+    elif voice_type == "edge":
+        from voice.edge.edge_voice import EdgeVoice
+
+        return EdgeVoice()
     raise RuntimeError