docs: update readme

Ensure compatibility for /wxcomapp URL with trailing slash

README.md

@@ -1,19 +1,26 @@
 # 简介
 
-> chatgpt-on-wechat（简称CoW）项目是基于大模型的智能对话机器人，支持微信公众号、企业微信应用、微信、飞书、钉钉接入，可选择GPT3.5/GPT4.0/Claude/Gemini/LinkAI/ChatGLM/KIMI/文心一言/讯飞星火/通义千问/LinkAI，能处理文本、语音和图片，通过插件访问操作系统和互联网等外部资源，支持基于自有知识库定制企业AI应用。
+> chatgpt-on-wechat（简称CoW）项目是基于大模型的智能对话机器人，支持微信公众号、企业微信应用、飞书、钉钉接入，可选择GPT3.5/GPT4.0/Claude/Gemini/LinkAI/ChatGLM/KIMI/文心一言/讯飞星火/通义千问/LinkAI，能处理文本、语音和图片，通过插件访问操作系统和互联网等外部资源，支持基于自有知识库定制企业AI应用。
 
 最新版本支持的功能如下：
 
--  ✅   **多端部署：** 有多种部署方式可选择且功能完备，目前已支持微信生态下公众号、企业微信应用、飞书、钉钉等部署方式
--  ✅   **基础对话：** 私聊及群聊的消息智能回复，支持多轮会话上下文记忆，支持 GPT-3.5, GPT-4, GPT-4o, Claude-3, Gemini, 文心一言, 讯飞星火, 通义千问，ChatGLM-4，Kimi(月之暗面)
+-  ✅   **多端部署：** 有多种部署方式可选择且功能完备，目前已支持微信公众号、企业微信应用、飞书、钉钉等部署方式
+-  ✅   **基础对话：** 私聊及群聊的消息智能回复，支持多轮会话上下文记忆，支持 GPT-3.5, GPT-4o-mini, GPT-4o,  GPT-4, Claude-3.5, Gemini, 文心一言, 讯飞星火, 通义千问，ChatGLM-4，Kimi(月之暗面), MiniMax
 -  ✅   **语音能力：** 可识别语音消息，通过文字或语音回复，支持 azure, baidu, google, openai(whisper/tts) 等多种语音模型
 -  ✅   **图像能力：** 支持图片生成、图片识别、图生图（如照片修复），可选择 Dall-E-3, stable diffusion, replicate, midjourney, CogView-3, vision模型
 -  ✅   **丰富插件：** 支持个性化插件扩展，已实现多角色切换、文字冒险、敏感词过滤、聊天记录总结、文档总结和对话、联网搜索等插件
 -  ✅   **知识库：** 通过上传知识库文件自定义专属机器人，可作为数字分身、智能客服、私域助手使用，基于 [LinkAI](https://link-ai.tech) 实现
 
+## 声明
+
+1. 本项目遵循 [MIT开源协议](/LICENSE)，仅用于技术研究和学习，使用本项目时需遵守所在地法律法规、相关政策以及企业章程，禁止用于任何违法或侵犯他人权益的行为
+2. 境内使用该项目时，请使用国内厂商的大模型服务，并进行必要的内容安全审核及过滤
+3. 本项目主要接入协同办公平台，推荐使用公众号、企微自建应用、钉钉、飞书等接入通道，其他通道为历史产物已不维护
+4. 任何个人、团队和企业，无论以何种方式使用该项目、对何对象提供服务，所产生的一切后果，本项目均不承担任何责任
+
 ## 演示
 
-https://github.com/zhayujie/chatgpt-on-wechat/assets/26161723/d5154020-36e3-41db-8706-40ce9f3f1b1e
+DEMO视频：https://cdn.link-ai.tech/doc/cow_demo.mp4
 
 ## 社区
 
@@ -25,7 +32,7 @@ https://github.com/zhayujie/chatgpt-on-wechat/assets/26161723/d5154020-36e3-41db
 
 # 企业服务
 
-<a href="https://link-ai.tech" target="_blank"><img width="800" src="https://img-1317903499.cos.ap-guangzhou.myqcloud.com/docs/github-linkai-intro-v2.png"></a>
+<a href="https://link-ai.tech" target="_blank"><img width="800" src="https://cdn.link-ai.tech/image/link-ai-intro.jpg"></a>
 
 > [LinkAI](https://link-ai.tech/) 是面向企业和开发者的一站式AI应用平台，聚合多模态大模型、知识库、Agent 插件、工作流等能力，支持一键接入主流平台并进行管理，支持SaaS、私有化部署多种模式。
 >
@@ -39,7 +46,11 @@ https://github.com/zhayujie/chatgpt-on-wechat/assets/26161723/d5154020-36e3-41db
 
 # 🏷 更新日志
 
->**2024.05.14：** [1.6.5版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.6.5)，新增 gpt-4o 模型支持
+>**2024.07.19：** [1.6.9版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.6.9) 新增 gpt-4o-mini 模型、阿里语音识别、企微应用渠道路由优化
+
+>**2024.07.05：** [1.6.8版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.6.8) 和 [1.6.7版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.6.7)，Claude3.5, Gemini 1.5 Pro, MiniMax模型、工作流图片输入、模型列表完善
+
+>**2024.06.04：** [1.6.6版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.6.6) 和 [1.6.5版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.6.5)，gpt-4o模型、钉钉流式卡片、讯飞语音识别/合成
 
 >**2024.04.26：** [1.6.0版本](https://github.com/zhayujie/chatgpt-on-wechat/releases/tag/1.6.0)，新增 Kimi 接入、gpt-4-turbo版本升级、文件总结和语音识别问题修复
 
@@ -77,7 +88,7 @@ https://github.com/zhayujie/chatgpt-on-wechat/assets/26161723/d5154020-36e3-41db
 
 > 默认对话模型是 openai 的 gpt-3.5-turbo，计费方式是约每 1000tokens (约750个英文单词 或 500汉字，包含请求和回复) 消耗 $0.002，图片生成是Dell E模型，每张消耗 $0.016。
 
-项目同时也支持使用 LinkAI 接口，无需代理，可使用 文心、讯飞、GPT-3、GPT-4 等模型，支持 定制化知识库、联网搜索、MJ绘图、文档总结和对话等能力。修改配置即可一键切换，参考 [接入文档](https://link-ai.tech/platform/link-app/wechat)。
+项目同时也支持使用 LinkAI 接口，无需代理，可使用 Kimi、文心、讯飞、GPT-3.5、GPT-4o 等模型，支持 定制化知识库、联网搜索、MJ绘图、文档总结、工作流等能力。修改配置即可一键使用，参考 [接入文档](https://link-ai.tech/platform/link-app/wechat)。
 
 ### 2.运行环境
 
@@ -139,7 +150,7 @@ pip3 install -r requirements-optional.txt
   "subscribe_msg": "感谢您的关注！\n这里是ChatGPT，可以自由对话。\n支持语音对话。\n支持图片输出，画字开头的消息将按要求创作图片。\n支持角色扮演和文字冒险等丰富插件。\n输入{trigger_prefix}#help 查看详细指令。",
   "use_linkai": false,                                        # 是否使用LinkAI接口，默认关闭，开启后可国内访问，使用知识库和MJ
   "linkai_api_key": "",                                       # LinkAI Api Key
-  "linkai_app_code": ""                                       # LinkAI 应用code
+  "linkai_app_code": ""                                       # LinkAI 应用或工作流code
 }
 ```
 **配置说明：**
@@ -160,11 +171,11 @@ pip3 install -r requirements-optional.txt
 
 + 添加 `"speech_recognition": true` 将开启语音识别，默认使用openai的whisper模型识别为文字，同时以文字回复，该参数仅支持私聊 (注意由于语音消息无法匹配前缀，一旦开启将对所有语音自动回复，支持语音触发画图)；
 + 添加 `"group_speech_recognition": true` 将开启群组语音识别，默认使用openai的whisper模型识别为文字，同时以文字回复，参数仅支持群聊 (会匹配group_chat_prefix和group_chat_keyword, 支持语音触发画图)；
-+ 添加 `"voice_reply_voice": true` 将开启语音回复语音（同时作用于私聊和群聊），但是需要配置对应语音合成平台的key，由于itchat协议的限制，只能发送语音mp3文件，若使用wechaty则回复的是微信语音。
++ 添加 `"voice_reply_voice": true` 将开启语音回复语音（同时作用于私聊和群聊）
 
 **4.其他配置**
 
-+ `model`: 模型名称，目前支持 `gpt-3.5-turbo`, `gpt-4o`, `gpt-4-turbo`, `gpt-4`, `wenxin` , `claude` , `gemini`, `glm-4`,  `xunfei`, `moonshot`
++ `model`: 模型名称，目前支持 `gpt-3.5-turbo`, `gpt-4o-mini`, `gpt-4o`, `gpt-4`, `wenxin` , `claude` , `gemini`, `glm-4`,  `xunfei`, `moonshot`等，全部模型名称参考[common/const.py](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/common/const.py)文件
 + `temperature`,`frequency_penalty`,`presence_penalty`: Chat API接口参数，详情参考[OpenAI官方文档。](https://platform.openai.com/docs/api-reference/chat)
 + `proxy`：由于目前 `openai` 接口国内无法访问，需配置代理客户端的地址，详情参考  [#351](https://github.com/zhayujie/chatgpt-on-wechat/issues/351)
 + 对于图像生成，在满足个人或群组触发条件外，还需要额外的关键词前缀来触发，对应配置 `image_create_prefix `
@@ -172,7 +183,7 @@ pip3 install -r requirements-optional.txt
 + `conversation_max_tokens`：表示能够记忆的上下文最大字数（一问一答为一组对话，如果累积的对话字数超出限制，就会优先移除最早的一组对话）
 + `rate_limit_chatgpt`，`rate_limit_dalle`：每分钟最高问答速率、画图速率，超速后排队按序处理。
 + `clear_memory_commands`: 对话内指令，主动清空前文记忆，字符串数组可自定义指令别名。
-+ `hot_reload`: 程序退出后，暂存微信扫码状态，默认关闭。
++ `hot_reload`: 程序退出后，暂存等于状态，默认关闭。
 + `character_desc` 配置中保存着你对机器人说的一段话，他会记住这段话并作为他的设定，你可以为他定制任何人格      (关于会话上下文的更多内容参考该 [issue](https://github.com/zhayujie/chatgpt-on-wechat/issues/43))
 + `subscribe_msg`：订阅消息，公众号和企业微信channel中请填写，当被订阅时会自动回复， 可使用特殊占位符。目前支持的占位符有{trigger_prefix}，在程序中它会自动替换成bot的触发词。
 
@@ -180,7 +191,7 @@ pip3 install -r requirements-optional.txt
 
 + `use_linkai`: 是否使用LinkAI接口，开启后可国内访问，使用知识库和 `Midjourney` 绘画, 参考 [文档](https://link-ai.tech/platform/link-app/wechat)
 + `linkai_api_key`: LinkAI Api Key，可在 [控制台](https://link-ai.tech/console/interface) 创建
-+ `linkai_app_code`: LinkAI 应用code，选填
++ `linkai_app_code`: LinkAI 应用或工作流的code，选填
 
 **本说明文档可能会未及时更新，当前所有可选的配置项均在该[`config.py`](https://github.com/zhayujie/chatgpt-on-wechat/blob/master/config.py)中列出。**
 
@@ -194,7 +205,7 @@ pip3 install -r requirements-optional.txt
 python3 app.py                                    # windows环境下该命令通常为 python app.py
 ```
 
-终端输出二维码后，使用微信进行扫码，当输出 "Start auto replying" 时表示自动回复程序已经成功运行了（注意：用于登录的微信需要在支付处已完成实名认证）。扫码登录后你的账号就成为机器人了，可以在手机端通过配置的关键词触发自动回复 (任意好友发送消息给你，或是自己发消息给好友)，参考[#142](https://github.com/zhayujie/chatgpt-on-wechat/issues/142)。
+终端输出二维码后，进行扫码登录，当输出 "Start auto replying" 时表示自动回复程序已经成功运行了（注意：用于登录的账号需要在支付处已完成实名认证）。扫码登录后你的账号就成为机器人了，可以在手机端通过配置的关键词触发自动回复 (任意好友发送消息给你，或是自己发消息给好友)，参考[#142](https://github.com/zhayujie/chatgpt-on-wechat/issues/142)。
 
 ### 2.服务器部署
 

bot/baidu/baidu_wenxin.py

@@ -1,6 +1,8 @@
 # encoding:utf-8
 
-import requests, json
+import requests
+import json
+from common import const
 from bot.bot import Bot
 from bot.session_manager import SessionManager
 from bridge.context import ContextType
@@ -16,9 +18,15 @@ class BaiduWenxinBot(Bot):
 
     def __init__(self):
         super().__init__()
-        wenxin_model = conf().get("baidu_wenxin_model") or "eb-instant"
-        if conf().get("model") and conf().get("model") == "wenxin-4":
-            wenxin_model = "completions_pro"
+        wenxin_model = conf().get("baidu_wenxin_model")
+        if wenxin_model is not None:
+            wenxin_model = conf().get("baidu_wenxin_model") or "eb-instant"
+        else:
+            if conf().get("model") and conf().get("model") == const.WEN_XIN:
+                wenxin_model = "completions"
+            elif conf().get("model") and conf().get("model") == const.WEN_XIN_4:
+                wenxin_model = "completions_pro"
+
         self.sessions = SessionManager(BaiduWenxinSession, model=wenxin_model)
 
     def reply(self, query, context=None):
@@ -94,7 +102,7 @@ class BaiduWenxinBot(Bot):
             logger.warn("[BAIDU] Exception: {}".format(e))
             need_retry = False
             self.sessions.clear_session(session.session_id)
-            result = {"completion_tokens": 0, "content": "出错了: {}".format(e)}
+            result = {"total_tokens": 0, "completion_tokens": 0, "content": "出错了: {}".format(e)}
             return result
 
     def get_access_token(self):

bot/bot_factory.py

@@ -2,7 +2,6 @@
 channel factory
 """
 from common import const
-from common.log import logger
 
 
 def create_bot(bot_type):
@@ -64,6 +63,10 @@ def create_bot(bot_type):
     elif bot_type == const.MOONSHOT:
         from bot.moonshot.moonshot_bot import MoonshotBot
         return MoonshotBot()
+    
+    elif bot_type == const.MiniMax:
+        from bot.minimax.minimax_bot import MinimaxBot
+        return MinimaxBot()
 
 
     raise RuntimeError

bot/chatgpt/chat_gpt_bot.py

@@ -171,24 +171,48 @@ class AzureChatGPTBot(ChatGPTBot):
         self.args["deployment_id"] = conf().get("azure_deployment_id")
 
     def create_img(self, query, retry_count=0, api_key=None):
-        api_version = "2022-08-03-preview"
-        url = "{}dalle/text-to-image?api-version={}".format(openai.api_base, api_version)
-        api_key = api_key or openai.api_key
-        headers = {"api-key": api_key, "Content-Type": "application/json"}
-        try:
-            body = {"caption": query, "resolution": conf().get("image_create_size", "256x256")}
-            submission = requests.post(url, headers=headers, json=body)
-            operation_location = submission.headers["Operation-Location"]
-            retry_after = submission.headers["Retry-after"]
-            status = ""
-            image_url = ""
-            while status != "Succeeded":
-                logger.info("waiting for image create..., " + status + ",retry after " + retry_after + " seconds")
-                time.sleep(int(retry_after))
-                response = requests.get(operation_location, headers=headers)
-                status = response.json()["status"]
-            image_url = response.json()["result"]["contentUrl"]
-            return True, image_url
-        except Exception as e:
-            logger.error("create image error: {}".format(e))
-            return False, "图片生成失败"
+        text_to_image_model = conf().get("text_to_image")
+        if text_to_image_model == "dall-e-2":
+            api_version = "2023-06-01-preview"
+            endpoint = conf().get("azure_openai_dalle_api_base","open_ai_api_base")
+            # 检查endpoint是否以/结尾
+            if not endpoint.endswith("/"):
+                endpoint = endpoint + "/"
+            url = "{}openai/images/generations:submit?api-version={}".format(endpoint, api_version)
+            api_key = conf().get("azure_openai_dalle_api_key","open_ai_api_key")
+            headers = {"api-key": api_key, "Content-Type": "application/json"}
+            try:
+                body = {"prompt": query, "size": conf().get("image_create_size", "256x256"),"n": 1}
+                submission = requests.post(url, headers=headers, json=body)
+                operation_location = submission.headers['operation-location']
+                status = ""
+                while (status != "succeeded"):
+                    if retry_count > 3:
+                        return False, "图片生成失败"
+                    response = requests.get(operation_location, headers=headers)
+                    status = response.json()['status']
+                    retry_count += 1
+                image_url = response.json()['result']['data'][0]['url']
+                return True, image_url
+            except Exception as e:
+                logger.error("create image error: {}".format(e))
+                return False, "图片生成失败"
+        elif text_to_image_model == "dall-e-3":
+            api_version = conf().get("azure_api_version", "2024-02-15-preview")
+            endpoint = conf().get("azure_openai_dalle_api_base","open_ai_api_base")
+            # 检查endpoint是否以/结尾
+            if not endpoint.endswith("/"):
+                endpoint = endpoint + "/"
+            url = "{}openai/deployments/{}/images/generations?api-version={}".format(endpoint, conf().get("azure_openai_dalle_deployment_id","text_to_image"),api_version)
+            api_key = conf().get("azure_openai_dalle_api_key","open_ai_api_key")
+            headers = {"api-key": api_key, "Content-Type": "application/json"}
+            try:
+                body = {"prompt": query, "size": conf().get("image_create_size", "1024x1024"), "quality": conf().get("dalle3_image_quality", "standard")}
+                submission = requests.post(url, headers=headers, json=body)
+                image_url = submission.json()['data'][0]['url']
+                return True, image_url
+            except Exception as e:
+                logger.error("create image error: {}".format(e))
+                return False, "图片生成失败"
+        else:
+            return False, "图片生成失败，未配置text_to_image参数"

bot/chatgpt/chat_gpt_session.py

@@ -67,7 +67,7 @@ def num_tokens_from_messages(messages, model):
     elif model in ["gpt-4-0314", "gpt-4-0613", "gpt-4-32k", "gpt-4-32k-0613", "gpt-3.5-turbo-0613",
                    "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613", "gpt-35-turbo-16k", "gpt-4-turbo-preview",
                    "gpt-4-1106-preview", const.GPT4_TURBO_PREVIEW, const.GPT4_VISION_PREVIEW, const.GPT4_TURBO_01_25,
-                   const.GPT_4o, const.LINKAI_4o, const.LINKAI_4_TURBO]:
+                   const.GPT_4o, const.GPT_4o_MINI, const.LINKAI_4o, const.LINKAI_4_TURBO]:
         return num_tokens_from_messages(messages, model="gpt-4")
     elif model.startswith("claude-3"):
         return num_tokens_from_messages(messages, model="gpt-3.5-turbo")

bot/claudeapi/claude_api_bot.py

@@ -130,4 +130,6 @@ class ClaudeAPIBot(Bot, OpenAIImage):
             return "claude-3-sonnet-20240229"
         elif model == "claude-3-haiku":
             return "claude-3-haiku-20240307"
+        elif model == "claude-3.5-sonnet":
+            return "claude-3-5-sonnet-20240620"
         return model

bot/gemini/google_gemini_bot.py

@@ -24,7 +24,9 @@ class GoogleGeminiBot(Bot):
         self.api_key = conf().get("gemini_api_key")
         # 复用文心的token计算方式
         self.sessions = SessionManager(BaiduWenxinSession, model=conf().get("model") or "gpt-3.5-turbo")
-
+        self.model = conf().get("model") or "gemini-pro"
+        if self.model == "gemini":
+            self.model = "gemini-pro"
     def reply(self, query, context: Context = None) -> Reply:
         try:
             if context.type != ContextType.TEXT:
@@ -35,7 +37,7 @@ class GoogleGeminiBot(Bot):
             session = self.sessions.session_query(query, session_id)
             gemini_messages = self._convert_to_gemini_messages(self.filter_messages(session.messages))
             genai.configure(api_key=self.api_key)
-            model = genai.GenerativeModel('gemini-pro')
+            model = genai.GenerativeModel(self.model)
             response = model.generate_content(gemini_messages)
             reply_text = response.text
             self.sessions.session_reply(reply_text, session_id)

bot/linkai/link_ai_bot.py

@@ -399,6 +399,7 @@ class LinkAIBot(Bot):
             return
         max_send_num = conf().get("max_media_send_count")
         send_interval = conf().get("media_send_interval")
+        file_type = (".pdf", ".doc", ".docx", ".csv", ".xls", ".xlsx", ".txt", ".rtf", ".ppt", ".pptx")
         try:
             i = 0
             for url in image_urls:
@@ -407,7 +408,7 @@ class LinkAIBot(Bot):
                 i += 1
                 if url.endswith(".mp4"):
                     reply_type = ReplyType.VIDEO_URL
-                elif url.endswith(".pdf") or url.endswith(".doc") or url.endswith(".docx") or url.endswith(".csv"):
+                elif url.endswith(file_type):
                     reply_type = ReplyType.FILE
                     url = _download_file(url)
                     if not url:

bot/minimax/minimax_bot.py

@@ -0,0 +1,151 @@
+# encoding:utf-8
+
+import time
+
+import openai
+import openai.error
+from bot.bot import Bot
+from bot.minimax.minimax_session import MinimaxSession
+from bot.session_manager import SessionManager
+from bridge.context import Context, ContextType
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from config import conf, load_config
+from bot.chatgpt.chat_gpt_session import ChatGPTSession
+import requests
+from common import const
+
+
+# ZhipuAI对话模型API
+class MinimaxBot(Bot):
+    def __init__(self):
+        super().__init__()
+        self.args = {
+            "model": conf().get("model") or "abab6.5",  # 对话模型的名称
+            "temperature": conf().get("temperature", 0.3),  # 如果设置，值域须为 [0, 1] 我们推荐 0.3，以达到较合适的效果。
+            "top_p": conf().get("top_p", 0.95),  # 使用默认值
+        }
+        self.api_key = conf().get("Minimax_api_key")
+        self.group_id = conf().get("Minimax_group_id")
+        self.base_url = conf().get("Minimax_base_url", f"https://api.minimax.chat/v1/text/chatcompletion_pro?GroupId={self.group_id}")
+        # tokens_to_generate/bot_setting/reply_constraints可自行修改
+        self.request_body = {
+            "model": self.args["model"],
+            "tokens_to_generate": 2048,
+            "reply_constraints": {"sender_type": "BOT", "sender_name": "MM智能助理"},
+            "messages": [],
+            "bot_setting": [
+                {
+                    "bot_name": "MM智能助理",
+                    "content": "MM智能助理是一款由MiniMax自研的，没有调用其他产品的接口的大型语言模型。MiniMax是一家中国科技公司，一直致力于进行大模型相关的研究。",
+                }
+            ],
+        }
+        self.sessions = SessionManager(MinimaxSession, model=const.MiniMax)
+
+    def reply(self, query, context: Context = None) -> Reply:
+        # acquire reply content
+        logger.info("[Minimax_AI] query={}".format(query))
+        if context.type == ContextType.TEXT:
+            session_id = context["session_id"]
+            reply = None
+            clear_memory_commands = conf().get("clear_memory_commands", ["#清除记忆"])
+            if query in clear_memory_commands:
+                self.sessions.clear_session(session_id)
+                reply = Reply(ReplyType.INFO, "记忆已清除")
+            elif query == "#清除所有":
+                self.sessions.clear_all_session()
+                reply = Reply(ReplyType.INFO, "所有人记忆已清除")
+            elif query == "#更新配置":
+                load_config()
+                reply = Reply(ReplyType.INFO, "配置已更新")
+            if reply:
+                return reply
+            session = self.sessions.session_query(query, session_id)
+            logger.debug("[Minimax_AI] session query={}".format(session))
+
+            model = context.get("Minimax_model")
+            new_args = self.args.copy()
+            if model:
+                new_args["model"] = model
+            # if context.get('stream'):
+            #     # reply in stream
+            #     return self.reply_text_stream(query, new_query, session_id)
+
+            reply_content = self.reply_text(session, args=new_args)
+            logger.debug(
+                "[Minimax_AI] new_query={}, session_id={}, reply_cont={}, completion_tokens={}".format(
+                    session.messages,
+                    session_id,
+                    reply_content["content"],
+                    reply_content["completion_tokens"],
+                )
+            )
+            if reply_content["completion_tokens"] == 0 and len(reply_content["content"]) > 0:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+            elif reply_content["completion_tokens"] > 0:
+                self.sessions.session_reply(reply_content["content"], session_id, reply_content["total_tokens"])
+                reply = Reply(ReplyType.TEXT, reply_content["content"])
+            else:
+                reply = Reply(ReplyType.ERROR, reply_content["content"])
+                logger.debug("[Minimax_AI] reply {} used 0 tokens.".format(reply_content))
+            return reply
+        else:
+            reply = Reply(ReplyType.ERROR, "Bot不支持处理{}类型的消息".format(context.type))
+            return reply
+
+    def reply_text(self, session: MinimaxSession, args=None, retry_count=0) -> dict:
+        """
+        call openai's ChatCompletion to get the answer
+        :param session: a conversation session
+        :param session_id: session id
+        :param retry_count: retry count
+        :return: {}
+        """
+        try:
+            headers = {"Content-Type": "application/json", "Authorization": "Bearer " + self.api_key}
+            self.request_body["messages"].extend(session.messages)
+            logger.info("[Minimax_AI] request_body={}".format(self.request_body))
+            # logger.info("[Minimax_AI] reply={}, total_tokens={}".format(response.choices[0]['message']['content'], response["usage"]["total_tokens"]))
+            res = requests.post(self.base_url, headers=headers, json=self.request_body)
+
+            # self.request_body["messages"].extend(response.json()["choices"][0]["messages"])
+            if res.status_code == 200:
+                response = res.json()
+                return {
+                    "total_tokens": response["usage"]["total_tokens"],
+                    "completion_tokens": response["usage"]["total_tokens"],
+                    "content": response["reply"],
+                }
+            else:
+                response = res.json()
+                error = response.get("error")
+                logger.error(f"[Minimax_AI] chat failed, status_code={res.status_code}, " f"msg={error.get('message')}, type={error.get('type')}")
+
+                result = {"completion_tokens": 0, "content": "提问太快啦，请休息一下再问我吧"}
+                need_retry = False
+                if res.status_code >= 500:
+                    # server error, need retry
+                    logger.warn(f"[Minimax_AI] do retry, times={retry_count}")
+                    need_retry = retry_count < 2
+                elif res.status_code == 401:
+                    result["content"] = "授权失败，请检查API Key是否正确"
+                elif res.status_code == 429:
+                    result["content"] = "请求过于频繁，请稍后再试"
+                    need_retry = retry_count < 2
+                else:
+                    need_retry = False
+
+                if need_retry:
+                    time.sleep(3)
+                    return self.reply_text(session, args, retry_count + 1)
+                else:
+                    return result
+        except Exception as e:
+            logger.exception(e)
+            need_retry = retry_count < 2
+            result = {"completion_tokens": 0, "content": "我现在有点累了，等会再来吧"}
+            if need_retry:
+                return self.reply_text(session, args, retry_count + 1)
+            else:
+                return result

bot/minimax/minimax_session.py

@@ -0,0 +1,72 @@
+from bot.session_manager import Session
+from common.log import logger
+
+"""
+    e.g.
+    [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Who won the world series in 2020?"},
+        {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
+        {"role": "user", "content": "Where was it played?"}
+    ]
+"""
+
+
+class MinimaxSession(Session):
+    def __init__(self, session_id, system_prompt=None, model="minimax"):
+        super().__init__(session_id, system_prompt)
+        self.model = model
+        # self.reset()
+
+    def add_query(self, query):
+        user_item = {"sender_type": "USER", "sender_name": self.session_id, "text": query}
+        self.messages.append(user_item)
+
+    def add_reply(self, reply):
+        assistant_item = {"sender_type": "BOT", "sender_name": "MM智能助理", "text": reply}
+        self.messages.append(assistant_item)
+
+    def discard_exceeding(self, max_tokens, cur_tokens=None):
+        precise = True
+        try:
+            cur_tokens = self.calc_tokens()
+        except Exception as e:
+            precise = False
+            if cur_tokens is None:
+                raise e
+            logger.debug("Exception when counting tokens precisely for query: {}".format(e))
+        while cur_tokens > max_tokens:
+            if len(self.messages) > 2:
+                self.messages.pop(1)
+            elif len(self.messages) == 2 and self.messages[1]["sender_type"] == "BOT":
+                self.messages.pop(1)
+                if precise:
+                    cur_tokens = self.calc_tokens()
+                else:
+                    cur_tokens = cur_tokens - max_tokens
+                break
+            elif len(self.messages) == 2 and self.messages[1]["sender_type"] == "USER":
+                logger.warn("user message exceed max_tokens. total_tokens={}".format(cur_tokens))
+                break
+            else:
+                logger.debug("max_tokens={}, total_tokens={}, len(messages)={}".format(max_tokens, cur_tokens, len(self.messages)))
+                break
+            if precise:
+                cur_tokens = self.calc_tokens()
+            else:
+                cur_tokens = cur_tokens - max_tokens
+        return cur_tokens
+
+    def calc_tokens(self):
+        return num_tokens_from_messages(self.messages, self.model)
+
+
+def num_tokens_from_messages(messages, model):
+    """Returns the number of tokens used by a list of messages."""
+    # 官方token计算规则："对于中文文本来说，1个token通常对应一个汉字；对于英文文本来说，1个token通常对应3至4个字母或1个单词"
+    # 详情请产看文档：https://help.aliyun.com/document_detail/2586397.html
+    # 目前根据字符串长度粗略估计token数，不影响正常使用
+    tokens = 0
+    for msg in messages:
+        tokens += len(msg["text"])
+    return tokens

bridge/bridge.py

@@ -19,38 +19,45 @@ class Bridge(object):
             "translate": conf().get("translate", "baidu"),
         }
         # 这边取配置的模型
-        model_type = conf().get("model") or const.GPT35
-        if model_type in ["text-davinci-003"]:
-            self.btype["chat"] = const.OPEN_AI
-        if conf().get("use_azure_chatgpt", False):
-            self.btype["chat"] = const.CHATGPTONAZURE
-        if model_type in ["wenxin", "wenxin-4"]:
-            self.btype["chat"] = const.BAIDU
-        if model_type in ["xunfei"]:
-            self.btype["chat"] = const.XUNFEI
-        if model_type in [const.QWEN]:
-            self.btype["chat"] = const.QWEN
-        if model_type in [const.QWEN_TURBO, const.QWEN_PLUS, const.QWEN_MAX]:
-            self.btype["chat"] = const.QWEN_DASHSCOPE
-        if model_type in [const.GEMINI]:
-            self.btype["chat"] = const.GEMINI
-        if model_type in [const.ZHIPU_AI]:
-            self.btype["chat"] = const.ZHIPU_AI
-        if model_type and model_type.startswith("claude-3"):
-            self.btype["chat"] = const.CLAUDEAPI
+        bot_type = conf().get("bot_type")
+        if bot_type:
+            self.btype["chat"] = bot_type
+        else:
+            model_type = conf().get("model") or const.GPT35
+            if model_type in ["text-davinci-003"]:
+                self.btype["chat"] = const.OPEN_AI
+            if conf().get("use_azure_chatgpt", False):
+                self.btype["chat"] = const.CHATGPTONAZURE
+            if model_type in ["wenxin", "wenxin-4"]:
+                self.btype["chat"] = const.BAIDU
+            if model_type in ["xunfei"]:
+                self.btype["chat"] = const.XUNFEI
+            if model_type in [const.QWEN]:
+                self.btype["chat"] = const.QWEN
+            if model_type in [const.QWEN_TURBO, const.QWEN_PLUS, const.QWEN_MAX]:
+                self.btype["chat"] = const.QWEN_DASHSCOPE
+            if model_type and model_type.startswith("gemini"):
+                self.btype["chat"] = const.GEMINI
+            if model_type in [const.ZHIPU_AI]:
+                self.btype["chat"] = const.ZHIPU_AI
+            if model_type and model_type.startswith("claude-3"):
+                self.btype["chat"] = const.CLAUDEAPI
 
-        if model_type in ["claude"]:
-            self.btype["chat"] = const.CLAUDEAI
+            if model_type in ["claude"]:
+                self.btype["chat"] = const.CLAUDEAI
 
-        if model_type in ["moonshot-v1-8k", "moonshot-v1-32k", "moonshot-v1-128k"]:
-            self.btype["chat"] = const.MOONSHOT
+            if model_type in ["moonshot-v1-8k", "moonshot-v1-32k", "moonshot-v1-128k"]:
+                self.btype["chat"] = const.MOONSHOT
 
-        if conf().get("use_linkai") and conf().get("linkai_api_key"):
-            self.btype["chat"] = const.LINKAI
-            if not conf().get("voice_to_text") or conf().get("voice_to_text") in ["openai"]:
-                self.btype["voice_to_text"] = const.LINKAI
-            if not conf().get("text_to_voice") or conf().get("text_to_voice") in ["openai", const.TTS_1, const.TTS_1_HD]:
-                self.btype["text_to_voice"] = const.LINKAI
+            if model_type in ["abab6.5-chat"]:
+                self.btype["chat"] = const.MiniMax
+
+            if conf().get("use_linkai") and conf().get("linkai_api_key"):
+                self.btype["chat"] = const.LINKAI
+                if not conf().get("voice_to_text") or conf().get("voice_to_text") in ["openai"]:
+                    self.btype["voice_to_text"] = const.LINKAI
+                if not conf().get("text_to_voice") or conf().get("text_to_voice") in ["openai", const.TTS_1, const.TTS_1_HD]:
+                    self.btype["text_to_voice"] = const.LINKAI
 
         self.bots = {}
         self.chat_bots = {}

channel/chat_channel.py

@@ -86,14 +86,14 @@ class ChatChannel(Channel):
             if e_context.is_pass() or context is None:
                 return context
             if cmsg.from_user_id == self.user_id and not config.get("trigger_by_self", True):
-                logger.debug("[WX]self message skipped")
+                logger.debug("[chat_channel]self message skipped")
                 return None
 
         # 消息内容匹配过程，并处理content
         if ctype == ContextType.TEXT:
             if first_in and "」\n- - - - - - -" in content:  # 初次匹配 过滤引用消息
                 logger.debug(content)
-                logger.debug("[WX]reference query skipped")
+                logger.debug("[chat_channel]reference query skipped")
                 return None
 
             nick_name_black_list = conf().get("nick_name_black_list", [])
@@ -111,12 +111,13 @@ class ChatChannel(Channel):
                         nick_name = context["msg"].actual_user_nickname
                         if nick_name and nick_name in nick_name_black_list:
                             # 黑名单过滤
-                            logger.warning(f"[WX] Nickname {nick_name} in In BlackList, ignore")
+                            logger.warning(f"[chat_channel] Nickname {nick_name} in In BlackList, ignore")
                             return None
 
-                        logger.info("[WX]receive group at")
+                        logger.info("[chat_channel]receive group at")
                         if not conf().get("group_at_off", False):
                             flag = True
+                        self.name = self.name if self.name is not None else ""  # 部分渠道self.name可能没有赋值
                         pattern = f"@{re.escape(self.name)}(\u2005|\u0020)"
                         subtract_res = re.sub(pattern, r"", content)
                         if isinstance(context["msg"].at_list, list):
@@ -130,13 +131,13 @@ class ChatChannel(Channel):
                         content = subtract_res
                 if not flag:
                     if context["origin_ctype"] == ContextType.VOICE:
-                        logger.info("[WX]receive group voice, but checkprefix didn't match")
+                        logger.info("[chat_channel]receive group voice, but checkprefix didn't match")
                     return None
             else:  # 单聊
                 nick_name = context["msg"].from_user_nickname
                 if nick_name and nick_name in nick_name_black_list:
                     # 黑名单过滤
-                    logger.warning(f"[WX] Nickname '{nick_name}' in In BlackList, ignore")
+                    logger.warning(f"[chat_channel] Nickname '{nick_name}' in In BlackList, ignore")
                     return None
 
                 match_prefix = check_prefix(content, conf().get("single_chat_prefix", [""]))
@@ -147,7 +148,7 @@ class ChatChannel(Channel):
                 else:
                     return None
             content = content.strip()
-            img_match_prefix = check_prefix(content, conf().get("image_create_prefix"))
+            img_match_prefix = check_prefix(content, conf().get("image_create_prefix",[""]))
             if img_match_prefix:
                 content = content.replace(img_match_prefix, "", 1)
                 context.type = ContextType.IMAGE_CREATE
@@ -159,17 +160,16 @@ class ChatChannel(Channel):
         elif context.type == ContextType.VOICE:
             if "desire_rtype" not in context and conf().get("voice_reply_voice") and ReplyType.VOICE not in self.NOT_SUPPORT_REPLYTYPE:
                 context["desire_rtype"] = ReplyType.VOICE
-
         return context
 
     def _handle(self, context: Context):
         if context is None or not context.content:
             return
-        logger.debug("[WX] ready to handle context: {}".format(context))
+        logger.debug("[chat_channel] ready to handle context: {}".format(context))
         # reply的构建步骤
         reply = self._generate_reply(context)
 
-        logger.debug("[WX] ready to decorate reply: {}".format(reply))
+        logger.debug("[chat_channel] ready to decorate reply: {}".format(reply))
 
         # reply的包装步骤
         if reply and reply.content:
@@ -187,7 +187,7 @@ class ChatChannel(Channel):
         )
         reply = e_context["reply"]
         if not e_context.is_pass():
-            logger.debug("[WX] ready to handle context: type={}, content={}".format(context.type, context.content))
+            logger.debug("[chat_channel] ready to handle context: type={}, content={}".format(context.type, context.content))
             if context.type == ContextType.TEXT or context.type == ContextType.IMAGE_CREATE:  # 文字和图片消息
                 context["channel"] = e_context["channel"]
                 reply = super().build_reply_content(context.content, context)
@@ -199,7 +199,7 @@ class ChatChannel(Channel):
                 try:
                     any_to_wav(file_path, wav_path)
                 except Exception as e:  # 转换失败，直接使用mp3，对于某些api，mp3也可以识别
-                    logger.warning("[WX]any to wav error, use raw path. " + str(e))
+                    logger.warning("[chat_channel]any to wav error, use raw path. " + str(e))
                     wav_path = file_path
                 # 语音识别
                 reply = super().build_voice_to_text(wav_path)
@@ -210,7 +210,7 @@ class ChatChannel(Channel):
                         os.remove(wav_path)
                 except Exception as e:
                     pass
-                    # logger.warning("[WX]delete temp file error: " + str(e))
+                    # logger.warning("[chat_channel]delete temp file error: " + str(e))
 
                 if reply.type == ReplyType.TEXT:
                     new_context = self._compose_context(ContextType.TEXT, reply.content, **context.kwargs)
@@ -228,7 +228,7 @@ class ChatChannel(Channel):
             elif context.type == ContextType.FUNCTION or context.type == ContextType.FILE:  # 文件消息及函数调用等，当前无默认逻辑
                 pass
             else:
-                logger.warning("[WX] unknown context type: {}".format(context.type))
+                logger.warning("[chat_channel] unknown context type: {}".format(context.type))
                 return
         return reply
 
@@ -244,7 +244,7 @@ class ChatChannel(Channel):
             desire_rtype = context.get("desire_rtype")
             if not e_context.is_pass() and reply and reply.type:
                 if reply.type in self.NOT_SUPPORT_REPLYTYPE:
-                    logger.error("[WX]reply type not support: " + str(reply.type))
+                    logger.error("[chat_channel]reply type not support: " + str(reply.type))
                     reply.type = ReplyType.ERROR
                     reply.content = "不支持发送的消息类型: " + str(reply.type)
 
@@ -265,10 +265,10 @@ class ChatChannel(Channel):
                 elif reply.type == ReplyType.IMAGE_URL or reply.type == ReplyType.VOICE or reply.type == ReplyType.IMAGE or reply.type == ReplyType.FILE or reply.type == ReplyType.VIDEO or reply.type == ReplyType.VIDEO_URL:
                     pass
                 else:
-                    logger.error("[WX] unknown reply type: {}".format(reply.type))
+                    logger.error("[chat_channel] unknown reply type: {}".format(reply.type))
                     return
             if desire_rtype and desire_rtype != reply.type and reply.type not in [ReplyType.ERROR, ReplyType.INFO]:
-                logger.warning("[WX] desire_rtype: {}, but reply type: {}".format(context.get("desire_rtype"), reply.type))
+                logger.warning("[chat_channel] desire_rtype: {}, but reply type: {}".format(context.get("desire_rtype"), reply.type))
             return reply
 
     def _send_reply(self, context: Context, reply: Reply):
@@ -281,14 +281,14 @@ class ChatChannel(Channel):
             )
             reply = e_context["reply"]
             if not e_context.is_pass() and reply and reply.type:
-                logger.debug("[WX] ready to send reply: {}, context: {}".format(reply, context))
+                logger.debug("[chat_channel] ready to send reply: {}, context: {}".format(reply, context))
                 self._send(reply, context)
 
     def _send(self, reply: Reply, context: Context, retry_cnt=0):
         try:
             self.send(reply, context)
         except Exception as e:
-            logger.error("[WX] sendMsg error: {}".format(str(e)))
+            logger.error("[chat_channel] sendMsg error: {}".format(str(e)))
             if isinstance(e, NotImplementedError):
                 return
             logger.exception(e)
@@ -342,7 +342,7 @@ class ChatChannel(Channel):
                     if semaphore.acquire(blocking=False):  # 等线程处理完毕才能删除
                         if not context_queue.empty():
                             context = context_queue.get()
-                            logger.debug("[WX] consume context: {}".format(context))
+                            logger.debug("[chat_channel] consume context: {}".format(context))
                             future: Future = handler_pool.submit(self._handle, context)
                             future.add_done_callback(self._thread_pool_callback(session_id, context=context))
                             if session_id not in self.futures:

channel/dingtalk/dingtalk_channel.py

@@ -4,20 +4,81 @@
 @author huiwen
 @Date 2023/11/28
 """
-
+import copy
+import json
 # -*- coding=utf-8 -*-
+import logging
+import time
+
+import dingtalk_stream
+from dingtalk_stream import AckMessage
+from dingtalk_stream.card_replier import AICardReplier
+from dingtalk_stream.card_replier import AICardStatus
+from dingtalk_stream.card_replier import CardReplier
+
+from bridge.context import Context, ContextType
+from bridge.reply import Reply, ReplyType
+from channel.chat_channel import ChatChannel
 from channel.dingtalk.dingtalk_message import DingTalkMessage
-from bridge.context import Context
-from bridge.reply import Reply
+from common.expired_dict import ExpiredDict
 from common.log import logger
 from common.singleton import singleton
+from common.time_check import time_checker
 from config import conf
-from common.expired_dict import ExpiredDict
-from bridge.context import ContextType
-from channel.chat_channel import ChatChannel
-import logging
-from dingtalk_stream import AckMessage
-import dingtalk_stream
+
+
+class CustomAICardReplier(CardReplier):
+    def __init__(self, dingtalk_client, incoming_message):
+        super(AICardReplier, self).__init__(dingtalk_client, incoming_message)
+
+    def start(
+            self,
+            card_template_id: str,
+            card_data: dict,
+            recipients: list = None,
+            support_forward: bool = True,
+    ) -> str:
+        """
+        AI卡片的创建接口
+        :param support_forward:
+        :param recipients:
+        :param card_template_id:
+        :param card_data:
+        :return:
+        """
+        card_data_with_status = copy.deepcopy(card_data)
+        card_data_with_status["flowStatus"] = AICardStatus.PROCESSING
+        return self.create_and_send_card(
+            card_template_id,
+            card_data_with_status,
+            at_sender=True,
+            at_all=False,
+            recipients=recipients,
+            support_forward=support_forward,
+        )
+
+
+# 对 AICardReplier 进行猴子补丁
+AICardReplier.start = CustomAICardReplier.start
+
+
+def _check(func):
+    def wrapper(self, cmsg: DingTalkMessage):
+        msgId = cmsg.msg_id
+        if msgId in self.receivedMsgs:
+            logger.info("DingTalk message {} already received, ignore".format(msgId))
+            return
+        self.receivedMsgs[msgId] = True
+        create_time = cmsg.create_time  # 消息时间戳
+        if conf().get("hot_reload") == True and int(create_time) < int(time.time()) - 60:  # 跳过1分钟前的历史消息
+            logger.debug("[DingTalk] History message {} skipped".format(msgId))
+            return
+        if cmsg.my_msg and not cmsg.is_group:
+            logger.debug("[DingTalk] My message {} skipped".format(msgId))
+            return
+        return func(self, cmsg)
+
+    return wrapper
 
 
 @singleton
@@ -39,11 +100,13 @@ class DingTalkChanel(ChatChannel, dingtalk_stream.ChatbotHandler):
         super(dingtalk_stream.ChatbotHandler, self).__init__()
         self.logger = self.setup_logger()
         # 历史消息id暂存，用于幂等控制
-        self.receivedMsgs = ExpiredDict(60 * 60 * 7.1)
-        logger.info("[dingtalk] client_id={}, client_secret={} ".format(
+        self.receivedMsgs = ExpiredDict(conf().get("expires_in_seconds"))
+        logger.info("[DingTalk] client_id={}, client_secret={} ".format(
             self.dingtalk_client_id, self.dingtalk_client_secret))
         # 无需群校验和前缀
         conf()["group_name_white_list"] = ["ALL_GROUP"]
+        # 单聊无需前缀
+        conf()["single_chat_prefix"] = [""]
 
     def startup(self):
         credential = dingtalk_stream.Credential(self.dingtalk_client_id, self.dingtalk_client_secret)
@@ -51,50 +114,112 @@ class DingTalkChanel(ChatChannel, dingtalk_stream.ChatbotHandler):
         client.register_callback_handler(dingtalk_stream.chatbot.ChatbotMessage.TOPIC, self)
         client.start_forever()
 
+    async def process(self, callback: dingtalk_stream.CallbackMessage):
+        try:
+            incoming_message = dingtalk_stream.ChatbotMessage.from_dict(callback.data)
+            image_download_handler = self  # 传入方法所在的类实例
+            dingtalk_msg = DingTalkMessage(incoming_message, image_download_handler)
+
+            if dingtalk_msg.is_group:
+                self.handle_group(dingtalk_msg)
+            else:
+                self.handle_single(dingtalk_msg)
+            return AckMessage.STATUS_OK, 'OK'
+        except Exception as e:
+            logger.error(f"dingtalk process error={e}")
+            return AckMessage.STATUS_SYSTEM_EXCEPTION, 'ERROR'
+
+    @time_checker
+    @_check
     def handle_single(self, cmsg: DingTalkMessage):
         # 处理单聊消息
         if cmsg.ctype == ContextType.VOICE:
-            logger.debug("[dingtalk]receive voice msg: {}".format(cmsg.content))
+            logger.debug("[DingTalk]receive voice msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.IMAGE:
-            logger.debug("[dingtalk]receive image msg: {}".format(cmsg.content))
+            logger.debug("[DingTalk]receive image msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.IMAGE_CREATE:
+            logger.debug("[DingTalk]receive image create msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.PATPAT:
-            logger.debug("[dingtalk]receive patpat msg: {}".format(cmsg.content))
+            logger.debug("[DingTalk]receive patpat msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.TEXT:
-            expression = cmsg.my_msg
-        cmsg.content = conf()["single_chat_prefix"][0] + cmsg.content
+            logger.debug("[DingTalk]receive text msg: {}".format(cmsg.content))
+        else:
+            logger.debug("[DingTalk]receive other msg: {}".format(cmsg.content))
         context = self._compose_context(cmsg.ctype, cmsg.content, isgroup=False, msg=cmsg)
         if context:
             self.produce(context)
 
+
+    @time_checker
+    @_check
     def handle_group(self, cmsg: DingTalkMessage):
         # 处理群聊消息
         if cmsg.ctype == ContextType.VOICE:
-            logger.debug("[dingtalk]receive voice msg: {}".format(cmsg.content))
+            logger.debug("[DingTalk]receive voice msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.IMAGE:
-            logger.debug("[dingtalk]receive image msg: {}".format(cmsg.content))
+            logger.debug("[DingTalk]receive image msg: {}".format(cmsg.content))
+        elif cmsg.ctype == ContextType.IMAGE_CREATE:
+            logger.debug("[DingTalk]receive image create msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.PATPAT:
-            logger.debug("[dingtalk]receive patpat msg: {}".format(cmsg.content))
+            logger.debug("[DingTalk]receive patpat msg: {}".format(cmsg.content))
         elif cmsg.ctype == ContextType.TEXT:
-            expression = cmsg.my_msg
-        cmsg.content = conf()["group_chat_prefix"][0] + cmsg.content
+            logger.debug("[DingTalk]receive text msg: {}".format(cmsg.content))
+        else:
+            logger.debug("[DingTalk]receive other msg: {}".format(cmsg.content))
         context = self._compose_context(cmsg.ctype, cmsg.content, isgroup=True, msg=cmsg)
         context['no_need_at'] = True
         if context:
             self.produce(context)
 
-    async def process(self, callback: dingtalk_stream.CallbackMessage):
-        try:
-            incoming_message = dingtalk_stream.ChatbotMessage.from_dict(callback.data)
-            dingtalk_msg = DingTalkMessage(incoming_message)
-            if incoming_message.conversation_type == '1':
-                self.handle_single(dingtalk_msg)
-            else:
-                self.handle_group(dingtalk_msg)
-            return AckMessage.STATUS_OK, 'OK'
-        except Exception as e:
-            logger.error(e)
-            return self.FAILED_MSG
 
     def send(self, reply: Reply, context: Context):
+        receiver = context["receiver"]
+        isgroup = context.kwargs['msg'].is_group
         incoming_message = context.kwargs['msg'].incoming_message
-        self.reply_text(reply.content, incoming_message)
+
+        if conf().get("dingtalk_card_enabled"):
+            logger.info("[Dingtalk] sendMsg={}, receiver={}".format(reply, receiver))
+            def reply_with_text():
+                self.reply_text(reply.content, incoming_message)
+            def reply_with_at_text():
+                self.reply_text("📢 您有一条新的消息，请查看。", incoming_message)
+            def reply_with_ai_markdown():
+                button_list, markdown_content = self.generate_button_markdown_content(context, reply)
+                self.reply_ai_markdown_button(incoming_message, markdown_content, button_list, "", "📌 内容由AI生成", "",[incoming_message.sender_staff_id])
+
+            if reply.type in [ReplyType.IMAGE_URL, ReplyType.IMAGE, ReplyType.TEXT]:
+                if isgroup:
+                    reply_with_ai_markdown()
+                    reply_with_at_text()
+                else:
+                    reply_with_ai_markdown()
+            else:
+                # 暂不支持其它类型消息回复
+                reply_with_text()
+        else:
+            self.reply_text(reply.content, incoming_message)
+
+
+    def generate_button_markdown_content(self, context, reply):
+        image_url = context.kwargs.get("image_url")
+        promptEn = context.kwargs.get("promptEn")
+        reply_text = reply.content
+        button_list = []
+        markdown_content = f"""
+{reply.content}
+                                """
+        if image_url is not None and promptEn is not None:
+            button_list = [
+                {"text": "查看原图", "url": image_url, "iosUrl": image_url, "color": "blue"}
+            ]
+            markdown_content = f"""
+{promptEn}
+
+!["图片"]({image_url})
+
+{reply_text}
+
+                                """
+        logger.debug(f"[Dingtalk] generate_button_markdown_content, button_list={button_list} , markdown_content={markdown_content}")
+
+        return button_list, markdown_content

channel/dingtalk/dingtalk_message.py

@@ -1,44 +1,84 @@
-from bridge.context import ContextType
-from channel.chat_message import ChatMessage
-import json
+import os
+
 import requests
-from common.log import logger
-from common.tmp_dir import TmpDir
-from common import utils
 from dingtalk_stream import ChatbotMessage
 
+from bridge.context import ContextType
+from channel.chat_message import ChatMessage
+# -*- coding=utf-8 -*-
+from common.log import logger
+from common.tmp_dir import TmpDir
+
+
 class DingTalkMessage(ChatMessage):
-    def __init__(self, event: ChatbotMessage):
+    def __init__(self, event: ChatbotMessage, image_download_handler):
         super().__init__(event)
-        
+        self.image_download_handler = image_download_handler
         self.msg_id = event.message_id
-        msg_type = event.message_type
-        self.incoming_message =event
+        self.message_type = event.message_type
+        self.incoming_message = event
         self.sender_staff_id = event.sender_staff_id
         self.other_user_id = event.conversation_id
         self.create_time = event.create_at
-        if event.conversation_type=="1":
+        self.image_content = event.image_content
+        self.rich_text_content = event.rich_text_content
+        if event.conversation_type == "1":
             self.is_group = False
         else:
             self.is_group = True
-        
 
-        if msg_type == "text":
+        if self.message_type == "text":
             self.ctype = ContextType.TEXT
-            
+
             self.content = event.text.content.strip()
-        elif msg_type == "audio":
-            
+        elif self.message_type == "audio":
             # 钉钉支持直接识别语音，所以此处将直接提取文字，当文字处理
             self.content = event.extensions['content']['recognition'].strip()
             self.ctype = ContextType.TEXT
+        elif (self.message_type == 'picture') or (self.message_type == 'richText'):
+            self.ctype = ContextType.IMAGE
+            # 钉钉图片类型或富文本类型消息处理
+            image_list = event.get_image_list()
+            if len(image_list) > 0:
+                download_code = image_list[0]
+                download_url = image_download_handler.get_image_download_url(download_code)
+                self.content = download_image_file(download_url, TmpDir().path())
+            else:
+                logger.debug(f"[Dingtalk] messageType :{self.message_type} , imageList isEmpty")
+
         if self.is_group:
             self.from_user_id = event.conversation_id
             self.actual_user_id = event.sender_id
+            self.is_at = True
         else:
             self.from_user_id = event.sender_id
+            self.actual_user_id = event.sender_id
         self.to_user_id = event.chatbot_user_id
         self.other_user_nickname = event.conversation_title
-        
-        user_id = event.sender_id
-        nickname =event.sender_nick
+
+
+def download_image_file(image_url, temp_dir):
+    headers = {
+        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36'
+    }
+    # 设置代理
+    # self.proxies
+    # , proxies=self.proxies
+    response = requests.get(image_url, headers=headers, stream=True, timeout=60 * 5)
+    if response.status_code == 200:
+
+        # 生成文件名
+        file_name = image_url.split("/")[-1].split("?")[0]
+
+        # 检查临时目录是否存在，如果不存在则创建
+        if not os.path.exists(temp_dir):
+            os.makedirs(temp_dir)
+
+        # 将文件保存到临时目录
+        file_path = os.path.join(temp_dir, file_name)
+        with open(file_path, 'wb') as file:
+            file.write(response.content)
+        return file_path
+    else:
+        logger.info(f"[Dingtalk] Failed to download image file, {response.content}")
+        return None

channel/feishu/feishu_channel.py

@@ -40,7 +40,7 @@ class FeiShuChanel(ChatChannel):
             self.feishu_app_id, self.feishu_app_secret, self.feishu_token))
         # 无需群校验和前缀
         conf()["group_name_white_list"] = ["ALL_GROUP"]
-        conf()["single_chat_prefix"] = []
+        conf()["single_chat_prefix"] = [""]
 
     def startup(self):
         urls = (

channel/terminal/terminal_channel.py

@@ -78,6 +78,7 @@ class TerminalChannel(ChatChannel):
                 prompt = trigger_prefixs[0] + prompt  # 给没触发的消息加上触发前缀
 
             context = self._compose_context(ContextType.TEXT, prompt, msg=TerminalMessage(msg_id, prompt))
+            context["isgroup"] = False
             if context:
                 self.produce(context)
             else:

channel/wechat/wechat_channel.py

@@ -109,7 +109,7 @@ class WechatChannel(ChatChannel):
 
     def __init__(self):
         super().__init__()
-        self.receivedMsgs = ExpiredDict(60 * 60)
+        self.receivedMsgs = ExpiredDict(conf().get("expires_in_seconds"))
         self.auto_login_times = 0
 
     def startup(self):

channel/wechatcom/wechatcomapp_channel.py

@@ -44,7 +44,7 @@ class WechatComAppChannel(ChatChannel):
 
     def startup(self):
         # start message listener
-        urls = ("/wxcomapp", "channel.wechatcom.wechatcomapp_channel.Query")
+        urls = ("/wxcomapp/?", "channel.wechatcom.wechatcomapp_channel.Query")
         app = web.application(urls, globals(), autoreload=False)
         port = conf().get("wechatcomapp_port", 9898)
         web.httpserver.runsimple(app.wsgifunc(), ("0.0.0.0", port))

channel/wechatmp/wechatmp_channel.py

@@ -140,6 +140,42 @@ class WechatMPChannel(ChatChannel):
                 media_id = response["media_id"]
                 logger.info("[wechatmp] image uploaded, receiver {}, media_id {}".format(receiver, media_id))
                 self.cache_dict[receiver].append(("image", media_id))
+            elif reply.type == ReplyType.VIDEO_URL:  # 从网络下载视频
+                video_url = reply.content
+                video_res = requests.get(video_url, stream=True)
+                video_storage = io.BytesIO()
+                for block in video_res.iter_content(1024):
+                    video_storage.write(block)
+                video_storage.seek(0)
+                video_type = 'mp4'
+                filename = receiver + "-" + str(context["msg"].msg_id) + "." + video_type
+                content_type = "video/" + video_type
+                try:
+                    response = self.client.material.add("video", (filename, video_storage, content_type))
+                    logger.debug("[wechatmp] upload video response: {}".format(response))
+                except WeChatClientException as e:
+                    logger.error("[wechatmp] upload video failed: {}".format(e))
+                    return
+                media_id = response["media_id"]
+                logger.info("[wechatmp] video uploaded, receiver {}, media_id {}".format(receiver, media_id))
+                self.cache_dict[receiver].append(("video", media_id))
+
+            elif reply.type == ReplyType.VIDEO:  # 从文件读取视频
+                video_storage = reply.content
+                video_storage.seek(0)
+                video_type = 'mp4'
+                filename = receiver + "-" + str(context["msg"].msg_id) + "." + video_type
+                content_type = "video/" + video_type
+                try:
+                    response = self.client.material.add("video", (filename, video_storage, content_type))
+                    logger.debug("[wechatmp] upload video response: {}".format(response))
+                except WeChatClientException as e:
+                    logger.error("[wechatmp] upload video failed: {}".format(e))
+                    return
+                media_id = response["media_id"]
+                logger.info("[wechatmp] video uploaded, receiver {}, media_id {}".format(receiver, media_id))
+                self.cache_dict[receiver].append(("video", media_id))
+
         else:
             if reply.type == ReplyType.TEXT or reply.type == ReplyType.INFO or reply.type == ReplyType.ERROR:
                 reply_text = reply.content
@@ -222,6 +258,38 @@ class WechatMPChannel(ChatChannel):
                     return
                 self.client.message.send_image(receiver, response["media_id"])
                 logger.info("[wechatmp] Do send image to {}".format(receiver))
+            elif reply.type == ReplyType.VIDEO_URL:  # 从网络下载视频
+                video_url = reply.content
+                video_res = requests.get(video_url, stream=True)
+                video_storage = io.BytesIO()
+                for block in video_res.iter_content(1024):
+                    video_storage.write(block)
+                video_storage.seek(0)
+                video_type = 'mp4'
+                filename = receiver + "-" + str(context["msg"].msg_id) + "." + video_type
+                content_type = "video/" + video_type
+                try:
+                    response = self.client.media.upload("video", (filename, video_storage, content_type))
+                    logger.debug("[wechatmp] upload video response: {}".format(response))
+                except WeChatClientException as e:
+                    logger.error("[wechatmp] upload video failed: {}".format(e))
+                    return
+                self.client.message.send_video(receiver, response["media_id"])
+                logger.info("[wechatmp] Do send video to {}".format(receiver))
+            elif reply.type == ReplyType.VIDEO:  # 从文件读取视频
+                video_storage = reply.content
+                video_storage.seek(0)
+                video_type = 'mp4'
+                filename = receiver + "-" + str(context["msg"].msg_id) + "." + video_type
+                content_type = "video/" + video_type
+                try:
+                    response = self.client.media.upload("video", (filename, video_storage, content_type))
+                    logger.debug("[wechatmp] upload video response: {}".format(response))
+                except WeChatClientException as e:
+                    logger.error("[wechatmp] upload video failed: {}".format(e))
+                    return
+                self.client.message.send_video(receiver, response["media_id"])
+                logger.info("[wechatmp] Do send video to {}".format(receiver))
         return
 
     def _success_callback(self, session_id, context, **kwargs):  # 线程异常结束时的回调函数

common/const.py

@@ -1,44 +1,73 @@
 # bot_type
 OPEN_AI = "openAI"
 CHATGPT = "chatGPT"
-BAIDU = "baidu"
+BAIDU = "baidu"  # 百度文心一言模型
 XUNFEI = "xunfei"
 CHATGPTONAZURE = "chatGPTOnAzure"
 LINKAI = "linkai"
-CLAUDEAI = "claude"
-CLAUDEAPI= "claudeAPI"
-QWEN = "qwen"
+CLAUDEAI = "claude"  # 使用cookie的历史模型
+CLAUDEAPI= "claudeAPI"  # 通过Claude api调用模型
+QWEN = "qwen"  # 旧版通义模型
+QWEN_DASHSCOPE = "dashscope"  # 通义新版sdk和api key
 
-QWEN_DASHSCOPE = "dashscope"
-QWEN_TURBO = "qwen-turbo"
-QWEN_PLUS = "qwen-plus"
-QWEN_MAX = "qwen-max"
 
-GEMINI = "gemini"
+GEMINI = "gemini"  # gemini-1.0-pro
 ZHIPU_AI = "glm-4"
 MOONSHOT = "moonshot"
+MiniMax = "minimax"
 
 
 # model
 CLAUDE3 = "claude-3-opus-20240229"
 GPT35 = "gpt-3.5-turbo"
-GPT4 = "gpt-4"
+GPT35_0125 = "gpt-3.5-turbo-0125"
+GPT35_1106 = "gpt-3.5-turbo-1106"
+
 GPT_4o = "gpt-4o"
-LINKAI_35 = "linkai-3.5"
-LINKAI_4_TURBO = "linkai-4-turbo"
-LINKAI_4o = "linkai-4o"
-GPT4_TURBO_PREVIEW = "gpt-4-turbo-2024-04-09"
+GPT4_TURBO = "gpt-4-turbo"
+GPT4_TURBO_PREVIEW = "gpt-4-turbo-preview"
 GPT4_TURBO_04_09 = "gpt-4-turbo-2024-04-09"
 GPT4_TURBO_01_25 = "gpt-4-0125-preview"
+GPT4_TURBO_11_06 = "gpt-4-1106-preview"
 GPT4_VISION_PREVIEW = "gpt-4-vision-preview"
+
+GPT4 = "gpt-4"
+GPT_4o_MINI = "gpt-4o-mini"
+GPT4_32k = "gpt-4-32k"
+GPT4_06_13 = "gpt-4-0613"
+GPT4_32k_06_13 = "gpt-4-32k-0613"
+
 WHISPER_1 = "whisper-1"
 TTS_1 = "tts-1"
 TTS_1_HD = "tts-1-hd"
 
-MODEL_LIST = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4", "wenxin", "wenxin-4", "xunfei", "claude", "claude-3-opus-20240229", "gpt-4-turbo",
-              "gpt-4-turbo-preview", "gpt-4-1106-preview", GPT4_TURBO_PREVIEW, GPT4_TURBO_01_25, GPT_4o, QWEN, GEMINI, ZHIPU_AI, MOONSHOT,
-              QWEN_TURBO, QWEN_PLUS, QWEN_MAX, LINKAI_35, LINKAI_4_TURBO, LINKAI_4o]
+WEN_XIN = "wenxin"
+WEN_XIN_4 = "wenxin-4"
+
+QWEN_TURBO = "qwen-turbo"
+QWEN_PLUS = "qwen-plus"
+QWEN_MAX = "qwen-max"
+
+LINKAI_35 = "linkai-3.5"
+LINKAI_4_TURBO = "linkai-4-turbo"
+LINKAI_4o = "linkai-4o"
+
+GEMINI_PRO = "gemini-1.0-pro"
+GEMINI_15_flash = "gemini-1.5-flash"
+GEMINI_15_PRO = "gemini-1.5-pro"
+
+MODEL_LIST = [
+              GPT35, GPT35_0125, GPT35_1106, "gpt-3.5-turbo-16k",
+              GPT_4o, GPT_4o_MINI, GPT4_TURBO, GPT4_TURBO_PREVIEW, GPT4_TURBO_01_25, GPT4_TURBO_11_06, GPT4, GPT4_32k, GPT4_06_13, GPT4_32k_06_13,
+              WEN_XIN, WEN_XIN_4,
+              XUNFEI, ZHIPU_AI, MOONSHOT, MiniMax,
+              GEMINI, GEMINI_PRO, GEMINI_15_flash, GEMINI_15_PRO,
+              "claude", "claude-3-haiku", "claude-3-sonnet", "claude-3-opus", "claude-3-opus-20240229", "claude-3.5-sonnet",
+              "moonshot-v1-8k", "moonshot-v1-32k", "moonshot-v1-128k",
+              QWEN, QWEN_TURBO, QWEN_PLUS, QWEN_MAX,
+              LINKAI_35, LINKAI_4_TURBO, LINKAI_4o
+            ]
 
 # channel
 FEISHU = "feishu"
-DINGTALK = "dingtalk"   
+DINGTALK = "dingtalk"

common/linkai_client.py

@@ -45,8 +45,11 @@ class ChatClient(LinkAIClient):
             elif reply_voice_mode == "always_reply_voice":
                 local_config["always_reply_voice"] = True
 
-        if config.get("admin_password") and plugin_config.get("Godcmd"):
-            plugin_config["Godcmd"]["password"] = config.get("admin_password")
+        if config.get("admin_password"):
+            if not plugin_config.get("Godcmd"):
+                plugin_config["Godcmd"] = {"password": config.get("admin_password"), "admin_users": []}
+            else:
+                plugin_config["Godcmd"]["password"] = config.get("admin_password")
             PluginManager().instances["GODCMD"].reload()
 
         if config.get("group_app_map") and pconf("linkai"):

common/time_check.py

@@ -1,7 +1,5 @@
-import hashlib
 import re
 import time
-
 import config
 from common.log import logger
 
@@ -10,31 +8,33 @@ def time_checker(f):
     def _time_checker(self, *args, **kwargs):
         _config = config.conf()
         chat_time_module = _config.get("chat_time_module", False)
+
         if chat_time_module:
             chat_start_time = _config.get("chat_start_time", "00:00")
-            chat_stopt_time = _config.get("chat_stop_time", "24:00")
-            time_regex = re.compile(r"^([01]?[0-9]|2[0-4])(:)([0-5][0-9])$")  # 时间匹配，包含24:00
+            chat_stop_time = _config.get("chat_stop_time", "24:00")
 
-            starttime_format_check = time_regex.match(chat_start_time)  # 检查停止时间格式
-            stoptime_format_check = time_regex.match(chat_stopt_time)  # 检查停止时间格式
-            chat_time_check = chat_start_time < chat_stopt_time  # 确定启动时间<停止时间
+            time_regex = re.compile(r"^([01]?[0-9]|2[0-4])(:)([0-5][0-9])$")
 
-            # 时间格式检查
-            if not (starttime_format_check and stoptime_format_check and chat_time_check):
-                logger.warn("时间格式不正确,请在config.json中修改您的CHAT_START_TIME/CHAT_STOP_TIME,否则可能会影响您正常使用,开始({})-结束({})".format(starttime_format_check, stoptime_format_check))
-            if chat_start_time > "23:59":
-                logger.error("启动时间可能存在问题，请修改!")
-
-            # 服务时间检查
-            now_time = time.strftime("%H:%M", time.localtime())
-            if chat_start_time <= now_time <= chat_stopt_time:  # 服务时间内，正常返回回答
-                f(self, *args, **kwargs)
+            if not (time_regex.match(chat_start_time) and time_regex.match(chat_stop_time)):
+                logger.warning("时间格式不正确，请在config.json中修改CHAT_START_TIME/CHAT_STOP_TIME。")
                 return None
+
+            now_time = time.strptime(time.strftime("%H:%M"), "%H:%M")
+            chat_start_time = time.strptime(chat_start_time, "%H:%M")
+            chat_stop_time = time.strptime(chat_stop_time, "%H:%M")
+            # 结束时间小于开始时间，跨天了
+            if chat_stop_time < chat_start_time and (chat_start_time <= now_time or now_time <= chat_stop_time):
+                f(self, *args, **kwargs)
+            # 结束大于开始时间代表，没有跨天
+            elif chat_start_time < chat_stop_time and chat_start_time <= now_time <= chat_stop_time:
+                f(self, *args, **kwargs)
             else:
-                if args[0]["Content"] == "#更新配置":  # 不在服务时间内也可以更新配置
+                # 定义匹配规则，如果以 #reconf 或者  #更新配置  结尾, 非服务时间可以修改开始/结束时间并重载配置
+                pattern = re.compile(r"^.*#(?:reconf|更新配置)$")
+                if args and pattern.match(args[0].content):
                     f(self, *args, **kwargs)
                 else:
-                    logger.info("非服务时间内,不接受访问")
+                    logger.info("非服务时间内，不接受访问")
                     return None
         else:
             f(self, *args, **kwargs)  # 未开启时间模块则直接回答

config.py

@@ -4,6 +4,7 @@ import json
 import logging
 import os
 import pickle
+import copy
 
 from common.log import logger
 
@@ -16,7 +17,8 @@ available_setting = {
     "open_ai_api_base": "https://api.openai.com/v1",
     "proxy": "",  # openai使用的代理
     # chatgpt模型， 当use_azure_chatgpt为true时，其名称为Azure上model deployment名称
-    "model": "gpt-3.5-turbo",  # 还支持 gpt-4, gpt-4-turbo, wenxin, xunfei, qwen
+    "model": "gpt-3.5-turbo",  # 可选择: gpt-4o, pt-4o-mini, gpt-4-turbo, claude-3-sonnet, wenxin, moonshot, qwen-turbo, xunfei, glm-4, minimax, gemini等模型，全部可选模型详见common/const.py文件
+    "bot_type": "",  # 可选配置，使用兼容openai格式的三方服务时候，需填"chatGPT"。bot具体名称详见common/const.py文件列出的bot_type，如不填根据model名称判断，
     "use_azure_chatgpt": False,  # 是否使用azure的chatgpt
     "azure_deployment_id": "",  # azure 模型部署名称
     "azure_api_version": "",  # azure api版本
@@ -33,14 +35,21 @@ available_setting = {
     "group_name_keyword_white_list": [],  # 开启自动回复的群名称关键词列表
     "group_chat_in_one_session": ["ChatGPT测试群"],  # 支持会话上下文共享的群名称
     "nick_name_black_list": [],  # 用户昵称黑名单
-    "group_welcome_msg": "",  # 配置新人进群固定欢迎语，不配置则使用随机风格欢迎 
+    "group_welcome_msg": "",  # 配置新人进群固定欢迎语，不配置则使用随机风格欢迎
     "trigger_by_self": False,  # 是否允许机器人触发
     "text_to_image": "dall-e-2",  # 图片生成模型，可选 dall-e-2, dall-e-3
+    # Azure OpenAI dall-e-3 配置
+    "dalle3_image_style": "vivid", # 图片生成dalle3的风格，可选有 vivid, natural
+    "dalle3_image_quality": "hd", # 图片生成dalle3的质量，可选有 standard, hd
+    # Azure OpenAI DALL-E API 配置, 当use_azure_chatgpt为true时,用于将文字回复的资源和Dall-E的资源分开.
+    "azure_openai_dalle_api_base": "", # [可选] azure openai 用于回复图片的资源 endpoint，默认使用 open_ai_api_base
+    "azure_openai_dalle_api_key": "", # [可选] azure openai 用于回复图片的资源 key，默认使用 open_ai_api_key
+    "azure_openai_dalle_deployment_id":"", # [可选] azure openai 用于回复图片的资源 deployment id，默认使用 text_to_image
     "image_proxy": True,  # 是否需要图片代理，国内访问LinkAI时需要
     "image_create_prefix": ["画", "看", "找"],  # 开启图片回复的前缀
     "concurrency_in_session": 1,  # 同一会话最多有多少条消息在处理中，大于1可能乱序
     "image_create_size": "256x256",  # 图片大小,可选有 256x256, 512x512, 1024x1024 (dall-e-3默认为1024x1024)
-    "group_chat_exit_group": False, 
+    "group_chat_exit_group": False,
     # chatgpt会话参数
     "expires_in_seconds": 3600,  # 无操作会话的过期时间
     # 人格描述
@@ -68,14 +77,14 @@ available_setting = {
     "claude_api_cookie": "",
     "claude_uuid": "",
     # claude api key
-    "claude_api_key":"",
+    "claude_api_key": "",
     # 通义千问API, 获取方式查看文档 https://help.aliyun.com/document_detail/2587494.html
     "qwen_access_key_id": "",
     "qwen_access_key_secret": "",
     "qwen_agent_key": "",
     "qwen_app_id": "",
     "qwen_node_id": "",  # 流程编排模型用到的id，如果没有用到qwen_node_id，请务必保持为空字符串
-    # 阿里灵积模型api key
+    # 阿里灵积(通义新版sdk)模型api key
     "dashscope_api_key": "",
     # Google Gemini Api Key
     "gemini_api_key": "",
@@ -86,8 +95,8 @@ available_setting = {
     "group_speech_recognition": False,  # 是否开启群组语音识别
     "voice_reply_voice": False,  # 是否使用语音回复语音，需要设置对应语音合成引擎的api key
     "always_reply_voice": False,  # 是否一直使用语音回复
-    "voice_to_text": "openai",  # 语音识别引擎，支持openai,baidu,google,azure
-    "text_to_voice": "openai",  # 语音合成引擎，支持openai,baidu,google,pytts(offline),azure,elevenlabs,edge(online)
+    "voice_to_text": "openai",  # 语音识别引擎，支持openai,baidu,google,azure,xunfei,ali
+    "text_to_voice": "openai",  # 语音合成引擎，支持openai,baidu,google,azure,xunfei,ali,pytts(offline),elevenlabs,edge(online)
     "text_to_voice_model": "tts-1",
     "tts_voice_id": "alloy",
     # baidu 语音api配置， 使用百度语音识别和语音合成时需要
@@ -95,13 +104,13 @@ available_setting = {
     "baidu_api_key": "",
     "baidu_secret_key": "",
     # 1536普通话(支持简单的英文识别) 1737英语 1637粤语 1837四川话 1936普通话远场
-    "baidu_dev_pid": "1536",
+    "baidu_dev_pid": 1536,
     # azure 语音api配置， 使用azure语音识别和语音合成时需要
     "azure_voice_api_key": "",
     "azure_voice_region": "japaneast",
     # elevenlabs 语音api配置
-    "xi_api_key": "",    #获取ap的方法可以参考https://docs.elevenlabs.io/api-reference/quick-start/authentication
-    "xi_voice_id": "",   #ElevenLabs提供了9种英式、美式等英语发音id，分别是“Adam/Antoni/Arnold/Bella/Domi/Elli/Josh/Rachel/Sam”
+    "xi_api_key": "",  # 获取ap的方法可以参考https://docs.elevenlabs.io/api-reference/quick-start/authentication
+    "xi_voice_id": "",  # ElevenLabs提供了9种英式、美式等英语发音id，分别是“Adam/Antoni/Arnold/Bella/Domi/Elli/Josh/Rachel/Sam”
     # 服务时间限制，目前支持itchat
     "chat_time_module": False,  # 是否开启服务时间限制
     "chat_start_time": "00:00",  # 服务开始时间
@@ -129,22 +138,21 @@ available_setting = {
     "wechatcomapp_secret": "",  # 企业微信app的secret
     "wechatcomapp_agent_id": "",  # 企业微信app的agent_id
     "wechatcomapp_aes_key": "",  # 企业微信app的aes_key
-
     # 飞书配置
     "feishu_port": 80,  # 飞书bot监听端口
     "feishu_app_id": "",  # 飞书机器人应用APP Id
     "feishu_app_secret": "",  # 飞书机器人APP secret
     "feishu_token": "",  # 飞书 verification token
     "feishu_bot_name": "",  # 飞书机器人的名字
-    
     # 钉钉配置
     "dingtalk_client_id": "",  # 钉钉机器人Client ID 
-    "dingtalk_client_secret": "",  # 钉钉机器人Client Secret 
+    "dingtalk_client_secret": "",  # 钉钉机器人Client Secret
+    "dingtalk_card_enabled": False,
     
     # chatgpt指令自定义触发词
     "clear_memory_commands": ["#清除记忆"],  # 重置会话指令，必须以#开头
     # channel配置
-    "channel_type": "wx",  # 通道类型，支持：{wx,wxy,terminal,wechatmp,wechatmp_service,wechatcom_app}
+    "channel_type": "",  # 通道类型，支持：{wx,wxy,terminal,wechatmp,wechatmp_service,wechatcom_app,dingtalk}
     "subscribe_msg": "",  # 订阅消息, 支持: wechatmp, wechatmp_service, wechatcom_app
     "debug": False,  # 是否开启debug模式，开启后会打印更多日志
     "appdata_dir": "",  # 数据目录
@@ -152,18 +160,21 @@ available_setting = {
     "plugin_trigger_prefix": "$",  # 规范插件提供聊天相关指令的前缀，建议不要和管理员指令前缀"#"冲突
     # 是否使用全局插件配置
     "use_global_plugin_config": False,
-    "max_media_send_count": 3,     # 单次最大发送媒体资源的个数
+    "max_media_send_count": 3,  # 单次最大发送媒体资源的个数
     "media_send_interval": 1,  # 发送图片的事件间隔，单位秒
     # 智谱AI 平台配置
     "zhipu_ai_api_key": "",
     "zhipu_ai_api_base": "https://open.bigmodel.cn/api/paas/v4",
     "moonshot_api_key": "",
-    "moonshot_base_url":"https://api.moonshot.cn/v1/chat/completions",
+    "moonshot_base_url": "https://api.moonshot.cn/v1/chat/completions",
     # LinkAI平台配置
     "use_linkai": False,
     "linkai_api_key": "",
     "linkai_app_code": "",
     "linkai_api_base": "https://api.link-ai.tech",  # linkAI服务地址
+    "Minimax_api_key": "",
+    "Minimax_group_id": "",
+    "Minimax_base_url": "",
 }
 
 
@@ -224,6 +235,30 @@ class Config(dict):
 config = Config()
 
 
+def drag_sensitive(config):
+    try:
+        if isinstance(config, str):
+            conf_dict: dict = json.loads(config)
+            conf_dict_copy = copy.deepcopy(conf_dict)
+            for key in conf_dict_copy:
+                if "key" in key or "secret" in key:
+                    if isinstance(conf_dict_copy[key], str):
+                        conf_dict_copy[key] = conf_dict_copy[key][0:3] + "*" * 5 + conf_dict_copy[key][-3:]
+            return json.dumps(conf_dict_copy, indent=4)
+
+        elif isinstance(config, dict):
+            config_copy = copy.deepcopy(config)
+            for key in config:
+                if "key" in key or "secret" in key:
+                    if isinstance(config_copy[key], str):
+                        config_copy[key] = config_copy[key][0:3] + "*" * 5 + config_copy[key][-3:]
+            return config_copy
+    except Exception as e:
+        logger.exception(e)
+        return config
+    return config
+
+
 def load_config():
     global config
     config_path = "./config.json"
@@ -232,7 +267,7 @@ def load_config():
         config_path = "./config-template.json"
 
     config_str = read_file(config_path)
-    logger.debug("[INIT] config str: {}".format(config_str))
+    logger.debug("[INIT] config str: {}".format(drag_sensitive(config_str)))
 
     # 将json字符串反序列化为dict类型
     config = Config(json.loads(config_str))
@@ -257,13 +292,11 @@ def load_config():
         logger.setLevel(logging.DEBUG)
         logger.debug("[INIT] set log level to DEBUG")
 
-    logger.info("[INIT] load config: {}".format(config))
+    logger.info("[INIT] load config: {}".format(drag_sensitive(config)))
 
     config.load_user_datas()
 
 
-
-
 def get_root():
     return os.path.dirname(os.path.abspath(__file__))
 
@@ -315,6 +348,4 @@ def pconf(plugin_name: str) -> dict:
 
 
 # 全局配置，用于存放全局生效的状态
-global_config = {
-    "admin_users": []
-}
+global_config = {"admin_users": []}

docker/docker-compose.yml

@@ -6,6 +6,7 @@ services:
     security_opt:
       - seccomp:unconfined
     environment:
+      TZ: 'Asia/Shanghai'
       OPEN_AI_API_KEY: 'YOUR API KEY'
       MODEL: 'gpt-3.5-turbo'
       PROXY: ''

docs/version/old-version.md

@@ -8,6 +8,6 @@
 
 2023.03.25： 支持插件化开发，目前已实现 多角色切换、文字冒险游戏、管理员指令、Stable Diffusion等插件，使用参考 #578。(contributed by @lanvent in #565)
 
-2023.03.09： 基于 whisper API(后续已接入更多的语音API服务) 实现对微信语音消息的解析和回复，添加配置项 "speech_recognition":true 即可启用，使用参考 #415。(contributed by wanggang1987 in #385)
+2023.03.09： 基于 whisper API(后续已接入更多的语音API服务) 实现对语音消息的解析和回复，添加配置项 "speech_recognition":true 即可启用，使用参考 #415。(contributed by wanggang1987 in #385)
 
 2023.02.09： 扫码登录存在账号限制风险，请谨慎使用，参考#58

plugins/config.json.template

@@ -40,5 +40,22 @@
             "max_file_size": 5000,
             "type": ["FILE", "SHARING"]
         }
+    },
+    "hello": {
+        "group_welc_fixed_msg": {
+            "群聊1": "群聊1的固定欢迎语",
+            "群聊2": "群聊2的固定欢迎语"
+        },
+        "group_welc_prompt": "请你随机使用一种风格说一句问候语来欢迎新用户\"{nickname}\"加入群聊。",
+
+        "group_exit_prompt": "请你随机使用一种风格跟其他群用户说他违反规则\"{nickname}\"退出群聊。",
+
+        "patpat_prompt": "请你随机使用一种风格介绍你自己，并告诉用户输入#help可以查看帮助信息。",
+
+        "use_character_desc": false
+    },
+    "Apilot": {
+        "alapi_token": "xxx",
+        "morning_news_text_enabled": false
     }
 }

plugins/linkai/linkai.py

@@ -9,6 +9,7 @@ from common.expired_dict import ExpiredDict
 from common import const
 import os
 from .utils import Util
+from config import plugin_config
 
 
 @plugins.register(
@@ -69,7 +70,7 @@ class LinkAI(Plugin):
             return
 
         if (context.type == ContextType.SHARING and self._is_summary_open(context)) or \
-                (context.type == ContextType.TEXT and LinkSummary().check_url(context.content)):
+                (context.type == ContextType.TEXT and self._is_summary_open(context) and LinkSummary().check_url(context.content)):
             if not LinkSummary().check_url(context.content):
                 return
             _send_info(e_context, "正在为你加速生成摘要，请稍后")
@@ -196,7 +197,7 @@ class LinkAI(Plugin):
         if context.kwargs.get("isgroup") and not self.sum_config.get("group_enabled"):
             return False
         support_type = self.sum_config.get("type") or ["FILE", "SHARING"]
-        if context.type.name not in support_type:
+        if context.type.name not in support_type and context.type.name != "TEXT":
             return False
         return True
 
@@ -253,6 +254,7 @@ class LinkAI(Plugin):
                     plugin_conf = json.load(f)
                     plugin_conf["midjourney"]["enabled"] = False
                     plugin_conf["summary"]["enabled"] = False
+                    plugin_config["linkai"] = plugin_conf
                     return plugin_conf
         except Exception as e:
             logger.exception(e)

plugins/plugin.py

@@ -18,6 +18,7 @@ class Plugin:
         if not plugin_conf:
             # 全局配置不存在，则获取插件目录下的配置
             plugin_config_path = os.path.join(self.path, "config.json")
+            logger.debug(f"loading plugin config, plugin_config_path={plugin_config_path}, exist={os.path.exists(plugin_config_path)}")
             if os.path.exists(plugin_config_path):
                 with open(plugin_config_path, "r", encoding="utf-8") as f:
                     plugin_conf = json.load(f)

plugins/source.json

@@ -22,11 +22,23 @@
     },
     "pictureChange": {
       "url": "https://github.com/Yanyutin753/pictureChange.git",
-      "desc": "利用stable-diffusion和百度Ai进行图生图或者画图的插件"
+      "desc": "1. 支持百度AI和Stable Diffusion WebUI进行图像处理，提供多种模型选择，支持图生图、文生图自定义模板。2. 支持Suno音乐AI可将图像和文字转为音乐。3. 支持自定义模型进行文件、图片总结功能。4. 支持管理员控制群聊内容与参数和功能改变。"
     },
     "Blackroom": {
       "url": "https://github.com/dividduang/blackroom.git",
       "desc": "小黑屋插件，被拉进小黑屋的人将不能使用@bot的功能的插件"
+    },
+    "midjourney": {
+      "url": "https://github.com/baojingyu/midjourney.git",
+      "desc": "利用midjourney实现ai绘图的的插件"
+    },
+    "solitaire": {
+      "url": "https://github.com/Wang-zhechao/solitaire.git",
+      "desc": "机器人微信接龙插件"
+    },
+    "HighSpeedTicket": {
+      "url": "https://github.com/He0607/HighSpeedTicket.git",
+      "desc": "高铁（火车）票查询插件"
     }
   }
 }

voice/ali/ali_api.py

@@ -8,6 +8,7 @@ Description:
 
 """
 
+import http.client
 import json
 import time
 import requests
@@ -61,6 +62,69 @@ def text_to_speech_aliyun(url, text, appkey, token):
 
     return output_file
 
+def speech_to_text_aliyun(url, audioContent, appkey, token):
+    """
+    使用阿里云的语音识别服务识别音频文件中的语音。
+
+    参数:
+    - url (str): 阿里云语音识别服务的端点URL。
+    - audioContent (byte): pcm音频数据。
+    - appkey (str): 您的阿里云appkey。
+    - token (str): 阿里云API的认证令牌。
+
+    返回值:
+    - str: 成功时输出识别到的文本，否则为None。
+    """
+    format = 'pcm'
+    sample_rate = 16000
+    enablePunctuationPrediction  = True
+    enableInverseTextNormalization = True
+    enableVoiceDetection  = False
+
+    # 设置RESTful请求参数
+    request = url + '?appkey=' + appkey
+    request = request + '&format=' + format
+    request = request + '&sample_rate=' + str(sample_rate)
+
+    if enablePunctuationPrediction :
+        request = request + '&enable_punctuation_prediction=' + 'true'
+
+    if enableInverseTextNormalization :
+        request = request + '&enable_inverse_text_normalization=' + 'true'
+
+    if enableVoiceDetection :
+        request = request + '&enable_voice_detection=' + 'true'
+        
+    host = 'nls-gateway-cn-shanghai.aliyuncs.com'
+
+    # 设置HTTPS请求头部
+    httpHeaders = {
+        'X-NLS-Token': token,
+        'Content-type': 'application/octet-stream',
+        'Content-Length': len(audioContent)
+        }
+
+    conn = http.client.HTTPSConnection(host)
+    conn.request(method='POST', url=request, body=audioContent, headers=httpHeaders)
+
+    response = conn.getresponse()
+    body = response.read()
+    try:
+        body = json.loads(body)
+        status = body['status']
+        if status == 20000000 :
+            result = body['result']
+            if result :
+                logger.info(f"阿里云语音识别到了：{result}")
+            conn.close()
+            return result
+        else :
+            logger.error(f"语音识别失败，状态码: {status}")
+    except ValueError:
+        logger.error(f"语音识别失败，收到非JSON格式的数据: {body}")
+    conn.close()
+    return None
+
 
 class AliyunTokenGenerator:
     """

voice/ali/ali_voice.py

@@ -15,9 +15,9 @@ import time
 
 from bridge.reply import Reply, ReplyType
 from common.log import logger
+from voice.audio_convert import get_pcm_from_wav
 from voice.voice import Voice
-from voice.ali.ali_api import AliyunTokenGenerator
-from voice.ali.ali_api import text_to_speech_aliyun
+from voice.ali.ali_api import AliyunTokenGenerator, speech_to_text_aliyun, text_to_speech_aliyun
 from config import conf
 
 
@@ -34,7 +34,8 @@ class AliVoice(Voice):
             self.token = None
             self.token_expire_time = 0
             # 默认复用阿里云千问的 access_key 和 access_secret
-            self.api_url = config.get("api_url")
+            self.api_url_voice_to_text = config.get("api_url_voice_to_text")
+            self.api_url_text_to_voice = config.get("api_url_text_to_voice")
             self.app_key = config.get("app_key")
             self.access_key_id = conf().get("qwen_access_key_id") or config.get("access_key_id")
             self.access_key_secret = conf().get("qwen_access_key_secret") or config.get("access_key_secret")
@@ -53,7 +54,7 @@ class AliVoice(Voice):
                       r'äöüÄÖÜáéíóúÁÉÍÓÚàèìòùÀÈÌÒÙâêîôûÂÊÎÔÛçÇñÑ，。！？,.]', '', text)
         # 提取有效的token
         token_id = self.get_valid_token()
-        fileName = text_to_speech_aliyun(self.api_url, text, self.app_key, token_id)
+        fileName = text_to_speech_aliyun(self.api_url_text_to_voice, text, self.app_key, token_id)
         if fileName:
             logger.info("[Ali] textToVoice text={} voice file name={}".format(text, fileName))
             reply = Reply(ReplyType.VOICE, fileName)
@@ -61,6 +62,25 @@ class AliVoice(Voice):
             reply = Reply(ReplyType.ERROR, "抱歉，语音合成失败")
         return reply
 
+    def voiceToText(self, voice_file):
+        """
+        将语音文件转换为文本。
+
+        :param voice_file: 要转换的语音文件。
+        :return: 返回一个Reply对象，其中包含转换得到的文本或错误信息。
+        """
+        # 提取有效的token
+        token_id = self.get_valid_token()
+        logger.debug("[Ali] voice file name={}".format(voice_file))
+        pcm = get_pcm_from_wav(voice_file)
+        text = speech_to_text_aliyun(self.api_url_voice_to_text, pcm, self.app_key, token_id)
+        if text:
+            logger.info("[Ali] VoicetoText = {}".format(text))
+            reply = Reply(ReplyType.TEXT, text)
+        else:
+            reply = Reply(ReplyType.ERROR, "抱歉，语音识别失败")
+        return reply
+
     def get_valid_token(self):
         """
         获取有效的阿里云token。

voice/ali/config.json.template

@@ -1,5 +1,6 @@
 {
-    "api_url": "https://nls-gateway-cn-shanghai.aliyuncs.com/stream/v1/tts",
+    "api_url_text_to_voice": "https://nls-gateway-cn-shanghai.aliyuncs.com/stream/v1/tts",
+    "api_url_voice_to_text": "https://nls-gateway.cn-shanghai.aliyuncs.com/stream/v1/asr",
     "app_key": "",
     "access_key_id": "",
     "access_key_secret": ""

voice/baidu/baidu_voice.py

@@ -62,7 +62,7 @@ class BaiduVoice(Voice):
         # 识别本地文件
         logger.debug("[Baidu] voice file name={}".format(voice_file))
         pcm = get_pcm_from_wav(voice_file)
-        res = self.client.asr(pcm, "pcm", 8000, {"dev_pid": self.dev_id})
+        res = self.client.asr(pcm, "pcm", 16000, {"dev_pid": self.dev_id})
         if res["err_no"] == 0:
             logger.info("百度语音识别到了：{}".format(res["result"]))
             text = "".join(res["result"])

voice/factory.py

@@ -46,4 +46,8 @@ def create_voice(voice_type):
         from voice.edge.edge_voice import EdgeVoice
 
         return EdgeVoice()
+    elif voice_type == "xunfei":
+        from voice.xunfei.xunfei_voice import XunfeiVoice
+
+        return XunfeiVoice()
     raise RuntimeError

voice/xunfei/config.json.template

@@ -0,0 +1,7 @@
+{
+  "APPID":"xxx71xxx",
+  "APIKey":"xxxx69058exxxxxx",
+  "APISecret":"xxxx697f0xxxxxx",
+  "BusinessArgsTTS":{"aue": "lame", "sfl": 1, "auf": "audio/L16;rate=16000", "vcn": "xiaoyan", "tte": "utf8"},
+  "BusinessArgsASR":{"domain": "iat", "language": "zh_cn", "accent": "mandarin", "vad_eos":10000, "dwa": "wpgs"}
+}

voice/xunfei/xunfei_asr.py

@@ -0,0 +1,209 @@
+# -*- coding:utf-8 -*-
+#
+#  Author: njnuko 
+#  Email: njnuko@163.com 
+#
+#  这个文档是基于官方的demo来改的，固体官方demo文档请参考官网
+#
+#  语音听写流式 WebAPI 接口调用示例 接口文档（必看）：https://doc.xfyun.cn/rest_api/语音听写（流式版）.html
+#  webapi 听写服务参考帖子（必看）：http://bbs.xfyun.cn/forum.php?mod=viewthread&tid=38947&extra=
+#  语音听写流式WebAPI 服务，热词使用方式：登陆开放平台https://www.xfyun.cn/后，找到控制台--我的应用---语音听写（流式）---服务管理--个性化热词，
+#  设置热词
+#  注意：热词只能在识别的时候会增加热词的识别权重，需要注意的是增加相应词条的识别率，但并不是绝对的，具体效果以您测试为准。
+#  语音听写流式WebAPI 服务，方言试用方法：登陆开放平台https://www.xfyun.cn/后，找到控制台--我的应用---语音听写（流式）---服务管理--识别语种列表
+#  可添加语种或方言，添加后会显示该方言的参数值
+#  错误码链接：https://www.xfyun.cn/document/error-code （code返回错误码时必看）
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+import websocket
+import datetime
+import hashlib
+import base64
+import hmac
+import json
+from urllib.parse import urlencode
+import time
+import ssl
+from wsgiref.handlers import format_date_time
+from datetime import datetime
+from time import mktime
+import _thread as thread
+import os
+import wave
+
+
+STATUS_FIRST_FRAME = 0  # 第一帧的标识
+STATUS_CONTINUE_FRAME = 1  # 中间帧标识
+STATUS_LAST_FRAME = 2  # 最后一帧的标识
+
+#############
+#whole_dict 是用来存储返回值的，由于带语音修正，所以用dict来存储，有更新的化pop之前的值，最后再合并
+global whole_dict
+#这个文档是官方文档改的，这个参数是用来做函数调用时用的
+global wsParam
+##############
+
+
+class Ws_Param(object):
+    # 初始化
+    def __init__(self, APPID, APIKey, APISecret,BusinessArgs, AudioFile):
+        self.APPID = APPID
+        self.APIKey = APIKey
+        self.APISecret = APISecret
+        self.AudioFile = AudioFile
+        self.BusinessArgs = BusinessArgs
+        # 公共参数(common)
+        self.CommonArgs = {"app_id": self.APPID}
+        # 业务参数(business)，更多个性化参数可在官网查看
+        #self.BusinessArgs = {"domain": "iat", "language": "zh_cn", "accent": "mandarin", "vinfo":1,"vad_eos":10000}
+
+    # 生成url
+    def create_url(self):
+        url = 'wss://ws-api.xfyun.cn/v2/iat'
+        # 生成RFC1123格式的时间戳
+        now = datetime.now()
+        date = format_date_time(mktime(now.timetuple()))
+
+        # 拼接字符串
+        signature_origin = "host: " + "ws-api.xfyun.cn" + "\n"
+        signature_origin += "date: " + date + "\n"
+        signature_origin += "GET " + "/v2/iat " + "HTTP/1.1"
+        # 进行hmac-sha256进行加密
+        signature_sha = hmac.new(self.APISecret.encode('utf-8'), signature_origin.encode('utf-8'),
+                                 digestmod=hashlib.sha256).digest()
+        signature_sha = base64.b64encode(signature_sha).decode(encoding='utf-8')
+
+        authorization_origin = "api_key=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"" % (
+            self.APIKey, "hmac-sha256", "host date request-line", signature_sha)
+        authorization = base64.b64encode(authorization_origin.encode('utf-8')).decode(encoding='utf-8')
+        # 将请求的鉴权参数组合为字典
+        v = {
+            "authorization": authorization,
+            "date": date,
+            "host": "ws-api.xfyun.cn"
+        }
+        # 拼接鉴权参数，生成url
+        url = url + '?' + urlencode(v)
+        #print("date: ",date)
+        #print("v: ",v)
+        # 此处打印出建立连接时候的url,参考本demo的时候可取消上方打印的注释，比对相同参数时生成的url与自己代码生成的url是否一致
+        #print('websocket url :', url)
+        return url
+
+
+# 收到websocket消息的处理
+def on_message(ws, message):
+    global whole_dict
+    try:
+        code = json.loads(message)["code"]
+        sid = json.loads(message)["sid"]
+        if code != 0:
+            errMsg = json.loads(message)["message"]
+            print("sid:%s call error:%s code is:%s" % (sid, errMsg, code))
+        else:
+            temp1 = json.loads(message)["data"]["result"]
+            data = json.loads(message)["data"]["result"]["ws"]
+            sn = temp1["sn"]
+            if "rg" in temp1.keys():
+                rep = temp1["rg"]
+                rep_start = rep[0]
+                rep_end = rep[1]
+                for sn in range(rep_start,rep_end+1):
+                    #print("before pop",whole_dict)
+                    #print("sn",sn)
+                    whole_dict.pop(sn,None)
+                    #print("after pop",whole_dict)
+                results = ""
+                for i in data:
+                    for w in i["cw"]:
+                        results += w["w"]
+                whole_dict[sn]=results
+                #print("after add",whole_dict)
+            else:
+                results = ""
+                for i in data:
+                    for w in i["cw"]:
+                        results += w["w"]
+                whole_dict[sn]=results
+            #print("sid:%s call success!,data is:%s" % (sid, json.dumps(data, ensure_ascii=False)))
+    except Exception as e:
+        print("receive msg,but parse exception:", e)
+
+
+
+# 收到websocket错误的处理
+def on_error(ws, error):
+    print("### error:", error)
+
+
+# 收到websocket关闭的处理
+def on_close(ws,a,b):
+    print("### closed ###")
+
+
+# 收到websocket连接建立的处理
+def on_open(ws):
+    global wsParam
+    def run(*args):
+        frameSize = 8000  # 每一帧的音频大小
+        intervel = 0.04  # 发送音频间隔(单位:s)
+        status = STATUS_FIRST_FRAME  # 音频的状态信息，标识音频是第一帧，还是中间帧、最后一帧
+
+        with wave.open(wsParam.AudioFile, "rb") as fp:
+            while True:
+                buf = fp.readframes(frameSize)
+                # 文件结束
+                if not buf:
+                    status = STATUS_LAST_FRAME
+                # 第一帧处理
+                # 发送第一帧音频，带business 参数
+                # appid 必须带上，只需第一帧发送
+                if status == STATUS_FIRST_FRAME:
+                    d = {"common": wsParam.CommonArgs,
+                         "business": wsParam.BusinessArgs,
+                         "data": {"status": 0, "format": "audio/L16;rate=16000","audio": str(base64.b64encode(buf), 'utf-8'), "encoding": "raw"}} 
+                    d = json.dumps(d)
+                    ws.send(d)
+                    status = STATUS_CONTINUE_FRAME
+                # 中间帧处理
+                elif status == STATUS_CONTINUE_FRAME:
+                    d = {"data": {"status": 1, "format": "audio/L16;rate=16000",
+                                  "audio": str(base64.b64encode(buf), 'utf-8'),
+                                  "encoding": "raw"}}
+                    ws.send(json.dumps(d))
+                # 最后一帧处理
+                elif status == STATUS_LAST_FRAME:
+                    d = {"data": {"status": 2, "format": "audio/L16;rate=16000",
+                                  "audio": str(base64.b64encode(buf), 'utf-8'),
+                                  "encoding": "raw"}}
+                    ws.send(json.dumps(d))
+                    time.sleep(1)
+                    break
+                # 模拟音频采样间隔
+                time.sleep(intervel)
+        ws.close()
+
+    thread.start_new_thread(run, ())
+
+#提供给xunfei_voice调用的函数
+def xunfei_asr(APPID,APISecret,APIKey,BusinessArgsASR,AudioFile):
+    global whole_dict
+    global wsParam
+    whole_dict = {}
+    wsParam1 = Ws_Param(APPID=APPID, APISecret=APISecret,
+                       APIKey=APIKey,BusinessArgs=BusinessArgsASR,
+                       AudioFile=AudioFile)
+    #wsParam是global变量，给上面on_open函数调用使用的
+    wsParam = wsParam1
+    websocket.enableTrace(False)
+    wsUrl = wsParam.create_url()
+    ws = websocket.WebSocketApp(wsUrl, on_message=on_message, on_error=on_error, on_close=on_close)
+    ws.on_open = on_open
+    ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
+    #把字典的值合并起来做最后识别的输出
+    whole_words = ""
+    for i in sorted(whole_dict.keys()):
+        whole_words += whole_dict[i]
+    return whole_words
+
+     

voice/xunfei/xunfei_tts.py

@@ -0,0 +1,163 @@
+# -*- coding:utf-8 -*-
+#
+#  Author: njnuko
+#  Email: njnuko@163.com
+#
+#  这个文档是基于官方的demo来改的，固体官方demo文档请参考官网
+#
+#  语音听写流式 WebAPI 接口调用示例 接口文档（必看）：https://doc.xfyun.cn/rest_api/语音听写（流式版）.html
+#  webapi 听写服务参考帖子（必看）：http://bbs.xfyun.cn/forum.php?mod=viewthread&tid=38947&extra=
+#  语音听写流式WebAPI 服务，热词使用方式：登陆开放平台https://www.xfyun.cn/后，找到控制台--我的应用---语音听写（流式）---服务管理--个性化热词，
+#  设置热词
+#  注意：热词只能在识别的时候会增加热词的识别权重，需要注意的是增加相应词条的识别率，但并不是绝对的，具体效果以您测试为准。
+#  语音听写流式WebAPI 服务，方言试用方法：登陆开放平台https://www.xfyun.cn/后，找到控制台--我的应用---语音听写（流式）---服务管理--识别语种列表
+#  可添加语种或方言，添加后会显示该方言的参数值
+#  错误码链接：https://www.xfyun.cn/document/error-code （code返回错误码时必看）
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+import websocket
+import datetime
+import hashlib
+import base64
+import hmac
+import json
+from urllib.parse import urlencode
+import time
+import ssl
+from wsgiref.handlers import format_date_time
+from datetime import datetime
+from time import mktime
+import _thread as thread
+import os
+
+
+
+STATUS_FIRST_FRAME = 0  # 第一帧的标识
+STATUS_CONTINUE_FRAME = 1  # 中间帧标识
+STATUS_LAST_FRAME = 2  # 最后一帧的标识
+
+#############
+#这个参数是用来做输出文件路径的
+global outfile
+#这个文档是官方文档改的，这个参数是用来做函数调用时用的
+global wsParam
+##############
+
+
+class Ws_Param(object):
+    # 初始化
+    def __init__(self, APPID, APIKey, APISecret,BusinessArgs,Text):
+        self.APPID = APPID
+        self.APIKey = APIKey
+        self.APISecret = APISecret
+        self.BusinessArgs = BusinessArgs
+        self.Text = Text
+
+        # 公共参数(common)
+        self.CommonArgs = {"app_id": self.APPID}
+        # 业务参数(business)，更多个性化参数可在官网查看
+        #self.BusinessArgs = {"aue": "raw", "auf": "audio/L16;rate=16000", "vcn": "xiaoyan", "tte": "utf8"}
+        self.Data = {"status": 2, "text": str(base64.b64encode(self.Text.encode('utf-8')), "UTF8")}
+        #使用小语种须使用以下方式，此处的unicode指的是 utf16小端的编码方式，即"UTF-16LE"”
+        #self.Data = {"status": 2, "text": str(base64.b64encode(self.Text.encode('utf-16')), "UTF8")}
+
+    # 生成url
+    def create_url(self):
+        url = 'wss://tts-api.xfyun.cn/v2/tts'
+        # 生成RFC1123格式的时间戳
+        now = datetime.now()
+        date = format_date_time(mktime(now.timetuple()))
+
+        # 拼接字符串
+        signature_origin = "host: " + "ws-api.xfyun.cn" + "\n"
+        signature_origin += "date: " + date + "\n"
+        signature_origin += "GET " + "/v2/tts " + "HTTP/1.1"
+        # 进行hmac-sha256进行加密
+        signature_sha = hmac.new(self.APISecret.encode('utf-8'), signature_origin.encode('utf-8'),
+                                 digestmod=hashlib.sha256).digest()
+        signature_sha = base64.b64encode(signature_sha).decode(encoding='utf-8')
+
+        authorization_origin = "api_key=\"%s\", algorithm=\"%s\", headers=\"%s\", signature=\"%s\"" % (
+            self.APIKey, "hmac-sha256", "host date request-line", signature_sha)
+        authorization = base64.b64encode(authorization_origin.encode('utf-8')).decode(encoding='utf-8')
+        # 将请求的鉴权参数组合为字典
+        v = {
+            "authorization": authorization,
+            "date": date,
+            "host": "ws-api.xfyun.cn"
+        }
+        # 拼接鉴权参数，生成url
+        url = url + '?' + urlencode(v)
+        # print("date: ",date)
+        # print("v: ",v)
+        # 此处打印出建立连接时候的url,参考本demo的时候可取消上方打印的注释，比对相同参数时生成的url与自己代码生成的url是否一致
+        # print('websocket url :', url)
+        return url
+
+def on_message(ws, message):
+    #输出文件
+    global outfile
+    try:
+        message =json.loads(message)
+        code = message["code"]
+        sid = message["sid"]
+        audio = message["data"]["audio"]
+        audio = base64.b64decode(audio)
+        status = message["data"]["status"]
+        if status == 2:
+            print("ws is closed")
+            ws.close()
+        if code != 0:
+            errMsg = message["message"]
+            print("sid:%s call error:%s code is:%s" % (sid, errMsg, code))
+        else:
+
+            with open(outfile, 'ab') as f:
+                f.write(audio)
+
+    except Exception as e:
+        print("receive msg,but parse exception:", e)
+
+
+
+# 收到websocket连接建立的处理
+def on_open(ws):
+    global outfile
+    global wsParam
+    def run(*args):
+        d = {"common": wsParam.CommonArgs,
+             "business": wsParam.BusinessArgs,
+             "data": wsParam.Data,
+             }
+        d = json.dumps(d)
+        # print("------>开始发送文本数据")
+        ws.send(d)
+        if os.path.exists(outfile):
+            os.remove(outfile)
+
+    thread.start_new_thread(run, ())
+
+# 收到websocket错误的处理
+def on_error(ws, error):
+    print("### error:", error)
+
+
+
+# 收到websocket关闭的处理
+def on_close(ws):
+    print("### closed ###")
+
+
+
+def xunfei_tts(APPID, APIKey, APISecret,BusinessArgsTTS, Text, OutFile):
+    global outfile
+    global wsParam 
+    outfile = OutFile
+    wsParam1 = Ws_Param(APPID,APIKey,APISecret,BusinessArgsTTS,Text)
+    wsParam = wsParam1
+    websocket.enableTrace(False)
+    wsUrl = wsParam.create_url()
+    ws = websocket.WebSocketApp(wsUrl, on_message=on_message, on_error=on_error, on_close=on_close)
+    ws.on_open = on_open
+    ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
+    return outfile
+     

voice/xunfei/xunfei_voice.py

@@ -0,0 +1,86 @@
+#####################################################################
+#    xunfei voice service
+#     Auth: njnuko
+#     Email: njnuko@163.com
+#
+#    要使用本模块, 首先到 xfyun.cn 注册一个开发者账号,
+#    之后创建一个新应用, 然后在应用管理的语音识别或者语音合同右边可以查看APPID API Key 和 Secret Key
+#    然后在 config.json 中填入这三个值
+#
+#    配置说明：
+# {
+#  "APPID":"xxx71xxx",
+#  "APIKey":"xxxx69058exxxxxx",  #讯飞xfyun.cn控制台语音合成或者听写界面的APIKey
+#  "APISecret":"xxxx697f0xxxxxx",  #讯飞xfyun.cn控制台语音合成或者听写界面的APIKey
+#  "BusinessArgsTTS":{"aue": "lame", "sfl": 1, "auf": "audio/L16;rate=16000", "vcn": "xiaoyan", "tte": "utf8"}, #语音合成的参数，具体可以参考xfyun.cn的文档
+#  "BusinessArgsASR":{"domain": "iat", "language": "zh_cn", "accent": "mandarin", "vad_eos":10000, "dwa": "wpgs"}  #语音听写的参数，具体可以参考xfyun.cn的文档
+# }
+#####################################################################
+
+import json
+import os
+import time
+
+from bridge.reply import Reply, ReplyType
+from common.log import logger
+from common.tmp_dir import TmpDir
+from config import conf
+from voice.voice import Voice
+from .xunfei_asr import xunfei_asr
+from .xunfei_tts import xunfei_tts
+from voice.audio_convert import any_to_mp3
+import shutil
+from pydub import AudioSegment
+
+
+class XunfeiVoice(Voice):
+    def __init__(self):
+        try:
+            curdir = os.path.dirname(__file__)
+            config_path = os.path.join(curdir, "config.json")
+            conf = None
+            with open(config_path, "r") as fr:
+                conf = json.load(fr)
+            print(conf)
+            self.APPID = str(conf.get("APPID"))
+            self.APIKey = str(conf.get("APIKey"))
+            self.APISecret = str(conf.get("APISecret"))
+            self.BusinessArgsTTS = conf.get("BusinessArgsTTS")
+            self.BusinessArgsASR= conf.get("BusinessArgsASR")
+
+        except Exception as e:
+            logger.warn("XunfeiVoice init failed: %s, ignore " % e)
+
+    def voiceToText(self, voice_file):
+        # 识别本地文件
+        try:
+            logger.debug("[Xunfei] voice file name={}".format(voice_file))
+            #print("voice_file===========",voice_file)
+            #print("voice_file_type===========",type(voice_file))
+            #mp3_name, file_extension = os.path.splitext(voice_file)
+            #mp3_file = mp3_name + ".mp3"
+            #pcm_data=get_pcm_from_wav(voice_file)
+            #mp3_name, file_extension = os.path.splitext(voice_file)
+            #AudioSegment.from_wav(voice_file).export(mp3_file, format="mp3")
+            #shutil.copy2(voice_file, 'tmp/test1.wav')
+            #shutil.copy2(mp3_file, 'tmp/test1.mp3')
+            #print("voice and mp3 file",voice_file,mp3_file)
+            text = xunfei_asr(self.APPID,self.APISecret,self.APIKey,self.BusinessArgsASR,voice_file)
+            logger.info("讯飞语音识别到了: {}".format(text))
+            reply = Reply(ReplyType.TEXT, text)
+        except Exception as e:
+            logger.warn("XunfeiVoice init failed: %s, ignore " % e)
+            reply = Reply(ReplyType.ERROR, "讯飞语音识别出错了；{0}")
+        return reply
+
+    def textToVoice(self, text):
+        try:
+            # Avoid the same filename under multithreading
+            fileName = TmpDir().path() + "reply-" + str(int(time.time())) + "-" + str(hash(text) & 0x7FFFFFFF) + ".mp3"
+            return_file = xunfei_tts(self.APPID,self.APIKey,self.APISecret,self.BusinessArgsTTS,text,fileName)
+            logger.info("[Xunfei] textToVoice text={} voice file name={}".format(text, fileName))
+            reply = Reply(ReplyType.VOICE, fileName)
+        except Exception as e:
+            logger.error("[Xunfei] textToVoice error={}".format(fileName))
+            reply = Reply(ReplyType.ERROR, "抱歉，讯飞语音合成失败")
+        return reply