FastGPT Redis/BullMQ SigNoz 告警实战从稳定日志合同到生产级告警规则【免费下载链接】FastGPTFastGPT is a knowledge-based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.项目地址: https://gitcode.com/GitHub_Trending/fa/FastGPT本文基于 FastGPT 仓库中的 Redis/BullMQ 告警手册 signoz-redis-alerts.md完整讲解如何以 DAL 层输出的 OTel 日志字段为基础在 SigNoz 中配置覆盖连接层、队列层、业务 Cache 与限流层的 10 类 P1/P2 告警规则。读完本文你可以理解每条规则的 Filter、阈值、分组字段与降噪策略背后的失败语义并能在源码中定位每条告警消息的确切产生位置与测试保障。一、设计思想匹配稳定日志消息分组只用低基数字段1.1 告警可用的 OTel 字段FastGPT 的 Redis 告警不依赖异常堆栈而是以 DAL 层fastgpt/dal主动输出的、经过测试固定的日志消息为匹配锚点。SigNoz 侧可用字段如下SigNoz 字段用途service.name应用/进程边界severity_texterror、warn、info、debugbody.__log_message稳定日志消息告警的主要匹配字段body.*role、name等低基数 metadatacategoryLogTape category目前不同 Cache/Runtime 尚未统一其中error对象、完整 Redis URL、teamId、userId、sessionId、shareId、jobId、key等高基数字段只用于排查不得用于通知分组或模板——按这些字段分组会生成无限增长的通知组造成告警风暴。1.2 为什么不匹配 categoryRuntime 与 BullMQ logger 当前通常继承[system]业务 Cache 使用各自领域 category。在 category 统一之前上述规则默认不增加 category 条件否则会漏掉 Runtime、BullMQ 和多数 Cache 日志。基础设施 category 统一后的目标状态见本文第五节。1.3 消息稳定性的测试保障日志消息不是随手写出的字符串DAL 的单元测试逐条断言了告警所依赖的消息文本。例如dailyActiveDedupe.test.ts 断言Daily active dedupe failed openlease.test.ts 断言Redis lease renew failedsession.test.ts 断言Invalid Redis session recordteamPoint.test.ts 断言Failed to get team point cacheworkflowStopSignal.test.ts 断言Workflow stop signal read failed openbullmq-services.test.ts 断言Failed to push collection update jobstreamResume.test.ts 断言Failed to mirror stream response to redis。这意味着“重命名日志消息”在仓库内会先被测试拦截也为手册中“消息重命名必须同步更新对应规则”的约束提供了工程基础。二、使用前提service.name告警示例统一使用service.name fastgpt-cn-client实际值由 OTEL service name 环境变量决定。packages/service/env.ts 中定义了LOG_OTEL_SERVICE_NAME、METRICS_OTEL_SERVICE_NAME、TRACING_OTEL_SERVICE_NAME等变量默认值均为fastgpt-client不同环境部署可将其设置为各自的应用名如示例中的fastgpt-cn-client。App 与 Pro 使用不同 service name 时分别创建规则不要为了复用查询合并两个进程——合并会掩盖单侧进程的故障边界也使service.name分组失去区分能力。三、通用配置所有规则共享以下默认配置参数默认值Evaluation windowRolling 5 分钟Evaluation frequency1 分钟No-dataNo alert / OKMatch typeat least onceRepeat notificationP1 15 分钟P2 60 分钟推荐 Group Bybody.__log_message、body.role、body.name、service.name发布、Redis 维护和故障演练使用 SigNoz maintenance window 或通知路由静默不修改 Filter 绕过一次发布。四、连接层规则R1–R3连接层日志统一由进程级 Redis Runtime 输出。connection.ts 中的RedisRuntime按command/blocking/queue/worker四种角色管理连接每个 ioredis 事件都映射为一条带rolemetadata 的稳定日志。R1. Redis 连接错误P1Filterservice.name fastgpt-cn-client AND severity_text error AND body.__log_message Redis connection errorAggregatecount()Group Bybody.role、service.name。阈值同一 role 5 分钟内 3。处理检查 Redis 实例、网络、认证和连接数并关联 R3/R4。源码依据connection.ts 在error事件上记录Redis connection error并附带role同时上报connection.errormetric。由于按role分组可以区分是命令连接、阻塞连接还是 BullMQ 队列/Worker 连接出问题。R2. Redis 进程关闭失败P1Filterservice.name fastgpt-cn-client AND severity_text error AND body.__log_message Redis runtime close failed after process shutdown signalAggregatecount()Group Byservice.name。阈值5 分钟内 1。处理关联部署信号、进程退出码、Next.js drain 和 BullMQ close 日志。源码依据shutdown.ts 注册了 SIGTERM/SIGINT 处理信号触发后执行runtime.close()成功路径输出Redis runtime closed after process shutdown signalinfo进程退出码 0失败路径输出本规则匹配的错误消息进程退出码 1。因此该消息天然与发布/缩容动作强相关单条即 P1处理时应先看部署平台侧的信号与退出码。R3. Redis 重连抖动P2Filterservice.name fastgpt-cn-client AND severity_text warn AND ( body.__log_message Redis connection closed OR body.__log_message Redis reconnect scheduled OR body.__log_message Redis reconnect requested by command error )Aggregatecount()Group Bybody.role、service.name。阈值同一 role 5 分钟内 5单条 close 不告警。处理R1 同时触发时按连接故障处理否则检查 failover、网络抖动和连接压力。源码依据Redis connection closed由 connection.ts 的close事件输出Redis reconnect scheduled与Redis reconnect requested by command error由 policy.ts 在重连策略层输出。阈值 5的降噪意图是偶发 failover 切换会产生 12 条 closereconnect只有持续抖动才值得通知。五、队列层规则R4–R5R4. BullMQ Queue/Worker 错误P1Filterservice.name fastgpt-cn-client AND severity_text error AND ( body.__log_message BullMQ queue error OR body.__log_message BullMQ worker error OR body.__log_message BullMQ worker restart failed, will retry )Aggregatecount()Group Bybody.name、service.name。阈值同一 queue/worker 5 分钟内 3。处理先排除 R1Redis 正常时检查 processor、job payload 和 BullMQ 版本。源码依据BullMQ queue errorqueue-manager.tsQueue 的error事件处理器namemetadata 即队列名。BullMQ worker errorworker-manager.ts。BullMQ worker restart failed, will retryworker-manager.ts 中 Worker 关闭后自动重启循环的失败分支——Worker 意外关闭会触发带restartDelayMs的重试连续重启失败才会打出这条 error。R5. BullMQ 生命周期异常P2非部署窗口可提升 P1Filter 消息8 条叠加severity_text warn用 OR 匹配body.__log_messageBullMQ worker closed, attempting restart BullMQ worker resume failed BullMQ queue close failed BullMQ worker close failed BullMQ queue connection forced disconnect failed BullMQ worker connection forced disconnect failed Failed to release Redis connection after BullMQ queue creation error Failed to release Redis connection after BullMQ worker creation errorAggregatecount()Group Bybody.name、body.__log_message。阈值5 分钟内 3close/forced-disconnect failure 可单独设置 1。降噪部署期间静默单条worker closed不代表故障。源码依据BullMQ worker closed, attempting restart与BullMQ worker resume failedworker-manager.ts前者是closed事件在 running 状态下触发自动重启的日志后者是paused后延迟 resume 失败。BullMQ queue close failedqueue-manager.tsBullMQ worker close failedworker-manager.ts两者均在带超时的close()失败后降级为强制断连。两条Failed to release Redis connection after BullMQ ... creation errorQueue/Worker 创建抛错时释放底层连接的兜底日志见 queue-manager.ts 与 worker-manager.ts。从源码结构看... forced disconnect failed由关闭兜底模块 close.ts 的forceDisconnect统一输出对应 queue/worker 主连接与 worker blocking 连接两类资源。六、业务 Cache 规则R6R6. Redis Cache 持续降级P2业务 Cache 的降级路径在 DAL 内刻意区分了 fail-open业务放行、记 warn与硬失败记 error这些消息共同构成“Redis 正在影响业务但连接层尚未全面故障”的信号Daily active dedupe failed open DingTalk accessToken cache read failed DingTalk accessToken cache write failed Invalid Redis session record Failed to delete invalid Redis session record Redis lease renew failed Redis lease acquire failed Redis lease release failed Failed to get team point cache Failed to set team point cache Failed to increment team point cache Failed to clear team point cache Workflow stop signal read failed open Workflow stop signal clear failed Failed to get team vector count cache Failed to set team vector count cache Failed to invalidate team vector count cacheFilter 叠加severity_text IN (error, warn)用 OR 匹配上述消息。Aggregatecount()Group Bybody.__log_message、service.name。阈值5 分钟内 10session 损坏或 lease acquire failure 可设 3。处理R1 未触发时检查单一 Cache 的数据格式、TTL 和降级路径。源码抽查印证了这些消息的落点例如Daily active dedupe failed opendailyActiveDedupe.tsRedis 读失败时选择放行failed open仅影响去重统计。Redis lease renew failed/Redis lease renew failed because token no longer matcheslease.ts租约续约是互斥与锁语义的基础因此可设更低的 3阈值。Invalid Redis session recordsession.ts属于数据损坏信号。Workflow stop signal read failed openworkflowStopSignal.ts读失败时无法确认工作流停止信号fail-open 会让停止操作可能失效。Failed to get team point cacheteamPoint.ts。特别说明Skipped invalid team point cache valueteamPoint.ts属于数据质量信号表示某条缓存值非法被跳过不代表 Redis 可用性问题不并入本告警。七、流恢复规则R7R7. Stream Resume 镜像失败P1内存压力状态变化为 P2FastGPT 支持会话流式响应中断后从 Redis 镜像恢复。P1 消息恢复链路硬失败Failed to clear stream resume redis keys before mirror Failed to mirror stream response to redis Failed to shrink stream resume redis ttl阈值合计 5 分钟内 5恢复能力属于核心 SLA 时可低频通知单次失败。P2 消息状态/检查类Disabling new stream resume mirrors due to Redis memory pressure Failed to inspect Redis memory pressure for stream resume mirror Failed to persist stream resume unavailable state cleanStaleGeneratingChats: failed to inspect stream resume activity阈值内存压力状态变化 1/10 分钟检查或状态写入失败 3/5 分钟。源码依据镜像写入与 key 清理失败日志在 streamResume.ts 中如Failed to mirror stream response to redis内存压力导致禁用新镜像的日志由 Service 层 resume.ts 输出说明该规则跨越 DAL 与 service 两层分组只用body.__log_message和service.name才能兼容两层日志。八、限流规则R8R8. 限流服务 Fail-closedP1限流是少数 Redis 故障直接转换为用户可见 429的路径因此定为 P1Filterservice.name fastgpt-cn-client AND severity_text error AND ( body.__log_message Fixed window rate limit failed closed OR body.__log_message Team QPM configuration lookup failed closed OR body.__log_message Team QPM rate limit failed closed )Aggregatecount()Group Bybody.__log_message、body.type。阈值5 分钟内 1高流量环境可使用 3。禁止按 teamId 或 key 分组。源码依据frequencyLimit.ts 中QPM 配置查询失败与限流计数失败都会打出对应的failed closed日志并立即返回code: 429Rate limit service unavailable。底层限流实现为固定窗口算法计数原子性由 DAL 的 rateLimit.ts 保证且该 Cache 的注释明确“Redis 执行错误向上抛出由 Service 层按场景映射故障策略”——这正是 fail-closed 语义的出处。需要说明的是当前仓库源码中可确认的是两条 Team QPM 消息Fixed window rate limit failed closed属于限流 Cache 层 fail-closed 场景的预留文案在配置规则时保留该分支不会造成误报。九、Pro 专属规则R9R9. Pro 企微订单 Cache 故障P1由支付错误率决定是否降为 P2Filterservice.name fastgpt-cn-client AND severity_text error AND ( body.__log_message Failed to get WeChat Work pending order cache OR body.__log_message Failed to set WeChat Work pending order cache OR body.__log_message Failed to clean WeChat Work pending order cache )Aggregatecount()Group Bybody.__log_message、service.name。阈值5 分钟内 1只关注持续故障时使用 3。禁止按 teamId 或 orderId 分组。按 DAL 设计文档 的所有权约定Pro 专属 Cache 保留在pro/admin/src/dal因此该规则仅适用于部署 Pro 侧企业微信/支付能力的环境订单缓存读写失败会直接打断“下单-轮询-回调”的支付闭环故阈值比 R6 更严格。十、业务任务规则R10R10. BullMQ 业务任务失败这类错误来自各队列的 service 层封装不能单独证明 Redis 不可用应与 R1/R4 分开配置Failed to push collection update job Failed to update collection Failed to check evaluation job status Failed to remove evaluation job Failed to enqueue skill creation job, marked skill creation failed Failed to resume pending skill creation jobs Failed to resume marked skill delete jobs Reply job failed Wechat getUpdates request failed getUpdates API error Schedule next poll (completed) failed Schedule next poll (failed) failed严重度和阈值按业务 SLA默认 3/5 分钟。Group By 使用稳定消息或业务 categoryshareId只能做去重计数不能形成通知组。R1/R4 未触发时按 processor、第三方 API 或 job payload 排查。源码落点均在 packages/dal/redis/bullmq/services/ 下Failed to push collection update job见 collectionUpdate.tsFailed to check evaluation job status见 evaluation.tsWechat getUpdates request failed属于第三方 API 拉取失败落在 Service 层 wechat/mq.ts。这类日志混有“Redis 推任务失败”和“任务处理器本身失败”两种根因所以必须与连接层规则解耦避免 Redis 恢复后仍被业务侧误判为缓存故障。十一、Dashboard 边界哪些只画曲线不通知以下消息属于正常生命周期与恢复信号只用于 dashboard 或趋势不直接通知Redis connection established Redis connection ready Redis runtime closed after process shutdown signal BullMQ worker ready BullMQ worker restarted successfully BullMQ worker paused Collection update job pushed Dataset sync scheduler reconcile finished Redis memory pressure recovered; stream resume mirror creation resumed幂等补偿、资源已不存在和 active job 无法删除等 warning 也属于业务 dashboard。只有它们与 R1/R4 或明确业务 SLA 同时满足时才升级。其中前四条可在 connection.ts 的connect/ready事件与 shutdown.ts 的正常关闭路径中找到对应输出BullMQ worker ready/restarted successfully/paused则对应 worker-manager.ts 的生命周期事件——把它们保留在曲线上恰好可以反向验证 R1R5 触发与否。十二、维护约束消息重命名与 category 统一日志消息重命名必须同步更新对应 SigNoz 规则且仓库内 DAL 测试会先拦截这类改动本文档刻意不维护一份与源码重复的全量消息清单规则以 signoz-redis-alerts.md 为唯一事实来源。基础设施 category 统一后Redis Runtime/health/shutdown 使用LogCategories.INFRA.REDISBullMQ Runtime 使用LogCategories.INFRA.QUEUE业务 Cache 保留领域 category并额外提供低基数component字段。在 SigNoz 中确认 category 的实际存储类型之前不批量修改现有规则避免规则静默失效。小结FastGPT 的 Redis 告警体系可以概括为四层防线连接层R1–R3回答“Redis 还能不能连上”队列层R4–R5回答“任务链路是不是断的”业务 Cache 层R6、R7、R8回答“降级是否已经影响用户”Pro/业务任务层R9、R10回答“具体业务闭环是否受损”。所有规则共享同一套低基数分组纪律与 5 分钟滚动窗口并通过维护窗口而非修改 Filter 的方式应对发布与演练。由于每条告警消息都有源码落点和单元测试钉住这套规则具备可审计、可回归的长期可维护性。【免费下载链接】FastGPTFastGPT is a knowledge-based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.项目地址: https://gitcode.com/GitHub_Trending/fa/FastGPT创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考