可观测性是把指标、日志、链路、告警、通知串成一条能驱动故障响应的流水线。本文基于 Spring Cloud + Node Exporter + Prometheus + Grafana + Alertmanager + PrometheusAlert 搭建完整链路，覆盖机器监控、微服务监控、告警规则，以及飞书、电话、邮箱三类告警消息推送。
一、必要性
微服务架构下，一个请求可能跨越网关、认证、订单、库存、支付、积分等多个环节。实现全链路资源占用、进程状态、连接数、线程池、GC详情、系统日志、链路追踪的可视化监控与告警，对线上环境的问题发现、故障恢复至关重要。可观测性的目标，就是把这些指标变成可采集、可查询、可告警、可通知。
二、整体架构
各组件职责：
Spring Cloud Actuator + Micrometer：暴露 JVM、HTTP、Feign、熔断器、自定义业务指标
Node Exporter：采集主机 CPU、内存、磁盘、网络、进程等系统指标
Prometheus：Pull 指标、存储时序数据、执行 PromQL、评估告警规则
Grafana：数据可视化，导入 Dashboard 快速出图
Alertmanager：告警分组、抑制、静默、路由分发
PrometheusAlert：告警通知网关，将 Webhook 转为飞书/电话/邮箱等多渠道
三、Spring Cloud 微服务接入
添加依赖（每个微服务）
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-actuator</artifactId></dependency><dependency><groupId>io.micrometer</groupId><artifactId>micrometer-registry-prometheus</artifactId></dependency>配置 application.yml
以 order-service 为例：
spring:application:name:order-servicemanagement:endpoints:web:exposure:include:health,info,prometheus,metricsmetrics:tags:application:${spring.application.name}env:proddistribution:percentiles-histogram:http.server.requests:true# 启用直方图，支持 P99 计算启动后访问 http://localhost:8080/actuator/prometheus 即可看到 Prometheus 格式的指标输出。
多服务统一标签
@ConfigurationpublicclassMetricsConfig{@BeanpublicMeterRegistryCustomizer<MeterRegistry>metricsCommonTags(){returnregistry->registry.config().commonTags("cluster","prod-cn").commonTags("env","production");}}自定义业务指标
@ComponentpublicclassOrderMetrics{privatefinalCounterorderSuccessCounter;privatefinalCounterorderFailCounter;privatefinalTimerorderProcessTimer;publicOrderMetrics(MeterRegistryregistry){this.orderSuccessCounter=Counter.builder("order_success_total").tag("application","order-service").register(registry);this.orderFailCounter=Counter.builder("order_fail_total").tag("application","order-service").register(registry);this.orderProcessTimer=Timer.builder("order_process_duration_seconds").tag("application","order-service").publishPercentiles(0.5,0.95,0.99).register(registry);}publicvoidrecordSuccess(){orderSuccessCounter.increment();}publicvoidrecordFail(Stringreason){orderFailCounter.increment();}publicvoidrecordProcessTime(longmillis){orderProcessTimer.record(millis,TimeUnit.MILLISECONDS);}}Spring Cloud 自动暴露的关键指标
http_server_requests_* HTTP 请求量、延迟、状态码分布
jvm_memory_used_bytes JVM 堆/非堆内存
jvm_gc_pause_seconds_* GC 停顿时间
hikaricp_connections_* 数据库连接池状态
resilience4j_circuitbreaker_* 熔断器状态
feign_* Feign 远程调用指标
四、Prometheus 配置
prometheus/prometheus.yml
global:scrape_interval:15sevaluation_interval:15sexternal_labels:cluster:prod-cnenv:productionalerting:alertmanagers:-static_configs:-targets:["alertmanager:9093"]rule_files:-"rules/alerts.yml"scrape_configs:# Prometheus 自身-job_name:"prometheus"static_configs:-targets:["localhost:9090"]# 主机监控-job_name:"node-exporter"static_configs:-targets:["node-exporter:9100"]labels:env:prod# Spring Cloud 微服务-job_name:"spring-cloud-apps"metrics_path:"/actuator/prometheus"static_configs:-targets:-"host.docker.internal:8081"# gateway-"host.docker.internal:8082"# order-service-"host.docker.internal:8083"# pay-service-"host.docker.internal:8084"# user-servicelabels:env:prod如果微服务部署在独立服务器上，把 host.docker.internal 换成实际 IP。生产环境推荐使用 Nacos 自动发现。
prometheus/rules/alerts.yml
groups:#========== 主机告警 ==========-name:host-alertsrules:-alert:NodeDownexpr:up{job="node-exporter"}== 0for:1mlabels:severity:criticalannotations:summary:"节点 {{ $labels.instance }} 已宕机"description:"节点已停止响应超过 1 分钟"-alert:HighCPUexpr:100-(avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)>90for:5mlabels:severity:warningannotations:summary:"节点 {{ $labels.instance }} CPU 使用率 > 90%"-alert:HighMemoryexpr:(1-node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100>85for:5mlabels:severity:warningannotations:summary:"节点 {{ $labels.instance }} 内存使用率 > 85%"-alert:DiskFullexpr:(1-node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100>85for:5mlabels:severity:warningannotations:summary:"节点 {{ $labels.instance }} 磁盘使用率 > 85%"# ========== 微服务告警 ==========-name:microservice-alertsrules:-alert:ServiceDownexpr:up{job="spring-cloud-apps"}== 0for:1mlabels:severity:criticalannotations:summary:"微服务 {{ $labels.instance }} ({{ $labels.application }}) 已宕机"-alert:HighErrorRateexpr:|sum by(application) (rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum by(application) (rate(http_server_requests_seconds_count[5m])) > 0.05for:2mlabels:severity:criticalannotations:summary:"服务 {{ $labels.application }} 5xx 错误率 > 5%"-alert:HighLatencyexpr:|histogram_quantile(0.99, sum by(le, application) (rate(http_server_requests_seconds_bucket[5m])) ) > 2for:5mlabels:severity:warningannotations:summary:"服务 {{ $labels.application }} P99 延迟 > 2s"-alert:HighGCPauseexpr:rate(jvm_gc_pause_seconds_sum[5m])>0.5for:5mlabels:severity:warningannotations:summary:"服务 {{ $labels.application }} GC 停顿过高"-alert:ThreadPoolExhaustedexpr:hikaricp_connections_active / hikaricp_connections_max>0.9for:3mlabels:severity:warningannotations:summary:"服务 {{ $labels.application }} 数据库连接池使用率 > 90%"五、Alertmanager 配置
alertmanager/alertmanager.yml
global:resolve_timeout:5mroute:group_by:["alertname","env","application"]group_wait:30sgroup_interval:5mrepeat_interval:4hreceiver:"prometheusalert"routes:-matchers:-severity="critical"receiver:"prometheusalert"group_wait:10srepeat_interval:1h-matchers:-severity="warning"receiver:"prometheusalert"repeat_interval:4hreceivers:-name:"prometheusalert"webhook_configs:-url:"http://prometheusalert:8080/prometheusalert"send_resolved:trueinhibit_rules:-source_matchers:-severity="critical"target_matchers:-severity="warning"equal:["alertname","env","application"]核心机制：
分组：同组告警合并为一条通知，避免告警风暴
抑制：critical 触发时自动抑制同组 warning
静默：维护窗口期通过 UI 或 API 临时关闭告警
路由：按 severity/env/team 分发到不同接收者
六、PrometheusAlert 告警网关（飞书 + 电话 + 邮箱）
PrometheusAlert 是一个开源的告警通知网关，支持将 Prometheus/Alertmanager 的 Webhook 转为多种渠道。
prometheusalert/app.conf
[common]httpport = 8080 copyrequestbody = true title = PrometheusAlert prometheus_cst_time = 1飞书告警
[feishu]open-feishu= 1 fsurl = https://open.feishu.cn/open-apis/bot/v2/hook/YOUR_FEISHU_TOKENfsurl 替换成你自己的飞书机器人 Webhook。创建方式：飞书群设置 → 群机器人 → 添加机器人 → 自定义机器人 → 获取 Webhook 地址。
电话告警
PrometheusAlert 支持电话语音通知，需要在配置中启用电话通道并填写服务商凭证。以阿里云语音服务为例：
[phone]open-phone= 1#阿里云 AccessKeyphone-al-access-key-id = YOUR_ALIYUN_ACCESS_KEY_ID phone-al-access-key-secret = YOUR_ALIYUN_ACCESS_KEY_SECRET#语音通知模板 code（需在阿里云控制台创建）phone-al-tpl-id = YOUR_TPL_CODE#被叫号码，多个用逗号分隔phone-al-called-numbers = 13800138000,13900139000电话告警适用于 critical 级别的告警，如服务宕机、核心链路中断。
使用前需在阿里云控制台开通语音服务并创建语音模板，模板中可包含告警名称、级别、时间等变量。
邮箱告警
[email]open-email= 1#SMTP 服务器地址email-host = smtp.qq.com email-port = 465 email-user = your-email@qq.com email-password = YOUR_SMTP_AUTH_CODE#收件人，多个用逗号分隔email-to = ops-team@company.com,dev-lead@company.com#发件人显示名称email-from= PrometheusAlertemail-password 通常是邮箱服务商提供的授权码，而非登录密码。
QQ 邮箱在「设置 → 账户 → POP3/IMAP/SMTP」中开启 SMTP 服务后获取授权码。
多通道路由策略
在 PrometheusAlert 中，可以通过不同的 Webhook 端点实现通道选择。更推荐的做法是在 Alertmanager 中按 severity 路由到不同的 PrometheusAlert 端点：
route:group_by:["alertname","env"]group_wait:30sgroup_interval:5mrepeat_interval:4hreceiver:"feishu-default"routes:-matchers:-severity="critical"receiver:"phone-critical"group_wait:10srepeat_interval:30m-matchers:-severity="warning"receiver:"feishu-warning"repeat_interval:4h-matchers:-severity="info"receiver:"email-info"repeat_interval:12hreceivers:#飞书 - 默认通道-name:"feishu-default"webhook_configs:-url:"http://prometheusalert:8080/prometheusalert?type=fs"
