grafana导入etcd监控界面展示地址URL：

https://grafana.com/grafana/dashboards/9733-etcd-for-k8s-cn/

1.2云原生 kube-proxy 监控

说明：kube-proxy默认暴露两个端口，一个是10249，在/metrics端点暴露监控指标，另一个是10256，默认监听127.0.0.1

1.2.1查看端口

ss -tunlp | grep kube-proxy

测试能否访问资源

curl 127.0.0.1:10249/metrics

1.2.2修改监听地址

说明：修改监听端口，监听所有端口。端口为本地，Prometheus如果没有部署在这台机子上，就无法监听到该端口。

kubectl edit configmap kube-proxy -n kube-system

添加以下内容：

metricsBindAddress: "0.0.0.0:10249"

重启Pod

说明：kube-proxy采用副本的方式构建，一旦删除，会立即重建

kubectl -n kube-system get pod|grep kube-proxy|awk '{system("kubectl -n kube-system delete pod "$1"")}' ss -tunlp | grep kube-proxy

1.2.3创建kube-proxy Service

vim kube-proxy-service.yaml

添加以下内容：

--- apiVersion: v1 kind: Service metadata: labels: k8s-app: kube-proxy name: kube-proxy namespace: kube-system spec: selector: k8s-app: kube-proxy #筛选标签 type: ClusterIP ports: - name: https-metrics port: 10249 targetPort: 10249 protocol: TCP

运行yaml

kubectl apply -f kube-proxy-service.yaml kubectl get svc kube-proxy -n kube-system

尝试抓取服务：

curl 10.110.8.62:10249/metrics

1.2.4创建kube-proxy servicemonitor 资源

1.编写yaml文件

vim kube-proxy-servicemonitor.yaml

添加以下内容：

--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-proxy namespace: monitoring labels: app: kube-proxy #Service Monitor标签 spec: jobLabel: kube-proxy endpoints: - interval: 30s port: https-metrics scheme: http selector: matchLabels: k8s-app: kube-proxy #匹配标签 namespaceSelector: matchNames: - kube-system

2.运行yaml文件

kubectl apply -f kube-proxy-servicemonitor.yaml kubectl get servicemonitor kube-proxy -n monitoring

1.2.5查看监控数据

1.2.6grafana监控界面展示

说明：prometheus本身自带granfan界面

1.3云原生 calico 网络插件监控

说明：Calico 中最核心的组件就是Felix，它负责设置路由表和 ACL规则等，以便为该主机上的 endpoints 资源正常运行提供所需的网络连接。同时它还负责提供有关网络健康状况的数据（例如，报告配置其主机时发生的错误和问题），这些数据会被写入etcd，以使其对网络中的其他组件和操作人员可见。由此可见，监控calico的核心便是监控felix，felix相当于calico的大脑。

1.3.1开启 Metrics 端口

说明：calico在启动时，默认没有打开Metrics监听

kubectl edit ds calico-node -n kube-system

添加以下内容：

spec: containers: - env: - name: FELIX_PROMETHEUSMETRICSENABLED value: "True" - name: FELIX_PROMETHEUSMETRICSPORT value: "9091" ports: - containerPort: 9091 name: http-metrics protocol: TCP

1.3.2创建podMonitor对象

说明：ServiceMonitor 通过 Service 发现目标，PodMonitor 直接选 Pod。前者适合有 Service 的服务，后者适合没有 Service 或不想建 Service 的场景。

1.创建yml文件

vim prometheus-podMonitorCalico.yaml

添加以下内容：

--- apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: labels: k8s-app: calico-node name: calico-node namespace: monitoring spec: podMetricsEndpoints: - interval: 15s path: /metrics port: http-metrics namespaceSelector: matchNames: - kube-system selector: matchLabels: k8s-app: calico-node #匹配标签

2.运行yml

kubectl apply -f prometheus-podMonitorCalico.yaml kubectl get podmonitor calico-node -n monitoring

1.3.3查看监控数据

1.3.4grafana监控界面展示

https://grafana.com/grafana/dashboards/12175-felix-dashboard-calico/

1.4云原生nginx-ingress监控

1.4.1开启 Metrics 端口

说明：helm部署时默认没有开启metrics采集服务

vim ingress-nginx.yml

添加以下内容：

global: image: registry: reg.westos.org controller: image: image: ingress-nginx/controller tag: "v1.13.3" digest: "" digestChroot: "" ingressClassResource: name: nginx default: true service: type: LoadBalancer admissionWebhooks: patch: image: registry: reg.westos.org image: ingress-nginx/kube-webhook-certgen tag: v1.6.3 digest: "" metrics: #开启metrics采集 enabled: true service: enabled: true labels: app: "ingress-nginx-metrics" serviceMonitor: enabled: true namespace: "monitoring" defaultBackend: enabled: true name: defaultbackend image: registry: reg.westos.org image: ingress-nginx/defaultbackend-amd64 tag: "1.5"

2.使用helm部署Ingress-nginx

helm -n ingress-nginx upgrade ingress-nginx -f ingress-nginx.yml ingress-nginx-4.13.3.tgz kubectl -n ingress-nginx get svc ingress-nginx-controller-metrics

测试

curl 10.107.130.97:10254/metrics

1.4.2修改service monitor 资源

说明：helm部署service monitor，修改匹配标签

kubectl -n monitoring edit servicemonitors ingress-nginx-controller

添加以下内容：

spec: endpoints: - interval: 30s port: metrics scheme: http namespaceSelector: matchNames: - ingress-nginx selector: matchLabels: app: ingress-nginx-metrics

kubectl apply -f rbac.yml

文件内容如下：

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: ingress-nginx rules: - apiGroups: [""] resources: ["endpoints", "pods", "services"] verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: ingress-nginx subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io

1.4.3查看监控数据

1.4.4grafana监控界面展示

https://grafana.com/grafana/dashboards/21336-nginx-ingress-controller/

二.非云原生应用监控

说明：非云原生应用不能使用metric暴露服务端口，需要使用外部的 exporter暴露

2.1 Linux 主机监控

说明：通过kube-prometheus技术栈安装的prometheus默认会以ds的形式在每一个节点上启动一个node-exporter pod，并且会监控一个9100端口。该端口用来采集宿主机的相关监控指标。

2.1.1检查暴露服务

ss -tunlp | grep 9100 kubectl get svc node-exporter -n monitoring -o yaml kubectl get servicemonitor node-exporter -n monitoring -o yaml

2.1.2查看监控数据

2.1.3grafana监控界面展示

默认面板：

中文面板

13105

2.2非云原生应用 Redis 集群监控

2.2.1部署redis集群

1.编辑配置文件

vim redis-values-metrics.yml

添加以下内容：

global: imageRegistry: "reg.westos.org" security: allowInsecureImages: true redis: password: "westos" metrics: enabled: true serviceMonitor: enabled: true namespace: "monitoring"

2.上传镜像

docker pull bitnami/redis-exporter docker tag bitnami/redis-exporter reg.westos.org/bitnami/redis-exporter

3.部署redis集群

helm install redis-cluster -f redis-values-metrics.yml redis-23.2.6.tgz -n redis kubectl get all -n redis

测试是否暴露服务：

curl 10.103.174.171:9121/metrics

2.2.2配置rbac权限

vim rbac.yml

添加以下内容：

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: prometheus-endpoints-view namespace: redis rules: - apiGroups: [""] resources: ["endpoints", "pods", "services"] verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: ["endpointslices"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: prometheus-k8s-endpoints namespace: redis subjects: - kind: ServiceAccount name: prometheus-k8s namespace: monitoring roleRef: kind: Role name: prometheus-endpoints-view apiGroup: rbac.authorization.k8s.io

运行yml

kubectl apply -f rbac.yml

2.2.3查看监控数据

2.2.4grafana监控界面展示

763

2.3非云原生应用 NGINX 监控

说明：nginx有自己的监控模块stub_status。

2.3.1编辑configmap文件

vim default.conf

添加以下内容：

server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; index index.html index.htm; } location /status { stub_status on; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } }

运行yml文件

kubectl create cm nginx-conf --from-file=./default.conf

2.2.2上传nginx-exporter镜像

docker pull nginx/nginx-exporter docker tsg nginx/nginx-exporter reg.westos.org/library/nginx-exporter docker push reg.westos.org/library/nginx-exporter

2.3.3部署Nginx资源

vim deploy-nginx.yml

添加以下内容：

--- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server spec: selector: matchLabels: app: nginx replicas: 3 template: metadata: labels: app: nginx annotations: prometheus.io/scrape: 'true' prometheus.io/port: '9113' spec: containers: - name: nginx image: reg.westos.org/nginx/nginx:1.29 ports: - containerPort: 80 volumeMounts: - name: nginxconf mountPath: /etc/nginx/conf.d - name: nginx-exporter image: reg.westos.org/nginx/nginx-prometheus-exporter:1.5.1 args: - '-nginx.scrape-uri=http://localhost/status' resources: limits: memory: 128Mi cpu: 500m ports: - containerPort: 9113 volumes: - name: nginxconf configMap: name: nginx-conf

kubectl apply -f deploy-nginx.yml kubectl get pod

2.3.4创建service

vim service-nginx.yml

添加以下内容：

--- apiVersion: v1 kind: Service metadata: name: webapp labels: app: nginx spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 name: web - protocol: TCP port: 9113 targetPort: 9113 name: metrics

运行yml文件

kubectl apply -f service-nginx.yml kubectl get svc webapp

查看抓取数据

curl 10.108.234.20/status curl 10.108.234.20:9113

2.3.5创建servicemonitor

vim nginx-servicemonitor.yml

添加以下内容：

--- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: nginx-exporter namespace: monitoring labels: k8sapp: nginx-exporter namespace: monitoring spec: jobLabel: k8s-app endpoints: - port: metrics interval: 30s scheme: http path: /metrics selector: matchLabels: app: nginx namespaceSelector: matchNames: - default

运行yml文件

kubectl apply -f nginx-servicemonitor.yml

2.3.6查看监控数据

2.3.7grafana监控界面展示

https://github.com/nginx/nginx-prometheus-exporter/blob/main/grafana/dashboard.json