Kubernetes日志收集

1 安装loki

1.1 更新配置文件

1.1.1 配置集群参数

memberlist:
  join_members: ["loki-1", "loki-2", "loki-3"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
ingester:
  chunk_idle_period: 3m
  chunk_block_size: 262144
  chunk_retain_period: 1m
  max_transfer_retries: 0
  lifecycler:
    join_after: 60s
    observe_period: 5s
    ring:
      kvstore:
        store: memberlist
      replication_factor: 2

1.1.2 配置持久化

persistence:
  enabled: true
  accessModes:
  - ReadWriteOnce
  size: 300Gi
  annotations: {}

helm upgrade --install loki --namespace=loki . --set limitsConfig.ingestionBurstSizeMb=40 --set limitsConfig.ingestionRateMb=30 --set fluent-bit.enabled=false,promtail.enabled=true,grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=true,prometheus.server.persistentVolume.enabled=true

注意这里用promtail，promtail.enabled=true

我这里是测试环境，就没配置那么多参数，建议下载下来自定义values.yaml

2 安装promtail

helm pull grafana/promtail
查看loki的svc信息

解压编辑values.yaml

1，配置loki地址

lokiAddress: http://loki-headless:3100/api/v1/push

主要添加监控网关的日志

2，只收集业务命名空间(dev与test的日志，一定要放最后

1
2
3

- source_labels: [__meta_kubernetes_namespace]
  action: keep
  regex: test|dev

clip_image111

3，默认的标签太多，目前用不到，因为只看业务日志

pipelineStages:
  - cri: {}
common:
  - action: replace
    source_labels:
      - __meta_kubernetes_pod_node_name
    target_label: node_name
  - action: replace
    source_labels:
      - __meta_kubernetes_namespace
    target_label: namespace
  - action: replace
    source_labels:
      - __meta_kubernetes_pod_name
    target_label: pod
  - action: replace
    replacement: /var/log/pods/*$1/*.log
    separator: /
    source_labels:
      - __meta_kubernetes_pod_uid
      - __meta_kubernetes_pod_container_name
    target_label: __path__
  - action: replace
    replacement: /var/log/pods/*$1/*.log
    regex: true/(.*)
    separator: /
    source_labels:
      - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
      - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
      - __meta_kubernetes_pod_container_name
    target_label: __path__

scrapeConfigs: |
  - job_name: kubernetes-pods
    pipeline_stages:
      {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: drop
        regex: finance-gateway
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - source_labels: [__meta_kubernetes_namespace]
        action: keep
        regex: test|dev
      {{- if .Values.config.snippets.addScrapeJobLabel }}
      - replacement: kubernetes-pods
        target_label: scrape_job
      {{- end }}
      {{- toYaml .Values.config.snippets.common | nindent 4 }}
      {{- with .Values.config.snippets.extraRelabelConfigs }}
      {{- toYaml . | nindent 4 }}
      {{- end }}

4，因为上面挂载网关的日志了，这里不收集网关的pod日志

1
2
3

- source_labels: [__meta_kubernetes_pod_label_app]
  action: drop
  regex: finance-gateway

clip_image101

5，新加个收集gateway任务，注释部分是隐藏ip地址最后一位，不需要所以注释了

- job_name: access
#  pipeline_stages:
#  - replace:
#      expression: '(?:[0-9]{1,3}\.){3}([0-9]{1,3})'
#      replace: '***'
  static_configs:
    - targets:
        - localhost
      labels:
        job: nginx_access_log
        host: appfelstrudel
        agent: promtail
        __path__: /var/log/access/*.log
    - targets:
        - localhost
      labels:
        job: xiaobaihua
        __path__: /var/log/access/xiaobaihua-gateway.access.log
    - targets:
        - localhost
      labels:
        job: reapal
        __path__: /var/log/access/reapal-gateway.access.log