一.现象

公司kubenetes生产环境使用的ingress控制器是traefik，默认是通过deployment部署的，现在研发上反馈不能获取客户的真实源IP地址，通过x_forward_for获取的IP地址都是kubernetes集群内部的IP地址。

二.解决思路

通过查找traefik的官方文档

Traefik EntryPoints Documentation - Traefik

traefik传输客户源地址到后端需要配置Forwarded Headers参数，用于x_forwarded_for保存客户源IP地址

具体配置如下

 yaml格式，用于启用的时候指定配置文件

## Static configuration
entryPoints:web:address: ":80"forwardedHeaders:insecure: true

TOML格式，用于traefik启动的时候指定配置文件，这个文件在kubernetes中可以以comfigmap存在

## Static configuration
[entryPoints][entryPoints.web]address = ":80"[entryPoints.web.forwardedHeaders]insecure = true

命名行格式

--entryPoints.web.address=:80
--entryPoints.web.forwardedHeaders.insecure

三.问题处理

1.公司现有环境配置

---
apiVersion: apps/v1
kind: Deployment
metadata:annotations:meta.helm.sh/release-name: traefikmeta.helm.sh/release-namespace: defaultlabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: traefikhelm.sh/chart: traefik-9.11.0name: traefiknamespace: defaultresourceVersion: '505763774'
spec:progressDeadlineSeconds: 600replicas: 6revisionHistoryLimit: 10selector:matchLabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/name: traefikstrategy:rollingUpdate:maxSurge: 1maxUnavailable: 1type: RollingUpdatetemplate:metadata:creationTimestamp: nulllabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: traefikhelm.sh/chart: traefik-9.11.0spec:containers:- args:- '--global.checknewversion'- '--global.sendanonymoususage'- '--entryPoints.traefik.address=:9000/tcp'- '--entryPoints.web.address=:8000/tcp'- '--entryPoints.websecure.address=:8443/tcp'- '--api.dashboard=true'- '--ping=true'- '--providers.kubernetescrd'- '--providers.kubernetesingress'image: 'traefik:2.3.3'imagePullPolicy: IfNotPresentlivenessProbe:failureThreshold: 3httpGet:path: /pingport: 9000scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 2name: traefikports:- containerPort: 9000name: traefikprotocol: TCP- containerPort: 8000name: webprotocol: TCP- containerPort: 8443name: websecureprotocol: TCPreadinessProbe:failureThreshold: 1httpGet:path: /pingport: 9000scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 2resources: {}securityContext:capabilities:drop:- ALLreadOnlyRootFilesystem: truerunAsGroup: 65532runAsNonRoot: truerunAsUser: 65532terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /dataname: data- mountPath: /tmpname: tmpdnsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext:fsGroup: 65532serviceAccount: traefikserviceAccountName: traefikterminationGracePeriodSeconds: 60volumes:- emptyDir: {}name: data- emptyDir: {}name: tmp
status:availableReplicas: 6conditions:- lastTransitionTime: '2022-10-10T07:58:50Z'lastUpdateTime: '2022-10-10T07:58:50Z'message: Deployment has minimum availability.reason: MinimumReplicasAvailablestatus: 'True'type: Available- lastTransitionTime: '2020-11-25T22:53:59Z'lastUpdateTime: '2022-11-17T10:44:40Z'message: ReplicaSet "traefik-54bf67c74d" has successfully progressed.reason: NewReplicaSetAvailablestatus: 'True'type: ProgressingobservedGeneration: 13readyReplicas: 6replicas: 6updatedReplicas: 6---
apiVersion: v1
kind: Service
metadata:annotations:meta.helm.sh/release-name: traefikmeta.helm.sh/release-namespace: defaultlabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: traefikhelm.sh/chart: traefik-9.11.0name: traefiknamespace: defaultresourceVersion: '505762379'
spec:clusterIP: 10.96.252.108externalTrafficPolicy: Clusterports:- name: webnodePort: 30079port: 80protocol: TCPtargetPort: webselector:app.kubernetes.io/instance: traefikapp.kubernetes.io/name: traefiksessionAffinity: Nonetype: NodePort
status:loadBalancer: {}

发现trafik启动的参数里面没有指定配置文件，也没有以comfigmap形式存在，所以只能考虑通过添加启动参数的方式（即命令行方式）解决问题

2.解决问题

在deployment部署中的traefik启动参数中添加 - '--entrypoints.web.forwardedHeaders.insecure'和- '--entrypoints.websecure.forwardedHeaders.insecure'启动参数，其中标红的web和websecure根据--entryPoints.web.address=:8000/tcp和--entryPoints.websecure.address=:8443/tcp要保持一致，有可能不一样。

修改后的derloyment的yaml文件如下：

---
apiVersion: apps/v1
kind: Deployment
metadata:annotations:meta.helm.sh/release-name: traefikmeta.helm.sh/release-namespace: defaultlabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: traefikhelm.sh/chart: traefik-9.11.0name: traefiknamespace: defaultresourceVersion: '505763774'
spec:progressDeadlineSeconds: 600replicas: 6revisionHistoryLimit: 10selector:matchLabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/name: traefikstrategy:rollingUpdate:maxSurge: 1maxUnavailable: 1type: RollingUpdatetemplate:metadata:creationTimestamp: nulllabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: traefikhelm.sh/chart: traefik-9.11.0spec:containers:- args:- '--global.checknewversion'- '--global.sendanonymoususage'- '--entryPoints.traefik.address=:9000/tcp'- '--entryPoints.web.address=:8000/tcp'- '--entryPoints.websecure.address=:8443/tcp'- '--api.dashboard=true'- '--ping=true'- '--providers.kubernetescrd'- '--providers.kubernetesingress'- '--entrypoints.web.forwardedHeaders.insecure'- '--entrypoints.websecure.forwardedHeaders.insecure'image: 'traefik:2.3.3'imagePullPolicy: IfNotPresentlivenessProbe:failureThreshold: 3httpGet:path: /pingport: 9000scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 2name: traefikports:- containerPort: 9000name: traefikprotocol: TCP- containerPort: 8000name: webprotocol: TCP- containerPort: 8443name: websecureprotocol: TCPreadinessProbe:failureThreshold: 1httpGet:path: /pingport: 9000scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 2resources: {}securityContext:capabilities:drop:- ALLreadOnlyRootFilesystem: truerunAsGroup: 65532runAsNonRoot: truerunAsUser: 65532terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /dataname: data- mountPath: /tmpname: tmpdnsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext:fsGroup: 65532serviceAccount: traefikserviceAccountName: traefikterminationGracePeriodSeconds: 60volumes:- emptyDir: {}name: data- emptyDir: {}name: tmp
status:availableReplicas: 6conditions:- lastTransitionTime: '2022-10-10T07:58:50Z'lastUpdateTime: '2022-10-10T07:58:50Z'message: Deployment has minimum availability.reason: MinimumReplicasAvailablestatus: 'True'type: Available- lastTransitionTime: '2020-11-25T22:53:59Z'lastUpdateTime: '2022-11-17T10:44:40Z'message: ReplicaSet "traefik-54bf67c74d" has successfully progressed.reason: NewReplicaSetAvailablestatus: 'True'type: ProgressingobservedGeneration: 13readyReplicas: 6replicas: 6updatedReplicas: 6---
apiVersion: v1
kind: Service
metadata:annotations:meta.helm.sh/release-name: traefikmeta.helm.sh/release-namespace: defaultlabels:app.kubernetes.io/instance: traefikapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: traefikhelm.sh/chart: traefik-9.11.0name: traefiknamespace: defaultresourceVersion: '505762379'
spec:clusterIP: 10.96.252.108externalTrafficPolicy: Clusterports:- name: webnodePort: 30079port: 80protocol: TCPtargetPort: webselector:app.kubernetes.io/instance: traefikapp.kubernetes.io/name: traefiksessionAffinity: Nonetype: NodePort
status:loadBalancer: {}

3.验证测试

通过和研发和业务验证测试，通过x_forwarded_for获取的客户端源Ip地址格式如下

 其中第一个公网IP地址是客户的真实IP地址，其他几个是经过的k8s节点和pod的IP地址，至此解决业务上获取真实客户IP地址的需求。