Istio mTLS

istio 配置 mtls

Posted by Luffyao on Friday, January 10, 2020

ISTIO MTLS SUMMARY

THIS IS BASED ON ISTIO OFFICIAL PACKAGE ISTIO-1.3.5(FOLLOWING ALL OF ISTIO DOCS MUST OVER THE WALL,YOU CAN REFERENCE THE LATEST VERDION DOCS), BECAUSE OUR ENV VERSION IS BASED ON THIS VERSION.

Test Flow

TODO

Update MeshPolicy from PERMISSIVE to STRICT

NOTE: More infor about this in our ENV, please look at following Question section Q2

Reference link MutlalTls-Mode

kubectl get meshpolicy
kubectl edit meshpolicy


///// then manually update mtls.mode to STRICT

Generate CA ,key and cert

You can reference this link secure-ingress-mount

git clone https://github.com/nicholasjackson/mtls-go-example
cd mtls-go-example

/////// Note: change <password>  to any value you like

./generate.sh your_fqdn <password>
mkdir ../your_fqdn && mv 1_root 2_intermediate 3_application 4_client ../your_fqdn && cd ..

Then all of CA key and cert moved into your_fqdn DIR.

NOTE: Following all of create sceret command must under the same DIR with your_fqdn.

Intergration MTLS

Incoming Traffic with MTLS (Ingressgateway)

  1. Deploy a httpbin service under default namespace and enabled sidecar injection

    Reference command: kubectl apply -f <(istioctl kube-inject -f httpbin.yaml)

    apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

  2. Create two secrets istio-ingressgateway-certs and istio-ingressgateway-ca-certs under istio-system namespace

    kubectl create -n istio-system secret tls istio-ingressgateway-certs --key your_fqdn/3_application/private/your_fqdn.key.pem --cert your_fqdn/3_applicationcerts/your_fqdn.cert.pem
kubectl create -n istio-system secret generic istio-ingressgateway-ca-certs --from-file=your_fqdn/2_intermediate/certs/ca-chain.cert.pem

  3. Define a Gateway and VirtualService and DestinationRule

    kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: MUTUAL
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      caCertificates: /etc/istio/ingressgateway-ca-certs/ca-chain.cert.pem
    hosts:
    - "*"
EOF

    kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "your_fqdn"
  gateways:
  - httpbin-gateway
  http:
  - match:
    - uri:
        prefix: /status
    - uri:
        prefix: /delay
    route:
    - destination:
        port:
          number: 8000
        host: httpbin
EOF

    kubectl apply -f - <<EOF
apiVersion: "networking.istio.io/v1alpha3"
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: "httpbin"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF

  4. Check Cert and CA

    kubectl exec -it $(kubectl get pods -n istio-system| grep ingress | awk -F " " '{print $1}') -n istio-system -- ls /etc/istio/ingressgateway-certs/
kubectl exec -it $(kubectl get pods -n istio-system| grep ingress | awk -F " " '{print $1}') -n istio-system -- ls /etc/istio/ingressgateway-ca-certs/

  5. Using CURL command to test it.

    NOTE: 31390 is NodePort, because in minikube we haven’t EXTERNAL-IP, so we must use Node port. detailed message please reference link ingress-control

    curl -HHost:your_fqdn --resolve your_fqdn:31390:10.120.20.63 --cacert  your_fqdn/2_intermediate/certs/ca-chain.cert.pem --cert your_fqdn/4_client/certs/your_fqdn.cert.pem --key your_fqdn/4_client/private/your_fqdn.key.pem https://your_fqdn:31390/status/418 -v

    The successful mark is you will see a teapot

Outgoing Traffic with MTLS (ServiceEntry)

  1. Create an external namespace

    kubectl create namespace external

  2. Create client secret under external namespace

    kubectl create -n external secret tls testsleep --key your_fqdn/4_client/private/your_fqdn.key.pem --cert your_fqdn/4_client/certs/your_fqdn.cert.pem

kubectl create -n external secret generic ca --from-file=your_fqdn/2_intermediate/certs/ca-chain.cert.pem

  3. Deploy sleep service under external namespace.

    NOTE: Following files ,you must install it under external namespace.

    • Deploy sleep under external namespace and enable injection.

      Reference command: kubectl apply -f <(istioctl kube-inject -f sleep.yaml) -n external

         ###contents of the sleep.yaml
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: sleep
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: sleep
     labels:
       app: sleep
   spec:
     ports:
     - port: 80
       name: http
     selector:
       app: sleep
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sleep
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sleep
     template:
       metadata:
         labels:
           app: sleep
         annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/userVolume: '{"client-secret": {"secret": {"secretName":"testsleep"}},"ca-secret": {"secret": {"secretName":"ca"}}}'
           sidecar.istio.io/userVolumeMount: '{"client-secret": {"mountPath":"/etc/client-cert/","readOnly":true},"ca-secret":{"mountPath":"/etc/ca-cert/"}}'
       spec:
         serviceAccountName: sleep
         containers:
         - name: sleep
           image: governmentpaas/curl-ssl
           command: ["/bin/sleep", "3650d"]
           imagePullPolicy: IfNotPresent
   ---

    • Deploy VS,DR, SE under external namespace

      Reference command: kubectl apply -f xxxx.yaml -n external

         #### contents of the XXXX.yaml
   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: test-external
     namespace: external
   spec:
     hosts:
     - your_fqdn
     http:
     - match:
       - port: 31380
       route:
       - destination:
           host: your_fqdn
           port:
             number: 31390
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     name: test-external
   spec:
     host: your_fqdn
     trafficPolicy:
       loadBalancer:
         simple: ROUND_ROBIN
       portLevelSettings:
       - port:
           number: 31390
         tls:
           mode: MUTUAL
           clientCertificate: /etc/client-cert/tls.crt
           privateKey: /etc/client-cert/tls.key
           caCertificates: /etc/ca-cert/ca-chain.cert.pem
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: test-external
   spec:
     hosts:
     - your_fqdn
     ports:
     - number: 31380
       name: http
       protocol: HTTP
     - number: 31390
       name: https
       protocol: HTTPS
     resolution: DNS
     location: MESH_EXTERNAL

  4. Check CA and Cert

    kubectl exec -it $(kubectl get pods -n external| grep sleep | awk -F " " '{print $1}') -c istio-proxy -n external -- ls /etc/client-cert/
kubectl exec -it $(kubectl get pods -n external| grep sleep | awk -F " " '{print $1}') -c istio-proxy -n external -- ls /etc/ca-cert/

  5. Test Client

    • kubectl exec into sleep container

      kubectl exec -it $(kubectl get pods -n external| grep sleep | awk -F " " '{print $1}') -c sleep -n external -- /bin/sh

    • Testing

      curl -HHost:your_fqdn --resolve your_fqdn:31380:10.120.20.63 -svL http://your_fqdn:31380/status/418
curl -HHost:your_fqdn --resolve your_fqdn:31390:10.120.20.63 -svL http://your_fqdn:31390/status/418

    The successful mark is you will see a teapot

Test Analysis(IP and FQDN)

  1. update VS, DR supports wildcard. please look at following example.

    apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: test-external
  namespace: external
spec:
  hosts:
  - "*.com"
  http:
  - match:
    - port: 31380
    route:
    - destination:
        host: "*.com"
        port:
          number: 31390
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: test-external
spec:
  host: "*.com"
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 31390
      tls:
        mode: MUTUAL
        clientCertificate: /etc/client-cert/tls.crt
        privateKey: /etc/client-cert/tls.key
        caCertificates: /etc/ca-cert/ca-chain.cert.pem

  2. update SE to support FQDN and IP

    • test wildcard

      NOTE: this scenario,“resolution” field must fills in “NONE”, otherwise appear an error when configuration.

         apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: test-external
   spec:
     hosts:
      - '*.com'
     ports:
     - number: 31380
       name: http
       protocol: HTTP
     - number: 31390
       name: https
       protocol: HTTPS
     resolution: NONE
     location: MESH_EXTERNAL

      # curl -HHost:your_fqdn --http2-prior-knowledge -svL http://10.120.20.63:31380/status/418
# curl -HHost:your_fqdn --http2-prior-knowledge -svL http://your_fqdn:31380/status/418

      Test result:(maybe need other configuration) :503.

    • test IP only

         apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: test-external
   spec:
     hosts:
      - '*.com'
     endpoints:
       - address: 10.120.20.63
     ports:
     - number: 31380
       name: http
       protocol: HTTP
     - number: 31390
       name: https
       protocol: HTTPS
     resolution: STATIC
     location: MESH_EXTERNAL

      # curl -HHost:your_fqdn  --http2-prior-knowledge -svL http://10.120.20.63:31380/status/418

      Test result: OK. haven’t DNS resolution.

      # curl   --http2-prior-knowledge -svL http://10.120.20.63:31380/status/418

      Test result: 502

    • test FQDN only

         apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: test-external
   spec:
     hosts:
      - '*.com'
     endpoints:
       - address: your_fqdn
     ports:
     - number: 31380
       name: http
       protocol: HTTP
     - number: 31390
       name: https
       protocol: HTTPS
     resolution: DNS
     location: MESH_EXTERNAL

      # curl -HHost:your_fqdn  --http2-prior-knowledge -svL http://10.120.20.63:31380/status/418

      Test result: OK. have DNS resolution.

    • test both FQDN and IP

         apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: test-external
   spec:
     hosts:
      - '*.com'
     endpoints:
       - address: your_fqdn
       - address: 10.120.20.63
     ports:
     - number: 31380
       name: http
       protocol: HTTP
     - number: 31390
       name: https
       protocol: HTTPS
     resolution: DNS
     location: MESH_EXTERNAL

      # curl -HHost:your_fqdn -svL http://your_fqdn:31380/status/418

      Test result: OK , have DNS resolution

      # curl -HHost:your_fqdn  --http2-prior-knowledge -svL http://10.120.20.63:31380/status/418

      Test result: OK, haven’t DNS resolution

      # curl   --http2-prior-knowledge -svL http://your_fqdn:31380/status/418

      Test result: OK.

      # curl   --http2-prior-knowledge -svL http://10.120.20.63:31380/status/418

      Test result: 502

  3. Test Analysis

    • IP Only.

      Even only we have IP, we still must override Host header to match VS host. so here has a problem.

    • FQDN Only.

      This is OK. if FQDN can match VS host.

    • Both IP and FQDN.

      Here still have a problem the same as only IP scenario.

Conclusion

C1. If want to transfor port from 31380 to 31390 as above client Test, you must add 31380 port message into related ServiceEntry,otherwise curl connection will be reset by peer(this is because if you didn’t add 31380 port message in SE, the sleep service doesn’t listen 0.0.0.0 31380.). so if we have many different port need transfor, then we must add port into related SE.

Questions

Q1. Testing ingressgateway HTTP Protocol is HTTP2.0, but Through sleep container, HTTP Protocol is HTTP1.1. Is this normal phenomenon?
A1. using "–http2-prior-knowledge" in the curl command ,then HTTP protocol is HTTP2. so this isn’t an issue.

Q2. If i set meshpolicy tls mode is PERMISSIVE . then i can’t install httpbin DR, the test also can success. but if i set it to STRICT, then test will failed. so maybe in our ENV, we don’t need set mode to STRICT(default is PERMISSIVE), because our ENV internal service communications don’t use TLS as our previous ARCH.

Q3. If we use directly IPs to send out to external service, we must override Host header to match VS host, otherwise we have 502 error.

CATALOG
    FEATURED TAGS
    c++17 client-go coredump design-pattern docker envoy gerrit git github golang hpack hugo iptables istio kafka kernel kubernetes linux mtls net netfilter operator-sdk profiling raft tcpdump 随笔