THREESCALE-12133 optimize configuration reload

[tkan145]

What

Fix THREESCALE-12133

NOTE:

• Try to download the flamegraph file and and open it in your browser (interactive graph).
• Latency was measured by reloading the static file, adding 2-3 seconds to the result if loading from a portal.
• JSON encoding/decoding still consumes a significant portion of the CPU, but latency has decreased to an acceptable level, and it requires more extensive changes, so I decided to defer the refactor to the next PR.

Before:

File read

bash-5.2# accessed-files -p `pgrep -f 'nginx: worker'` -t 60        
Neither -r nor -w options are specified.                            
bash-5.2# accessed-files -p `pgrep -f 'nginx: worker'` -r -t 60     
Tracing 3599829 (/usr/local/openresty/nginx/sbin/nginx)...          
Please wait for 60 seconds.                                         
                                                                    
=== Top 10 file reads ===                                           
#1: 117280 times, 233684323 bytes reads in file apicast-policy.json.
#2: 44 times, 31357920 bytes reads in file config.json.             

Latency

HTTP                                                                                
http_req_duration..............: med=42.46ms p(95)=48.14ms p(99)=2.37s p(99.9)=2.76s
  { expected_response:true }...: med=42.46ms p(95)=48.14ms p(99)=2.37s p(99.9)=2.76s
http_req_failed................: 0.00%  0 out of 9000                               
http_reqs......................: 9000   47.464397/s

Flamegraph

After:

File read

bash-5.2# accessed-files -p `pgrep -f 'nginx: worker'` -r -t 60
Tracing 3605605 (/usr/local/openresty/nginx/sbin/nginx)...     
Please wait for 60 seconds.                                    
                                                               
=== Top 10 file reads ===                                      
#1: 44 times, 31357920 bytes reads in file config.json.        

Latency

HTTP                                                                                
http_req_duration..............: med=42.66ms p(95)=44.6ms p(99)=68.14ms p(99.9)=205.02ms
  { expected_response:true }...: med=42.66ms p(95)=44.6ms p(99)=68.14ms p(99.9)=205.02ms
http_req_failed................: 0.00%  0 out of 9147                                   
http_reqs......................: 9147   49.677724/s  

Flamegraph

Verification steps

• Install 3scale
• Setup products, applications, backend, etc...

Details

#!/bin/bash

# Get the ADMIN_URL and ADMIN_ACCESS_TOKEN from apimanager and system-seed secret
DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')
ADMIN_ACCESS_TOKEN=$(oc get secret system-seed -n 3scale-test -o jsonpath="{.data.ADMIN_ACCESS_TOKEN}"| base64 --decode)
oc project 3scale-test
# Create the required secrets for Accounts, products and backends. 
oc apply -f - <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: mytenant
type: Opaque
stringData:
  adminURL: https://3scale-admin.$DOMAIN
  token: $ADMIN_ACCESS_TOKEN
EOF
# user secret
oc apply -f - <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: myusername01
stringData:
  password: "123456"
EOF
# Developer User
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: DeveloperUser
metadata:
  name: developeruser01
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  developerAccountRef:
    name: developeraccount01
  email: [email protected]
  passwordCredentialsRef:
    name: myusername01
  providerAccountRef:
    name: mytenant
  role: admin
  username: myusername01
EOF
sleep 30
# TODO check for developer user completed
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: DeveloperAccount
metadata:
  name: developeraccount01
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  orgName: 3scale-test
  providerAccountRef:
    name: mytenant
EOF
deploy httpbin and use it as the backend
oc new-project httpbin
oc new-app quay.io/trepel/httpbin
oc get svc
oc scale deployment/httpbin --namespace httpbin --replicas=1 
oc project 3scale-test

# create backend
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: Backend
metadata:
  name: backend1-cr
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  mappingRules:
    - httpMethod: GET
      increment: 1
      last: true
      metricMethodRef: hits
      pattern: /
    - httpMethod: POST
      pattern : "/"
      metricMethodRef: hits
      increment: 1    
  name: backend1
  privateBaseURL: 'http://httpbin.httpbin.svc:8080'
  systemName: backend1
EOF

TOTAL_PRODUCTS=600

for ((n=1; n<=$TOTAL_PRODUCTS; n++))
do
# Product
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: Product
metadata:
  name: product$n-cr
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  applicationPlans:
    plan01:
      name: "My Plan 01"
  deployment:
    apicastHosted:
      authentication:
        userkey:
          authUserKey: token
  name: product$n
  backendUsages:
    backend1:
      path: /
  mappingRules:
    - httpMethod: GET
      pattern : "/"
      metricMethodRef: hits
      increment: 1
    - httpMethod: POST
      pattern : "/"
      metricMethodRef: hits
      increment: 1
  policies:
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo1"
            value: "test"
            value_type: "plain"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo2"
            value: "test"
            value_type: "plain"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo3"
            value: "test"
            value_type: "plain"
    - name: logging
      version: builtin
      enabled: true
      configuration:
        custom_logging: "[{{time_local}}] {{host}}:{{server_port}} {{remote_addr}}:{{remote_port}} \"{{request}}\" {{status}} {{body_bytes_sent}} ({{request_time}}) {{post_action_impact}} AND {{upstream_response_time}}"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo4"
            value: "test"
            value_type: "plain"
    - name: headers
      version: builtin
      enabled: true
      configuration:
        set_headers:
          - name: "echo5"
            value: "test"
            value_type: "plain"
    - name: apicast
      version: builtin
      enabled: true
      configuration: {}
EOF

# application
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: Application
metadata:
  name: application$n-cr
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  accountCR: 
    name: developeraccount01
  applicationPlanName: plan01
  productCR: 
    name: product$n-cr
  name: testApp
  description: further testing
EOF
# TODO proxy promote
oc apply -f - <<EOF
---
apiVersion: capabilities.3scale.net/v1beta1
kind: ProxyConfigPromote
metadata:
  name: product$n-v1-production
  namespace: 3scale-test
  annotations:
    "insecure_skip_verify": "true"
spec:
  productCRName: product$n-cr
  production: true
  deleteCR: true
EOF
done


sleep 30
echo Product Route: 
echo "https://$(oc get routes | grep product1 |grep production| awk '{print $2}')" 
echo
echo User_key: 
echo $(curl -s -X 'GET' "https://3scale-admin.$DOMAIN/admin/api/applications.xml?access_token=$ADMIN_ACCESS_TOKEN&page=1&per_page=500&service_id=3" -H 'accept: */*' | grep -oP '<user_key>\K[^<]+' | sed 's/\s//g')

• Once the script finish, visit the admin portal and wait for all products to be populated (going to take a while)
• Create an token
• Inside APIcast repo, move into dev-environments folder

▲ lib/3scale/APIcast cd dev-environments/http-proxy-plain-http-upstream/

• Update the docker-compose.yml file as follow. Update <token> and <3scale_admin_portal> to match values from previous step.

diff --git a/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml b/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml
index f7b311a5..5032bc13 100644                                                                                                                     
--- a/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml                                                                            
+++ b/dev-environments/http-proxy-plain-http-upstream/docker-compose.yml                                                                            
@@ -12,11 +12,12 @@ services:                                                                                                                       
     - two.upstream                                                                                                                                 
     environment:                                                                                                                                   
       THREESCALE_CONFIG_FILE: /tmp/config.json                                                                                                     
-      THREESCALE_DEPLOYMENT_ENV: staging                                                                                                           
-      APICAST_CONFIGURATION_LOADER: lazy                                                                                                           
+      THREESCALE_PORTAL_ENDPOINT: "https://<token>:<3scale_admin_portal>"                                                                          
+      THREESCALE_DEPLOYMENT_ENV: production                                                                                                        
+      APICAST_CONFIGURATION_LOADER: boot                                                                                                           
       APICAST_WORKERS: 1                                                                                                                           
-      APICAST_LOG_LEVEL: debug                                                                                                                     
-      APICAST_CONFIGURATION_CACHE: "0"                                                                                                             
+      APICAST_LOG_LEVEL: notice                                                                                                                    
+      APICAST_CONFIGURATION_CACHE: "15"                                                                                                            
     expose:                                                                                                                                        
       - "8080"                                                                                                                                     
       - "8090"                                                                                                                                     

• Start the gateway

make gateway IMAGE_NAME=quay.io/3scale/apicast:latest 

• Get the gateway's IP

 ▲ lib/3scale/3scale-operator docker inspect \                                                         
  -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' http-proxy-plain-http-upstream-gateway-1

• In another terminal, create k6 script with the following content

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '3m', target: 100 },
    { duration: '6m', target: 200 },
    { duration: '2m', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], 
    http_req_failed: ['rate<0.01'], 
  },
};

export function setup() {
  // This runs once for the whole test run, after the init context and before default
  const msg = `[${new Date().toISOString()}] Starting load test`;
  console.log(msg);
}

export default function () {
  const url = 'http://<APICast_gateway_IP>:8080/echo?user_key=<user_key>'
  const params = {
    headers: {
      'Host': '<hostname>'
      'Content-Type': 'application/json',
    },
  };

  const res = http.get(url, params);
  if(res.timings.duration > 1000) {
    const msg = `[${new Date().toISOString()}] Response time above 1s: ${String(res.timings.duration)} ms`;
    console.log(msg);
  }

  check(res, {
    'status 200': (r) => r.status === 200
  });

  sleep(1);
}

Replace:

• APICast_gateway_IP, user_key with value from previous steps

• Visit the admin portal and replace the hostname with the hostname of one product of your choice

• Start k6 test

k6 run --summary-trend-stats="med,p(95),p(99),p(99.9)" script.js 

• Once the test is finish, stop the gateway

CTRL-C

• Checkout this branch and build a new runtime-image

make runtime-image IMAGE_NAME=quay.io/3scale/apicast:reload-patch

• Start the gateway again with new image

make gateway IMAGE_NAME=quay.io/3scale/apicast:reload-patch 

• Start the test again, you should see the delay drop to sub 1s

k6 run --summary-trend-stats="med,p(95),p(99),p(99.9)" script.js 

[akostadinov]

Reading lua for the first time so excuse me if I miss something.

Whenever the manifest is not found, here we insert a failure somewhere. But if it is missing, should we continue with the rest of the method? Shouldn't we just return to avoid further processing?

Perhaps this would not be performance critical because we don't expect errors to happen often. But it occurred to me to ask you if that makes sense.

[akostadinov]

A nit, it looks like the method is only called in gateway/src/apicast/configuration.lua without parameters. So perhaps makes sense to simplify by removing the argument from it as well the if clause.

[akostadinov]

These are some excellent improvements! Just a couple of minor suggestions that you can ignore if they don't make sense to you.

Now learning a little bit more about apicast architecture, I think that your suggestion of looking at increment updates makes a lot of sense so we can discuss the options to implement.