I find the reason why my cpu high in my test machine

[werbenhu]

hi.
I created a issue a few days ago.
etcd cpu high in my test machine #6576

I use the default configuration, and the CPU becomes very high every 30 seconds.

now i find the reason now.
in core/config_etcd.lua _automatic_fetch() will sync_data() for each work processes.
and sync_data() call watchdir() to watch keys of apisix in etcd such as /apisix/upstreams, /apisix/consumers, /apisix/consumers, /apisix/routes, /apisix/plugin_metadata, /apisix/global_rules, /apisix/proto, etc.

The reason for my situation is that all processors will call watchdir() for all keys at the same time（every 30 seconds） . If I configure 8 worker processors, there are 8 keys here to watch, watchdir() is called 8x8=64 times at the same time.

I modified the timeout of watchdir() so that timeout = 20 + rand(20) for different keys. now cpu is normal.

I'm not sure if this is a bug, I just found out what happened to me.

[tzssangglass]

In fact, watch_dir already has a rand operation, see:

apisix/apisix/core/config_etcd.lua

Lines 586 to 589 in 51eec3e

	-- etcd watch timeout is an expected error, so there is no need for resync_delay
	if err ~= "timeout" then
	ngx_sleep(self.resync_delay + rand() * 0.5 * self.resync_delay)
	end

is the rand interval not large enough?

[werbenhu]

yes, but the err is always "timeout" and ngx_sleep() has never been executed.

[werbenhu]

I find that it doesn't make any sense to call ngx_sleep() after sync_data().

When the next ngx_timer_at(_automatic_fetch) came, all sync_data() would still be executed at the same time.

Calling ngx_sleep() before sync_data() seems to have an effect.

You can see all this by adding some print log.

[tzssangglass]

yes, but the err is always "timeout" and ngx_sleep() has never been executed.

why skip resync_delay while etcd watch timeout(watch timeout): #6259

[tzssangglass]

seems to be no solution to this problem

[werbenhu]

i set resync_delay default value to 20 , and call ngx_sleep before sync_data,

the cpu is now stable, and always below 4%.

before this change, the cpu would be very high every 30 seconds.

local function _automatic_fetch(premature, self)
    ...
    while not exiting() and self.running and i <= 32 do
        local ok, err = xpcall(function()
            ...
            //werben add 
            ngx_sleep(self.resync_delay + rand() * 0.5 * self.resync_delay)
            local ok, err = sync_data(self)
            ...
        end, debug.traceback)
    ...
    end

    if not exiting() and self.running then
        ngx_timer_at(0, _automatic_fetch, self)
    end
end

[tokers]

Large resync delay might enhance the latency to react the configuration changes.

[werbenhu]

Yes, I know that's a problem too.