Evaluate & Support using dedicated disks for WAL to mitigate IOPS contention

[unmarshall]

How to categorize this issue?

/area control-plane
/area performance
/area scalability
/kind enhancement

What would you like to be added:
Today etcd-druid deploys an etcd cluster with a single SSD that is shared to store WAL and snapshot files. All these SSDs come with IOPS limits. For clusters where the etcd read/write activity is LOT, there is possibility that etcd slows down significantly which then causes timeouts from kube-apiserver.

Trace[1428471700]:  ---"Txn call failed" err:etcdserver: request timed out 7015ms (06:23:13.521)]
E0729 06:23:13.532618       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"

Details of one such occurrence can be seen in Live Issue#7539.

It is a recommendation from upstream etcd to have dedicated disk for WAL (https://etcd.io/docs/v2.3/admin_guide/). Since these SSDs have an associated cost this should be made configurable via Etcd resource.

Why is this needed:
etcd clusters rely heavily on extremely fast SSDs and their response times are sensitive to disk performance. For large/busy etcd clusters the IOPS can easily cross the limits for the SSD used. In order to prevent timeouts from the kube-apiserver which results in an outage it is essential to provide an option to use multiple SSDs by individual etcd members.

[ishan16696]

It is a recommendation from upstream etcd to have dedicated disk for WAL (https://etcd.io/docs/v2.3/admin_guide/)

This is very old doc which is written for etcd v2.3, which is no longer supported by etcd. For newer version etcd, I don't find such recommendations https://etcd.io/docs/v3.5/tuning/#disk, anyway I found rancher does that to improve performace.

Is it a good idea to use dedicated disk for wal file ? may be it is but we have to prove it by performance testing .
Moreover, right now I can think couple of things which need to be taken care of:

1. Dedicated disk for wal means more no. of disk per etcd cluster, which will also increase our cost ... we have to calculate how much ,and we have to analysis the trade off b/w performance improvement vs cost increase due to more no. of disk, whether will it be worth the cost or not ?
2. Now, backup-restore's restoration sub-command have to be compatible to restore the etcd with 2 disk as it need to know while during the restoration that it has to restore/create wal file on separate disk, and db file on separate disk, then do the restoration.

[unmarshall]

Dedicated disk for wal means more no. of disk per etcd cluster, which will also increase our cost ... we have to calculate how much ,and we have to analysis the trade off b/w performance improvement vs cost increase due to more no. of disk, whether will it be worth the cost or not ?

Of course, that is the reason it will be configurable. Human operators can take a call for which clusters they are ready to pay an additional cost. This also has an impact on node utilization as well since there are only a limited number of network volumes that can be attached to a Node. Having more than one volume will result in more fragmented nodes, thus increasing the cost.
However, you need to check if exceeding the IOPS limit is a regular occurrence and if it is then for those clusters one can choose to pay this extra cost to ensure availability of the control plane.

[unmarshall]

Now, backup-restore's restoration sub-command have to be compatible to restore the etcd with 2 disk as it need to know while during the restoration that it has to restore/create wal file on separate disk, and db file on separate disk, then do the restoration.

Will there be a difference? These are mounted volumes. I only imagine a possible change in the directory path which will be specified via --wal-dir and --data-dir configuration.

[ishan16696]

Will there be a difference?

yeah, it should be fine, just a wal dir path need to be passed in restore config of restore api call

[vlerenc]

Indeed, using a second disk (instead of setting/increasing the iops) is an option, but comes with the above mentioned downsides. @ishan16696 @shreyas-s-rao and I had a similar discussion.

While Gardener adopters can charge for more iops, it will be difficult to incorporate pool utilization / inefficiency (if we need even more volume attachments). At least, that's a lot harder/impossible to calculate and charge for iops.

Also, iops scales probably better. A second disk, in theory, would only yield a factor of 2 improvement (ideal case), while with iops, we can go as fast/high as the cloud provider offers.