Monitor system

[str]

Multiple times, we've seen eosdac-client not showing the right data. We think it's an issue with the API. We've seen some times the API has been down, or not updating the data, and other times we think it's because of too much traffic in main net.

A proposal was to have a monitor system to track when the API is up or down, but we agreed that just tracking that is not enough as the API could be up but outdated. We need to:

• create a server/vps to monitor the API
• trigger notifications via email
• trigger notifications via discord
• add new endpoints to track more information, like last block read (@michaeljyeates)
• create multiple metrics about the API
• track state in jungle
• increase the interval between tests
• track state in mainet
• add a default filter to block too many results in query
• solve issue with API getting down
• see if msigs not showing updated msig count #11 is related

[str]

So, to document, today we saw that

• msigs had outdated information about the signatures per msig
• later some msigs were updated with the right signatures count, but others, that were signed at the same time weren't
• later the executed msigs were still shown as open

[str]

I've just confirmed a new custodian just signed all proposals, but the dacclient still shows the wrong numbers. API data still has wrong info.

[str]

I'm now tracking if the API returns any data at all. That now tracks if the API is up or down.

But that gives us not much detail about how old the data in the API is.

[str]

OK, mail notifications enabled

[str]

OK, now notifications of API down are sent to tech channel

[str]

Mainnet API has not been answering.

We now have more information confirming the API is down

time curl -v -I https://api.eosdac.io/v1/eosdac/msig_proposals
*   Trying 79.137.68.20:443...
* TCP_NODELAY set
* Connected to api.eosdac.io (79.137.68.20) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=eu.eosdac.io
*  start date: May 11 17:43:45 2020 GMT
*  expire date: Aug  9 17:43:45 2020 GMT
*  subjectAltName: host "api.eosdac.io" matched cert's "api.eosdac.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55cb00af4db0)
> HEAD /v1/eosdac/msig_proposals HTTP/2
> Host: api.eosdac.io
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 404 
HTTP/2 404 
< access-control-allow-origin: *
access-control-allow-origin: *
< content-type: application/json; charset=utf-8
content-type: application/json; charset=utf-8
< content-length: 60
content-length: 60
< date: Fri, 19 Jun 2020 22:00:32 GMT
date: Fri, 19 Jun 2020 22:00:32 GMT
< 
* Connection #0 to host api.eosdac.io left intact
real	1m48.382s
user	0m0.005s
sys	0m0.009s

[str]

We've been monitoring for 1h and it looks better now: