[New instance] https://invidious.nerdvpn.de

[Sommerwiesel]

URL

https://invidious.nerdvpn.de

Mandatory checks

• Instance has a domain name
• Instance is served over HTTPS
• Statistics (/api/v1/stats) are enabled
• Instance is properly configured (including the mandatory post install configuration)
• Instance has an automatic hourly restart setup

Maintainer chart

• Ensure that my instance is up to date (less than one month old)
• Ensure a proper uptime of my instance (around 90%)

Host country

Germany

Man in the Middle

None

Source code URL

https://github.com/Sommerwiesel/invidious

Analytics

None

Additionnal informations

• The only source code changes as of 05-22-2022 are css changes and there are no plans in changing anything else.
• The instance is open for new user registrations.
• The banner text is german only.
• The server has a planned downtime of up to 15min every day around 1am (UTC) for backups to an encrypted external storage and updates.
• The hourly restart is done via cron at 42 * * * *

[Sommerwiesel]

https://stats.uptimerobot.com/mn6k9upm1g

[unixfox]

Added on uptime robot: https://stats.uptimerobot.com/89VnzSKAn/791917306

[Sommerwiesel]

I have a question about the hourly restart.
I understand why it's neccessary to circumvent the inevitable "invidious breaking down" after a while.
However, I find it hard to believe it's the best solution to this.

Can someone explain what exactly happens when invidious crashes?
Do we get any kind of status/error/log?
Is there some way to monitor this and only restart invidious the moment such a crash happens?

I have full root access to my server, surely there is some better way a cronjob/monitor can watch the invidious service/website/whatever and react accordingly.
I use the official docker image with a custom systemd unit starting it.

The problem I have with the hourly restart is, that it also breaks active sessions. I often watch videos which are longer than an hour and this results in the video breaking multiple times because of the restart.

Also, what if invidious breaks only 15min after the last restart, that would leave the service with a downtime of 45min until the next.

I would really like to implement a better solution that "just" a hourly restart.

Thanks for any input.

[unixfox]

Nowadays if you don't have use_quic enabled invidious will never really crash but it can become sluggish due to some unknown memory leaks: iv-org/invidious#1438

This is why we recommend restarting it every hour because a public instance will receive way more traffic than a private one and will become sluggish more quickly.

If you don't want to break "active sessions", you can also run two invidious processes at the same time behind a reverse proxy and just restart one at a time. Do note that it's not normal if videojs (the player of invidious) doesn't retry if a request fails due to invidious being offline for a short period of time, you should probably open a bug report for that on invidious.

[SamantazFox]

Can someone explain what exactly happens when invidious crashes?

I have absolutely no clue, sorry :/ If you find out why, please open an issue with your findings !

Do we get any kind of status/error/log? Is there some way to monitor this and only restart invidious the moment such a crash happens?

I have full root access to my server, surely there is some better way a cronjob/monitor can watch the invidious service/website/whatever and react accordingly. I use the official docker image with a custom systemd unit starting it.

Nope.

The phenomenon is weird in many aspects. It's not as simple as "invidious breaks" like your program have a segfault. Invidious becomes sluggish (slow to respond) as the time goes on, and you get more occurences of weird things, like slow response times, subscriptions not refreshing anymore, database queries failing, etc..

I don't know if it's related to the language itself (maybe the internal Crystal scheduler doesn't handle that many fibers?), to one of the libraries (shards) that we're using, to the garbage compiler, or to our own code. There are so many things that I don't fully understand in the chain that could cause those problems.

As unifox said, one big source of problems was QUIC, which is a monster (it embeds its own SSL library). Since we changed the default behavior from "use it" to "not use it", it also reduced the amount of crashes encountered by users / instance admins.

I would really like to implement a better solution that "just" a hourly restart.

Thanks for any input.

I'd love to, too, but at the moment I don't have the knowledge nor the time to dig in depth in valgrind, strace, or other memory analysis tools usage.

Technically, invidious can run fine for multiple hours (generally > 6h on a medium sized instance) but in order to reduce the amount of problems (= issues) we have to handle, we decided that an 1h restart was mandatory. This indeed helped with a lot of common problems.

Until a definitive fix is found, that requirement will stay.

[Sommerwiesel]

Hello there,
thanks a lot for the detailed answer and explaining why the whole thing isn't that simple.
Huge kudos for taking your time writing this 👍🏻

As of now, I'll just accept this weird bug and keep the hourly restart active.

Have a nice week everyone :)

[TheFrenchGhosty]

@Sommerwiesel Instance is properly configured and has good uptime.

Please confirm that you indeed have auto-restart setup?

If yes, are you ready for it to be added to the instance list?

PS: Some, in my opinion, issues with your CSS:

• the RSS button isn't visible on the channel/playlist page
• the "video/playlist/community" category "switcher" isn't visible on the channel page
• the "watch on youtube" button isn't visible anywhere (on purpose I guessing) except on a playlist page

If you explicitly want those "issues", it's not a problem.