Map Jitsi package version to stable release version

[Rpi-Dude-ghma]

Hi @damencho ,

As of now stable releases, such as 8c3f7ca , tag image versions, which are built on release day.

However, if one rebuilds the images at a later date, using a tag in this repo, latest packages from https://download.jitsi.org are used in newly built images. These latest packages could be unstable or incompatible with the configurations in the tagged release.

This MR/ commit proposes a way to embed package version names, for a tagged release.

If one needs to rebuild images (say to use old known good Jitsi packages, but updating Debian base image packages), one can

• checkout the tag
• uncomment the version ARG lines in various docker files
• build required images (make build_base, make build_base-java, make build_jvb etc)

A change will be required in release process, where along with updating stable version number, version ARG lines are updated as well.

Current set of version numbers in this commit have been found by following:

apt-cache policy jibri jicofo jigasi jitsi-autoscaler-sidecar jitsi-meet-web jitsi-upload-integrations jitsi-videobridge2 jitsi-meet-prosody | grep '^[a-z]\|Candidate:' | tr '\n:' ' ' | sed -e 's/ji/\nji/g' | grep -v '^$' |awk '{printf "%s=%s \n", $1, $3}'
jibri=8.0-184-g4f57e90-1
jicofo=1.0-1167-1
jigasi=1.1-395-g3bb4143-1
jitsi-autoscaler-sidecar=1.0-50-g5fef5f5-1
jitsi-meet-web=1.0.8936-1
jitsi-upload-integrations=0.15.15-1
jitsi-videobridge2=2.3-260-gea11cd36c-1
jitsi-meet-prosody=1.0.8936-1

[saghul]

While I understand the desire some may have for reproducibility, in general I'm -1 on this approach because:

• It complicates the unstable build

• It complicates each stable build as well

• The upside is not that big, you can always layer a custom image on top of ours

[Rpi-Dude-ghma]

Hi @saghul ,

Thank you for review.

• It does not complicate unstable build, since by default these *_VERSION ARGs are empty, and thus current behavior of unstable builds in not impacted at all. (I have tested and can be tested by others as well by running existing unstable build commands/ pipelines)
• Stable release may take a bit more time for sure, but from complexity point of view, looking at stable release related commit pairs (such as 8c3f7ca + 3dc1ea5 and 3bc1ebc + 839c483) similar manual tagging is already done, and extra effort and complexity is quite low for once a month stable releases. We might be able to automate this a bit more to reduce effort, but that would make ci more complex.
• Yes, additional layer could be added, but that increases final image size (increase is 100s of MBs). Being able to build fresh image from the tagged version allows for more frequent builds for longer time period, and helps reduces one place where things could go wrong.

I feel this extra effort per stable release gives significant benefits. I am willing to discuss and answer any additional concerns.

If for some reason above proposal is considered a definite no-go, may I request to update release notes (such as https://github.com/jitsi/docker-jitsi-meet/releases/tag/stable-10655) to include package version numbers? This would add some level of assurance by having mapping of docker-jitsi-meet stable release to Jitsi packages in each stable release.

[saghul]

• It does not complicate unstable build, since by default these *_VERSION ARGs are empty, and thus current behavior of unstable builds in not impacted at all. (I have tested and can be tested by others as well by running existing unstable build commands/ pipelines)

I missed that, thanks for clarifying!

• Stable release may take a bit more time for sure, but from complexity point of view, looking at stable release related commit pairs (such as 8c3f7ca + 3dc1ea5 and 3bc1ebc + 839c483) similar manual tagging is already done, and extra effort and complexity is quite low for once a month stable releases. We might be able to automate this a bit more to reduce effort, but that would make ci more complex.

The process it currently automated here: https://github.com/jitsi/docker-jitsi-meet/blob/master/.github/workflows/release-stable.yml

If you can automate it by getting the versions from the meta repository I'd be more receptive to the idea :-)

• Yes, additional layer could be added, but that increases final image size (increase is 100s of MBs). Being able to build fresh image from the tagged version allows for more frequent builds for longer time period, and helps reduces one place where things could go wrong.

I've honestly never found the image size argument compelling.

If for some reason above proposal is considered a definite no-go, may I request to update release notes (such as https://github.com/jitsi/docker-jitsi-meet/releases/tag/stable-10655) to include package version numbers? This would add some level of assurance by having mapping of docker-jitsi-meet stable release to Jitsi packages in each stable release.

That would require doing it manually. If we automate that, then we might as well go all in and do it like you suggested. The starting point would be to get the jitsi meet web, jicofo and jvb version from here: https://github.com/jitsi/jitsi-meet-debian-meta for the matching release number.

@damencho WDYT?

[Rpi-Dude-ghma]

Thank you for pointing to the release automation workflows. Most recent commit duplicates the logic, from unstable, to get version information in stable jobs, and update the stable commit message to include version numbers. This way no change is done to release process (no additional/ manual work), except for updating the commit message. I hope this is an acceptable solution.

( I see in unstable release, tags are using application versions (e.g

docker-jitsi-meet/.github/workflows/unstable.yml

Line 189 in a9c0a0c

${{ secrets.JITSI_REPO }}/jibri:${{ needs.version.outputs.jibri_version }}-${{ matrix.config.arch }}

), so it seems there was some effort ongoing in this direction already, and thus not making any wide spread changes.)

Image size in itself is not so much of a problem, except when automated information security assessment tools get involved, which not only scan the artifacts in latest layer, but the old ones too, and cause red marks in audit reports (and a lot of followup work, which can be avoided by getting rid of old layers/ artifacts). I understand there are other tools that can merge layers to get only the most recent artifacts, but still sometimes remnants are left since there is some version number appended to the file.

Would it be ok to leave ARG comments, so it is easier for those re-building images on their own?
I can work on another MR later (in a few months), if we would like to fully automate the process to use specific version numbers at build time in the docker images.

[Rpi-Dude-ghma]

Hi @saghul ,

I have now used version handling approach from the unstable release and updated stable job steps to fix versions of various components in stable release commit, which are revered back to bring back latest ones in unstable builds.

This ensures that current workflows are not affected for developers and yet one might be able to rebuild stable images with fixed component versions number at any later time.

[Rpi-Dude-ghma]

@damencho , @saghul ,
may you have a look please.

[emrahcom]

Hi @Rpi-Dude-ghma,

Out of curiosity, why do you need reproducible images?

It is not complicated to provide this by manually editing Dockerfile if it is rarely needed. And you will still need to update Dockerfile even if this PR is merged.

[Rpi-Dude-ghma]

Hi @emrahcom ,

To be honest, I just need an easy way to map docker-jitsi-meet tag to Jitsi components version, I am not looking necessarily for full re-producibility. (in fact re-producibility is not even possible with fat base image with all distribution packages... reason 2 below). Reasons why I need this mapping are

1. Stable docker-jitsi-meet tags are used as "Jitsi" version, though there are different Jitsi components that have their own versioning scheme. This coupled with high development velocity means that a combination of configurations of various components could be out-of-date/ incompatible with other components having a different version. By pinning Jitsi component versions against docker-jitsi-meet we can ensure we can rule out incompatible versions accidentally being used together.
2. Fat base image means when ever there are trivy security scans that report vulnerabilities (which are almost always in base image distribution package), we need to rebuild image to have no critical CVEs in the running image. When we need to re-build image, right now we need to fiddle with current image, extract individual package version numbers and build image by editing Dockerfile. This is working, but seems un-necessarily grunt work that should not be there. We frequently update images, but not necessarily when security scans cause us to just re-build.

Given these reason, it seemed to be best if I could get this in upstream repo and possibly this is useful for others in community too. Also, this is kind of being done with unstable, so why not we have it with stable too, where stable docker-jitsi-meet tag has good know jitsi component versions.

(sorry for the delayed reply)

[damencho]

Checking the versions for jitsi-meet is easy by just visiting:
https://github.com/jitsi/jitsi-meet-release-notes/blob/master/CHANGELOG-WEB.md

But we do not have answer at the moment for the rest of the components ...

[damencho]

As a middle ground and first step why we just don't put the versions in the release notes when those are generated for the Github release?

[Rpi-Dude-ghma]

Hello,

I implemented full automation, re-using unstable logic in stable, given @saghul 's suggestion

That would require doing it manually. If we automate that, then we might as well go all in and do it like you suggested.

We can start with putting version numbers in release notes, but given this is already implemented fully (most of the logic is copied from unstable, so not new), we could may be start using it. I have verified Dockerfile changes, but of course having one extra check will be good.