Custom TLS root certificates for Debian package sources

[tobiasjakobi-lr]

Hello,

I already tried getting feedback for this on the ML, but no response for two weeks now, so I'm trying my luck again here.

Original ML thread: https://lists.linutronix.de/pipermail/elbe-devel/2025-April/007699.html

Original message:

Hello dear ELBE team,

we are currently encountering some problems with our ELBE setup. We
are operating our own repository server, i.e. some Docker that is
serving Debian packages via an Apache. Due to certain IT department
requirements, we are forced to use the server via HTTPS. Again, due to
the IT department, the TLS server certificate of the server is
directly signed by some root cert and not by a cert chain with
intermediate cert. The root cert, you probably have guess it already,
again, comes from team IT.

This in itself wouldn't be a problem. We can easily put this root cert
in /usr/local/share/ca-certificates, call update-ca-certificates, and
everything works. We have done this on the host itself, and inside the
ELBE initvm. So calling curl https://our-debian-repo/ works fine, both
inside and outside the initvm.

What no longer works, is when we now do a initvm submit
some-machine.xml, which uses this server as source for packages. The
root cause is that ELBE doesn't care about the content of
/usr/local/share/ca-certificates when it construct the actual chroot
inside the initvm. We have checked this by doing a initvm submit, and
then chrooting manually into the active chroot. There is nothing in
/usr/local/share/ca-certificates inside the chroot, and of course,
curl no longer works (because it can't verify due to missing root
cert). Which then results in Debian tooling removing the sources entry
(due to failing TLS verification) and then later failing because it
doesn't find some packages.

I have briefly gone over the ELBE sourcecode, in particular the
daemon, but I don't find anything that does any kind of cert transfer
from the initvm to the chroot. I believe this functionality simply
does not exist. We have thought about replicating such functionality
by using the fine-tuning mechanism. But fine-tuning happens much too
late in the whole process. We would need something that runs just
before ELBE does the chroot(), namely copying the content of the
initvm's /usr/local/share/ca-certificates to the corresponding
directory inside the chroot (or bind mounting). And then some
mechanism that runs early in the chroot, namely
update-ca-certificates.

I currently lack the required knowledge about the ELBE codebase to
determine where one would implement such functionality. Maybe someone
from your team can advise?

With best wishes,
Tobias Jakobi

[t-8ch]

Hi @tobiasjakobi-lr thanks for the report and sorry for missing your mail.
I'm on vacation for two weeks and will look at your issues afterwards.

[tobiasjakobi-lr]

@t-8ch Thanks Thomas, looking forward to hearing from you. Enjoy your vacation!

[t-8ch]

The fact that ELBE is using the apt methods/transports from within the target system from the apt of the initvm also leads to other issues.
See for example https://lists.linutronix.de/pipermail/elbe-devel/2025-May/007705.html

Mounting directories from the initvm into the target system could work but has some drawbacks:

• It is intransparent
• It makes modifications of these directories during the build error-prone
• It is not clear that the set of directories will be stable

I am currently evaluating to proxy the apt methods from the initvm into the target.
This should fix yours and the other problem.

[tobiasjakobi-lr]

Since our IT department seems to be pushing this topic again (getting rid of non-encrypted HTTP traffic in our network), would it be possible to give a rough timeline when this feature is going to be implemented?

[t-8ch]

I don't have a timeline yet. One workaround would be running a local apt-cacher-ng providing a local http repository proxied from the remote https repository.