Rebase to a new version of Fedora (N=45)
At previous Fedora major release
Open tickets to track related work for this release
At the first change checkpoint
Untag old packages
koji untag N-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process:
f32key=12c944d0
key=$f32key
echo > untaglist # create or empty out file
for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do
if koji buildinfo $build | grep -i $key 1>/dev/null; then
echo "Adding $build to untag list"
echo "${build}" >> untaglist
fi
done
Now we have a list of builds to untag. But we need a few more sanity checks.
f32key=12c944d0
key=$f32key
podman run -it --rm quay.io/fedora/fedora-coreos:testing-devel rpm -qai | grep -i -B 9 $key
podman rmi quay.io/fedora/fedora-coreos:testing-devel
If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed. One example of this is the shim RPM where the same build could be used for many Fedora releases. In this case you'll need to untag the RPM from coreos-pool, run a koji distrepo, which will remove that RPM from the repo metadata, and then re-tag it into the pool. The RPM in the repo will now be signed with a newer signing key.
# use xargs so we don't exhaust bash string limit
cat untaglist | xargs -L50 koji untag-build -v coreos-pool
At Branching
Branching is when a new stream is "branched" off of rawhide. This eventually becomes the next major Fedora (N).
Release engineering changes
coreos-installer changes
Example PR: coreos/coreos-installer#1113
Update rawhide stream
Enable branched stream
At Fedora (N) Beta
Prepare Fedora CoreOS (N) Beta announcement
Ship rebased next
Preparing for Fedora (N) GA
Do these steps as soon as we have a Go confirmation for GA, usually the Thursday of the week before GA.
Ship a final next release
If the packages in next-devel don't exactly match the last next release that was done, we need to do a release with the final GA content. This ensures that what we'll promote to testing has the exact content in GA (plus version fast-tracks). This usually happens on the Thursday of the announcement of Go.
Build rebased testing and final stable release on N-1
At Fedora (N) GA
Do these steps on GA day.
Release rebased testing and final stable release on N-1
Disable next-devel stream if not needed
We prefer to disable next-devel when there is no difference between testing-devel and next-devel. This allows us to prevent wasting a bunch of resources (bandwidth, storage, compute) for no reason. After the switch to N if next-devel and testing-devel are in lockstep, then disable next-devel.
Switch upstream packages to shipping release binaries from Fedora (N)
Disable the fedora-candidate-compose repo
After Fedora (N) GA
Ship rebased stable
Open ticket for the next Fedora rebase
Update the FCOS Meeting-Action Template
Now that Fedora N is GA, we need to start tracking the release schedule of Fedora N+1 during the weekly Fedora CoreOS Community Meeting.
Miscellaneous container updates
These are various containers in use throughout our ecosystem. We should update or open a ticket to track updating them once a new Fedora release is out. If you open a ticket instead of doing the update add a link to the ticket as comment.
Rebase to a new version of Fedora (N=45)
At previous Fedora major release
Open tickets to track related work for this release
At the first change checkpoint
Untag old packages
koji untagN-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process:Now we have a list of builds to untag. But we need a few more sanity checks.
Nbased FCOS. Check by running:If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed. One example of this is the shim RPM where the same build could be used for many Fedora releases. In this case you'll need to untag the RPM from
coreos-pool, run akoji distrepo, which will remove that RPM from the repo metadata, and then re-tag it into the pool. The RPM in the repo will now be signed with a newer signing key.Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit.
Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 33/34/35 keys. You'll most likely have to get someone from releng to run the second command (
edit-tag).koji taginfo coreos-poolkoji edit-tag coreos-pool -x tag2distrepo.keys="9570ff31 45719a39 9867c58f"At Branching
Branching is when a new stream is "branched" off of
rawhide. This eventually becomes the next major Fedora (N).Release engineering changes
Verify that a few tags were created when branching occurred:
f${N+1}-coreos-signing-pendingf${N+1}-coreos-continuousAdd and tag a package (any package) which is in the stable repos into the continuous tag. This will create the initial yum repo that's used as input for building the COSA container.
koji add-pkg --owner ${FAS_USERNAME} f${N+1}-coreos-continuous $PKGkoji add-pkg --owner dustymabe f36-coreos-continuous fedora-releasefedora-releaseRPM, but it could be any other.koji tag-build f${N+1}-coreos-continuous $BUILDkoji tag-build f36-coreos-continuous fedora-release-36-0.16Add the N+1 signing key short hash (usually found here) to the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 32/33/34/35 keys. You'll most likely have to get someone from releng to run the second command (
edit-tag). An example request looks like this.koji taginfo coreos-poolkoji edit-tag coreos-pool -x tag2distrepo.keys="12c944d0 9570ff31 45719a39 9867c58f"coreos-installer changes
Example PR: coreos/coreos-installer#1113
Update
rawhidestreamVERSION,MUTATE_OS_RELEASE,BUILDER_IMGin build-args.conf (example PR)Enable
branchedstreamVERSION,MUTATE_OS_RELEASE,BUILDER_IMGin build-args.conf (example PR)branchedstream definition (example PR)At Fedora (N) Beta
Update fedora-coreos-config
next-develBump
VERSION,MUTATE_OS_RELEASE,BUILDER_IMGinbuild-args.confAdd the
fedora-candidate-composerepo inmanifest.yaml(example PR)Update the repos in
manifest.yamlif neededRun
cosa fetch --dry-run --update-lockfilebump-lockfileruns.cosa fetchdirectlyPR the result
Re-enable
next-develif needed (docs)Disable
branchedstream since it is no longer needed.branchedstream definition.Prepare Fedora CoreOS (N) Beta announcement
Ship rebased
nextnextnext. In the barrier entry set a link to the docs. See discussionPreparing for Fedora (N) GA
Do these steps as soon as we have a Go confirmation for GA, usually the Thursday of the week before GA.
Ship a final
nextreleaseIf the packages in
next-develdon't exactly match the lastnextrelease that was done, we need to do a release with the final GA content. This ensures that what we'll promote totestinghas the exact content in GA (plus version fast-tracks). This usually happens on the Thursday of the announcement of Go.nextrelease has GA contentBuild rebased
testingand finalstablerelease on N-1stable; promote it from thetestingbranch, which should still be on N-1. Don't release it yet (i.e. don't run thereleasejob).testing; promote it from thenextbranch instead oftesting-devel. Don't release it yet (i.e. don't run thereleasejob).Update fedora-coreos-config
testing-develVERSION,MUTATE_OS_RELEASE,BUILDER_IMGinbuild-args.confmanifest.yamlif needednext-develci/buildroot/Dockerfiletests/kola/data/commonlib.shAt Fedora (N) GA
Do these steps on GA day.
Release rebased
testingand finalstablerelease on N-1releasejob for the stagedtestingandstablebuilds and start rollout.testing. In the barrier entry set a link to the docs. See discussionDisable
next-develstream if not neededWe prefer to disable
next-develwhen there is no difference betweentesting-develandnext-devel. This allows us to prevent wasting a bunch of resources (bandwidth, storage, compute) for no reason. After the switch to N ifnext-develandtesting-develare in lockstep, then disablenext-devel.next-develSwitch upstream packages to shipping release binaries from Fedora (N)
Disable the
fedora-candidate-composerepomanifest.yamlofnext-develthefedora-candidate-composerepoAfter Fedora (N) GA
Ship rebased
stablestablestable. In the barrier entry set a link to the docs. See discussionOpen ticket for the next Fedora rebase
FNlabel whereNis the Fedora version.Update the FCOS Meeting-Action Template
Now that Fedora N is GA, we need to start tracking the release schedule of Fedora N+1 during the weekly Fedora CoreOS Community Meeting.
Miscellaneous container updates
These are various containers in use throughout our ecosystem. We should update or open a ticket to track updating them once a new Fedora release is out. If you open a ticket instead of doing the update add a link to the ticket as comment.