The build of the RPM package in Fedora Copr failed, and opening builder-live.log.gz resulted in a 404 Not Found error.

[EvernightFedora]

I tried built the RPM package,Build Failed and click the builder-live.log.gz get 404 not found error.

[FrostyX]

Thank you for the report @EvernightFedora,
I think this may be #4184 and/or #4182 and/or #3116. @praiskup is working on this.

[FrostyX]

@praiskup, this must be related to Pulp. The initial Pulp repository for the project was not created - https://console.redhat.com/api/pulp-content/public-copr/

Does this actually have the same root cause as #4186?

[mineiro]

Hi, I'm hitting what looks like the same root cause on my project (mineiro/hyprland).

I recently enabled aarch64 on it, and all builds fail immediately — the backend spends ~60s polling for the copr_base repo and then gives up:

 Backend process error: Giving up waiting for copr_base repository, please try to
  manually regenerate the DNF repository (e.g. by 'copr-cli regenerate-repos <project_name>')

The repomd.xml for the aarch64 chroots was never created — they all 404 via Pulp:

$ curl -s -o /dev/null -w '%{http_code}' -L \
      'https://download.copr.fedorainfracloud.org/results/mineiro/hyprland/fedora-43-aarch64/repodata/repomd.xml'
  404

The x86_64 chroots work fine.

I ran copr-cli regenerate-repos mineiro/hyprland and also tried the website "Regenerate Repositories" button — neither created the missing aarch64 repo metadata. The project is confirmed "storage": "pulp".

Example failed build: https://copr.fedorainfracloud.org/coprs/mineiro/hyprland/build/10179486/

The builder-live.log.gz is also 404 since mock never actually starts — same symptom as the OP.

Just wanted to add another data point.

Best regards!

[FrostyX]

Thank you @mineiro.

Also, let me link a relevant discussion here - https://discussion.fedoraproject.org/t/failed-shortly-after-creating-a-new-copr-project-and-uploading-the-spec-file-to-start-the-build/182110

[FrostyX]

Sooo, my theory is that this is in fact the Pulp timeout problem. See, we now have a lot of waiting actions:

coprdb=# select count(*) from action where result=0;
┌───────┐
│ count │
├───────┤
│  1166 │
└───────┘
(1 row)

and the @mineiro action to add new chroots is among them:

coprdb=# select * from action where id=2558962;
┌─────────┬─────────────┬─────────────┬───────────┬───────────┬───────────┬────────┬─────────┬────────────┬──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│   id    │ action_type │ object_type │ object_id │ old_value │ new_value │ result │ message │ created_on │ ended_on │                                                                                                                 data
├─────────┼─────────────┼─────────────┼───────────┼───────────┼───────────┼────────┼─────────┼────────────┼──────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ 2558962 │           3 │ repository  │         0 │           │           │      0 │         │ 1772246166 │          │ {"ownername": "mineiro", "projectname": "hyprland", "project_dirnames": ["hyprland"], "chroots": ["fedora-43-aarch64",
└─────────┴─────────────┴─────────────┴───────────┴───────────┴───────────┴────────┴─────────┴────────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
(1 row)

I think there is nothing wrong with those actions, I think they are just not being processed by the action dispatcher, because we already have max number of actions running:

[root@copr-be ~][PROD]# ps -eo pid,etime,cmd |grep action
    571 80-15:14:23 [btrfs-transaction]
 369353 10-03:10:52 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2551144 --worker-id action_worker:2551144
 400793 61-22:39:42 tail -f /var/log/copr-backend/action_dispatcher.log /var/log/copr-backend/actions.log /var/log/copr-backend/analyze-results.log /var/log/copr-backend/backend.log /var/log/copr-backend/build_dispatcher.log /var/log/copr-backend/change-storage-delete.log /var/log/copr-backend/change-storage.log /var/log/copr-backend/hitcounter.log /var/log/copr-backend/hitcounter-logrotate.log /var/log/copr-backend/hitcounter-s3.log /var/log/copr-backend/logger.log /var/log/copr-backend/modifyrepo.log /var/log/copr-backend/pruner.log /var/log/copr-backend/spawner.log /var/log/copr-backend/terminator.log /var/log/copr-backend/vmm.log /var/log/copr-backend/worker.log
 849097 22-02:24:15 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2529000 --worker-id action_worker:2529000
 849099 22-02:24:15 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2529006 --worker-id action_worker:2529006
 849140 22-02:24:14 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2528996 --worker-id action_worker:2528996
 849144 22-02:24:14 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2528997 --worker-id action_worker:2528997
 849759 22-02:24:09 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2529001 --worker-id action_worker:2529001
 850857 22-02:23:50 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2529003 --worker-id action_worker:2529003
 850860 22-02:23:50 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2529002 --worker-id action_worker:2529002
 850863 22-02:23:50 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2529005 --worker-id action_worker:2529005
 864979 22-02:21:09 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2529224 --worker-id action_worker:2529224
 995170 22-01:56:25 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2530105 --worker-id action_worker:2530105
1205326 13-22:49:36 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2547534 --worker-id action_worker:2547534
1221272 22-00:58:17 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2532338 --worker-id action_worker:2532338 [locking] [locking] [locking] [locking] [locking] [locking]
1363827 22-00:24:40 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2533958 --worker-id action_worker:2533958
1416688 22-00:08:16 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2534796 --worker-id action_worker:2534796
1775075 21-22:36:06 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2536741 --worker-id action_worker:2536741
1775081 21-22:36:06 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2536740 --worker-id action_worker:2536740
2170166  1-11:40:33 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2558438 --worker-id action_worker:2558438
2888394       00:00 grep --color=auto action
3035766 11-08:25:35 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2549904 --worker-id action_worker:2549904 [locking] [locking] [locking] [locking]
3206463 10-06:56:48 /usr/bin/python3 /usr/bin/copr-backend-process-action --daemon --task-id 2551068 --worker-id action_worker:2551068

and they are all stuck for days ... This puts a big urgency on those timeout tickets now.