Summary

Period Beginning: 2021-09-22 07:48:34.025+00

Period Ending: 2022-12-03 10:38:36.204+00

Number of Jobs: 2691961

Node Descriptions

• UO Frank Node Descriptions

Shortcuts

• UO Runners, Last 100 Completed Jobs
• AWS Runners, Last 100 Completed Jobs

• Job Times, Last 4 Hours
• Job Times, Last Week
• Job Times, Overview, All


• Runner System Failures, by Runner, Last 4 Hours
• Runner System Failures, by Runner, Last Week


• Runner System Failures, by Type, Last 4 Hours
• Runner System Failures, by Type, Last Week

Job Times, Last 4 Hours

name	total_runtime	avg_runtime	n	pct_uo	pct_aws
rebuild-index	0.02 hr	0.97 min	1	100%	0%
rebuild, x86_64	18.59 hr	278.84 min	4	75%	25%

Job Times, Last Week

name	total_runtime	avg_runtime	n	pct_uo	pct_aws
sign-pkgs	19.79 hr	12.63 min	94	0%	100%
rebuild-index	43.27 hr	5.37 min	483	50%	38%
rebuild, x86_64	3900.52 hr	8.31 min	28167	31%	69%
rebuild, other	437.88 hr	5.69 min	4614	2%	85%
rebuild, aarch64	292.07 hr	6.13 min	2857	0%	100%
protected-publish	4.47 hr	22.33 min	12	0%	92%
protected-generate	41.51 hr	5.44 min	458	19%	80%
pr-generate	499.47 hr	7.09 min	4227	12%	82%
no-specs-to-rebuild	28.47 hr	0.45 min	3827	49%	46%

Job Times, Overview

name	total_runtime	avg_runtime	n	pct_uo	pct_aws
sign-pkgs	9.65 days	6.88 min	2078	0%	96%
rebuild-index	110.75 days	6.34 min	25194	70%	24%
rebuild, x86_64	7479.74 days	5.19 min	2076419	42%	57%
rebuild, ppc64le	156.21 days	8.81 min	25867	83%	0%
rebuild, other	635.51 days	6.07 min	150766	2%	89%
rebuild, aarch64	600.37 days	6.62 min	130650	0%	99%
protected-publish	12.66 days	70.67 min	258	0%	99%
protected-generate	30.86 days	5.07 min	8989	21%	75%
pr-generate	501.16 days	5.13 min	140834	35%	61%
other	20.80 days	4.34 min	6915	82%	17%
no-specs-to-rebuild	29.91 days	0.39 min	111246	62%	32%
copy-mirror	0.94 days	1.87 min	723	0%	100%
cleanup	8.56 days	1.03 min	12022	88%	12%

Runner System Failures, by Runner, Last 4 Hours

runner	n_jobs	n_system_failures	system_failure_rate	facility
empty

Runner System Failures, by Runner, Last Week

runner	n_jobs	n_system_failures	system_failure_rate	facility
spack-package-signing-gitlab-runner-6b75b5ffb5-ddqts	58	23	39.66%	other
spack-package-signing-gitlab-runner-6b75b5ffb5-72rjh	36	14	38.89%	other
x86-v4-pub-gitlab-runner-857fd77f57-8vbb6	404	55	13.61%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-j2m4c	354	36	10.17%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-8vwpl	146	14	9.59%	AWS
x86-v2-prot-gitlab-runner-644c6bd7f5-7hvmj	35	3	8.57%	other
x86-v2-pub-gitlab-runner-7684c89569-vqdd9	12	1	8.33%	AWS
x86-v2-prot-gitlab-runner-644c6bd7f5-9xdmc	84	7	8.33%	other
graviton2-pub-gitlab-runner-857859d58-2zbhg	73	6	8.22%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-lc8md	627	50	7.97%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-j5865	454	35	7.71%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-k7lb5	194	14	7.22%	AWS
x86-v3-prot-gitlab-runner-6b77445575-x98nr	14	1	7.14%	other
x86-v2-pub-gitlab-runner-7684c89569-c8szp	284	20	7.04%	AWS
x86-v2-pub-gitlab-runner-7684c89569-lpsmc	344	24	6.98%	AWS
x86-v2-prot-gitlab-runner-644c6bd7f5-scwzr	73	5	6.85%	other
x86-v2-prot-gitlab-runner-644c6bd7f5-mjsxf	98	6	6.12%	other
x86-v3-pub-gitlab-runner-6bfbdb987f-kcknv	135	8	5.93%	AWS
x86-v2-pub-gitlab-runner-7684c89569-f5cfb	441	25	5.67%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-p7mcm	18	1	5.56%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-bh595	345	19	5.51%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-87dd5	704	38	5.40%	AWS
x86-v2-prot-gitlab-runner-644c6bd7f5-pb6zv	56	3	5.36%	other
x86-v2-pub-gitlab-runner-7684c89569-x6wks	264	14	5.30%	AWS
x86-v2-pub-gitlab-runner-7684c89569-jb864	155	8	5.16%	AWS
x86-v2-pub-gitlab-runner-7684c89569-5zzvv	394	16	4.06%	AWS
x86-v2-prot-gitlab-runner-644c6bd7f5-slzdj	50	2	4.00%	other
x86-v2-pub-gitlab-runner-7684c89569-flrbv	291	11	3.78%	AWS
graviton2-pub-gitlab-runner-857859d58-bb8rg	82	3	3.66%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-298dh	830	30	3.61%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-9hz4f	827	29	3.51%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-lfxhh	350	11	3.14%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-s5l6r	845	26	3.08%	AWS
x86-v2-pub-gitlab-runner-7684c89569-fkdw6	500	15	3.00%	AWS
x86-v2-pub-gitlab-runner-7684c89569-kfn82	538	16	2.97%	AWS
graviton3-pub-gitlab-runner-5fb75f7574-kl89b	172	5	2.91%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-9qvlz	419	12	2.86%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-qrtnw	934	26	2.78%	AWS
x86-v4-prot-gitlab-runner-85fb75c797-wztz7	74	2	2.70%	other
x86-v2-pub-gitlab-runner-7684c89569-jndw6	489	12	2.45%	AWS
x86-v2-pub-gitlab-runner-7684c89569-7f6g5	83	2	2.41%	AWS
x86-v3-prot-gitlab-runner-6b77445575-bk9kg	42	1	2.38%	other
graviton3-prot-gitlab-runner-587cc887fb-f8c4k	216	5	2.31%	other
x86-v2-pub-gitlab-runner-7684c89569-vh6dc	87	2	2.30%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-6tx5n	1014	23	2.27%	AWS
graviton2-pub-gitlab-runner-857859d58-mqvb9	136	3	2.21%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-6z9ff	371	8	2.16%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-hnjbx	434	9	2.07%	AWS
x86-v4-prot-gitlab-runner-85fb75c797-6lhxv	305	6	1.97%	other
graviton3-pub-gitlab-runner-5fb75f7574-j2rpm	368	7	1.90%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-bv68c	363	6	1.65%	AWS
graviton3-pub-gitlab-runner-5fb75f7574-kqb57	248	4	1.61%	AWS
graviton3-pub-gitlab-runner-5fb75f7574-lqv6j	319	5	1.57%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-ksn4d	779	12	1.54%	AWS
graviton3-prot-gitlab-runner-587cc887fb-hdq52	65	1	1.54%	other
x86-v4-prot-gitlab-runner-85fb75c797-jkfnq	66	1	1.52%	other
graviton3-pub-gitlab-runner-5fb75f7574-kfz8l	334	5	1.50%	AWS
x86-v4-prot-gitlab-runner-85fb75c797-274pk	70	1	1.43%	other
graviton3-pub-gitlab-runner-5fb75f7574-k7drq	220	3	1.36%	AWS
x86-v4-prot-gitlab-runner-85fb75c797-2mkln	222	3	1.35%	other
graviton3-prot-gitlab-runner-587cc887fb-7vlcx	86	1	1.16%	other
x86-v3-prot-gitlab-runner-6b77445575-np2f8	87	1	1.15%	other
x86-v2-prot-gitlab-runner-644c6bd7f5-sk5fc	87	1	1.15%	other
x86-v2-pub-gitlab-runner-7684c89569-khcmf	352	4	1.14%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-vc2h2	877	10	1.14%	AWS
graviton3-pub-gitlab-runner-5fb75f7574-2b8zh	301	3	1.00%	AWS
graviton2-pub-gitlab-runner-857859d58-jtrtt	101	1	0.99%	AWS
graviton3-pub-gitlab-runner-5fb75f7574-tzrr2	302	3	0.99%	AWS
x86-v4-pub-gitlab-runner-857fd77f57-gbfrz	889	8	0.90%	AWS
x86-v3-pub-gitlab-runner-6bfbdb987f-bwmsk	114	1	0.88%	AWS
x86-v4-prot-gitlab-runner-85fb75c797-gkvpd	115	1	0.87%	other
x86-v4-pub-gitlab-runner-857fd77f57-kz4gr	812	7	0.86%	AWS
x86-v4-prot-gitlab-runner-85fb75c797-c7tsp	120	1	0.83%	other
uo-gilgamesh	1119	9	0.80%	UO
graviton3-prot-gitlab-runner-587cc887fb-k8f6r	136	1	0.74%	other
x86-v4-prot-gitlab-runner-85fb75c797-9dt9k	385	2	0.52%	other
graviton2-pub-gitlab-runner-857859d58-8x76p	197	1	0.51%	AWS
uo-illyad	1203	5	0.42%	UO
x86-v2-pub-gitlab-runner-7684c89569-mnm6q	251	1	0.40%	AWS
graviton2-pub-gitlab-runner-857859d58-w5v59	258	1	0.39%	AWS
x86-v4-prot-gitlab-runner-85fb75c797-9d2wl	568	2	0.35%	other
x86-v3-prot-gitlab-runner-6b77445575-bfl8c	300	1	0.33%	other

Runner System Failures, by Type, Last 4 Hours

failure_type	n	n_aws	n_uo	percent
empty

Runner System Failures, by Type, Last Week

failure_type	n	n_aws	n_uo	percent
pod-unready	541	541	0	66.14%
pod-not-found	256	256	0	31.30%
other	9	0	9	1.10%
dial-backend	6	6	0	0.73%
exit-137	3	0	3	0.37%
aborted-terminated	2	0	2	0.24%
connection-upgrade	1	1	0	0.12%