Summary

Period Beginning: 2021-09-22 07:48:34.025+00

Period Ending: 2022-06-24 23:27:35.05+00

Number of Jobs: 1365088

Number of Failed Jobs, all types: 44257

Number of Failed Jobs, system failures only: 11714

Node Descriptions

• UO Frank Node Descriptions

Shortcuts

• UO Runners, Last 100 Completed Jobs
• AWS Runners, Last 100 Completed Jobs

• Job Times, Last 4 Hours
• Job Times, Last Week
• Job Times, Overview, All
• Job Times, Detailed, All


• Runner System Failures, by Runner, Last 4 Hours
• Runner System Failures, by Runner, Last Week
• Runner System Failures, by Runner, All


• Runner System Failures, by Type, Last 4 Hours
• Runner System Failures, by Type, Last Week
• Runner System Failures, by Type, all


• Runner System Failures, Last 20

Job Times, Last 4 Hours

name	total_runtime	avg_runtime	n	pct_uo	pct_aws
empty

Job Times, Last Week

name	total_runtime	avg_runtime	n	pct_uo	pct_aws
rebuild	2187:33:56.39	00:10:14.91656	12807	26%	73%
generate	139:39:31.59	00:04:54.190515	1709	26%	74%
service	34:27:53.147	00:01:20.934864	1533	66%	30%
other	10:49:59.143	00:04:48.882541	135	0%	100%

Job Times, Overview

name	total_runtime	avg_runtime	n	pct_uo	pct_aws
rebuild	89961:16:10.286	00:04:21.437208	1239610	49%	50%
generate	4681:31:54.518	00:04:38.381832	60620	69%	31%
service	1601:19:04.67	00:01:30.065692	64082	83%	16%
other	45:57:26.77	00:03:38.267507	776	15%	82%

Job Times, Detailed

name	total_runtime	avg_runtime	n	pct_uo	pct_aws
rebuild	89961:16:10.286	00:04:21.437208	1239610	49%	50%
e4s-pr-generate	1851:32:29.875	00:11:17.875508	9833	71%	29%
rebuild-index	1145:30:40.426	00:05:08.716906	13387	86%	13%
radiuss-pr-generate	693:06:46.021	00:04:17.822486	9678	67%	33%
data-vis-sdk-pr-generate	424:07:42.458	00:02:36.376737	9764	65%	35%
tutorial-pr-generate	359:37:46.615	00:02:58.009984	7273	64%	36%
build_systems-pr-generate	351:52:56.345	00:02:09.052195	9816	68%	32%
e4s-develop-generate	275:17:38.763	00:11:40.890214	1414	84%	16%
no-specs-to-rebuild	250:28:33.342	00:00:23.325401	38673	81%	19%
cleanup	205:19:50.902	00:01:01.650617	12022	88%	12%
e4s-on-power-pr-generate	111:53:28.902	00:26:13.472273	256	95%	0%
radiuss-develop-generate	91:17:54.829	00:03:58.689055	1377	82%	18%
aws-ahug-aarch64-pr-generate	84:23:22.809	00:08:06.863476	624	0%	100%
aws-isc-aarch64-pr-generate	76:01:29.049	00:06:52.182303	664	0%	100%
aws-ahug-pr-generate	50:56:46.418	00:04:50.659933	631	88%	12%
data-vis-sdk-develop-generate	48:10:36.448	00:02:05.405964	1383	81%	19%
other	45:57:26.77	00:03:38.267507	776	15%	82%
build_systems-develop-generate	40:50:34.558	00:01:46.546781	1380	83%	17%
tutorial-develop-generate	36:56:25.243	00:02:15.28509	983	77%	23%
aws-isc-pr-generate	32:56:09.233	00:02:58.299598	665	88%	12%
e4s-mac-pr-generate	27:32:02.133	00:00:31.477337	3220	98%	0%
e4s-protected-generate	16:28:56.635	00:07:13.114124	137	0%	100%
aws-isc-aarch64-protected-generate	16:02:39.862	00:08:45.089655	110	0%	100%
aws-ahug-aarch64-protected-generate	13:00:22.113	00:07:09.56067	109	0%	100%
tutorial-protected-generate	10:02:01.959	00:04:16.184106	141	0%	100%
aws-e4s-pr-generate	09:53:15.981	00:08:51.283299	67	79%	21%
radiuss-protected-generate	09:23:39.875	00:04:05.071558	138	0%	100%
data-vis-sdk-protected-generate	08:55:46.949	00:03:52.948906	138	0%	100%
python-pr-generate	08:08:29.55	00:22:12.252273	22	59%	41%
aws-ahug-protected-generate	07:35:30.002	00:04:13.055574	108	86%	14%
build_systems-protected-generate	06:18:36.26	00:02:47.031324	136	0%	100%
aws-e4s-aarch64-pr-generate	05:37:58.224	00:10:33.6945	32	0%	100%
aws-isc-protected-generate	05:13:34.23	00:02:54.205833	108	86%	14%
e4s-on-power-develop-generate	03:49:57.882	00:32:51.126	7	71%	0%
e4s-mac-develop-generate	01:42:10.363	00:00:25.437191	249	97%	0%
e4s-mac-protected-generate	01:29:08.582	00:00:41.142938	130	100%	0%
radiuss-aws-aarch64-pr-generate	00:24:26.641	00:04:04.440167	6	0%	100%
dev-tools-pr-generate	00:24:11.16	00:02:00.93	12	67%	33%
aws-e4s-aarch64-develop-generate	00:10:46.19	00:10:46.19	1	0%	100%
radiuss-aws-pr-generate	00:08:56.63	00:01:29.438333	6	100%	0%
aws-e4s-develop-generate	00:07:01.841	00:07:01.841	1	100%	0%
dev_tools-pr-generate	00:00:13.89	00:00:13.89	1	100%	0%

Runner System Failures, by Runner, Last 4 Hours

runner	n_jobs	n_system_failures	system_failure_rate	facility
empty

Runner System Failures, by Runner, Last Week

runner	n_jobs	n_system_failures	system_failure_rate	facility
large-x86-pub-gitlab-runner-7c5d6645c5-s85zc	1029	19	1.85%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-qqlfl	942	10	1.06%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-psbtj	933	33	3.54%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-7g6cr	913	3	0.33%	AWS
large-arm-pub-gitlab-runner-b58985c75-vd845	890	15	1.69%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-n7pjx	822	15	1.82%	AWS
large-arm-pub-gitlab-runner-b58985c75-gc2pr	713	6	0.84%	AWS
large-arm-pub-gitlab-runner-b58985c75-89sjg	699	6	0.86%	AWS
large-arm-pub-gitlab-runner-b58985c75-h8psf	622	4	0.64%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-fkmnz	530	6	1.13%	AWS
large-arm-pub-gitlab-runner-b58985c75-t6mw7	508	3	0.59%	AWS
large-arm-pub-gitlab-runner-b58985c75-krvws	456	1	0.22%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-xmzrl	401	2	0.50%	AWS
uo-godzilla	244	1	0.41%	UO
large-arm-prot-gitlab-runner-6f778bd5b9-wzfxf	240	6	2.50%	other
spack-omnia	231	4	1.73%	other
large-arm-prot-gitlab-runner-6f778bd5b9-p86n6	190	2	1.05%	other

Runner System Failures, by Runner

runner	n_jobs	n_system_failures	pct_system_failures	facility
large-x86-pub-gitlab-runner-75d954648d-pphmp	79318	201	0.25%	AWS
large-x86-pub-gitlab-runner-64fcfb66b-w2gnl	56770	430	0.76%	AWS
large-x86-pub-gitlab-runner-788666b4-hd469	38800	127	0.33%	AWS
large-x86-pub-gitlab-runner-586fc54475-5d6kr	30678	133	0.43%	AWS
large-x86-pub-gitlab-runner-586fc54475-qb5kt	16547	50	0.30%	AWS
large-x86-pub-gitlab-runner-64fcfb66b-28pd5	16326	277	1.70%	AWS
large-x86-pub-gitlab-runner-596fbc486b-4j9sc	14584	543	3.72%	AWS
large-x86-pub-gitlab-runner-596fbc486b-mpv22	14441	533	3.69%	AWS
large-x86-pub-gitlab-runner-596fbc486b-46rjf	14427	533	3.69%	AWS
large-x86-pub-gitlab-runner-596fbc486b-9v8tl	14109	511	3.62%	AWS
large-x86-pub-gitlab-runner-596fbc486b-wthd7	11990	494	4.12%	AWS
large-x86-pub-gitlab-runner-856f65bf79-9wx9g	9086	418	4.60%	AWS
xlarge-x86-pub-gitlab-runner-f9f45999f-ld7hk	7653	14	0.18%	AWS
large-x86-pub-gitlab-runner-596fbc486b-vcwxm	6435	426	6.62%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-n7pjx	4845	45	0.93%	AWS
xlarge-x86-pub-gitlab-runner-7c58cc9d47-k87vf	4629	181	3.91%	AWS
medium-x86-pub-gitlab-runner-5b4ddf9758-n62pg	4563	95	2.08%	AWS
medium-x86-pub-gitlab-runner-5b4ddf9758-bxjvf	4002	110	2.75%	AWS
large-x86-pub-gitlab-runner-586fc54475-fr4vn	3627	4	0.11%	AWS
large-x86-pub-gitlab-runner-596fbc486b-phm94	3188	32	1.00%	AWS
xlarge-x86-pub-gitlab-runner-5d8f87c7dd-mklgd	3101	9	0.29%	AWS
large-x86-pub-gitlab-runner-75d954648d-9528c	2940	4	0.14%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-psbtj	2806	64	2.28%	AWS
large-x86-pub-gitlab-runner-546548c6d8-jj8v8	2795	29	1.04%	AWS
large-arm-pub-gitlab-runner-b58985c75-vd845	2732	22	0.81%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-s85zc	2729	53	1.94%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-7g6cr	2659	10	0.38%	AWS
large-x86-pub-gitlab-runner-586fc54475-5lktb	2548	2	0.08%	AWS
large-x86-pub-gitlab-runner-596fbc486b-xxktt	2459	44	1.79%	AWS
large-arm-pub-gitlab-runner-b58985c75-ztndw	2427	23	0.95%	AWS
large-x86-pub-gitlab-runner-596fbc486b-f88ns	2398	36	1.50%	AWS
large-arm-pub-gitlab-runner-b58985c75-89sjg	2242	20	0.89%	AWS
large-arm-pub-gitlab-runner-b58985c75-gc2pr	2158	14	0.65%	AWS
large-arm-pub-gitlab-runner-b58985c75-pwnkt	2158	10	0.46%	AWS
large-arm-pub-gitlab-runner-b58985c75-g728s	2154	34	1.58%	AWS
large-arm-pub-gitlab-runner-b58985c75-xgvnw	2147	25	1.16%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-99j84	2142	14	0.65%	AWS
large-arm-pub-gitlab-runner-b58985c75-h8psf	2123	17	0.80%	AWS
large-arm-pub-gitlab-runner-b58985c75-q6rsj	2103	27	1.28%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-fkmnz	2089	42	2.01%	AWS
large-arm-pub-gitlab-runner-b58985c75-zg996	1991	42	2.11%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-t4zvc	1917	8	0.42%	AWS
large-x86-pub-gitlab-runner-596fbc486b-s98p5	1881	38	2.02%	AWS
medium-x86-pub-gitlab-runner-6d6b484d7c-dx4cn	1841	5	0.27%	AWS
xlarge-x86-pub-gitlab-runner-7c58cc9d47-6rh2n	1802	106	5.88%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-65bwx	1771	8	0.45%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-6lf8j	1764	0	0.00%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-pk5cs	1711	33	1.93%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-q4dl7	1697	4	0.24%	AWS
large-arm-pub-gitlab-runner-b58985c75-t6mw7	1666	13	0.78%	AWS
medium-x86-pub-gitlab-runner-77f9fc59f6-22fv5	1642	33	2.01%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-69qbw	1523	10	0.66%	AWS
medium-x86-pub-gitlab-runner-5b4ddf9758-9qjtp	1400	114	8.14%	AWS
large-arm-pub-gitlab-runner-b58985c75-krvws	1370	9	0.66%	AWS
xlarge-x86-pub-gitlab-runner-78bf7464f5-fz8sp	1352	14	1.04%	AWS
glr-testing-x86-pub-gitlab-runner-67c47869cb-k5tqv	1315	7	0.53%	AWS
medium-x86-pub-gitlab-runner-5449c66658-86bl2	1297	198	15.27%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-zrkfc	1290	7	0.54%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-qrpd2	1277	11	0.86%	AWS
glr-testing-x86-pub-gitlab-runner-67c47869cb-zl94w	1203	8	0.67%	AWS
large-x86-pub-gitlab-runner-586fc54475-mm59w	1178	204	17.32%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-qqlfl	1163	10	0.86%	AWS
xlarge-x86-pub-gitlab-runner-5d8f87c7dd-5rbmf	1052	5	0.48%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-b252v	938	5	0.53%	AWS
large-x86-pub-gitlab-runner-546548c6d8-8gzhq	901	829	92.01%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-qkm5p	852	1	0.12%	AWS
large-x86-pub-gitlab-runner-5fd66c6468-95xmb	841	838	99.64%	AWS
glr-testing-x86-pub-gitlab-runner-766f5dc44c-5ld8t	824	12	1.46%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-gtqlt	717	3	0.42%	AWS
huge-x86-pub-gitlab-runner-6ff68d7cdf-tfr6v	675	5	0.74%	AWS
xlarge-x86-pub-gitlab-runner-5d8f87c7dd-f8bz2	639	6	0.94%	AWS
glr-testing-x86-pub-gitlab-runner-55f6847674-8rdlz	585	1	0.17%	AWS
huge-x86-pub-gitlab-runner-54cb9dd65f-74hbd	551	550	99.82%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-mlzwx	518	0	0.00%	AWS
xlarge-x86-pub-gitlab-runner-988db589c-rh9hr	489	49	10.02%	AWS
large-x86-pub-gitlab-runner-586fc54475-cnfk8	454	46	10.13%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-hrzrj	443	7	1.58%	AWS
medium-x86-pub-gitlab-runner-d7ff69878-9w9t7	411	3	0.73%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-xmzrl	401	2	0.50%	AWS
xlarge-x86-pub-gitlab-runner-6ffc5954f-fpgmg	399	7	1.75%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-5s5p2	383	0	0.00%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-55bgd	366	0	0.00%	AWS
huge-x86-pub-gitlab-runner-56dcc9f4d4-8pn85	351	3	0.85%	AWS
huge-x86-pub-gitlab-runner-658c9ff576-h7czw	301	6	1.99%	AWS
large-x86-pub-gitlab-runner-6564fcd648-nx89m	260	3	1.15%	AWS
large-x86-pub-gitlab-runner-586fc54475-9zpvs	255	0	0.00%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-wqmxz	250	5	2.00%	AWS
large-x86-pub-gitlab-runner-596fbc486b-gvdt6	248	16	6.45%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-n8cp8	248	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-dcxdk	235	6	2.55%	AWS
xlarge-x86-pub-gitlab-runner-7c58cc9d47-7b4sl	200	3	1.50%	AWS
large-arm-pub-gitlab-runner-b58985c75-9b8gk	187	0	0.00%	AWS
large-x86-pub-gitlab-runner-596fbc486b-hpgg2	183	12	6.56%	AWS
xlarge-x86-pub-gitlab-runner-6b59cc5b7f-s9g9j	173	8	4.62%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-h86mf	172	0	0.00%	AWS
large-x86-pub-gitlab-runner-5fd66c6468-24jfs	169	0	0.00%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-w4xsw	163	17	10.43%	AWS
medium-x86-pub-gitlab-runner-79488dbc4-mgvxz	162	0	0.00%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-r7l8m	145	0	0.00%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-n7kgs	131	0	0.00%	AWS
large-x86-pub-gitlab-runner-84589cff4f-swxl9	129	1	0.78%	AWS
huge-x86-pub-gitlab-runner-854685d854-klg2k	114	56	49.12%	AWS
huge-x86-pub-gitlab-runner-68c974fc48-z8sr5	113	1	0.88%	AWS
huge-x86-pub-gitlab-runner-854685d854-ndl59	94	0	0.00%	AWS
large-x86-pub-gitlab-runner-596fbc486b-ctnv7	86	0	0.00%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-dlb5k	81	0	0.00%	AWS
medium-x86-pub-gitlab-runner-75b685dd9f-f4dll	79	79	100.00%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-lj92t	66	0	0.00%	AWS
xlarge-x86-pub-gitlab-runner-89c9d4c99-w9xdl	66	66	100.00%	AWS
large-x86-pub-gitlab-runner-6cf4d66cf8-57fnf	55	1	1.82%	AWS
large-x86-pub-gitlab-runner-6cf4d66cf8-tb2v6	55	2	3.64%	AWS
large-x86-pub-gitlab-runner-6cf4d66cf8-mvtcd	47	6	12.77%	AWS
large-x86-pub-gitlab-runner-6cf4d66cf8-r7fhn	43	1	2.33%	AWS
large-x86-pub-gitlab-runner-6cf4d66cf8-mrgd8	39	1	2.56%	AWS
huge-x86-pub-gitlab-runner-854685d854-hlwlj	38	1	2.63%	AWS
large-x86-pub-gitlab-runner-6cf4d66cf8-tb2bt	38	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-59vm8	38	0	0.00%	AWS
large-x86-tmp-pub-gitlab-runner-8576848c95-cfhc2	37	0	0.00%	AWS
large-x86-pub-gitlab-runner-6564fcd648-7n9nb	36	0	0.00%	AWS
huge-x86-pub-gitlab-runner-5c76cc9d55-78l2n	35	0	0.00%	AWS
large-x86-tmp-pub-gitlab-runner-66494cc84c-k2r2v	32	1	3.13%	AWS
large-x86-tmp-pub-gitlab-runner-5674bd848-7bmm7	31	0	0.00%	AWS
large-x86-testing-pub-gitlab-runner-d7589dcb4-bdkkt	28	24	85.71%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-k6jtv	28	0	0.00%	AWS
huge-x86-pub-gitlab-runner-85854fd676-xb62d	28	28	100.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-zpcd5	25	0	0.00%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-nrptl	24	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-2jn2t	18	0	0.00%	AWS
huge-x86-pub-gitlab-runner-854685d854-b9477	18	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-ljklb	17	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-bkns2	17	0	0.00%	AWS
large-x86-pub-gitlab-runner-5fd66c6468-cppwp	17	17	100.00%	AWS
glr-testing-x86-pub-gitlab-runner-5d75b975bb-6r5wg	15	15	100.00%	AWS
large-x86-pub-gitlab-runner-7c5d6645c5-ggxjb	15	0	0.00%	AWS
xlarge-x86-pub-gitlab-runner-89c9d4c99-jtts2	15	15	100.00%	AWS
large-x86-pub-gitlab-runner-75d954648d-qxft4	12	0	0.00%	AWS
huge-x86-pub-gitlab-runner-854685d854-4bqgt	11	0	0.00%	AWS
glr-testing-x86-pub-gitlab-runner-7f9b984b56-dpbk2	10	10	100.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-bpd6c	9	0	0.00%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-b94l5	8	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-mbbl2	7	1	14.29%	AWS
huge-x86-pub-gitlab-runner-5c76cc9d55-5ql99	7	1	14.29%	AWS
large-x86-tmp-pub-gitlab-runner-6d66cfcf79-xpmmx	7	0	0.00%	AWS
huge-x86-pub-gitlab-runner-b4d474c54-xvbql	7	3	42.86%	AWS
huge-x86-pub-gitlab-runner-854685d854-sh2hj	7	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-zfptz	4	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-rmmqx	4	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-b4dps	3	0	0.00%	AWS
medium-x86-pub-gitlab-runner-75b685dd9f-5fqxf	3	3	100.00%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-j6nb6	2	0	0.00%	AWS
huge-x86-pub-gitlab-runner-5c76cc9d55-kvdg2	2	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-jnjfz	2	0	0.00%	AWS
huge-x86-pub-gitlab-runner-b4d474c54-qwzxh	2	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-kw5dk	1	0	0.00%	AWS
large-x86-pub-gitlab-runner-546548c6d8-n89h5	1	0	0.00%	AWS
huge-x86-pub-gitlab-runner-5c76cc9d55-6qw7g	1	0	0.00%	AWS
huge-x86-pub-gitlab-runner-5c76cc9d55-llzrg	1	0	0.00%	AWS
large-x86-tmp-pub-gitlab-runner-f4c8644d6-rkbck	1	0	0.00%	AWS
medium-x86-pub-gitlab-runner-5576f87d5b-7cg8q	1	1	100.00%	AWS
huge-x86-pub-gitlab-runner-5c76cc9d55-69lbs	1	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-vl7x4	1	0	0.00%	AWS
medium-x86-pub-gitlab-runner-6dd47d4f9d-rx2j6	1	0	0.00%	AWS
uo-jupiter	108119	19	0.02%	UO
uo-gilgamesh	96897	35	0.04%	UO
uo-illyad	93711	454	0.48%	UO
uo-instinct	77323	23	0.03%	UO
uo-saturn	65648	9	0.01%	UO
uo-reptar	58876	6	0.01%	UO
uo-godzilla	55892	138	0.25%	UO
uo-voltar	37540	13	0.03%	UO
uo-cygnus	26205	2	0.01%	UO
uo-mothra	25431	3	0.01%	UO
uo-delphi	24389	10	0.04%	UO
uo-pegasus	16475	0	0.00%	UO
uo-omicron	7017	0	0.00%	UO
uo-eagle	3370	1	0.03%	UO
uo-vina	3210	0	0.00%	UO
uo-minotaur	1816	13	0.72%	UO
uo-centaur	1671	34	2.03%	UO
uo-typhon	13	0	0.00%	UO
uo-medusa	9	0	0.00%	UO
large-x86-pub-4-gitlab-runner-5c4d87479c-m8qc5	33825	173	0.51%	other
large-x86-pub-2-gitlab-runner-85bdcf44b-m644j	33208	165	0.50%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-5m76d	32838	135	0.41%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-ltnls	16464	48	0.29%	other
large-x86-pub-2-gitlab-runner-85bdcf44b-t4h84	11708	272	2.32%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-x4kvn	11031	275	2.49%	other
large-x86-pub-2-gitlab-runner-85bdcf44b-tmxpz	9866	34	0.34%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-8jbkk	5069	41	0.81%	other
large-x86-prot-gitlab-runner-677dccf647-snhlt	3711	8	0.22%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-f2hhf	2863	13	0.45%	other
large-x86-prot-gitlab-runner-677dccf647-zrsq8	2744	3	0.11%	other
large-x86-prot-gitlab-runner-677dccf647-kgbwc	2181	3	0.14%	other
large-x86-prot-gitlab-runner-677dccf647-nfq5t	2120	5	0.24%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-7w6q6	2065	6	0.29%	other
large-x86-prot-gitlab-runner-677dccf647-cqzbq	1937	4	0.21%	other
large-arm-prot-gitlab-runner-6f778bd5b9-xkwzj	1706	11	0.64%	other
large-arm-prot-gitlab-runner-6f778bd5b9-wzfxf	1661	14	0.84%	other
large-x86-prot-gitlab-runner-677dccf647-t57b2	1609	0	0.00%	other
large-x86-prot-gitlab-runner-677dccf647-c4jgf	1604	1	0.06%	other
large-x86-prot-gitlab-runner-677dccf647-lwml5	1586	0	0.00%	other
large-x86-prot-gitlab-runner-677dccf647-7zstx	1529	0	0.00%	other
large-x86-prot-gitlab-runner-677dccf647-j4mcs	1410	0	0.00%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-f2k6z	1112	233	20.95%	other
large-arm-prot-gitlab-runner-6f778bd5b9-h2qhw	890	1	0.11%	other
large-arm-prot-gitlab-runner-6f778bd5b9-p86n6	760	7	0.92%	other
large-x86-prot-gitlab-runner-677dccf647-qvzsh	749	0	0.00%	other
large-x86-prot-gitlab-runner-64f7f5d6cf-j9zcj	731	0	0.00%	other
large-x86-prot-gitlab-runner-64f7f5d6cf-7vcxl	725	1	0.14%	other
large-x86-prot-gitlab-runner-677dccf647-tsfjh	687	2	0.29%	other
large-x86-prot-gitlab-runner-64f7f5d6cf-sd9n8	675	1	0.15%	other
large-x86-prot-gitlab-runner-64f7f5d6cf-hw6rz	673	0	0.00%	other
large-x86-prot-gitlab-runner-64f7f5d6cf-rn9s4	558	1	0.18%	other
large-x86-prot-gitlab-runner-64f7f5d6cf-9n87j	484	0	0.00%	other
large-x86-prot-gitlab-runner-677dccf647-ntt4q	353	0	0.00%	other
spack-omnia	308	4	1.30%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-6nfxm	307	0	0.00%	other
spack-package-signing-gitlab-runner-657765f47f-7jf9p	246	0	0.00%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-jzlv8	193	0	0.00%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-qfqcq	84	1	1.19%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-7jlwk	52	0	0.00%	other
spack-package-signing-gitlab-runner-657765f47f-rczgn	40	0	0.00%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-jmsw4	35	0	0.00%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-bh4q4	29	0	0.00%	other
large-x86-pub-3-gitlab-runner-fbdbc6b99-t62mw	19	0	0.00%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-kqxrb	15	0	0.00%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-hpswz	12	0	0.00%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-b7b5f	11	0	0.00%	other
large-x86-pub-4-gitlab-runner-5c4d87479c-p9fnt	4	0	0.00%	other
spack-package-signing-gitlab-runner-657765f47f-6jjrb	1	0	0.00%	other
spack-package-signing-gitlab-runner-657765f47f-hsr7g	1	0	0.00%	other

Runner System Failures, by Type, Last 4 Hours

failure_type	n	n_aws	n_uo	percent
empty

Runner System Failures, by Type, Last Week

failure_type	n	n_aws	n_uo	percent
pod-unready	84	84	0	61.76%
pod-not-found	42	42	0	30.88%
connection-upgrade	5	5	0	3.68%
aborted-terminated	4	0	0	2.94%
docker-daemon	1	0	1	0.74%

Runner System Failures, by Type

failure_type	n	n_aws	n_uo	percent
pod-unready	7494	7455	0	63.97%
pod-not-found	1592	1578	0	13.59%
configmap-failed	1133	1019	0	9.67%
docker-daemon	656	0	656	5.60%
dial-backend	505	501	0	4.31%
exit-137	126	80	46	1.08%
connection-upgrade	93	93	0	0.79%
aborted-terminated	42	0	38	0.36%
other	38	24	13	0.32%
pull-failed	28	28	0	0.24%
no-space-on-device	7	0	7	0.06%

Runner System Failures, Last 20

id	name	finished_at	web_url	failure_type	runner
2722128	(specs) openfoam/gxgqqi3 2112_220610 gcc@7.3.1 linux-amzn2-graviton2 AWS Packages	2022-06-24 11:25:55.684+00	https://gitlab.spack.io/spack/spack/-/jobs/2722128	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-vd845
2722378	(specs) qmcpack/23yh7w2 3.14.0 gcc@7.3.1 linux-amzn2-graviton2 AHUG ARM HPC User Group	2022-06-24 09:40:52.431+00	https://gitlab.spack.io/spack/spack/-/jobs/2722378	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-vd845
2722125	(specs) salmon/ol6vr2u 1.4.0 gcc@7.3.1 linux-amzn2-aarch64 AWS Packages	2022-06-24 09:35:52.626+00	https://gitlab.spack.io/spack/spack/-/jobs/2722125	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-vd845
2722401	(specs) elmerfem/gku35u7 9.0 gcc@7.3.1 linux-amzn2-aarch64 AHUG ARM HPC User Group	2022-06-24 09:35:52.049+00	https://gitlab.spack.io/spack/spack/-/jobs/2722401	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-vd845
2722367	(specs) abyss/ufggmb2 2.3.1 gcc@7.3.1 linux-amzn2-graviton2 AHUG ARM HPC User Group	2022-06-24 09:25:51.252+00	https://gitlab.spack.io/spack/spack/-/jobs/2722367	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-t6mw7
2722113	(specs) flux-sched/pqez67d 0.23.0 gcc@7.3.1 linux-amzn2-aarch64 AWS Packages	2022-06-24 09:20:53.091+00	https://gitlab.spack.io/spack/spack/-/jobs/2722113	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-krvws
2722109	(specs) cgal/lmvd2fg 4.13 gcc@7.3.1 linux-amzn2-aarch64 AWS Packages	2022-06-24 09:20:52.62+00	https://gitlab.spack.io/spack/spack/-/jobs/2722109	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-vd845
2722124	(specs) salmon/i6dvn4u 1.4.0 gcc@7.3.1 linux-amzn2-graviton2 AWS Packages	2022-06-24 09:20:52.262+00	https://gitlab.spack.io/spack/spack/-/jobs/2722124	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-vd845
2720241	(specs) warpx/jjnt23l 22.06 gcc@7.5.0 linux-ubuntu18.04-x86_64 Spack E4S CI Testing	2022-06-23 19:18:36.634+00	https://gitlab.spack.io/spack/spack/-/jobs/2720241	pod-not-found	large-x86-pub-gitlab-runner-7c5d6645c5-xmzrl
2721290	(specs) pruners-ninja/ohdujek 1.0.1 gcc@11.2.0 linux-ubuntu22.04-x86_64 New PR testing workflow	2022-06-23 19:13:24.435+00	https://gitlab.spack.io/spack/spack/-/jobs/2721290	aborted-terminated	spack-omnia
2720753	(specs) py-requests/mauszai 2.26.0 gcc@11.2.0 linux-ubuntu22.04-x86_64 New PR testing workflow	2022-06-23 19:13:23.474+00	https://gitlab.spack.io/spack/spack/-/jobs/2720753	aborted-terminated	spack-omnia
2720190	(specs) slate/5xsllli 2022.06.00 gcc@7.5.0 linux-ubuntu18.04-x86_64 Spack E4S CI Testing	2022-06-23 19:13:22.778+00	https://gitlab.spack.io/spack/spack/-/jobs/2720190	aborted-terminated	spack-omnia
2720312	(specs) py-hpbandster/wc44kk3 0.7.4 gcc@7.5.0 linux-ubuntu18.04-x86_64 Spack E4S CI Testing	2022-06-23 19:13:22.23+00	https://gitlab.spack.io/spack/spack/-/jobs/2720312	aborted-terminated	spack-omnia
2720708	(specs) mercury/p7jzykh 2.1.0 gcc@11.2.0 linux-ubuntu22.04-x86_64 New PR testing workflow	2022-06-23 19:06:09.342+00	https://gitlab.spack.io/spack/spack/-/jobs/2720708	pod-not-found	large-x86-pub-gitlab-runner-7c5d6645c5-psbtj
2720709	(specs) metall/wpwgtx6 0.20 gcc@11.2.0 linux-ubuntu22.04-x86_64 New PR testing workflow	2022-06-23 19:06:07.161+00	https://gitlab.spack.io/spack/spack/-/jobs/2720709	pod-not-found	large-x86-pub-gitlab-runner-7c5d6645c5-psbtj
2719956	(specs) adios2/y6yv2fr 2.8.1 gcc@7.5.0 linux-ubuntu18.04-x86_64 Spack E4S CI Testing	2022-06-23 18:56:20.001+00	https://gitlab.spack.io/spack/spack/-/jobs/2719956	pod-unready	large-x86-pub-gitlab-runner-7c5d6645c5-psbtj
2720735	(specs) py-kiwisolver/i2l3ngq 1.3.2 gcc@11.2.0 linux-ubuntu22.04-x86_64 New PR testing workflow	2022-06-23 18:55:31.878+00	https://gitlab.spack.io/spack/spack/-/jobs/2720735	pod-not-found	large-x86-pub-gitlab-runner-7c5d6645c5-s85zc
2721153	(specs) gotcha/q6ucpu7 1.0.3 gcc@11.2.0 linux-ubuntu22.04-x86_64 New PR testing workflow	2022-06-23 17:54:35.537+00	https://gitlab.spack.io/spack/spack/-/jobs/2721153	docker-daemon	uo-godzilla
2719303	(specs) salmon/pyao5zx 1.4.0 gcc@7.3.1 linux-amzn2-aarch64 AWS Packages	2022-06-23 17:20:30.3+00	https://gitlab.spack.io/spack/spack/-/jobs/2719303	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-vd845
2719726	aws-isc-aarch64-pr-generate	2022-06-23 17:20:30.256+00	https://gitlab.spack.io/spack/spack/-/jobs/2719726	pod-not-found	large-arm-pub-gitlab-runner-b58985c75-h8psf