Summary

Beginning: 2021-09-22 12:48 AM PDT
Ending: 2026-02-24 07:50 PM PDT
Total Jobs: 10,268,053

Navigation

Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
GPU Usage, UO, Last Week
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours

GPU Usage, UO, Last Week

GPU Type	# Jobs
AMD MI100	168
AMD MI210	29
AMD MI300A	90
AMD MI50	14
INTEL A770	133
INTEL DATA CENTER MAX 1100	246
INTEL DG1	28
NVIDIA A100	60
NVIDIA A2000	14
NVIDIA GH200	87
NVIDIA H100	18
NVIDIA RTX5080	83

Job Times, Last 24 Hours

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AWS Packages	3.02 hr	0.4%	8.62 min	21	29%	71%
App CI - E4S - Adios2	5.77 hr	0.8%	1.03 min	335	90%	0%
App CI - E4S - ExaGO	.07 hr	0.0%	1.50 min	3	100%	0%
App CI - E4S - ExaWind	1.38 hr	0.2%	82.65 min	1	100%	0%
App CI - E4S - HPCToolkit	.31 hr	0.0%	1.56 min	12	100%	0%
App CI - E4S - Kokkos	83.95 hr	12.3%	39.05 min	129	100%	0%
App CI - E4S - TAU	30.44 hr	4.5%	83.01 min	22	100%	0%
App CI - E4S - Trilinos	9.64 hr	1.4%	44.51 min	13	100%	0%
Build Systems	.34 hr	0.1%	10.30 min	2	50%	50%
Copy	.02 hr	0.0%	1.16 min	1	0%	100%
Data and Vis SDK	35.41 hr	5.2%	30.79 min	69	12%	88%
Developer Tools aarch64-linux-gnu	.6 hr	0.1%	12.08 min	3	33%	67%
Developer Tools x86_64_v3-linux-gnu	.81 hr	0.1%	16.16 min	3	0%	100%
Developer Tools, Darwin	.57 hr	0.1%	17.03 min	2	100%	0%
E4S	241.53 hr	35.5%	11.93 min	1,215	15%	85%
E4S ARM Neoverse V2	25.85 hr	3.8%	4.53 min	342	27%	73%
E4S OneAPI	24.19 hr	3.6%	8.74 min	166	19%	81%
E4S ROCm External	3.84 hr	0.6%	7.95 min	29	10%	90%
High Energy Physics	85.15 hr	12.5%	6.66 min	767	20%	80%
Machine Learning	68.46 hr	10.1%	5.84 min	703	51%	49%
RADIUSS	9.36 hr	1.4%	7.69 min	73	34%	66%
Spack CI	.08 hr	0.0%	4.91 min	1	100%	0%
Spack Tutorial	3.08 hr	0.5%	3.19 min	58	9%	91%
Windows Visualization (Kitware)	7.7 hr	1.1%	17.77 min	26	0%	100%
generate	25.29 hr	3.7%	3.45 min	440	68%	32%
no-specs-to-rebuild	2.6 hr	0.4%	0.27 min	573	0%	100%
other	.48 hr	0.1%	1.20 min	24	4%	96%
rebuild-index	7.12 hr	1.0%	5.03 min	85	0%	100%
sign-pkgs	2.73 hr	0.4%	12.59 min	13	0%	100%
TOTAL	679.8 hr	--	--	5,131	32%	67%

Job Times, Last Week

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AWS Packages	31.69 hr	0.5%	4.55 min	418	20%	80%
App CI - E4S - Adios2	39.64 hr	0.7%	1.01 min	2,361	89%	0%
App CI - E4S - ExaGO	.46 hr	0.0%	1.63 min	17	100%	0%
App CI - E4S - ExaWind	9.78 hr	0.2%	83.86 min	7	100%	0%
App CI - E4S - HPCToolkit	2.45 hr	0.0%	1.75 min	84	100%	0%
App CI - E4S - Kokkos	312.79 hr	5.2%	34.82 min	539	100%	0%
App CI - E4S - TAU	150.35 hr	2.5%	64.44 min	140	100%	0%
App CI - E4S - Trilinos	28.68 hr	0.5%	18.70 min	92	100%	0%
Build Systems	2.65 hr	0.0%	3.01 min	53	70%	30%
Copy	.02 hr	0.0%	1.16 min	1	0%	100%
Data and Vis SDK	376.04 hr	6.3%	16.33 min	1,382	46%	53%
Developer Tools aarch64-linux-gnu	8.4 hr	0.1%	11.72 min	43	37%	56%
Developer Tools x86_64_v3-linux-gnu	5.11 hr	0.1%	8.29 min	37	54%	46%
Developer Tools, Darwin	7.87 hr	0.1%	7.05 min	67	100%	0%
E4S	2,148.89 hr	35.8%	10.07 min	12,798	32%	68%
E4S ARM Neoverse V2	163.13 hr	2.7%	3.74 min	2,619	22%	78%
E4S OneAPI	115.2 hr	1.9%	5.01 min	1,380	21%	78%
E4S ROCm External	43.99 hr	0.7%	5.39 min	490	25%	75%
High Energy Physics	768.08 hr	12.8%	14.76 min	3,123	43%	57%
Machine Learning	1,294.21 hr	21.6%	7.93 min	9,793	59%	40%
RADIUSS	56.35 hr	0.9%	4.90 min	690	42%	58%
Spack CI	3.23 hr	0.1%	4.12 min	47	70%	30%
Spack Tutorial	17.49 hr	0.3%	3.26 min	322	41%	59%
Windows Visualization (Kitware)	96.77 hr	1.6%	20.23 min	287	0%	100%
generate	215. hr	3.6%	3.64 min	3,540	61%	39%
no-specs-to-rebuild	19.13 hr	0.3%	0.33 min	3,455	0%	100%
other	4.84 hr	0.1%	1.27 min	228	8%	92%
rebuild-index	57.64 hr	1.0%	5.53 min	625	0%	100%
sign-pkgs	19.28 hr	0.3%	10.15 min	114	0%	100%
TOTAL	5,999.15 hr	--	--	44,752	42%	57%

Job Times, Overview

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AHUG ARM HPC User Group	27.39 days	0.1%	5.75 min	6,858	14%	85%
AWS Packages	1,689.02 days	4.1%	6.28 min	387,214	40%	57%
Build Systems	40.7 days	0.1%	4.37 min	13,416	73%	25%
Copy	16.4 days	0.0%	7.04 min	3,356	17%	82%
Data and Vis SDK	1,293.66 days	3.2%	7.95 min	234,468	54%	44%
Developer Tools	35.13 days	0.1%	5.51 min	9,179	67%	32%
Developer Tools Manylinux2014	52.05 days	0.1%	6.02 min	12,454	63%	33%
Developer Tools aarch64-linux-gnu	8.62 days	0.0%	3.57 min	3,480	61%	38%
Developer Tools x86_64_v3-linux-gnu	12.81 days	0.0%	4.41 min	4,180	61%	39%
Developer Tools, Darwin	10.72 days	0.0%	3.64 min	4,237	100%	0%
E4S	8,997.15 days	21.9%	8.95 min	1,447,867	51%	47%
E4S ARM Neoverse V2	914.53 days	2.2%	6.27 min	210,072	100%	0%
E4S Cray	267.46 days	0.7%	3.75 min	103,218	100%	0%
E4S OneAPI	1,405.69 days	3.4%	5.74 min	352,759	48%	50%
E4S ROCm External	712.73 days	1.7%	10.41 min	98,626	55%	42%
GCP Packages	.21 days	0.0%	3.78 min	81	0%	99%
GPU Testing	35.1 days	0.1%	6.09 min	8,376	70%	28%
High Energy Physics	66.31 days	0.2%	5.41 min	17,636	61%	39%
Machine Learning	4,836.83 days	11.8%	7.45 min	935,580	57%	41%
RADIUSS	297.62 days	0.7%	4.74 min	90,407	56%	41%
Spack CI	9.82 days	0.0%	2.87 min	4,941	67%	33%
Spack Deprecated CI	1.15 days	0.0%	3.96 min	417	76%	21%
Spack Tutorial	205.13 days	0.5%	5.37 min	55,017	67%	31%
Windows Visualization (Kitware)	29.3 days	0.1%	14.02 min	3,009	0%	87%
cleanup	8.56 days	0.0%	1.03 min	12,022	88%	12%
generate	2,782.68 days	6.8%	7.63 min	525,572	56%	42%
no-specs-to-rebuild	289.97 days	0.7%	0.66 min	635,504	27%	70%
pr-generate	1,186.42 days	2.9%	6.53 min	261,741	35%	63%
protected-generate	84.93 days	0.2%	6.61 min	18,725	25%	70%
protected-publish	41.74 days	0.1%	38.04 min	1,580	0%	98%
rebuild, aarch64	1,029.06 days	2.5%	5.51 min	269,139	0%	97%
rebuild, cray, zen4	.58 days	0.0%	1.16 min	851	100%	0%
rebuild, macOS	23.29 days	0.1%	1.45 min	23,191	100%	0%
rebuild, other	1,610.36 days	3.9%	6.41 min	363,018	2%	77%
rebuild, ppc64le	953.52 days	2.3%	9.81 min	140,428	97%	0%
rebuild, x86_64	9,865.17 days	24.0%	5.86 min	2,423,539	47%	53%
rebuild, x86_64 (p)	1,912.96 days	4.7%	7.46 min	369,092	3%	97%
rebuild-index	204.81 days	0.5%	3.01 min	97,870	32%	65%
sign-pkgs	107.58 days	0.3%	7.75 min	20,075	0%	96%
TOTAL	41,067.14 days	--	--	9,169,195	45%	52%

Runner System Failures, by Type, Last 24 Hours

Failure Type	N	# AWS	# UO	% of All System Failures
pod-unready	120	120	0	98.36%
pod-not-found	2	2	0	1.64%

Runner System Failures, by Runner, Last 24 Hours

Runner	Total Jobs	System Failures #	System Failures %	Facility
runner-graviton3-pub-gitlab-runner-55bc68f847-m9sxp	2	1	50.00%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-q7mw6	135	26	19.26%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-88vbw	128	18	14.06%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-ksn79	114	10	8.77%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-48gc4	116	10	8.62%	AWS
runner-x86-v4-prot-gitlab-runner-6f459b48c8-n74h9	77	4	5.19%	other
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-fgr8r	60	3	5.00%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-gwwtf	86	4	4.65%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-rvj5h	136	6	4.41%	AWS
runner-x86-v3-prot-gitlab-runner-74687b8fd9-9xsc6	50	2	4.00%	other
runner-x86-v3-prot-gitlab-runner-74687b8fd9-9qt6q	202	8	3.96%	other
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-z5h8z	131	5	3.82%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-gzpvz	79	3	3.80%	AWS
runner-x86-v3-prot-gitlab-runner-74687b8fd9-wcx5j	80	3	3.75%	other
runner-x86-v4-prot-gitlab-runner-6f459b48c8-fwx7d	55	2	3.64%	other
runner-x86-v4-pub-gitlab-runner-54bb8f997-44stb	180	6	3.33%	AWS
runner-x86-v2-pub-gitlab-runner-858b8c9d77-cxjjx	60	2	3.33%	AWS
runner-x86-v3-prot-gitlab-runner-74687b8fd9-nl88h	179	4	2.23%	other
runner-x86-v4-prot-gitlab-runner-6f459b48c8-dlq49	59	1	1.69%	other
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-vvnrv	61	1	1.64%	AWS
runner-x86-v4-prot-gitlab-runner-6f459b48c8-57phq	62	1	1.61%	other
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-bgpjc	96	1	1.04%	AWS
runner-x86-v2-pub-gitlab-runner-858b8c9d77-gkhkt	177	1	0.56%	AWS