Summary

Beginning: 2021-09-22 12:48 AM PDT
Ending: 2026-03-05 03:53 AM PDT
Total Jobs: 10,268,053

Navigation

Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
GPU Usage, UO, Last Week
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours

GPU Usage, UO, Last Week

GPU Type	# Jobs
AMD MI100	244
AMD MI210	40
AMD MI300A	134
AMD MI50	17
INTEL A770	217
INTEL DATA CENTER MAX 1100	398
INTEL DG1	9
NVIDIA A100	87
NVIDIA A2000	17
NVIDIA GH200	128
NVIDIA H100	27
NVIDIA RTX5080	128

Job Times, Last 24 Hours

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AWS Packages	.42 hr	0.1%	3.13 min	8	38%	63%
App CI - E4S - Adios2	11.91 hr	3.1%	1.56 min	459	74%	0%
App CI - E4S - ExaGO	.11 hr	0.0%	2.20 min	3	100%	0%
App CI - E4S - ExaWind	0. hr	0.0%	0.04 min	1	100%	0%
App CI - E4S - HPCToolkit	.7 hr	0.2%	1.74 min	24	100%	0%
App CI - E4S - Kokkos	56.53 hr	14.5%	28.27 min	120	100%	0%
App CI - E4S - TAU	.47 hr	0.1%	2.03 min	14	100%	0%
App CI - E4S - Trilinos	23.72 hr	6.1%	88.95 min	19	100%	0%
Build Systems	.07 hr	0.0%	2.07 min	2	100%	0%
Data and Vis SDK	17.26 hr	4.4%	11.64 min	89	49%	51%
Developer Tools aarch64-linux-gnu	.37 hr	0.1%	3.13 min	7	100%	0%
Developer Tools x86_64_v3-linux-gnu	.35 hr	0.1%	2.97 min	7	86%	14%
Developer Tools, Darwin	.56 hr	0.1%	4.80 min	7	100%	0%
E4S	175.23 hr	44.9%	8.29 min	1,268	17%	83%
E4S ARM Neoverse V2	1.58 hr	0.4%	0.95 min	100	27%	73%
E4S OneAPI	.37 hr	0.1%	1.69 min	13	69%	31%
E4S ROCm External	.22 hr	0.1%	3.37 min	4	75%	25%
High Energy Physics	16.94 hr	4.3%	19.18 min	53	30%	70%
Machine Learning	29.3 hr	7.5%	5.29 min	332	23%	77%
RADIUSS	.58 hr	0.1%	2.34 min	15	67%	33%
Spack CI	.9 hr	0.2%	3.39 min	16	63%	38%
Spack Tutorial	.22 hr	0.1%	2.60 min	5	100%	0%
Windows Visualization (Kitware)	2.46 hr	0.6%	49.24 min	3	0%	100%
generate	36.88 hr	9.5%	4.03 min	549	70%	30%
no-specs-to-rebuild	1.87 hr	0.5%	0.22 min	501	0%	100%
other	.29 hr	0.1%	0.52 min	33	6%	94%
rebuild-index	6.22 hr	1.6%	11.66 min	32	13%	88%
sign-pkgs	4.46 hr	1.1%	24.34 min	11	0%	100%
TOTAL	389.99 hr	--	--	3,695	37%	60%

Job Times, Last Week

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AWS Packages	48.78 hr	1.4%	4.68 min	626	38%	62%
App CI - E4S - Adios2	48.28 hr	1.4%	1.19 min	2,433	83%	0%
App CI - E4S - ExaGO	.56 hr	0.0%	1.60 min	21	100%	0%
App CI - E4S - ExaWind	.01 hr	0.0%	0.05 min	7	100%	0%
App CI - E4S - HPCToolkit	3.7 hr	0.1%	1.68 min	132	100%	0%
App CI - E4S - Kokkos	438.14 hr	12.3%	30.86 min	852	100%	0%
App CI - E4S - TAU	33.19 hr	0.9%	12.77 min	156	100%	0%
App CI - E4S - Trilinos	56.1 hr	1.6%	36.19 min	109	100%	0%
Build Systems	1.27 hr	0.0%	2.93 min	26	54%	46%
Data and Vis SDK	178.03 hr	5.0%	9.33 min	1,145	43%	57%
Developer Tools aarch64-linux-gnu	1.76 hr	0.0%	4.39 min	24	38%	63%
Developer Tools x86_64_v3-linux-gnu	3.05 hr	0.1%	5.08 min	36	44%	56%
Developer Tools, Darwin	10.03 hr	0.3%	4.82 min	125	100%	0%
E4S	910.76 hr	25.7%	6.03 min	9,055	26%	74%
E4S ARM Neoverse V2	31.94 hr	0.9%	2.39 min	803	26%	74%
E4S OneAPI	41.14 hr	1.2%	3.21 min	770	19%	81%
E4S ROCm External	7.43 hr	0.2%	5.06 min	88	56%	44%
High Energy Physics	182.57 hr	5.1%	9.53 min	1,149	42%	58%
Machine Learning	997.09 hr	28.1%	6.17 min	9,690	50%	50%
RADIUSS	43.64 hr	1.2%	3.57 min	733	37%	63%
Spack CI	3.15 hr	0.1%	3.86 min	49	80%	20%
Spack Tutorial	31.49 hr	0.9%	2.99 min	631	66%	34%
Windows Visualization (Kitware)	5.41 hr	0.2%	27.05 min	12	0%	100%
generate	386.58 hr	10.9%	4.78 min	4,854	68%	32%
no-specs-to-rebuild	17.14 hr	0.5%	0.24 min	4,279	0%	100%
other	3.76 hr	0.1%	0.84 min	267	6%	94%
rebuild-index	42.55 hr	1.2%	7.19 min	355	1%	99%
sign-pkgs	21.06 hr	0.6%	14.20 min	89	0%	100%
TOTAL	3,548.61 hr	--	--	38,516	42%	56%

Job Times, Overview

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AHUG ARM HPC User Group	27.39 days	0.1%	5.75 min	6,858	14%	85%
AWS Packages	1,689.02 days	4.1%	6.28 min	387,214	40%	57%
Build Systems	40.7 days	0.1%	4.37 min	13,416	73%	25%
Copy	16.4 days	0.0%	7.04 min	3,356	17%	82%
Data and Vis SDK	1,293.66 days	3.2%	7.95 min	234,468	54%	44%
Developer Tools	35.13 days	0.1%	5.51 min	9,179	67%	32%
Developer Tools Manylinux2014	52.05 days	0.1%	6.02 min	12,454	63%	33%
Developer Tools aarch64-linux-gnu	8.62 days	0.0%	3.57 min	3,480	61%	38%
Developer Tools x86_64_v3-linux-gnu	12.81 days	0.0%	4.41 min	4,180	61%	39%
Developer Tools, Darwin	10.72 days	0.0%	3.64 min	4,237	100%	0%
E4S	8,997.15 days	21.9%	8.95 min	1,447,867	51%	47%
E4S ARM Neoverse V2	914.53 days	2.2%	6.27 min	210,072	100%	0%
E4S Cray	267.46 days	0.7%	3.75 min	103,218	100%	0%
E4S OneAPI	1,405.69 days	3.4%	5.74 min	352,759	48%	50%
E4S ROCm External	712.73 days	1.7%	10.41 min	98,626	55%	42%
GCP Packages	.21 days	0.0%	3.78 min	81	0%	99%
GPU Testing	35.1 days	0.1%	6.09 min	8,376	70%	28%
High Energy Physics	66.31 days	0.2%	5.41 min	17,636	61%	39%
Machine Learning	4,836.83 days	11.8%	7.45 min	935,580	57%	41%
RADIUSS	297.62 days	0.7%	4.74 min	90,407	56%	41%
Spack CI	9.82 days	0.0%	2.87 min	4,941	67%	33%
Spack Deprecated CI	1.15 days	0.0%	3.96 min	417	76%	21%
Spack Tutorial	205.13 days	0.5%	5.37 min	55,017	67%	31%
Windows Visualization (Kitware)	29.3 days	0.1%	14.02 min	3,009	0%	87%
cleanup	8.56 days	0.0%	1.03 min	12,022	88%	12%
generate	2,782.68 days	6.8%	7.63 min	525,572	56%	42%
no-specs-to-rebuild	289.97 days	0.7%	0.66 min	635,504	27%	70%
pr-generate	1,186.42 days	2.9%	6.53 min	261,741	35%	63%
protected-generate	84.93 days	0.2%	6.61 min	18,725	25%	70%
protected-publish	41.74 days	0.1%	38.04 min	1,580	0%	98%
rebuild, aarch64	1,029.06 days	2.5%	5.51 min	269,139	0%	97%
rebuild, cray, zen4	.58 days	0.0%	1.16 min	851	100%	0%
rebuild, macOS	23.29 days	0.1%	1.45 min	23,191	100%	0%
rebuild, other	1,610.36 days	3.9%	6.41 min	363,018	2%	77%
rebuild, ppc64le	953.52 days	2.3%	9.81 min	140,428	97%	0%
rebuild, x86_64	9,865.17 days	24.0%	5.86 min	2,423,539	47%	53%
rebuild, x86_64 (p)	1,912.96 days	4.7%	7.46 min	369,092	3%	97%
rebuild-index	204.81 days	0.5%	3.01 min	97,870	32%	65%
sign-pkgs	107.58 days	0.3%	7.75 min	20,075	0%	96%
TOTAL	41,067.14 days	--	--	9,169,195	45%	52%

Runner System Failures, by Type, Last 24 Hours

Failure Type	N	# AWS	# UO	% of All System Failures
pod-unready	12	12	0	92.31%
aborted-terminated	1	0	1	7.69%

Runner System Failures, by Runner, Last 24 Hours

Runner	Total Jobs	System Failures #	System Failures %	Facility
gilgamesh-docker-nvidia-h100-trilinos	5	1	20.00%	other
runner-x86-v3-pub-gitlab-runner-6bb57b456d-9tlrc	16	1	6.25%	AWS
runner-x86-v4-pub-gitlab-runner-7c87647f4b-9z2j2	18	1	5.56%	AWS
runner-x86-v4-prot-gitlab-runner-59bd4646dc-stmdz	88	3	3.41%	other
runner-x86-v3-prot-gitlab-runner-8cfbbdb86-67shm	135	2	1.48%	other
runner-x86-v3-prot-gitlab-runner-8cfbbdb86-6s49f	92	1	1.09%	other
runner-x86-v4-prot-gitlab-runner-59bd4646dc-rflj9	101	1	0.99%	other
runner-x86-v4-prot-gitlab-runner-59bd4646dc-29zq8	106	1	0.94%	other
runner-x86-v3-prot-gitlab-runner-8cfbbdb86-kckps	131	1	0.76%	other
runner-x86-v4-prot-gitlab-runner-59bd4646dc-t4zkl	144	1	0.69%	other