Summary

Beginning: 2021-09-22 12:48 AM PDT
Ending: 2026-02-11 01:46 PM PDT
Total Jobs: 10,268,053

Navigation

Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
GPU Usage, UO, Last Week
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours

GPU Usage, UO, Last Week

GPU Type	# Jobs
AMD MI100	159
AMD MI210	24
AMD MI300A	81
AMD MI50	16
INTEL A770	131
INTEL DATA CENTER MAX 1100	225
INTEL DG1	28
NVIDIA A100	69
NVIDIA A2000	14
NVIDIA GH200	72
NVIDIA H100	21
NVIDIA RTX5080	72

Job Times, Last 24 Hours

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AWS Packages	17.56 hr	1.6%	5.16 min	204	25%	75%
App CI - E4S - Adios2	6.96 hr	0.6%	1.25 min	333	90%	0%
App CI - E4S - ExaGO	.16 hr	0.0%	3.27 min	3	100%	0%
App CI - E4S - ExaWind	1.39 hr	0.1%	83.12 min	1	100%	0%
App CI - E4S - HPCToolkit	1.01 hr	0.1%	2.54 min	25	96%	0%
App CI - E4S - Kokkos	26.85 hr	2.4%	38.35 min	44	100%	0%
App CI - E4S - TAU	18.44 hr	1.7%	58.23 min	19	100%	0%
App CI - E4S - Trilinos	6.66 hr	0.6%	24.97 min	16	100%	0%
Build Systems	.77 hr	0.1%	3.29 min	14	64%	36%
Data and Vis SDK	23.32 hr	2.1%	3.20 min	437	33%	67%
Developer Tools aarch64-linux-gnu	.04 hr	0.0%	1.20 min	2	0%	100%
Developer Tools x86_64_v3-linux-gnu	2.69 hr	0.2%	2.41 min	67	18%	82%
Developer Tools, Darwin	1.24 hr	0.1%	9.32 min	8	100%	0%
E4S	485.48 hr	43.9%	3.74 min	7,780	16%	84%
E4S ARM Neoverse V2	77.85 hr	7.0%	2.60 min	1,795	23%	77%
E4S OneAPI	25.66 hr	2.3%	3.20 min	481	17%	83%
E4S ROCm External	8.21 hr	0.7%	5.13 min	96	25%	75%
High Energy Physics	56.96 hr	5.1%	3.40 min	1,005	9%	91%
Machine Learning	247.1 hr	22.3%	4.24 min	3,494	18%	82%
RADIUSS	12.47 hr	1.1%	4.18 min	179	29%	71%
Spack CI	3.36 hr	0.3%	2.40 min	84	25%	75%
Spack Tutorial	5.52 hr	0.5%	2.61 min	127	38%	62%
Windows Visualization (Kitware)	19.66 hr	1.8%	20.69 min	57	0%	100%
generate	48.49 hr	4.4%	3.89 min	748	28%	72%
no-specs-to-rebuild	5.29 hr	0.5%	0.53 min	594	0%	100%
other	1.05 hr	0.1%	0.88 min	71	10%	90%
rebuild-index	1.39 hr	0.1%	1.10 min	76	0%	100%
sign-pkgs	1.42 hr	0.1%	17.04 min	5	0%	100%
TOTAL	1,107. hr	--	--	17,765	20%	80%

Job Times, Last Week

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AWS Packages	60.35 hr	1.4%	6.61 min	548	25%	75%
App CI - E4S - Adios2	47.03 hr	1.1%	1.20 min	2,358	90%	0%
App CI - E4S - ExaGO	.55 hr	0.0%	1.84 min	19	100%	0%
App CI - E4S - ExaWind	7.92 hr	0.2%	67.92 min	7	100%	0%
App CI - E4S - HPCToolkit	4.37 hr	0.1%	2.50 min	109	96%	0%
App CI - E4S - Kokkos	204.23 hr	4.8%	31.18 min	409	100%	0%
App CI - E4S - TAU	129.22 hr	3.0%	55.38 min	146	100%	0%
App CI - E4S - Trilinos	32.83 hr	0.8%	21.65 min	103	100%	0%
Build Systems	4.13 hr	0.1%	2.88 min	86	27%	73%
Copy	.01 hr	0.0%	0.87 min	1	0%	100%
Data and Vis SDK	211.97 hr	4.9%	9.05 min	1,405	35%	65%
Developer Tools aarch64-linux-gnu	5.4 hr	0.1%	2.33 min	139	8%	92%
Developer Tools x86_64_v3-linux-gnu	8.11 hr	0.2%	3.69 min	132	35%	65%
Developer Tools, Darwin	2.71 hr	0.1%	4.40 min	37	100%	0%
E4S	1,737.26 hr	40.4%	5.80 min	17,965	24%	76%
E4S ARM Neoverse V2	209.47 hr	4.9%	2.76 min	4,551	27%	73%
E4S OneAPI	123.43 hr	2.9%	3.08 min	2,401	19%	81%
E4S ROCm External	38.13 hr	0.9%	4.90 min	467	41%	59%
High Energy Physics	208.73 hr	4.9%	5.39 min	2,325	14%	86%
Machine Learning	684.63 hr	15.9%	4.98 min	8,255	30%	70%
RADIUSS	46.98 hr	1.1%	3.50 min	806	37%	63%
Spack CI	4.83 hr	0.1%	2.68 min	108	29%	71%
Spack Tutorial	53.71 hr	1.2%	2.86 min	1,128	56%	44%
Windows Visualization (Kitware)	40.5 hr	0.9%	15.99 min	152	0%	100%
generate	347.3 hr	8.1%	4.34 min	4,796	44%	56%
no-specs-to-rebuild	28.41 hr	0.7%	0.41 min	4,132	0%	100%
other	5.64 hr	0.1%	0.82 min	414	5%	95%
rebuild-index	35.7 hr	0.8%	3.24 min	661	0%	100%
sign-pkgs	15.17 hr	0.4%	9.39 min	97	0%	100%
TOTAL	4,298.74 hr	--	--	53,757	29%	70%

Job Times, Overview

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
AHUG ARM HPC User Group	27.39 days	0.1%	5.75 min	6,858	14%	85%
AWS Packages	1,689.02 days	4.1%	6.28 min	387,214	40%	57%
Build Systems	40.7 days	0.1%	4.37 min	13,416	73%	25%
Copy	16.4 days	0.0%	7.04 min	3,356	17%	82%
Data and Vis SDK	1,293.66 days	3.2%	7.95 min	234,468	54%	44%
Developer Tools	35.13 days	0.1%	5.51 min	9,179	67%	32%
Developer Tools Manylinux2014	52.05 days	0.1%	6.02 min	12,454	63%	33%
Developer Tools aarch64-linux-gnu	8.62 days	0.0%	3.57 min	3,480	61%	38%
Developer Tools x86_64_v3-linux-gnu	12.81 days	0.0%	4.41 min	4,180	61%	39%
Developer Tools, Darwin	10.72 days	0.0%	3.64 min	4,237	100%	0%
E4S	8,997.15 days	21.9%	8.95 min	1,447,867	51%	47%
E4S ARM Neoverse V2	914.53 days	2.2%	6.27 min	210,072	100%	0%
E4S Cray	267.46 days	0.7%	3.75 min	103,218	100%	0%
E4S OneAPI	1,405.69 days	3.4%	5.74 min	352,759	48%	50%
E4S ROCm External	712.73 days	1.7%	10.41 min	98,626	55%	42%
GCP Packages	.21 days	0.0%	3.78 min	81	0%	99%
GPU Testing	35.1 days	0.1%	6.09 min	8,376	70%	28%
High Energy Physics	66.31 days	0.2%	5.41 min	17,636	61%	39%
Machine Learning	4,836.83 days	11.8%	7.45 min	935,580	57%	41%
RADIUSS	297.62 days	0.7%	4.74 min	90,407	56%	41%
Spack CI	9.82 days	0.0%	2.87 min	4,941	67%	33%
Spack Deprecated CI	1.15 days	0.0%	3.96 min	417	76%	21%
Spack Tutorial	205.13 days	0.5%	5.37 min	55,017	67%	31%
Windows Visualization (Kitware)	29.3 days	0.1%	14.02 min	3,009	0%	87%
cleanup	8.56 days	0.0%	1.03 min	12,022	88%	12%
generate	2,782.68 days	6.8%	7.63 min	525,572	56%	42%
no-specs-to-rebuild	289.97 days	0.7%	0.66 min	635,504	27%	70%
pr-generate	1,186.42 days	2.9%	6.53 min	261,741	35%	63%
protected-generate	84.93 days	0.2%	6.61 min	18,725	25%	70%
protected-publish	41.74 days	0.1%	38.04 min	1,580	0%	98%
rebuild, aarch64	1,029.06 days	2.5%	5.51 min	269,139	0%	97%
rebuild, cray, zen4	.58 days	0.0%	1.16 min	851	100%	0%
rebuild, macOS	23.29 days	0.1%	1.45 min	23,191	100%	0%
rebuild, other	1,610.36 days	3.9%	6.41 min	363,018	2%	77%
rebuild, ppc64le	953.52 days	2.3%	9.81 min	140,428	97%	0%
rebuild, x86_64	9,865.17 days	24.0%	5.86 min	2,423,539	47%	53%
rebuild, x86_64 (p)	1,912.96 days	4.7%	7.46 min	369,092	3%	97%
rebuild-index	204.81 days	0.5%	3.01 min	97,870	32%	65%
sign-pkgs	107.58 days	0.3%	7.75 min	20,075	0%	96%
TOTAL	41,067.14 days	--	--	9,169,195	45%	52%

Runner System Failures, by Type, Last 24 Hours

Failure Type	N	# AWS	# UO	% of All System Failures
pod-unready	71	71	0	85.54%
connection-upgrade	5	5	0	6.02%
pod-not-found	4	4	0	4.82%
other	3	0	3	3.61%

Runner System Failures, by Runner, Last 24 Hours

Runner	Total Jobs	System Failures #	System Failures %	Facility
gilgamesh-docker-nvidia-h100-trilinos	3	3	100.00%	other
runner-x86-v4-prot-gitlab-runner-6f459b48c8-dlq49	68	4	5.88%	other
runner-x86-v4-prot-gitlab-runner-6f459b48c8-cztq2	98	4	4.08%	other
runner-x86-v4-prot-gitlab-runner-6f459b48c8-n74h9	83	3	3.61%	other
runner-x86-v4-prot-gitlab-runner-6f459b48c8-m4f7q	86	3	3.49%	other
runner-graviton3-pub-gitlab-runner-55bc68f847-z5xgc	70	2	2.86%	AWS
runner-x86-v4-prot-gitlab-runner-6f459b48c8-57phq	108	2	1.85%	other
runner-x86-v3-prot-gitlab-runner-74687b8fd9-wcx5j	70	1	1.43%	other
runner-x86-v4-pub-gitlab-runner-54bb8f997-q7mw6	849	9	1.06%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-44stb	959	9	0.94%	AWS
runner-x86-v2-pub-gitlab-runner-858b8c9d77-hp8b2	405	3	0.74%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-rvj5h	987	7	0.71%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-gwwtf	682	4	0.59%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-fgr8r	769	4	0.52%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-88vbw	1018	5	0.49%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-ksn79	1015	5	0.49%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-z5h8z	640	3	0.47%	AWS
runner-x86-v4-pub-gitlab-runner-54bb8f997-48gc4	941	4	0.43%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-s566l	749	3	0.40%	AWS
runner-graviton4-pub-gitlab-runner-96895665c-z7lsl	259	1	0.39%	AWS
runner-graviton4-pub-gitlab-runner-96895665c-fpk4t	287	1	0.35%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-bgpjc	703	2	0.28%	AWS
runner-x86-v3-pub-gitlab-runner-bc8cf7fd7-vvnrv	737	1	0.14%	AWS