Summary

Beginning: 2021-09-22 12:48 AM PDT
Ending: 2023-10-03 03:46 PM PDT
Total Jobs: 4,837,545

Navigation

Pipeline Summary
Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours

Job Times, Last 24 Hours

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
sign-pkgs	2.26 hr	0.2%	4.67 min	29	0%	100%
rebuild-index	1.65 hr	0.1%	1.04 min	95	1%	99%
rebuild, x86_64 (p)	206.49 hr	17.5%	6.65 min	1,862	35%	65%
rebuild, ppc64le	49.06 hr	4.2%	7.71 min	382	100%	0%
rebuild, other	11.83 hr	1.0%	3.78 min	188	0%	100%
rebuild, aarch64	12.49 hr	1.1%	3.88 min	193	0%	100%
other	113.94 hr	9.7%	5.98 min	1,144	68%	32%
no-specs-to-rebuild	8.36 hr	0.7%	0.55 min	918	6%	94%
generate	74.07 hr	6.3%	3.43 min	1,295	62%	38%
Spack Tutorial, x86_64	5.93 hr	0.5%	2.97 min	120	78%	22%
RADIUSS, x86_64	6.44 hr	0.5%	3.31 min	117	53%	47%
Machine Learning, x86_64	179.49 hr	15.2%	5.49 min	1,962	71%	29%
Machine Learning MPS, macOS aarch64	24.86 hr	2.1%	4.16 min	359	100%	0%
E4S, x86_64	204.42 hr	17.4%	7.59 min	1,615	51%	49%
E4S Power, ppc64le	167.44 hr	14.2%	7.13 min	1,410	100%	0%
E4S OneAPI, x86_64	37.87 hr	3.2%	7.33 min	310	40%	60%
E4S Cray, RHEL8, zen4	5.04 hr	0.4%	5.49 min	55	100%	0%
Data and Vis SDK, x86_64	41.85 hr	3.6%	6.57 min	382	73%	27%
AWS Packages, x86_64	10.39 hr	0.9%	2.98 min	209	75%	25%
AWS Packages, neoverse_n1	6.84 hr	0.6%	2.95 min	139	0%	100%
AWS Packages, aarch64	6.33 hr	0.5%	2.69 min	141	0%	100%
TOTAL	1,177.05 hr	--	--	12,925	58%	42%

Job Times, Last Week

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
sign-pkgs	10.01 hr	0.1%	4.69 min	128	0%	98%
rebuild-index	12.09 hr	0.2%	1.09 min	665	0%	100%
rebuild, x86_64 (p)	209.22 hr	3.0%	6.43 min	1,951	37%	63%
rebuild, ppc64le	49.06 hr	0.7%	7.71 min	382	100%	0%
rebuild, other	11.83 hr	0.2%	3.78 min	188	0%	100%
rebuild, aarch64	12.49 hr	0.2%	3.88 min	193	0%	100%
protected-publish	6.51 hr	0.1%	15.61 min	25	0%	100%
protected-generate	.84 hr	0.0%	3.36 min	15	20%	80%
other	388.78 hr	5.5%	5.43 min	4,304	33%	67%
no-specs-to-rebuild	35.32 hr	0.5%	0.48 min	4,404	6%	94%
generate	430.5 hr	6.1%	4.28 min	6,122	57%	42%
Spack Tutorial, x86_64	68.43 hr	1.0%	5.50 min	746	78%	22%
RADIUSS, x86_64	128.3 hr	1.8%	4.73 min	1,627	66%	34%
Machine Learning, x86_64	1,906.78 hr	27.0%	6.34 min	18,045	60%	40%
Machine Learning MPS, macOS aarch64	244.43 hr	3.5%	3.75 min	3,915	100%	0%
E4S, x86_64	1,216.41 hr	17.2%	7.64 min	9,547	66%	34%
E4S Power, ppc64le	922.56 hr	13.0%	6.54 min	8,465	100%	0%
E4S OneAPI, x86_64	353.63 hr	5.0%	4.67 min	4,540	72%	28%
E4S Cray, SLES, x86_64	56.87 hr	0.8%	2.84 min	1,289	93%	0%
E4S Cray, RHEL8, zen4	67.12 hr	0.9%	3.54 min	1,138	100%	0%
Data and Vis SDK, x86_64	525.36 hr	7.4%	7.00 min	4,502	65%	35%
AWS Packages, x86_64	153.87 hr	2.2%	5.60 min	1,648	69%	31%
AWS Packages, neoverse_n1	132.78 hr	1.9%	5.34 min	1,493	0%	100%
AWS Packages, aarch64	130.29 hr	1.8%	5.15 min	1,518	0%	100%
TOTAL	7,073.49 hr	--	--	76,850	61%	38%

Job Times, Overview

Job Type	Runtime	% Total Runtime	Average Runtime	N	% UO	% AWS
sign-pkgs	44.79 days	0.2%	8.62 min	7,539	0%	95%
rebuild-index	143.9 days	0.8%	4.79 min	43,275	61%	38%
rebuild, x86_64 (p)	1,876.65 days	10.4%	7.52 min	359,516	3%	97%
rebuild, x86_64	9,865.17 days	54.6%	5.86 min	2,423,539	47%	53%
rebuild, ppc64le	943.81 days	5.2%	9.85 min	138,467	97%	0%
rebuild, other	1,600.89 days	8.9%	6.41 min	360,972	2%	77%
rebuild, macOS	23.29 days	0.1%	1.45 min	23,191	100%	0%
rebuild, cray, zen4	.58 days	0.0%	1.16 min	851	100%	0%
rebuild, aarch64	1,024.28 days	5.7%	5.51 min	267,717	0%	97%
protected-publish	17.73 days	0.1%	46.67 min	547	0%	97%
protected-generate	84.26 days	0.5%	6.57 min	18,675	25%	70%
pr-generate	1,185.76 days	6.6%	6.53 min	261,588	35%	63%
other	44.67 days	0.2%	6.26 min	10,275	32%	68%
no-specs-to-rebuild	113.11 days	0.6%	0.63 min	258,280	52%	47%
generate	332.46 days	1.8%	8.31 min	57,638	48%	52%
copy-mirror	.94 days	0.0%	1.87 min	723	0%	100%
cleanup	8.56 days	0.0%	1.03 min	12,022	88%	12%
Spack Tutorial, x86_64	10.25 days	0.1%	5.85 min	2,524	71%	29%
RADIUSS, x86_64	18.41 days	0.1%	5.21 min	5,086	57%	43%
Machine Learning, x86_64	180.4 days	1.0%	8.38 min	30,988	54%	46%
Machine Learning MPS, macOS aarch64	29.67 days	0.2%	1.87 min	22,796	100%	0%
E4S, x86_64	203.65 days	1.1%	10.70 min	27,409	53%	47%
E4S Power, ppc64le	157.69 days	0.9%	10.10 min	22,472	100%	0%
E4S OneAPI, x86_64	38.04 days	0.2%	6.83 min	8,022	60%	40%
E4S Cray, SLES, x86_64	1.2 days	0.0%	3.61 min	480	100%	0%
E4S Cray, RHEL8, zen4	3.27 days	0.0%	2.56 min	1,834	100%	0%
Data and Vis SDK, x86_64	51.71 days	0.3%	9.91 min	7,516	54%	46%
AWS Packages, x86_64	30.01 days	0.2%	7.48 min	5,778	55%	44%
AWS Packages, neoverse_n1	25.36 days	0.1%	6.26 min	5,836	0%	99%
AWS Packages, aarch64	24.01 days	0.1%	6.29 min	5,500	0%	99%
TOTAL	18,084.52 days	--	--	4,391,056	39%	59%

Runner System Failures, by Type, Last 24 Hours

Failure Type	N	# AWS	# UO	% of All System Failures
pod-unready	10	10	0	100.00%

Runner System Failures, by Runner, Last 24 Hours

Runner	Total Jobs	System Failures #	System Failures %	Facility
runner-x86-v3-pub-gitlab-runner-599577d4db-75bn2	70	1	1.43%	AWS
runner-x86-v3-prot-gitlab-runner-65f46dcb4b-tm6fn	142	2	1.41%	other
runner-x86-v4-pub-gitlab-runner-6c4774d87d-t8dlp	121	1	0.83%	AWS
runner-x86-v4-pub-gitlab-runner-6c4774d87d-2t6np	131	1	0.76%	AWS
runner-x86-v3-prot-gitlab-runner-65f46dcb4b-97c94	174	1	0.57%	other
runner-x86-v4-prot-gitlab-runner-6d8b5748d-6whx2	182	1	0.55%	other
runner-x86-v3-prot-gitlab-runner-65f46dcb4b-vfwjt	181	1	0.55%	other
runner-x86-v4-prot-gitlab-runner-6d8b5748d-444lc	200	1	0.50%	other
runner-x86-v4-prot-gitlab-runner-6d8b5748d-k452l	205	1	0.49%	other