Summary

Beginning: 2021-09-22 12:48 AM PDT
Ending: 2024-03-28 07:40 AM PDT
Total Jobs: 6,340,877

Navigation

Pipeline Summary
Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours

Job Times, Last 24 Hours

Job Type Runtime % Total Runtime Average Runtime N % UO % AWS
sign-pkgs6.64 hr0.5%5.45 min730%100%
rebuild-index3.77 hr0.3%1.24 min18296%4%
other216.39 hr17.8%15.11 min85950%42%
no-specs-to-rebuild7.33 hr0.6%0.44 min1,00789%11%
generate167.17 hr13.8%8.32 min1,20564%36%
Spack Tutorial, x86_647.55 hr0.6%6.66 min6878%21%
RADIUSS, x86_6418. hr1.5%6.07 min17854%46%
Machine Learning, x86_64190.97 hr15.7%9.12 min1,25751%49%
Machine Learning MPS, macOS aarch6425.87 hr2.1%5.39 min288100%0%
E4S, x86_6463.15 hr5.2%13.63 min27864%33%
E4S ROCm External130.63 hr10.8%15.37 min51060%38%
E4S Power, ppc64le90.86 hr7.5%23.50 min232100%0%
E4S OneAPI, x86_64119.36 hr9.8%8.68 min82560%39%
E4S Cray, SLES, x86_646.54 hr0.5%8.01 min49100%0%
E4S Cray, RHEL8, zen410.8 hr0.9%5.58 min116100%0%
Data and Vis SDK, x86_6485.02 hr7.0%9.22 min55353%47%
AWS Packages, x86_6431.3 hr2.6%9.99 min18858%42%
AWS Packages, neoverse_n115.54 hr1.3%5.86 min15979%21%
AWS Packages, aarch6415.65 hr1.3%6.02 min15681%19%
TOTAL1,212.53 hr----8,18366%33%

Job Times, Last Week

Job Type Runtime % Total Runtime Average Runtime N % UO % AWS
sign-pkgs26.95 hr0.3%6.17 min2620%100%
rebuild-index17.19 hr0.2%1.29 min79742%57%
protected-publish7.33 hr0.1%73.30 min60%100%
other2,774.9 hr26.1%10.53 min15,81650%49%
no-specs-to-rebuild47.18 hr0.4%0.51 min5,58441%58%
generate932.12 hr8.8%8.35 min6,69861%39%
Spack Tutorial, x86_6483.28 hr0.8%6.39 min78271%29%
RADIUSS, x86_64129.81 hr1.2%6.32 min1,23352%47%
Machine Learning, x86_641,280.92 hr12.1%9.80 min7,84044%55%
Machine Learning MPS, macOS aarch64161.66 hr1.5%5.68 min1,707100%0%
E4S, x86_641,050.53 hr9.9%10.08 min6,25244%55%
E4S ROCm External552.88 hr5.2%13.21 min2,51152%47%
E4S Power, ppc64le1,576.64 hr14.8%16.45 min5,751100%0%
E4S OneAPI, x86_64686.26 hr6.5%8.19 min5,02745%54%
E4S Cray, SLES, x86_6474.74 hr0.7%4.62 min971100%0%
E4S Cray, RHEL8, zen485.21 hr0.8%4.72 min1,083100%0%
Data and Vis SDK, x86_64412.58 hr3.9%9.56 min2,59046%53%
AWS Packages, x86_64362.48 hr3.4%8.88 min2,44954%45%
AWS Packages, neoverse_n1185.12 hr1.7%5.82 min1,90947%53%
AWS Packages, aarch64174.25 hr1.6%5.85 min1,78853%47%
TOTAL10,622.03 hr----71,05656%44%

Job Times, Overview

Job Type Runtime % Total Runtime Average Runtime N % UO % AWS
sign-pkgs76.88 days0.3%8.89 min12,5050%96%
rebuild-index166.39 days0.6%3.86 min62,10144%55%
rebuild, x86_64 (p)1,912.23 days7.4%7.47 min368,6033%97%
rebuild, x86_649,865.17 days38.3%5.86 min2,423,53947%53%
rebuild, ppc64le953.52 days3.7%9.81 min140,42897%0%
rebuild, other1,610.1 days6.2%6.41 min362,8602%77%
rebuild, macOS23.29 days0.1%1.45 min23,191100%0%
rebuild, cray, zen4.58 days0.0%1.16 min851100%0%
rebuild, aarch641,028.77 days4.0%5.51 min268,9770%97%
protected-publish33.94 days0.1%39.41 min1,2400%98%
protected-generate84.3 days0.3%6.57 min18,69025%70%
pr-generate1,186.42 days4.6%6.53 min261,74135%63%
other918.29 days3.6%7.49 min176,68128%70%
no-specs-to-rebuild157.78 days0.6%0.58 min394,18936%62%
generate1,043.29 days4.0%6.73 min223,35155%44%
copy-mirror.94 days0.0%1.87 min7230%100%
cleanup8.56 days0.0%1.03 min12,02288%12%
Spack Tutorial, x86_6481.19 days0.3%5.73 min20,41171%28%
RADIUSS, x86_64129.72 days0.5%5.19 min35,99158%41%
Machine Learning, x86_641,686.4 days6.5%7.67 min316,51152%47%
Machine Learning MPS, macOS aarch64222.68 days0.9%3.99 min80,299100%0%
E4S, x86_641,705.24 days6.6%10.18 min241,29856%43%
E4S ROCm External205.43 days0.8%10.72 min27,59055%43%
E4S Power, ppc64le983.49 days3.8%8.93 min158,590100%0%
E4S OneAPI, x86_64448.31 days1.7%6.39 min101,00154%45%
E4S Cray, SLES, x86_6434.73 days0.1%3.51 min14,37499%0%
E4S Cray, RHEL8, zen464.43 days0.2%3.64 min25,837100%0%
E4S ARM75.52 days0.3%7.33 min14,8460%100%
Data and Vis SDK, x86_64503.75 days2.0%8.39 min86,46255%44%
AWS Packages, x86_64211.17 days0.8%6.99 min43,52158%41%
AWS Packages, neoverse_n1183.88 days0.7%6.32 min41,8704%94%
AWS Packages, aarch64180.21 days0.7%6.27 min41,4144%94%
TOTAL25,786.61 days----6,001,70742%56%

Runner System Failures, by Type, Last 24 Hours

Failure Type N # AWS # UO % of All System Failures
pod-unready4545078.95%
connection-upgrade88014.04%
dial-backend3305.26%
pod-not-found1101.75%

Runner System Failures, by Runner, Last 24 Hours

Runner Total Jobs System Failures # System Failures % Facility
runner-graviton2-pub-gitlab-runner-8658f48c96-prgjm9111.11%AWS
runner-graviton3-pub-gitlab-runner-7cbff6f797-ntg8k1119.09%AWS
runner-graviton3-pub-gitlab-runner-7cbff6f797-chjqr1417.14%AWS
runner-x86-v4-prot-gitlab-runner-857d9b5648-p8vc79666.25%other
runner-x86-v4-prot-gitlab-runner-857d9b5648-b6fbf14596.21%other
runner-x86-v3-prot-gitlab-runner-549c98c65c-ppfjr15663.85%other
runner-x86-v3-prot-gitlab-runner-549c98c65c-tj48k13053.85%other
runner-x86-v3-prot-gitlab-runner-549c98c65c-mk5nb15053.33%other
runner-x86-v3-prot-gitlab-runner-549c98c65c-d4jz915253.29%other
runner-x86-v3-prot-gitlab-runner-549c98c65c-6qqbr15553.23%other
runner-x86-v4-prot-gitlab-runner-857d9b5648-5shcl12843.13%other
runner-graviton3-prot-gitlab-runner-6fff4694bd-cwdz910921.83%other
runner-x86-v3-prot-gitlab-runner-549c98c65c-6whxd16731.80%other
runner-spack-package-signing-gitlab-runner-5bd4885799-7f5b47311.37%other
runner-x86-v4-prot-gitlab-runner-857d9b5648-65q5z9311.08%other
runner-x86-v4-prot-gitlab-runner-857d9b5648-4v5nf10410.96%other
runner-graviton3-prot-gitlab-runner-6fff4694bd-g2xgm11610.86%other