Summary

Beginning: 2021-09-22 12:48 AM PDT
Ending: 2024-12-03 03:29 PM PDT
Total Jobs: 8,489,009

Navigation

Pipeline Summary
Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours

Job Times, Last 24 Hours

Job Type Runtime % Total Runtime Average Runtime N % UO % AWS
sign-pkgs2.13 hr0.2%5.55 min250%88%
rebuild-index2.26 hr0.2%0.95 min1420%99%
other324.76 hr24.4%7.99 min2,44035%63%
no-specs-to-rebuild9.31 hr0.7%0.90 min6220%100%
generate109.78 hr8.3%8.33 min79753%44%
Spack Tutorial, x86_6411.97 hr0.9%5.13 min14074%24%
RADIUSS, x86_6417.87 hr1.3%4.83 min22265%33%
Machine Learning, x86_64254.11 hr19.1%10.27 min1,48447%52%
E4S, x86_64254.98 hr19.2%7.68 min1,99344%56%
E4S ROCm External22.38 hr1.7%11.68 min11561%37%
E4S Power, ppc64le110.98 hr8.3%9.86 min675100%0%
E4S OneAPI, x86_6470.31 hr5.3%4.45 min94935%63%
Data and Vis SDK, x86_6469.18 hr5.2%9.65 min43052%48%
AWS Packages, x86_6430.85 hr2.3%6.81 min27264%35%
AWS Packages, neoverse_n121.42 hr1.6%5.08 min25345%54%
AWS Packages, aarch6417.54 hr1.3%4.26 min24759%37%
TOTAL1,329.81 hr----10,80645%54%

Job Times, Last Week

Job Type Runtime % Total Runtime Average Runtime N % UO % AWS
sign-pkgs6.59 hr0.1%6.08 min670%94%
rebuild-index12.94 hr0.1%1.27 min6130%97%
other2,716.54 hr25.5%6.52 min25,01334%61%
no-specs-to-rebuild27.89 hr0.3%0.46 min3,6400%96%
generate618.23 hr5.8%8.01 min4,63850%47%
Spack Tutorial, x86_6482.09 hr0.8%4.14 min1,19167%29%
RADIUSS, x86_64120.45 hr1.1%3.58 min2,01759%37%
Machine Learning, x86_641,497.8 hr14.1%7.47 min12,03346%50%
E4S, x86_642,268.21 hr21.3%6.23 min21,85449%49%
E4S ROCm External288.43 hr2.7%8.25 min2,09862%34%
E4S Power, ppc64le1,082.18 hr10.2%7.20 min9,012100%0%
E4S OneAPI, x86_64742.3 hr7.0%4.14 min10,74748%50%
Data and Vis SDK, x86_64590.51 hr5.6%8.70 min4,07357%39%
AWS Packages, x86_64274.58 hr2.6%5.67 min2,90854%42%
AWS Packages, neoverse_n1177.06 hr1.7%5.56 min1,91036%58%
AWS Packages, aarch64128.09 hr1.2%4.33 min1,77646%48%
TOTAL10,633.89 hr----103,59048%48%

Job Times, Overview

Job Type Runtime % Total Runtime Average Runtime N % UO % AWS
sign-pkgs93.3 days0.3%8.31 min16,2360%97%
rebuild-index183.15 days0.6%3.33 min79,19340%58%
rebuild, x86_64 (p)1,912.96 days5.8%7.46 min369,0923%97%
rebuild, x86_649,865.17 days29.9%5.86 min2,423,53947%53%
rebuild, ppc64le953.52 days2.9%9.81 min140,42897%0%
rebuild, other1,610.36 days4.9%6.41 min363,0182%77%
rebuild, macOS23.29 days0.1%1.45 min23,191100%0%
rebuild, cray, zen4.58 days0.0%1.16 min851100%0%
rebuild, aarch641,029.06 days3.1%5.51 min269,1390%97%
protected-publish41.56 days0.1%37.93 min1,5780%98%
protected-generate84.93 days0.3%6.61 min18,72525%70%
pr-generate1,186.42 days3.6%6.53 min261,74135%63%
other2,470.18 days7.5%7.76 min458,66838%59%
no-specs-to-rebuild222.35 days0.7%0.63 min508,86834%64%
generate1,906.84 days5.8%7.52 min365,50658%41%
copy-mirror.94 days0.0%1.87 min7230%100%
cleanup8.56 days0.0%1.03 min12,02288%12%
Spack Tutorial, x86_64145.23 days0.4%5.69 min36,78570%29%
RADIUSS, x86_64218.12 days0.7%5.13 min61,24758%40%
Machine Learning, x86_642,666.46 days8.1%8.27 min464,12552%47%
Machine Learning MPS, macOS aarch64370.33 days1.1%4.56 min116,905100%0%
E4S, x86_642,627.47 days8.0%9.48 min399,19054%45%
E4S ROCm External485.02 days1.5%10.52 min66,41057%41%
E4S Power, ppc64le1,798.94 days5.4%9.67 min267,952100%0%
E4S OneAPI, x86_64834.52 days2.5%6.46 min185,94552%46%
E4S Cray, SLES, x86_6476. days0.2%3.84 min28,618100%0%
E4S Cray, RHEL8, zen4124.81 days0.4%3.75 min48,262100%0%
E4S ARM75.52 days0.2%7.33 min14,8460%100%
Data and Vis SDK, x86_64862.57 days2.6%8.16 min152,26855%43%
AWS Packages, x86_64493.18 days1.5%7.96 min89,16459%40%
AWS Packages, neoverse_n1335.69 days1.0%6.03 min80,18626%72%
AWS Packages, aarch64302.77 days0.9%5.80 min75,17727%71%
TOTAL33,009.81 days----7,399,59845%53%

Runner System Failures, by Type, Last 24 Hours

Failure Type N # AWS # UO % of All System Failures
pod-unready244210081.61%
dial-backend3931013.04%
pod-not-found6402.01%
connection-upgrade5401.67%
other5501.67%

Runner System Failures, by Runner, Last 24 Hours

Runner Total Jobs System Failures # System Failures % Facility
runner-spack-package-signing-gitlab-runner-65dcdf4599-bjwsw2150.00%other
runner-graviton2-prot-gitlab-runner-78f5fdb4f4-gnvrc1345238.81%other
runner-x86-v4-prot-gitlab-runner-95f7bc676-lgcnm331236.36%other
runner-graviton2-prot-gitlab-runner-78f5fdb4f4-d25mz1334634.59%other
runner-x86-v3-pub-gitlab-runner-6564cf46d-plqz48225.00%AWS
runner-x86-v2-pub-gitlab-runner-78f8c998ff-pl8v98225.00%AWS
runner-graviton2-pub-gitlab-runner-5bd7f7f788-zwfln481225.00%AWS
runner-x86-v4-prot-gitlab-runner-659c7dd994-l26nw4125.00%other
runner-x86-v3-prot-gitlab-runner-775b867b79-2rvz820525.00%other
runner-x86-v3-pub-gitlab-runner-6564cf46d-sd9rs4125.00%AWS
runner-graviton2-pub-gitlab-runner-5bd7f7f788-4jnr738923.68%AWS
runner-graviton2-pub-gitlab-runner-5bd7f7f788-h5qv518422.22%AWS
runner-graviton3-prot-gitlab-runner-6599b79548-gl7z637821.62%other
runner-graviton2-pub-gitlab-runner-5bd7f7f788-g9n4r14321.43%AWS
runner-graviton2-pub-gitlab-runner-5bd7f7f788-hzsq920420.00%AWS
runner-graviton2-pub-gitlab-runner-5bd7f7f788-zw4hk18316.67%AWS
runner-x86-v2-pub-gitlab-runner-78f8c998ff-pfgxq32515.63%AWS
runner-graviton3-pub-gitlab-runner-7494b74c85-2264k14214.29%AWS
runner-x86-v2-pub-gitlab-runner-78f8c998ff-js9ll36513.89%AWS
runner-x86-v2-pub-gitlab-runner-78f8c998ff-7pmrx36513.89%AWS
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-c2qhn25312.00%AWS
runner-graviton2-pub-gitlab-runner-7b9c95694-55rsw10110.00%AWS
runner-x86-v3-pub-gitlab-runner-67b8df5756-qlqmh3139.68%AWS
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-wbzxs4449.09%AWS
runner-x86-v3-pub-gitlab-runner-67b8df5756-g7sqj6158.20%AWS
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-hbbxs6035.00%AWS
runner-x86-v3-pub-gitlab-runner-67b8df5756-qhzhx8544.71%AWS
runner-x86-v4-pub-gitlab-runner-5c9f88985-rh5rk6123.28%AWS
runner-graviton2-pub-gitlab-runner-7b9c95694-dm6bf6323.17%AWS
runner-graviton3-prot-gitlab-runner-6599b79548-44p5m28493.17%other
runner-x86-v3-prot-gitlab-runner-775b867b79-chss225172.79%other
runner-x86-v4-prot-gitlab-runner-95f7bc676-k7p5p25762.33%other
runner-x86-v4-pub-gitlab-runner-5c9f88985-skkf64912.04%AWS
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-dqqwx6011.67%AWS
runner-graviton3-pub-gitlab-runner-7494b74c85-hj4pp14021.43%AWS
runner-graviton3-prot-gitlab-runner-6599b79548-2sdgb30341.32%other
runner-x86-v3-prot-gitlab-runner-775b867b79-2d9p523331.29%other
runner-x86-v3-pub-gitlab-runner-67b8df5756-n7p6t7911.27%AWS
runner-x86-v3-prot-gitlab-runner-775b867b79-gc4ws24131.24%other
runner-x86-v3-pub-gitlab-runner-67b8df5756-9rzlv8911.12%AWS
runner-graviton3-pub-gitlab-runner-7494b74c85-wcrhp9211.09%AWS
runner-x86-v4-prot-gitlab-runner-95f7bc676-pgwp628331.06%other
runner-x86-v3-prot-gitlab-runner-775b867b79-kfz8c22310.45%other
runner-x86-v4-prot-gitlab-runner-95f7bc676-m778h24810.40%other
runner-x86-v2-pub-gitlab-runner-6d88c8459c-9rpmx27610.36%AWS