Beginning: 2021-09-22 12:48 AM PDT
Ending: 2024-12-03 03:29 PM PDT
Total Jobs: 8,489,009
Pipeline Summary
Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours
Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
---|---|---|---|---|---|---|
sign-pkgs | 2.13 hr | 0.2% | 5.55 min | 25 | 0% | 88% |
rebuild-index | 2.26 hr | 0.2% | 0.95 min | 142 | 0% | 99% |
other | 324.76 hr | 24.4% | 7.99 min | 2,440 | 35% | 63% |
no-specs-to-rebuild | 9.31 hr | 0.7% | 0.90 min | 622 | 0% | 100% |
generate | 109.78 hr | 8.3% | 8.33 min | 797 | 53% | 44% |
Spack Tutorial, x86_64 | 11.97 hr | 0.9% | 5.13 min | 140 | 74% | 24% |
RADIUSS, x86_64 | 17.87 hr | 1.3% | 4.83 min | 222 | 65% | 33% |
Machine Learning, x86_64 | 254.11 hr | 19.1% | 10.27 min | 1,484 | 47% | 52% |
E4S, x86_64 | 254.98 hr | 19.2% | 7.68 min | 1,993 | 44% | 56% |
E4S ROCm External | 22.38 hr | 1.7% | 11.68 min | 115 | 61% | 37% |
E4S Power, ppc64le | 110.98 hr | 8.3% | 9.86 min | 675 | 100% | 0% |
E4S OneAPI, x86_64 | 70.31 hr | 5.3% | 4.45 min | 949 | 35% | 63% |
Data and Vis SDK, x86_64 | 69.18 hr | 5.2% | 9.65 min | 430 | 52% | 48% |
AWS Packages, x86_64 | 30.85 hr | 2.3% | 6.81 min | 272 | 64% | 35% |
AWS Packages, neoverse_n1 | 21.42 hr | 1.6% | 5.08 min | 253 | 45% | 54% |
AWS Packages, aarch64 | 17.54 hr | 1.3% | 4.26 min | 247 | 59% | 37% |
TOTAL | 1,329.81 hr | -- | -- | 10,806 | 45% | 54% |
Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
---|---|---|---|---|---|---|
sign-pkgs | 6.59 hr | 0.1% | 6.08 min | 67 | 0% | 94% |
rebuild-index | 12.94 hr | 0.1% | 1.27 min | 613 | 0% | 97% |
other | 2,716.54 hr | 25.5% | 6.52 min | 25,013 | 34% | 61% |
no-specs-to-rebuild | 27.89 hr | 0.3% | 0.46 min | 3,640 | 0% | 96% |
generate | 618.23 hr | 5.8% | 8.01 min | 4,638 | 50% | 47% |
Spack Tutorial, x86_64 | 82.09 hr | 0.8% | 4.14 min | 1,191 | 67% | 29% |
RADIUSS, x86_64 | 120.45 hr | 1.1% | 3.58 min | 2,017 | 59% | 37% |
Machine Learning, x86_64 | 1,497.8 hr | 14.1% | 7.47 min | 12,033 | 46% | 50% |
E4S, x86_64 | 2,268.21 hr | 21.3% | 6.23 min | 21,854 | 49% | 49% |
E4S ROCm External | 288.43 hr | 2.7% | 8.25 min | 2,098 | 62% | 34% |
E4S Power, ppc64le | 1,082.18 hr | 10.2% | 7.20 min | 9,012 | 100% | 0% |
E4S OneAPI, x86_64 | 742.3 hr | 7.0% | 4.14 min | 10,747 | 48% | 50% |
Data and Vis SDK, x86_64 | 590.51 hr | 5.6% | 8.70 min | 4,073 | 57% | 39% |
AWS Packages, x86_64 | 274.58 hr | 2.6% | 5.67 min | 2,908 | 54% | 42% |
AWS Packages, neoverse_n1 | 177.06 hr | 1.7% | 5.56 min | 1,910 | 36% | 58% |
AWS Packages, aarch64 | 128.09 hr | 1.2% | 4.33 min | 1,776 | 46% | 48% |
TOTAL | 10,633.89 hr | -- | -- | 103,590 | 48% | 48% |
Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
---|---|---|---|---|---|---|
sign-pkgs | 93.3 days | 0.3% | 8.31 min | 16,236 | 0% | 97% |
rebuild-index | 183.15 days | 0.6% | 3.33 min | 79,193 | 40% | 58% |
rebuild, x86_64 (p) | 1,912.96 days | 5.8% | 7.46 min | 369,092 | 3% | 97% |
rebuild, x86_64 | 9,865.17 days | 29.9% | 5.86 min | 2,423,539 | 47% | 53% |
rebuild, ppc64le | 953.52 days | 2.9% | 9.81 min | 140,428 | 97% | 0% |
rebuild, other | 1,610.36 days | 4.9% | 6.41 min | 363,018 | 2% | 77% |
rebuild, macOS | 23.29 days | 0.1% | 1.45 min | 23,191 | 100% | 0% |
rebuild, cray, zen4 | .58 days | 0.0% | 1.16 min | 851 | 100% | 0% |
rebuild, aarch64 | 1,029.06 days | 3.1% | 5.51 min | 269,139 | 0% | 97% |
protected-publish | 41.56 days | 0.1% | 37.93 min | 1,578 | 0% | 98% |
protected-generate | 84.93 days | 0.3% | 6.61 min | 18,725 | 25% | 70% |
pr-generate | 1,186.42 days | 3.6% | 6.53 min | 261,741 | 35% | 63% |
other | 2,470.18 days | 7.5% | 7.76 min | 458,668 | 38% | 59% |
no-specs-to-rebuild | 222.35 days | 0.7% | 0.63 min | 508,868 | 34% | 64% |
generate | 1,906.84 days | 5.8% | 7.52 min | 365,506 | 58% | 41% |
copy-mirror | .94 days | 0.0% | 1.87 min | 723 | 0% | 100% |
cleanup | 8.56 days | 0.0% | 1.03 min | 12,022 | 88% | 12% |
Spack Tutorial, x86_64 | 145.23 days | 0.4% | 5.69 min | 36,785 | 70% | 29% |
RADIUSS, x86_64 | 218.12 days | 0.7% | 5.13 min | 61,247 | 58% | 40% |
Machine Learning, x86_64 | 2,666.46 days | 8.1% | 8.27 min | 464,125 | 52% | 47% |
Machine Learning MPS, macOS aarch64 | 370.33 days | 1.1% | 4.56 min | 116,905 | 100% | 0% |
E4S, x86_64 | 2,627.47 days | 8.0% | 9.48 min | 399,190 | 54% | 45% |
E4S ROCm External | 485.02 days | 1.5% | 10.52 min | 66,410 | 57% | 41% |
E4S Power, ppc64le | 1,798.94 days | 5.4% | 9.67 min | 267,952 | 100% | 0% |
E4S OneAPI, x86_64 | 834.52 days | 2.5% | 6.46 min | 185,945 | 52% | 46% |
E4S Cray, SLES, x86_64 | 76. days | 0.2% | 3.84 min | 28,618 | 100% | 0% |
E4S Cray, RHEL8, zen4 | 124.81 days | 0.4% | 3.75 min | 48,262 | 100% | 0% |
E4S ARM | 75.52 days | 0.2% | 7.33 min | 14,846 | 0% | 100% |
Data and Vis SDK, x86_64 | 862.57 days | 2.6% | 8.16 min | 152,268 | 55% | 43% |
AWS Packages, x86_64 | 493.18 days | 1.5% | 7.96 min | 89,164 | 59% | 40% |
AWS Packages, neoverse_n1 | 335.69 days | 1.0% | 6.03 min | 80,186 | 26% | 72% |
AWS Packages, aarch64 | 302.77 days | 0.9% | 5.80 min | 75,177 | 27% | 71% |
TOTAL | 33,009.81 days | -- | -- | 7,399,598 | 45% | 53% |
Failure Type | N | # AWS | # UO | % of All System Failures |
---|---|---|---|---|
pod-unready | 244 | 210 | 0 | 81.61% |
dial-backend | 39 | 31 | 0 | 13.04% |
pod-not-found | 6 | 4 | 0 | 2.01% |
connection-upgrade | 5 | 4 | 0 | 1.67% |
other | 5 | 5 | 0 | 1.67% |
Runner | Total Jobs | System Failures # | System Failures % | Facility |
---|---|---|---|---|
runner-spack-package-signing-gitlab-runner-65dcdf4599-bjwsw | 2 | 1 | 50.00% | other |
runner-graviton2-prot-gitlab-runner-78f5fdb4f4-gnvrc | 134 | 52 | 38.81% | other |
runner-x86-v4-prot-gitlab-runner-95f7bc676-lgcnm | 33 | 12 | 36.36% | other |
runner-graviton2-prot-gitlab-runner-78f5fdb4f4-d25mz | 133 | 46 | 34.59% | other |
runner-x86-v3-pub-gitlab-runner-6564cf46d-plqz4 | 8 | 2 | 25.00% | AWS |
runner-x86-v2-pub-gitlab-runner-78f8c998ff-pl8v9 | 8 | 2 | 25.00% | AWS |
runner-graviton2-pub-gitlab-runner-5bd7f7f788-zwfln | 48 | 12 | 25.00% | AWS |
runner-x86-v4-prot-gitlab-runner-659c7dd994-l26nw | 4 | 1 | 25.00% | other |
runner-x86-v3-prot-gitlab-runner-775b867b79-2rvz8 | 20 | 5 | 25.00% | other |
runner-x86-v3-pub-gitlab-runner-6564cf46d-sd9rs | 4 | 1 | 25.00% | AWS |
runner-graviton2-pub-gitlab-runner-5bd7f7f788-4jnr7 | 38 | 9 | 23.68% | AWS |
runner-graviton2-pub-gitlab-runner-5bd7f7f788-h5qv5 | 18 | 4 | 22.22% | AWS |
runner-graviton3-prot-gitlab-runner-6599b79548-gl7z6 | 37 | 8 | 21.62% | other |
runner-graviton2-pub-gitlab-runner-5bd7f7f788-g9n4r | 14 | 3 | 21.43% | AWS |
runner-graviton2-pub-gitlab-runner-5bd7f7f788-hzsq9 | 20 | 4 | 20.00% | AWS |
runner-graviton2-pub-gitlab-runner-5bd7f7f788-zw4hk | 18 | 3 | 16.67% | AWS |
runner-x86-v2-pub-gitlab-runner-78f8c998ff-pfgxq | 32 | 5 | 15.63% | AWS |
runner-graviton3-pub-gitlab-runner-7494b74c85-2264k | 14 | 2 | 14.29% | AWS |
runner-x86-v2-pub-gitlab-runner-78f8c998ff-js9ll | 36 | 5 | 13.89% | AWS |
runner-x86-v2-pub-gitlab-runner-78f8c998ff-7pmrx | 36 | 5 | 13.89% | AWS |
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-c2qhn | 25 | 3 | 12.00% | AWS |
runner-graviton2-pub-gitlab-runner-7b9c95694-55rsw | 10 | 1 | 10.00% | AWS |
runner-x86-v3-pub-gitlab-runner-67b8df5756-qlqmh | 31 | 3 | 9.68% | AWS |
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-wbzxs | 44 | 4 | 9.09% | AWS |
runner-x86-v3-pub-gitlab-runner-67b8df5756-g7sqj | 61 | 5 | 8.20% | AWS |
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-hbbxs | 60 | 3 | 5.00% | AWS |
runner-x86-v3-pub-gitlab-runner-67b8df5756-qhzhx | 85 | 4 | 4.71% | AWS |
runner-x86-v4-pub-gitlab-runner-5c9f88985-rh5rk | 61 | 2 | 3.28% | AWS |
runner-graviton2-pub-gitlab-runner-7b9c95694-dm6bf | 63 | 2 | 3.17% | AWS |
runner-graviton3-prot-gitlab-runner-6599b79548-44p5m | 284 | 9 | 3.17% | other |
runner-x86-v3-prot-gitlab-runner-775b867b79-chss2 | 251 | 7 | 2.79% | other |
runner-x86-v4-prot-gitlab-runner-95f7bc676-k7p5p | 257 | 6 | 2.33% | other |
runner-x86-v4-pub-gitlab-runner-5c9f88985-skkf6 | 49 | 1 | 2.04% | AWS |
runner-x86-v4-pub-gitlab-runner-7945ddd9f7-dqqwx | 60 | 1 | 1.67% | AWS |
runner-graviton3-pub-gitlab-runner-7494b74c85-hj4pp | 140 | 2 | 1.43% | AWS |
runner-graviton3-prot-gitlab-runner-6599b79548-2sdgb | 303 | 4 | 1.32% | other |
runner-x86-v3-prot-gitlab-runner-775b867b79-2d9p5 | 233 | 3 | 1.29% | other |
runner-x86-v3-pub-gitlab-runner-67b8df5756-n7p6t | 79 | 1 | 1.27% | AWS |
runner-x86-v3-prot-gitlab-runner-775b867b79-gc4ws | 241 | 3 | 1.24% | other |
runner-x86-v3-pub-gitlab-runner-67b8df5756-9rzlv | 89 | 1 | 1.12% | AWS |
runner-graviton3-pub-gitlab-runner-7494b74c85-wcrhp | 92 | 1 | 1.09% | AWS |
runner-x86-v4-prot-gitlab-runner-95f7bc676-pgwp6 | 283 | 3 | 1.06% | other |
runner-x86-v3-prot-gitlab-runner-775b867b79-kfz8c | 223 | 1 | 0.45% | other |
runner-x86-v4-prot-gitlab-runner-95f7bc676-m778h | 248 | 1 | 0.40% | other |
runner-x86-v2-pub-gitlab-runner-6d88c8459c-9rpmx | 276 | 1 | 0.36% | AWS |