Beginning: 2021-09-22 12:48 AM PDT
Ending: 2024-11-20 03:35 PM PDT
Total Jobs: 8,310,965
Pipeline Summary
Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
Job Times, Last 24 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours
| Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
|---|---|---|---|---|---|---|
| sign-pkgs | 3.45 hr | 0.2% | 4.71 min | 44 | 0% | 100% |
| rebuild-index | 3.53 hr | 0.2% | 1.22 min | 173 | 0% | 95% |
| other | 341.12 hr | 23.1% | 7.58 min | 2,700 | 28% | 62% |
| no-specs-to-rebuild | 5.2 hr | 0.4% | 0.35 min | 889 | 0% | 98% |
| generate | 134.47 hr | 9.1% | 7.33 min | 1,101 | 54% | 40% |
| Spack Tutorial, x86_64 | 9.44 hr | 0.6% | 7.98 min | 71 | 56% | 41% |
| RADIUSS, x86_64 | 5.94 hr | 0.4% | 6.26 min | 57 | 51% | 44% |
| Machine Learning, x86_64 | 242.5 hr | 16.4% | 11.21 min | 1,298 | 48% | 41% |
| Machine Learning MPS, macOS aarch64 | 19.72 hr | 1.3% | 5.19 min | 228 | 100% | 0% |
| E4S, x86_64 | 274.39 hr | 18.6% | 9.89 min | 1,665 | 57% | 38% |
| E4S ROCm External | 21.72 hr | 1.5% | 17.61 min | 74 | 66% | 27% |
| E4S Power, ppc64le | 159.24 hr | 10.8% | 10.26 min | 931 | 100% | 0% |
| E4S OneAPI, x86_64 | 149.76 hr | 10.1% | 3.82 min | 2,355 | 63% | 25% |
| Data and Vis SDK, x86_64 | 62.06 hr | 4.2% | 8.60 min | 433 | 59% | 37% |
| AWS Packages, x86_64 | 21.08 hr | 1.4% | 6.91 min | 183 | 61% | 34% |
| AWS Packages, neoverse_n1 | 11.89 hr | 0.8% | 4.99 min | 143 | 35% | 62% |
| AWS Packages, aarch64 | 10.18 hr | 0.7% | 3.32 min | 184 | 36% | 61% |
| TOTAL | 1,475.68 hr | -- | -- | 12,529 | 49% | 43% |
| Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
|---|---|---|---|---|---|---|
| sign-pkgs | 17.11 hr | 0.1% | 5.83 min | 176 | 0% | 97% |
| rebuild-index | 19.31 hr | 0.2% | 1.14 min | 1,012 | 0% | 99% |
| protected-publish | 4.33 hr | 0.0% | 129.95 min | 2 | 0% | 50% |
| other | 2,972.94 hr | 24.5% | 6.04 min | 29,540 | 38% | 57% |
| no-specs-to-rebuild | 26.66 hr | 0.2% | 0.31 min | 5,134 | 0% | 99% |
| generate | 928.96 hr | 7.6% | 8.01 min | 6,959 | 52% | 46% |
| Spack Tutorial, x86_64 | 131.66 hr | 1.1% | 4.58 min | 1,723 | 56% | 41% |
| RADIUSS, x86_64 | 145.41 hr | 1.2% | 4.00 min | 2,183 | 43% | 53% |
| Machine Learning, x86_64 | 1,561.2 hr | 12.9% | 8.99 min | 10,424 | 33% | 63% |
| Machine Learning MPS, macOS aarch64 | 340.41 hr | 2.8% | 5.51 min | 3,709 | 100% | 0% |
| E4S, x86_64 | 1,909.86 hr | 15.7% | 7.47 min | 15,344 | 38% | 59% |
| E4S ROCm External | 214.56 hr | 1.8% | 10.77 min | 1,195 | 34% | 58% |
| E4S Power, ppc64le | 1,118.13 hr | 9.2% | 9.05 min | 7,409 | 100% | 0% |
| E4S OneAPI, x86_64 | 930.2 hr | 7.7% | 4.85 min | 11,503 | 43% | 53% |
| E4S Cray, SLES, x86_64 | 25.68 hr | 0.2% | 2.94 min | 524 | 100% | 0% |
| E4S Cray, RHEL8, zen4 | 79.42 hr | 0.7% | 3.51 min | 1,357 | 100% | 0% |
| Data and Vis SDK, x86_64 | 738.04 hr | 6.1% | 6.68 min | 6,631 | 45% | 52% |
| AWS Packages, x86_64 | 418.46 hr | 3.4% | 5.67 min | 4,428 | 41% | 55% |
| AWS Packages, neoverse_n1 | 320.26 hr | 2.6% | 4.14 min | 4,637 | 28% | 68% |
| AWS Packages, aarch64 | 241.68 hr | 2.0% | 3.52 min | 4,122 | 34% | 63% |
| TOTAL | 12,144.28 hr | -- | -- | 118,012 | 44% | 53% |
| Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
|---|---|---|---|---|---|---|
| sign-pkgs | 93.3 days | 0.3% | 8.31 min | 16,236 | 0% | 97% |
| rebuild-index | 183.15 days | 0.6% | 3.33 min | 79,193 | 40% | 58% |
| rebuild, x86_64 (p) | 1,912.96 days | 5.8% | 7.46 min | 369,092 | 3% | 97% |
| rebuild, x86_64 | 9,865.17 days | 29.9% | 5.86 min | 2,423,539 | 47% | 53% |
| rebuild, ppc64le | 953.52 days | 2.9% | 9.81 min | 140,428 | 97% | 0% |
| rebuild, other | 1,610.36 days | 4.9% | 6.41 min | 363,018 | 2% | 77% |
| rebuild, macOS | 23.29 days | 0.1% | 1.45 min | 23,191 | 100% | 0% |
| rebuild, cray, zen4 | .58 days | 0.0% | 1.16 min | 851 | 100% | 0% |
| rebuild, aarch64 | 1,029.06 days | 3.1% | 5.51 min | 269,139 | 0% | 97% |
| protected-publish | 41.56 days | 0.1% | 37.93 min | 1,578 | 0% | 98% |
| protected-generate | 84.93 days | 0.3% | 6.61 min | 18,725 | 25% | 70% |
| pr-generate | 1,186.42 days | 3.6% | 6.53 min | 261,741 | 35% | 63% |
| other | 2,470.18 days | 7.5% | 7.76 min | 458,668 | 38% | 59% |
| no-specs-to-rebuild | 222.35 days | 0.7% | 0.63 min | 508,868 | 34% | 64% |
| generate | 1,906.84 days | 5.8% | 7.52 min | 365,506 | 58% | 41% |
| copy-mirror | .94 days | 0.0% | 1.87 min | 723 | 0% | 100% |
| cleanup | 8.56 days | 0.0% | 1.03 min | 12,022 | 88% | 12% |
| Spack Tutorial, x86_64 | 145.23 days | 0.4% | 5.69 min | 36,785 | 70% | 29% |
| RADIUSS, x86_64 | 218.12 days | 0.7% | 5.13 min | 61,247 | 58% | 40% |
| Machine Learning, x86_64 | 2,666.46 days | 8.1% | 8.27 min | 464,125 | 52% | 47% |
| Machine Learning MPS, macOS aarch64 | 370.33 days | 1.1% | 4.56 min | 116,905 | 100% | 0% |
| E4S, x86_64 | 2,627.47 days | 8.0% | 9.48 min | 399,190 | 54% | 45% |
| E4S ROCm External | 485.02 days | 1.5% | 10.52 min | 66,410 | 57% | 41% |
| E4S Power, ppc64le | 1,798.94 days | 5.4% | 9.67 min | 267,952 | 100% | 0% |
| E4S OneAPI, x86_64 | 834.52 days | 2.5% | 6.46 min | 185,945 | 52% | 46% |
| E4S Cray, SLES, x86_64 | 76. days | 0.2% | 3.84 min | 28,618 | 100% | 0% |
| E4S Cray, RHEL8, zen4 | 124.81 days | 0.4% | 3.75 min | 48,262 | 100% | 0% |
| E4S ARM | 75.52 days | 0.2% | 7.33 min | 14,846 | 0% | 100% |
| Data and Vis SDK, x86_64 | 862.57 days | 2.6% | 8.16 min | 152,268 | 55% | 43% |
| AWS Packages, x86_64 | 493.18 days | 1.5% | 7.96 min | 89,164 | 59% | 40% |
| AWS Packages, neoverse_n1 | 335.69 days | 1.0% | 6.03 min | 80,186 | 26% | 72% |
| AWS Packages, aarch64 | 302.77 days | 0.9% | 5.80 min | 75,177 | 27% | 71% |
| TOTAL | 33,009.81 days | -- | -- | 7,399,598 | 45% | 53% |
| Failure Type | N | # AWS | # UO | % of All System Failures |
|---|---|---|---|---|
| pod-unready | 163 | 151 | 0 | 75.81% |
| dial-backend | 35 | 35 | 0 | 16.28% |
| connection-upgrade | 11 | 11 | 0 | 5.12% |
| pod-not-found | 4 | 4 | 0 | 1.86% |
| other | 2 | 0 | 0 | 0.93% |
| Runner | Total Jobs | System Failures # | System Failures % | Facility |
|---|---|---|---|---|
| runner-graviton2-pub-gitlab-runner-7b9c95694-59ntr | 1 | 1 | 100.00% | AWS |
| runner-graviton2-pub-gitlab-runner-7b9c95694-5xfhd | 1 | 1 | 100.00% | AWS |
| runner-graviton2-pub-gitlab-runner-7b9c95694-dm6bf | 1 | 1 | 100.00% | AWS |
| runner-graviton2-pub-gitlab-runner-5fcd4dc657-tsh8x | 167 | 38 | 22.75% | AWS |
| runner-graviton2-pub-gitlab-runner-5fcd4dc657-wrqv2 | 181 | 37 | 20.44% | AWS |
| runner-graviton2-pub-gitlab-runner-5fcd4dc657-pnc2k | 178 | 31 | 17.42% | AWS |
| runner-graviton2-pub-gitlab-runner-5fcd4dc657-hd9kh | 248 | 42 | 16.94% | AWS |
| runner-graviton2-pub-gitlab-runner-5fcd4dc657-9hxv7 | 63 | 5 | 7.94% | AWS |
| runner-graviton3-pub-gitlab-runner-78bb9fcdd4-vmqtq | 136 | 6 | 4.41% | AWS |
| runner-graviton2-pub-gitlab-runner-5fcd4dc657-9xjj7 | 324 | 11 | 3.40% | AWS |
| runner-x86-v3-pub-gitlab-runner-6b45c99db5-cs2wv | 69 | 2 | 2.90% | AWS |
| runner-x86-v3-pub-gitlab-runner-6b45c99db5-kcwz7 | 78 | 2 | 2.56% | AWS |
| runner-spack-package-signing-gitlab-runner-bd87f8896-7w68w | 44 | 1 | 2.27% | other |
| runner-graviton3-pub-gitlab-runner-78bb9fcdd4-fm52k | 137 | 3 | 2.19% | AWS |
| runner-x86-v2-pub-gitlab-runner-5f78688ffc-bdc8c | 48 | 1 | 2.08% | AWS |
| runner-x86-v4-pub-gitlab-runner-658f977f55-hg4t5 | 171 | 3 | 1.75% | AWS |
| runner-x86-v4-pub-gitlab-runner-658f977f55-vjfs7 | 311 | 4 | 1.29% | AWS |
| runner-x86-v4-pub-gitlab-runner-658f977f55-p9j8r | 91 | 1 | 1.10% | AWS |
| runner-x86-v2-pub-gitlab-runner-5f78688ffc-xhm78 | 105 | 1 | 0.95% | AWS |
| runner-graviton3-pub-gitlab-runner-78bb9fcdd4-59gfr | 289 | 2 | 0.69% | AWS |
| runner-x86-v2-pub-gitlab-runner-5f78688ffc-h67pt | 149 | 1 | 0.67% | AWS |
| runner-x86-v3-pub-gitlab-runner-6b45c99db5-hnx95 | 240 | 1 | 0.42% | AWS |
| runner-x86-v3-pub-gitlab-runner-6b45c99db5-hzg7r | 262 | 1 | 0.38% | AWS |
| runner-graviton3-pub-gitlab-runner-78bb9fcdd4-xs5kg | 268 | 1 | 0.37% | AWS |
| runner-x86-v4-pub-gitlab-runner-658f977f55-54s2g | 304 | 1 | 0.33% | AWS |
| runner-x86-v4-pub-gitlab-runner-658f977f55-mbn9r | 302 | 1 | 0.33% | AWS |
| runner-x86-v3-pub-gitlab-runner-6b45c99db5-fkxcm | 308 | 1 | 0.32% | AWS |
| runner-graviton3-pub-gitlab-runner-78bb9fcdd4-nrjcr | 366 | 1 | 0.27% | AWS |