Beginning: 2021-09-22 12:48 AM PDT
Ending: 2026-05-21 05:34 AM PDT
Total Jobs: 11,358,897
Pipeline failures over time
Jobs per pipeline, overview
Summary of Pipeline Errors
---
UO Frank Node Descriptions
---
UO Runners, Last 100 Completed Jobs
AWS Runners, Last 100 Completed Jobs
---
GPU Usage, UO, Last Week
Job Times, Last 72 Hours
Job Times, Last Week
Job Times, Overview, All
Runner System Failures, by Type, Last 24 Hours
Runner System Failures, by Runner, Last 24 Hours
| GPU Type | # Jobs |
|---|---|
| AMD MI100 | 95 |
| AMD MI210 | 211 |
| AMD MI300A | 296 |
| AMD MI50 | 14 |
| INTEL A770 | 95 |
| INTEL DATA CENTER MAX 1100 | 272 |
| NVIDIA A100 | 125 |
| NVIDIA A2000 | 14 |
| NVIDIA GB10 | 113 |
| NVIDIA GH200 | 116 |
| NVIDIA H100 | 155 |
| NVIDIA P100 | 115 |
| NVIDIA RTX5080 | 113 |
| NVIDIA RTX6000 | 16 |
| NVIDIA V100 (Power9) | 7 |
| Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
|---|---|---|---|---|---|---|
| AWS Packages | 51.9 hr | 1.8% | 8.67 min | 359 | 65% | 35% |
| App CI - E4S - Adios2 | 23.16 hr | 0.8% | 1.42 min | 982 | 100% | 0% |
| App CI - E4S - ExaGO | .24 hr | 0.0% | 1.59 min | 9 | 100% | 0% |
| App CI - E4S - ExaWind | 0. hr | 0.0% | 0.04 min | 3 | 100% | 0% |
| App CI - E4S - HPCToolkit | 5.56 hr | 0.2% | 4.22 min | 79 | 94% | 0% |
| App CI - E4S - Kokkos | 264.27 hr | 9.0% | 33.24 min | 477 | 100% | 0% |
| App CI - E4S - LLVM OpenMP Offloading | 71.34 hr | 2.4% | 17.40 min | 246 | 100% | 0% |
| App CI - E4S - TAU | 16.9 hr | 0.6% | 11.66 min | 87 | 100% | 0% |
| App CI - E4S - Trilinos | 15.56 hr | 0.5% | 19.45 min | 48 | 100% | 0% |
| Build Systems | .31 hr | 0.0% | 4.70 min | 4 | 50% | 50% |
| Data and Vis SDK | 117.59 hr | 4.0% | 8.35 min | 845 | 56% | 44% |
| Developer Tools aarch64-linux-gnu | 1.31 hr | 0.0% | 11.23 min | 7 | 71% | 29% |
| Developer Tools x86_64_v3-linux-gnu | 5.65 hr | 0.2% | 6.92 min | 49 | 67% | 33% |
| Developer Tools, Darwin | 2. hr | 0.1% | 7.49 min | 16 | 100% | 0% |
| E4S | 1,061.67 hr | 36.3% | 8.37 min | 7,613 | 34% | 66% |
| E4S ARM Neoverse V2 | 76.75 hr | 2.6% | 4.75 min | 970 | 35% | 65% |
| E4S Cray RHEL | 5.46 hr | 0.2% | 3.72 min | 88 | 100% | 0% |
| E4S OneAPI | 133. hr | 4.5% | 7.56 min | 1,055 | 28% | 72% |
| E4S ROCm External | 10.98 hr | 0.4% | 9.98 min | 66 | 59% | 41% |
| High Energy Physics | 119.96 hr | 4.1% | 7.66 min | 940 | 40% | 60% |
| Machine Learning | 478.55 hr | 16.3% | 6.02 min | 4,773 | 77% | 23% |
| RADIUSS | 53.13 hr | 1.8% | 3.54 min | 901 | 59% | 41% |
| Spack CI | 1.23 hr | 0.0% | 3.51 min | 21 | 95% | 5% |
| Spack Tutorial | 28.88 hr | 1.0% | 3.97 min | 437 | 68% | 32% |
| Tools SDK | 4.42 hr | 0.2% | 2.50 min | 106 | 69% | 31% |
| Windows Visualization (Kitware) | 52.3 hr | 1.8% | 8.44 min | 372 | 70% | 30% |
| dotenv | 1.33 hr | 0.0% | 0.76 min | 105 | 0% | 100% |
| generate | 290.07 hr | 9.9% | 8.90 min | 1,956 | 70% | 30% |
| no-specs-to-rebuild | 8.77 hr | 0.3% | 0.31 min | 1,699 | 76% | 24% |
| other | 4.4 hr | 0.2% | 7.13 min | 37 | 100% | 0% |
| rebuild-index | 14.84 hr | 0.5% | 5.00 min | 178 | 0% | 100% |
| sign-pkgs | 5.47 hr | 0.2% | 7.45 min | 44 | 0% | 100% |
| TOTAL | 2,927.01 hr | -- | -- | 24,572 | 57% | 43% |
| Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
|---|---|---|---|---|---|---|
| AWS Packages | 69.6 hr | 1.3% | 6.96 min | 600 | 66% | 34% |
| App CI - E4S - Adios2 | 40.87 hr | 0.8% | 1.09 min | 2,250 | 100% | 0% |
| App CI - E4S - ExaGO | .57 hr | 0.0% | 1.62 min | 21 | 100% | 0% |
| App CI - E4S - ExaWind | 0. hr | 0.0% | 0.04 min | 7 | 100% | 0% |
| App CI - E4S - HPCToolkit | 9.05 hr | 0.2% | 2.97 min | 183 | 97% | 0% |
| App CI - E4S - Kokkos | 447.55 hr | 8.4% | 33.78 min | 795 | 100% | 0% |
| App CI - E4S - LLVM OpenMP Offloading | 87.84 hr | 1.6% | 12.70 min | 415 | 100% | 0% |
| App CI - E4S - TAU | 23.2 hr | 0.4% | 6.86 min | 203 | 100% | 0% |
| App CI - E4S - Trilinos | 34.95 hr | 0.7% | 18.24 min | 115 | 100% | 0% |
| Build Systems | 2.12 hr | 0.0% | 2.65 min | 48 | 73% | 27% |
| Data and Vis SDK | 241.39 hr | 4.5% | 8.34 min | 1,736 | 62% | 38% |
| Developer Tools aarch64-linux-gnu | 2.96 hr | 0.1% | 5.55 min | 32 | 81% | 19% |
| Developer Tools x86_64_v3-linux-gnu | 8.21 hr | 0.2% | 6.84 min | 72 | 61% | 39% |
| Developer Tools, Darwin | 12.35 hr | 0.2% | 3.61 min | 205 | 100% | 0% |
| E4S | 2,006.28 hr | 37.5% | 7.77 min | 15,487 | 40% | 60% |
| E4S ARM Neoverse V2 | 98.57 hr | 1.8% | 4.88 min | 1,213 | 41% | 59% |
| E4S Cray RHEL | 8.83 hr | 0.2% | 3.63 min | 146 | 100% | 0% |
| E4S OneAPI | 155.59 hr | 2.9% | 6.50 min | 1,437 | 33% | 67% |
| E4S ROCm External | 25.03 hr | 0.5% | 8.03 min | 187 | 56% | 44% |
| High Energy Physics | 399.38 hr | 7.5% | 4.77 min | 5,028 | 60% | 40% |
| Machine Learning | 931.72 hr | 17.4% | 6.58 min | 8,491 | 70% | 30% |
| RADIUSS | 63.73 hr | 1.2% | 3.68 min | 1,040 | 60% | 40% |
| Spack CI | 4.5 hr | 0.1% | 3.81 min | 71 | 85% | 15% |
| Spack Tutorial | 31.43 hr | 0.6% | 3.94 min | 479 | 67% | 33% |
| Tools SDK | 5.81 hr | 0.1% | 2.68 min | 130 | 69% | 31% |
| Windows Visualization (Kitware) | 76.09 hr | 1.4% | 9.76 min | 468 | 69% | 31% |
| dotenv | 2.57 hr | 0.0% | 0.63 min | 246 | 0% | 100% |
| generate | 503.47 hr | 9.4% | 6.74 min | 4,484 | 72% | 28% |
| no-specs-to-rebuild | 15.46 hr | 0.3% | 0.24 min | 3,939 | 79% | 21% |
| other | 7.76 hr | 0.1% | 5.48 min | 85 | 100% | 0% |
| rebuild-index | 27.82 hr | 0.5% | 3.39 min | 492 | 0% | 100% |
| sign-pkgs | 9.89 hr | 0.2% | 7.61 min | 78 | 0% | 100% |
| TOTAL | 5,354.57 hr | -- | -- | 50,183 | 60% | 40% |
| Job Type | Runtime | % Total Runtime | Average Runtime | N | % UO | % AWS |
|---|---|---|---|---|---|---|
| AHUG ARM HPC User Group | 27.39 days | 0.1% | 5.75 min | 6,858 | 14% | 85% |
| AWS Packages | 1,795.34 days | 3.7% | 6.13 min | 421,669 | 41% | 56% |
| Build Systems | 45.98 days | 0.1% | 4.08 min | 16,225 | 72% | 26% |
| Copy | 20.21 days | 0.0% | 7.24 min | 4,022 | 14% | 85% |
| Data and Vis SDK | 1,622.84 days | 3.3% | 8.14 min | 287,124 | 54% | 44% |
| Developer Tools | 35.13 days | 0.1% | 5.51 min | 9,179 | 67% | 32% |
| Developer Tools Manylinux2014 | 52.08 days | 0.1% | 5.99 min | 12,510 | 63% | 33% |
| Developer Tools aarch64-linux-gnu | 19.87 days | 0.0% | 3.20 min | 8,934 | 64% | 35% |
| Developer Tools x86_64_v3-linux-gnu | 26.81 days | 0.1% | 4.01 min | 9,641 | 67% | 33% |
| Developer Tools, Darwin | 24.35 days | 0.0% | 3.42 min | 10,255 | 100% | 0% |
| E4S | 11,266.4 days | 23.1% | 8.54 min | 1,900,239 | 49% | 49% |
| E4S ARM Neoverse V2 | 1,143.24 days | 2.3% | 5.67 min | 290,450 | 86% | 14% |
| E4S Cray | 313.65 days | 0.6% | 3.74 min | 121,434 | 100% | 0% |
| E4S OneAPI | 1,645.91 days | 3.4% | 5.34 min | 444,174 | 47% | 51% |
| E4S ROCm External | 856.71 days | 1.8% | 9.97 min | 123,684 | 55% | 43% |
| GCP Packages | .21 days | 0.0% | 3.78 min | 81 | 0% | 99% |
| GPU Testing | 35.1 days | 0.1% | 6.09 min | 8,376 | 70% | 28% |
| High Energy Physics | 560.15 days | 1.1% | 9.10 min | 88,600 | 55% | 45% |
| Machine Learning | 6,314.29 days | 13.0% | 7.57 min | 1,200,720 | 56% | 42% |
| RADIUSS | 341.58 days | 0.7% | 4.47 min | 109,983 | 57% | 42% |
| Spack CI | 25.6 days | 0.1% | 2.53 min | 14,572 | 75% | 25% |
| Spack Deprecated CI | 1.15 days | 0.0% | 3.96 min | 417 | 76% | 21% |
| Spack Tutorial | 247.04 days | 0.5% | 4.89 min | 72,785 | 65% | 34% |
| Tools SDK | 3.26 days | 0.0% | 2.63 min | 1,785 | 66% | 34% |
| Windows Visualization (Kitware) | 106.84 days | 0.2% | 16.87 min | 9,120 | 0% | 95% |
| cleanup | 8.56 days | 0.0% | 1.03 min | 12,022 | 88% | 12% |
| dotenv | 3.08 days | 0.0% | 0.58 min | 7,685 | 0% | 100% |
| generate | 3,824.64 days | 7.8% | 7.19 min | 766,838 | 58% | 41% |
| no-specs-to-rebuild | 336.39 days | 0.7% | 0.57 min | 853,709 | 22% | 76% |
| other | 897.75 days | 1.8% | 7.33 min | 178,092 | 90% | 0% |
| pr-generate | 1,186.42 days | 2.4% | 6.53 min | 261,741 | 35% | 63% |
| protected-generate | 84.93 days | 0.2% | 6.61 min | 18,725 | 26% | 70% |
| protected-publish | 41.74 days | 0.1% | 38.04 min | 1,580 | 0% | 98% |
| rebuild, aarch64 | 1,029.06 days | 2.1% | 5.51 min | 269,139 | 0% | 97% |
| rebuild, cray, zen4 | .58 days | 0.0% | 1.16 min | 851 | 100% | 0% |
| rebuild, macOS | 23.29 days | 0.0% | 1.45 min | 23,191 | 100% | 0% |
| rebuild, other | 1,610.36 days | 3.3% | 6.41 min | 363,018 | 2% | 77% |
| rebuild, ppc64le | 953.52 days | 2.0% | 9.81 min | 140,428 | 97% | 0% |
| rebuild, x86_64 | 9,865.17 days | 20.2% | 5.86 min | 2,423,539 | 47% | 53% |
| rebuild, x86_64 (p) | 1,912.96 days | 3.9% | 7.46 min | 369,092 | 3% | 97% |
| rebuild-index | 286.84 days | 0.6% | 3.23 min | 128,009 | 25% | 73% |
| sign-pkgs | 151.1 days | 0.3% | 8.57 min | 25,460 | 0% | 97% |
| TOTAL | 48,747.53 days | -- | -- | 11,015,956 | 46% | 52% |
| Failure Type | N | # AWS | # UO | % of All System Failures |
|---|---|---|---|---|
| other | 135 | 0 | 135 | 65.53% |
| pod-unready | 69 | 69 | 0 | 33.50% |
| pod-not-found | 2 | 2 | 0 | 0.97% |
| Runner | Total Jobs | System Failures # | System Failures % | Facility |
|---|---|---|---|---|
| gilgamesh-nccl | 1 | 1 | 100.00% | other |
| gilgamesh-docker-nvidia-h100-trilinos | 3 | 3 | 100.00% | other |
| gilgamesh-docker-nvidia-h100 | 3 | 3 | 100.00% | other |
| uo-windows-4 | 128 | 128 | 100.00% | UO |
| runner-graviton3-prot-gitlab-runner-5c87ffd587-j9b9h | 49 | 5 | 10.20% | other |
| runner-graviton3-pub-gitlab-runner-69bfff4f4c-2n2sp | 28 | 2 | 7.14% | AWS |
| runner-graviton3-prot-gitlab-runner-5c87ffd587-srm5n | 30 | 2 | 6.67% | other |
| runner-graviton3-pub-gitlab-runner-69bfff4f4c-sp6mz | 21 | 1 | 4.76% | AWS |
| runner-spack-package-signing-gitlab-runner-69cb7f96fb-lwpv4 | 21 | 1 | 4.76% | other |
| runner-x86-v4-prot-gitlab-runner-79b8b5975-6k5pv | 210 | 9 | 4.29% | other |
| runner-x86-v4-prot-gitlab-runner-79b8b5975-fr97p | 183 | 7 | 3.83% | other |
| runner-x86-v4-pub-gitlab-runner-c4f7c56f6-hkdpz | 120 | 4 | 3.33% | AWS |
| runner-x86-v3-prot-gitlab-runner-6c475ccbc7-cscnc | 151 | 5 | 3.31% | other |
| runner-x86-v4-prot-gitlab-runner-79b8b5975-gkcxq | 212 | 6 | 2.83% | other |
| runner-x86-v4-pub-gitlab-runner-c4f7c56f6-9zx2v | 106 | 3 | 2.83% | AWS |
| runner-x86-v4-prot-gitlab-runner-79b8b5975-jpsdn | 199 | 5 | 2.51% | other |
| runner-graviton3-prot-gitlab-runner-5c87ffd587-t4n8g | 40 | 1 | 2.50% | other |
| runner-x86-v4-prot-gitlab-runner-79b8b5975-f6k9r | 203 | 5 | 2.46% | other |
| runner-x86-v4-prot-gitlab-runner-79b8b5975-5ltx6 | 176 | 4 | 2.27% | other |
| runner-x86-v3-prot-gitlab-runner-6c475ccbc7-vzlbg | 143 | 3 | 2.10% | other |
| runner-x86-v4-pub-gitlab-runner-c4f7c56f6-q6dw9 | 117 | 2 | 1.71% | AWS |
| runner-x86-v3-pub-gitlab-runner-6d566597b-99fpx | 87 | 1 | 1.15% | AWS |
| runner-x86-v4-pub-gitlab-runner-c4f7c56f6-5676h | 92 | 1 | 1.09% | AWS |
| runner-x86-v3-prot-gitlab-runner-6c475ccbc7-d4nj7 | 122 | 1 | 0.82% | other |
| runner-x86-v3-pub-gitlab-runner-6d566597b-l7xq7 | 125 | 1 | 0.80% | AWS |
| runner-x86-v3-prot-gitlab-runner-6c475ccbc7-v4899 | 132 | 1 | 0.76% | other |
| runner-x86-v3-prot-gitlab-runner-6c475ccbc7-bkjmm | 141 | 1 | 0.71% | other |