What changed
AWS introduced enhanced observability in Amazon SageMaker HyperPod, a new capability that helps users diagnose GPU training issues faster. This feature brings together diagnostic and profiling capabilities for training workloads running on both Amazon Elastic Kubernetes Service (Amazon EKS) and Slurm orchestrators.
What this means in practice
The enhanced observability collects telemetry at two levels: default metrics and advanced metrics. Default metrics are enabled automatically and cover cluster health, CPU/disk/OS-level usage, GPU utilization and temperature, and per-job resource utilization. These metrics are collected every 30 seconds and sent to the chosen metric store with approximately 1–2% CPU overhead and 250–300 MB of memory. Advanced metrics are optional and provide deeper visibility, including full node exporter suite metrics, NVIDIA GPU DCGM and AWS Neuron metrics, and Elastic Fabric Adapter (EFA) metrics.
The diagnostic summary uses Amazon CloudWatch’s DetectGroupOutliers API to compare GPUs within a training job across utilization, memory usage, temperature, power draw, and SM clock frequency. It flags GPUs that deviate from their peers, helping identify issues such as GPU hardware faults, EFA throughput drops, thermal throttling, and overheating.
Limitations and availability
Enhanced observability is delivered as an add-on for EKS-orchestrated clusters and as an extension for Slurm-orchestrated clusters. Users control who can install or change it through AWS Identity and Access Management (IAM). The feature is available through the AWS Management Console, API, or AWS CLI. Metric stores can be chosen between Amazon CloudWatch (new default) or Amazon Managed Service for Prometheus. Advanced metrics are optional due to their higher telemetry volume and cost implications. Most customers running small jobs on small clusters may not need them, but they are essential for long-running training jobs on large clusters.
