Reproducing NVIDIA MLPerf v5.0 Training Scores for LLM Benchmarks

The previous post, NVIDIA Blackwell Delivers up to 2.6x Higher Performance in MLPerf Training v5.0, explains how the NVIDIA platform delivered the fastest time to train across all seven benchmarks in this latest MLPerf round. This post provides a guide to reproduce the performance of NVIDIA MLPerf v5.0 submissions of Llama 2 70B LoRA fine-tuning and Llama 405B pretraining. Submission repositories also include README files to reproduce the scores. See, for example, those for the Llama 2 70B LoRA fine-tuning benchmark and the Llama 3.1 405B benchmark. 

Prerequisites

Running NVIDIA benchmarks requires your system to have the following:

Container preparation, dataset/checkpoint download and preprocessing Docker A Hugging Face access token (for dataset/checkpoint download) At least 2.5 TB of disk space for Llama 3.1, 300 GB for LoRA fine-tuning Hardware requirements Llama 2 70B LoRA: An NVIDIA DGX B200 or NVIDIA GB200 NVL72 system, or multiple GB200 NVL72 systems connected with InfiniBand for scales larger than 72 GPUs. The smallest NVIDIA submission for this benchmark is eight GPUs. Llama 3.1 405B: At least four GB200 NVL72 systems connected with InfiniBand. The smallest NVIDIA submission for this benchmark is 256 GPUs. 

Cluster setup

Running NVIDIA MLPerf Training benchmarks requires:

Environment based on Slurm, Pyxis, and Enroot Networking with NVIDIA NVLink and InfiniBand Fast local storage set up in RAID0 configuration to minimize data loading bottlenecks

NVIDIA submission clusters do not support running workloads with Docker. The clusters are governed by the NVIDIA Base Command Manager (BCM). Follow the official instructions to properly set up a BCM SLURM cluster.

After a proper setup, you should be able to log in to the head node and access SLURM commands (sinfo, squeue, srun, sbatch) to launch jobs on the compute nodes.

Running benchmarks

The steps necessary to start benchmarking any model include the following:

Build a Docker container. Run the container on any machine with Docker to download and process the dataset and the checkpoint. This step can be done on any machine, in the cluster or not, and it generally doesn’t require a system with a GPU. Make sure the data is accessible by the compute nodes. Preferably the data is stored locally on the node; alternatively, it is accessible through a fast (parallel) file system. Launch the training and parse the logs.

Llama 2 70B LoRA

To run benchmarks for Llama 2 70B LoRA, follow the instructions in this section.

Build the container

Clone the mlcommons/training_results_v5.0 GitHub repo. cd NVIDIA/benchmarks/llama2_70b_lora/implementations/tyche_ngpu72_ngc25.04_nemo. Docker build -t mlperf-nvidia:llama2_70b_lora-pyt. If you have a registry you would like to push the image to, add the registry name to the image name.

Download the dataset and model

This benchmark uses the GovReport dataset and a Hugging Face checkpoint. Both the dataset and the checkpoint require preprocessing to be used by NVIDIA NeMo. You need a Hugging Face token to download the checkpoint.

To download and preprocess, do the following:

If the model download failed, you might need to export your HF_TOKEN before you call the download_model.py script:

After conversion you should see the following files in the /data directory:

You may exit the container at this point.

Launch the benchmarking

NVIDIA uses SLURM to launch benchmarks on the compute nodes. Two files are used to facilitate the job launch process:

A configuration file (config_*.sh) that describes the hyperparameters of the model, including the number of nodes, walltime, and so on. Organizing these in a single file per submission enables easy configuration of the workload to run at desired scale with optimal hyperparameters. A fixed run.sub file that contains srun commands to launch the training, passing all the hyperparameters from the config to the Python script.

To take a look at a typical config file, in this case config_GB200_18x4x1xtp1pp1cp8.sh, the name describes the size and the type of the system:

GB200: Designed to run on a GB200 machine 18×4: A system setting that can be decoded as NNODES x NGPUS NNODES: Number of GB200 nodes NGPUS: Number of GPUs per node x1xtp1pp1cp8 is a parallelization schema x1 is GradientAccumulation, here equal to 1, meaning no GA TP1: No TensorParallel PP1: No PipelineParallel CP8: 8-way ContextParallel

This means a 72-GPU configuration will be used, running on a single GB200 NVL72 rack. The benchmark will run GA1TP1PP1CP8: Global batch size (GBS) = 9.

Next, take a closer look at the content of the config file. The first part sources config_common.sh, which contains hyperparameters and optimization flags used by all configs. Some cases override a flag from config_common.sh. Set the max steps, learning rate, gradient accumulation (MINIBS), and the aforementioned parallelization schema.

Next is a section to add system-specific optimizations and override the common flags if needed.

Next is a system-level setting to pass to SLURM.

The configs are being tuned for a particular system size, both in terms of optimization flags (impacting performance) as well as hyperparameters (impacting convergence). It is possible to modify a given config to run on a system of a different size, but that requires careful consideration and is not guaranteed to be as performant as the original config.

To start the actual training, you need to tell the script where the dataset/model is, where you want to have the logfile stored, which container you want to use, source the config files, and run the sbatch command:

Parse the logs

The logfile will contain a lot of output from the initialization and other info lines. The MLPerf-relevant lines start with the MLPerf logger prefix: :::MLLOG. There are a few interesting markers, as shown below.

Initialization starts:

Here, you can see that the Python script has started. Below, you can see the hyperparameters that you have selected using the config file, along the default (immutable) ones. 

After the initialization is completed, and the model is warmed up, the init_stop and run_start markers are printed:

The run_start line marks the start of the timing clock. The following lines show the progress of the training, including evaluation. You can see that the evaluation loss is decreasing, marked by the eval_accuracy marker.

When the evaluation accuracy (evaluation loss in reality) drops below the threshold of 0.925, the training stops and the run_stop marker is printed:

If the benchmark fails to converge, the run_stop status will display as ‘aborted’. The MLPerf score is a difference between the timestamp of run_stop and run_start. In this case:

Score [milliseconds] = (1745067009420 – 1745066917961) = 91459
Score [minutes] = 91459/60000 = 1.524

Keep in mind that because the convergence is nondeterministic, the final score has to be deduced from multiple runs, because the number of samples to converge may vary. Here, the benchmark converged at 3,072 samples, while on average, it should converge at around 3,100-3,200 samples.

Llama 3.1 405B

To run benchmarks for Llama 3.1 405B, follow the instructions in this section.

Build the container

Clone the mlcommons/training_results_v5.0 GitHub repo. cd NVIDIA/benchmarks/llama31_405b/implementations/tyche_ngpu512_ngc25.04_nemo. docker build -t mlperf-nvidia:large_language_model-pyt. If you have a registry you would like to push the image to, feel free to add the registry name to the image name.

Download the dataset and model

For instructions on how to download the dataset and the tokenizer, see the Llama 3.1 405B reference README.

Environment variable PREPROCESSED_PATH points to the preprocessed dataset. Downloaded files should end with .idx and .bin.

Environment variable TOKENIZER_PATH points to the tokenizer used in this benchmark. Downloaded files include:

You can clean up unnecessary files by running the cleanup script:

The final PREPROCESSED_PATH directory should contain:

Checkpoint

In the benchmarking region, resume training from the Meta official Hugging Face checkpoint. Refer to the instructions in the reference README to download the BF16 model checkpoint. Note that before you proceed, make sure that your current working directory is able to hold >1.5 TB of data.

Assuming that you are running the download command under a given directory, with its location stored under LOAD_CHECKPOINTS_PATH environment variable. After the checkpoint is downloaded, you should be able to find a 405B folder which holds a context and weights subfolder under the current directory:

Launch the benchmarking

NVIDIA uses SLURM to launch benchmarks on the compute nodes. To facilitate the job launch process, similarly to Llama 2 70B LORA, two files are used:

A configuration file (config_*.sh) that describes the hyperparameters of the model, including the number of nodes, walltime, and so on. Selecting a proper file enables you to easily configure the workload to run at desired scale with optimal hyperparameters. A fixed run.sub file that contains srun commands to launch the training, passing all the hyperparameters from the config to the Python script.

To take a look at a typical config file, in this case config_GB200_128x4x112xtp4pp8cp2_cg_dplast.sh, the name describes the size and the type of the system:

GB200: Designed to run on a GB200 machine 128×4: A system setting that can be decoded as NNODES x NGPUS NNODES: Number of GB200 nodes NGPUS: Number of GPUs per node x112xtp4pp8cp2 is a parallelization schema: x112 is GradientAccumulation, here equal to 112 TP4: 4-way TensorParallel PP8: 8-way PipelineParallel CP2: 2-way ContextParallel

This means that a 512-GPU configuration will be used running on eight GB200 NVL72 racks, with 64 GPUs being used from each rack. The benchmark will run GA112TP4PP8CP2: Global batch size (GBS) = 896.

To take a closer look at the content of the config file, the first part sources configs containing hyperparameters and optimization flags used by: all configs, configs using Blackwell GPUs, and configs using CUDA Graphs. In some cases, a flag from config_common.sh is overridden. Later, set the gradient accumulation (MINIBS), parallelization schema, micro batch size, the model size (frozen), and max steps.

Next, set performance optimization flags:

Next is a system-level settings to pass to SLURM:

To start the actual training, you need to tell the script where the dataset/checkpoint is, where you want to have the logfile stored, which container you want to use, source the config files, and run the sbatch command:

Parse the logs

The logfiles should largely resemble the Llama 2 70B LoRA logs. The target accuracy (evaluation loss) is 5.6. The training will stop once the target has been reached, and print the run_stop marker.