{"id":606,"date":"2024-12-05T05:37:17","date_gmt":"2024-12-05T05:37:17","guid":{"rendered":"https:\/\/www.alanknipmeyer.phd\/?p=606"},"modified":"2024-12-05T05:37:58","modified_gmt":"2024-12-05T05:37:58","slug":"adding-a-2nd-p40","status":"publish","type":"post","link":"https:\/\/www.alanknipmeyer.phd\/index.php\/2024\/12\/05\/adding-a-2nd-p40\/","title":{"rendered":"Adding a 2nd P40"},"content":{"rendered":"\n

Having already added a 2nd P100 to the other GPU server, it was time to maximize the 2nd R720 empty PCI slot and add a 2nd P40. This GPU whilst old, still does pretty well, the main thing for me the amount of available NVRAM to work on larger models – now a combined 48G of NVRAM dedicated to the Tesla Generation GPU. I didn’t really have a script that could run a simple GPU\/Memory stress test so about this so I could incremently make sure the memory and dual GPU function worked correctly. This is on Ubuntu 24.04, but should equally work on all CUDA based Python 3 platforms.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Create venv, install necessary pip packages<\/p>\n\n\n\n

python3 -m venv work\nsource work\/bin\/activate\npip install torch transformers\n\n<\/code><\/pre>\n\n\n\n
Pytorch script to stress test available CPUs – adjust BATCH_SIZE to the amount of available memory<\/p>\n\n\n\n
import torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader, Dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Parameters for stress test\nMODEL_NAME = \"gpt2\"  # You can use \"bert-base-uncased\" or another model\nNUM_GPUS = torch.cuda.device_count()\nBATCH_SIZE = 16  # Adjust to push GPU memory limits\nSEQ_LEN = 128    # Sequence length for dummy data\nSTEPS = 200      # Number of steps for the stress test\n\n# Dummy dataset for training\nclass DummyDataset(Dataset):\n    def __init__(self, size, seq_len, tokenizer):\n        self.size = size\n        self.seq_len = seq_len\n        self.tokenizer = tokenizer\n\n    def __len__(self):\n        return self.size\n\n    def __getitem__(self, idx):\n        text = \"This is a dummy sentence for stress testing GPUs.\"\n        tokens = self.tokenizer(text, max_length=self.seq_len, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n        return tokens[\"input_ids\"].squeeze(0), tokens[\"attention_mask\"].squeeze(0)\n\n# Create the model and move it to GPUs\ndef create_model():\n    model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)\n    if NUM_GPUS > 1:\n        model = nn.DataParallel(model)  # For multi-GPU usage\n    model = model.to(\"cuda\")\n    return model\n\n# Training loop\ndef train(model, dataloader, optimizer, steps):\n    model.train()\n    criterion = nn.CrossEntropyLoss()\n    for step, (input_ids, attention_mask) in enumerate(dataloader):\n        if step >= steps:\n            break\n        input_ids, attention_mask = input_ids.to(\"cuda\"), attention_mask.to(\"cuda\")\n        \n        # Shift inputs for causal LM\n        labels = input_ids.clone()\n        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)\n        loss = outputs.loss\n        \n        optimizer.zero_grad()\n        loss = loss.mean()\n        loss.backward()\n        optimizer.step()\n\n        if step % 10 == 0:\n            print(f\"Step {step}\/{steps}, Loss: {loss.item()}\")\n\n# Main\ndef main():\n    model = create_model()\n    print(f\"Using {NUM_GPUS} GPUs for stress test.\")\n    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n    if tokenizer.pad_token is None:\n      tokenizer.add_special_tokens({'pad_token': '[PAD]'})\n      model.module.resize_token_embeddings(len(tokenizer))  # Resize model embeddings to account for new token\n    else:\n        model.resize_token_embeddings(len(tokenizer))\n\n    dataset = DummyDataset(size=1000, seq_len=SEQ_LEN, tokenizer=tokenizer)\n    dataloader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True)\n    \n    optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)\n    \n    train(model, dataloader, optimizer, steps=STEPS)\n\nif __name__ == \"__main__\":\n    main()<\/code><\/pre>\n\n\n\n
I then logged the results, adjusting the batch size until the GPUs ran out of memory, using nvtop to capture stats in real time.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
BATCH_SIZE<\/td>Memory Use max<\/td>Step Output<\/td><\/tr>
16<\/td>8G<\/td>Step 0\/200, Loss: 13.902944564819336
Step 10\/200, Loss: 10.364654541015625
Step 20\/200, Loss: 9.427839279174805
Step 30\/200, Loss: 8.033140182495117
Step 40\/200, Loss: 6.651190280914307
Step 50\/200, Loss: 5.285730361938477
Step 60\/200, Loss: 3.734560966491699<\/code><\/td><\/tr>
32<\/td>19G<\/td>Step 0\/200, Loss: 14.08126449584961
Step 10\/200, Loss: 10.370573043823242
Step 20\/200, Loss: 9.409626007080078
Step 30\/200, Loss: 8.021991729736328<\/code><\/td><\/tr>
64<\/td>21G<\/td>Step 0\/200, Loss: 14.005016326904297
Step 10\/200, Loss: 10.352890014648438<\/code><\/td><\/tr>
128<\/td>OOM<\/td>torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.07 GiB. <\/code>
GPU 0 has a total capacity of 22.38 GiB of which 1.89 GiB is free. <\/code>
Process 2856 has 250.00 MiB memory in use. Including non-PyTorch memory, this process has 20.24 GiB memory in use.<\/code><\/td><\/tr>
110<\/td>32G<\/td>Step 0\/200, Loss: 13.981659889221191<\/td><\/tr>
115<\/td>34G<\/td>Step 0\/200, Loss: 13.970483779907227<\/code><\/td><\/tr>
120<\/td>OOM<\/td>torch.OutOfMemoryError: CUDA out of memory. <\/code>
Tried to allocate 2.88 GiB. <\/code>
GPU 0 has a total capacity of 22.38 GiB of which 2.16 GiB is free.<\/code><\/td><\/tr>
119<\/td>OOM<\/td><\/td><\/tr>
118<\/td>OOM<\/td><\/td><\/tr>
116 <\/td>OOM<\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
<\/p>\n\n\n\n
Following the stress testing, I’m confident that these are both genuine NVIDIA GPU’s and will be able to handle the workloads I use them for ! I’m doing some more ‘lab tidying’ and will take thru the lab in upcoming post.<\/p>\n","protected":false},"excerpt":{"rendered":"
Having already added a 2nd P100 to the other GPU server, it was time to maximize the 2nd R720 empty PCI slot and add a 2nd P40. This GPU whilst old, still does pretty well, the main thing for me the amount of available NVRAM to work on larger models – now a combined 48G […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-606","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"_links":{"self":[{"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/posts\/606","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/comments?post=606"}],"version-history":[{"count":2,"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/posts\/606\/revisions"}],"predecessor-version":[{"id":609,"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/posts\/606\/revisions\/609"}],"wp:attachment":[{"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/media?parent=606"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/categories?post=606"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alanknipmeyer.phd\/index.php\/wp-json\/wp\/v2\/tags?post=606"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}