When I was completing my masters thesis, I used EC2 Machine p3 instances along side my ‘on prem’ devices. As I was moving various code\/models all the time these instances where on, the bills where racking up – sure it was great to have such powerful GPU’s on demand and I could easily setup EKS (Kubernetes) with GPU nodes via IaC and CI\/CD Tooling, but when the bill arrived it brought home how important it is to have good MLOps methodologies in having efficient use of delivering models for production usage.<\/p>\n\n\n\n
Not wanting to incur such regular costs and knowing i would need a server for at least the next 2 to 3 years, I explored how I could improve upon my ‘on prem’ setup, which consisted of a ASUS Tuf gaming laptop with a RTX-3050 and 4G of NVRAM and a Server Tower with a RTX-2080 with 12G of RAM.<\/p>\n\n\n\n
What I needed was a GPU(s) that was good at inference and a training, with enough capacity to hold the models I am developing and some of the widely used LLM’s available. Doing some research it became apparent that the Dell R720 was a server built for hosting graphics cards and the NVIDIA P100 & P40 where good matches for my research requirements.<\/p>\n\n\n\n
Alot of help came from the ‘the data daddi’ you tube channel, his videos are pretty well put together and easy to follow.<\/p>\n\n\n\n