Cloud vs. on prem is going to come down basically to your duration of use and average/peak utilization ratio. For instance, if you want to run large parallel experiments 1 hour a day then 23 hours a day your compute is sitting idle then that will favor cloud more. Fundamentally on-prem you incur capital costs proportional to peak use whereas cloud opex is proportional to average use.
Once you have your intended compute lifecycle figured out you can compute the cloud cost and hardware cost and compare. Given you’re mentioning k8s I’m assuming this might be a continuous load in which case you’d amortize your hardware capital costs much faster.
Along with that, if you can find other users inside the company/research area/whatever that need compute power too, splitting that cost and getting higher usage numbers will also probably be favorable compared to cloud offerings then. It'd also let you split up the hardware and maintenance costs of keeping it on premises too.
Another really important factor here is workload size. If you NEED massive, parallel compute 1 hour a day, public cloud is probably your best option.
But if you can break that up and batch process it over 24h, and you don't have a need to scale up and down, a VPS from OVH is going to give you almost an order of magnitude lower cost than Azure, AWS, and GCP will for a box running 24h/day. What $5/mo gets you on OVH will run you close to $50/mo running 24/7 in a public cloud.
Once you have your intended compute lifecycle figured out you can compute the cloud cost and hardware cost and compare. Given you’re mentioning k8s I’m assuming this might be a continuous load in which case you’d amortize your hardware capital costs much faster.