What speed should we expect from the model on consumer hardware? I tried a 8 bit...

veselin on May 16, 2023 | parent | context | favorite | on: StarCoder and StarCoderBase: 15.5B parameter model...

What speed should we expect from the model on consumer hardware? I tried a 8 bit quantized version on 4090 and got it to generate 100 tokens for 13 second, which seems a bit slow to me.