Hey! I think all of those points are 100% valid and I understand where you're coming from.
I'd like to preface that the product you see today isn't the end goal we're trying to reach, it's very much just the beginning of the vision of easy to use ML we want to accomplish, and we hope to meet the rest of the pains you've outlined in the future.
1. Data cleaning is the most important and painful thing we experience, and all the companies we talk to experience as well. "Data cleaning" is often a very broad term and can mean a lot of different things across different companies. It's something we absolutely want to tackle, but it's not on the immediate roadmap (it's still a bit vague of how to bring all the different problems companies face together).
2. This is something we hope to move into in the short term, we understand that production does not mean "ML runs on sever and returns requests that look right". We know that it means that the model scores well on offline validation data, online data, and is able to notify appropriately of shifting distributions of input data. We hope to have tools in place that can allow teams to understand the performance of the model both in an offline and continual online setting, and be alerted when the model is failing due to unseen feedback loops or drastic difference between training and inference distributions.
Where we think this stands today is that any engineer on any team can get started quicker with validating a ML PoC and get it deployed into production faster. While I understand there's a ton of nuance to "production", sometimes "good enough" is fine for the short term (and there's a lot of tools that can support them with the problems outlined above as well!), while we work out how to support customers in the long term even better.
As for accuracy, one of our core values is transparency, you can read about the technical workings here: https://medium.com/modeldepot/percept-whats-inside-the-ml-co.... This should arm a team with the appropriate terms to search for and understand if our product is the right fit for their use case. We hope to only expand the capabilities and options customers can tune as well if this exact solution won't work for them out of the box. Additionally, users are free to experiment around with how to understand the accuracy of the model (beyond a single accuracy metric), we even go a bit into this at the end of our guide here: https://medium.com/modeldepot/apples-oranges-a-machine-learn...
As for performance, since it's deployed on your own infrastructure, you can scale horizontally as much as you'd like without hitting arbitrary rate limits. In our own internal benchmarks we can get around ~6 sustained QPS on a c5.4xlarge and about ~12 sustained QPS on a c5.9xlarge. If you'd like to learn more, we can go more into details about p90s and whatnot across various concurrent connection settings.
I hope you have a chance to check out the product and leave us more feedback, we're striving to be different than other MLaaS offerings out there with more transparency in what we do, to empower engineers to leverage ML in a more meaningful way.
How do you enforce request limits when the code is not hosted on your server? And why should I pay tiered based on number of requests if I’m the one paying the infrastructure cost anyway?
(Sorry if I’m misunderstanding your pricing model, only had a quick glance)
The pricing model is to help support our continual development costs of the platform, it definitely takes a bit of time to develop the tech, and continually test it with our users and refine the experience so that it's easy to use out of the box for you. We have a pay-as-you-go structure to help make it easier for users that aren't sure about their usage yet to try it out at a very low cost, and decide if it'll work for them, we didn't want a steep initial pricing structure to be a barrier of entry for people exploring ML. The requests are loosely enforced through our central key server (though there is no request rate limit). While absolutely no training/inference data leaves the container, we do occasionally send back usage metrics to help keep track of usage. If you're interested in a totally isolated solution, we can talk about having on-prem deployed key servers so that your cluster can be completely isolated.
I hope that clarifies everything, let me know if you have any feedback or further questions :)
Usage based pricing seems very misplaced for a self hosted solution. Questions of enforceability aside (all I need to do is remove the telemetry code to get it free), the issue is that usage based pricing is meant to scale your revenue alongside costs. When your customer is hosting the infrastructure, you have the same costs regardless how many requests they make, so as a customer, it doesn’t feel right to pay you for each request that my own servers are solving.
A competitor can easily undercut you here. I would also be interested in hearing of any other company that charges per-request for self hosted software, because it’s certainly not a model I’ve heard. Typically the way to approach this is a licensing fee for running the software self-hosted.
I don't think usage based pricing is esoteric, if you look towards products in the enterprise space (Gitlab, Splunk, Mongo, etc.) They're all based off of a usage metric of some sort (in our case API requests). We're making on-prem ML accessible at SaaS prices (try negotiating a on-prem contract at other MLaaS). Our costs are continual as we continue to improve the product over time, and those improvements will be passed down to you as a user.
If you're interested in running the product on a licensing fee instead of pay-as-you-go, feel free to shoot us an email at hi@modeldepot.io :) We found that licensed pricing model to be more restrictive for new ML users and increased barriers to entry. If you're not interested in using our paid product, you can check out our primary pre-trained ML platform at https://modeldepot.io/browse and hopefully find a ML solution that works for you without paying a cent.
I'd like to preface that the product you see today isn't the end goal we're trying to reach, it's very much just the beginning of the vision of easy to use ML we want to accomplish, and we hope to meet the rest of the pains you've outlined in the future.
1. Data cleaning is the most important and painful thing we experience, and all the companies we talk to experience as well. "Data cleaning" is often a very broad term and can mean a lot of different things across different companies. It's something we absolutely want to tackle, but it's not on the immediate roadmap (it's still a bit vague of how to bring all the different problems companies face together).
2. This is something we hope to move into in the short term, we understand that production does not mean "ML runs on sever and returns requests that look right". We know that it means that the model scores well on offline validation data, online data, and is able to notify appropriately of shifting distributions of input data. We hope to have tools in place that can allow teams to understand the performance of the model both in an offline and continual online setting, and be alerted when the model is failing due to unseen feedback loops or drastic difference between training and inference distributions.
Where we think this stands today is that any engineer on any team can get started quicker with validating a ML PoC and get it deployed into production faster. While I understand there's a ton of nuance to "production", sometimes "good enough" is fine for the short term (and there's a lot of tools that can support them with the problems outlined above as well!), while we work out how to support customers in the long term even better.
As for accuracy, one of our core values is transparency, you can read about the technical workings here: https://medium.com/modeldepot/percept-whats-inside-the-ml-co.... This should arm a team with the appropriate terms to search for and understand if our product is the right fit for their use case. We hope to only expand the capabilities and options customers can tune as well if this exact solution won't work for them out of the box. Additionally, users are free to experiment around with how to understand the accuracy of the model (beyond a single accuracy metric), we even go a bit into this at the end of our guide here: https://medium.com/modeldepot/apples-oranges-a-machine-learn...
As for performance, since it's deployed on your own infrastructure, you can scale horizontally as much as you'd like without hitting arbitrary rate limits. In our own internal benchmarks we can get around ~6 sustained QPS on a c5.4xlarge and about ~12 sustained QPS on a c5.9xlarge. If you'd like to learn more, we can go more into details about p90s and whatnot across various concurrent connection settings.
I hope you have a chance to check out the product and leave us more feedback, we're striving to be different than other MLaaS offerings out there with more transparency in what we do, to empower engineers to leverage ML in a more meaningful way.