Cortex: Deploy machine learning models in production

[ChefboyOG]

https://github.com/cortexlabs/cortex/tree/v0.9.0

[kuu]

How does this work under the hood? Is the model loaded every time it receives a request? Is it run in a docker or a lambda? How does it work after "uploading it" to amazon?

[ospillinger → kuu]

Each model is loaded into a Docker container, along with any Python packages and request handling code. The cluster runs on EKS on your AWS account. Cortex takes the declarative configuration from 'cortex.yaml' and creates it every time you run 'cortex deploy' so the containers don’t change unless you run 'cortex deploy' again with updated configuration. This post goes into more detail about some of our design decisions: https://towardsdatascience.com/inference-at-scale-49bc222b3a...

[kuu → ospillinger]

Thank you!

[oli5679]

If your model can be exported as PMML, this is really nice. Fast, minimalist, battle-tested and with very clean API.

When I've tested, it's up to 10x faster than Flask + serialised model object and uses far less CPU resources.

Plays nicely with lightgbm and Xgboost.

https://github.com/openscoring/openscoring

[isubasinghe]

This is basically my startup idea that I worked on for a while now (https://aiscalr.isub.dev)

Looks like I am going to have to scrap that entire project now, seems pointless to keep working on it given how similar this is.

[sixhobbits → isubasinghe]

Similarity should be taken as validation, not a negative thing at all.

[ovi256 → isubasinghe]

You should do customer development and find people willing to pay for your product.

If they're willing to pay, they'll even tell you why they can't use the open source tool.

[sails]

How does this compare to MLflow [0]?

Considering MLflow has a few components, I suppose you are building something closer to MLflow Models? How do they compare?

[0] https://mlflow.org/docs/latest/index.html

[ospillinger → sails]

From the MLflow Models docs: "An MLflow Model is a standard format for packaging machine learning models that can be used in a variety of downstream tools—for example, real-time serving through a REST API or batch inference on Apache Spark. The format defines a convention that lets you save a model in different “flavors” that can be understood by different downstream tools."

Cortex is what they are referring to as a downstream tool for real-time serving through a REST API. In other words, MLflow helps with model management and packaging, whereas Cortex is a platform for running real-time inference at scale. We are working on supporting more model packaging formats and I think it's a good idea to support the MLflow format as well.

[lettergram]

Was discussed three months ago: https://news.ycombinator.com/item?id=20579166

[ecnahc515]

Not to be confused with weaveworks/CNCF Cortex project for high scale Prometheus monitoring https://github.com/cortexproject/cortex.

[punnerud]

Seems to work with: Pytorch, TensorFlow, Keras, XGBoost, sklearn

[ospillinger → punnerud]

Yes, Cortex uses ONNX Runtime (https://github.com/microsoft/onnxruntime) under the hood so any model that can be exported to ONNX can be deployed.

[solidasparagus → ospillinger]

Is it only able to handle ONNX models? That's a pretty massive limitation compared to a hosted SageMaker endpoint.

[vishalbollu → solidasparagus]

Contributor here - Cortex supports Tensorflow saved models in addition to ONNX. PyTorch support is on the roadmap. Do you have specific frameworks in mind that you would like Cortex to support?

[solidasparagus → vishalbollu]

Perfect. Nothing in particular other than TF.

[kevinmershon]

Unfortunately, a somewhat popular Clojure library for machine learning on GitHub is also called Cortex, because this is going to make discussing machine learning APIs in the context of Clojure that much more confusing.

[waz0wski → kevinmershon]

there's also a prometheus storage backend called Cortex

https://github.com/cortexproject/cortex

[bermanoid → kevinmershon]

And I imagine many more machine learning tools will take the same name in the years to come, since it's about the most obvious one you could think of other than "brain".

Whatever is popular will survive...

[notus → kevinmershon]

Last commit was like 2 years ago...

[brennebeck → kevinmershon]

Couldn’t you just search ‘clojure cortex’? As this isn’t actually clojure?

[ChefboyOG → brennebeck]

It also looks like the last release for the Clojure library was in 2017

[TaupeRanger]

Stop naming things single word neuroscience terms. There are like 50 projects called "Cortex".

[mzanchi]

Calling it an alternative to SageMaker might be a bit misleading, as SageMaker is also a platform for training the models in automatically allocated EC2 resources, even on spot instances.

[dang → mzanchi]

We've changed the title from "Cortex: An open source alternative to SageMaker" to the page's own title, as the HN guidelines request.

https://news.ycombinator.com/newsguidelines.html

[deliahu → mzanchi]

Cortex contributor here - you're right, I would say we can be compared to SageMaker model deployment. We are currently working on supporting spot instances for serving, and training is on our roadmap.

[manojlds]

Sagemaker has notebooks, training and serving. This seems to be only about the serving.