We've been running a stripped down version of numpy + scipy + matplotlib in lambda. We'd build the environment in a docker container with Amazon linux, manually remove unneeded shared objects and then rezip the local environment before uploading to s3.
A similar method is described here: https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/
Layers should make this entire process easier.