Livebook already has a SQL Smart cell (https://livebook.dev/integrations/sql). It doesn't integrate with Explorer yet, but it's already possible to reference a database connection, run a SQL query, and visualize the results in a table.
Can you make a pitch to a Python/R user to give this a try?
What you’ve built looks very nice and heard nothing but good things about elixir elsewhere, but would take a lot to leave those much more robust ecosystems. Do you hope to grow into that over time? Is there enough in terms of viz, statistical models, and ml to survive?
José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.
My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).
In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:
* Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box
* Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.
Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.
Jose's reply suggests the basics have Elixir equivalents. I can't really speak to that side but I can say the usability story is much much better.
The last time I gave Jupyter notebooks a go it was a full session of installing and updating various Python tools: pip, conda, jupyter then struggling with Python versions. You end up piecing together your own bespoke setup based on other people's outdated bespoke setups you find while searching for your error messages. Maybe that's better now, this was a few years ago. For Livebook it's "download the app and run it." Other options exist and are well documented and straight forward. I set up a livebook server on our k8s dev cluster with a pretty simple Deployment I wrote just from looking at the livebook README notes on docker. We've made livebooks that connect to the elixir app running in a different namespace on the cluster. Very cool.
Once you have Livebook going the `.livemd` file is both version control friendly AND very readable markdown file rather than the big json objects used in `.ipynb`.
For Livebook rebuilding cells is a lot more repeatable. It also does a good job of determining if a cell re-execution is necessary or not if a previous cell is modified which can save you a lot of time. Likewise the dependencies installed are captured at the top so I've never had a problem when sharing a livebook. The other person always gets the same results that I had. I don't remember how it worked for Jupyter but it's really cool to collaborate with someone by both going to the same notebook session. It's like working on the same Google Doc but you are writing and executing code.
Now with the Publish functionality I can see using a livebook to throw together some functionality and share it with non-technical users in your org, while having it backed up to git for posterity.
I avoided Smart Cells for a while because I didn't like the "magic-ness" of the UI hiding what the code was doing, but as Jose has shown in the launch videos this week you can easily see the code they are backed with and replace the cell with the code if you want to take full control. Maybe it was always like that but I didn't realize it at first. They really make setting up stuff very easy without limiting you later on.
I'm pitching it to the data science department of the company I work for (huge insurance company in my Country) next week.
They do a lot of prototyping from CSV/parquet sources in Python and R.
I've waited to show them Livebook because Elixir syntax is somewhat alien to many, but now that the Livebook team has integrated ML models and dataframes (Explorer through polar.rs) as smart cells, I think they have a killer feature in their hands, much like Liveview was for Phoenix framework.
Let's see how it goes, I'm fairly optimistic about it.
Dealing with sensitive data in code notebooks is a pain point [0]. That's why we've been improving secret management in Livebook for a while. And now we have more news we're excited to share.
In this new release, we added notebook stamping. Together with the other Livebook features, this helps to make notebooks readable, open, and safe to share.
Let me know if you have any questions about those new features or anything related to Livebook's launch week. :)
Super excited to share our first-ever Livebook Launch Week with you all! Each day this week, we'll announce new features and capabilities in Livebook 0.9.
Kicking off Day 1, we're introducing the ability to deploy Livebook notebooks as apps!
Now, you can transform your interactive notebooks into user-friendly web apps, making them accessible to a broader audience.
Livebook is an open-source code notebook built with Elixir and Phoenix LiveView. This release has cool new features, like visualizing message-passing between Elixir processes and an interactive UI for Elixir pipelines.
