For those who use R, I strongly recommend looking into R Notebooks (http://rmarkdown.rstudio.com/r_notebooks.html), as there is a lot more versatility involved, especially over the Jupyter/IRKernel approach. Although it's R only, unfortunately (you can run Python code in it but not the way you expect)
I'd like to see some things ported into Jupyter from R Notebooks, like JavaScript data tables and the separation of code and output, making it easy to version control only the code. (Atleast this 5.0 release makes tables nonugly)
It would be great to sit with the Rstudio-notebooks developers and user and discuss how we could get the two platform to converge and be more interoperable. I guess both team have a lot on their plate, and we would need more manpower, but understanding and collaborating with other project for the good of user is always something the Jupyter team is happy to do.
The jupyter team has been working on jupyter lab which is a more r studio style environment. It incorperates notebooks, terminals, file editing, traditional ipython console, plus some other cool stuff in a web ide. You should be able to use an R jupyter kernel with it.
It's in alpha right now but they're making crazy fast progress on it. I've been using it and it's awesome
Yes I know I'm one of the Jupyter dev :-) there is still a difference in notebook format and protocols. In that sens JupyterLab is not more compatible with Rstudio than the normal notebook.
In addition to what I've already said, R Notebooks have native support for widgets, autogenerated TOC, and themes (although all of those can be added to Jupyter I believe). There's also the meta-support from being embedded in an IDE. (package management/variable explorer/etc)
Thanks for the example! That definitely includes a bunch of things I think Jupyter should have, but "purists" might consider out of scope (theming, ToC, something as simple as being able to hide cells interactively).
Currently, Jupyter works well for development, but the result is often hard to read because code gets in the way. (unless you use nbconvert, but that sometimes defeats the purpose)
> That said, both Jupyter and R Markdown Notebooks are but a pale shadow of the support offered by Org-mode (seriously!).
Yes, but the large number of excellent notebooks available for Jupyter (and R as well) all over the web as well as support for CUDA and all kinds of extremely powerful libraries such as tensorflow) give those a serious edge over Org-mode, even though Org-mode is super powerful by itself.
If you setup a source block, you can use TRAMP to actually execute the command in a source block on a remote machine. So:
#+BEGIN_SRC sh :dir /user@remotemachine.com:~/remotedir
ls
#+END_SRC
The above will run ls on "remotemachine.com" and put the results in an output block below it.
Edit: Meant to add that you can do the same with docker. Just use "/docker:dockerId:" as the dir, and it will execute in a docker instance locally. Using multihop addresses, this can get extreme.
It would be great if Microsoft would open up a bit about what their plans are for this service, to the extent that it could be something I can rely on in a commercial context.
Hi, fair question. Lots of people are using the service in edu as well as commercial contexts. We haven't put any restrictions in that regard on it.
Other than that the plan is to keep this (or something similar) running as long as there is sufficient interest and so far there seems to be quite a bit!
Note that it does run on docker which means ultimately it's not fully "secure", but we hope to switch to hyperv-linux when it's available.
Just the other day I discovered that there exist a kernel (SOS kernel) that allows to use Python and R simultaneously in the same notebook (on a per-cell basis) and even has primitives for data exchange between the two.
Does it work in the same client-server fashion? (With Jupyter you can run the notebook on a server or the head machine of a cluster and then access the notebook remotely from your laptop or desktop.)
It's very easy to set up an RStudio server. I used to run one on Azure. Took minutes to set up and you get the whole RStudio suite, including R notebooks.
Myself and all the Data Scientists I know use Python, while the Data Analysts in my company use R. Does R have anything like the Natural Language Tool Kit that Python has? Or SpaCy for fast NLP? Also, between PYKE, ProbLOG, and python-constraint, a lot of my logic programming needs are satisfied in Python. Does R have anything like that for logic programming?
The only thing I want from Jupyter notebook is easy version control workflow - not sure what's the best option is right now but last time I checked it was 'delete all outputs before commit' which is great in many cases.
From the docs: "nbdime provides tools for diffing and merging Jupyter notebooks."
It includes both graphical diff and merge tools, command line diff and merge tools, VCS integration for git (so git uses the nbdime diff and merge for notebooks), etc.
Might be worth looking into if that is what you are looking for.
This is something I've been giving some thought to recently. I came to the conclusion that the best way to go currently is git-clean filters built using jq. It's still a bit hacky though. Gentle write-up here: http://timstaley.co.uk/posts/making-git-and-jupyter-notebook...
Use nbstripout[0] as a git filter. Then you have seamless git control of notebooks. There's talk of automatically saving code only separate versions in future releases.
This looks pretty good for handling diffing/merging of notebooks. I'm just getting started with jupyter, but it seems like somewhat of a pain to have to ssh into a server to manage versioning with git while I work on the code in the browser. I'd prefer if I could do all of my jupyter work directly in a notebook, and commit from the notebook webpage.
I'm not sure how others work with this stack, so there is likely tooling I don't know about. I'd love to hear suggestions.
This very much. I'd like for Jupyter to have an option to keep the cell code in individual python files in th backend, so you can just check in those to VCS. It would furthermore allow outside python code to import code from notebooks if jupyter would just provide a simple __init__.py file together with the cell files.
Jupyter will become like the Google of software development then ;)
To be fair, wasn't Jupyter designed to be an IDE - experiment with code, make tweaks and get the final version out in a text editor - rather than a repository of production-ready code?
Not really, you rarely have IDE designed to be around live-code. IDE are more meant to develop full-featured software. We are moving in this direction (as many users request it), but we still want to stay focus on Data Exploration.
Smalltalk people have thought otherwise since the 70s. The live code part is a big reason to use Jupyter. It makes me think the ST (and Lisp) people were on to something.
Did anybody succeed in deploying Jupyter into a multi-user environment? I know there is jupyterhub and some people deployed it for selected trustworthy users (e.g. in a university course context or a company's intranet), but AFAIK it hasn't been deployed in production to potential untrustworthy users. Current approaches seem to rely on the idea to sandbox you in a Docker environment, but mostly these are work-in-progress solutions and I am not sure how much more secure they are, beside the fact that also authentication and persistent storage issues have to be solved.
The obvious problem is that accessing Jupyter is technically similar to allowing full shell-access and you have to deal with local privilege escalation, but I wonder if there has been any progress. I evaluated to use it as a UI for domain-specific applications that give users some kind of graphical shell, but in the end I decided against it, because of security concerns.
https://cloud.sagemath.com deploys Jupyter into a multi-user environment. It's used by hundreds of courses for teaching across the world. Our Jupyter deployment also supports realtime synchronization (multiple people editing at once like Google docs) and recording of the complete history of the document. I've also spent the last few weeks on a complete rewrite of Jupyter from scratch using React to provide more robust realtime sync support, faster startup, and better integration with the rest of our platform; this rewrite is of course not live yet.
This is incredible. I have been wondering what a react port of jupyter would look like. Is this opensource ?
Any learning from multi-user deployment ? We are trying to do this internally inside our company and jupyter hub is a little hard to grok.
I know of a lot of people who would pay for a faster jupyter that can also be run as a standalone dashboard/script - without the heavy duty interactive kernels. Basically reduce the prototype-deploy loop.
Yes, that jupyter+react implementation will be open source. It's not clear though how the feature parity will be. At least the document is exactly the same.
I know sage math - in fact it was the first platform I found many years ago that provided an open alternative to Maple and Mathematica back in the days when Jupyter was still in its infancy.
It convinced me that there really exist viable open alternatives to the proprietary closed systems and I am extremely thankful for your efforts - I will definitely check out sagemathcloud.
http://mybinder.org/ does that, as well as https://notebooks.azure.com/ – well this one is authenticated, but it's easy to get a microsoft live account, and it does not even needs to be validated so that count (for me) as non trustworthy users. Technically giving access to Jupyter does not give full shell access. Only if the installed kernel give full shell access (which the default Python one does). The terminal feature can be deactivated as well.
You can use systemd or docker with JupyterHub, and make it as protected (or not) as you want. We could also write a spawner that spawns full fledged VMs, which would give you 'real' untrusted isolation - would that be something interesting to you?
You're right, it does use that. The security against untrusted users is whatever you get from docker - each user's code runs in an individual container. In the case of try.jupyter.org a breakout would not be catastrophic, though, because those servers shouldn't be holding any sensitive data.
Hands down the best and most used tool in my portfolio. Well done, guys!
Does anyone know if there's a plan to introduce multi-kernel support in single notebooks like what Zeppelin does? Not that I have a strong preference for it but it appears to hold a lot of appeal in Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark.
See project like metakernel [1], in the end a kernel can be "just" controlling process that manage multiple language. So why you won't get "multiple kernel" you can get "one kernel with multiple language". Its just asking for "multiple kernel" which not the best way to describe what you want to do. In the end it the same excpet you do the language dispatch in the backend instead of the frontend. It's also more efficient for data sharing.
[2] show you haw Hydrogen (based on Jupyter as well), does it.
And [3] (mine), show you how in the the same notebook to use Python, R, C, Rust, Fortran, Cython and Julia with data sharing and sending functions back and forth between languages. I was definitively lazy and did not include things like SQL, javascript and a few others. I haven't used spark in a while, and definitively never from scala directly, but I doubt it would be much harder to do as the C/Rust/Fortran/Cython took me an afternoon to write.
We are actually working on improving the polyglot support in Jupyter by porting/merging work from Beaker Notebook (http://beakernotebook.com) into Jupyter and especially into Jupyter Lab. This new effort is called BeakerX: https://github.com/twosigma/beakerx. We have autotranslation working in one direction now, should have multi-way in a couple of weeks.
Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark.
What exactly do you mean here? Are you referring to the parts of Spark which don't currently have a Python API? Because those are becoming smaller and smaller.
There is also Jupyter magics to let you change languages within a notebook. See %Rpush and %RPull from [1]. Not sure if there is a way to have a Scala kernel running and sharing the same Spark context though.
Yes, that's what I exactly meant. And you're right - that's a pretty small subset and that's what I tell others too.
But, the ease with which you can load interpreters on Zeppelin (apart from the pre-loaded ones) is impressive. I imagine it comes at a cost of some instability because it hangs more often than Jupyter.
Jupyter is super useful, together with anaconda it's a winning team. I love anaconda because it takes the sting out of all the dependencies and cruft that stops python packages from installing cleanly (besides the v3 / v2 mess and installing as a regular user rather than root).
I absolutely love Jupyter. Thank you guys and gals for your great efforts. I've had this idea about how it could become a very simple and powerful rapid application development tool based upon the widgets you already have: What if you could preview and bundle it in a mode with hidden code to standalone executable files on all platforms? There's already many tools in python to do this, all it would take is some export feature to tie it all together.
We're going to be exploring building dashboarding solutions more with JupyterLab. See https://github.com/jupyterlab/jupyterlab/issues/1640 for some of our current discussion. That said, the deployment issue you've talked about has been experimented with in the IBM dashboarding features (i.e., one-click to deploy an application-like interface).
Does anyone know how to setup a multi user notebook? Because of the nature of the notebook and how it maintains state ... that a single instance is usable by a single person.
We are building dashboards in Jupiter and really would love it to be multi user... Without getting into the hub and stuff (way too complex to set it up)
Sorry to hear you found the hub too complex. We're working on making easier-to-use hub setups that fit different use cases. Can you tell us a little more about what your use case was and (optionally) which parts of the hub setup you found too complex?
So it's a bunch of different technologies - nodejs,etc. I'm kinda wondering if it can be built in Python itself. Make it part of a normal jupyter install, so just a "jupyter hub start " will work ?
EDIT: adding to that, you have built a nodejs based http proxy - can you not build it within Python (for uniformity) or nginx (for performance as well as mind share) ? Do you even need to mandate a http proxy ?
Second question is that can it run in a multiprocess - I don't want to run it in interactive mode, but just straight top to bottom. Perhaps there's huge memory savings there.
I think yuvi wrote the CHP on nginx (https://github.com/yuvipanda/jupyterhub-nginx-chp) when we first wrote CHP, node was the only viable solution to have a dynamic websocket proxy. Nowdays Go, or Python 3 with AsyncIO may be potential contenders. It may be possible to rewrite in Python but time is limitted. I'm unsure about your second question.. run notebook top to bottom ? `nbconvert --to notebook --execute --inplace yournotebook.ipynb` ?
Well the second question was also related to multi-user deployment. From what I understand, jupyterhub will spawn multiple kernels every time someone logs in. But a lot of the time (most of the time?), you don't intend people logging into your jupyter notebook to be doing interactive stuff - maybe they just want to run the whole thing as a dashboard.
So it becomes a traditional webapp use case. Do you need all the proxy/websocket, stuff to do this ? Your nbconvert command still needs every user to spawn his own kernel right ?
About the first part - it would be great to have a simpler jupyterhub. One of the steps is to have everything in Python.
JupyterHub isn't really setup to do a 'dashboard' style web application - is purely intended for interactive use. The design choices made reflect this.
I just wanted to make sure you guys were aware that it is a large component of the use case. The very typical prototype in jupyter .. to ..Rewrite in production code is shortened significantly by doing this.
All the tools already exist in jupyter - except one: lightweight multiuser. I would argue that building this is going to be a fairly trivial thing for you guys (as compared to other features you build), but the end user benefit is immense.
Jupyter becomes much more than an interactive scratchpad - it becomes a full blown prototyping environment for data science and reporting. I would say, you would even go against tableau in a lot of use cases.
Please do think about it. My company will be happy to contribute to a gofundme on this.
Reply to this plus 2 comments up. Hub spawns _servers_, not kernels. There are a lot of indirection layers, and indeed, being able to _view_ a notebook without starting a kernel is on the todo list. The multi user collaboration is in progress, it's more complicated than it looks. One of the issue is that if this is a "solved" [with many quotes] problem for static documents, as soon as you have code execution it becomes really tricky. The kernel need to run as someone, but who ? The owner of the document ? What are the permission you give to who and how ? There are some case where there are possible answers, but which are really hard to tackle in a generic way across programming languages and various kind of deployments. Ian has an already well advance JupyterLab Prototype that you can connect to Google Drive for live editing. If your company is interested in funding something like that, feel free to write to any of us privately (git log, and grep to find emails), and we can likely setup a contract with numfocus (non profit that handle our funds), the advantage will be that it will be tax deductible for your company (unlike most of gofundme campaigns).
I wish we had the kind of money to fund the full development - I mentioned gofundme because a small startup in India will not be able to do that, even though we want to. But I have a feeling that a lot of us will want to us as well.
I'm talking specifically about the usecase of code-execution (especially dashboards).
Here's a small point from me - perhaps you are overcomplicating the usecase for 90% of us. Give a proper ssl/tls+bcrypted password setup and roles: Editor and User.
I dont think you should be worried here in the context of people wanting to run a full on sagemath cloud kind of a thing.
If you can give me a low resource way of letting 100 "Users" on a dashboard form and one "Editor" (who can actually edit the underlying notebook), I'm golden. And I'm willing to bet that so will 90% of your audience.
Maybe it's just me, but the former table style seems much more readable and, well, compact, even if you claim otherwise. I hope, this behavior is customizable?
I agree. It could be because our brains are used to the old style but there are some objective drawbacks.
The new style right aligns everything which is good for numbers but bad for text. Also, it might just be the screenshot, but the contrast is poorer and the column sizes aren't as fitted.
It's a pity, I would rather have a decent default than to customize every notebook.
I'd like to see some things ported into Jupyter from R Notebooks, like JavaScript data tables and the separation of code and output, making it easy to version control only the code. (Atleast this 5.0 release makes tables nonugly)