> We have the following major components in the PyPy repo:
>> An "emscripten" build platform definition, which teaches pypy's rpython toolchain how to compile things with emscripten: ./deps/pypy/rpython/translator/platform/emscripten_platform/.
>> An rpython JIT backend that emits asmjs at runtime: ./deps/pypy/rpython/jit/backend/asmjs/.
>> A "js" builtin module for the resulting interperter, to allow interaction with the host javascript environment: ./deps/pypy/pypy/module/js/.
>Along with these wrappers to help working with the resulting interpreter:
>> A wrapper to load up the compiled VM and expose it via a nice javascript API: ./lib/pypy.js.
>> A script for bundling python modules into an indexed format that can be easily loaded into the browser: ./tools/module_bundler.py.
And at 3.26 seconds to initialize from cache on a 4-year-old machine, 5MB gzipped (not too much worse than your average high-quality gif) and only 1.5x slower than CPython... this is actually approaching viable if you needed to run Python business logic in the browser. Which I never thought I would see.
Glorious indeed. At least three levels of abstraction, two JITs, and a host of supporting code. Can we add some more layers like an emulator or get recursive by getting something like Pyasm running inside it?
But... Hardware is cheap, right? Hardware gets faster all the time, right? Sure, the end user rarely sees any noticeable improvement because we just use the speed increase as an excuse to write ever more inefficient code, but at least you get to use your favorite language in the browser.
No, because it's "AOT" compiled when it's downloaded or executed (since you have one JIT, you can't AOT compile what it produces). So you can think about it as "just" one JIT, but a very inefficient one
So look. PyPy compiles bits and pieces of code generating assembler. This assembler then gets to be turned into Javascript (asm.js) which needs to be parsed and compiled. All of this happens at runtime and while you can technically call the second piece of it AOT, it's essentially a double JIT or at least you pay the cost of double jitting. The equivalent in C would be generating C at runtime and sending it through gcc/clang.
Since PyPy compiles RPython to C, couldn't one write a web app in RPython, compile it to C, then to JS (with asm.js) and enjoy a quick page load and smaller file size?
By the way, is RPython a nice language to program in?
Thanks but to be honest, I'm not proficient in Python, so it's hard to guess if those restrictions would hurt, although they seem fairly small. I was more asking for personnal opinions and projects using it. I wasn't able to quickly find any, it even seems pretty much nobody writes RPython code or maybe people call it Python code so googling is hard.
Looking through that list it doesn't look dreadful. There are a couple of things that jump out though - you can't use kwargs in function definitions which is probably used quite a lot in dynamic settings.
Eg in Python one might do (warning; crazy, contrived example):
You could avoid it, and generally in Python you're better off being explicit about everything. But it's definitely a feature that gets used a fair amount, especially in libraries where you don't explicitly know what the api call takes concerning user data.
I don't know of any real projects outside of interpreters that use it. It has downsides (long compile times come to mind) and I'm not sure how well you can utilize the strengths (like the JIT) for non-interpreters.
EDIT: Writing it is more restricted than normal Python, so you have to think about what you do (especially type hints, to make sure it gets what type your variables are) and the long feedback cycles due to compile time are annoying, but still very comfortable. And for building interpreters, the more or less "free" JIT is magic :)
A little frustrating that github's down, as I'd be interested to see their build process - with some trickery I was able to get empython [0] down to 2.5MB gzipped from 7MB, though I'm aware pypy as a binary is quite a bit larger anyway.
I also wish github was up so I could look at the benchmark code - startup looks much faster with empython, but I'd like to know how much slower interpreting would be.
I watched the presentation in Pycon 2015[1], and I really liked how honest/aware they are about the challenges and tradeoffs (so far, the performance it's pretty bad compared to javascript, and the download size is still an issue). At the end is answered why python can't be directly included in Firefox or other browsers(the presenter/main developer works for Mozilla)
For historical context, early IE browsers weren't hardcoded to use JavaScript. We know about VBScript, but really there was a generic framework called Active Scripting that would allow any other language to be plugged in.
Python was one of those languages. You could download an ActivePython distribution and start being able to run python in your web page with a <script language="PythonScript"> tag.
I'm not sure if this still works in current IE versions. It might.
"If we ship Python inside of Firefox, no one is going to use it because it's not available in Chrome, it's not available in Internet Explorer." Oh yeah? Try me.
Well, Google more or less tried the same with Dart and has since abandoned that plan in favor of targeting JS as Dart's primary runtime. Then again, the Dart VM never shipped in mainline Chrome, so who knows how a more committed attempt would have played out.
That's a bit of a fig leaf: dart never had the clout and popularity Python has, and it was clearly something no other browser vendor could get behind. Python has significant cross-vendor mindshare and wouldn't be perceived as fostering anybody's agenda.
It's true though that getting even two major browser vendors to agree on supporting another scripting language in addition to JS, is really really hard. I don't see it happening unless Nadella's "Really Good, Honest! (TM)" Microsoft makes some sort of grand gesture towards Google or (more likely) Mozilla.
Python might have clout, but in many way it's even harder to optimize than JavaScript, whereas Dart was designed by JavaScript VM experts to be easily optimizable and has a very fast, very small VM.
Python is also only just starting to explore optional typing which is obviously a prime feature of Dart and JS derivatives like TypeScript.
It's very clear that this is not a technical problem, but a political one. Dart could be the second coming of Knuth and still nobody but Google would implement it in a major browser.
Significant whitespace (in all languages, really, its a difference of degree rather than kind) constrains, but does not prevent, minification. (Idiomatic python code can be reduced by three chars per indentation level on each line with no impact on semantics, and of course other transformations that aren't whitespace elimination that are past of minification are unaffected.)
Heck, there are existing python minification tools.
Of course it could. But that would defeat the point, because support wouldn't be widespread enough for any real use (unless you're just using it for an intranet site or something).
As usual, everyone devalues the benefits of progressive enhancement. Add-on would be a great way to get the ball rolling, as long as your website works without it too.
But we're not talking about some optional feature you can just shim in 1kb of JavaScript, we're talking about the language used to write a web app. Sure, you could use PyPy.js as a fallback, but you'd essentially create a horrible loading experience for >99.99% of your users while you wait for this addon to catch on (which won't happen, because there's no reason for an end-user to install it). For a language like Python to be natively supported on the web, at least one major browser has to support it. And that's not likely to happen given the current state of affairs.
> python can't be directly included in Firefox or other browsers
So Python is a bad candidate, but there are better candidates out there. For example, jsofocaml (using OCaml in the browser) looks quite promising.
In the end, we need a lightweight yet properly defined and highlevel language, and the ML languages as well as the LISP languages are great candidates for that.
I believe the issue is politics. I.e. browser vendor A (any vendor) can't unilaterally introduce a programming language and expect others to follow. (MS tried with VBScript, Google gave up on doing so with Dart, for example)
Purescript is a Haskell-like language that is designed to output to javascript. It looks quite nice and has seen a lot of attention lately. It's even in GSoC.
The problem I have with "output to Javascript" languages is that debugging becomes hell. Line numbers no longer correspond to each other, for example.
A slightly less-hellish but still annoying problem is that I often poke at things directly from the console when bugs happen, and this helps me a lot in debugging. However poking at the console requires using JavaScript, which is fine, but it would drive me nuts to have to code in one language (e.g. CoffeeScript, TypeScript, Python) and debug in another (JavaScript) and switch my brain back and forth every few minutes. So I've ended up largely sticking with writing code directly in JavaScript.
Any good solutions to this? Are there replacements to the Chrome JavaScript console that supports those other languages?
Are you using source maps? If so, you can actually put breakpoints in your other source and Chrome will still break on it. I've even done this with Scalajs.
Purescript is interesting, I've been trying to build a project in it over the last few months. Since it's still pre 1.0, it has proven difficult but understandably so. It's mostly due to arcane/vague error messages (which they're in the process of dealing with), painfully long compile times, (imo) a few missing features, and incomplete young libraries.
On the flip side, with the state of libraries and all, it's provided me with a real incentive to get involved in open source, by having to contribute patches to said libraries and getting actively involved in general discussion.
Edit:
Just so I don't come off as mostly negative, I do enjoy having purescript as another statically typed option on the client side with close semantics to javascript. It's also been nice having a some kind of parsec like library on the client side (purescript-parsing), which was the main reason I chose purescript for this project in the first place.
plus its semantics lie pretty close to javascript. Most importantly because it's strict instead of lazy. Eventhough its strict it has a very haskelly feel. Both in syntax and type system. Also I think strict semantics is the way to go for efficient Functional Reactive Programming.
Furthermore, Purescript's FFI is really nice to use and it's really easy to bind to existing javascript libraries.
PureScript is community is nice and active. They are also discussing a lot with the `virtual-dom` guys to bring react/om/mercury-like functionality to purescript [0] and it's quite freakin awesome.
Seriously I'd take purescript for javascript any day. I think it's great that we FP folks can finally do frontend dev :-)
Out of curiosity, what would make Lua valuable to add in browsers by default? It is similarly prototype-based, has the same closure design, is dynamically typed, has an object system fairly similar to JS (with the type of arrays and hashtables both mixed), and its number system is also based on floating-point numbers…
I honestly don't see how we can justify the cost of adding Lua to browsers. I see the merit of Lua-to-JS compilers (existing codebases or sheer love of the syntax), but built-in?
Lua 5.1 (2006) has lexical scoping, tail calls, multiple return vales with assignment lists, semicolons are actually optional & most importantly: coroutines. ES6 adds some of these features.
Lua 5.3 has both integer and floating-point types.
I didn't downvote, but I don't think the problem has anything to do with specific language features. It's purely a political issue: you have to get at least two or three main vendors to adopt the new language at the same time, and none of them must perceive it as a threat to its own agenda. For that to happen, all of them must release compatible implementations that put everyone on an equal level at the exact same time, which is really hard to accomplish. It was very hard even to get javascript where it is now (remember the browser wars and all the incompatible extensions it spawned... ?), starting again from scratch with any new language would be extremely difficult, regardless of how "nice" the actual language is.
Besides the big download size, it does several things very "right": Full Python data model, access to the full javascript model (albeit slowly currently).
Also PyPy is the right choice as the basis for this (as opposed to CPython) because its ecosystem is generally more "pure" Python than CPython's. C Extensions would only get in the way.
it's systematically translating the PyPy interpreter to the Javascript, so it has to get semantics right. However I'm not sure if the drawbacks can be addressed in the current JS model, notably: very slow warmup, huge download size (that's primarily PyPy fault though), the lack of good JS access.
> However I'm not sure if the drawbacks can be addressed in the current JS model, notably: very slow warmup, huge download size
Only for the first time loading it.
It would be great if some big organization (looks at Google) hosts the "standard" version of the file, much like jquery, so that it would be cached and ready to be used on the local machine most of the time.
So does this mean that since I'm better at Python than Javascript, I can now use this to make my web frontends in Python? Because that would be pretty sweet.
That would be great as one of the things that puts me off of front end development is lack of language choice. I adore Python as a language and it would certainly make a great choice for a front end language if supported. Maybe if this project takes off, browser vendors would ship a Python interpreter alongside the JS one. I'd bet that Apple, Google, Mozilla, etc. could make Python fast if they threw as much money at it as they do their JS interpreters. And PyPy is already pretty fast -- it's certainly way faster than JS engines were before V8 and JavaScriptCore hit the market.
Being a Python guy myself, the problem was never the lack of Python in the browser, it's always the lack of a Python ecosystem in the browser. If I'm going to use Python and awkwardly interface with JS, I might as well just use JS.
It makes me want to gouge my eyes out, but it's not really that hard to get productive in. CoffeeScript is even better, if you aren't averse to adding another dependency on top of everything else.
Pyjamas/pyjs is actually two parts: the transpiler and accompanying widget set. Only the latter is related to GWT. The transpiler can be used independently.
So apparently introspection doesn't quite work with the wrapped js. Also, if you try: "[ i for i in js.globals]" (or equivalently iterate/loop over the globals-object) - the whole repl/tab hangs.
Not really an issue for running business logic in the browser, but if this was a full python repl-interface (with ipython support!) to the whole browser DOM -- that'd be a fantastic tool.
The introspection bit is because js.globals is implemented with a fancy __getattr__ but doesn't have a __dir__. In general, dir() is not reliable in the face of custom __getattr__ implementations unless the implementor goes out of the way to make it work.
I think there is no point in trying to emulate different language semantics on top of JavaScript.
asm.js is a weird hack that lives in a totally different world than actual browser APIs (e.g. the DOM), and you have to emulate your entire runtime and environment to get anything useful, which means your binary is going to be huge (as in this example, but similar things apply).
Emulating better semantics than JS based on JS usually leaves you in this uncanny valley where you either trade off performance and size for nicer semantics (e.g. killing the null/undefined dichotomy, or 64 bit integer math), or you end up with odd semantics that don't quite fit the language you're compiling from, or a mix of both, which is a usability disaster.
So the only real chance to improve on the state of JS is being a syntactical shim on top of JS and help users with better syntax, static analysis, and compile time tooling. That'd be TypeScript.
>>> from datetime import datetime
>>> datetime.now()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/pypyjs/lib_pypy/datetime.py", line 1548, in now
return cls.fromtimestamp(t, tz)
File "/lib/pypyjs/lib_pypy/datetime.py", line 1522, in fromtimestamp
result = cls(y, m, d, hh, mm, ss, us, tz)
File "/lib/pypyjs/lib_pypy/datetime.py", line 1459, in __new__
hour, minute, second, microsecond)
File "/lib/pypyjs/lib_pypy/datetime.py", line 312, in _check_time_fields
raise ValueError('microsecond must be in 0..999999', microsecond)
ValueError: ('microsecond must be in 0..999999', 1716000)
>>> from datetime import datetime
>>> datetime.now()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/pypyjs/lib_pypy/datetime.py", line 1548, in now
return cls.fromtimestamp(t, tz)
File "/lib/pypyjs/lib_pypy/datetime.py", line 1511, in fromtimestamp
us = _round(frac * 1e6)
File "/lib/pypyjs/lib_pypy/datetime.py", line 28, in _round
return int(_math.floor(x + 0.5) if x >= 0.0 else _math.ceil(x - 0.5))
ValueError: cannot convert float NaN to integer
I was more disappointed (but not surprised!) that "import pip" didn't work.
If it did (and it has big ramifications for how much of "python" (for some definition of "python" that includes most of the standard library) works. Eg sockets, interacting with some sort of "local" "file-system" etc) one could (this is handy if working on windows, and running python via cmd-r python.exe):
import pip
# pip.main expects a list of arguments, hence .split()
pip.main("install ipython".split())
# Only needed first
# time, as it installs ipython under site-packages.
# Only works if the user has write access, obviously
import IPython
IPython.start_ipython()
> That would be a total security nightmare and hopefully it's never going to happen.
Why? You're assuming a lot of things about a hypothetical implementation that is going to have heavy restrictions on in anyway since it's running in the javascript sandbox.
I can imagine a way to include language extensions in the browser and have them loaded up lazily in a sandbox. And a checkbox that says "Allow websites to run code" or something. Then you could develop an extension for your favourite language and either install them manually or have some popular ones bundled with the browser.
If we're going to run everything in the browser, why use javascript as the low-level language that stuff compiles to?
I know portability and existing standards are a limitation, but those can be influenced by the 3 (?) major browsers if needed.
I think asm.js is an awesome cute little thing, but the fact that it's taken seriously is worrisome to me.
"vendors are too scared (for whatever reason, legitimate or otherwise) to add extra languages to the mix, like python."
Tried python server-side, mod_python and got lots of issues with formatting. Not syntax or mixed tab/spaces but one-off spacing errors. I realised then, Python used this way has problems. Could be fixed though.
ages ago @zo1, I do see similar problems now if I shift from *nux/vim toolchain to win/idle which by default uses 'Python standard: 4 spaces!' but still throws random format error parsing on default code edits.
I don't like the fact python code stops working randomly if I write it in one environment then touch (edit) in another. Never have this problem with C or JS.
I have only ever had issues like that when I worked on an actual Python checker where I had to keep BAD code in my tests to ensure the checks worked.
I've never ( read 99.999% of the time I've performed the task ) had issues with Python formatting that weren't my own fault, everyone makes mistakes after all. I routinely edit the same Python files in Vim, Nano, Sublime Text, Atom, and PyCharm, and even jumping between all those editors, they have never "damaged" the formatting and created syntax errors in my code.
I can only surmise that some of the things you used or even just your vim config was setup poorly for Python work.
" I routinely edit the same Python files in Vim, Nano, Sublime Text, Atom, and PyCharm, and even jumping between all those editors, they have never "damaged" the formatting and created syntax errors in my code."
PyPy.js is not a transpiler, it is a JITted interpreter. This has correctness advantages since the major work is upfront, and potential throughput advantages, but is results in huge upfront download and warmup costs.
I'm curious -- since the custom emitter is tied to the RPython JIT and not to the Python interpreter, could one easily plug in the existing Ruby or PHP interpreters written in RPython and get similar performance?
Unfortunately, this is of little use to me. Why? Well, with my Swedish keyboard layout, some alt-combinations are crucial (mainly square brackets, []). And these keypresses do not seem to work.
I was talking with some contributors at PyCon last month and there was a consensus that Python 3 is actually preferable. There's issues to be resolved that are more important than switching the version to 3 so the work is focused on these for now.
It boils down to the argument that "if there is no existing Python code relying on Python 2 semantics, why even bother making a Python 2 version, why not skip straight to Python 3 and not introduce the switching problem later?" There's a small matter of the Python 3 version not being the prime target of PyPy but they are updating it and progressing it forward so it's not any kind of roadblock just a little less than perfect.
> We have the following major components in the PyPy repo:
>> An "emscripten" build platform definition, which teaches pypy's rpython toolchain how to compile things with emscripten: ./deps/pypy/rpython/translator/platform/emscripten_platform/.
>> An rpython JIT backend that emits asmjs at runtime: ./deps/pypy/rpython/jit/backend/asmjs/.
>> A "js" builtin module for the resulting interperter, to allow interaction with the host javascript environment: ./deps/pypy/pypy/module/js/.
>Along with these wrappers to help working with the resulting interpreter:
>> A wrapper to load up the compiled VM and expose it via a nice javascript API: ./lib/pypy.js.
>> A script for bundling python modules into an indexed format that can be easily loaded into the browser: ./tools/module_bundler.py.
In https://github.com/rfk/pypy/blob/master/rpython/jit/backend/... alone, there are 2887 lines of custom Python code to emit asm.js for function and block JITing.
And at 3.26 seconds to initialize from cache on a 4-year-old machine, 5MB gzipped (not too much worse than your average high-quality gif) and only 1.5x slower than CPython... this is actually approaching viable if you needed to run Python business logic in the browser. Which I never thought I would see.