A few months ago I created/worked on a project that involved separating PNaCl (the portable native client) from the Chromium project, rewriting parts of it for use as a generic
JIT compiler, and building a small PoC distro around it where all of userspace were PNaCl binaries except for the loader/JIT (which was a regular old architecture-specific ELF that existed read-only in memory and loaded by a slightly-patched Linux kernel) Only PNaCl executables were allowed to run in userspace, which allowed for secure execution of user binaries thanks to the sandboxing features of the modified portable native client. However, another benefit of the PNaCl is that binaries are compiled to a performant architecture-agnostic IL (similar to LLVM's IL) and then quickly finalized by the PNaCl loader into a regular architecture-specific binary and loaded into secure memory when they are started.
This meant, in a nutshell, I could have a completely architecture-agnostic root filesystem, with the only architecture-specific binaries being the kernel and PNaCl runtime, which sat in the initial ramdisk.
Wow! I actually wasn't aware that TIMI existed until just now! All I've ever really known is ARM's work on the Jazelle hardware subsystem. I'll have to check this out more in the morning though!
If you're willing to accept some slightly more amount of native code for booting, you should be able to setup the loader to work entirely as a bimfmt_misc setup. This'd also let it be distributed to other distros easier.
Did you tackle dynamic linking? How did you abstract the syscall interface (i.e. how did you choose syscall numbers...take the x64 ones or the new "standard" ones shared by all modern ports?)
Based on interest here I'm planning on cleaning it up this weekend and shoving it on my GitHub (http://github.com/joshumax) by the end of this week or earlier...
Unfortunately getting dynamic linking working is a WIP and so far everything has to be statically linked (yeah it's terrible I need to get dynamic linking working; I have some ideas). As for abstracting the syscall interface between architectures...afaik the PNaCl handled that for me based on how process isolation works...
Imagine a hybrid x64/ARM system where you could run simpler, low-intensity tasks on your low-power ARM core (Twitter clients, Facebook, etc) but more intense tasks or entire applications (Photoshop, development IDEs, etc) on your x64-compatible chip.
Or a system where you take your ARM core + storage with you in your phone and dock it with an x64 core in your desktop.
It was also done the other way around with RISC workstations like Sun's running UNIX on SPARC coming with x86 cards to run their apps. I never checked it out to see what it was like but it was an advertised feature of SunBlade workstations. Additionally, IBM made PCI cards that could run their mainframe code on regular systems.
I'm not sure I understand. Does this mean that multiple linux kernels are split up, for example, on the graphics card cores such that each core on the GPU and CPU gets a "mini-kernel"? And then from that point all programs can run selectively on certain bits of hardware?
Popcorn Linux main idea started when trying to run Barrelfish OS (from ETHZ) we asked ourselfs "can I apply the same ideas to Linux"? Then we ended up in taking a somewhat different route in terms of system design for different reasons, some were technical. My personal perspective on the project was to try to understand if I can power up and down heterogeneous cores at any time, scaling indefinitely in the numbers of cores ... I changed that idea a little bit since I started the project (2012).
The current version of Popcorn Linux supports aarch64 + x86_64. Popcorn Linux runs only on OS-capable processors (no GPUs, no FPGAs). The kernel is not split up. You run different instances of the kernel on different ISA processors - one kernel per processor island (that is a set of processor cores with homogeneous characteristics). When a kernel boots up, if it is not the first one, it will join the community of Popcorn kernels. Operating system services are distributed, therefore things like memory are globally managed.
An applications to run among kernels must be compiled with the Popcorn Linux toolchain, and must be executed inside the Popcorn namespace (just an additional Linux namespace), in which the applications can see all the available memory and CPU cores. When your applications is in the namespace the only thing that it has to do is to call sched_setaffinity to migrate to another core - as easy as POSIX/SMP. You can migrate between different ISA at any point in the code (there is no limitation such as in offloading, that you have to go back as soon as the offloaded function is finished) - this is thanks to the stack rewriter (see below).
The current way Popcorn compiles an application to be run on aarch64 + x86_64 is to produce 2 binaries, one for aarch64 and one for x86_64. For now, the compiler toolchain is modified to align all symbols in each binary at the same address, and all data types have the same sizes; all alignments and paddings are forced the same way. For this reason we used musl, and yes, you need to recompile each library (although this is not strictly necessary in some cases). The executable is statically linked. The operating system cannot cross-ISA migrate your application at arbitrary points, the migration should be explicit in the code itself (but we added some OS-suggested migration, at automatically placed migration points). (If you want to cross-ISA migrate with OS control, you need emulation until a valid migration point, you can check the work from Venkat + Tullsen.) Finally, because the executable code of different ISAs sees the address space identically there is no need of state transformation during migration - this enables the developer to migrate a single thread, or a subset of threads of your application. The only thing that Popcorn transforms during cross-ISA migration is the stack, thanks to Rob that wrote the stack transformation routine. We decided to do not implement a single stack layout because we believed that the current stack ABI is optimized for performance, but as far as I remember we never tested the alternative option.
This meant, in a nutshell, I could have a completely architecture-agnostic root filesystem, with the only architecture-specific binaries being the kernel and PNaCl runtime, which sat in the initial ramdisk.