If rendering UI to a canvas fits your needs then by all means do it.
Just be aware that when it comes time to localize to other languages it starts getting hard to support things like CJK where in order to properly work there's an Input Method Editor (http://i.imgur.com/JmmYwyi.gif) that needs to be able to read the text. There's also cut/copy/paste which users can't do from text on a canvas. There's also "define" which is built into OSX and iOS (others?) letting the user select a word and look up its definition in the dictionary. Finally I'm sure there are accessibility issues. And don't forget right to left like Arabic. (https://www.flickr.com/photos/u-e/2680759761/)
UI in a game (the most common place to render a UI yourself) usually doesn't care about these things. I'd hazard a guess though that most apps do. I know I get really annoyed when I can't copy small pieces of text from a native app. For example copying a link to a browser, copying a phone number to my phone, copying an address to maps, etc.
React Canvas author here. I outlined some of the reasons why it is not always the right answer: http://engineering.flipboard.com/2015/02/mobile-web/#Practic... On Flipboard we use DOM where appropriate for text input and areas of the application where performance is less critical.
You are correct that the DOM just works. But we need a better option on the web in order to achieve the kinds of experiences that people have come to expect from native applications. The hope is that by pushing the browser beyond its limits that we can make progress in this area.
Please don't take this wrong. I think what you're doing is great. I like perf too. I'm just kind of thinking outloud.
For example copy/paste and "define" require you to support selecting text on your canvas and then telling the OS about it when the user picks "copy" or "define". Otherwise you get the bad behavior that 99.99% of all native apps have which is that they don't support selecting anything except in editable textareas so if they display an address I can't copy it into maps.
You could argue the browser should add APIs so you can provide what the user has selected to the OS but, it means every developer has to be perfect. Developers who get it wrong end up making apps that don't follow any OS conventions, don't handle the edge cases, don't handle things when running in other languages, etc etc etc.
To take it to an extreme, maybe the browser should just be a rectangle of pixels and JavaScript. No DOM whatsoever. Imagine we had that world. There'd be no search engines and every page would have a totally unique UI. Some pages you'd use the keyboard to move a cursor to select something. Others would use the mouse. Still others might use a joystick. Some would require IJKL instead of the cursor keys. None of them would have native OS widgets. They'd all use a different font system. Most would probably only support ASCII and maybe ANSI at best.
Instead we have HTML and a DOM and most of these issues are solved and that uniformity has some arguable benefits.
I'm not saying stop your work. I'm just saying I'm there are trade offs. I think an argument can be made either way. Maybe rather than going back to a rect of pixels we should also be lobbying for ways to make the DOM better so we can keep the good parts (universality?) and still do better?
Understood. We just happened to have unconventional constraints that suggested an unconventional solution. There are many ways to lobby for improvements for things (especially Web standards) and perhaps with us showing the limitations of the DOM, we can influence change for the better.
For accessibility you can look into what pdf.js does. They put a transparent text on-top of the pdf text (canvas). So you can select it. Pretty clever :)
> But we need a better option on the web in order to achieve the kinds of experiences that people have come to expect from native applications.
I think this is a really cool project and a technical achievement, but I disagree with this one statement. We don't actually need to "achieve the kinds of experiences". This is my biggest problem with designers I've encountered, they do not see the benefits of the platform and try to design everything the same way. The web has distinct advantages that makes it special and I think way too much effort is put into making it behave like native applications (trying to catch up to its benefits rather than leaning on the web's).
You're entirely on point. This loses touch with the platform; one wouldn't use Material design in an iOS app and expect as much success as a more native experience, and vice versa; the same works for the web. It's a different platform with different goals and expectations.
Also, hey everyone, before you downvote the parent, please recall this article that you voted so highly just a few weeks ago, and consider its implications on building a web ui on top of canvas: https://news.ycombinator.com/item?id=8965048
Thats bad for the web ecosystem in general. We want the web to be the place people go for content instead of native walled gardens.
Great user experiences make that happen.
Rubbish user experiences that look and feel like a lame 2nd cousin to native apps actively harm the web ecosystem (look at mobile facebook; lots of people just use the app and dont even use the browser on their phone)
I know I usually prefer apps to the web not because of lack of fancy graphical effects at 60 FPS, but because web pages are so larded up w/ ads and other crap, it's hard to actually read the content. Or they try to do tricky things which somehow break on an iPhone's screen. Etc.
I absolutely disagree with you. Do you know what app was a huge hit in 1999? Yahoo Mail. I probably don't need to tell you what the "user experience" was like on Yahoo Mail in 1999 vs. Eudora and other popular desktop mail clients.
The difference was that then there was a meaningful functional difference between a cloud app and a local desktop app. Cloud apps were better, and people appreciated that.
I'm not convinced that web apps, which have worse UI, not access to native apis (camera, notifications, contacts, etc) offer a meaningful benefit over native apps.
You use mail.google.com on your phone? I don't.
How about Twitter? Facebook? Instagram? App or website?
It's because the native app is better.
The web is not special, it just happens to exist. If you keep building experiences for it that are meaningfully less functional than native apps, people will continue to flock to native apps.
sure... it's not all about hitting 60fps animations on a web app; but web apps that are closer to the look and feel of native apps is the way forward if you're building for the web.
apps are the dominant paradigm on phones and tablets because the web experience on touch only devices with tiny screens was terrible when phones and tablets took off. the web experience has improved but still lags behind because the investment in native apps far outstrips the investment in making webapps better on mobile
trying to emulate native apps in the browser is the same losing battle, but the inverse. good luck with canvas
I'm glad someone else said this, you've echoed a lot of my thoughts. I'm a huge react fan, and I think this is super neat - once again showing the flexibility and power of React's component and render model. It'll probably have a big impact on game UI for the web.
That said, doing this on a content-driven site is absolute madness. It breaks so many things, and throws away so many fundamental tenets of the web. by the time they've reimplemented everything they'll have probably lost all performance wins. The fancy animations are a miniscule gain for such a great loss of functionality
Excited to play with this though, since I'm learning canvas and graphics!
These are great points. It also breaks SEO and clipboard and ctrl-f.
With that said you could easily build a component library on top of this that renders to DOM or react-native or canvas or HTML (server-side) depending on what your use case is (just use dependency injection to get a reference to your underlying components). This way you can more easily manage these tradeoffs.
I think this project is so, so exciting and awesome.
The hardest part is actually layout managers, i.e. what happens if you want to support more than one pixel width. With all its clunkiness, CSS & DOM actually solve that.
edit: they actually have their own layout implementation(s). I wonder how those compare to the built in browser versions (performance, correctness, ...)?
It's css-layout: https://github.com/facebook/css-layout. The same engine used in React Native. It turns out that you only need a portion of CSS for apps. It's very performant.
Yes, but it's hard to do your own layout on the web because in times that want to width of an element that has text with `flex: 0`, you can't do that. It's mainly useful for things like this where you control everything.
Well said. We also experimented with non-DOM rendering, in particular with WebGL-sprites. We achieved very nice result running constantly at 60fps even on low-end mobiles but it's nothing worth if one doesn't have all the DOM amenities you mentioned or a full integration with the browser environment.
Each of these features can be implemented on top of a canvas renderer. Not saying it's easy, but can be done. The browser itself does it, everything is 2D textures in the end.
I'm building a complicated canvas app[1] for visualizing Bitcoin markets. I resorted to building my own framework on top of canvas, exactly for performance reasons. It's fast, but as a framework I was never too happy with it because it's a little half-baked.
More recently I found and fell in love with React for other frontend work, and use it exclusively now for any view-related work. I might have to go back and rewrite the ugliest parts of my code with this, it looks awesome. Kudos to Flipboard for open-sourcing it.
Very cool app. I'm going to pass it around to friends who are more into cryptocurrencies than me. :)
Sorry to hijack this thread, but I have a question: you seem to have been successful in stopping left/right swipe from navigating back and forward. The app I'm working on supports horizontal scrolling, but it can be a bit confusing if you mean to scroll and accidentally navigate.
I've tried a few things and looked around for answers, but so far the solutions I've found haven't worked out. Can I ask what your secret is? :)
1) Your site looks very similar to bitcoinwisdom ( which i love). Did that come about by coincidence or design? Are there any open source tools that you were able to build off of that bitcoinwisdom also uses?
2) Could you build out support for visualizing financial markets like stocks and the price of oil? The bitcoin ecosystem has a bunch of great UI, but it's difficult to find good web products for actual securities
Thanks. I definitely looked at Wisdom when I was starting the project, but I've gone in different directions than they have. For example, a strong emphasis of my service is the ability to integrate your accounts on various Bitcoin exchanges and visualize your orders and trades (http://i.imgur.com/KxxtGFA.gifv). I'm actually slowly turning it into a proper trading platform, with order execution. On the other hand, BitcoinWisdom are still happy just serving charts with ads.
I'm not sure what they use to build their app. No part of mine is open source except tiny bits of d3, and jQuery.
Wrt supporting other types of markets, I'd love to. The reason Bitcoin has had a wave of nice web-based interfaces is because the exchanges all offer free-to-use HTTP APIs. The "old-world" financial markets all charge money for data access, and it's generally not as easy to consume either. Bitcoin markets were born into the web era.
I did the same a while back, which paired Backbone with Easel / Create.js for both interface and game elements. It worked ~ok~, but was never quite what I wanted and a lot of cleanup was still required in order to achieve solid performance.
Looking at the author, not surprising. I was an intern in Yahoo! Mobile's web team in 2009, and sat beside Michael Johnston. He was incredibly productive and talented, and was the backbone of the whole mobile team. Everyone came to him for advice. I remember he wrote a sophisticated color picker in an afternoon, and the code was a work of art.
It's worth mentioning that much of the code that integrates with React internals is based on the work in react-art: https://github.com/reactjs/react-art
This is the same technique used by Netflix for their React-based television UI.
React is traditionally used for outputting DOM, but in this case components are overriding its hooks to perform another side effect—updating the canvas. This just shows the power of the component paradigm. Just like React Native does it with iOS views. Other view frameworks (Ember, Angular, Riot, whatever) don't even come close to being able to express this, because they're so tied to DOM itself.
You need to know Javascript's quirks. I generally recommend [1] for experienced developers. Everything on that page is important to know (edit: except the perf stuff, which changes as the js engines get better). I'll second the Eloquent Javascript recommendation in another comment if you're looking for a more involved book.
With the JS quirks out of the way, I would recommend building something in React and muddle through things with SO/google.
The hardest part of frontend dev for an experienced backend dev is the CSS. CSS is conceptually simple but there's a LOT of random domain knowledge you need to get layouts you want. You can bypass a lot of it by using a sass framework (e.g. foundation or bourbon) at the cost of adding a build step and another abstraction layer on top of CSS.
I have a similar path, and honestly found the FE dev tools being the slowest thing to learn about (too many, too frequently updated/breaking things).
I think you can build a lot on top of React with minimal knowledge of js (so, yes, start), but you definitely want to go deep and understand how it works inside (so, yes, also study js).
There's no need to go deep diving on learning JavaScript.
As you're using a high level framework, you're interacting with that more than the details of JavaScript. So in essence you're learning 'React' not 'JavaScript'.
So just start building in React and the JS will come along naturally.
That's not really true. You really need to understand JS to work with React, especially since you need to combine it with other libs since it's only the view layer.
Thanks for the heads up about RxJS, didn't know that existed.
Seeing as one of the benefits of React is in server-side rendering, and now with the new OSS nature of .NET, it's worth considering standard Rx as well ( https://msdn.microsoft.com/en-gb/data/gg577609.aspx ).
Rx is very new to me, so I may have this description wrong, but it essentially seems to be well-supported event-driven programming library. Worth a look if you're interested in FRP.
Rx isn't FRP, and doesn't really solve the same problem that react does (view management). Its grew for pushing discrete events around, but doesn't provide for any signals and continuous binding.
Rx programming is so great. It's one of those things that you seldom find a use case for but when you do it leads to 50x easier and more readable code. It truly is for events what Linq was for collections Mostly only useful with more advanced ui and network programming though
A stronger indictment of the DOM than finding it necessary to combine a CSS parser written in Javascript, a virtual DOM and a canvas based renderer because it is faster than updating the DOM would be difficult to come up with...
It looks like amazing work, but it's quite depressing that it's necessary.
Don't worry, there will be a new flag `document.useStricter` that will turn off all the cruft that makes the DOM slow so you don't need to patch around it with libraries. It'll become a Chrome experimental flag after the editor's draft is released in 2022.
Yes they can.
If you are interested here is how the rendering works.
The rendering happens in batches, the first render is always an expensive operation and includes both "paint" & "layout".
- The "layout"-operation recalculates the tree of visible elements.
- The "paint"-operation paints the element and is commonly done using a software rasterizer.
Every subsequent render happens if an DOM-element triggers a layout or a paint (or both). A re-paint occurs when changes are made to an elements skin that changes visibility, but do not affect its layout. Examples of this include outline, visibility, or background color.
When you use GPU-accelerated CSS properties such as opacity, translate, rotate & scale, both layout- and paint-operations are not executed. The GPU handles the changes for those properties. This is how famo.us wants to achieve its "60fps".
React by itself can cause expensive operations too if you trigger a CSS property that is not GPU-accelerated. In the process of rendering, a DOM-element can be destroyed and recreated when your state or model changes. So be vary of third-party React-components that promise "fluid" or "fast" speeds. IMO animation is still a research-topic for React. Their manual currently lists a method for basic CSS animations and transitions, but it's projects like React Canvas that make React a worthwhile contender for performant animations.
Mobile browsers are no different to Desktop browsers, almost every smartphone has a GPU of some sort. So you have 2 main-bottlenecks with every device. The first is the CPU (& RAM), the next one is the GPU (& the GPU-RAM).
Honest question, would studying immediate mode rendering systems improve my understanding of React's system of doing things? If so, do you have a resource I could use?
The first half of my career was in graphics and games. I think at a high-level, React is similar to a rendering engine and React + Flux is similar to a game engine.
It was behind its times with mxmlc. mxmlc made it painfully clear, even with incremental compile, that you were building a client-server application in a web age.
Once you wanted to build custom UIs, you lost the "WYSIWYG" GUI rapid development environment that was the whole selling point that Adobe came in with.
The data synchronization system worked but it was opaque and how (at the time) they made their money.
Yes. It is funny to observe how engineers implement technology A by means of technology B and then go backwards and implement B through A. The spiral continues.
The fun part is tweaking text layout algorithm and letting Hot Loader pick up changes when replacing components. Without browser refreshing, of course.
Last time I checked, measuring font sizes using the canvas was horribly slow. So, I wonder how they perform measurements necessary for automatic line-wrapping, et cetera. Are they doing these measurements fully in javascript?
> Custom fonts are not currently supported but will be added in a future version.
Ok, that is a pity. I wonder if it has something to do with the complexity of implementing measureText fully in javascript.
Custom fonts are supported on flipboard.com using the same rendering engine. It was just a matter of time to extract the Flipboard specific bits out. This is coming soon.
What are the accessibility implications of drawing all controls of a web app to the canvas using React? Are there ARIA annotations available for React already?
I don't know the specifics of how canvas accessibility does or doesn't work, but React definitely supports adding ARIA attributes to any tags it renders.
That's one of the problems with rendering UIs on canvas: it doesn't work with a11y. And since it's not rendering tags, there's nowhere to hang the ARIA attributes.
holy flipboard, did you see they re-implemented a subset of CSS into JS? https://github.com/facebook/css-layout I was literally thinking about doing something similar the other day for an experimental/learning project.
React is taking over the web, and I'm excited for it - coming from a person who is incredibly cynical about frameworks. React is proving itself as a powerful library, not an assumption driven framework.
The XML-like syntax is just less work, mentally and physically: no need for commas between props, the props object and children, no need to carefully manage all those commas when editing, adding, deleting, moving and removing stuff later on, plus the closing tag syntax makes nesting nicer by providing context via the tag name; all of which adds up to make it more pleasant to work with.
The second approach is what I was doing in various flavours for years prior to JSX. I ended up having to adopt comma-first to stay sane in larger/deeper templates, ( e.g. https://gist.github.com/insin/8e72bed793772d82ca8d ). The first thing I did when I saw JSX was start to write a React.DOM emitting version of my then-usual JSON-ML type stuff, but then I tried it and... never going back.
Thanks for sharing the examples -- really appreciated.
I'm not convinced that HTML is preferable to other representations of the DOM, particularly for representing UI's and applications (rather than documents). The clojure script wrappers for React I've seen seem to believe, for example, that lisp is better at representing the DOM than HTML and I tend to agree. Especially since any configured IDE should be pointing out missed commas and have rainbow parens.
One thing that is different between your examples and the DOM I've been creating with React is that I tend to make custom tags, and I don't end up with tons of repetitive looking stuff. Your last example is quite readable to my eyes, and would be even more so if perhaps decomposed into some helper functions and, of course, if loops were used and the DOM was populated from some data structure.
For me, I find it much easier to get an understanding of the structure of the components by looking at hierarchical markup vs. plain old JS. Everyone has a different idea of how they want to lay out their JS, but most people know/agree on what XML should look like.
Having a preference for reading HTML seems fair, though I prefer closing parens to closing tags for short bits, and the JS formatting problem is one you've already got in the rest of your code base. Also, you'll be formatting HTML + JS bits and not just static HTML like in the example. JSX seems like a good deal of machinery to support this preference.
60fps does matter. We've just all been taught that 60fps and sane use of CSS and the DOM (as currently standardized), so as to allow browser features to work properly, are mutually incompatible... which might be true. Using canvas certainly sucks for the reasons you describe; I wouldn't do it for more than a proof of concept. But long term, do we really need to hang the success the whole Web on the backs of some old and very ugly APIs? Can't someone come up with an alternative?
Exactly. This is a horrible misuse of the canvas element, alas, the moronic "war against native" destroyed all the good we had achieved ten years ago. I think then we had the golden age of web standards, progressive enhancement, graceful degradation, accessibility, and what not. Now we just have a race who can rape poor HTML/CSS/JS further.
This is a great project indeed. However, as many here have already pointed out, there are many limitations to this approach, SEO and lack of ctrl+f for example.
Which reminds me of flash. Most flash controls had similar drawbacks. I think web development shouldn't repeat that mistake (again).
It also reminds me that Google Docs took similar approach. As far as I remember they use a lot of canvas with a bit of DOM intertwined where necessary.
This is great. Can't wait for the WebGL version ;-)
Also, it would be really cool to run everything except rendering in a worker thread and send rendering commands to the UI thread, like react-native does.
I'm guessing http://flipboard.com/explore should be a good demo (in mobile safari, upper-right link off the home page), but it's just infinitely loading for me.
Posting b/c someone from Flipboard is probably monitoring this thread ;-)
Looks really good. My only real question would be about something drab but practical: SEO.
As the components are rendered to canvas, would the metadata that usually feeds SEO be harder to generate? Is that something you could fix through use of the parallel DOM (it's mentioned briefly on the page)?
> There is experimental support for using css-layout to style React Canvas components.
Just an experiment, but that could be very powerful. A lot of canvas-eligible projects are done in the dom because CSS makes it easier that messing with hardcoding all the style in js functions.
So one of my friends says that a potential issue is accessibility compliancy. And that due to lack of compliancy Flipboard may find themselves in a lawsuit. I know next to nothing about this type of thing. Anyone have thoughts on the issue?
Lawsuit seems unlikely, but it is certainly hostile to screen readers (though, it's quite possible to create a set of hidden DOM nodes that expose the structure of the page to screen readers. See google docs)
It's a shame that so many custom android roms either do not or have only very limited support for hardware acceleration in the browser. I do not know what percentage of users would fall into this category, but worth keeping in mind.
From their blogpost[1] "Head on over to flipboard.com on your phone to see what we’ve built, or if you don’t have a Flipboard account, check out a couple[2] of magazines[3] to get a taste of Flipboard on the web."
I was rather impressed with flipboard.com on my mobile. Although it seemed to hit a jank on a few scrolls and I wonder if there are UX implications for not being able to highlight and copy some of the text.
When I went to the example links given at the end of the post with my phone, I jut got a "spinner" for several minutes before I gave up waiting...
On https://flipboard.com/explore I get some exciting but very distracting rendering artefacts (blocks on the images changing intensity at a high speed until I start scrolling fast - if I scroll through slowly, the flickering continues)..
If I click through on one of the example pages, scrolling seemingly randomly stops working smetimes (after a few tries, it started snapping back to where it was, even when there clearly is more content below).
This is with the Android browser on 4.2.2, admittedly on a rather obscure phone (a Kingzone K1 Turbo) .
If you're interested in more details, my e-mail is on my profile.
Oh man, it's a small thing, I love how well the back button works. Feels totally like a native iOS app. (Though not so much if you use the swipe from left gesture, sadly.)
err, WebForms is the exact opposite. WebForms is about taking something stateless (HTML) and making is stateful. React takes something stateful (the DOM) and makes it stateless.
Just be aware that when it comes time to localize to other languages it starts getting hard to support things like CJK where in order to properly work there's an Input Method Editor (http://i.imgur.com/JmmYwyi.gif) that needs to be able to read the text. There's also cut/copy/paste which users can't do from text on a canvas. There's also "define" which is built into OSX and iOS (others?) letting the user select a word and look up its definition in the dictionary. Finally I'm sure there are accessibility issues. And don't forget right to left like Arabic. (https://www.flickr.com/photos/u-e/2680759761/)
UI in a game (the most common place to render a UI yourself) usually doesn't care about these things. I'd hazard a guess though that most apps do. I know I get really annoyed when I can't copy small pieces of text from a native app. For example copying a link to a browser, copying a phone number to my phone, copying an address to maps, etc.
In general DOM makes all of these just work.