That's because all the text is inside <div> elements, so there is no hierarchy for a screenreader to latch on to: https://imgur.com/a/D78Tbgk
...And the three nav links are made via empty, transparent <a> containers absolutely positioned over <div> text. Focusing, therefore, has nothing to read.
N.b both left- and right-clicking activates these anchors, because navigation is implemented as a delegated `mousedown` event on the document.
Sorry, maybe I should have phrased this differently: Of course that should be fixed! (I have spent the last half year or so fixing a11y issues on a highly frequented website, mainly thanks to the EU Accessibility Act, so I'm starting to get a feeling for how awful non-accessible websites must be.)
What I meant to say was: The lack of a11y doesn't seem to be a deficiency of the library itself but rather of a hastily written demo application/renderer.
My surrounding description serves as alt text! More directly: that is a depiction of the flat hierarchy of the page, which also shows empty text anchors.
It doesn't render anything for me in Firefox, just a blank white page. After a moment, a message shows up: "This page is slowing down Firefox. To speed up your browser, stop this page."
yeah, i have the same problem. it's been hanging for almost an hour for me now. probably an infinite-loop bug rather than a working but slow algorithm?
Lots of other issues also such as page up/page down on keyboard not working and mouse scroll wheel is very slow(it behaves as if acceleration is turned off).
Yeah the scrolling is handled by Clay rather than being native (everything has position: absolute and there's almost no nesting). It's very cool but I don't know if I'd want to ever use it for a website. Native apps though, might be worth trying out.
Speaking of games, one of the most accessible I've ever run across was Gears 5.
It allows you to add ping to various obstacles, has both TTS and STT, and so on and... I could actually play it, whilst blind [1]. I could play a shooter, and not suck, without my sight.
A lot of what they did with the engine was really simple, but I so wish that there was a wider adoption of those kinds of techniques.
[1] My blindness comes and goes. Some days I have 0% vision, other days 75%.
A big improvement over CSS, but still seems pretty manual and finnicky, I wonder if a constraint solver based system and syntax would be ideal for laying out UI.
For instance, being able to set the constraint "element.[x,y] = other.[x,y]+other.[width,height]/2;", instead of working with "attachment" objects.
Apple adopted such a constraint-based system about 10 years ago in iOS and macOS. It's called Auto Layout.
It's powerful but not trivial to adopt. In particular, the design experience in Interface Builder has gone backwards in usability. The old system of resizing rules visualized as "springs and struts" was easier to understand in a visual design tool.
One might argue that's the cost of progress, and that designers using Interface Builder become better UI engineers when they have to figure out how to express themselves in constraints. But it seems to me that the reality is that a lot of people just stopped using IB.
IB used to be a crown jewel of NeXT's development suite in the 1990s. It was simple and focused, and allowed you to build surprisingly powerful UIs that connected to high-performance native code (unlike its mainstream competitor Visual Basic).
I don't think a lot of people have such fond feelings about Apple's current IB. Something was lost along the way.
This seems to have originated in something called Visix Galaxy, and supposedly done better there than in Interface Builder. See here: https://wiki.c2.com/?SpringsAndStruts
I tried finding any documentation on this tool/SDK, but no luck. Any one else has any more information on what this Galaxy looked like?
i don't know when visix galaxy existed, but springs and struts are how tex did text layout in 01978 and how tex and latex still do it today. 'visix' sounds like a name from after 01978, though i could be wrong
i read it when i was 18 and taking motel reservations in a call center. it has 27 chapters totaling 304 pages, thus averaging 11.3 pages per chapter. you can read a chapter each night before bed in 15 minutes and you'll be done by october. doing the exercises will take you longer, of course, and probably require you to sudo apt install texlive (i didn't have a computer to do that on)
I sense a parallel between that and the dropoff of SQL over lossy and impedance mismatched but "easy" ORMs and document stores. People don't realize they're trying to have their cake and eat it too maybe?
I wonder how we can stimulate "expert" tools and systems for those who don't want to take the greedy path of least resistance, and are tired of painting themselves into corners that way.
(The dark ages of data processing for personal use)
- Use a text file: Fine, you have to write your own read & write logic, but for small amounts of data this works.
- Use a CSV file: less custom logic than plaintext
- Use a JSON file: really nice to have structured data!
- Use a Python pickle file: the idea is you can “pick up where you left off”, but it’s slow, clunky, and inflexible
(Finally learning to use a database)
- Use Google Sheets: oh, it’s nice to be able to index things without needing to read/write the entire dataset! You can also do searches and stuff, it’s great.
- Django ORM + MongoDB: Oh my god, so horrible. MongoDB was supposed to be simple. This set up was slow and complicated. Migrations were a constant pain. And we didn’t even have any users.
- Postgres: It all makes sense. SQL is great. You can think about and query your data in reasonable ways. And it’s fast.
- DynamoDB: Yeah whatever, as long as you do validation on every read/write you’ll probably be fine.
Stimulate, indeed. How do we convince others to keep prototyping languages/technologies for prototypes? And to move from prototypes to actual products using languages that focus more on correctness and static guarantees, rather than endless runtime flexibility and dynamism?
This is a topic about which I have long simmering opinions. I suspect that several things happened.
First, expectations and requirements went up. Laying stuff out with a mouse and snap-to alignment guides is ok for simple UI, but the more complex the design, the more I would end up fighting the vector-art style design interface. I remember staying up late fixing pixel alignment errors in nib files. Often you would need to move objects around to inspect another underneath and then play undo games to get things back exactly where they were.
The interesting parallel here is that the designers were doing all the designs in vector art apps, and were also frequently missing dynamic aspects of the design requirements.
It was one thing to design complex dialog boxes for desktop; think of a photoshop filter control pane. You target a minimum size screen, and then work with a fixed pane and lay things out. When the iPhone came out, IB worked ok for early versions and stock components. Screens were crowded but fixed width.
Once bigger screens came out, more designs started needing variable width layout computation. The APIs for dynamic layout were (as I recall now) more subtle than they sounded in the docs, and I recall joining several teams that misused them. “sizeToFit” and “sizeThatFits” were two culprits. Perhaps it made more sense in the simpler NeXT days, and perhaps the docs degraded. If you didn’t read Apple’s “Programming Guide” docs, it was hard to know how these things were supposed to work together, and those were like mini books. The guides got increasingly ignored as the iPhone’s UIKit took center stage. I was always a fan of NSAutoresizingMaskOptions but rarely saw others use them.
Second, modern version control made the binary nib files unmanageable, and merging the xml xib files was also awful.
Third, and to the parent comment’s point, there were quality problems with Interface Builder. New stuff got heaped on every year. The data model was proprietary but looking at the xml clearly just got more complicated and version-encumbered over time.
In summary, it is tempting to glorify the original NeXT tools, but I never used them. I started toying with IB in 2004. It never felt like a brilliant system because I couldn’t express the underlying logic of designs in the visual box dragging paradigm. Ironically auto layout pushed me further towards UI in code. But to each their own.
Did you make any progress on this for macOS or iOS/iPadOS?
Out of total hatred of Xcode and Interface Builder I started experimenting with writing Apple UI stuff in C (calling Cocoa methods through libobjc), but there's precious few resources on doing so-called "Nibless/Xibless" development beyond the basics.
I'd love to find a decent-sized open-source macOS app written in [Objective-]C/C++ but with a good assortment of common UI paradigms, all done in code. I shudder whenever I see that dreaded .xcodeproj directory...
There's a fair deal of resources for writing code-only UIs in iOS apps with UIKit, but not nearly as much for Mac AppKit apps.
I suspect that this is because the average Mac window/view nib file is a great deal less complex than its counterpart view nib on iOS and still manageable to edit in Interface Builder, and so a lot of Mac devs still use nibs.
I've toyed with code-only Mac AppKit stuff but generally have found that it's not quite as clean as code only iOS UIKit. For example if I recall correctly, there's no initializer for NSWindow that sets all of the flags that make it behave like a normal window because it's assumed that you'll be using nibs. It's not difficult to write an extension to NSWindow to fix this, but it has to be done for reasonable productivity, and these papercuts are strewn throughout AppKit. In contrast, most UIKit controls can be initialized with few or no arguments and still behave as expected.
AbiWord appears to use IB (there are *.nib directories and plenty of `IBAction` and `IBOutlet` annotations), but it's kinda woven into AbiSource's own cross-platform application framework, and that actually makes it more interesting/useful as a learning resource.
Thanks for the suggestion! Any others you can think of?
I always thought that Apple should have really pushed the whole "Applescript Studio" thing so as to create something even better and more approachable than Visual Basic --- it could have been a true HyperCard replacement.
Auto Layout was just such a terrible system - as evidenced by it not being present in SwiftUI. It was slow, unintuitive, and not easy to handle UI updates
Maintainability. Think about having a list. You add constraints between all the rows etc. then remove a row in the middle. Now you have to update all the constraints
i think that in cassowary that involves removing roughly one constraint for each thing on the row, then adding a new constraint for the newly adjacent rows? then because it uses an incremental solver it can reuse the part of the previous solution for the rows above the deleted row, just recomputing the positions of the rows below
but that's a question of performance. 'maintainability' would be removing the code that adds the row to the list, which is trivial with cassowary
I'd been working on a similar syntax for a while, mostly was inspired by some work I did in iOS back in 2015/2016. They used a constraint system and it's honestly pretty great, though some of the more complex features can be daunting.
I think CSS's own internal logic prevents this sort of syntax from working, since you could be talking about any element matching `red-box`. There's no good way to refer to a specific match of a selector, unless you enforce that the selector only selects 1 thing, even then, that would be conceptually hard to deal with.
I think flexbox and grid handle a good chunk of what you'd want here, but having to handle constraints at the "group" level. Either flexbox, so the browser finds the best way to place elements into a line based on your rules, or grid for 2 dimensions:
boxes {
display: grid;
/* there are only 2 columns, 100px each */
grid-template-columns: 100px 100px;
/* all rows are 100px tall */
grid-auto-rows: 100px;
> * {
/* All children take up 1 col, and 1 row */
grid-column-end: span 1;
grid-row-end: span 1;
}
}
blue-box {
/* blue-box must start at col #2 and row #2 */
grid-column-start: 2;
grid-row-start: 2;
}
The problem with this specific API is that it depends on source order. To have element-b anchored to element-a, element-a must come first in source. I'm not really sure if it was designed that way or just implemented that way in Chrome, but it's the behavior I experienced when I played around with it.
Tailwind’s group selectors may be able to help some here, not sure and I’m on mobile so can’t play with it. I am working on a as-close-to-pure CSS solution for a list with content to one side. When you scale to mobile, the list is the only displayed element. If you click a list item, the details become the full view with a button to go back to the list.
Currently we are detecting mobile based on a JS library and branching the template based on that, which I abhor. Using Tailwind’s media prefixes and some data- group selectors, I’ve gotten a rough version working with just two small JS listeners to toggle the open state.
I think you could apply group rules to target a specific child or sibling in such a way that you can apply specific rulesets, and maybe CSS variables to dynamically base those on sibling values.
Yeah it would be nice if it felt like saying "this goes here, that goes there." I wanted it to mirror how someone might verbally speak about layout. It gets harder when you consider multi-dimensional layouts like grids, but for 1D things like headers or lists, it would significantly reduce complexity.
qt lost it's golden momentum taking way too long to relax licensing in the 90s. had they done it before the gtk shift the world would be very different.
btw, gtk is based on tcl/tk which i believe is the, or one of the, original auto layout engines
I literally thought of building this kind of thing recently, although I have some differing ideas on how to implement it. But the general idea of a single header C-like file that compiles to wasm and outputs primitive drawing commands is exactly what I was thinking.
What I really want is something like the old Flash/ActionScript display list. Just a 2d scene graph with the option to output draw commands, text or sprites. Things like containers (with things like border/backgrounds/etc.) and layouts can be built on top of that, so you could have two separate header files, one for the display list and another for a layout library.
Has anybody made an attempt at making a library like this, but with cross platform user input, and support for accessibility? From personal experience, if you can output triangles and text, writing a UI library like this is maybe 2-3 days of work. The fun starts when you consider that younger people are touch-first.
The accessibility is the hardest part. My custom library (in C# atop a weird rendering stack) has partial narration support and full touch/gamepad/mouse/keyboard navigation, but getting all the way to integrating with native screen readers is basically impossible at this point - from investigating it, it'd probably take me at least 3 months to get it working at all, and it wouldn't be portable.
One thing you have to do for reasonable accessibility is maintain a retained model behind the scenes even if you have an immediate mode API, so that's what I did. The immediate mode API does a bunch of caching in order to construct and maintain an appropriate retained mode tree across frames, which makes it possible to cleanly handle things like focus, selection and narration for invisible controls, etc. You also have to bake accessibility into the API from the start, for example by making certain every single widget has a description or a reasonable approximation of one, and by making sure there is an approximation of roles for every widget too.
A simple 'read a text description of the focused/clicked control' also doesn't get you far enough for narration - for example if there's a slider or textarea, you don't want to read the description and then the new value every time it changes, your narration has to be 'smart' and know to only read the description initially.
Have you seen libAgar (https://libagar.org)? Cross platform support is certainly there, covering everything from windows XP (and earlier) to *bsd and SGI IRIX. I'm not sure what all having support for accessibility requires as I've never had to worry about it, but am curious if 1, agar has what's needed, and 2, what exactly is required of a GUI library for accessibility. Screen reader support? (Are there SR standards for desktop applications)? Dynamic scaling? High contrast?
(For embedded and/or touch first UI, LVGL is pretty nice, but probably lacking any semblance of accessibility features apart from keyboard navigation, but you could hook that yourself).
I wonder if you can get better performance than the built-in browser engine for certain complex layouts by first calculating the layout using Clay and then absolutely positioning the elements with HTML/CSS.
There was some news feed web app that used <canvas /> for better scrolling performance.
if you can guarantee that layouts don't change during interactions, i guess it _might_ save some time for the browser (and thus battery, for low power devices).
If layouts change during interaction (e.g., orientation swap), then you will have a roundtrip to the server to recalculate. I assume this would cost more time than letting the browser css engine do their thing.
I wonder if this would be more efficient than the browser's impl. But i guess if clay does less complex layouts (but which is still sufficient for applications) than css, it might be faster than the browser's own.
> Fast enough to recompute your entire UI every
frame
Yet, when I scroll the front page, made with Clay, it stutters and feels like it can barely handle smooth scrolling, even on a modern Apple Silicon laptop.
Author here. I'm sorry that it performs poorly on your machine - if it makes any difference, it's the rendering that is slow, not the layout.
The HTML examples are more meant as a demo than anything else, as the library actually doesn't do any rendering itself at all, it's exclusively a layout tool.
I'm honestly not sure why the performance differs significantly between machines - I'm on an M1 mbp / firefox and it scrolls at 120 fps for me.
Hi, thanks for the response. It’s not actually rendering using native browser features, is it? I think that’s the reason. For example, on mobile, in which it also has problems for me in addition to the laptop, if I do a pinch zoom on the webpage, it all messes up and is not usable. I frequently do this on mobile to zoom in on text that I have trouble reading or images, etc.
On mobile, iPhone 15 PM, feels janky as well. It’s subtle but it’s there. Thought it was just me, but checked the comments to find similar experiences.
i'm guessing that this is because safari doesn't allow the layout code (or anything else) to run while it's scrolling, which with normal websites (which don't use fixed positioning for everything) results in less janky experiences, and was famously critically important to get reasonable scrolling on the original iphone. it doesn't matter how fast clay's algorithms are if they're not allowed to run
(otoh when i try to load the web page it doesn't work at all, not even jankily, if we're talking about https://www.nicbarker.com/clay)
Author here - yes you caught me. I'm building an asm IDE in C (which is why I ended up building this layout library in the first place) and the screenshot is from that application, specifically while building a NES game for Pikuma's NES/6502 course :)
For what it's worth, my experience with the linked website was:
- Text selection isn't possible, except on the final slide when I change to HTML Renderer and then it works very strangely (randomly selects all texts sometimes)
- The page crashed: "Error code: STATUS_ACCESS_VIOLATION"
Yoga is pretty good. Used to have a few bugs that stayed around way too long when it was abandoned for a few years. Development has picked up again though
