I've tried so many static generators, but they just all fail miserably past a certain point. For a simple blog, just text, etc, they're great. But once you get into adding images, rss, or any sort of complex layout and routing, none of them are really designed to handle that much complexity. Metalsmith came the closest because of it's plugin system, but only because I ended up writing my own plugins and just basically writing a static generator on top of it. But it can't do live previewing at all which is really annoying.
Recently I tried Nuxt because I really like Vue, but although these new types of generators are great for handling layout, and the live reloading rocks, they're not really designed for a truly static site, the pre-rendering includes a lot of unnecessary bloat. There's basically a copy of every post both in the html and the javascript, it's also super hard to control it so it's only the posts for that one page. You can remove the scripts manually but then you lose all js functionality. I gave up trying to find a sane workaround. They're not really meant for content that's truly static that you don't need to update.
I've given up on this point and have just started to write my own, focusing on fast live reloading and easy config (like Vue's cli) because that's one of the other huge problems with all generators.
Have you tried Gatsby? I have tried many SSGs and really like Vue as well, but I tried Nuxt out a while ago and it wasn't flexible enough for my liking. I'm not sure I 100% understand your issue, but I haven't found anything that I can't do in Gatsby so far and I'm blown away by the sources/graphql feature.
No, I don't really like react, and it's probably bound to have the same problem as Nuxt. The issue is that the site, at least with Nuxt, is pre-rendered to HTML, right, but it still needs to have Vue (because usually you would want to then use it to then refresh the content). The thing is this, apart from having to have Vue and a bunch of JS, it also adds your posts inline to the script (because Vue can't just extract the content from the DOM (well, it technically can, but this would be a mess for varying reasons). This is a bunch of bloat since I want the site to be static to be fast. You can delete the script tags after the rendering (using something like cheerio), but this gets rid of all your scripts (which I still need some small ones to for example, lazy load images). I could keep them separate and then insert them back in but then I lose the pleasure of using Vue. It's kind of impossible to solve as far as I see.
Ok I see what you're saying but I don't see why it's a big deal. The JavaScript is non-blocking right? So the static page is still displayed just as quickly. Gatsby does code splitting so the bundles are pretty small and keep in mind it takes advantage of preload/prefetch. Using JS to render a new page is going to be faster than loading the entire static page. Maybe you should try testing out real world performance vs something like Jekyll.
Technically, yes it's non-blocking, that's not the problem. For something like, for example, a store front, I agree, these framework renderers are great because/when:
- You have little data that needs to be pre-loaded (e.g. item name, price, description, thumbnail, etc).
- If you do have a lot of data, then your website is of the type that users revisit a lot (e.g. a documentation site).
- The data is modular (it's not one chunk of data) and can benefit from using a framework to organize it.
- Very little content changes from page to page (in a store front, the majority of the layout is re-used).
- There's a lot of DOM manipulation you want to do (e.g. checkout, cart, etc)
This is not true of content heavy blogs:
- It's a lot of data which must at a minimum be duplicated twice.
- The data is not modular. In one markdown file you have to process the text, images, videos, embeds, etc., all at once. So, for example, you can't take advantage of your framework to generate image tags.
- There's almost zero DOM manipulation needed (at most I might have a portfolio page which needs some js, a lazy-load script, and some share buttons).
- A lot of content changes from page to page (it overshadows any content they might have in common).
- The content is not exactly of the type users would revisit often.
With Nuxt (again don't know with Gatsby, and I think they're working in Nuxt to fix this), you also have to jump through hoops to pre-convert your markdown to html because otherwise you're converting on the client side. Found this out when I looked at the bundles and there were a bunch of emojis in it, turns out it was my markdown converter getting imported into the bundle.
Now, let's suppose this wasn't that difficult, there was some function were you could pre-convert this data and it wasn't including the libraries to do it in the bundle.
Except this is THE pain point of SSGs for blogs. The majority of the problems are here and you would essentially end up writing a static generator inside this function (when I realized this I gave up on Nuxt and similar framework renderers). I would have gained very little for that headache, and how big I would be willing to have the bundle size is a LOT smaller.
And while, yes, bundle sizes are usually small and don't matter for the cases I outlined in the beginning, this is not true, like I said, of a content heavy blog. I don't have a nice a proper benchmark, but I do have some notes from when I tried nuxt. These are the sizes of a single test index page (plain text + a couple lines of css for basic styling) and the full loading times with unthrottled and fully throttled (though this is a bit deceiving* ):
Nuxt w/o Content ~170kb - 250ms - 7.8s
Nuxt w/ 10 Blog Posts ~500kb - 70ms - 14s
Nuxt w/ 10 Blog Posts w/ All Scripts Removed ~110kb - 8ms - 4.3s
You can see how my text is about ~100kb, framework is ~170kb, but the script it produces is 500kb because, I actually forgot, it's reproducing my content 3 times (did not bother to investigate why, just twice is bad enough imo)!
* Now, yes technically the throttled load times don't matter because the static html is loaded (they all paint, even throttled at around the same time), so there's no waiting for the page to load, but who knows what the rest of the background loading does to interactivity (on a proper site, with images, buttons, etc, this was just text) on slow phones with slow internet. Why would I do this to my users? It's not like it's even pre-loading anything useful (this test was with a single page, no other existing pages), imagine if it were.
> Using JS to render a new page is going to be faster than loading the entire static page.
Yes, the lack of a refresh is nice, I'll give you that and it's true that subsequent navigations are faster, but in my case I don't think it's worth the overhead. Plus, so much content changes per page, I could always just fetch the next page and replace the body manually without the overhead of a framework.
Don't get me wrong though, I would probably use a framework based SSG for say, a code blog, where it has more pros than cons. So I can see where most developers are coming from when they use them / design them.
Yea I think Nuxt must have started out with different priorities/tradeoffs than Gatsby did because Gatsby has the idea of sources which gives it a nice structure for specifying what processing/data fetching needs to be done at compile time vs at run time. It's super easy to just add a markdown source plugin (actually markdown support might be built-in) and it will compile it to HTML when your run the build and not on the client. There a ton of source plugins, but if you had to, I think it would be much easier to write one and use the provided abstractions than to write the pre-convert function you are thinking of.
You mentioned TTI (time to interactive) as a concern but that is a metric the Gatsby team seems to really be aware of from the beginning and I think they've done a pretty good job of dealing with it so far, see: https://www.gatsbyjs.org/blog/2017-09-13-why-is-gatsby-so-fa...
In general though I've found client-side hydration to work pretty seamlessly in both React and Vue. I'll be shipping my first (non-trivial) Vue Server-Side Rendered app to production soon.
I think Gatsby would do a pretty good job even for content-heavy sites. It's built to try to only render the critical-path items. If you have a content-heavy site, you most likely have implemented some sort of summary list and pagination and only the first set of links on each static page are going to be prefetched by the browser. So it's not like you have to load all the data up front (and you probably shouldn't for UX reasons), you just have to organize the data appropriately like you would for any other kind of site. I've found that Gatsby's graphql interface makes doing these things a breeze compared to my previous React experience where I've had to mess around with state management libraries like Redux.
I'm confused by what you meant by "you can't take advantage of your framework to generate image tags". Is this in the context of Nuxt processing markdown on the client? Like I mentioned previously, this shouldn't be an issue with Gatsby and there are some cool plugins that, for example, allow you to process images in markdown during the build so that they are lazy loaded and initially show a low-res geometric representation of the image.
I can understand your hesitation to want to work with React though. I was hesitant at first. I would get annoyed at writing JSX, but once I got used to it, I found it tolerable and on rare cases I actually prefer it now.
So you got me to look more into Gatsby because it's good they're interested in performance, and what's interesting is it seems to automate the pre-fetching and lazy-prefetch based on if the link is visible (that I can get more on board with). That's really nice and also overall (especially with the GraphQL querying) it seems to be better suited for blogs than Nuxt so I can see where you're coming from. I wouldn't use it because React, but it does sound nice.
> I'm confused by what you meant by "you can't take advantage of your framework to generate image tags".
I mean that the framework needs to be handed the final html. It has no knowledge about the formatting inside a post. You
have to rely on plugins, often third-party, which don't play well with each other.
I know I've gone on about bundle size (that was just the straw that broke the camel's back) but my main problem is that these frameworks don't help in going from the markdown to the final html which is 90% of the work. I'm not saying they should or can, just they don't, so they don't add enough value imo to be worth the overhead. This is true of all SSGs I've tried though. Yeah they always have a markdown plugin and, for example, that image loading plugin does sound really cool, but can I trust them to work nicely with other plugins? Do the plugins I need even exist? Will they be maintained? I can just feel I will end up fighting them and I've wrestled with too many SSGs already. I'd rather just try to roll my own at this point.
> But once you get into adding images, rss, or any sort of complex layout and routing, none of them are really designed to handle that much complexity.
Could you give examples of these pain points? As a maintainer of one of the SSGs on the list, I'm wanting to better understand users pain points, even if they aren't my users.
Well just off the top of my head, here's what I remember from most to least painful. Some of these are not 100% fixable though or I'm not sure the best way to fix them yet without adding more complexity or some sort of editor to my SSG.
- Layout & Routing - Having "special" pages is usually impossible with "structured" SSGs (like Hugo). For example, say you want a page that's your portfolio and only contains posts with x and y tags and you want it to look different than normal pages. Also see rss below. Or for example archives, where "/year" = "/year/index.html" but also "/year/month/" = "/year/month/index.html" and "/year/month/postname" = "year/month/postname.index.html".
- Sitemaps - I have videos on my blog and usually you want to add those to your sitemap properly. To do this I had to patch together a plugin to fetch the video info from the youtube api.
- Smart tags/embeds - Similarly I had to do something similar to fetch embeds from just the link like "[instagram]link[/instagram]". You have to do this with every provider you want, ugh, awful. Also means fighting the markdown converter you chose sometimes.
- Images - I run a blog with lots of images and different views of those images, so I need to have them all in at least thumbnail size and the regular size, sometimes the original as well, but not always. I used to batch convert new ones by hand but that was a pain. I eventually patched together a metalsmith plugin for this to automate the conversion of originals to "regular" (max post width) and thumbnails but it needs improvement. Edit: And I forgot, actually adding the correct attributes needed for lazy-loading, etc, to the image tags was also a pain.
- RSS, Most do not come equipped to handle rss, and available plugins are too simple (for routing that is, like having an rss feed per tag, per archive page, etc).
- Layout Language - Depends on the SSG, most allow choosing or these days are super flexible (e.g. Nuxt) but those that don't and just use one template language, argh! Template languages like handlebars are just not flexible enough. This was the easiest to fix though. I just use ejs now, basically allows any valid javascript, love it.
I can answer to your comment with respect to Hugo.
> - Layout & Routing - Having "special" pages is usually impossible with "structured" SSGs (like Hugo). For example, say you want a page that's your portfolio and only contains posts with x and y tags and you want it to look different than normal pages.
Have you looked at the "layout" and "type" front-matter?
> Or for example archives, where "/year" = "/year/index.html" but also "/year/month/" = "/year/month/index.html" and "/year/month/postname" = "year/month/postname.index.html".
The "/foo/ = /foo/index.html" or pretty URLs is the default behavior in Hugo, which you can also disable.
Regarding archíves, there are quite a few examples in the Discourse forum. I believe I have seen people implement the yearly/monthly archives using taxonomies.
>- Sitemaps - I have videos on my blog and usually you want to add those to your sitemap properly. To do this I had to patch together a plugin to fetch the video info from the youtube api.
You can have a custom layout for your sitemap. Also, you can fetch data from an API during Hugo builds. Look at the Data feature.
> - Smart tags/embeds - Similarly I had to do something similar to fetch embeds from just the link like "[instagram]link[/instagram]". You have to do this with every provider you want, ugh, awful. Also means fighting the markdown converter you chose sometimes.
Using API like embed.ly and such, and can get such data in real time during the site builds. Alternatively, you can create JSON/TOML/YAML/CSV of all such data, save it to the data/ dirnl before the Hugo builds and use that instead if you want to save on thousands of API calls during each build. This is the same Data feature.
> - Images - I run a blog with lots of images and different views of those images, so I need to have them all in at least thumbnail size and the regular size, sometimes the original as well, but not always. I used to batch convert new ones by hand but that was a pain. I eventually patched together a metalsmith plugin for this to automate the conversion of originals to "regular" (max post width) and thumbnails but it needs improvement. Edit: And I forgot, actually adding the correct attributes needed for lazy-loading, etc, to the image tags was also a pain.
Hugo has inbuilt image processing for a while now. Look at Image Processing in docs. It can do automatic centering, cropping, resolution adjustment, etc during site builds. You can choose to do that afresh during each build, or reuse the altered images from the resources cache.
> - RSS, Most do not come equipped to handle rss, and available plugins are too simple (for routing that is, like having an rss feed per tag, per archive page, etc).
Hugo has this inbuilt. You can choose to have RSS for each "list" page like home page, tag page, category page, etc., and even for each individual post if you like. All of that is configurable.
> - Layout Language - Depends on the SSG, most allow choosing or these days are super flexible (e.g. Nuxt) but those that don't and just use one template language, argh! Template languages like handlebars are just not flexible enough. This was the easiest to fix though. I just use ejs now, basically allows any valid javascript, love it.
Yes, Hugo mainly supports Go Templating. (And I don't have any complaints against that :))
I tried Hugo a long time ago. I believe some of the features you mentioned did not exist at the time. Also if I remember correctly, I did find workarounds for most of what I mentioned with Hugo, but the workarounds felt really messy and inflexible. I didn't mean to imply some things weren't possible, I mostly meant to mention it as an example of a "structured" SSG (by which I mean ones that aren't very flexible with their routes or how content is organized, i.e. they enforce a lot of structure).
Regarding archives, the problem if I remember correctly was getting them to nest properly, not just a "/archive" with all your posts, but lists at "/year/", "/year/month/", etc. until you reached the post. So no path was empty. I don't think I ever got that working.
Regarding embeds, didn't know about embed.ly. Seems great, but it's not free. If there's some free alternative I will definitely use it next time around. One of the main reasons I use an SSG is to keep my costs close to 0.
> Yes, Hugo mainly supports Go Templating. (And I don't have any complaints against that :))
Maybe I'm weird, but I can't stand 99% of template languages. They're too watered down. I want the full power of a programming language, or alternatively, some easy way to manipulate the data the template gets. Correct me if I'm wrong, but I don't remember Hugo having the latter either.
> I mostly meant to mention it as an example of a "structured" SSG (by which I mean ones that aren't very flexible with their routes or how content is organized, i.e. they enforce a lot of structure).
I still don't understand this point. You are free to put all your files in a flat structure like:
content/
post1.md
post2.md
But if you want structure on your site, just set a structure in the content/ dir, and that will be mirrored on your site:
content/
posts/
post1.md
post2.md
about.md
> Regarding archives, the problem if I remember correctly was getting them to nest properly, not just a "/archive" with all your posts, but lists at "/year/", "/year/month/", etc. until you reached the post. So no path was empty. I don't think I ever got that working.
You will find many themes, at least right now, that implement "bread crumbs". Even searching bread crumbs in the Discourse forum will yield a lot of results.
> Regarding embeds, didn't know about embed.ly. Seems great, but it's not free. If there's some free alternative I will definitely use it next time around. One of the main reasons I use an SSG is to keep my costs close to 0.
I just arbitrarily threw that name out there. While I am not aware of 100% free API services, there are a few I know of that provide that service for free with rate limits.
> Maybe I'm weird, but I can't stand 99% of template languages. They're too watered down. I want the full power of a programming language, or alternatively, some easy way to manipulate the data the template gets. Correct me if I'm wrong, but I don't remember Hugo having the latter either.
What kind or degree of manipulation are you looking for. The replaceRE template is pretty powerful for my purposes.. I use it to insert anchor links next to headings as I don't like using JS solutions to do that.
My problem is the mirroring itself. I don't want, taking your example, /posts/post1 just because post1 is under posts. You can get around this with some SSGs but this doesn't change the fact that they're path oriented. For example, I think it was possible to turn this off in Hugo, but then you had to do some workarounds for specifying templates because normally the templates have to be aligned with the paths. Plus it has a bunch of weird separated page "types" (archetypes, taxonomies, etc)
For comparison I now use my own metalsmith plugin which is the opposite of these types of SSGs. It only relies on the post metadata. I can basically say, pick all posts that match x criteria, use x route (it can dynamically generate them based on metadata variables), use x template for this route, and set basic settings (posts per page, etc). All from one place. Files can be organized however I think makes sense. There aren't different types of pages or anything. The only "special" thing I had to do was for /year/month/date/ archives, but I plan to remove that exception with the SSG I'm writing now.
Regarding the archives, bread crumbs are not what I mean exactly. I want the pages to also be paginated. So /year/ is not just a list of links to all posts or links to all months, it should paginate the posts list for that year as well if they pass the posts per page limit. Seems like this has never been implemented, see this four year old issue I found: https://github.com/gohugoio/hugo/issues/448
> What kind or degree of manipulation are you looking for.
I often want array manipulations. For example, to create tag clouds. I'm sure it's possible, but I doubt it's pretty. With something like EJS, I can just have the logic in plain javascript, right above the div that iterates through it. I will grant you, I'm not sure this is the best solution. With the SSG I'm writing, I've been playing around with the idea of having a header or footer section in javascript for the templates, where you can run this type of logic and the variables will be available in the template. Kind of how when you use a JS frameworks (e.g. Vue which I love) if you have something complicated to iterate over, you remove the logic from the template. This would make simple templating languages attractive again.
> none of them are really designed to handle that much complexity
Be careful what you wish for! I had used Nikola, but every release kept handling more complexity. Which meant an ever increasing number of options, but most annoyingly a large number of warnings due to config options that had changed, been obsoleted, interacted differently etc. Nikola is one of those tools that generates an initial config file for you, which exacerbates the problem since you probably don't use all the config items, and then have this option warning mess due to those items.
Yes you can install a version pinned version of the tool, but then have to do the same with all the dependencies. Before you know it, Wordpress seems easy and trivial.
Never tried Nikola, but I'm painfully aware of the problem. I think I've tried and migrated through enough SSGs at this point though that I think I can write something better.
The main problem IS the config imo. That and plugins (and their configs). Don't know if you've tried Vue's new CLI, but my idea is if I don't want to or can't abstract away the complexity without imposing limits on flexibility, the next best thing is to "visually" abstract it by making the configuration happen in a UI like Vue's CLI. This will get rid of errors because you would be able to stop the user choosing conflicting actions, and things that cannot be abstracted in a config, usually CAN be in a UI. Main one, for example, and the one I'm focusing on is routing. There's tons there that can be abstracted in a UI there. You can also add hints, links, etc in a UI. There would be no need to be looking at the docs all the time. In the future hopefully I can expand the idea further to include an editor, but I have to keep my goals reasonable or I will never finish.
As for how many features and how complex, after migrating my site so many times I feel the following should be included (because if they're third party plugins they end up un-maintained): RSS, Sitemaps, Image Resizing, and maybe Embeds
PS: I tried wordpress with a static generator plugin once, it was horrifying, never going back. Even ignoring the static generator plugin, it still did barely better than the previous SSG I'd been using. The plugins were all of wildly varying quality. In particular I was constantly fighting the one that resized my images.
That's why I've started a web framework with the goal of being entirely flexible. I've made all code ejectable, minimized/simplified the glue code (so that you eject beautiful code) and made the stack flexible (you can remove/add a frontend/backend/ORM/database to your app easily and at any time). That should give you the flexibility you are looking for.
This looks interesting for making SPAs and more app-like things, but then I haven't found things like Nuxt to be inflexible for that (never actually created a SPA with it though, just attempted to use it as an SSG).
Doesn't look like it would work for me though for the same reasons Nuxt doesn't. It's not really lock-in pain that's my problem btw, but more config fatigue and missing features, and if the SSG has a plugin system to counteract this, then there's the problem that most don't work well or age quickly (imo this is usually because the plugin system is not strict enough), and regardless of whether they work, you end up with this patchwork of plugins, with a bunch of hacks and glue code to make them interact nicely, it's awful.
For me, lack of nice (ie. controllable, but 'automagic') image handling is a huge bug bear in nearly every cms out there, static or otherwise. Perhaps I didn't find the right platform/plugin...
In my case, I wanted to have 1 source image that was resized at various (author controllable) sizes and then saved them, while also generating supporting HTML (eg. 'srcset' to automatically swap between them). I ended up with a couple of ways to template it in [0]:
I think this is a task that is handled by an optional CMS, not the static site generator itself, though I agree it would be handy to have a little helper snippet to do that during build.
There are CMS solutions for static sites which will probably handle images, and all kinds of content, easily. Check out Dato[1], Siteleaf[2] or Netlify CMS[3].
Alternatively you can use external tools to prepare your images and simply rely on the standard HTML srcset within your static site generator of choice. I use RetroBatch[4] to prepare my images.
Exactly what cmroanirgo describes, also see my reply to epage (https://news.ycombinator.com/item?id=17961353) for that and my problems with rss, and some other things I didn't mention. For images, I ended up writing my own metalsmith plugin to resize images to different sizes and add all the attributes to the tags properly.
Recently I tried Nuxt because I really like Vue, but although these new types of generators are great for handling layout, and the live reloading rocks, they're not really designed for a truly static site, the pre-rendering includes a lot of unnecessary bloat. There's basically a copy of every post both in the html and the javascript, it's also super hard to control it so it's only the posts for that one page. You can remove the scripts manually but then you lose all js functionality. I gave up trying to find a sane workaround. They're not really meant for content that's truly static that you don't need to update.
I've given up on this point and have just started to write my own, focusing on fast live reloading and easy config (like Vue's cli) because that's one of the other huge problems with all generators.