Great article! I'm the product manager for the headless feature, and I'm unsure why `binary.add_command_line_options('-headless')` doesn't work for you, as the flag does work when invoking Firefox directly:
Thanks! That was guess as well, since running the command you posted from cmd does work. I dug around selenium's source code a bit, but decided to take the pragmatic path and use the MOZ_HEADLESS environment variable instead.
Is there a linux headless mode already? Selenium chrome driver headless is pretty much useless in linux because important methods like `send_keys` need X anyway for keyboard mappings or something like that.
It's not very clear how to actually use this in a test environment like Selenium etc. At least with headless Chrome there are libraries now to drive it via the remote debug protocol like https://github.com/LucianoGanga/simple-headless-chrome
This feels a little nicer than Selenium as it's one less layer of abstraction.
EDIT: guess from other comments WebDriver is the right method to access.
SlimerJS [0] and Selenium WebDriver are the main APIs supported at the moment. Setting MOZ_HEADLESS=1 or providing the --headless flag to Firefox will launch in headless mode.
It should be mentioned that this is an area of active development. I don't believe headless mode has been "officially" announced yet; documentation should follow when that happens. Optimization, more APIs, more platforms, and easy deployment are in the pipeline. The meta bug for tracking is here: https://bugzilla.mozilla.org/show_bug.cgi?id=1338004
(Disclaimer: I'm an intern at Mozilla, working with 'brendandahl on headless right now.)
The Chrome Devtools API is really useful. However, if you're wanting to use Selenium, a coworker of mine wrote this up to show how to use headless Chrome with Python + Selenium https://duo.com/blog/driving-headless-chrome-with-python
I would imagine the same approach could be used here with minimal changes.
Would people find it helpful if I add a `headless` capability to `moz:firefoxOptions` in geckodriver so that passing in `{capabilities: {alwaysMatch: {moz:firefoxOptions:{headless: true}}}}` when starting a session will start the browser in headless mode, if supported? This doesn't seem like something that's compelex enough to warrant a whole blogpost to explain how to get it working.
Oh, apparently there's already a flag you can pass to the binary to enable headless mode, and geckodriver already supports setting those, so adding a separate option seems unnecessary. I thought it was just an environment variable.
Well, yes, a low level protocol is going to have at least the same features as a higher level protocol implemented in terms of it. However I think you should mention the significant disadvantages of using a nonstandard API:
* Doesn't generalise to driving other browsers. Using a standardised API gives you the possibility to run your automation against any top tier browser. If you are testing a website this ought to be a top consideration.
* Uncertain compatibilty story. I assume that the devtools protocol can change at the whim of the Chrome team. Using something standardised means that your tests and client are more likely to continue working.
* More limited selection of clients. WebDriver in particular has a large number of production-grade clients that are actively maintained.
I'm not claiming that WebDriver is perfect; certainly if I designed it from scratch it would look very different. And yes, there is a period of churn as it moves toward being a standard. But I think the advantages of cross-browser support are a compelling reason for it to be the default tool for remote automation tasks that are within the scope of its featureset. By all means, if you have something that cannot be done with a standardized technology look at the proprietary solutions. But I don't think it should be your first pot of call.
(Note: I'm the product manager for the headless browsing feature of Firefox.)
I chose to prioritize WebDriver support because it's a popular way to drive headed Firefox, and I wanted to make it as easy as possible for existing WebDriver users to use headless.
For use cases that are not well-served by WebDriver, I've considered supporting the Chrome DevTools Protocol, as @brendandahl noted, although I haven't yet made a decision to do so.
I'm interested to hear more about the use cases you've considered for which WebDriver is a suboptimal solution. Like @brendandahl, I'm in the Bay Area and would be happy to meet in SF or MV.
On the Firefox side, we are considering supporting the protocol or a subset of it. It seems many people have had lots of trouble with WebDriver/Selenium in the past and are hesitant to use it. However, it is nice that WebDriver has a W3C standard which could provide a nice path forward for headless cross browser use. This would probably require browser vendors to make it a first class citizen and work out some the kinks of the spec though.
BTW, I'm in Mozilla San Francisco office, so I'd be happy to chat about the headless cross browser future if anyone from the headless Chrome team is around.
1. You can use the WebDriver protocol[0] without selenium
2. Firefox's WebDriver endpoint geckodriver[1] is a layer above Firefox/Gecko's native Marionette protocol[2]
So you doubly don't need to use selenium, you can write your own WebDriver client (which should be cross-browser assuming the browsers either support WebDriver natively or have a WD layer of some sort installed) or you can use raw Marionette (either a hand-rolled client or an existing client)
Patch author here: To add to the other comments, you could also use SlimerJS[1]. Though I don't know if I'd consider that lower level. We're open to supporting other ways of controlling FIrefox and have been gauging feedback on what to support.
I was wondering if I could use this (using a plugin) or Chrome to generate PDF files on the web server. Most of the PDF generation software out there are quite expensive.
You can. In my experience, there are very few systems as good at producing accurate renders as well as Chrome. There are a few Chrome/webkit based options.
PhantomJS made the process pretty easy, but that's probably not a good option these days as it's on the way out.
NightmareJS is pretty much the same and can do it (though I haven't tested myself) [0]
If you want to get a little deeper you can use CEF (Chromium Embedded Framework [1
]. It basically turns Chrome into a library and there are a bunch of projects that build on top of it (eg Electron, which is what NightmareJS uses underneath).
Here's a project that just uses CEF for converting to PDF [2].
it has a lot of quirks, and, frankly, webkit is pretty bad at printing in general as it lacks a ton of printing specific features like re-printing table headers across page boundaries.
There's also only very rudimentary support for page headers and footers.
wkhtmktopdf is probably ok if you just need a quick single page invoice printout, but the moment you need more (in our case, we definitely do), I can highly recommend paying for https://www.princexml.com/ which has support for a lot of printing specific features.
We used node-html-pdf[1] for this, but it depends on phantomjs which uses an ancient version of webkit. We migrated to electron and its contents.printToPDF(options, callback), but that has the drawback that we have to start a X-server[2] with our backend code. I have hopes with a headless chrome we can ditch the xvfb-server and have one less dependencies. The setup is a pain, but the results are very good and looking very decent.
Really not the same thing (e.g. pandoc for html -> pdf via LaTeX) as having a webkit based renderer. They do not look the same at all; sometimes it is good to have a LaTeX output but most of the time you are going to want a more accurate representation of your HTML.
The original question didn't state that he was starting from HTML. If you want to generate an invoice or such from plain data, then LaTeX seems like a good fit. Rendering HTML is a whole different bag.
That is true. TeX is a typesetting engine, and allows you exact control over the final rendering, while HTML is a markup language, the rendering of which depends on very many factors.
What are the biggest challenges when implementing headless mode? I'm asking cause this feature took some time to be delivered both in Firefox and Chrome and I always assumed that it should be 'pretty' straightforward to implement.
Is it that both engines were coupled with GUI libs?
There are a bunch of things which are tied to native GUI libs in ways that make headless a little annoying to implement. Some examples, from memory and looking at the patches/dependencies in the Firefox bugs on this:
1) Theming of form controls to make them look like native widgets.
2) Clipboard.
3) Printing.
4) Fonts and font rendering.
5) General "now it's time to repaint" machinery.
This is in no way an exhaustive list. In practice what you have to do for headless is create the relevant GUI bits anyway, but not actually show them on screen, then deal with whatever quirks your GUI library has when its bits are not shown.
Patch author here. bzbarsky's comment covered most of it. It hasn't turned out to be that difficult. On linux, it was mostly playing whack-a-mole of avoiding calls in to gtk/x11 and then creating headless implementations of the platform specific code.
One slight complication of headless has been wiring up the headless "widgets" (in Firefox, we refer to most of the platform specific code as widgets e.g. there's gtk widgets, cocoa widgets, windows widgets). Usually the widget type is defined statically at build time, but in the case of headless we wanted Firefox to either use the headless widgets or the platform specific widgets at runtime. Luckily, some of the work on multi-process Firefox work also added another type of widget and made it much easier to support multiple types of widgets.
Overall, the hardest part has been trying to replicate all the events that would normally be triggered by the platform gui code. I've also found that these events and order can vary per platform which makes it hard to do in a non-platform specific way. We're still working out some issues here.
When you run the browser, but it doesn't actually display anything on screen.
Uses include rendering a screenshot (e.g. a preview thumbnail) of some website on a server, or running automated UI tests that click around a web app and fill out forms to verify its functionality.
Firefox audio developer here. I haven't heard from the folks doing this, and I would have been the person doing the code reviews so I suppose it's left untouched.
Edit: looking at the code, it will work just fine, and that means I can even use it myself to run Firefox's unit and integration tests, I usually use xvfb on Linux.
Firefox 55 introduced major changes in the process model. My scrolling on the Mac is slow and not smooth. Others have reported the same. Maybe focusing on stabilizing this big change over one or two releases would be a good thing?
It is good to see some competition, I would hate to see automated tests being done purely in Chrome simply because it is the simplest browser to setup in headless mode.
But can you really run windows headless? I thought even the Server Core and IoT versions wanted some sort of GPU even just to display a blue/blank screen.
I think the goal is to have a browser that renders web pages and is fully interactive, but just doesn't render to screen and is interactive through an API rather than input devices.
So it's going to need a UI rendering and interaction library.
You don't actually need to have X11 etc running to use headless mode, the libraries are just necessary for linking as there aren't separate headless and non-headless Firefox binaries shipped. The possibility of dynamically loading these libraries to remove this dependency is being looked into.
As minor counterpoint: Firefox is not strictly speaking developed "by Mozilla" but by the community. Mozilla's the curator, and has a fair bit of staff also working on it, but vast swathes of the codebase are community contributions. If you need FF to do a thing, and you think you have good enough ideas to contribute the solution there, then rather than upvoting a bug, or bumping it to ask what's happening, be the person to make the happening happen =)
Of course, but bug upvotes don't hurt that cause, either. I have a few reasons that I work on open source projects. First, to fix a bug or add a feature that affects me directly. Second, to learn something new. If I'm just doing the latter, I'd prefer to tackle a bug with more interest/votes, to help others out while I'm at it and increase the odds of finding a willing reviewer.
If your running this on a server sure great. On desktop that is next to pointless due to the need for those dependencies for everything besides cure apps.
It should be pretty similar on Linux, and probably macOS when it comes around.