D3.js is, to me, one of those tools that makes things possible rather than making things easy (although I only use javascript once every year or two so it's always a slog). This is big: it's like a zero-to-one step, rather than incremental progress.
This fall I made an interactive visualization of the fine-grained plate tectonic model I have built for the Himalaya and Tibet for a research project[0]. The model building process (i.e., finding the relative motions of the plates) is largely solving a system of linear equations, so it's not immediately apparent what the results really mean and whether they are acceptably realistic.
D3.js allows me to explore and understand my own scientific results so much more thoroughly than the static quiver plots that I was making in Matplotlib--being able to pan and zoom a globe (not just a projected map!) and see the tectonic plates move through time is phenomenal. Being able to share the results so trivially as a web page is a huge force multiplier as well.
That’s also one of my favorite examples, but for sentimental reasons: it was the first time I met Shan Carter, who came to my talk on D3’s force layout. This eventually led to me working for The New York Times, and then some years later, Shan joining me at Observable.
I love this graph. I had just landed my first programming gig in a statistical startup when this was published and I remember how inspired we were at seeing it. It really opened our eyes to how useful animation can be in data visualization. So much so that we started migrating many of the graphs we had in R to D3.
Here is an example of a graph we did where we blatantly ripped off one of mbostock’s other NYTimes graphs:
That’s an amazing example! I’ve had the pleasure of trying out the mapping part of D3 for a client job recently and while it’s been a bit overwhelming, seeing demonstrations like that really show how powerful it is. And, to some extent, why it’s so overwhelming!
A little D3 war story: I wrote a solver for the Schrödinger equation, which would render a wavefunction with D3 and SVG - cool right.
It was very slow to render. Eventually I found out why: D3 was round-tripping the SVG through text! The render time was dominated by the browser parsing the thousands of 15+ digit floating point numbers.
I added a call to round the floats to two decimal places, so they would be textually shorter, and it was a huge speedup. What a strange optimization.
I’ve written some animations in React and Svelte recently which edit SVG properties to move objects, and I found myself making the same rounding optimisation (along with using a single path to draw many circles, rather than many circle objects). I think it’s pretty wild that the only way to position objects in the DOM is with strings.
I love D3 and am always shocked by Mike Bostock’s willingness to help. Ten years of D3, yes, but ten years also of Mike answering questions in great detail across GitHub, SO, Twitter, etc. And that’s on top of maintenance + running a startup. Some people just have unbelievable drive.
In other news, I’ve enjoyed using D3 over the years, and always jump on visualization projects as a result. Here’s my favorite thing I’ve made while I was learning D3: https://www.jwilber.me/permutationtest/
I found this so useful. I'm building a suite of open source tools at https://datasette.io/ and the section "Teaching is the most impactful aspect of tool building" was something of a light bulb for me.
"To maximize your impact, then, teaching must be central to your strategy. This means documentation, tutorials, examples, videos, tweets, and more."
I have pretty comprehensive documentation ( https://docs.datasette.io/ ) but almost nothing in the way of tutorials - and I added the first demo video to the website just last week.
This has given me a ton to think about. Thanks, Mike!
I specifically recommend Divio's "Documentation System" structure, and the Vue docs "Writing Style Guide" section as being particularly good resources:
> Of all forms of documentation, examples seem to be the most effective.
This is applicable in almost all types software, not just graphical ones like D3. Many software lack this and can be made much better/attract more adoption if there are more examples.
I love this post. I can't tell you how many times I've started trying to use d3 and gave up after a couple hours because "all I wanted was a bar graph" (at least 3). Learning it is still on my bucket list because it is truly an amazing library.
These days I’d recommend sticking to a framework you already know (like React or Svelte) for DOM manipulation and using D3 as a library on top of that. It eliminates a lot of the learning curve that was associated with D3’s elegant but esoteric approach to DOM manipulation. D3 is still an excellent way to do scales, force-directed graphs, grids and ticks, etc. and it’s modular enough that it works well whether it owns the DOM nodes or not.
Last time I looked into this, React and D3 didn't really work together, since D3 manipulates the DOM and React has its own Shadow DOM. Has this changed?
D3 still manipulates the DOM if you ask it to, but only a few of its subpackages actually manipulate the DOM directly (mostly the selection, axes, dragging, and zooming APIs IIRC). Everything else either transforms data structures within JavaScript, or generates SVG path strings which you can pass down into path elements through JSX.
The one place I’ve found it doesn’t quite play nice with React is not the DOM at all, but that the force simulation insists on modifying data in place where React wants things to be immutable, but it’s not a dealbreaker.
This is an excellent advice. And as you do more advanced graphing you learn more about SVG, which opens even more doors for even more advanced or customized graphing.
With SVG as your canvas and a good framework for DOM manipulation (in my case it is mostly vue), and the shapes and scales available from D3, it really starts to feel like you have as much freedom as you would with a pen and paper.
Check out Amelia Wattenberger’s work. She’s done a lot of writing and has created a course and written a book. I can’t speak directly to the book but I think her work in general is a good representation of the modern way that D3 is used in conjunction with other front-end frameworks.
Second this, reading her work is how I was able to build some really cool things using those ideas. Once you get it, it really unlocks the potential to create some unique things
This is so hard to mesh existing examples into your framework of choice though. That could do well with a lot more examples around to spot the line between framework, DOM and D3
My go-to for that has been Vega these days. Just love the conceptual model and I've actually been able to hold the key parta in my head, compared to most such tools.
For standard graphs, there are plenty of simple libraries built on top of d3 or just pure javascript. Use those. D3 is for when you want to create something custom or complex.
> The only way to recognize — and then bridge — the gulf of understanding between you and people using your tool is to observe their struggles and talk to them. It’s amazing how quickly glaring flaws are revealed this way. Answering questions on Stack Overflow (or GitHub, Twitter, Slack, or wherever else) is not a selfless act of altruism; it’s a chance to learn, to find where people struggle and hear their perspective. Each question is an opportunity to help one individual, but that question can also inspire a tutorial, an example, or even a feature that prevents others from hitting the same issue.
What I love about being a front-end engineer is seeing how people use what I build, which almost always brings up "obvious" issues and improvements that weren't so obvious when you're working on it yourself.
Now that I'm mostly building developer tooling, this gets even better: not only are the users like me so I can empathise with them better, but I know where they're going with their questions and observe them "in the wild", and easily figure out what help they need without needing to set up specific UX research sessions.
But this primarily gets fun if you use it as a way to scale up your support. Which means it's important to keep the following in mind:
> But there’s also a limit. As one person, you can’t help everyone. And you can’t make everyone happy because your tool is only a tiny part of their life. Don’t make that your goal: focus on learning and broadening your perspective. If it stops being constructive for you, then stop. Period. And don’t feel bad!
Don't know a damn thing about visualization or how to use d3.js but I have a shitload of respect for Bostock's work on it, and his role in using it at the NYT.
Love this window into his thought process.
I'm also happy to see someone as accomplished as him say that randos on the internet are also capable of hurting his feelings. It makes me feel substantially less alone in that regard.
I owe a lot of my career to people like Mike Bostock who have made open-source libraries like d3 free and available to the world. I'm sure I'm just one of countless developers who feel the same like me. Eternally grateful
In some ways I miss my time as an R programmer. Back then I could type R and make various sorts of rough useful plots of data quickly. 10 years of nonstop programming later, I no longer use R, and I cannot make rough useful plots quickly in either Python or JS. (There is nothing quick about activating the right virtualenv, starting a jupyter server, getting a notebook running, and trying to remember how to import and use matplotlib. And JS is worse).
Have you tried the python port of ggplot, plotnine? I screencast doing live data analyses with jupyter notebooks, virtualenv, and plotnine. I'm definitely much quicker in R, but it's not too bad!
> To maximize your impact, then, teaching must be central to your strategy. This means documentation, tutorials, examples, videos, tweets, and more.
I think this is one of the main reasons the FastAPI framework is very successful. The documentation teaches you about everything and doesn't assume any prerequisites besides general Python knowledge. I've used it recently, and it's such a delight to work with.
This might be slightly off-topic, but I can't seem to find a direct solution.
I'm looking to build a complex dependency/timeline graph for a web of interconnected stories in a media franchise. Each story has a release date where it needs to appear, and an arbitrary list of dependencies for previous episodes/stories/interludes which would have a line drawn to that dependency. The end result would be a sort of hybrid digraph/timeline, enabling a reader to get the 'minimum path' to catch up to a certain story in the collection.
D3 looked promising, but there's no direct equivalent I can see in the examples. For those that are experienced with D3, is that sort of thing achievable? Should I be looking at D3 or something else?
MIT's Exhibit timeline widget used to be really nice for this, or at least extremely customizable timelines, but I am not sure of the state of that project... http://www.simile-widgets.org/timeline/
this should be feasible with D3. The way I understand it, is that you are trying to make node graph along a time series. So you basically just have one axis (date; usually plotted on the x-axis).
The way I would plot it is to have a long horizontal scroll along the x-axis where a series of nodes are scattered in the single date dimension. Hovering (or clicking) over this node would expand a list of dependencies into the second dimension (distance from the original node might represent time; perhaps in two columns, past [depends on] and future [depended by]). Clicking on one of these dependency nodes would scroll you smoothly where this dependency nodes reside on the main axis.
All of this is possible (and almost trivial) with D3. However you might need to be innovative in implementation as I don’t think there are easy examples for you to follow.
I used d3.js for one of my first programming projects. I was grateful and impressed that almost every question on Stack Overflow was answered by Mike Bostock or Jason Davies.
I'm (a little) older and (a little) wiser, and know this kind of support is a huge amount of work. It sounds like
"focus[ing] on learning and broadening your perspective" is what fueled all those answers I so appreciated.
This fall I made an interactive visualization of the fine-grained plate tectonic model I have built for the Himalaya and Tibet for a research project[0]. The model building process (i.e., finding the relative motions of the plates) is largely solving a system of linear equations, so it's not immediately apparent what the results really mean and whether they are acceptably realistic.
D3.js allows me to explore and understand my own scientific results so much more thoroughly than the static quiver plots that I was making in Matplotlib--being able to pan and zoom a globe (not just a projected map!) and see the tectonic plates move through time is phenomenal. Being able to share the results so trivially as a web page is a huge force multiplier as well.
Thanks so much, Mike!
[0]: http://rocksandwater.net/blog/2020/11/tibet-blocks-js/