> The conventional approach to writing audio software is an challenging process requiring expertise in a wide breadth of topics– C++, multi-threading and thread safety, lock-free programming, and realtime thread safety come to quickly to mind, and we haven't even mentioned digital signal processing itself. Moreover, the tooling, workflow, and developer experience that we have in this approach pales in comparison to that of the web software industry, for example. All together, the reality of these complications is that the conventional approach for writing audio software is difficult and time consuming. We believe it doesn't have to be that way.
This is then followed by a screenshot of some code that looks like a SuperCollider script from 15 years ago.
There are at least a dozen audio "scripting" languages that free someone from all the issues mentioned above. Many have been mentioned in the thread already. The only things that Elementary Audio appears to have in its favor are that it leverages JS and it runs in the browser (sort of).
For some people, those two alone will be interesting. Have at it, people!
But for people who write plugins designed to run inside DAWs, or modules inside Rack, and in particular for the people who write DAWs or Rack (or Rack-like things), this is of approximately zero interest. It's a web-centric version of SuperCollider, which is cool but ultimately so bounded by the limitations of the platform that for many things, SuperCollider or something similar remains more powerful and more flexible.
That was my impression too, it looks very much like an SC style computer music language. Which begs the question, who is this for? Someone who writes plugins doesn't need or want the high level SC style language. Someone seriously into computer music/audio isn't going to want the limitations, and has a lot of options already to choose from that are mature, if relatively unknown outside music circles.
If it were my company, I would be marketing this squarely at people with the need for high level procedural audio in a web context. This looks like it could be great for browser-based game devs, for example. I do know that Pure Data and Andy Farnell's great sound design book for it were a hit in the game dev world of procedural audio, FWTW.
It's probably for people who are smart enough to know DSP, but only know JavaScript and refuse to learn anything else. If you squint it looks like an empty set.
SC is typical OOP language. From this perspective, the raw Web Audio API or Tone.js are more relevant for SC.
This Elementary Audio prefers FP, which makes it quite different and I think it has the potential to be very friendly for beginners. But the docs need a lot of more works there.
If that seems "quite different" to you, then good luck!
Personally speaking, SC's approach seems more amenable to composition (in the programming, not the musical sense). Collapsing everything down to core.render() puts the emphasis in the wrong place IMO. The oscillator is better considered to be an independent entity, and the "render" step really ought to be implicit. But to each, their own!
for example, "1 + 1" in FP is different from "1 + 1" in OOP.
In FP, everything is func. 1 is func; + is a combinator.
In OOP, 1 is an obj, + is the method of that obj that takes another obj.
It looks like you really enjoy SC's syntax. It's great. But essentially, the language is just one part of SuperCollider.
SuperCollider = sclang + scsynth
sclang -> OSC msg -> scsynth
The scsynth is actually the soul of SuperCollider to many computer musicians and that is what drives many to develop different language front-ends for scsynth.
Language is just a wrapper, with different programming paradigms. But down to the lower abstraction, it's the audio graph. I have no connection with Elementary js at all but I think it does a good job for providing a new angle for Web audio. Yet I understand your concern of using it for developing vst plugins.
I guess another thought, mind you, is that Elementary doesn't need to succeed for hard core audio devs in order for it to succeed as a business - it just needs enough people who think they will succeed. If this turns out to be a great tool for casual or beginner plugin devs, I could see that being a valid market. Witness the explosion of casual Ableton Live plugin devs from Max for Live. I'm sure the vast majority of JUCE licenses don't wind up becoming sustaining businesses too, and the barrier to entry is a lot higher there. Whether those folks will get to the point of paying a license fee is the million dollar question I guess.
Ain't nothing wrong with selling pickaxes to miners and all that....
Still, I think that we have an obligation towards non-hard-core audio devs (whatever that actually means) to remind them that if you want to find gold you likely need a pickaxe, but if you want to turn over a field, you need a plough.
It's not important that everyone's efforts become successful businesses. What seems more important to me is two-fold:
1. the existence of suitable entry points for "non-hard-core audio devs"
2. if it isn't possible to provide those entry points using the same tools that the "hard-core audio devs" would use, to be sure that the "non-hard-core" folks understand that there is a difference and that there is a smooth migratory pathway between the two worlds.
On that front, Elementary looks fairly good, other than its fairly strong claims that this is a suitable technology for creating commercial audio plugins.
Let's be honest: the main thing that gets in the way for "old school" technologies these days is just compilation. Perry Cook's synthesis toolkit is a very old bundle of C++ modules that lets you do more or less the exact same thing that Elementary does. But being C++ means that there's a build step (which may generate errors). For people more familiar with interpreted languages like JS and its cousins, this appears to have become an almost insurmountable obstacle. If you can't just type it and run it, it's crap.
I think that's wrong, but I understand the appeal of "type and run" for people starting out on the path of "make the computer do what I want".
As a total outsider to the audio programming world, what's the best in class framework / environment to be writing plugins in in 2022 if one were to start today?
I agree with the first half. For the second half, I think for beginners, the examples are very important. From this perspective, many Rust projects comes with examples:
Once following the readme, it is very easy to get it work in your own machine. Then beginners can play around the examples, tweaking parameters, etc. while learning new stuffs with books or online resources.
Rust audio has also got a very helpful Discord community where beginners can always ask questions.
For the GUI part, I am not an expert, but there are more and more Rust GUI libraries (egui, iced, druid, rui): among them, egui-rs and iced-rs can all be used for VST. Still, there are some examples to get started with.
The visual programming languages tend to have really thorough guides on the DSP - max/msp's built in docs in particular, but pure data and external libraries are super in depth too. They also tend to have horrible GUI practices...
Actually I am teaching both pd and glicol in the University of Oslo this semester. It's even quite surprising to me that most students seem to gain more logic programming ideas from pd (such as the counter and metro controlled sequencer) while learning more dsp from glicol and its browser console based docs: https://glicol.org
Of course, Miller's book is always there http://msp.ucsd.edu/techniques.htm and the examples are in pd, which is always helpful for learning dsp.
JUCE is also reasonably popular among GPL'ed plugins, since it does allow for a GPL option. It's not perfect, and in fact has many important issues, but it's probably the least bad of a bunch of not-very-good options (no matter what type of license you're planning to use).
Well, if you plan to release a commercial product based on writing JS, compiling that to WASM, running somehow inside or alongside a C/C++ core engine also compiled to WASM, both running in some standalone WASM environment ... good luck!
Elementary looks like a nice playground for audio if you have a web-leaning bent. Just like SuperCollider is a nice playground for audio if you are familiar with some its fundamental assumptions.
Neither of them seem appropriate to be the basis of products I would charge someone for, other than maybe as a one-off custom tool of some sort.
The entire source code of Ardour is posted on the Internet. Nevertheless, it has no known bad-intent copies, and generates more than US $200k/yr. It is also the basis of two products generally viewed as "commercial", even though they are also GPL-licensed.
Almost all of the source code of VCV Rack is posted on the Internet. There have been a few forks for specific reasons, but it's almost certainly the most widely used software modular synthesis in existence, and has created both a paying job for its creator and a mixed libre/paid ecosystem for module developers.
The entire source code of Vital, one of the most acclaimed software synthesizers of 2021 is posted on the Internet. The author (Matt Tytel) makes a living from his work.
The definition of commercial might be a little wider than you think.
Finally, there are several ways to use SC in a product that do not involve the GPL. You fire up sclang as a separate process, feed it SC code, and sclang controls the server. Nothing you write is linked to anything GPL licensed.
I'm glad to hear Ardour is working well for you. I honestly had assumed it wouldn't be making much income, though I now realize that was based on just about no information (the sole data point I had was that it worked poorly for me when I attempted a trial on macOS about two years ago). I'm glad to hear I was mistaken about that.
I wasn't aware that VCV Rack was actually profitable either, nor that there's a mixture of libre and paid modules available for it. I'd vaguely considered that as a business model for OSS audio stuff in years past but didn't think it was likely to work in practice. Apparently I may be wrong about that - do you know if anyone's actually making decent income selling closed modules for VCV Rack?
You make a good point about running sclang as a standalone process to firewall the GPL from the rest of a product. Somehow that hadn't occurred to me, but it seems like it would work.
Overall - thanks for the rebuttal. You make a strong case that open source audio software is more compatible with commercial sales than I thought, and I appreciate you taking the time to do it.
Paul, I am very happy to hear that Ardour is succeeding in supporting you. While I'm not really using Linux these days for audio as I'm doing my Scheme in Max stuff now, your contributions were very influential to me back in the oughts when I was, and I remember learning a lot from your posts on various linux audio forums. It's heartwarming to know that the work is succeeding in making a good lifestyle business for you. :-)
This is a bad faith argument and is effectively FUD.
The list of open source projects that people use in a commercial setting is long. Linux, Apache, Nginx, mysql/postgres/SQLite, Redis, Git, nearly every major programming language (C, PHP, Go, Python, Node), Firefox/Chromium, to name a tiny fraction.
Businesses that have libre/free/open source as their core product are also numerous: SQLite, Gitlab (open core), Red Hat, Docker, Nginx, Npm, Redis, Wordpress to name a few of the big players. There is a long tail and companies like Draw.io, Gratipay, Overleaf, Jitsi
and the list goes on.
The list of open source hardware companies is also long: Adafruit, Arduino, Sparkfun, Seeed Studios, Gaudi Labs, LittleBits, Olimex, Zynthian, to name a few.
GPLv3 is a so-called "viral" license, making sure that any changes are given back to the community. This is done to prevent bad actors from taking the code, modifying it and then claiming it as their own intellectual property. From what I understand, this has the effect of preventing larger businesses with a more "classic" intellectual property stack from using smaller business's code base without giving back.
I have heard rumors that larger businesses, like Amazon, specifically use GPLv2 so as to not trigger the viral licensing, effectively benefiting from the commons without giving back to the broader community at large. I have also heard companies like NVidia use MIT licensing for code given to potential clients/customers/internally and strongarm everyone involved to not release it, creating a situation where they're not violating the licensing terms but never releasing it to the public.
Though a cultural or psychological effect, from what I understand, smaller businesses tend not to be too successful at "stealing" competitors code as the brand awareness and domain intelligence looks to be the more relevant factor to success. For businesses that do use others libre/free/open source code, there tends to be a virtuous cycle, where they each build on the others success and widen the market. Perhaps it's the other way around that success in one business allows for others to expand the market further.
Making money from business is difficult to begin with regardless of whether the core product is libre/free/open. Understanding how to create a business that has libre/free/open source products at its core is still being discovered, so it's not like there's a definitive answer but to say that it kills the opportunity is flat out incorrect.
> This is a bad faith argument and is effectively FUD.
As the one who made the argument, I can assure you that it was not made in bad faith.
I'm a big fan of open formats, open source, and libre software, and have made small contributions to various projects for a number of years (usually code contributions and bug reports, but occasionally financial donations too).
The argument was made from my sincere belief that, difficult as it is to make a sustainable software business (which I've seen up close, having participated in multiple failed startups), it's much harder to make a successful business with an open source product.
I would love to find out my belief is wrong, but it's still the one I've got until persuaded otherwise.
Accusing me of bad faith added nothing to your argument and violated the site's guidelines. Maybe skip that next time?
> The list of open source projects that people use in a commercial setting is long. Linux, Apache, Nginx, mysql/postgres/SQLite, Redis, Git, nearly every major programming language (C, PHP, Go, Python, Node), Firefox/Chromium, to name a tiny fraction.
Yep, OSS is everywhere and used for commercial purposes all the time. Large swaths of what you've listed above aren't commercial products, though, AFAIK.
I think most commercial software products use open source tools somewhere in their stack.
I should have been more specific with my claim - I think if the core components that form your software's value proposition are OSS, it's very hard to make a sustainable business out of selling that software.
The fact that open core products are a popular and successful way of monetizing OSS tools suggests as much, as there's some proprietary "secret sauce" that the users pay to get access to.
Open hardware works much better, I think, because people instinctively understand they've got to pay for someone to build the physical artifacts. Somehow, even developers seem to undervalue the effort it takes to build and maintain a useful piece of software.
> but to say that it kills the opportunity is flat out incorrect.
I don't believe I said it kills the opportunity. Just that it makes it much more difficult.
> Accusing me of bad faith added nothing to your argument and violated the site's guidelines. Maybe skip that next time?
I called out bad faith argument then steel-manned your argument and responded in good faith. In this case I felt calling out a bad faith argument and FUD was concisely pointing out part of your argument, which I take to be part of being a critical reader. I assume good faith until evidence suggests otherwise, which it did in this case.
> Large swaths of what you've listed above aren't commercial products, though, AFAIK.
I point out that there are many open source projects that fuel businesses and then go on to explicitly list businesses that have as their core product a piece of open source software. You conveniently left out the line directly underneath it listing those businesses. I qualified each of the sentences correctly.
> I don't believe I said it kills the opportunity. Just that it makes it much more difficult.
This is the original sentence you wrote:
"""
SuperCollider's license effectively stops you from designing commercial products. Even the free version of Elementary allows that.
"""
---
You continue to argue in bad faith. Me pointing it out highlights that you are engaging in behavior that is undermining constructive debate.
> This is the original sentence you wrote: """ SuperCollider's license effectively stops you from designing commercial products. Even the free version of Elementary allows that. """
I forgive you, and appreciate the apology and recognition that you needed to step away and take a breather.
I do think you make a decent case for open source being a viable way to make a profitable business. There are certainly businesses that achieve that, and I would love to see more of them.
I wasn't trying to give a blow-by-blow rebuttal to everything you wrote, so it's certainly fair to say I was cherrypicking.
I was just trying to explain some of why I think making a business's core product open source ticks the difficulty level up a notch. I believe AWS has made a practice of snagging OSS systems and making them effectively their own, thus undercutting the original developers' attempts at running a business. I may be mistaken about that, though.
As I said, though, you make a number of good points (most of which I already agree with). There are a number of ways to make it work, and I may well have overstated the difficulty of building businesses with GPL software.
Maybe I'm missing something from my quick read, but the idea of using this for plugins seems like a real misfire. I'm curious whether the devs come from the pro audio world at all. It's one area where prioritizing for development time by using high level garbage collected languages just isn't done because your user is always concerned with how many instances they can run and what the lowest latency they can get is, and that latency constraint is going to be applied across your entire system. So if there is one plugin that hoses the latency by needing a time slice big enough to run a GC in the audio dsp routine, pro audio buyers won't touch it.
It looks like a nice way to do interactive web audio work, but plugins are a different beast all together. If you want a place where old school skating-with-razor-blades pointer coding is still common practice, it's DSP code.
You're totally right that in this domain you have to be extremely careful with performance and latency. Elementary takes great care to deliver that; it's a JavaScript "frontend" API but all of the actual handling of audio is done natively with high quality realtime constraints. The core engine is all native C/C++, and we run in the web by compiling to wasm and running inside a web worker. For delivering a desktop app or audio plugin, the native engine is compiled directly into the target binary.
So, while JavaScript and garbage collectors can bring a performance concern, we're only using JavaScript there for handling the lightweight virtual representation of the underlying engine state, and for that role JavaScript is plenty fast enough and the garbage collector quite helpful!
That's great on paper but how do I actually use it? If I want to play back a wavetable you have el.table() but it's missing the oversampling and decimation tools required for antialiasing. The el.lowpass() is a biquad which is not suited well for modulation. How can this compete with JUCE when the documentation and features are so sparse?
You should really fix the initial impressions you make from the site to get that front and centre though. Experienced audio devs are going to dismiss this unless you make it clear very quickly.
Ah cool, after I posted I was wondering if this was the case. I was just thinking "maybe the renderer is in WASM"?
That's pretty cool, it will be interesting to watch. I do something very vaguely similar in my computer music work where Scheme orchestrates C level objects.
Personally, I wouldn't want to use JS as the high level orchestrator myself as it's such a dog's breakfast of a language, but I can see that many might.
This is my new favorite comment for illustrating the perilous future of general computing and how it needs to be taken away from JavaScript if we have any chance of survival. Electron, Webassembly fetishism, the pursuit of language synchrony at the expense of common sense, it all gets you this. This comment. Right here. This is the future of software and it should scare the shit out of you.
Let me get this straight: you realized latency was a concern, so you wrote in C/C++ (which, exactly?), then turned it into wasm so you can run it in a Web worker? What the hell was the point of that? That’s like saying you bought an M6 and converted it to a foot-powered bicycle. What exactly do you think wasm does? You seem to be implying that you think the native engineering you invested in continues to matter, in the same way, after you do that. You also imply heavily that you understand the wasm you’re executing to still be native. Do you think that? Do you understand what you’re giving up in the Web worker? As in, directly tied to latency and real-time requirements, your whole reason to go native in the first place?
Whatever your response is, deep down, you and I both know it’ll be justification to do Web things for Web’s sake. I know this because everyone I’ve had this discussion with has played the same notes (imagine the deployment model!) while failing to understand that they’re justifying their preference. The only people who build Web stuff want to build Web stuff. In the high performance sphere, of which DSP is a very mature practice, this looseness with the fundamentals of software is going to put you out of the game before you’re even started.
I'm a web person defending web things but providing something on the web has a significant advantage.
I know you've heard this again and again, but I can't emphasize it enough.
You can use the site from most platforms, including PCs and mobiles.
You don't have to install software, a single click is enough.
Of course, browsers have considerable limitations, and serious users will eventually choose other tools, but providing such an accessible software is a really huge advantage for me.
> You can use the site from most platforms, including PCs and mobiles. You don't have to install software, a single click is enough. Of course, browsers have considerable limitations, and serious users will eventually choose other tools, but providing such an accessible software is a really huge advantage for me.
In the audio world that is cool for toys.
There are a lot of really cool things to play with in the browser/multi media space, maybe this is another.
But when it comes to getting shit done, writing plugins and transports, creating instruments and rigs for live use (my hobby these days) the quite demeaning comments here are on the mark.
This "Elementary" is prioritising the wrong things. And there are a lot of frame works that do not require you know C or C++
Pure Data, and Sonic Pi are two I have played with (the former much more than the latter).
Platform independence is simply not an issue when building these systems. Installing software is not an important issue.
Sorry. This is, on the face of it, a waste of time. I hate saying that, but if it were me doing this I would pitch it as a fun toy.
Do you understand how WASM and Web workers work? Do you understand that low-enough-latency audio doesn't take a super computer anymore? Yeah, if you were working on DSP stuff in the 1990s, you were a hot shit programmer. Nowadays, it doesn't really say much at all. And it certainly doesn't justify talking about it as if it were a moral failure to not treat DSP with respect.
> Do you understand that low-enough-latency audio doesn't take a super computer anymore
It never did. Low latency audio has almost nothing to do with CPU power. Here's a summary of some of the issues faced on modern general purpose computers:
I know how WASM and Web workers work. Since nothing you can do in WASM or a web worker has anything to do with either (a) realtime scheduling priority (b) actual audio hardware i/o, they don't have much to do with solving the hard parts of this. Browsers in general do not solve it: they rely on relatively large amounts of buffering between them and the platform audio API. Actual music creation/pro-audio software requires (sometimes, not always) much lower latencies than you can get routing audio out of a browser.
And even when we set it, we don't get it, because we blithely read a "latency" label in a GUI instead of measuring the round-trip latency on the specific device in question.
That wouldn't be correct either, at least half the time. Problem is that "latency" is used with different meanings, at least two:
1. time from an acoustic pressure wave reaching a transducer (microphone), being converted to a digital representation, being processed by a computer, being converted back to an analog representation and finally causing a new acoustic pressure wave care of another transducer (speaker).
2. time between when someone uses some kind of physical control (mouse, MIDI keyboard, touch surface, many others) to indicate that they would like something to happen (a new note, a change in a parameter) and an acoustic pressure wave emerging somewhere that reflects that change.
The first one is "roundtrip" latency; the second one is playback latency.
my friend is running an independent studio, i'm a bedroom musician, and we're planning some elementary plugins together. we tried some stuff with max and it wasn't expressive enough, we tried the steinberg sdks but they were too much of a lift, elementary is good enough for our workflows and in that perfect zone with enough control and enough familiarity that we feel we can be productive. we'll see in due time if it's too good to be true, but we're excited!
or even if you can't! I use Max and to a lesser extent Pd, but > 90% of my actual work is done in Scheme or in C running inside them. Many advanced computer music people wind up doing this sort of thing because the graphical environments just can't be beat for quickly lashing together an environment for interaction, where interaction could be GUI, midi, network/OSC, customer hardware over serial, etc.
What a crazy coincidence. I just spent the last couple of days doing a survey of audio engines with a mind to find those that could run in wasm.
Did you write your own engine or are you built on top of csound, supercollider, ChucK or something else? How do you compare against the higher level or pattern based implementations like Sonic Pi, Tidal Cycles, Overtone and Cabbage Audio? How about libraries like LibPD or even Max/MSP?
I was even considering this space myself and on my list of todo's was checking out the licenses of the above to see what the ramifications of deploying a commercialized version of them would be. However, I must admit that the market for audio software is incredibly split. On the one end are the high-end audio effects with eye-watering prices. On the other OSS end many of the devs are literally begging for scraps on Patreon barely making enough to survive. There doesn't seem to be an in-between. I am very curious what you believe your market will be.
You should check Glicol:
https://github.com/chaosprint/glicol
For the marketing part, frankly speaking, I don't know. Since most of these projects you mentioned are based on academics, including myself.
I definitely did checkout Glicol, sorry I didn't call it out along with the others but I wanted to focus one the "uses existing old-school C/C++ audio engine compiled to wasm as a server" question. I understand you wrote your own custom engine in Rust. I saw your post in the comment section related to Overtone and was actually going to reply saying to you that I appreciated your work. I think I watched all of your Youtube video demos, even the ones on QuaverSeries.
The other small project that was interesting to me (that seems to have partially disappeared) was Sporth. The webpage for that has an unfortunate message saying the project was taken down to respect csound community.
Wow, thanks for your interest! I haven't maintained QuaverSeries for a while as I kinda feel that there are some issues on Tone.js that cannot be fixed in the foreseeable future.
For those language you mentioned csound, supercollider, Chuck (should add Faust here), I think these languages focus more on different music programming philosophy. Of course, SC is probably the most famous for reusing its audio engine for other high-level langs like the Tidal, Sonic Pi you mentioned. Also, SuperCollider's scsynth can run in browsers now:
https://scsynth.org/t/webassembly-support/3037
If you are interested, https://strudel.tidalcycles.org/ is an ongoing work to port Tidal to browsers. Perhaps at some point it can be connected with Glicol audio engine.
That supercollider/scsynth wasm thing was actually what I was thinking about. Unfortunately the implementation is still in a fork and hasn't been merged upstream. [1],[2]
Thanks for the other links, sema-engine was new to me and not in my original investigation. I think I skimmed over Maximilian but it is probably worth a closer look.
One of the benefits of csound/supercollider in my mind are their existing communities and the number of available open source modules. I've seen entire repos full of .scd definitions which would give any project starting with supercollider as its engine a big head start. The main downside is the esoteric languages these projects tend to use, which is why I think there are so many projects that build on top of it. This is one of the benefits of the client/server model I suppose.
I also agree that this whole area is full of different philosophies. At the low end you have straight DSP kind of stuff which is mostly floating point buffer manipulation, FFTs/convultions, etc. and at the other you have sequencers, scale pattern generators and composition tools. And of course the live performance aspect comes into play as well as interfacing/syncing with devices/DAW using OSC/Midi. I'm very excited to see where it all is going to end up.
> At the low end you have ... and at the other you have ...
Totally missing out on the actual DAWs where most people do most of this stuff.
Yes, there is some interest in and use of standalone tools. But for all kinds of reasons, the majority of new audio software development occurs with the assumption that it will run within a DAW (or at least, a plugin host).
One of the reasons for that is to offload almost all of the architectural issues to the host, and leave the developer to pay attention to the sorts of things you describe in the paragraph I was quoting from.
For my personal interest (probably no where near the majority) I am interested in these libraries primarily for use outside of a DAW. I mean, I already have a wealth of plugins for most conceivable purposes to use in one of the several DAWs I have installed. I am also in no hurry to move the kind of things I do in a DAW to a web based app. And frankly, none of the examples I've seen so far come within a country mile of the audio quality of the higher-end plugins I have.
I'm interested in these libraries to see how we can expand on musical creativity outside of DAWs. The first thing that comes to my mind was the cute HTML 5 game Friday Night Funkin'[1]. Or games such as Lumines [2] where the rhythmic music is integral to the gameplay. And other kinds of interactive media where advanced control of synthesized audio might enhance the experience. I think it would be interesting to give people a simplified/gamified interface to a suite of audio tools with the intent to unlock creativity, especially aimed at a very young age (e.g. toddlers and pre-teens). This is a place I don't think traditional DAWs will ever attempt to target and where web-first experiences tend to do extremely well. Being able to quickly develop and deploy "good-enough" audio effects that have an element of interactivity for those kind of products/experiences would be nice to have.
I work on this space a lot, and maintain a couple of libraries you might be interested in. They are very much focused on light quick "good enough" web audio music/sound effects.
Excellent and interesting response! The idea of responsive audio within a browser-based game had slid right by my old-fart brain. Will be interesting to see how that turns out.
I have been playing with Faust and WebAudio lately. It works really quite well - e.g. the online Faust IDE [1] and playground [2] use the WebAudio/WASM implementation to run in the browser. The Faust team are currently working on moving to an improved WebAudio architecture (with support for WebAudio Modules etc.) but the current stuff should work well enough for most cases.
It’s a cool language too, very functional based. I’m a total newcomer to DSP but it was pretty straightforward to figure out how to build a simple synth in not much code at all after reading a few docs and looking at examples. There are loads of high quality implementations of effects etc. in the standard library and examples too so you can get something sounding pretty good without too much work.
One of the biggest problem I have with WebAudio API is managing nodes. The API is extremely verbose, when you consider that anything interesting will need to be composed of dozens of basic nodes combined in different ways.
You're given a very basic set of primitive nodes, from which you can construct pretty much anything else. But those things you construct will never be "nodes" themselves. You can't create synthetic nodes or subclass node types. But then many of the basic nodes are useful on their own. So you end up in a bind of having two different provenance branches of audio "node-like" things and need to know which is which before you can connect any one of Column A to any other of Column B, or vice versa.
You're also on your own in terms of keeping track of those connections. Once a node is connected to another node, there is no in-API way of telling what nodes a node is connected to, or what nodes connect to a given node. There's a whole, huge, potentially cyclical graph structure sitting around in memory that you can manipulate but never inspect.
With how hard everything is to keep track of what is where, you can easily end up in a situation where old, now-disconnected or unused nodes stay resident in memory and never get garbage collected. Some developers then punt on tracking connections at all and expect the user to reload the page if there is a major change in audio setup (coughcoughGoogleOmnitonecoughcough).
So when people talk about how Web-based gaming really hasn't taken off or replaced the haunting specter of Adobe Flash, and then point at WebGL as the problemm, I kind of laugh. WebGL is the least of your worries.
Anyway.
Elementary Audio seems to end-run around all that by A) only letting you declaritively define your audio setup, and B) running everything in a single Audio Worklet, thereby bypassing the majority of the Web Audio API.
Yes, managing webaudio nodes is pretty hellish. If you're interested I maintain a small library for abstracting them - basically you pass the library an object describing what oscillators and filters you want, and what timing envelopes you want attached to their params, and then the library creates and destroys everything. It has no concept of plugins or processing, but it makes it convenient to play around with arbitrary configurations of basic nodes.
I couldn't agree with this more. Even beyond the web audio API, we have similar graph APIs in the low level/native C++ world that become unmanageable in their complexity simply due to an inability to compose and manage transitions with ease.
This is exactly one of the difficulties that Elementary aims to address, whether you're running in the browser (where indeed, Elementary uses almost none of the actual web audio api), or natively.
Just a heads-up that I could see both the project name and specifically the domain causing some confusion with the pre-existing ElementaryOS project (whose domain is https://elementary.io).
It's probably helpful to have some background context for the development of this
The author of this library is also the author of the React JUCE renderer. It's a toolkit that re-uses React Native's rendering engine to allow you to use JS and React to render JUCE components:
So at this point, you have a (presumably) C++ JUCE app that is using JS/TS to render the UI. Why not work to move DSP stuff into JS as well, so that you're gluing minimal bits of C++ together?
It's not a far leap from there to here when you follow this line of thinking down the road.
I see several comments along the lines of "who is this even for", which may be classic HN negativity or maybe the docs don't make it clear enough. I've actually played with this library (and many others), and I'm super psyched to see it ship!
To me this is the equivalent of React native for audio: describe your audio declaratively, and let the platform handle the rest. The benefits are:
- Write once, deploy everywhere (building DAW plugins sucks, and most web audio is not performant)
- Lower barrier to learning DSP (yes, DSP engineers already know low level languages that can do this, AND now more people can build audio software)
- Modern dev environment and faster feedback loop. I can now build a React app (or React native) with a high performance audio engine built right in.
I'm pretty skeptical of people defending the status quo approach here... Like, where's the Figma for DAWs? Why don't more more people to write their own VST plugins? And why is audio software so UGLY?
Audio software design has been stuck in the 90's for too long and I'm psyched to see it open up and become more accessible!
And Glicol engine will use LCS to analyse the difference, and avoid updating the whole graph.
Since Glicol uses Rust, WASM and SharedArrayBuffer, there is no GC, and support sample level control.
I haven't checked the source code of your project, but since I know CodePen quite well, it looks like that you need to add SAB, although it requires CORS during deployment.
Also, I think it may need more consideration to promote it with "Write once, run anywhere" at this stage. For example, how about Teensy? For Glicol, although it has POC to show that it can run as VST or on Bela board (https://github.com/chaosprint/glicol), there are still a long way to go (for both of us). I need to have Rust no_std from some of Glicol's dependencies. Yet I don't know how JS can be used in Teensy/STM32. But this is definitely not the end of the world. Sometimes, to have a better scope is actually a good thing for a project. If I would be to promote Elementary Audio, I would call it a Tone.js alternative. It would still be a significant contribution, since Tone.js has been so influential in Web Audio, especially for beginners. How do you think?
I think for Tone.js, the time management and scheduling is not very satisfying, perhaps you can introduce more on this topic for Elementary?
All being said, I think it is very exciting to see a new project in audio community. Good luck and please feel free to discuss more.
I wouldn’t run a full DAW in the browser yet (too spoiled with Logic) though BandLab is having a good go at it. But makes no sense to poo poo those attempts either. Both have their uses (browser based apps and native apps)
Hey Nick! I remember you showcasing this in the audio dev community. It's a paradigm shift in the way audio software is written. Congratulations on your launch and I wish you success!
I find it hard to imagine people the desktop/commercial audio world wanting to write plugins in JS. At a glance, this seems like an interesting webaudio project that is trying to be too much. Plugins are used in a very different situation where low latency and high performance trump pretty much everything. Reconciling that with web dev priorities seems optimistic.
Yeah. I got lost at JS. Buffers and buffer manipulation are everything in audio processing, I don’t see anything stealing C/C++’s thunder here except Rust.
Also, as another user here has already pointed out, the "plugin development kit" is still macos only, though I anticipate releasing the windows version within hopefully a week or two.
Last time I checked, a perpetual JUCE license was priced similar to 3 days from a C++ freelancer. I've never heard it being called out on price before.
I was interested but I took a look at the documentation, and it seems like VST plugins using this can only be compiled for Mac right now. Is that right? It doesn't seem very useful for building plugins if that's the case.
So with this I could make some fun js audio stuff? I dont really get it, why should I use this instead of howler js, or are they even comparable. either way seems pretty interesting and there is a free/lite version
the webpages require a little digging, but they have an examples page.
looks like it's a set of dsp functions where the filter graph is constructed by composing the functions in functional style and then passing it to a render function at the end.
so kinda like big data style composition of functions (like the scala stuff people often write to run on spark clusters) but with dsp/audio building blocks running in the browser instead.
Hey all, Nick from Elementary Audio here. I spent some time in the discussion below yesterday and coming back to it again now I can see this continued to get way more attention than I expected! Thanks for all of your thoughts.
There's a lot of discussion here, and although I'd love to visit every comment in specific I think it makes sense to speak in summary to a few of the major themes/underlying questions here. In general, the discussion here surrounds totally valid concerns that I'll try to cover, and I understand after this thread that my messaging needs some serious work to speak to those concerns.
Before I get to those questions, I want to start with some of the things that Elementary is _not_ well suited for, or some areas where Elementary currently has some limitations.
* It's a young project; there are plenty of standard library features that are not implemented yet (oversampling, several more filter topologies, iFFT, etc). That might mean Elementary is currently ill-suited for your project
* It uses a generalized, block-based graph rendering algorithm. If you have hard performance requirements or need highly specific and hand tuned DSP optimizations, Elementary is currently not ready for you.
* Because of the current rendering algorithm, you cannot specify signal flow graphs that have 1-sample feedback loops.
With that out of the way, let's talk through some of the questions here that seem to underly many of the comments.
1. Why JavaScript?
* It's inarguably one of the most widely used languages in the world
* It's more than fast enough for the role that it plays in Elementary
* It's the language of the web; which isn't going anywhere any time soon, and at the same time the industry is showing us demand for audio software that runs in the web.
* JavaScript/web tech has pioneered the way that we write app UIs over the last decade, and continues to do so
* JavaScript/web tech has pioneered oen of the best developer experience/workflows in software development
2. How can JavaScript possibly be fast enough for this?
* Elementary uses JavaScript only to _describe_ audio processes, not execute them
* All of the actual audio processing is handled in the underlying C++ audio engine
* This engine is compiled to native, platform-specific targets in the appropriate cases (i.e. plugins, cli), and only compiled to WASM when appropriate for running in the browser
* The description of the audio process is done with an extremely lightweight virtual representation for which JavaScript is more than fast enough, and for which garbage collection and seamless integration with application logic and user interface programming is desirable
3. What about latency?
* The core audio engine adds no audio latency overhead whatsoever; if you're running a platform/native target, you're running with the same latency that your drivers/plugin host are delivering to Elementary. Until, of course, you describe an audio graph which introduces new latency; that's on you.
* When compiled to WASM to run in the browser, of course there's additional implications here, but that's a reality of targeting the web in the first place. Elementary doesn't force you to the web, or force you to wasm, but if you want to run in the web, accepting the latency implications yourself for your own apps, Elementary can go there with you
4. What about bottom line performance? Professional, commercial software??
* Elementary will not make the claim that for executing a static graph it will outperform native C/C++ hand-tailored to the application demands. That would be crazy; it's probably fair to say that highly specialized code can always outperform generalized algorithms.
* Elementary will make the claim that for plenty of applications, generalized graph processing is surely fast enough.
* The generalized graph processing algorithm is fast enough because the audio processing is purely native (or compiled to wasm); we do not execute any JavaScript itself on the audio processing thread
5. What about Faust/SuperCollider/PureData/ChucK/Max...?
* I have direct experience with many of these languages, and have reviewed almost all of the others in detail. I think they're excellent pieces of software for prototyping and exploring. I also think they're generally weaker for shipping complete applications.
* Of course you can use Faust/Max Gen or similar to generate C++ and then bolt on your own application harness, and I'd encourage you to try. For non-trivial applications, interfacing between the UI and the DSP becomes cumbersome. Elementary aims to provide a better experience here.
6. Why do any of this at all?
The current dev process and workflow for writing audio software has stagnated; it takes so much time to turn a prototype into a product and we regularly accept complexities and limitations that for so many types of apps we no longer need to accept. Elementary offers a new way of prototyping and shipping those apps more easily and more quickly.
Thanks again for all the discussion, I hope this clears up some of the details.
> The conventional approach to writing audio software is an challenging process requiring expertise in a wide breadth of topics– C++, multi-threading and thread safety, lock-free programming, and realtime thread safety come to quickly to mind, and we haven't even mentioned digital signal processing itself. Moreover, the tooling, workflow, and developer experience that we have in this approach pales in comparison to that of the web software industry, for example. All together, the reality of these complications is that the conventional approach for writing audio software is difficult and time consuming. We believe it doesn't have to be that way.
This is then followed by a screenshot of some code that looks like a SuperCollider script from 15 years ago.
There are at least a dozen audio "scripting" languages that free someone from all the issues mentioned above. Many have been mentioned in the thread already. The only things that Elementary Audio appears to have in its favor are that it leverages JS and it runs in the browser (sort of).
For some people, those two alone will be interesting. Have at it, people!
But for people who write plugins designed to run inside DAWs, or modules inside Rack, and in particular for the people who write DAWs or Rack (or Rack-like things), this is of approximately zero interest. It's a web-centric version of SuperCollider, which is cool but ultimately so bounded by the limitations of the platform that for many things, SuperCollider or something similar remains more powerful and more flexible.