The overall article is extremely clearly written and well-diagrammed, as you'd expect from someone who writes software for test/metrology equipment. It does look pretty different from your usual web-dev retrospective, and shares more in common with "$BROWSER Internals" style articles.
The thesis:
> I recently spent some time building a browser-based signal analyzer (spectrogram + oscilloscope) as part of one of my projects. I ended up using some very modern browser APIs and technologies that I'd not worked with before, and I discovered a lot of really interesting patterns and techniques that I'd never seen before in a web app.
> Building this application has made it clear to me that the modern web is extremely well-suited to building complex multi-threaded and graphics-intensive applications.
Firefox is lagging behind on a lot of the more esoteric web APIs. I think it's a choice by the Firefox team, and a valid one given the effort required, Mozilla's relatively limited resources, and how few devs actually want or need these APIs. Firefox is more like a "website viewer" than a "web app host" like Chrome and Safari. I think that's what a lot of Firefox users want, and why they choose that browser. It might even be useful to differentiate between them instead of calling everything a 'web browser' in the future.
To that end I don't think it's a fair criticism of web app developers or Firefox that apps like this don't work in your browser.
The specs that firefox chooses not to implement are stuff like web midi for security concerns. Firefox does not try to market itself as a "website viewer" that does not support "web apps"
I can't speak on the specific details of web midi, but after the spate of poorly designed browser features with intrinsic security vulnerabilities (eg canvas fingerprinting), it's reasonable to be wary of things coming down the pipeline.
That always makes good sense. I think Chrome got this one right though, have a reasonable protection against abuse and trust that the end user is capable of weighing the risks.
The issue around fingerprinting is secondary because it isn't a security issue but a privacy issue, and this is a problem anyway for which there isn't a really good technical solution.
The security issues that were mentioned (someone bricking synths on purpose via SysEx messages) has afaik never actually happened.
This works fine for me (Firefox, Ubuntu, tracking protection set to max, uBlock origin on, privacy badger on, resistFingerprinting on). You need to click the play button to make it go, of course, but then it just works.
Maybe there's a setting you've disabled previously and have since forgotten about?
Hmm it should be working in Firefox - as long as you're using the latest version instead of ESR it should be good; works for me on Firefox 113.0 on Linux.
Some browsers have anti-autoplay-ad protections built in that prevent audio from playing until you interact with the page. You might have to click the page or press a key on the keyboard to get it started.
FYI, it supports decoding ultrasound wav files. I use a Dodotronic UM250K mic to capture ultrasound up to 125 kHz. LCD monitor and other electronics emits some interesting sounds. For some reason, getUserMedia doesn't allow high sample rates, but ffmpeg works:
That Webfft is fun. Try recording yourself making a few long vowel sounds A I U E O AE UH , and take a look at how the harmonics/formants/ the pattern of peeks at different frequencies is distributed between sounds.
Being able to run an FFT on audio signal in real time in a background thread seems really cool. Not sure how feasible it is but could be the basis for speech recognition engine running in browser (I'm not expert but I know Speech detection machine learning models are trained on range of FFT derived signals) - if you can pipe the FFT output to ML endpoint you could potentially have speech to text generation running in the browser - or something like automatic closed caption generation from an incoming audio signal.
Made by the same guy who created the popular llama.cpp LLM library. The model uses log-mel spectrograms as input.
Using modern algorithms, FFT is actually really fast to compute. Definitely dwarfed by the evaluation of the model itself, even when using many threads and Wasm SIMD.
Yep looks like it - wasn't expecting browsers to have it baked in. I'm not all that familiar with what is available in the world of audio.
I've done some rudimentary signal analysis on other types of sensors - accelerometer (for detecting vibrations of specific frequency) and millivolt probes on a hardware instrument - I've used stuff like Matlab or Python to do the signal analysis.
I think the biggest problem with SharedArrayBuffer is that you need to set the Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy headers for it to work at all (due to Spectre), which means you need a host or a web server where you can configure that.
I mean… Pro Tools, Reaper, Ableton, Logic, Cubase… these all run just fine without an RTOS. But you did say “audiophile” specifically and not “audio professional” so you might be right, but it likely doesn’t matter in practice.
Does your audio input device run an RTOS? Probably. Audio output device? Also likely. Does the intermediate processing need to be an RTOS? Or does it just need to be able to operate fast enough to consume the input buffers and keep the output buffers from starving?
The thesis:
> I recently spent some time building a browser-based signal analyzer (spectrogram + oscilloscope) as part of one of my projects. I ended up using some very modern browser APIs and technologies that I'd not worked with before, and I discovered a lot of really interesting patterns and techniques that I'd never seen before in a web app.
> Building this application has made it clear to me that the modern web is extremely well-suited to building complex multi-threaded and graphics-intensive applications.