Very happy to see the Global Explorer ROV still out there, hard at work. This vehicle was Chris Nicholson's baby and it looks like the crew at Oceaneering are keeping his legacy going.
Chris was one of the early inventors / developers of ROV technology and he was famous in the ocean engineering world. He passed away suddenly in 2015.
Alternatively, sed script below will give you just the article html, images, and embedded youtube along with all the json data enclosed in <pre></pre> tags.
I'm not entirely sure I understand what is happening, I have some grasp of the aim/purpose. Also have pulled pages using CURL before. I'm just curious, does this mean you have to analyze their page (like console) and find the links like the ones you posted above for cloudfront,etc... and then assemble it so you can format it? Not my field but I use ublock/adblock plus and it's not enough all the time, some videos have an overlay which I swear has a double-click counter where you have to open 2 ads before you can actually push the play button to play the video.
It is interesting to grab data and package it yourself through your own reader but I wonder if it loses the site's design/feel... but you're probably just after the information anyway.
"... I use adblock plus and it's not enough all the time..."
Next time this happens it could be useful to make a submission to HN about it.
We might be able to identify and/or solve the problem.
I do not use an ad blocker nor do I use python or youtube-dl yet I never see any ads and I download all videos before watching them. Indeed I access the web for the information not the inconsistent design/feel.
What I posted above illustrates examples of two alternative approaches:
1. Block sources of undesired resources: ads, tracking, etc.
The "links" listed are domains used for undesired resources.
2. Only make requests to sources of desired resources: the article and its accompanying images and video. The script extracts only what we want from the html page.
I use approach #2 more than #1.
As far as I know ad blockers use blocking, #1, exclusively. They need to maintain a list of domains to block.
I wasn't sure if you were serious about posting to HN about adblock haha, I complain enough as it is about my pathetic life.
In the case of the videos, they're sourced from other domains, it's funny the site's argument is "Google does this too technically so why are we different?" But I think the embedded iFrame's that contain the video players have their own ads. I don't know, it's hard to read JS code when it's minfied (un-minify it).
If you are serious and you want a script like this for any given page that you want de-crufted and reformatted, let me know. This applies to any HN reader.
I make these for myself all the time.
I do one-offs in sed and if it is something I will reuse I redo it in C using flex.
If you want something generalized to all web pages that removes google analytics, doubleclick, scorecardresearch, etc. of course that is possible too. I am happy to make this for anyone who wants it.
Another solution is to run a proxy on your computer that filters out garbage. Such proxies used to be more popular when advertising began to take over the web, but most of these projects seem to have been abandoned and forgotten (junkbusters, etc.).
Nowadays it seems like people try to accomplish this via Javascript running in the browser.
Turning off Javascript is also highly effective at disabling advertising. But note this may not stop things like img src tags pointing at tracking pixels, malicious iframes, and other elements that graphical browsers will load automatically.
Also, tcp clients do not normally store or send cookies. And even if they try to provide this functionality, e.g., curl or wget, unlike a graphical web browser, cookies are off by default. Consider the case where the user retrieves the page with a tcp client and then views it in a browser.
For each use case that is not a free browsing I create an electron app, that never executes any code from the web or uses any external style. It only uses XHR to fetch html pages/json data/other static stuff and then transforms that data and uses it in the custom UI designed for the use case.
> Another solution is to run a proxy on your computer that filters out garbage.
That's what I was thinking; a proxy server that can run on a pi. Plug it into the wall, configure wifi, connect to it, and bam - instant safety. (Assuming you trust the code.)
