A few years ago when I was a Junior engineer, I worked for a company which would provide us with free tickets to any sports team in our city. I'm not a sports fan, but my stepdad loves hockey. When the tickets went up, they got reserved pretty fast.
I noticed that the site we used to reserve the tickets had a predictable slug like: `{company}-hockey-tickets-2016-season`, and `{company}-hockey-tickets-2015-season`. The sites also didn't change between each refresh.
So, when it was nearly time for the 2017 season to start, I wrote a script which hashed a GET request of the site at `{company}-hockey-tickets-2017-season`. If the hash changed, it would send me a text message using twilio, and I would know to immediately get on a machine and reserve the tickets we wanted. After a couple false positives, the page eventually went up before it was announced and I reserved the tickets I wanted.
Unfortunately, the office manager who did these things told me I couldn't jump the gun and un-reserved my tickets. After the official announcement went up, the spots I had originally reserved quickly went to someone else
It seems like bad company policy to let a bunch of techies fight over a limited resource via web requests. It's like network inspector gladiators. Maybe that was the point? Lol
I assume they meant they were hashing the response rather than hashing the request. Then you don't have to keep the entire response around for comparison, only its hash.
I think mhb's point is that the size of the request content is so small, it can be compared byte-for-byte directly instead of hashing and comparing hashes. Many of us have a kind of muscle memory where we always hash everything for comparison, but for small data there is not really a point.
Good stuff. Niche idea for ya: consider government contractors. Gov sites tend to be atrocious. Manual change tracking can mean you find a $1M RFP weeks earlier, which increases chance of winning. (Consider mda.mil. There's no email signup. What they consider a 'newsletter' is in fact a list of PDFs hosted on their site.)
And that's federal gov contracting. State/local is even more of a Wild West. So much so that many, maany companies just never have the bandwidth to even find RFPs and navigate those ancient sites.
While that may be true if contractors/competitors for contracts were generally unaware of the contract until posting, that’s usually not the reality. A lot of government contracting advice from experienced contractors is that if you’re discovering it via a public source (i.e. the website) you’re too late.
This I am afraid is not true either, though. I know many dual-use companies who discovered an RFP (not just on SAM), late -- i.e. with a proposal deathmarch -- and got through.
My understanding is that those scenarios are the exception, and finding obscure contracts that have gone undiscovered isn’t really an untapped fount of free, missed, opportunity.
That said, aggregators like bonfire etc. can get you pretty far if that’s your strategy!
Yeah it's probably true that such things are the exception. But for a startup, a chance at e.g. a $256k SBIR Phase I and then $1.9M Phase II and the modern follow-on options in e.g. the Army (called the CATALYST program and other things) -- this one exception turns the company into something real.
So because the amounts are large (compared to commercial, in terms of when a customer first buys from a small company), those cases really make a huge difference.
I have to counter yet again, apologies. :) SBIRs yes are considered grants, but I've seen the same phenomenon for e.g. CRADAs and OTAs. Functionally this is all similar: we find them often on bad gov websites, and an automated tool to track changes can greatly help many companies to win them.
How can you be "too late?" Isn't the point of RFP postings that they're the first step in a process of soliciting proposals (bids) for the requested project? They're certainly not supposed to be granted on any kind of first-come, first-served basis... combatting that sort of corrupt insider dealing is the whole reason there are legal requirements to post them in the first place. So while I don't doubt the premise of your cynical truism that "if you're discovering it via a public source you're too late," I hope that this is a separate issue from the RFP process itself, because any project where that's the case must be a project where the RFP was meaningless in the first place. And that behavior is also explicitly illegal, even if it's unfortunately prevalent and infrequently prosecuted.
I'm OP. I'm not the developer of the project, but I think it deserved a post on HN because it's pretty an useful tool to track website changes. Setup can be done in a few minutes and the lead developer is always looking for solid feedbacks.
It's actually refreshing to see end-user documentation and answering the basic questions of “what can this do” and “why should I care” in the readme. Most projects jump straight into how to build it locally, with not even a screenshot of what they’ve created.
They should add an url into the description text. This will render as backlink any there are many pages out there which replicate github-topic pages which will then give your site many links for free.
I've used https://urlwatch.readthedocs.io but this definitely seems much easier to use—though possibly not quite as powerful regarding page filtering. But at least ChangeDetection supports jq which is already quite a nice feature in that department.
On the theme of alternatives, I've been using https://www.followthatpage.com/ on and off for years. In particular for my partner's job hunt, watching some local trade association billboards or single school's `/jobs.html` pages.
Also has a reasonable free plan for personal use (20 pages daily + 20 pages weekly + 1 page hourly).
Definitely much simpler in terms of diffing (only text), but it has this 2000s vibe.
You can install it for free on your machine. I think it's a decent price, considering that they offer you to check a website even with a ratio of seconds.
Interesting! I've been making some scripts privately for specific tasks, like an Nvidia video card I wanted to buy. However many sites are really hostile to scraping these days. I'll give it a try,.
This is interesting. I did not log into it for ages but sure still get the alerts for my existing trackers. That's why I assumed that they are still around.
I noticed that the site we used to reserve the tickets had a predictable slug like: `{company}-hockey-tickets-2016-season`, and `{company}-hockey-tickets-2015-season`. The sites also didn't change between each refresh.
So, when it was nearly time for the 2017 season to start, I wrote a script which hashed a GET request of the site at `{company}-hockey-tickets-2017-season`. If the hash changed, it would send me a text message using twilio, and I would know to immediately get on a machine and reserve the tickets we wanted. After a couple false positives, the page eventually went up before it was announced and I reserved the tickets I wanted.
Unfortunately, the office manager who did these things told me I couldn't jump the gun and un-reserved my tickets. After the official announcement went up, the spots I had originally reserved quickly went to someone else