Hacker News new | past | comments | ask | show | jobs | submit | dgranda's comments login

Based on my experience:

A.- I would also add "nofollow, noarchive" tags [1] to your X-Robots-Tag header:

- "nofollow" -> do not to follow (i.e., crawl) any outgoing links on the page.

- "noarchive" -> prevents Google from showing the Cached link for a page.

B.- I would specify in Search Console (former Webmaster Console) how should Google handle "query" parameter [2]

C.- Prevent those spam searches by blocking source IP address, User-Agents, combinations of both, etc.

Good luck!

[1] https://support.google.com/webmasters/answer/79812?hl=en

[2] https://www.google.com/webmasters/tools/crawl-url-parameters...


Thanks for the suggestions!

A) I'm going to add the nofollow and noarchive to see if that helps the issue.

B) I've already set the search console to ignore the query parameter but I'm still getting new spam results coming in.

C) I've been looking into this but so far the meta information for the spam requests is not consistent and it's tricky to identify so far.

Thanks for the help and the luck, I think I'll need it!


I am not sure if it matches what you are looking for, but backtrader [1] is a nice tool for trading strategies, worth a try.

[1] https://www.backtrader.com/


I have tried to set normal Firefox (55) for Android browser to open links by default in private mode but unfortunately I don't know how.

Why "browser.privatebrowsing.autostart" set to "true" does not work on Android version?

PS: I know I can clear everything (cookies, history, etc.) on exit.


With GPG's symmetric encryption ($ gpg -c <filename>) you just need a passphrase (no keys involved)


If you are not an extensive user of plugins, another option to block ads and tracking when browsing the Internet is Brave [1]

[1] https://play.google.com/store/apps/details?id=com.brave.brow...


I use both on free tier and I am happier with PostMark.

I had several issues when sending to Microsoft (hotmail, msn, etc.) domains via MailGun because someone exceeded granted quota for outbound IP address and since then I balance outgoing emails using both.

As mentioned by others, MailGun support doesn't care much about queries from free tier users so I would go for PostMark if I had to pay (more intensive use) or recommend to someone.


There is much better option in some countries if you want to change some law or government policy. For example, in the UK any British citizen or UK resident can create an online petition to be discussed in UK Parliament [1].

If a petition gets 10,000 signatures, the government will respond.

If a petition gets 100,000 signatures, it will be considered for debate in Parliament.

And source code is available on GitHub [2]

[1] https://petition.parliament.uk/

[2] https://github.com/alphagov/e-petitions


I can't recall any instance where this has had any remotely noticeable effect. Can you name an instance?

EDIT#1: Just to be absolutely clear: This is an incredibly transparent distraction ploy. There's no actual intention of acting on any of this.

EDIT#2: See also the recent FCC "public consultation" on Net Neutrality.


(Related to my company) I've spoken with a half dozen government officials in the US and they all said they do not consider petitions in policy making. The biggest problem they have is not knowing easily what fraction of the signers are in their constituency. Beyond this, they see petitions as a marketing ploy by organized groups and hence further discount it.

Btw, interesting article on change.org's challenges with business model: https://www.pastemagazine.com/articles/2017/05/change-dot-ug...


I can concur. In the US, representatives care about 1st, how many people will vote for them in the next election, and 2nd who is paying. That's the extend of their moral and logical debate (if any).


The UK's petition system (https://petition.parliament.uk) specifically collects postcode information to group votes by constituency, and even provides a handy heat map showing % of constituents who voted.


That's a little disingenuous of them. Every petition I've ever seen in the US required not only a name but an address as well, specifically for this purpose.


The challenge for the official is to sort the signatures easily to identify their constituents. Not so easy without zip+4 data (5-digit zip not always accurate on election districts).


Unfortunately, whether or not the petition is debated in Parliament, it won't actually change the law or government policy, because the government doesn't in the slightest bit care about the petitions.


I'm not sure that is totally true, it would be silly to ignore something that got a strong response.


They've debated something like a dozen petitions in Parliament. I'm not aware of anything meaningful having happened as a result.


Like the million people marching against the Iraq war in london 2003?


Marching in the streets sends a much stronger message per person than does signing an online petition.


It had fuck all effect though.


We don't really know what the impact is further down the line, the government has lost a vote over going to war since.


Trump's still been invited on a state visit.


Are people in the UK really signing a petition that amounts to "don't invite the President of the United States to the UK on an official visit"?


I wondered the same, so I looked it up: https://petition.parliament.uk/petitions/171928


Yes, and when 3% of the entire country's population has gone out of their way to vote for that, it's worth serious consideration.

It's often said that one letter written and sent is representative of the opinion of 1,000 people. Perhaps with online petitions it's more like 5.


He was invited before any petition.


The petition was to convert it to a regular foreign visit, rather than a State Visit where he meets the Queen, etc.


Politicians are kings of being silly, then...


In Brazil too[0]. The numbers are different, but if a petition reaches a certain number of signatures our Congress has to lead to the plenary. You could use the official government website for that. Recently we had a Law called 10 mediated against the corruption[1] that followed this script. Unfortunately, our Congress has greatly changed the proposal. I participated actively in this campaign and I am participating in some other proposals that are being built. What I have discovered is that getting in touch with a congressman so that he or she will sponsor the proposal and defend it in its originality is much faster and with a higher success rate. Unfortunately, we do not yet have a possibility to prevent congressmen from altering the public petition and this allows they to transform something positive into something very bad.

[0]https://www12.senado.leg.br/ecidadania/principalmateria (in portuguese) [1]http://www.dezmedidas.mpf.mp.br/campanha/documentos (in portuguese)


The European Union has a similar. A European Citizens Initiative has been available since the 2007 Treaty of Lisbon. If 1 million EU citizens (~0.2% of the population) sign it, the European Commission must respond, maybe with legislation. It hasn't been used much though.

https://en.wikipedia.org/wiki/European_Citizens%27_Initiativ...


can't tell if this is in jest, is "considered for debate" really a better option for bringing about change?

At the end of the day influence (and by proxy money since it buys influence) is pretty much all that matters in changing policy.


The point of physical petitions is they're hard. Showing a politician a few thousand names, with verified voter-registration statuses, who wrote out their addresses, and maybe copied a single sentence expressing their support for your cause, is much stronger than a million people who clicked an Internet button. The former can be organized to the politicians benefit (or against him, to his detriment). The latter probably can't.


I would tend to take the cynical view that the point of petitions in general is that it lets people feel like they're making making their voice heard and acting as a pressure relief valve of sorts for all but the most serious of grievances. Most protests today fulfill the same purpose.


Have you been involved in a legislative process? (I haven't, directly, at the federal level.) At the local and state levels, hard petitions are huge. (In some jurisdictions, they're required to get on the ballot.) Protests are meaningful inasmuch as they are willing to show up, time and again.

Why? Off-season elections, which most of these politicos must win to keep office, are not games for numbers. They're games to motivate people to inconvenience themselves by voting.

Remember why politicians like PAC and campaign contributions. They spend that money on turning out voters. If you can turn out a bloc of potential voters, or people whom you can reasonably claim are upset enough to vote against them, you'll catch attention. The active minority wins against the disinterested majority. Internet petitions represent the latter; hard petitions and protests high in repeat attendance demonstrate the former.


I am sure it is much more effective (and expensive) to lobby, but regular citizens can't afford it. There is a long way to go and "considered for debate" is just the first step.

Change.org only guarantees that all data you have provided (including what you are not aware of) will be sold to highest bidder.


The US has something similar. The issue is that (even in the UK) the response can be "no" and the consideration can also be "no".


The US used to have something similar. The site still exists, but under Trump the petitions are no longer reviewed. You can't take democracy for granted.


I don't think there is a single instance of a petition leading to legislation, though.


Not a very good idea to believe a government controlled platform for something like this.


Not sure for-profit is appropriate either though (even if it's a b-corp, as Change.org is).


What are your reservations about a B-corp?

Nonprofit legal status is hardly a guarantee of anything about corporate behaviour, aside from what paperwork they fill out.

To be clear, I'm actually curious about it, especially because Bs are becoming mildly trendy, and I don't know that much about them.


I've got nothing against b-corps. In fact, I think they should be the default corporate form rather than one you need to opt into with specific language in the articles of incorporation (as in Delaware). But b-corps are still owned by investors. They are legally entitled to take into account considerations other than maximizing shareholder return. Whether they do or not is a separate matter; they are not obligated (although this can depend on state and type of entity, I believe). They receive funding as investments (with expectation of returns) rather than donations. Again, no problem with that as such. But if my choice is between government operated petition site and a petition site accountable to private investors (even if a b-corp), I wouldn't reflexively favor the latter.


I fail to see how the UK's policy is better. For starters, its a government policy which can be revoked at will. Change.org is a for-profit entity whose objectives are clear and cuts across government and private enterprise.


> Change.org is a for-profit entity

Wikipedia claims that in summary information, but the text contradicts it; I suspect the people saying that (including Wikipedia) are making the error of failing to recognize that "for-profit", "not-for-profit", and "nonprofit" are three different categories (not two, with the last two being aliases), and that you can't distinguish between the first and second categories by just looking at the legal formalities that identify the third category.


It's not actually. It's decided by parliament, which is not the same as the government. It specifically exists in order to challenge the government.


Work hard. Be strong. Don't complain.


I used to have delivery problems (throttling errors caused by another customer) sending messages to Microsoft related domains (hotmail, msn, live, etc.) when using MailGun.

There was no progress at all when discussing this topic with MailGun's support so I created another account at PostMark [1] to choose provider depending on recipient's domain. No more problems when sending messages to Microsoft related domains.

Anyway, "silently discarded" sounds like you need to put in place DMARC reports [2] to troubleshoot what is going on there.

PS: I chose PostMark because I like its online documentation. Currently a happy customer, not affiliate at all.

[1] https://postmarkapp.com/ [2] https://dmarc.postmarkapp.com/faq/


I run https://www.fortsu.es (also https://www.fortsu.co.uk, https://www.fortsu.de and https://www.fortsu.com) a price comparison website for running shoes. Original one is focused on spanish market while expanding into interesting ones.

It started as side project some years ago when I wanted to buy running shoes online and it has been improved over the time. To-Do list never ends ;)


Very nice! Couple of questions if you could answer please. How do you host your site? How do you compare the similar products and aggregate them within your database? Unique identifies etc.


Thanks!

> How do you host your site?

TL;DR: entry level Virtual Private Server (VPS) per country.

Based on some SEO articles, I decided to go for Top Level Domains depending on target market (.es for Spain, .de for Germany, etc.) so I needed properly geolocated IP addresses for each country. Each server is around 10 USD/month making this approach affordable. I do all operations.

> How do you compare the similar products and aggregate them within your database? Unique identifies etc.

Each product is manually added (using self made web based template) into database basically because of custom description, picture and specifications. Similar products are linked based first on category and then using properties like weight and drop.


This is good advice. I'm going to be building a site and replicating across TLD and IPs like this across countries. My only question is, if you are using the same content (ie, you write content on the blog for SEO) across each site do you run into trouble with Google for copying existing content?


I am sorry I can't provide a reliable answer to this because my english based sites need more traction, but Google (GWT) does not complain as long as hreflang attributes are properly set.


I'm interested in this kind of projects I have a lot of experience working with scrapers. How do you earn money from them? via affiliate sales?


Yes, affiliate marketing is the main source of income.

I am slightly introducing kind of business inteligence in order to sell reporting to merchants willing to know more.


I am interested in building something like this for Android phone comparison.

Can you give some indication of stats: visitors, income etc?


Regarding Android mobile phones, check https://www.kimovil.com

Numbers change much from country to country but average conversion rate is around 2,3%


How do you market this? Is it mostly SEO or ads as well? Must be expensive if you want to compete with Nike and Adidas for clicks...


At the beginning it was basically word of mouth and niche related forums on the internet.

Then I started reading about SEO and advertising. Organic search more or less work but I got almost no traffic from a couple of banners on related pages during few months. I didn't try advertising networks like AdWords.

I don't see big brands as my competition. I have partnered with some but I don't think they sell much on the internet (typically higher price tags) compared to full equipped city centre stores.


Very cool, I've been thinking of doing an affiliate website for a while. But it seems like it could take a lot of time and patience to get it profitable. Thanks for sharing!


The sooner you start it, the quicker you see some results ;)


To get the prices, do you go and copy them manually or have you written some crawlers?


Price data is retrieved from datafeeds parsing, triggering requests against APIs like Amazon [1] and using custom crawling.

[1] https://affiliate-program.amazon.com/gp/advertising/api/deta...


i'd recommend adding sitewide hreflang across all your ccTLDs to the list ;)


hreflang attributes are in place, aren't they?

It took some time to get them properly in Google Webmaster Tools 0:)


At a glance, I can only see them in place on your respective homepages, where it only has limited value.

It should ideally be across all pages of all sites, so as to declare all 'sibling' pages, e.g.

https://www.fortsu.co.uk/running-shoes/all/trail

https://www.fortsu.de/laufschuhe/alle/trail

https://www.fortsu.com/running-shoes/all/trail

you can also use XML sitemaps to implement it, which tends to be preferable, especially if you don't want the extra bloat in the HTML of your pages.


I will double check category pages (those you point out) but I am sure hreflang attributes are in place for product pages ;)

I will check the XML sitemap stuff, thanks for the tip.


Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: