I have a truly naive question about the distributed web: what makes the supporters of it think it will be any different from the original web. I mean, isn't it likely that at some point, there will be the need for a centralized search engine for it? Isn't it unavoidable that big companies like facebook runs their own non-distributed subnetwork, so that it can deliver standard functionality to all its users? The original web IS distributed already, isn't it? It's just that organically, the way people use it has become a lot more centralized, no? Or am I missing the main argument for a distributed architecture?
The difference is IPFS (or similar) would distribute content. Currently communication is distributed, but content is not. IPFS is like https, but where every page is a bittorrent. There are challenges for security in distributing services and monetization. However, the key is that content would be distributed. YouTube probably would not switch to ipfs, but DNS (what was targeted recently) would make sense. A DNS host list should be more distributed. Even with DNS there is a dynamic content challenge. When a DNS entry changes, it would still need to propagate.
This brings it back to a distributed communication problem. If you could distribute content more consistently and 100,000 computers could step in as dns providers with authentic data rather than concentrating dns at dyn or Google it would help. That will still require protocol updates so that the fail over uses local content. Really ipfs and distributed content could be a component to help with distribution. It definitely isn't a complete solution alone (yet -- there is a push to make it the solution).
With the web going more mobile with less storage space. Why would anyone would want to be a node in such a network where you have to cache content for others.
Well apart from battery, If I was an ISP, I´d like it if a Youtube movie would flow from one persons mobile to the hotspot of the train said person is sitting on and on towards another person on the train. This scenario would require significantly less bandwidth overall. Same goes for neighbors accessing the same information. And this can be extrapolated to many situations.
The IPFS powered internet would perhaps lead to more uploads but it would scale significantly better and will be cheaper to maintain. Hence cost can also go down for end users/consumers.
... and that's pretty much the deal killer for this on mobile, even ignoring everything else.
> If I was an ISP, I´d like it if a Youtube movie
... except you are not allowed to download videos from YouTube or most (if not all) the popular video content services.
> a Youtube movie would flow from one persons mobile to the hotspot of the train said person is sitting on and on towards another person on the train.
This would only work as long as person A is running IPFS, connected to the hotspot and has any cached content somebody else is concurrently interested in. The hotspot is very unlikely to run IPFS and to have any storage, so cache hit ratios would not only be low they would be dependent on person A's transit schedule.
> This scenario would require significantly less bandwidth overall.
No, the same amount of bandwidth would be consumed, but perhaps over a cheaper radio bearer.
> The IPFS powered internet would perhaps lead to more uploads but it would scale significantly better and will be cheaper to maintain.
Somehow I doubt that. Total costs would most likely go up, but they would perhaps be more spread out.
> Hence cost can also go down for end users/consumers.
Consumers don't spend anything for accessing content on the Internet, so I really doubt there are any cost savings to be had.
> ... and that's pretty much the deal killer for this on mobile, even ignoring everything else.
It is, for now, indeed.
> Except you are not allowed to download videos from YouTube
I think youtube would be interested in some load balancing if they could keep the income from commercials. Wait, what, there wouldn't be a need for YouTube. We'd only need a way to pay content creators build into the system (Ethereum coupling somehow?? I'm not sure, can views be tracked in IPFS?).
> This scenario would require significantly less bandwidth overall.
The same amount of bits are pumped around but they have to cover significantly less physical distance and hubs. This reduces bandwidth.
> Somehow I doubt that. Total costs would most likely go up, but they would perhaps be more spread out.
Me pumping bits from my neighbor's house to mine instead of us both via the backbone from some server in a central location requires less (expensive) infrastructure between me and that central location.
> Consumers don't spend anything for accessing content on the Internet
indirectly they (we) pay for the copper and the fiber.
> I think youtube would be interested in some load balancing if they could keep the income from commercials.
Firstly, it's not up to Google. Secondly, how would that even work?
> Wait, what, there wouldn't be a need for YouTube.
Yes, there would. YouTube isn't the solution to a technical problem.
> We'd only need a way to pay content creators build into the system
Why?
That's also a pretty big "only".
> The same amount of bits are pumped around but they have to cover significantly less physical distance and hubs. This reduces bandwidth
No, it doesn't. What it reduces is bitmiles. Which may or may not be significant. Mostly not, but it may have some significant if we can lower the usage of some scarce and expensive radio bearer.
> Me pumping bits from my neighbor's house to mine instead of us both via the backbone from some server in a central location requires less (expensive) infrastructure between me and that central location.
Not really. You are still going to need that infrastucture, so no cost savings.
> indirectly they (we) pay for the copper and the fiber.
So we do, but using IPFS isn't going to result in us getting a check in the mail. Costs are still zero and savings likewise.
> except you are not allowed to download videos from YouTube or most (if not all) the popular video content services.
If you're not allowed to download videos, then how come they show up on my screen? That information wasn't on my device before I pressed the play button.
Streaming _is_ downloading. It's just downloading _continuously_ as the content is presented. Whether or not you seed is an entirely separate issue and has nothing to do with streaming vs downloading.
No, it is not. Not from a legal perspective, which is all that matters in this context. With seeding you are making things worse for yourself, as you are not merely downloading but also distributing. Seeding is also relevant with regards to downloading, as IPFS automatically starts seeding what you download.
Depends, I'm wondering if it could be cheaper energy wise if everybody was tapping into each other phones nearby instead of reaching for the cell tower.
By allowing your phone to be used as a server, other people will let you use their phones as a server, so you can download faster. If this sharing doesn't emerge naturally, then it can be directly incentivized with ratio tracking (like private torrent trackers), or even with money like the Karma WiFi service.
Also, I have WiFi on always and my battery lasts all day, so I doubt this will be a problem.
Having wifi on all day is one thing, actually using it is totally different. Just activate your wifi hotspot on your phone and start using it from your computer. See how fast the battery depletes.
No matter what incentive scheme you device , it's not going to help you with your battery.
> No matter what incentive scheme you [devise], it's not going to help you with your battery.
Obviously with certain incentives, battery would become irrelevant. If this hypothetical "IPFS ratio" were as valuable as (for example) what.cd ratio, I'd have no problem carrying around car battery to continue seeding. Thankfully, I don't think it would be possible for my phone to consume that much power in a day.
> Obviously with certain incentives, battery would become irrelevant.
Sure, cold hard cash would suffice, but imaginary Internet points won't do. But even with cash there comes a point where you would turn off IPFS, because the alternative would be to have a dead phone for the rest of the day.
> Thankfully, I don't think it would be possible for my phone to consume that much power in a day.
Perhaps not that much battery, but it is quite easy to deplete your battery in much less than a day by having your hotspot on constantly, which is what IPFS would require.
Let’s not forget the context here. We’re talking over-WiFi (not cellular) opportunistic sharing between two phones running an IPFS node.
If you have anything to share at this point this is probably a good indication that both clients have some content in common. So first of all, transfers over WiFi while the phone is in use anyway are not that expensive, and more importantly there’s an opportunity to save a lot of battery, and data, that results in substantial net positive gain.
To illustrate imagine the likely scenario of two people using some form of Youtube with a distributed IPFS cache while on a plane. If those two people have anything in common—and certainly there are things like news etc. that are always in common, those can be shared instantly over the WiFi.
So for an insignificant battery investment, you can potentially save a lot of data and modem power in the long run.
Plus a plane, bus, or train could easily have 50 people within range. Assuming everyone has 8GB of cache space on their phones, you could access about 0.4TB of content directly. With a mesh network that supports packet forwarding, you could connect with everyone on the plane, meaning 100-700 people, which would give you 0.8-5.6TB of content.
Obviously content would be highly duplicated between peers, but you'd have access to a significant chunk of the "popular" internet, and would be able to avoid in-flight WiFi.
For such a mesh network, you don't even necessarily need to connect over WiFi - lower energy Bluetooth could be used too.
> Assuming everyone has 8GB of cache space on their phones
That feels optimistic. I doubt people have that much free space on their phones, especially those that don't have flash cards. Heck, a basic iPhone won't have. Even a 16GB model has something like only 12 GB free when it's empty.
> For such a mesh network, you don't even necessarily need to connect over WiFi - lower energy Bluetooth could be used too.
IPFS doesn't (currently) work over Bluetooth.
However, my main point is, that I don't think most people would be altruistic enough to participate unless they could plug in their phones on the plane.
> So for an insignificant battery investment, you can potentially save a lot of data and modem power in the long run.
Having run many a phone into the ground while hotspotting, I do not consider the battery investment insignificant. It's annoying enough when I drain my own battery when using the hotspot, I would never allow complete strangers to do that to me.
Isn't that the direct antithesis of IPFS? Might as well just Airdrop whatever you want to share with your buddy if you are going to require a rendezvous mechanism.
Opportunistic as in piggyback on active WiFi connection and non-idle state.
P2P networks like Bittorrent pride themselves on being available despite high churn, so this is right up their alley.
The network works even if people only seed during a download. Obviously, that’s less than ideal. For mobile, though, this is perfectly fine.
The point of IPFS is you can do an ‘airdrop’ without having to coordinate it. You lookup content by hash— If it’s available locally? Great! If not just get it from remote and add to cache. Simple as that.
> Opportunistic as in piggyback on active WiFi connection and non-idle state.
That's a reasonable approach, but due to the synchronous requirement I guess the offload factor would be minimal. When I want to download something I might as well enable wifi and see if it's available locally. After I'm done there's really no incentive for me to keep seeding or even have wifi turned on.
> P2P networks like Bittorrent pride themselves on being available despite high churn, so this is right up their alley.
This really isn't applicable to opportunistic mobile use of IPFS, as you are unlikely to have a large enough local swarm to guarantee uptime or availability.
> This really isn't applicable to opportunistic mobile use of IPFS, as you are unlikely to have a large enough local swarm to guarantee uptime or availability.
IPFS content is chunked, your swarm is lan+wan.
> That's a reasonable approach, but due to the synchronous requirement I guess the offload factor would be minimal.
A room full of people using their phones is what I have in mind. Remember, a node’s cache is not exclusive to just the content that’s being currently consumed. Anyway, the point was whatever’s the offload factor it would still be a net positive.
Also one can imagine if IPFS gets popular, just as ISPs collocate CDN caches in the present model, WiFi routers could come with a local node of their own. The “Web Accelerator.” And now we’re talking!
There’s opportunity for caching at many network levels, but there’s no incentive to do so with HTTP.
Yes, but for offload, all that matters is your local lan swarm.
> A room full of people using their phones is what I have in mind.
That might work to some extent, but how often do you (i) spend your time in rooms full of people and (ii) have no wifi.
> Anyway, the point was whatever’s the offload factor it would still be a net positive.
Not quite. If the offload factor is sufficiently small there is no rational reason to burn battery on it.
> Also one can imagine if IPFS gets popular, just as ISPs collocate CDN caches in the present model, WiFi routers could come with a local node of their own. The “Web Accelerator.” And now we’re talking!
Possible, but unlikely. Web cache hit ratios are abysmal, normally it's much cheaper to just up the bandwidth. A lot of business models would also have to change for the (large) and interesting content to become third party cacheable.
> There’s opportunity for caching at many network levels, but there’s no incentive to do so with HTTP.
There are also many business models where there is no incentive to cache either.
> If the offload factor is sufficiently small there is no rational reason to burn battery on it.
Sure, but there’s also no cost to it, if you go with the piggyback/opportunistic approach. That’s the point. It’s either net positive or just neutral. Whether the local offload is effective or not— that’s a different argument.
Anyway, the driving reason for my interest in IPFS is re-decentralization of the Web. If it can, theoretically, save some data on mobile—so much better.
What it needs, though, is simply work on mobile at least as well as HTTP. There’s no doubt IPFS has no fundamental architectural problems in that regard.
Though, full disclosure, it does have some major implementation issues at the moment.
The idea is that companies or people backing the content could easily provision more nodes to meet the demand. To create an IPFS "host", its super simple:
The internet may run on content (storage) and communication, but both those things run on computation. Until computation can be safely distributed real changes can't happen
> In which way is IPFS immune to content takedowns?
In the way that, if one node is forced to take down some information, that information may still be accessible from other nodes under the same address as before. It's not quite immunity: They can still harrass individuals. Call it "resilience", maybe.
> IPFS provides no anonymity, so it won't take long before DMCA notices to start arriving.
Because mirroring content on IPFS is trivial and transparent to the consumer. All it takes is a single user (or an arbitrary number) outside of US jurisdiction and the content is more or less DMCA-immune.
The equivalent is not practical in traditional HTTP, where content at risk of being taken down is scraped from the server and that snapshot-in-time is hosted as a mirror, generally at a different domain.
> Because mirroring content on IPFS is trivial and transparent to the consumer.
Which is also why a regular consumer should never use IPFS and why prosumer should immediately disable caching and seeding upon install.
> All it takes is a single user (or an arbitrary number) outside of US jurisdiction and the content is more or less DMCA-immune.
Perhaps so, but if your plan for resiliency is based on the kindness of strangers in countries where the DMCA nor censorship does not apply, it's not much of a plan.
Plus you'll still have to seed the original which opens you up to liability, as IPFS does not provide any kind of anonymity.
So, while better than traditional HTTP, IPFS isn't really immune to takedowns nor very resilient.
>Which is also why a regular consumer should never use IPFS and why prosumer should immediately disable caching and seeding upon install.
Unless they changed policy, seeding is strictly a manual, opt-in process.
>Perhaps so, but if your plan for resiliency is based on the kindness of strangers in countries where the DMCA nor censorship does not apply, it's not much of a plan.
I disagree, IPFS is a pretty reasonable plan for resiliency in the case of static web content, as previously discussed.
If instead of "kindness of strangers" you frame it as a "market with incentives" to maintain information availability it is both less condescending and more accurate to how situations are likely to play out with things like DMCA'd content.
>So, while better than traditional HTTP
(which is all it needs to be)
>IPFS isn't really immune to takedowns nor very resilient.
That wasn't the goalpost we originally set, nor one of the project's longterm objectives, so I'm not sure the relevance.
> Unless they changed policy, seeding is strictly a manual, opt-in process.
Pinning may be manual, but is not content automatically cached and seeded (until purged from the cache) once any content is retrieved?
> I disagree, IPFS is a pretty reasonable plan for resiliency in the case of static web content, as previously discussed.
How is it a reasonable plan to exploit the naive and the uninformed or to depend only on those that are outside your juridistiction and unassailable by their own?
> If instead of "kindness of strangers" you frame it as a "market with incentives" to maintain information availability it is both less condescending and more accurate to how situations are likely to play out with things like DMCA'd content.
What incentives exactly would those be?
I also find it interesting that you object to kindness from strangers.
>>So, while better than traditional HTTP
> (which is all it needs to be)
That might not be sufficient for IPFS to get enough adoption, tho.
>>IPFS isn't really immune to takedowns nor very resilient.
> That wasn't the goalpost we originally set, nor one of the project's longterm objectives, so I'm not sure the relevance.
Fair enough. It is however relevant in the sense that it both removes a use case and acts as a disincentive for users to participate, as there is no liability shielding.
> Perhaps so, but if your plan for resiliency is based on the kindness of strangers in countries where the DMCA nor censorship does not apply, it's not much of a plan.
Torrents work for content that is heavily seeded, but you also have a lot of content that has ~0-5 seeders with very bad bandwidth.
I don't see how IPFS tackles the problems of guaranteed availability and enough redundancy while relying on volunteers to mirror and serve content.
Also, if you want to DDoS content, it sounds like all you need to do is find the IPs of the nodes that host the content you want to target (which might not be many, and I guess become visible to you once you access/download that content) and knock them out
It's not like the grandparent is advocating DDOSing, he is just pointing out that doing so is trivial for less popular content. This is something that IPFS doesn't address, but that their marketing fluff implies would IPFS would take care of.
Good luck adding all DDOS bot IPs to your null route.
I've come to the conclusion compute needs to be separated from content and users using a new non-financial based business model. That's the only way to ensure game theory based models don't fuck up the infra on which everything depends.
I am a novice, but I'll do my best to answer. IPFS isn't a replacement for the existing web like many of its predecessors, for the purposes of your question, it really works more like a drop-in shared caching system. The site host doesn't have to participate in IPFS for this to work.
As an example: You host a blog. As an IPFS user, I surf to the blog and store your content in my cache. When another IPFS user attempts to access your blog they may pull directly from my cached version, or from the original host (depending which is fastest). Merkle DAGs are used to hash content for quick locating, to ensure content is up-to-date, and to build a linked line of content over time.
This gets more interesting if there's a widespread service outage. IPFS nodes will continue to serve the most up-to-date version of the web even if the web is fragmented. As new information becomes available it is integrated into the existing cache and then propagated to the rest of the fragments.
I still struggle to understand how this works with databased content, but I do believe IPFS addresses this content.
Yeah - there's a ton of immutable URLs on the web. All of CDNJS/JSDelivr/Google Hosted Libraries, most of raw.github.com, all Imgur & Instagram images, all YouTube video streams (excluding annotations & subtitles), and all torrent file caching sites (like itorrents.org). There are probably some large ones I'm forgetting about, but just mapping immutable URLs to IPFS could probably cover 1/3rd of Internet traffic.
IPFS archives is the effort that's going on right now to archive sites. Eventually there will be a system for automatically scraping & re-publishing content on IPFS.
Right now storage space and inefficiencies in the reference IPFS implementation are the biggest problems I've hit. Downloading sites is easy enough with grab-site, but my 24TB storage server is getting pretty full :( ... Gotta get more disks.
Say you grab a site. How do you announce that fact, verify that it is an unmodified copy, sync/merge/update copies and deduplicate assets between different snapshots?
Say I have a site that sells Awesome Products. L337Hacker mirrors my site, does a DDOS on the original and lets IPFS take over, redirecting the shopping cart to his own site.
Is this a potential scenario? If so, is there any way to prevent it?
If I'm reading this correctly, there are a few ways IPFS could be used in support of distributed, fraud-resistant, commerce.
First: publishing the catalog isn't the same as processing the shopping request. Online commerce is largely an update to catalog + mail-order shopping as it existed from ~1880 - 1990. If someone else wants to print and deliver your (PKI-authenticated, tamper-resistant) catalog, that's fine.
Second: The catalog isn't the transaction interface, it's the communications about product availability. The present e-commerce world is hugely hamstrung on numerous points, but one of these is the idea separating the catalog and product presentation itself from ordering. So long as you're controlling the order-request interface, you're good. A payment processing system which authorised payments via the bank rather than from the vendor would be helpful. Also a move away from account-info sharing.
The key is in knowing who the valid merchant is, and in establishing that the fraudulent merchant has misrepresented themselves as the valid merchant. Perhaps authentication within the payment system would help.
Taking the shopping cart's payment mechanism out of the shopping cart would help.
All IPFS URLs contain the hash of the content, so you can't change it. There's a mechanism to allow for URLs which can point to varying bits of content, but I'm not aware of a paper which shows its security properties.
Upon further reading, it appears that it may be impossible to verify the security of an IPFS cached page, simply because the hash is calculated post-fetch on the client. That allows any sort of shenanigans to be performed on the original content before it's stored.
If content is created specifically for IPFS-caching (similar to Freenet or Onion), then it may be possible to be authoritative, but content cached from the web should never be considered so.
> Upon further reading, it appears that it may be impossible to verify the security of an IPFS cached page
Not at all, rather the opposite, it's very easy to verify a page since the hash is based on the content.
You have file "ABC" that you want to download. So you fetch it and once you have it locally, you hash it yourself and you compare the hashes. If they are the same, you know you have the right thing. If they are different, someone is sending you bad content.
Re-read the original question. If someone is preventing access to the original page and the alternate is being served thru IPFS, there is no way to compare the original. The IPFS cached page becomes the authoritative page and could contain altered content, which the hash takes into account.
If the original page can perform the hash and embed it, that would somewhat alleviate the issue during the fetch, but do nothing to prove that the IPFS-served page was trustworthy or not, unless some third-party knows the original hash, as well.
If the page was served to the IPFS network, to be cached, by a neutral, trusted third-party, that would somewhat alleviate the problem, although there arises the problem of trust again.
The only way to minimize the trust issue is if the page originates from inside the IPFS network and is not a cached version of page originally served outside the network.
You're right. I misread your parent's comment, which suggested that web pages could magically be cached securely into IPFS from the public web without any involvement from the website's owner, which is nonsense.
1) The web as a whole is decentralized, but any given web site is centralized by default. A webmaster can "push" a modification or deletion from a central server and have it trickle down through caches/proxies unless someone like archive.org makes a deliberate decision to archive it. In systems like IPFS, content deletion and modification follows a model more like garbage collection, where pointers to old data can hang around and continue working even if an "authoritative" pointer is pointing to something else. Basically, nobody has the authority to break a link (law/policy-mandated filters notwithstanding).
2) The web doesn't really have any built-in mechanisms to promote redundancy for durability, availability, or performance purposes. By default, there's one daemon serving one copy. Everything else (CDNs, forward/reverse proxies, failover) is some kind of optimization tacked on after-the-fact.
The idea is that we run that centralized search engine via Smart Contracts.
In other words, using a decentralized currency such as Bitcoin and decentralized execution environment like Ethereum, we create softwares which pays people for providing CPU time and disk space (facilitated by IPFS).
Few missing pieces of this puzzle are homomorphic encryption implemented on Smart Contracts level which will allow that CPU service.
Once that's done then we can have the distributed web of the next level.
A search engine via ethereum is not going to scale, is not easily upgraded, and will fall far short of any properly centralized search engine. If you want a competitve decentralized search engine, you're going to need a different technology.
Homomorphic encryption as best I'm aware is more than a decade away from practical use. I believe the best libraries today have blowups of 100,000,000x in resource usage, meaning it's not even practical for basic crypto operations.
Multiparty computation makes some security sacrifices (breaks if M out of N people cheat, breaks undetectably iirc), but I think the blowup is less than 10x for some operations and less than 1000x for the others.
Still not pretty, but at least makes certain things possible. But not competitive search engines.
Does the issue of digital-currency volatility apply?
If you're hosting content at a certain rate, then the rate drops so that it's less profitable or no longer profitable, wouldn't the proper strategy be to drop that content and then find something worth hosting?
>what makes the supporters of it think it will be any different from the original web.
Disclaimer: I have worked on building the Internet since before the web, and have a BOFH kind of point of view of things, 2 or 3 times over now ..
I support IPFS - and things like it - as a technologist/hacker/punk-ass because, in fact, we have built an utter monstrosity of a beast of a spaghetti god of an Internet, and we can always do better.
Software - and by extension, inter-networking - is like music. There will always be better ways to do it, beyond the horizon of what is current and now. Just because what we've got "basically works, even though its all broken", doesn't mean we can't 'fix whats broken'.
To those of us in the group of your postulate, whats broken is the fact that its all so damned hierarchical, and requires canonical/authority issue and agency in multiple forms before the bits can flow. We can't just set up an IP address - we also have to have DNS, serve content on it on some port, etc. And, on top of this, we end up having to sort it out all the way down the OSI stack, at the hardware layer, by keeping as many hosts as possible up and running as are required to service the customers, here and now, who are using our service.
All of that is what is broken about the current Internet, but of course the irony is that it is all that works about the Internet at the same time.
So, IPFS/distributed/etc. means this: we can un-bork all of the above, and just have all the things route down to the real, hard, user of the system, i.e. the original content provider. IPFS, and its ilk, is all about putting the original content provider in control, while also having the network - as a function of its operating capability - contribute to helping those content providers who become, eventually, popular.
So, we don't have a lot of rigid hierarchy - we have instead multi-variate spread spectra of responsibility/duty to serve up the bits of it all - packets at a time, as part of participation in the whole - to those who want something, and those who know how to find it, from those who have something, and know how to name it.
In short, "its not about the network any more, its the people..."
I worry that this is another example of throwing technology at a social and political problem.
That the current web is centralized has little to do with its technical design, and everything to do with economic and structural incentives that have made it that way.
It's tempting to say "start afresh", but we'll just be trading our current problems for a new set of problems IPFS introduces. It's a law of nature that problems are always conserved.
I would rather we do the hard work of fixing the web we've got, in particular the hard issue of how to re-decentralize it.
> in particular the hard issue of how to re-decentralize it.
Isn't that exactly the problem ipfs is attempting to address. What kind of solution do you want and why does this one not work for you? It's easier to discard someone else's idea than it is to evaluate it's merits. I'm not sure this is the solution but it looks interesting and it appears to interoperate with the existing Web via gateways even though the author does not make this clear early on in the documentation.
I'm not trying to pee all over IPFS. But it's also the case that it's easier to throw out mature, crufty old systems and start afresh. Every programmer knows this temptation.
We've had the web for twenty years now, and are just beginning to understand what it is and how it changes the world. If you're suggesting using IPFS (or any other system) as a way to expand and grow the web, I'm with you. But replacing the web means throwing out a lot of hard-won experience and intuition, in favor of a system that's bound to have its own issues down the line.
If there was something dramatically better for every use case (which is not IPFS) it would slowly displace HTTP in new deployments and there would still be HTTP for a very, very long time. The Web we have, in other words, isn't going anywhere.
> I worry that this is another example of throwing technology at a social and political problem.
I'm so lost at this assertion. How is this a social / political problem?
IPFS is not permanent hosting. It is not an attempt to take power away from others and distribute it to the masses. It is purely a technical problem, which i think is objectively true.
I have a feeling that you believe IPFS is partially designed to subvert governments/etc. And to an extent, that may be possible, but it is being built with DMCA's in mind. IPFS wants to allow governments / copyright holders to take content down from the network. It is not trying to become a pirate haven. Why? Well, mainly because piracy inhibits adoption. There may not be a way to DMCA currently (i haven't checked in months), but there are Github issue(s) on it. The creators don't want to see IPFS blocked in countries because it can permanently host illegal content. Which makes a ton of sense.
The problem IPFS is trying to solve, as i see it, is purely technical. As advertised in pretty much every pitch i've seen for this technology, we are simply scaling too fast in storage and not fast enough in bandwidth. It appears impossible to handle the infrastructure load of files we are creating. IPFS solves this by automatically pulling content as locally as possible, reducing network overhead. Furthermore, it means that a DDOS/viral/etc of a central resource will not kill it. It's safely and mostly permanently (barring DMCA/etc requests) available.
So with that said, what political and social problem do you see IPFS trying to solve? I don't see any.
"The Internet has been one of the great equalizers in human history and a real accelerator of innovation. But the increasing consolidation of control is a threat to that.
"IPFS remains true to the original vision of the open and flat web, but delivers the technology which makes that vision a reality."
This presents IPFS as the technological cure for the "consolidation of control" that threatens the Internet.
Well, that is fair. I'm mainly talking about "normal" adoption though.
Eg, for a long time BitTorrent was being branded as a pirate haven. The only major players i saw adopt BitTorrent were open source projects needing to distribute large ISOs. IPFS is something that they're trying to get into standards, into browsers, into operating systems, etc. If it's known for distributing kiddy porn on the dark net it will be much more difficult.
I know piracy spreads like wildfire, but it's not really the target audience for what IPFS is trying to do. In short, they want YouTube, not PirateBay. Or so i gather, at least.
They are addressing it, but it's a future feature. Last i saw on Github issues (it likely has evolved since then), they plan to have a central authority which allows for DMCAs/reports/etc of illegal content. The network then propagates an ignore list of sorts (or possibly watches a central list).
Any nodes on the network which abide by this will simply not download and/or unpin the data.
It doesn't prevent you from downloading some kiddy porn that a stranger sent you.. but neither does the internet. This, is purely to allow law to take place, and hopefully allow lawmakers to work with IPFS, and not block the whole damn thing.
It's ironic that IPFS plans on using a central service to censor and purge it's distributed system.
Not that this central authority will solve this issue either. It just opens another can of worms.
Who will review claims and reports? Who will be authorized to add hashes to the blacklist? How will counternotices be handled? How will different juridistictions and conflicting laws be handled? Who will be liable for content that makes it past the filters? Who will pay for all this?
The world is a messy place and IPFS does nothing to protect its users.
> The world is a messy place and IPFS does nothing to protect its users.
I'm a bit confused, why is it the job of IPFS to "protect its users"? IPFS has no liability here, it's simply trying to make IPFS a law-abiding software as a whole. So it is not outlawed.
IPFS doesn't have any reason to "protect" me from things i download anymore than HTTP or TPC does. It's a technology for localizing data. What makes you feel IPFS has to protect its users?
It sounds like you think IPFS somehow spreads files onto your system without your consent. You only download what you choose to, just like with HTTP/TPC/UPD and the whole tech stack.
IPFS changes nothing for your personal security. It's not intended to. It's a technology to lower bandwidth usage from the production of mass data in the current age. And in regards to it being ironic for having a central authority, - sure - but the problem is bandwidth that IPFS is aiming to solve is, not law and/or DMCAs.
IPFS must have some seriously bad marketing (i blame the whole "good for humanity angle"), because i feel like you and a lot of people have the entirely wrong idea about IPFS. It's a technology, nothing more.. especially in it's current form. The only thing it might "save the world" from is bandwidth, lol.
IPFS needs to protect its users, if it is to have any users.
The creators of IPFS may not have any liability, but they open up its users to severe issues of liability due to the way it operates.
IPFS will become toxic real quick, if it becomes know that IPFS can make you an unintentional distributor of kiddy porn and pirated content.
The core problem is that IPFS' default mode of operation is actively user hostile. This is a big problem, and the reason IPFS needs to protect its users. IPFS not only leaks what you retrieve, it also makes you an (unknowing) distributior of everything you retrieve. All it takes for you to commit a crime and/or copyright infringement, not only as a consumer but also a as a distributor, is to be tricked into retrieving one hash.
In conclusion, IPFS needs not only to not be outlawed, it also needs to be safe to use. As it is IPFS is not safe to use. Hence IPFS needs to do something to become safe to use, i.e. to protect its users.
> IPFS will become toxic real quick, if it becomes know that IPFS can make you an unintentional distributor of kiddy porn and pirated content.
Yea, this is where you're mistaken. It cannot make you distribute anything! You only distribute what you download. Don't download kiddy porn, and you won't distribute kiddy porn.
> All it takes for you to commit a crime and/or copyright infringement, not only as a consumer but also a as a distributor, is to be tricked into retrieving one hash.
That is fair - but the same could be said for every p2p system on the planet. Has anyone solved this problem? How does BitTorrent protect me from unknowingly downloading kiddy porn?
> Yea, this is where you're mistaken. It cannot make you distribute anything!
Oh, really?
"In some cases, nodes must work for their
blocks. In the case that a node has nothing that its peers want (or nothing at all), it seeks the pieces its peers want, with lower priority than what the node wants itself. This incentivizes nodes to cache and disseminate rare pieces, even if they are not interested in them directly."
What's to say BitSwap won't proactively decide to download and start seeding some rare pieces of kiddy porn?
> You only distribute what you download.
As if that wasn't bad enough.
> Don't download kiddy porn, and you won't distribute kiddy porn.
How are you to know if a hash contains kiddy porn or not?
You need to know if the hash contains kiddy porn before you download it, but to know if it contains kiddy porn you have to download it. Classic Catch-22.
The only rational response to a system like this is not to use it.
> How does BitTorrent protect me from unknowingly downloading kiddy porn?
By not randomly downloading stuff from the Internet. By only downloading torrents, I have explicitly chosen to retrieve (hopefully from vetted and reputable sources). By giving me complete control and feedback on what I'm seeding.
Obviously not ideal, but far better than what IPFS does.
I also don't get how this central censorship authority is going to work. I very much doubt IPFS wants to become the Internet Police for Saudi Arabia.
The whole thing is a futile attempt to appease lawmakers and dictators, since any filtering will be done clientside. Nothing is stopping the client from ignoring any blacklists.
Any great or small firewall will just find it easier to block the whole thing than do some silly whack-a-mole deep packet inspection.
How is IPFS not going to become a haven for kiddy porn aficionados?
If the censorship authority publishes a list of blacklisted hashes, that's the same as publishing a directory of all the available kiddy porn. Oops!
If the censorship authority requires each hash to be queried individually, it becomes a real honeypot for the state surveillance machinery.
> What's to say BitSwap won't proactively decide to download and start seeding some rare pieces of kiddy porn?
Nice catch, that is interesting. Granted, i feel like this could be tweaked as it is simply a mechanic to attempt to reduce leeching on the network.
> You need to know if the hash contains kiddy porn before you download it, but to know if it contains kiddy porn you have to download it. Classic Catch-22.
How do you know if that link to a pdf from a friend contains kiddy porn? You need to download it to check the md5 / contents.
I'm really lost on how this is any different than HTTP/BitTorrent/etc. Any link on the web can be kiddy. Any bittorrent link can be kiddy. Any IPFS link can be kiddy. They're all equally dangerous on that front as i see it.
The only argument i feel like can be made on this front, is that the hash can make for a hard to visually decipher file. Ie, `foo.com/bar.pdf` is inherently more "trust worthy" than `/ipfs/d3b07384d113edec49eaa6238ad5ff00`. With that said though, that's a better reason to not use `/ipfs/d3b07384d113edec49eaa6238ad5ff00` than it is to use `/ipns/foo.com/bar.pdf` (fake domain/example for easy discussion).
Again, stay away from random hashes just like you stay away from sketchy torrents and random http files.
> Granted, i feel like this could be tweaked as it is simply a mechanic to attempt to reduce leeching on the network.
How do you propose tweaking it without either publishing a directory of all the kiddy porn on IPFS or creating a bottleneck which incidentally is ideal for tracking and surveillance?
> How do you know if that link to a pdf from a friend contains kiddy porn?
You don't, but at least your browser won't shout out to the world: "Hey, everybody! This guy is downloading kiddy porn! IPFS does that and to add insult to injury then automatically goes on to seed it.
On the web you also have a sense of where you are. With IPFS you don't, it's all a bunch of hashes. There is no there there. There are no clues to tell you that you are in a seedy neighborhood. Everything is just available, one hash away, from kiddy porn to fine art. That makes navigating IPFSpace such a minefield, the next hash could take you anywhere.
> I'm really lost on how this is any different than HTTP/BitTorrent/etc.
Let me recap:
- BitSwap
- metadata leakage
- automatic seeding
- opacity and lack of control
> Again, stay away from random hashes just like you stay away from sketchy torrents and random http files.
How? Is there some kind of sniff test I don't know about?
Fwiw, I appreciate your insight on this matter. I know we disagree on some points, but it was fruitful for me to see your perspective. Anyway:
> How do you propose tweaking it without either publishing a directory of all the kiddy porn on IPFS or creating a bottleneck which incidentally is ideal for tracking and surveillance?
Well this is off the cuff, because i wasn't aware of BitSwap's desire to seed in the event of no swap data.. but with that said, i was mainly just referring to alternate methods to help inhibit leeching.
Eg, allow leeching but only for the first (tiny) %x. How willing a host is to allow leeching should be up to them regardless. If i'm hosting a file i want to distribute for an open source network, it's in my interest to host it, and i'd rather not inhibit others from downloading what i'm providing them based on some BitSwap algorithm trying to make things "fair".
P2P is a tough challenge though, which is what that whole sketchy BitSwap section stems from. I imagine a beer and an hour and you could come up with a dozen alternate methods to discourage leeching. Whether or not they're as effective as requiring "work", is impossible for me to say.. but i think we both agree that the work outlined in that PDF is a bit.. sketchy.
> How? Is there some kind of sniff test I don't know about?
Well, i don't know about you - but i don't download random files on the internet. I download from trusted sources only. IPFS would be no different. A trusted web TLD or a trusted IPNS namespace would be good enough for me to feel it is trusted, in both accounts. Outside of trusted domains on the web, i don't download it. Eg, would you download 555.555.555.555/foo.pdf? I certainly wouldn't.
I think with IPFS it will become a bit defacto to trust IPNS rather than IPFS directly. Not only are hashes potentially dangerous, but they're simply a terrible UX. A parallel would be like downloading `555.555.555.555/foo.pdf` - you have no idea if that's kiddy porn or not, it's effectively a random hash. Don't download it.
A domain (like that IPNS is aiming to provide, i believe) would be more akin to `https://trusted-site.com/foo.pdf`. Long term, i don't think anyone should be using hashes directly. But that's just my 2c, we'll see how it plays out.
This brings us back to my original point. Even with trusted IPNS namespaces, IPFS needs to protect its users. If nothing else IPFS should have some kind of content policy engine that warns or blocks the user from unintentionally leaving safe namespaces and/or retrieving random hashes.
P.S. 555.555.555.555 isn't really a great example IP :)
> I'm so lost at this assertion. How is this a social / political problem?
How is it a technical problem? Current web technologies allow decentralised web (except for maybe root name servers). Everyone can host their own content, control their own servers, basically everything below your hostname. The implementation has led down a centralised path.
> The implementation has led down a centralised path.
Sure, but choosing another implementation is.. a technical choice, is it not? Are you saying that if i choose to implement a distributed method it's not technical? What is my choice then, political?
Because i have no political motives.. purely technical. I want less data choking the pipes. Which is all IPFS is ultimately doing. Seems pretty technical (instead of social/political, atleast) to me.
> I worry that this is another example of throwing technology at a social and political problem. [...] I would rather we do the hard work of fixing the web we've got
There are multiple flaws in your reasoning: for one, the web was also a technical solution to a social problem; also, IPFS is someone's hard work aimed at fixing the web we've got.
That was the initially scoped solution. It turned out that a lot of other actors had information management & sharing problems of a similar scope. Well before commercialisation of the Web.
> It's a law of nature that problems are always conserved.
I'd agree that some social issues cannot be solved by technology, but many have been.
E.g., without looking at the information age: in London before the Clean Air Act thousands people literally died of the smog, but the CAA was made possible only by increased availability of electricity and gas heating.
Your example is one where political and technical changes together helped solve a problem. You're literally citing a law as an example of a 'social issue solved by technology'.
My point is that replacing a complex technical system with a different technical system only reshuffles the set of problems that will need solving. The social and political problems that made the web what it is today won't disappear due to a redesign.
He is actually saying that the law could only change due to technological advance. That law (if memory serves) banned burning coal, which was how people heated their homes. There had to be something that could heat their homes instead of coal, before they could be compelled to use it.
I think there are multiple issues in many contexts wrt decentralised net. You probably won't require Facebook or Twitter for many reasons. There's both a social and tech issue.
But there are some things that mirroring and content addressing would be perfect for. Replacement for the current script and files CDN for example. Package/software repos, firmware downloads, news sites, etc. Anything accessed by lots of people and not updated every second.
And there's also the location context. If you live on the west coast US, maybe it doesn't matter. But if half of the web traffic could move to ipfs, Australia for example should get an amazing speed increase by unblocking the big pipes.
I've recently had issues accessing most CDNs from inside of China without a VPN, accessing directly doesn't have this issue. So maybe IPFS could help there as well.
Technology may help by making it easy for people to host/share their data. But technology is driven by politics. So we kind of need both.
The solution that I see, is transitioning from social media, where people use some centralized service to "socialize", to social infrastructure, where people run their own servers and connect to the servers of their friends making a real social network. IPFS may be a step in this direction.
Id say we need a political movement to push through a technical solution to problems like this, so I agree with you on that. But I think there is definitely space for things like IPFS. Technical solutions can give more guarantees than people can, so without it, what would we really have? Promises easily broken in the future?
The problem with things like IPFS, is that they require political backing or economical incentive to really spread, otherwise they are most likely just going to remain a nice demo for a few technical people.
So IMHO technology needs to go hand in hand with economical or political incentive, but how is an open question.
> I worry that this is another example of throwing technology at a social and political problem.
And here i was just listening to Adam Curtis talk about his latest creation, Hypernormalisation, where he touched on the issue of thinking one can solve social issues with engineering.
There are plenty of technical reasons why the web is centralized.
I cannot even host a server on my residential service without violating the ToS agreement. Even if I did, there is the problem of not enough IP addresses so my IP is dynamic. This is solved with dynamic dns, etc but is another technical hurdle to overcome. And then there is NAT creating yet another layer to hosting your own server. And I won't even go into security.
These are all solvable, but to say there is little to do with the technical design ignores all of the problems that led us to where we are today.
The key is economic. The cloud and SaaS have taken over largely because the cloud is DRM and because centralization creates opportunities for ad and ad-driven surveillance business models.
There is merit in doing the technical work to make a decentralized paradigm easier, but the lack of an economic model is the tougher of the two problems.
One is discoverability. It's tough to find a bunch of scattered systems all over the place. That's a problem Facebook, Reddit, and HN address.
Another is identity. Tracking who's doing what (and what their permissions/authorities are) over a bunch of sites is ... a pain. I'm still mulling over a comment a few days back on HN that a user had 770 different password-based identities in their accounts-management storage (personal).
Search (related to the first) is a third -- spiders suck a lot of bandwidth. Unless there's some way to distribute search or allow nodes to self-report search elements (which will probably require some form of auditing).
And there's administration. It's really easy create systems any idiot can use, but then, any idiot will, and DDoS Brian Krebs and Dyn. Bulletproof administration is a real pain point.
There's the million-dollar question. I've been bashing my head against it for a few years now.
I think an important thing is to give everyone a say in how these large computer networks affect their lives. How to do that in the context of a broken political system isn't clear to me at all.
You are clearly fond of democracy as a solution to conflicting social goals. Democracy has some nice properties, but also downsides: notably it can only try a single solution at a time.
The idea behind decentralized architecture is: instead of trying to find a better way to agree on one thing, how about we assume from the start that different people will want different things? What do we actually need to agree on?
In the old days we had to agree on everything, because it was technologically infeasible to coordinate plans more than a few times per year. If you need to move bodies to communicate, democracy is what happens.
But we live in a new time where messaging is close to free, so we don't have to agree on everything ahead of time. In a totally decentralized world, we wouldn't have to agree on anything ahead of time. I would meet you in the forest and our respective AIs would go to work helping us quickly and safely negotiate an entire legal framework and set of treaties governing our interactions.
I don't think we will ever get there, I think it will always be a mixture of decentralized and centralized norms. But right now the decentralized architecture is brand new, so it has a huge headroom for growth, so it just makes more sense putting effort into it.
This image of AIs in the forest is really arresting. It reminds me of something Joi Ito said recently about us nerdles all trying to pass the thorny problems of living with one another as human beings up to a higher intelligence, which is a way of reducing them to (the world's hardest) programming problem.
I imagine that our AI companions in the forest would just play hyperdimensional space chess with one another, or talk shop about how best to make paper clips, and leave us in the same position we are now. They'll have their own cares and woes.
I don't know if in 2016 I'm a big believer in democracy, but I do believe in community, particularly now that everyone regardless of technical ability is using the web.
That said, I think you make a very good point about variety and trying multiple avenues at once.
I'm not talking about autonomous AIs, I'm talking about controllable AIs.
And I absolutely don't consider it a programming problem. You'll notice I said they will help us negotiate, not negotiate on our behalf. It is a design challenge, which makes it partly a social problem and partly and engineering problem.
I'm slowly realising a fundamental truth of economics: whatever it is you're making cheaper, you're going to have one hell of a lot more of.
And as you increase the quantity, you'll start bumping into limiting factors elsewhere. E.g., cheaper transport => far more congestion. Cheaper communications => more advertising. Cheaper storage => vastly more surveillance and monitoring. Cheaper telecoms => vastly more phone solicitations and scams.
There's the question of what people really need to access -- and here I think it's key to go back to Maslow's Hierarchy. And to acknowledge our human limitations of information capacity per day. Several sources I've seen (Stephen Wolfram, Walt Mossberg) suggest about 150 - 300 email messages/day, and something less than even a cursory glance at 800 comments (The New York Times's comments moderation team). How do you provide a manageable and significant set of information to people?
I've also started noting that our-so-called discussion systems themselves (Slashdot, Facebook, Reddit, even HN) are pretty poor at self-supporting useful user feedback. Almost as if that's not their true function.
We wanted to turn around the incentive structure where right now senders have a lot of power and receivers have basically nothing, they have to eat what they're fed. https://www.youtube.com/watch?v=Yth7O6yeZRE&t=23890
The solution does not have to be technical. If you want to restore control to as wide a group as possible, start by running a centralised service with a membership structure. Not everybody is clever enough to run their own stuff, but almost everyone can use a website or an app. Allowing them a role in governance rather than operations is the key factor. I wish more people thought so.
> "Not everybody is clever enough to run their own stuff"
It's possible to make home servers easy enough to use they're effectively plug and play. I've got my own ideas of how this can be done, others have different ideas, but the goal of easy home server infrastructure is the same. Once you have that, it becomes painless to take part in IPFS or similar.
I used to think that home servers could be the best solution, but I see more and more obstacles in the way of adoption. First, even if you can create a plug and play home server box, it will still need some configuration to make it available from the internet, which while very obvious to anyone a bit techie can be difficult to even explain in layman's terms. Secondly, it is not clear if there is an acceptable solution to security issues. In such a system that by design will be accessible from the internet, this a major problem. Auto updates could be part of the answer, but it seems that human intervention would still be needed from time to time and that imply an effort of the user to keep informed and aware of such operations. Or an entity could offer a remote administration service but that would displace entirely control of the system and create a single point of failure. At that point you might as well have that entity administer a single server used for a moderate number of people. If this entity is a non-profit where governance is shared between all members it could very well be an ideal solution. In France the "association loi 1901" legal structure is a good fit and is known and understood by the vast majority of the population.
> "First, even if you can create a plug and play home server box, it will still need some configuration to make it available from the internet"
Will it? The device I had in mind would plug directly into a router, occupying two LAN ports for connectivity, and one USB port for power (USB ports are increasingly common on routers these days, even the free ones ISPs give out). In addition, IPv6 removes the need for NAT and port forwarding. The device could even come with a default DNS name setup, so it could be literally plug and play.
> "Secondly, it is not clear if there is an acceptable solution to security issues."
You build the security into the device, e.g. firewalls and process isolation. You could also design the device so that flashing new firmware required a physical button on the device to be pressed to enable this feature for a short period of time, limiting the window of opportunity for the device to be infected with malware.
> "Auto updates could be part of the answer, but it seems that human intervention would still be needed from time to time and that imply an effort of the user to keep informed and aware of such operations."
Information about security updates could be done via email or SMS, which are both easy enough for mass market users. You would enter these details at the point of purchasing the device, so you'd have nothing extra to setup once you had it. If you had to update your contact details, this would be easy enough to set via the home server GUI.
So I've been thinking about creating a basic site running on IPFS and here's my dillema. The hash of each page is a sha256 of the contents right? So lets say you have 3 pages A, B and C, A links to B, B links to C, C links to A. How do you create all 3 pages with correct links to each other?
When you create page A you have to have the SHA of page B, but then to create page B you have to have the SHA of page C and finally to create it you need the SHA of page A. You get into this cyclical loop where you can't generate any page and link to others. What is the solution to this problem?
Well, IPFS allows you to upload trees of files to a single IPFS node.
So you just create three webpages, put them into a single folder and use `ipfs add` on this folder. Your webpages will be available with relative links, being exposed at addresses like `/ipfs/<Sha>/page_a.html` and `/ipfs/<Sha>/page_b.html`.
Under the hood those pages will be standalone ipfs Duke nodes with separate hash addresses, and your root folder will be actually a set of pointers to those hash addresses — but you don't need to keep that in mind when uploading a site.
One can also study ipfs project websites (ipfs.io, dist.ipfs.io and so on).
Those are just static bunches of files (html and resources) hosted in ipfs and served with a standard ipfs gateway (located on gateway.ipfs.io address). Gateway just looks up _dnslink TXT records for corresponding domains, and serves files from the ipfs hash address specified there.
> Each network node stores only content it is interested in [...]
Isn't that the issue here? Storing data that will maybe be there later isn't really storing data. People want to publish something that must always be available, so why inject data into the IPFS network and hope it will be there in a year, rather than set up a $10/yr VPS?
> With video delivery, a P2P approach could save 60% in bandwidth costs.
In my opinion, this may be true, but total costs will be greater. P2P solutions are awesome because they are resilient, not because they are cheaper. Distributing pirated movies by dumping them on public FTP servers is much cheaper than BitTorrent. BitTorrent appeared because the centralized method was not resilient enough against adversaries, not because it was cheaper (quite the contrary).
If I get it right, the idea is that if you want to make sure it's there one year later, you set up your own server and host the files on your own. IPFS just allows others to distribute a mirror of the files.
Why would nodes in the IPFS network voluntarily act as a load balancer for my VPS? Usually I have to pay for load balancing, so I'm a bit suspicious when someone claims I can get it for free from IPFS.
> Why would nodes in the IPFS network voluntarily act as a load balancer for my VPS?
Why would nodes in the BitTorrent network voluntarily act as a load balancer for your movies?
Same reason. Maybe you and I cannot explain the reason, but that does not mean that it doesn't happen. People don't always act in a 100% rational, for-profit manner, no matter what economics textbooks tell you.
(Edit: Of course the likelihood of your content being mirrored rises with the quality of your content.)
There is a lot of (mostly free) content that people are willing to distribute freely.
Take Wikipedia for example, even though it's changing very fast, I imagine a huge portion of it is static. If the latest version of each page was served via IPFS, somebody could setup a local mirror and people nearby would automatically use that.
I don't disagree with the fact that IPFS has value. I just don't like the way it's presented as a generic transport protocol that's "superior" to HTTP.
It's more resilliant than HTTP: When you grab a torrent, you can download from any seeder. With HTTP, the host is specified, so if that host goes down, good luck.
In either case, if all the hosts go down and you didn't mirror it, you're screwed. However, IPFS makes it easier to mirror things.
While this sounds like it "should" be true. I think managing that will turn into something of a nightmare. IPFS is making awesome inroads into achieving that, but its not clear of the benefit over something like say freenet (other than freenet is very slow because it prefers privacy and resilience over speed).
I can see it being "really" useful as a backbone for something like Open Cobalt - http://www.opencobalt.net/ And I'm really looking forward to seeing that mature.
But the last time Open Cobalt seems to have been updated is 2010 - that's quite some time ago.
I wonder if it's being held up by patent trolls, but really all of them feels like more like a solution looking for a problem. That can take a very very long time to mature. Which is a shame.
Alpha 22 was actually released just last year, and it's still actively being worked on AFAIK. It's just that the website hasn't been updated, save for the downloads.
> Why would nodes in the IPFS network voluntarily act as a load balancer for my VPS?
(a) People currently downloading and viewing the site. Like bittorrent, the more popular a site/page is, the more people will be seeding it. (b) The Internet Archive.
If I try to host a javascript application that uses LocalStorage for saving data, it would be visible to any other ipfs JavaScript application because they all exist under the same domain, right? Have you thought about having the URLs be something like ipfs://<hash>/index.html instead of http://local host/<hash>/index.html so browsers keep the LocalStorage for each ipfs hash separated?
Putting a hash in the hostname/domain would also be using same-origin policy. The issue you linked refers to using suborigins, which seems to still be a draft proposal with no implementer buy-in outside of Chromium [1].
Could be wrong but I think [beaker](https://beakerbrowser.com/) supports using the protocol like that. Of course its very experimental so if security is a concern then it probably has its own issues.
Well, as I understood IPNS is a bit limited; it only has one resource and there is only one authority that can change content, the node in this case.
It does protect against DDOS, since old content remains visible and online but if the private key is leaked or the node is malicious, it has exactly 0 protection.
Even worse, if the private key is stolen, you can't do much about it unless you have DNS setup... which brings you back to centralized.
IPNS is bug Proof of Concept that mutable content is possible in IPFS, there is long running goal of InterPlanetary Record System that would include much more complex validity schemes including revokeation.
I'm curious what the impact will be of the upcoming improvements to easily support multiple IPNS entries on a single node, along with navigating the Ethereum blockchain, which I'm guessing will make more complex authority schemes easier?
If the key is leaked, it means someone can alter what the name points to, but all previous versions would still be available as far as I understand it.
No content is changed, only the pointer to the "newest" data.
That is true but most external or IPFS-external sites will probably want to point at the IPNS link, bookmarks are gonna point at the IPNS link, etc.
In the same way you could still use the old IP + header manipulation if someone took over a Domain in DNS, it's possible and you can use the old site but in reality you're gonna have lots of people on the malicious attacker's site.
I think that's an important distinction though, and it doesn't mesh with what I thought you meant by
> since old content remains visible and online but if the private key is leaked or the node is malicious, it has exactly 0 protection
The old content remains visible and online if the private key is leaked, afaik.
I think there was some discussion about having there be a chain, so each updated entry would point back to the previous one. I'm not sure where that is (and if it is) on the roadmap now, but that would also allow users to go back to a known good version. This mitigates some, though not all, problems.
>The old content remains visible and online if the private key is leaked, afaik.
The problem is that content that is not access becomes garbage collected and that links and listings will most likely not point towards the hash itself and instead at the IPNS entry.
It doesn't have much more protection than a DNS entry if the private key is stolen; you could use the last known IP but it's not very practical and might eventually become dead if the IP changes.
In the same way, the old content may remain online after an attacker takes over and people could continue using it but it only remains online if people are using it. If the attack remains unknown for long enough there is no previous version IIRC.
>I think there was some discussion about having there be a chain
Still same problem, if the private key gets leaked, a malicious attacker can manipulate the entries on the chain.
The real solution is something like Swarm or Filecoin where Users can actually encourage keeping old versions around, something IPFS lacks atm. Theoretically any IPFS link can become dead once it's no longer used, which is a problem.
Merely putting the entries on chain solves little, you need to actually encourage archiving of data to get resilient against attacks.
Even then, you would also need some sort of PKI, where someone can set a master key which authorizes other nodes to manipulate content and can also revoke that. This allows to easily fight off attacks based on leaked private keys.
However, the PKI should remain largely private, not much different to Bitcoin Cold and Hot Wallets.
I'm not sure what attack you're talking about then. If someone steals my key and points my IPNS location to a new file, I'd likely still be hosting the files right?
> you need to actually encourage archiving of data to get resilient against attacks.
Yes, but this feels like a distinct issue. Neither of these things are enough on their own (encouraging mirrors or providing the tech to distribute the hosting).
>And anyone pointing to a specific version will still see that rather than the new one.
That's an IPFS Hash tho, not IPNS. Differences matter.
>I think that's a bit strong.
Over time your IPNS could very well become your identity, linked everywhere in the net.
>IPNS doesn't give you the ability to edit content.
That is correct, but you can point at arbitrary content.
It's basically the equivalent to controlling the DNS A entry to a website, they can't modify the content and it's available under the same IP but they can still do a shitload of bad stuff because few people bother to actually link IP addresses.
How do hosting providers fit in here, if at all? E.g., if I want to host a website on IPFS, do I publish it from my own machine and then wait a healthy amount of time for the content to be absorbed by the ether, or is there some way I can encourage other nodes to pick it up without requiring end-users to actively seek out my fresh material?
It seems that the role of a hosting provider could be: I pay you <amount> per month to fetch <hash> or <name> on <N> peers with bandwidth <bandwidth>.
Of course, the economics are quite different. In the normal web, you need to add more resources when a page becomes more popular. In a distributed web, more resources are a side-effect of a page becoming more popular.
So if you add a file to IPFS, it's not gonna be automatically picked up by the network. Only people who request it, will have the file.
In the future, there will be hosting providers that allows you to upload a file to their IPFS "cloud" either as free hosting or paying for it, just as today. It'll just be easier for everyone to pass around a hash of the file rather than the file itself.
Maybe there will even be an API that you can hook up into your application, so content will be seeded that way.
Suppose I'm poking around IPFS and unintentionally download some unauthorized copyrighted content. Is my computer going to automatically start sharing this content, exposing me and my ISP to legal action?
Or if there is a way to prevent sharing particular content that I've accessed, what's to stop me from leeching everything and never sharing anything?
(Edit: Ah, now I see "BitSwap" as possibly addressing my second question, but I'm still concerned about the first.)
Since it's impossible to know who owns, claims to own, or has licensed any particular content, #1 is a fatal flaw unless users can officially receive immunity (which seems doubtful). Otherwise we'll have a situation like speeding on the highway, where "everyone does it" at the risk of a random ticket, but in this case the fines are thousands to millions of dollars.
I don't see how it's bad: the hosting user receives no credit, and local mirroring keeps retrieval speeds high. Your users effectively become a giant CDN for you. Who would sue?
I wouldn't sue anyone. But I can watch music videos on YouTube without contacting all of the the relevant rights holders to ensure both the uploader and YouTube have permission to show me the video. On a distributed file share lacking anonymity and/or immunity, I have to ensure that I have those rights before viewing (thus distributing) any content.
Immunity is not likely to happen, as it would amount to granting redistribution rights to anyone who downloads a piece of content. So the political solution is out. Anonymity is a requirement.
That's not to say IPFS is useless. It's just not going to "replace HTTP" or any significant part of the web as long as it's possible to see who's sharing what. Or if it does, the sharks will feed.
No, that doesn't add up: You only host data that's already available on a public network, pretty much by definition. If the rights distributor didn'f want it to be globally accessible, they wouldn't have put it up on IPFS.
I'm happy to play the naivete card when it comes to downloading. (If the producers didn't want me watching this CAM footage of a new release on YouTube, they wouldn't have uploaded it, wink wink.) I'm a bit more cautious when it comes to uploading.
But this is very different: This is more like the original copyright holder putting something on BitTorrent, inviting people to download it freely, and them suing anybody who seeds it: By putting the data on a P2P protocol, you've kind of implicitly stated that you're okay with people seeding, or whatever the local terminology for the same is.
You have no guarantees that the content put up via IPFS is by the original copyright owner.
Thus immediately after you receive the infringing content over IPFS and start seeding it, you are infringing on the copyright owners rights. You have no recourse and no legal defenses to protect you.
Well, then, as soon as you receive notification, you can immediately stop hosting the data: it's like a DMCA takedown, but you have to send it to way more people.
From what I understand, the plan is to have blacklists that you can subscribe to, so if you don't, then you can be legally presumed to have knowingly pirated material.
This seems to be a flaw in IPFS. If it reshares content automatically then other users can infer what content you have viewed. If it doesn't reshare content automatically then the likelihood of content actually being "distributed" rather than "centralized" is low because few users are going to actively choose content to reshare while browsing the web.
As a devops guy, I sort of think ipfs seems more useful as a private, backend sort of solution where you trust all the nodes. I'm sort of vaguely imagining it running as a shared file system in AWS, running on docker containers.
Why do you think that's needed? Of course if majority of nodes cheated and said "sure, I've got that file" and send you random garbage instead so you have to retry, that would be bad. But if majority are running proper implementation, you don't really need to trust anyone.
IPFS is content-addressed. That means that it's trivial to discard tampered content. Lying majority, sure, will impact your content retrieval speed — but dealing with cheaters seems to be as simple as adding their addresses in some ignore list (which can itself be IPFS-shared, if needed).
I'm not sure of this blacklisting is implemented in current go-ipfs out js-ipfs codebase — but hash verification itself certainly is.
You need a reasonable number of nodes. Likely a large majority. Otherwise you'll just keep redownloading garbage data and discarding it. And likely doing it slower than bad nodes can leave/rejoin the network with new ips.
Frustratingly I couldn't reproduce this and the bandwidth-using part of the IPFS daemon appears to be completely opaque - i.e. I had no clue what was using all the bandwidth (other than the fact there were 200+ connections).
No visibility into an app means it's unlikely to stay on my system.
You seem to have glanced over the qualifier I used: accidentally.
Given a hash for an IPFS resource, how do you know it does not contain child porn before you retrieve it?
Once you have downloaded it you have committed a crime. There are no take backs with strict liability crimes and IPFS provides no anonymity, so there is a record of you downloading and seeding child porn.
> In IPFS you do not automatically relay things, you have to explicitly decide to.
Really?
Correct me if I'm wrong, but I was under the impression that immediately once you retrieve a hash via IPFS, you cache it and start seeding it.
> Given a URL for an HTTP resource, how do you know it does not contain child porn before you retrieve it?
As clueless404 stated, on the http web you don't automatically seed the content that you accidentally retrieved. On IPFS you're distributing it.
As an example, another posted here provided an IPFS link to resources, some of which are a copyright violation in some regions. In IPFS you can discover the nodes that are seeding it:
> Given a URL for an HTTP resource, how do you know it does not contain child porn before you retrieve it?
You don't, but then again regular browsing does not automatically download the illegal content, start seeding it and announce to the world that you have it on your computer. Big difference.
Also since IPFS deals with content hashes, you not only could and should, you absolutely must ensure any content you download via IPFS does not contain known instances of illegal content.
Private networks is something that we're working on (track it here: https://github.com/ipfs/notes/issues/146) but it would be private-private, only access+upload between your nodes. "Accessible by anyone - upload only by you" is not gonna be supported since it doesn't really make sense. If someone has a hash of a file, they can rehost it because they can access it. In IPFS you're not locating files based on WHERE they are but their name rather.
> Private networks is something that we're working on
Pretty sure you can already do it by setting up your a node on e.g. a Digitalocean droplet, removing all bootstrap nodes from its config file, and having everyone on the network replace the bootstrap node addresses in their config file with your server's address. Granted that's not a very user-friendly way to do it, but when I tried it it seemed to work.
Yeah, there is two things that is missing for it to be good. First is encryption of content in case something happens to leave the network and secondly rejecting the connections from anyone who is not supposed to connect to you.
With your current setup, if/when a node gets connected to someone else (maybe someone connects directly to you), your entire network will connect from that, since the connections will be shared with the peers.
Content distribution. Many people however don't even realise this is a problem, because they have been spoiled by free-as-beer services provided by Google, CloudFlare, Github and various other companies.
The idea is that some fraction of users will be configured to reshare content, which helps distribution for popular content scale. This seems to work in practice for bittorrent.
Yes, but only for suitably popular content, as all the dead torrents will testify. This doesn't make IPFS very appealing to Joe Blow, content maker unextraordinaire.
Regardless of whether I think this is a bit of a far fetched use case or not, this really does nothing for Joe Blow and his content distribution needs.
But let's roll with it anyway. How does IPFS solve this problem, and more specifically how does IPFS solve this problem any better than just publishing a torrent of Joe Blows collected ramblings?
I don't think that word means what you think it means.
Asking what problem a technology solves is a very practical question. Countering with "IPFS is a nice and fun technology", is the very opposite, i.e. alluding to that there might be some theoretical benefits to it, but that your mainly into it for gits and shiggles.
ZFS is less flexible in deduplication because it's not true content addressing, and git has lousier protocol. Rsync and unison also have lousier protocol and don't deduplicate storage.
There's an interesting emphasis on developing nations not engaging with the Internet, but I think that might be partially cultural too. What tools have we given the developing world to really engage with the internet? The easy-to-use publishing platform often require an email and usually a real name. Both of these things may be unavailable to countries where being connected to thoughts posted online could be dangerous.
Most content is not written in simple english, and there's just not much incentive for somebody who may not know how to think critically/complexly (due to lack of western education) to engage with the internet.
I think distributed web is an interesting idea, and that IPFS really lists out some issues with the internet that we'd all win in solving, but I think maybe some of these, like developing nation web access, are solvable with current tech, and more culturally based solutions
I think the better solutions come from within the society, rather than being imposed from without (Cuba's flash-drive sneakernet). An outsider might provide the gross implementation, but it should be up to the locals to work out the messy details, even if it means they don't connect to the net, as a whole, on a regular basis. It's their lives on the line, they're going to have the best ideas on how to preserve them. An outsider is going to be, at best, somewhat ill-informed and at worst, inimical to the solution.
I think it'd have to be saved statically. Which actually makes this question interesting: How do you store comments on an IPFS site? Constantly updating a single file on IPFS?
Interesting question. I know nothing about IPFS other than what I've read in the past five minutes on Wikipedia, but:
1. HTML wants badly for a nested relational document format. Essentially tin or mutt's "in reply to" and "references" headers.
2. A comments stream is a) a parent document to which b) multiple child documents, themselves possibly having c) parent-child relationships, d) are associated. Rather than thinking of "threaded discussion == single document" think "threaded discussion == a set of related documents".
That gives the option of having a discussion "occurring" across multiple sites, with some form of trust, whitelist, blacklist, or other mechanisms for reflecting what you do or don't include in the discussion. Individual comments, as their own docs, could also be freestanding instances.
Finding children from the parent becomes an interesting question.
There's also the matter of versioning a document. Tying a git-like capability to this could be interesting.
Interestingly some links to copyrighted material end in ¨Unavailable for Legal Reasons¨ however, running the daemon and issuing an ¨ipfs get hash¨ the download does start.
Hey I have a related question: so with IPFS we all host bits of the internet, and with IPv6 our machines are all directly world-accessible, right? So how do we prevent this from turning into a huge pwn-fest? If routers aren't doing NAT and a bit of firewalling along with that, would each machine be completely responsible for its own security?
> with IPv6 our machines are all directly world-accessible
world-addressable != world-accessible
IPv6 just gives each device a unique address, which is orthogonal to deciding which packets you allow into your local network. In most situations, a stateful firewall (probably on a home router) will still block inbound packets by default, even if they are addressed properly to a valid local IPv6 address.
> would each machine be completely responsible for its own security?
Every single device is already and will continue to be responsible for its own security. A firewall doesn't remove the need for any security protections on each individual host. Border security adds defense in depth, which is great, but you have to assume an attack can bypass the firewall or originate from another host on the local network.
Assuming the firewall will offer sufficient protection and skipping the hard work of properly securing a device is how we get easily compromised trash like the cameras that recently DDoSed Dyn/etc.
Home routers are doing IPv6 firewalling by default, no need for NAT. NAT is strictly inferior to firewalling.
(Of course you shouldn't put things on your internet-connected network that need the firewall, just look at it as a porous defense-in-depth element, just like with IPv4)
Normal static address is assigned. Whether it comes from the external range or from the link-local range doesn't really matter. You could potentially randomise the source at the exit to prevent identification of your hosts, but you don't have to.
For most practical cases just imagine you're getting a big ipv4 range. Whatever you can do with it - you can do the same with ipv6. NAT, no NAT, filtering, static or dynamic assignment.
That's what I'm asking. IPv6 feels like a regressiin over NATv4 because it can leak which internal device made a request. Is there a standard way to randomize addresses that works with ofd-the-shelf router firmware. Also, are link-local IPv6 leaking MAC addresses?
Yes, there is a standard since about 10 years ago. It's not dependent on your router firmware, in accordance with IP's end-to-end design philosophy (keep the network dumb, and hosts smart). Here's some links to get you started:
As for link-local IPv6 addresses, those aren't even accepted by the socket API[1] in place of normal routable addresses. They're only used for low level things like neighbour discovery (IPv6 equivalent of ARP) and apps that go out of their way to use them. They aren't routable outside your L2 segment.
[1] as in:
$ telnet fe80::something:something:42
Trying fe80::something:something:42...
telnet: Unable to connect to remote host: Invalid argument
This sounds like distributed Geocities, where you can have any content you like so long as it's static, or at least, changes in iterations of static files like a HTML-generator blog.
If you do anything that needs a central server, suddenly its advantages vanish. I could imagine Wikipedia using this; I couldn't imagine gmail doing so.
What I'd personally like to see is built in monetisation such that hosting and serving other peoples pages becomes a socialised cost and benefit although one would guess that such as feature would have to be deeply designed into the system itself, and cannot be added as an after-thought?
Connecting and indexing documents has been the challenge of a few internet generations. Creating a document at a point of filing is a subtle but potentially large shift.
Hopefully this lands on homebrew soon to aid it's growth.
I heard about IPFS at the Decentralized Web Conference in SF last spring. It sounded promising, long term. Anyone here using it right now? What are the costs for running it on a VPS, for example, bandwidth, storage, and CPU load?
ipfs is fantastic, but it is half the solution. We also need a distributed p2p application framework, with which nodes can securely communicate and allow building distributed apps, like search.
We can think differently with ipfs. Traditional web allows everyone to publish content somewhere, hoping that search engines will index it.
With ipfs, the same file (with the same content) is only indexed/stored once and then you reference the hash to get to the content.
This fact changes the problem of search.
Take all the world's movies. With ipfs + p2p network, you only need one back end in the form of a distributed search index, which can index all the movies in the world.
Same with the world's music. You only need one back end which can index all the music.
The index can be as simple as {"movie title": [sha256]}, where the array contains the hashes of different 'encodings' of the same content (eg. 'dvd rip', 'blue ray' or 'mp3').
Content can be indexed by all kinds of properties of course and it can grow organically over time to include more and more details.
With ipfs plus the p2p network we'll build 'apps', not 'pages'. People can have a list of 'apps' running on their machines - which are node instances in various distributed applications, sharing the same p2p network and using ipfs as storage.
Apps can have 'backend' and 'front end' parts - the back end is the part which participates in the p2p network, while the 'front end' provides a human interface to the back end, were users can search/browse/view the content.
Apps are distributed as git repositories stored in ipfs, while the 'core' running on the user's machine compiles the sources (inside a build vm) and loads the resulting binaries into containers running in virtual machines.
This would make it easy for devs to write and publish new distributed apps, making the network totally decentralised and virtually unstoppable.
Ps. If you feel that this insanity could work, then I'd love to discuss it in more depth - delegate78@gmx.com
I work with IPFS so it would be very handy to know what you feel is slow exactly. We're working our hardest to make it as fast as possible but sometimes we fall through, so more information so we can fix it would be wonderful!
Ah, I understand. Yeah, it's not much we can do about that but in the future it can be expected that you'll be able to use some sort of service to help you with the initial seeding of the content.
So instead of you sharing a file from your home connection and five friends have to share that, you'll share your file with some hosting provider and once they have the file, you'll just send the same hash to your friends and you'll have at least 2 nodes providing the file.
Everybody that pins it or retrieved the file from someone else (second one to maximum cache size that is configurable). Files and folders are referred to by hash.
Usually you still need to run a server to seed the files, but if a file becomes popular it will be served from anybody who also retrieved it and likely persist in the network even if your server goes down.
Think of it as BitTorrent for the web. It has Gateways that allow current browsers to access the network and pointers to update content with a new version (that's /ipns/ not /ipfs/ - the old version persists as long as someone hosts it).
What's also interesting is that due to the nature of the hashing algorithm (it's a merkle tree), you can "ipfs add" a directory, preserving the file structure inside, so websites on IPFS can use relative paths.
It's impossible to keep everything. A system like this one keeps the things people access, rather than the current system where content remains available only at the will (and expense) of the publisher.
IPFS appears here every 6 months, every 6 months the same questions get asked, the same problems get raised, the same collective sigh of bewilderment/disappointment appears to emanate from the comments, and it goes away again for another 6 months. Everybody wants something this clever and community-spirited to work, but the basic problem is, I don't want my data to be vulnerable to slow, unreliable endpoints, or people switching off their IPFS servers. I can't really trust an unremunerated volunteer system with my data, and I don't believe that my keeping your data is remuneration enough for you to keep mine forever.
Peer-to-peer is excellent for ephemeral streaming stuff like chat, file transfer, even gaming. But it is not good for permanence unless some monetary remuneration gets involved, either via a centralizing entity asking for payments (dropbox et al), or a distributed monetization system like bitcoin. Somewhere, somehow, someone needs to get paid to keep the system running.
...Which is why you keep your server up. It's not perfect, but it's still better than HTTP. And no doubt a cottage industry will spring up in paying nodes to pin specific data.
> IPFS appears here every 6 months, every 6 months the same questions get asked, the same problems get raised, the same collective sigh of bewilderment/disappointment appears to emanate from the comments, and it goes away again for another 6 months.
I would say the impermanence is one of the most appealing aspects of the system. Content that no one uses and that no one is willing to sustain gets culled from the network, what more could you ask for?
It is distributed. The website should have a copy at /ipfs/QmRaS4AZriMzw9nekub7hojTnvQYsVTDqkYG7BggQsexNt (check TXT record).
You just need to access it over the right protocol.
Right, ok, but then ipfs.io is already using ipfs-js itself? Because some time ago I checked and the reference impl was python, and the ipfs-js port was not complete yet.
js-ipfs, the Javascript implementation of IPFS, has made a lot of progress in the past 6 months. It's still early but totally usable.
We've been working on go-ipfs and js-ipfs interop so that browser nodes can talk to "native" nodes. It's not fully ready yet but soon. This will open a lot of doors for a more advanced network and applications using IPFS.
Is there a working ipfs demo page anywhere? I've had a look around and can't find one (the closest appears to be Orbit, but the ipfs-js version is down).
I'm particularly interested in using ipfs pubsub capabilities. It looks like an interesting way of allowing multiple users of a web app to talk to each other without needing (much) server side support.
Unfortunately there's no proper demo page :/ We're working on improving the docs (we know this is big issues atm).
There's an old version (from June) of Orbit at http://orbit.libp2p.io which you can try. Much has happened since and we're working on bringing the js-ipfs version Orbit back to a working state.
Can I just take a moment to talk about how happy I am that the IPFS team is actually taking protocol documentation seriously, allowing for multiple implementations and the ability to understand how the protocol works?
IPFS and Matrix seem to be the only teams doing this: Dat and SSB have vague protocol documentation at best, only documenting bits of the protocol at all.
