Wrong that OSM should be a service provider and not a database - I think it should be a database, and corporations are good for OSM. The post calls for OSM to essentially become Mapbox. It's insanely expensive to be Mapbox, and Mapzen just failed trying to be a less corporate version of Mapbox. And then the usage policy part of the article is moot or at least not that consequential if you don't believe OSM should be a service provider.
Right: Moderation tools need to be created/fixed. New mappers need to be encouraged/onboarded better. Vandalism is a problem. OSMF culture is toxic. Hidden gatekeepers are toxic. Imports need to be supported. Bots need to be enabled, and we need to consider people who contribute code/tech/imports first class OSM contributors and not just glorify the individual mapper.
Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data.
Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.
> The post calls for OSM to essentially become Mapbox
The problem is that most companies using OSM are using Mapbox. This leads to a few problems, including attribution, which I decided not to touch on, but is a serious problem that Mapbox doesn't address sufficiently.
If they could use OSM directly, they'd be engaged with OSM directly. They'd care about OSM. Right now OSM shoves these users away and makes them into customers of some third party (maybe Mapbox, maybe someone else) instead of caring about OSM itself.
> Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.
How about letting users experiment and seeing what cool stuff they can come up with?
> The problem is that most companies using OSM are using Mapbox.
Is that even true? And is number of companies even a good metric? Based on revenue/market cap, the bulk of companies use OSM directly.
The world's biggest company (Apple) uses OSM directly. Telenav uses it directly. Don't Tesla and Uber use OSM directly too? And we use Mapbox at Gaia GPS, but we also use OSM data directly.
> How about letting users experiment and seeing what cool stuff they can come up with?
In my experience building a company, if you focus on too many things, you get everything wrong. This smells to me of being not on target for making OSM as strong as possible, even though it has alluring upside.
In the post, he says this is ok for some (implying the giants you mention) but it discourages smaller companies. That second (and likely much bigger) set of people use Mapbox.
Well, "biggest" can mean a number of different things but, given that this is an Internet comment board and not an audited financial report, market cap seems one perfectly reasonable measure.
Market cap would be a reasonable measure for "biggest public company".
It's an awful measure for "biggest company", since private companies have no market cap. This was discussed a while ago in relation to Saudi Aramco's potential partial IPO. Saudi Aramco is huge, much bigger than Apple, with no market cap.
They absolutely do. They're just updated less frequently and called "valuations". Closely-held private companies also have valuations, they're just even less frequently made visible.
Its technically not "market cap" since there are no stocks and no public offerings, and therefore no "bet" on their future. A valuation and market cap are entirely different things.
> A valuation and market cap are entirely different things
In the way the Fahrenheit and Celsius scales "are entirely different things." Both "valuation" and "market cap" are measures on the aggregated estimated present value of future cash flows, i.e. net asset value, from/of a security.
Not to be rude, but do you even know what you're talking about? What does "biggest" company mean? Most employees? Largest land area covered? Or market capitalization?
Revenue? Wallmart gets that making pennies in every sale. And with 2.3M employees -- Apple does half that revenue for 1/20 the employees -- and has 2-3 times the market cap (Walmart is not even in the top 10: https://en.wikipedia.org/wiki/List_of_public_corporations_by... ).
I mean, I realize hosting a service is expensive, but can't there be an open source set of service APIs that's self hosted? That way you can chose to get a full copy of the data (and be responsible for pulling updates) and the API, or just pay someone else for access to their hosted version of the API?
can't there be an open source set of service
APIs that's self hosted
OSM does have open source self-hosted options.
Unfortunately, the developer experience is much more painful than using Google Maps.
With Google Maps, if you can add Javascript to a web page you're basically good to go. Data updates, software updates, server redundancy, server support and security, all taken care of. Even if you're exclusively a front-end dev, you're making a static site, and you've never heard of Linux.
Compared to that, self-hosting OSM tile generation requires a bunch of extra knowledge, equipment and maintenance.
Is it something that could be containerized? Maybe not as simple as a JS embed, but deploying a self hosted docker container is far easier than installing most things I've encountered.
With vector tiles one doesn't have to containerize anything - just shard them onto a bunch of servers and server them over https. Put an accelerator/cache in front and proxy to the desired region backend.
With image tiles it's a hurdle. You have to build 18 zoom levels of tiles. The most detailed layers have an astronomic number if tiles. You also need a huge amount if storage.
Did I mention the data being huge? What's the point of having 15Tb container, which takes forever to deploy? You can certainly containerize the services, but there's no benefit containerizing the data, since you have to replace it all at once rather than synchronizing the differences.
Depends on the degree of complexity of the tiles. We tried it (using tilecache or tilestache and mapproxy) and it becomes unresponsive when you rasterize several tiles (9). It can take several secons for a busy tile. If the user gets impacient like I do, zooms in and out and pans the map, it freezes to a halt. It's doable over clear water and maybe on the highway, but not in busy cities, at least not with hundreds of users requesting on demand tiles and without an insane amount of rendering resources. My impression is that it works if you're the only user and a really patient one at that.
Why not use vector tiles? I can generate a vector tile package of the whole planet in Esri .vtpk format and it takes up around 30Gb. A map service can be published from it easily. Takes me about 2 days for initial translation and about 1 week ti generate the vtpk. Neither of these have been optimised for speed.
I have an old Galaxy S3 upstairs running an app that displays this vtpk - so I can have the entire world down to house level on my phone.
> you can chose to get a full copy of the data (and be responsible for pulling updates)
Even if you could make the general setup and maintenance incredibly easy somehow, just syncing the map data from osm regularly would require decent hardware: the extra cost of each dev doing this themselves just doesn't compare to using a central tile host.
On top of that, if every dev was regularly updating their own OSM data, OSM would have a lot of extra strain on their data download servers to deal with. Not sure if this could be comparable to the hardware overhead of providing a tile hosting service though.
> just syncing the map data from osm regularly would require decent hardware
We need servers with at least 24 CPU threads and 256 GB RAM and moving the data into PostGIS, region by region takes a week or two. Plus you manually have to fix the process when it just spits out an error and aborts after 15 hours of churning.
Agreed. I can load with osm2pgsql on my home PC (AMD FX 8370, 32gb ram, 1tb SSD) in a couple of days, and that's generating 78 output layers. imposm is supposed to be faster still.
> Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data.
I have a hobby site about railways [0] I'm picking up again after a decade of inactivity. I wanted to display a route map of each railway, with clickable stations and the route highlighted.
I can not work out how to do that. OSM has the route and station data (there and in Open Railway Map [1]). I expected to find a layer called 'Railways' with sub-layers, one for each railway, and I could export the data and highlight it on another map.
Instead I have raster tiles, and no way to highlight the railway I'm talking about. I know the data must be there somewhere but how on earth do I get it?
The type of data you want for routes is probably in KMZ format. I took a few minutes looking around openrailwaymap and their wiki but nothing immediately obvious stood out.
A brief google search came back with this; I haven't inspected the data or even tried to use it to see what it contains, but the page description indicates it may not be exact routes you are looking for-but it's a start. Implementing the data is a bit more work in order to display route maps, something like the previously mentioned MapBox does this very interactively.
My email address is in the profile, feel free to shoot over an email if you'd like some help with it. Depending on the complexity, I can probably devote a few cycles this weekend to show you a one-off map with railway routes, and show you how it's done.
KML/KMZ is decently useful as an interchange format and simple rendering but almost all web GIS software has better support for GeoJSON. It’s richer and easier to work with.
I’ve never worked on a GIS system that used KML/KMZ in prod for rendering. It’s usually shapefiles, PostGIS Geometry, ESRI’s proprietary GDB feature, WKB/WKT or GeoJSON on the backend being served up to clients.
I have used it as a lightweight way to deliver simple data in the past though for use cases where there was no proper client renderer (we would just tell the end user to download google earth and open up the file)
Thanks for the offer, I got 75% through writing you an email and finally managed to get some data exported via https://overpass-turbo.eu/, so I'll work on that for now. Thanks though!
In some sense there is a layer, features related to railways will all have a railway tag (or be listed as a member of one of those features). There's some description here:
In the sense of being easy to retrieve I guess it isn't a layer. Global railway data will be pretty huge, so Overpass-API may not work out if that is what you want. In that case, the strategy is to grab a planet file (a snapshot of the db) and filter it locally:
And then you probably have to translate the resulting data into a format that you would use for your map.
As to why you have to do that work instead of just pulling layers, it comes down to the core data model being focused on editing and there being many different end use cases (making it difficult to anticipate and serve each one).
OSM is essentially a database of points and lines, each with a (loosely defined) soup of tags attached to it. Highly flexible (annoyingly so, if you've got a single focus area) and not in a "layer" format.
Investigate Overpass and in particular Overpass Turbo.
And really, any strict layering system in OSM would quickly just fall apart into debates of exactly what belonged in each layer. The range of detail captured by OpenStreetMap users in different areas is really far too nuanced to work crammed into coarse, crude "layer" classifications.
How can that be the case? Even with a range of detail, the points on a map have to represent one of a fairly limited selection of types. Even if you keep it high level like "buildings", "roads", etc. it's still a massive improvement over having no layers whatsoever.
> the points on a map have to represent one of a fairly limited selection of types
Absolutely not. Not only is the selection of types huge and full of shades of grey, but an element can be tagged as things from multiple different categories at once (and this is a wonderful feature). Take a browse of the OpenStreetMap wiki sometime. OpenStreetMap data is far richer than a layer model could cope with.
However, let's not confuse this with an editor having a layers-like interface feature to help the user focus on a particular subset of the data. JOSM for instance does this very successfully with the "filters" tool, which I use heavily.
In OSM objects have zero or more tags, a set of key/values (with both being strings). That's all. With that, the OSM community has come up with lots of ways to classify and tag things.
I'm currently doing something very similar for my school project and it's been quite a journey, the OSM ecosystem sure isn't simple.
My current solution is based on OpenMapTiles and uses their stack for creating, and subsequently rendering, vector tiles. First, I obtain an extract of OSM data via the Overpass API. Then I convert it from XML to protobuf, as it's a much more efficient format. Afterwards I import it into a PostGIS database using imposm3 and define some SQL functions that transform that data. Then I generate some vector tile definition from the OMT format into tm2source format. After that I finally generate the vector tiles in mbtiles format using tilelive, server them as GeoJSON using tileserver-gl and style them using maputnik and mapbox-gl styles.
There seems to be some bug with imposm3 and relations, because I can't get them to import. Or I'm just not doing it right.
You make the point OSM should be a database and not a service, but that’s a false dichotomy as nothing inherently prevents it from being both.
The main point against a service seems to be that’s it’s an expensive thing to do, however I don’t see how we can draw any conclusion from that without weighing the costs against the upsides.
I don’t see how the expense argument can be made with confidence without first establishing some basic data points. For example:
1) How expensive is expensive?
2) Have the benefits and advantages been throughly explored ie what’s the value proposition?
3) Given this value proposition, what level of fund raising success would be plausible?
As an aside, the ineffectiveness of leadership and how it is seemingly blocking progress is quite striking.
Off the top of my head, I can’t think of any actively developed software with a non-deprecated api that hasn’t changed in so many years.
OSM has gained countless new capabilities since moving to API 0.6 in 2009. World-class, super-fast routing. Beautiful vector maps. A much easier raster map API. A powerful and performant query language. To say that progress is being "blocked" is not at all accurate.
Sure, these aren't provided by what's formally known as the "OSM API". They are provided by OSRM/Graphhopper, Mapbox GL/Tangram, Leaflet/OpenLayers 3, and Overpass respectively. Because that's, deliberately, how OSM works: it's the centre of an ecosystem, not a single service provider.
The OSM API is for the use of map editing programs. It's an implementation detail for the authors of iD, JOSM, Vespucci and a few others. It isn't something that anyone who isn't writing an editor needs to concern themselves about; and I honestly haven't heard many comments from editor maintainers that they want the API to do things it doesn't at present (though, Bryan, Dirk, Simon, do jump in and correct me if I'm wrong).
Do you have any connection to any company involved with OSM tech, just so I understand your perspective?
Mentioning OSM has gained functionality is irrelevant, the article’s point is the API which serves a different purpose. What other companies offer is irrelevant that’s not OSM.
Your other points seem to attempt to diminish the importance of the API, yet the state of modern software has become such that almost any significant software project can benefit from a rich, well implemented and maintained API.
Not to mention maintaining a high quality API often in and of itself leads to improvements in the overall architecture of software, due to how it encourages careful thought about the purpose and organization of the system.
The "OSM API" is an editing API, for editing the map.
The OSM ecosystem has gained some new, much more powerful APIs, which Doctor_Fegg mentions. OSRM's API, GraphHopper's API, Overpass's API, these are all better than the APIs that were available in 2009
Again I would ask, how does that invalidate the article’s premise that the core software API hasn’t changed in ~10 years?
What other significant, widely used software, depends only on the “ecosystem” for APIs development, while having no active API efforts in the core project?
I can think of none. Maybe some examples exist somewhere, but at the least it’s unusual, especially given the overall context of the article.
The core editing API hasn't changed in years. HTTP/1.1 come out in 1997 and HTTP/2 in 2015. Was web programming in 2014 the same as web programming in 1998?
Have you ever designed and implemented a modern api? It doesn’t work like that.
HTTP itself is fundamental and a different level of abstraction. We would not want monthly updates.
A closer analogy is Wikipedia (see wikimedia api). It hasn’t had the fastest evolution, but it makes OSM look like the stone age. You could look at the github api as a project that’s invested more frequently.
And it’s not just because these are projects with vast riches (even though WP’s multi-million budget is actually tight). There are countless smaller projects that keep a well maintained API. I just list these because they are well known.
The role and importance of APIs in software has changed dramatically over the last 20 years. It’s true at one point it was more of a luxury and just wasn’t as central to many projects.
Those days are far enough in the past and so anachronistic that a ~10 year stale API at this point should be considered architectural malpractice.
This discussion might be more productive if you laid out what you believe the OSM API being discussed already does and what you think it should do in addition to that.
And then you could lay out why those things should be done centrally rather than on separate servers. For example, there is a sophisticated data query api https://wiki.openstreetmap.org/wiki/Overpass_API that keeps up with edits within a minute or two but is free to lay out the backing database in a way that is optimized for queries and people can run their own instance and so on.
Oh yes, I agree. In fact, I’m taking a close look at the architecture, organizational efficiency and results to form an opinion of the bigger picture. Indeed, I haven’t, scratch that, can’t, even assert an opinion on the article as a whole without doing so.
Then how is this productive you ask? Because for now I’m only drawing much smaller conclusions, which also generalize to most actively developed software. Rich, well maintained APIs tend to be useful. As of yet I see no real explanation of why that doesn’t apply to OSM.
To start, not sure why you mention a central server vs. “separate servers”. SaaS users shouldn’t care, or ideally even need to know whether something runs on 10 or 100 servers.
Maybe instead of separate servers, you meant different companies? Knowing which organization is the provider of each service is paramount. For each provider it’s also important to know something about their business model, reputation, roadmap, and the pros/cons of using multiple providers vs. one stop shopping. It’s important enough that companies like Gartner are paid a truckload of money by SaaS users to analyze these details.
So back to my original point: Most significant and active projects benefit from investing in rich and well maintained APIs.
Take that fact, and add in a domain expert taking the time to write an article asserting it’s indeed a problem relevant to OSM. One person’s opinion is not always a smoking gun. But combined with the fact a 10 year old stale API is so unusual, and the fact that what other companies do is a red herring argument being made, at minimum it sure seems like a bright red flag that warrants better understanding of what’s going on over there.
I mention a central server vs. “separate servers”. because that's pretty much the only way you could come to the incorrect conclusion of 10 year old stale API.
There's a particular service running on some OSM servers that provides data to editing software and receives changes from editing software and applies those changes to the master database. The primary version number on that api is 10 years old. It has still evolved over that time. But it is not the entirety of the OSM API, it is the OSM editing API, which probably benefits from being at least somewhat stable.
I mean, I guess the basic data model could be changed all the time so that all the tooling had to chase after it, but the only big proposal along those lines is to add an explicit "area" type which would simplify some things (it wouldn't really create room for any new capabilities).
Take a look at the vector tile stack Mapbox has put together if you want to see a new way of working with OSM data that has come about over that same 10 year period. Mapzen and OpenMapTiles have also worked on vector tile stacks.
Mapzen tried to be a less corporate version of Mapbox but the quality of their data/tools was just not there.
I had to implemented a specialized geocoder/administrative boundaries referential last year. Mapzen was really compelling, the documentation about how they handled hierarchical data was clear and interesting but execution was lacking. I am French, handling European data was important and I quickly hit several pain points:
- Their github repository organization was a mess. A lot of small ones without a clear map of which did what.
- The vagrant setup to experiment locally was unmaintained
- I found a basic unicode normalization issue in the geocoder several hours after playing with it.
- Manipulating their dataset via git-lfs was a massive pain
- Their geocoder was nicely presented but relations to administrative levels was done with neightbourhood search to administrative area centroids, instead of (even approximated) geometries.
- The data was not as comprehensive and up to date as OSM one, at least in France. For instance, France second level administrative areas (Régions) were redesigned at the beginning of 2016. They were defined in OSM but progress seemed fairly slow in Mapzen.
Now, I appreciate Mapzen was a very ambitious project and handling cartographic data at this scope and scale is a difficult job, but as pure data consumer, it was not comparable to OSM.
I eventually used OSM data and while their documentation/tools are quite a mess too, wiki style, it is workable. Datasets are large but there are a lot of way to retrieve them efficiently. Their non-protobuf binary format hits a sweet spot between size efficiency and ease of processing. And at least in Europe, the data is fairly good, at least for my requirements.
> Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.
Maybe I'm misunderstanding the whole domain, but if OSM would get layers, that would lead to better composability, then wouldn't the question of transient/frivolous data kinda become moot? The way I see it, in that scenario I could make and host my own whatever pokemon layer and OSM proper could provide the base layer, and then they could be easily merged at usage point; whereas based on the discussion now without layers you'd need to essentially make hard fork of the OSM data to make your own pokemon map?
We wanted to contribute forest data on Africa back in 2015. It wasn't possible. We'd have to add it via JOSM or the web based editor. So we gave up since it was a no go. Now we just discard the OSM forest data and add our own when updating the map data.
> It wasn't possible. We'd have to add it via JOSM or the web based editor.
That's strange. What were you trying to do? If you convert your data to OSM's relatively simple XML format, you can open that file in JOSM, and just press the upload button. That's what most people do with imports. It's possible to communicate with the OSM API directly and make API calls, but that's essentially what JOSM does. JOSM also has a validator, which can spot some problems with your data.
Are you not violating the licence in this case? Or do you make the new dataset available under the same licence? I think the licence is one of OSM’s big problems.
Transit data should probably be supported in a meta-sense. A machine readable bounded region with annotations about where to fetch transit data and how to parse it should likely be added. That way the OSM database links to external data sources.
I'm the DRI for most of these problems. Random thoughts:
I think David Ogilvy said something like "Search all the parks in your city, you'll find no statues of committees". Everyone was super keen to make OSM in to a web of committees, and this is one root of the problem from an organizational point of view. It's not super clear to me what the OSMF has done in the last (to pick a number) 5 years. (I don't regard "doing the same thing as last year" as relevant to anything, I just have a personal bias towards new things).
From a technical point of view we merged sysadmin (who inherently want stability) with development (who inherently want change) and stability won (it doesn't involve arguments or as much work, I guess). That's why the API looks the same way I designed it X number of years ago.
It's still a wonderful project, but more of a work of art than anything else. It's not like wikipedia didn't have all the same problems. Making a project that really evolves and grows over large periods of time is really hard. Like - most of the things on the S&P 500 won't be there in 10 or 20 years or something.
I think what really happens is that projects go through a lifecycle and something new comes along to replace them, rather than anything getting fixed. I'm not sure if this is a good example, but, you could try to "fix" Apache, or you could just do this new thing, "nginx". For all kinds of reasons, the new thing is more efficient. One of the few counter examples I think is going to be Amazon.
The world has moved on from making vector maps and OSM solved almost all the related problems (just not geocoding, which annoys me no end). It moved on, because of OSM! All the money and interest in maps is now around autonomous navigation, which means 3D maps and other things outside of OSMs area.
I just spent several minutes of my life trying to figure out what a DRI is, and I still don't know.
I agree with you in regards to the OSMF. There has been a lack of motivation and a lack of desire to experiment.
> From a technical point of view we merged sysadmin (who inherently want stability) with development (who inherently want change)
Agreed, operations and development are often at odds. I've done both, and there are organizational ways to solve this, but they require a great deal of work.
> It's still a wonderful project
I completely agree. A prominent OSMer just sent me a message saying how he never wants to hear from me again, but this is not me writing against OSM, but rather me getting down and dirty into exactly where I think OSM doesn't do a good job.
I can love something (especially software) and advocate for it to change.
Steve, you and I don't always agree, but I think we both agree that we want the project to succeed and are frustrated by the current situation.
> I think what really happens is that projects go through a lifecycle and something new comes along to replace them, rather than anything getting fixed.
I'd hate for this to be the case, but it seems that if the project can't move forward, this is what will happen.
It would be possible to start by stiching together data from existing libre data sources, and then using tools to augment them. I'm not a computer vision expert, but I bet much of what we (you and I and many of our bretheren) were doing tracing roads and buildings could be done better by computer.
BTW, if you're Liking my post- like the parent post, since Steve is the primary founder of OSM. I mention him in the post.
(I agree with most of your post by the way (except for the bit about it being art, which would mean it has no function except to be art), I'm just unclear on what you meant here.)
Yeah that's why I don't go for those websites, there are hundreds of meanings for each TLA and you'll rarely know which one was meant (both because you don't feel like scrolling through that many, and because more than one usually fits). I made that mistake once or twice before.
...A-and we still don't have a single decent map people can use. And positively undoubtedly we don't have a single decent map people can use, that is owned by society and free in a sense Wikipedia is. I use Google Maps on a daily basis, and I hate Google.
> Everyone was super keen to make OSM in to a web of committees, and this is one root of the problem from an organizational point of view.
Oh come come Steve, you know better than me that's not true. There is one genuine committee in OSM (OSMF board, which does nothing, which is probably a good thing given how well their attempts to do something work out) and three things that are nominally committees but in reality just a small number of self-motivated volunteers doing thankless grunt work (Licensing Working Group, Data Working Group, State of the Map [conference] Working Group).
OSM is about the least hierarchical, least committee-fied project there is. Compare to Wikimedia - or, you know, any form of government - if you want to see what committees are really like.
> "Search all the parks in your city, you'll find no statues of committees"
This doesn't necessarily mean that committees did not play a vital role in various historical events, just that people don't like to make statues of them.
Agreed. While the quote has a certain elegance to it, and certainly some truth, I suspect many people who have statues were on committees of some sort.
For all their flaws committees are a primary way in which people pool their collective experience and effort which is ultimately how anything significant is built.
Open maps used to be the job of the USGS. They stopped doing it because of Google.
I was touring the USGS in Palo Alto about 10 years ago, and the librarian there was lamenting the existence of Google. He said that while it was nice that Google was doing all this work, it was causing the USGS to stop doing it, since they could get the data from Google.
He predicted that the USGS would get severely cut in their funding for mapping activities because of it, and he was right. He was worried that all the best data would be locked up in Google's servers, and he was right about that too.
I used to work in the building with the USGS map store. It was very sad to see it close. It wasn't commercial map providers that caused it to close, it was the avilablilty of USGS topo maps online, and a faster data collection to map pipeline. You can still order maps printed on demand directly from the USGS.
I remember just a few years ago, I got a new phone and was interested in replacing as many Google apps with OSS maps that I could. (K9 instead of Google's mail app, etc.)
Naturally I turned to OSM instead of Google Maps. But I was really surprised by how strange and difficult it was to install and use. As the author mentioned, OSM itself doesn't have any apps, which is very confusing. After some research I figured out that you have to download a 3rd party app. The 3rd party apps that do exist all use different terminologies, require downloading huge amounts of data and picking what maps you want ahead of time, and had different levels of functionality. The UIs were universally terrible (at the time at least).
Needless to say I quickly gave up on using OSM and turned back to Google Maps--and I'm their perfect use case, a skilled techie with a deep interest in using free software! I can't imagine trying to explain how to use OSM to my mom, even if I somehow managed to install it for her.
I can't comment on the rest of the article but its usability worries are on point.
> Needless to say I quickly gave up on using OSM and turned back to Google Maps--and I'm their perfect use case, a skilled techie with a deep interest in using free software! I can't imagine trying to explain how to use OSM to my mom
My mom got an iPhone gifted from my dad, so there's mistake number one (I don't know of any decent OSM apps on that proprietary platform), but for what it's worth, my non-technical girlfriend has no trouble with it. Then again, that was 2014ish, not sure what time you're talking about.
> require downloading huge amounts of data and picking what maps you want ahead of time
It's not easy to find, I agree, but you can use it without downloading a single map. If being online all the time floats your boat, you can do that (at least in Osmand, the de facto app for Android).
> The UIs were universally terrible (at the time at least)
OsmAnd invested a lot of time in this. I didn't mind the old UI that much and the new one has some things I don't like (but are probably easier for newbies), but it might be worth a try again. Especially if you live in Europe, and definitely if you live in the Netherlands or west Germany, the map data is much more complete than Google's on everything but business information.
Maps.me is pretty killer, though. The only issue I see with OSM is lack of traffic data. Generally when I drive with Google Maps, I already know the route but want Google to step in when there's traffic.
> Generally when I drive with Google Maps, I already know the route but want Google to step in when there's traffic.
You should try Waze. I too mostly use Google to route me around traffic, but Waze is an order of magnitude better at it. It has more frequently updated traffic data and is willing to take you on side streets.
For example the other day I was driving down a major road, and Google said to just stay on it, but Waze took me down a side street for two blocks and then back to the main road, saving me four minutes, which was 25% of the total trip time.
Also, when making a long drive, Google chooses a route that is optimal right now for the entire route. Waze accounts for the fact that it will be an hour later when you get to the second half of your trip, and uses that to make better choices.
Waze is pretty great but it has a habit of running you through hospital parking lots to save 2 minutes or turning multiple times to avoid a intersection that isn't really that busy.
What I wish Waze would do is have a mode that allowed me to learn a route. Maps, and Waze both 'optimize' so much that its sometimes hard to learn a route from point a -> b because every time you go it sends you on different paths.
Many of the path optimizations seem to save you _seconds_ and when I'm driving I don't care about seconds, I care about minutes in greater quantities than 10.
I feel that Waze quite literally reflects the community - i.e. it's not that Waze is trying to shave off seconds by some sort of math analysis of possible routes, but rather that it's showing a shortcut that quite a few people actually are using and that's working for them.
Google might route me through a turn that's theoretically allowed according to street signs but takes forever to make in heavy traffic; Waze might route me through a turn or "street" that's theoretically not usable but all the locals are using it anyway.
If it knows routes used by locals, that means your positioning data is always sent back to Waze, meaning Google.
I'm using Here for longer trips. Google has the habbit of making questionable routing decisions. We ended up on roads that looked like after bombardment using it.
If you run them side by side, you will often find that Google Maps does not try as hard as Waze to save seconds. It prefers routes that are easier to learn, as you yourself like them, with fewer turns (and thus fewer interruptions). It caters to the 90% of people like you that don't obsess about seconds. Of course, it also has get traffic into account. Perhaps, in your case, the intersection isn't that busy, but traffic information is noisy.
I think there must be a factor like local data quality: I stopped following Google Maps as closely because it does things like an unprotected left turn on a major arterial road to save 100 feet going to the next light.
I remember trying to use Google Maps in San Francisco at rush hour. In terms of utility, efficiency, and practicality, it was indistinguishable from some bus-riding programmer's idea of a practical joke.
No, Google, I am not going to make a left turn here, because I have someplace I need to be by Wednesday.
Another problem with Waze, by the way, is that the algorithm is designed to route you away from a busy intersection or a congested street without regard to whether or not going out of your way will take more time than just slogging through the street/intersection. Waze also likes to do stupid things like insisting you make a left onto a six-lane divided highway without a light.
And their pathing is just bad: it insists on cutting through a gated community to get to my place, something that's physically impossible. And even worse, the entrances to that gated community and my townhouse complex are on different arterials. It also doesn't recognize the entrance to my townhouse complex as a valid entrance too (if you try to enter via the entrance, Waze will shriek at you to turn around, get back on the arterial, and find the gated community). I've also seen it straight-up get lost before when trying to go to one bar that's near my office.
Because of this, I long ago decided that whenever I request a Lyft, and I'm going either to or from home, I will call the driver and say "My name is Amy, I requested a ride. I'm calling to confirm that you are not using Waze.", and I will hang up and cancel if they say they're using Waze, if they don't speak enough English to understand the question (I live in a pretty diverse city, so this is common), or if they demand why I'm asking and then start angrily grilling me about every detail related to my destination.
And, yes, the third scenario is sadly common. I have never seen people get angry in real life like Waze partisans do except where actual religion is involved... imagine the worst kinds of vim/emacs warriors, Linux/Mac/Windows fanboys, PlayStation/Xbox/Nintendo/PC zealots, iPhone/Android partisans, etc., except in real life instead of on the Internet. It's actually why I started calling ahead: I used to just wait for the driver to get here, get in the car, and then as soon as they start the navigation and Waze comes up, I would calmly say "please don't use Waze, it can't get to where I'm going". The reaction was, more often than not, angry and aggressive. They treat "Waze can't get to where I'm going" as an attack on their lifestyle. After dealing with abusive drivers who will call me a liar to my face, screaming "that's the only thing I have!" at the top of their lungs, getting violent (I had one lady who punched her steering wheel while screaming the aforementioned), and sometimes getting straight-up kicking me out of the car (my response is always "Gladly!"), I've learned to call ahead and filter them out, as I have no desire to come within 10 feet of nasty, aggressive religious warriors. Oh, sure there have been plenty of Waze users who will let me navigate them manually (I can navigate from my office to my home in my sleep) or who will let me open Google Maps on my own phone and listen to the voice nav from there, but the number of angry fanboys and fangirls with no emotional maturity means that I'm not willing to risk it.
And it's because of the aggressive religious warriors who I'm afraid to share a vehicle with, and to a smaller degree because Waze's pathing sucks in general, I've started to call ahead to make sure they're not using Waze even when I'm not going to or from my house.
Google maps does update route in real time. They also integrate Waze data since they own them. Waze was taking heat for routing onto residential side streets which makes me wonder if Google is taking a softer approach. Interesting that you mapped your route with Google maps and Waze at the same time and compared them. I'll have to give that a shot.
They update in real time, but Waze predicts traffic, which is better. For example, in the Bay Area, if you are coming from Sunnyvale and going to Berkeley, you could go up the peninsula side or the east side. GMaps might tell you to take the peninsula route, because right now, the bridge is clear. Waze however will predict that the bridge will be crowded in 40 minutes when you get there, so it will take you up the east bay side.
GMaps may notice the bridge is crowded when you get there, but at the point it can't reroute you, because you're already on the wrong side of the bay, so it will just update your ETA.
What I've found is that the Waze ETA is almost always accurate when I start the trip, whereas the GMaps one keeps updating as I go.
I actually find the opposite. :) Google Map's estimates tend to be close for my work commute (within 5-10 minutes).
Waze on the other hand will often give me an estimate that's 20 minutes faster than Google Maps, and with the traffic that eventually pop up on that route, take me 20 minutes longer than what Google Maps estimated.
In my area Waze does a better job at anticipating future traffic patterns. For example, every Thursday I have to get my son from school and drive across town starting at 5pm. It's a 20 minute drive if there's no traffic and 30-35 minutes typically and we only have 40 minutes to get from A to B. Google will pick a route that is fast at 5pm, but builds a lot of traffic at the Nike headquarters by the time we reach it 15 minutes later. Waze seems to understand this better and has never routed me into that traffic jam.
Is Waze's traffic really updated more frequently? Depending on the area, Google Maps can be down to roughly once per minute. (Open a view at a busy time of the day and figure how often the traffic data refreshes.) Perhaps it's a matter of reacting more rapidly to new data.
Yes, Waze is willing to take you on side streets, by design, just like Google Maps isn't, deliberately. Waze is Gentoo, Google Maps is Ubuntu. One is tuned for the quickest drive, the other for directions that are simpler and thus easier to follow or remember (e.g., it doesn't reroute you if it knows it will save you just 2-3 minutes).
Waze is extremely regional. I was among the first adopter and mappers here in Italy, and was involved with various misfortune with the local bureaucrats which locked all the dialogue and started making terrible, terrible decision for a mapping software: for example to save mappers time they decided not to map service roads nearby larger mains, except in places like Milan if you need to know when to get from the main into the service road since not all left turns from the main are legal and not always the service road can merge back into the main.
As a direct result Waze was unusable in large Italian cities for the longest time, long enough that the gap in free turn to turn navigation that was once there got filled by google maps on ios and everyone forgotten about that experiment. They eventually conceded and started mapping all the details as needed by turn to turn navigation, but too little, too late.
This is more a matter of degree. Google Maps does occasionally suggest surface streets when there's major traffic, and will also suggest alternate routes during a trip.
In my experience, though, traffic has to be very bad for it to do this. Maybe it does it less often?
The underlying map data is the "real world", so there certainly is a mapping between the google and waze map data, even if the mapping is non-trivial :)
There are thing like OpenLR, which were invented exactly to solve these kinds of problems.
My experience with traffic rerouting is that you usually are 5% faster than just waiting. You're driving, which feels nice, but it's also stressful because you're already late and of course the country roads are now full of people routing around... Aside from major accidents, I think traffic info is quite overrated.
Not that it wouldn't be nice to have in OSM, but it's just not something that fits into what OSM is. It's a whole separate service, even if such a service could be integrated with the third party apps that use OSM data.
I've found OSMAnd quite buggy and temperamental, plus it heavily pushes you to the paid version if you want up to date maps (so you could effectively ending up paying a third party to access your own mapping work unreliably).
Although I've had similar experiences, have you tried it recently? It seems to be much better. And the F-Droid version is nag-free. I've not found a more flexible mapping program. Unusally for an OSM program, it even supports online tiles - I have mine configured with offline maps overlaid on Bing satellite imagery if data is available.
I use the paid for product but, from what I remember, you get the same maps with the free version. The difference between the two is that you're limited to 10 (?) maps download in the free version.
10 downloads through the app. You can download them yourself from their website and place them in the correct directory, and it will just work (and this is allowed and even mentioned somewhere).
They changed this sometime in the last few months. As well as reducing the limit to 7 map downloads total, the maps paid users have access to are now updated faster than the free ones.
What. Reading this thread I was thinking maybe it's time to give OSMAnd another go. Currently using a combination of Maps.me and Google Maps.
But with only 7 map downloads I might as well not bother. Even just the Netherlands is more than 10 map downloads and you probably want at least a piece of Germany too.
On the one hand, OSMAnd felt like it was more accurate or well-behaved in some sense. But the interface was just so confusing (a few years ago), there were a couple of unlabeled icons for its major functions that were (to me) just too generic and could mean any of map/directions/route/search/location related. Even after a long period of use I found myself confusing these icons, having to tap-and-try and navigate (hah) myself through the navigation app.
... so did I understand correctly that the F-Droid version of OSMAnd does not have the 7 map limitation either?
It's $10 normally, $5 right now! Not expensive for anyone with a European income (And the same for many other regions).
Not wanting to pay for that is fine, whatever, but it's not exactly over the top for the people building the app and running the map infrastructure to charge a fee.
That's not quite true. You pay a small amount once and you get it for life, and you can opt to support development and mappers by a really tiny amount (€1.20) monthly, but you don't have to.
> I can't imagine trying to explain how to use OSM to my mom
But why would your mum have heard of OSM and be trying to download it in the first place?
She might well have heard of a third-party app - in certain countries and communities OSM-based apps are massive and have significant brand recognition - and download that. Or she might be using a name brand like Apple or Facebook or Garmin or whoever, all of whom use OSM under the hood.
OSM isn't a consumer-facing brand and doesn't aim to be.
I had the same experience a few years ago, in the end I stuck with Google Maps, with Maps.me as a backup. But ideally, I'd like mapbox to release a Google Maps like app, as their maps are just stunning compared to anything else that renders OSM data. (I havn't yet found any other company using mapbox data to create a useful map I could use, usually it's just built in maps like uber)
Exactly. If the alternative is Google Maps (or even Apple Maps), it’s not good enough to make me pay $5 to download a map of 1 country and forcing me to keep 500mb of data on my device permanently.
It shouldn't really matter if I want to use it with some other application on Android or to test some routing libraries, the point is that it's not easy to get.
I work for a company that syndicates location data for businesses. We had a couple of businesses that wanted to syndicate to OSM.
It. Was. A. Nightmare.
We have addresses for every store. We don't know where the building is. Any store we place is going to be easy to place on a street near the building. If you've not placed it inside of the strip mall on the right spot, other people delete it. And what do you do with NYC? Suppose that you have a store that is somewhere on the second floor of a skyscraper. We don't know where it is in that building.
Have you tried to map out thousands of department stores? We know hours, phone numbers, and addresses for each department. We have NO idea what the actual store layout is. Without the layout, OSM basically didn't seem to want the rest of what we can give them.
If a customer is willing to pay us to deal with OSM, we're going to discourage them and ask for top dollar if they insist. It is that painful.
Are you complaining that OSM contributors rejected wrong or imprecise location data about your client’s stores?
If you are, I would like you to also consider this from the point of view of (maybe volunteer) contributors who prefer less data but correct data in OSM. Maybe draw a parallel with Wikipedia where a page that is not up to the standards might be deleted.
(Not a contributor to OSM myself, but I have helped the French Drupal community and cofounded the Museomix community. Also, I appreciate precise data in OSM)
No. I am stating that OSM has no setup for, or desire to, accept useful but incomplete data. And unless they do, they will not have data available that other services have.
I know where Google, Bing, Facebook, Yelp, Foursquare, etc get phone numbers and current business hours for our clients. They are straightforward for us to work with, and we are happy to give it to them. OSM is not easy to work with us, so don't get that data.
OSM has a choice. Be easy to work with, or have limited data. OSM has chosen to be hard to work with. That is a valid choice, but then you have to live with the result.
I don’t want to argue, I understand your position of trying to publish this data and being frustrated by the data being deemed not useful or in scope by other contributors.
I’m not an OSM expert at all — but I can guess that opening hours or phone numbers for a store that has no proper geographical location (and thus no geo-existence) would be perceived as not having its place in geographical database.
I suppose OSM is really a GIS (geographical information system) more than, for example, a tourist map, so it’s not totally absurd that volunteers would have issues with integrating data not properly aligned with the ideal way of working of a GIS (everything is geo-localized).
Google Maps has obviously diverging goals and incentives in that regards (serving ads, being an intermediate to stores).
As Google Maps and other online maps are trying to be many things, including the yellow pages, they probably don’t care about being an ideal GIS.
Open communities of contributors are often misunderstood when they enforce limits to contributions, and at the same time seek more contributors! But it makes sense when your success is defined on building a digital commons—scope and values is really what define the project, not much else.
(The Wikipedia community has to explain constantly that the goal is an encyclopedia, for example to people trying to use it for publishing their own findings, personal biography or fan pages. The Linux kernel won’t accept some driver contributions too.)
Of course it now seems that the OSM community of contributors is itself divided on what is the scope of OSM.
What qualifies as useful but incomplete data is relative (and to a degree -- a matter of opinion). If am told "there is a big mountain here, give-or-take 1 km" that's probably useful. But if I'm told "there is a water fountain here give-or-take 1 km" that is pretty darn useless.
Your data is probably not near as bad as that, but if you put a POI on a map, and somebody goes there, but doesn't find it because its a couple hundred metres away, they're going to assume that it moved/went out of business (and remove the data).
I can think of two ways around this. One is to make a note about the incomplete data you have and hope that somebody finds the actual location and adds it. The second is to go-ahead and add the thing with a "fixme" tag saying "location approximate" or something. I've seen both techniques used in the past.
No, that's the opposite of what we are looking for. We're not looking for data that other people have uploaded. We've got that data straight from the companies before it is publicly announced. We're publishing it to everywhere the companies want it published to.
OSM is not one of the places that we want to deal with providing that data to.
Hello. I am the creator of alltheplaces and Max is absolutely correct. Alltheplaces is a bunch of scrapers right now because it was the fastest way to get the data, but if you have data in a machine readable format already then I don't have to scrape it. I can just download your data file and distribute it to whomever wants to use it (people interested in importing to OSM, geocoders/search engines, etc.).
Comming from the other side I've dealt with a lot of companies trying to use Openstreetmap, and they are all really bad at it. Two things I say to everyone who want to import data to Openstreetmap. These seems to be very hard things for people to manage, do you have any idea why that would be?
1. just make it available for everyone to download, try to add as much metadata as possible.
2. make it clear what kind of license you have for this data
The broader point is that there's a structural issue here, as there's no ability to link data to addresses in a form that doesn't require literally knowing the location of a store within a building ahead of time.
I think in this case his criticism is merited. There is plenty of open data for street addresses available, and almost certainly already in osm. Instead of linking the most basic form of physical site identification -- which is basically a legal data set maintained by governments -- to their data import functionality/process, they just refuse data. Sounds off to me.
I am certainly not an expert on international data access regulations. But at the same time, there is more than enough of that information publicly available to make it a basic first step in a mapping application/dataset.
Open address has a good map that show this (http://results.openaddresses.io/), but for any area that is grey on that map you are going to be a long way from that "basic first step" you talk about. Adress data is hard, there are so many shitty collections with bad licenses.
Thanks for that link. I dropped their coverage chart into a spreadsheet to come up with 1,184,252,000 (surely estimated) people covered by their address data - not to mention covered commercial addresses, which started this thread. My point was that there is enough green on that map, and enough obvious, critical demand, to make it a pretty fundamental feature for a mapping process, not that it would work most of the time.
osm.org is one live database. There is no staging server - I'm not sure how there could be given the massive amount of free geodata in the world. So yes, stuff you should put in there should be correct.
That absolutely isn't to say that the OSM community isn't interested in your data. Plenty of companies have offered their data to OSM, and OSM volunteers have done the hard work of finding the correct location for each store (or whatever) and adding it in. As I write, the UK community is integrating data on every Shell petrol station in the country; the result is much better than if we'd simply dumped the raw Shell data into OSM.
Hi, we at Maps.Me do this for Brandify: we've recently helped them import 4k of Walmart stores across the US, and next week there will be a few hundred fitness centers. While importing third-party data is hard, it is possible, and I am working to make that a norm for OSM. Google "OSM Conflator" for details, write to iz@maps.me if you need help.
I've tried to build an open check-in app based on OSM data, and I've ran into many of the problems mentioned in the article:
• Tags and relations are a total mess. There are many different and overlapping schemes. There is obviously outdated/sub-standard data, but automated fixes are forbidden. There are some neat tools to make sense of the tag data, but ultimately I had to deal with all that mess and complexity on my end.
• The lack of moderation or a way to officially act as an intermediary for edits (other than clunky oAuth flow) meant I couldn't expose an Add/Edit/Fix functionality for my users. Similarly, if I wanted to let my users upload a photo of a venue, there was no service to store it, no identifier or tag to associate it.
• The API returns only nodes entirely within an area, rather than everything intersecting a given area. If you fetch small lat/long area, you won't get a node for the country/city/district it's in, unless you stumble upon a point in its border or label. That meant I couldn't fetch things from OSM API as needed, and had to have my own copy of the data and the DB and custom search.
• The map looks ugly without buildings. There was free building data available for areas I was interested in, but I was told to copy them by hand.
> The API returns only nodes entirely within an area, rather than everything intersecting a given area.
The osm.org API is meant for editing. It's not a map rendering and querying API. There are lots of alternatives for that.
> There was free building data available for areas I was interested in, but I was told to copy them by hand.
OSM is open to imported data. It just has to be done carefully because "with great power comes great responsibility" - add millions of items to the OSM database and you can royally screw it up (there is an argument to say that most of OSM's problems in the US stem from this). But there's a well documented imports process and lots of people with experience in importing buildings to help you through it.
Can you clarify which API I should be using for data access? My last project involved determining locations parks and rivers, and I ran in to a lot of the same problems. But, if I just want GPS coordinates, I'm not sure where else to go. Mapbox is find for rendering but less so if I want the raw data and use it myself (for a phone application for example).
The Overpass API should be the first call. Alternatively, you can download OSM data and feed it into your own database, which gives you much more flexibility to write the queries you want (http://download.geofabrik.de/ is a good place to download the data for the area you want; osm2pgsql and Osmosis are the two standard tools for loading OSM into a database, usually Postgres/PostGIS, though there are others).
In that case you wouldn't have to add the buildings to OSM to show them, just merge the data during your map preparation.
Nominatim does have a reverse geocode api which returns the sort of place information you want in your third point (given a coordinate, returns an address).
> There is obviously outdated/sub-standard data, but automated fixes are forbidden.
Please consider arms race encouraging automated edits would begin. A lot of people have a lot of awfully different views about how map data should be recorded (you thought source code formatting was contentious?)
> I couldn't expose an Add/Edit/Fix functionality for my users.
Lots of tools have managed this.
> if I wanted to let my users upload a photo of a venue, there was no service to store it
No. That's not what OpenStreetMap is. They aren't there to give out free image storage.
> The API returns only nodes entirely within an area
Yes, it's meant to be a low-level API which makes few assumptions about the meaning of any of the data. For the ability to make more complex queries you could have looked at Overpass. And possibly there is a space for a third party service to provide an API with such an ability. Again, I don't see that as OSM's business - OSM here should be considered more like the UN.
> There was free building data available for areas I was interested in, but I was told to copy them by hand.
And yet NYC have managed somehow to get most city building data into OSM, so perhaps you haven't investigated every possible option for getting what you want here...?
Thanks for posting this here! I feel like this is mostly good constructive criticism.
I have given a few talks about using OSM data in the US government and it comes down to authoritative data sources and validation. It's a great starting point, but there needs to be another layer (or two) of validation before it can be used at the enterprise level. The National Map Corps has some good examples of how this can be implemented with data stewardship.
I am also working on a few projects to take public domain data and put it into OSM. While it's useful and mostly open, it's definitely less open than the original data sources. Another issues is that as soon as someone edits the imported data, it will become less authoritative as well.
Like with Wikipedia, where you often hear that people prefer to cite a single person as source (because "anyone could edit Wikipedia", as if no single person can setup a website or write a paper), at least Wikipedia has had thousands of reads and often dozens or hundreds of reviews. People are scared of the idea "it could be anyone", while the most likely editor is just a peer with good intentions.
As soon as I started reading your article I actually started thinking of Jimmy and how he's been a part of a real government entity (the national park service) sending data directly into OSM.
I've found that in terms of supporting any kind of additional analysis, OSM data to be unreliable to the point of uselessness, which isn't to say there is a better alternative. I think this quote from your article:
"One of the most significant technical problems with OSM is the lack of a review model, that is for a change to the map to be staged and then reviewed before being applied. Not having this functionality caused ripples of problems throughout the system, some of which I'll discuss here."
is the primary reason I can't regularly use OSM data. Trust me when I say there are millions of instance where having reliable data regarding streets could save me time/ stress.
Do you have any thoughts on how to improve the quality of OSMaps? Conceptually, I feel like the product is indispensable. In practice, I feel like the product is utterly unusable.
As an occasional OSM editor, I wouldn't really feel good about moderation. My edits are limited basically to my home town (10.000 inhabitants, about 10 sq km) where I am adding unmapped roads (usually agricultural, sometimes new construction) that I "discover" while running. I started doing that simply because Runkeeper's website uses OSM and it was showing me running in the middle of nowhere, while there was actually a small road leading to my grandfather's place. (And FWIW Google doesn't have agricultural roads at all, which kinda sucks for running even slightly out of the beaten path).
My edits are probably not perfect, but while somebody cared about mapping the town in the first place, it was out of date enough that probably nobody would bother to approve or perfect my changes. Having them rejected would probably have caused me to stop contributing too.
What kinds of issues and vandalisms did you experience?
I've contributed small-time to the Musicbrainz cd database, and I think they do it pretty well. New additions are automatically approved. Edits are accepted or rejected after a voting period (3 days, iirc). If nobody votes, they're approved by default; I believe (but am not certain) they are rejected in the case of a tie.
I think the beauty of this system is that it gives complete autonomy with no oversight to individuals dealing with obscure data that nobody else cares about, while more popular data is subjected to higher scrutiny.
This is more or less how OSM works now, except there is no waiting period. But undoing a recent edit is pretty straightforward, so there is a similar effect.
I may be a bit late to this post but as a contributor to both projects, I think the main difference here is that MusicBrainz is all about trying to prevent data deletions in favour of data merges, which will be resolved to the new object while OSM pretty much only cares about the look in the end.
So in OSM it's perfectly acceptable to delete an object and re-create it whereas in MusicBrainz deletions are very much frowned upon.
This means that you often can't really track changes to a single OSM object which would be pretty much necessary for implementing robust rollback behaviour.
I used to work on user generated content in maps. Business vandalism is a big problem once you reach a certain scale. It's possible that OSM doesn't have this problem yet because it's not as valuable a target as Google/Apple/Bing. I'm not super familiar with what's available on OSM these days, but an unscrupulous locksmith might try to make competing locksmiths unreachable by moving their address to a distant location (locksmiths and taxi companies were notorious for attempted vandalism in the past).
Map additions are probably generally safe, although there are some spammy ones for various reasons. Map edits are basically all suspect because you never know when someone is trying to sabotage something.
Business vandalism is certainly a plausible threat for OSM. Something we've also had to deal with are political issues escalating into edit wars, i.e. where the border between countries is supposed to be, which language the labels on a contested part of the map should be in and so on.
I think it's good to keep in mind that OSM already has moderation, though, just not in the shape of a formalized approval process. Instead, edits go live instantly – but are inspected, discussed and possibly undone afterwards. So while vandalism does get through at first, it will hopefully be short-lived enough that it does not pay off for the vandal.
Ultimately, I believe it comes down to weighing priorities based on the project's current needs. OSM's current approach lowers the bar for contributors and rewards them with instant gratification when the change is visible on the map right away. Because of that, I believe it makes sense when your focus is on growing the community. Stricter moderation makes sense once the primary focus switches from growth to preserving the valuable database you've already built.
Looking at Wikipedia, for example, they started out with a setup resembling OSM's current model, and switched to a more strict moderation process only once they had grown large enough to become the world's default encyclopedia.
Based on my personal experiences in OSM, vandalism is not yet a big enough problem to justify stricter moderation practices. But of course that is a matter of opinion to some degree.
It's already a problem. Go look at the history tab for any moderately large town in OSM. You'll find gigantic edits overlapping the area you're trying to look at. The reason for that is someone posting changes/additions onto map locations on separate countries and/or continents, so it intersects with your small town regardless if the changes were near it.
Those edits are usually people adding useless businesses and phone-numbers. I've seen people advertising bitcoin-related business and general spam.
I'd be surprised if your edits would be rejected. More likely, whoever was doing QA would make a few minor alignment adjustments based on satellite data and any county data they could glean (the latter is likely where all the road data came from).
Moderation isn't the worst idea, OSM in many areas basically has large scale edit wars, wasting editors time as they fight over road and building alignment with other editors that don't realize Microsoft's satellite data is 20ft off from reality.
This was a hard post to make. OSM was a big part of my life for many years, and I have friends in the project. I like my friends (even the ones I disagree with).
But where OSM has failed most in my view (and maybe I need to add this to the article) is that there are now free geographic datasets from local governments and without the ability to integrate them into OSM easily, and then update them (which we basically can't do), I fear OSM is going to ultimately fail at its mission
Where do you have experience with OSM data anyway? Because your view of the project differs from mine as day differs from night. The quality in western Europe is equal to or better than commercial maps. And the one time I looked at a random, small village in the Sahara, OSM was also better: https://twitter.com/lucb1e/status/522491538912604160
Is OSM not doing this? I swear I've seen them importing and updating street level address data in my city - it directly referenced some city GIS server. Maybe not officially, but someone must be running scripts that do this?
Somebody might be doing that for their area but for example I haven't seen that happen in Estonia where the government provides absolutely excellent data but there's no easy and meaningful way getting that data into OSM
I would stress the "meaningful" part. I assume that every country in EU gives out CC0 (or so) licensed data for at least geocoding that have the same EU-wide high-level hierarchical format. Problem is that this hierarchy does not map well to OSM tags (which is one instance of problems caused by using tags instead of layers) and often also does not map well to local users' expectations (ie. converting the object names to/from handwritten address is non-trivial problem). Also sometimes the government provided data tends to be stale.
Stale or not, this seems like a ridiculously complicated problem without some set of rules/boundaries to govern it. I.e. You're doing a diff with spatial data that not only differs in the spatial component, but in the semantic/descriptive domain as well.
> I assume that every country in EU gives out CC0 (or so) licensed data for at least geocoding that have the same EU-wide high-level hierarchical format.
It's frequently a licensing issue and then also a resources issue.
There's relatively generalized tools for diffing data against OSM and outputting a change file, but lots of data is released using ambiguous or slightly incompatible licensing.
How would a layered maintainer model similar to the one of the Linux kernel work on a project like OSM? Have a Continent maintainer, a region maintainer, a country maintainer, a town maintainer, etc... all pulling only "good" edits. I think this approach could work, couldn't it?
If Apple has trouble keeping feature parity with Google Maps, there is no way that an endeavor like Open Street Maps has any real chance of making something with the breadth, features, and usability of Google Maps.
> there is no way that an endeavor like Open Street Maps has any real chance
Before making such an insulting claim to so many volunteers' hard work, get your facts straight. We did it.
In the few countries I have experience with the data (western Europe), the quality is great. It could be more complete on business information, but for the rest (think hiking trails, footpaths, cycling paths, etc.), OSM is more complete in every country I know of.
The data is also more current: when a tunnel nearby here was opened on a Thursday at 8pm, I checked the next weekend to see if someone had mapped it yet. Turns out, at 7am the next morning someone already did it. A few subsequent edits completed the thing. People here care and are jumping on projects like that. Google Maps, the winner (according to you) took a few weeks to get their facts straight. People drove wrong so many times to our place and we keep telling them, just use OSM... Some of our friends do, but not everyone. It's a marketing problem, not a lack of talent or manpower.
I'm honestly not even sure what half this article is about. I'm an active mapper and have yet to discover a single case where it's vandalism beyond doubt of stupidity. I've heard of cases, but never found any. Even stupidity, only in very few cases does it truly mess something up (like like disconnecting roads). I'm guessing OP is not from Europe, and local quality varies on the number of mappers, availability of open data, etc. That it's failing in the USA because it's a completely different world (people seem to think differently, more commercially) doesn't mean the project failed as a whole.
I agree that OSM is great for roads and on the whole at least as complete and up to date as gmaps. However when it comes to buildings and landmarks it's way behind in my experience (at least in Sweden). Having roads is great, but it would be nice if it also had my destination.
For me in Sweden that relly depends on where in the contry we are, most maps (except OSM) have imported a registry of companies (PRV or Skatteverket) that keeps the address for most small companies as the owners home address. So the restaurang shows up in a village a few miles away from it's actual location. This made gmaps and foursquare useless for finding the destination. For me OSM is better of having this right, but the Nomatrim search still makes one unable to find it :D
Justin O'Beirne's argument is essentially that Google Maps is much better at being Google Maps than any other map provider.
Which is true, but Google doesn't get to define "maps".
Indeed, Google Maps as expounded by Mr O'Beirne is the very antithesis of OSM. He argues that Google is unstoppable because of its technology: that it can derive building shapes, heights, etc. from imagery much better than anyone else can.
That's almost certainly true (though Facebook is doing interesting ML-based stuff in this area with OSM) but it's not what OSM is. If Google is the apotheosis of technology, OSM is the apotheosis of human curiosity. It's the collected works of everyone with a particular interest that can be recorded geographically. You're interested in cycle routes? Add them in. Long-distance hiking paths? Go for it. Detailed coverage of the area where you live? We'd love to have you.
So for cycle paths, for long-distance hikes, for areas where we have committed contributors, OSM is streets ahead of Google. It doesn't always work. We struggle with stuff that no-one is interested in (principally addressing, which is just insanely boring to survey) and in areas where people aren't motivated to add stuff (basically, poor areas: OSM is a very middle-class pursuit and this is our biggest problem by a long way). Still, OSM is pretty much the opposite of Google Maps, and any comparison will fall down on that.
(Footnote: in this regard, Google's biggest advantage is that they get to draw on the entire Google search corpus. For POIs, that can be the equal of the best OSM can do, and I don't think we can easily match it. That said, Google hasn't yet worked out how to scrape linear/polygon data, and their attempt at mobilising a community to do this - Google Map Maker - ended in fairly spectacular failure.)
Goes both ways: Openstreetmap is better for some things already, and I don't see Google catching up to it at them.
A big issue is consistency though: OpenStreetMap varies way more in quality between regions, to the point that detailed conversations almost make no sense without disclaimers about where you are talking. OpenStreetMap is a really good map where a strong community exists (either locally or dedicated remote interest) and a pretty bad map if no large effort has touched the area.
Getting to my house. I got tired of people ending with a "you have arrived" more than a mile from my house so I went to open street maps and fixed it. (the automatically imported data for my street had the street name wrong - I'm guessing google imported the same data instead of actually driving the back road). While I was at it marked a few of the roads as requiring 4 wheel drive (legal roads, but only used by farmers going to the fields along it), thus preventing people from getting directed to a road they can't drive on.
I checked the "I consider my work public domain" box, 6 months latter google started giving better directions around my neighborhood. I doubt this is coincidence. However the ability to fix directions to your personal house is something OSM does better than google today.
Display flexibility: I can pick a rendering that works best for what I want (e.g. optimized for high contrast or print, prioritizing/de-prioritizing specific types of paths, or public transport). If you consider derived maps, there is even more choice, e.g. I can get a topographic map based on it.
Usefulness while on foot. In well-mapped areas, OpenStreetMap has every little footpath between buildings, through parks and forests, ..., with information if it is publicly accessible or not. Google Maps is missing tons of them.
Offline functionality. (Google could change this, but I somehow don't see them allowing to download entire countries at once)
Another: dynamic updates - in case of emergencies and disasters OSM data has often been updated by volunteers faster than other sources to reflect the on ground situation. For current examples See:
Kurviger is a routing service for motorcyclists. Pick a point, select a direction, and a distance, and it'll generate a round trip that doesn't go on the same road twice, you can go direct, or sorta curvy, or very curvy roads. https://kurviger.de/en
You mean few small (relatively) communities did this for themselves, because it was obvious no one will think about them and in most cases ordinary user doesn't care much about their input. I am talking about an average map user, globally. If you want global coverage and relying on people to input data, then good luck. I do really like the idea and I wish with all my heart that people would be that generous with their time and effort but we have to face the truth - people don't care, people want a working and complete service. Unless there is some automation in place and free data exchange OSM will never be first choice for general public. Idea is beautiful, execution... Well, Tokyo city, population goes in millions, buildings and addresses not available farther than few blocks away from the train stations. You can see roads but roads in Japan have no meaning. And yes, I could input my house (just 10 minutes from station, blank point on map) but sad truth is, I am unable to input all of them and as you can see - people don't care. They just use Google Maps.
Means - if you have an address, road name or topography won't help you. Most roads don't have a name, anyways. Not sure about US but in EU it's pretty common that houses are bound to roads so the address encloses road name and house number on this road. In Japan it is different. City is divided in areas, which are divided into smaller areas, all the way down to the building. Even locals often don't know the layout and are unable to tell how to reach the house on next square just by looking at the address.
True, at least for the moment, although that doesn't mean we shouldn't try.
One nice thing about open data is it can outlast. What do you think will survive and be readily accessible on the internet longer, Wikipedia or Quora? My money's on Wikipedia 100%.
With OSM things are similar. Obviously geo data has additional ways it can grow stale, but still. Google is big today but there's no guarantee for the future.
Anecdotal, but still:
In Paris, new streets had been created in the 13th district. One was an important administrative destination for Parisians (building permits admin), allée Claude Levi-Strauss.
OSM had the street in its database from the start.
It took Google Maps more than a year (?) to catch up, and consequently in 2015 for example an Uber driver at the time would try and drive you to the opposite side of Paris where a museum had a garden of the same name… not good when you had an appointment with an official there :-)
The why doesn't Apple do what other companies and industries typically do in a situation where there is one dominant player? Get everyone else to join a single open solution and then start attacking Google for not being open with its maps.
Get Microsoft, Tom Tom, Samsung - everyone else - to work on OSM.
Right now, Apple is basically building the "Windows Phone" of map solutions, because it "wants to be different", and making the same mistake Nokia did in smartphones.
They should get over their own egos and actually choose the option with the highest chance of success against Google Maps.
Microsoft and Samsung have both made pretty big investments in OpenStreetMap. Samsung funded Mapzen for a couple years and Microsoft has long provided both permission to use their aerial imagery and the server resources for that to happen.
If it's to their advantage, it will happen. See the AV1 codec, which Apple joined along with other industry giants as an alternative to patent-encumbered video codecs.
> there is no way that an endeavor like Open Street Maps has any real chance of making something with the breadth, features, and usability of Google Maps
But does it need to? Isn't there a market for basic mapping, without all the needless feature bloat?
I'd rather a mapping service concentrate on getting roads right, and not whether there is a Howard Johnson's nearby.
Most people are using those roads to get to somewhere, be it a HoJo or a street address. So the geotagging ability is very, very important. I suspect a map system with vastly superior routing information (just roads and coordinates) and slightly inferior geotagging would still lose to one with the opposite balance.
Im my opinion the biggest problem is the data model (or lack of). Tags i.e. key-value pairs are used to add properties to geometry. Mappers use different combinations of keys/values to describe the same thing. The post touches on this, but does not offer solutions other than adding "layers".
There should be a strict schema that would be community managed through some process.
Like what, tag proposals where people vote yay or nay? We have those. As an active mapper, I very rarely find two ways of tagging the same thing, so the process seems to work fine.
And if you do find a situation where there are multiple options, it takes only a quick check on taginfo to find the more commonly used one. If you go with that, it should converge soon.
Currently there is nothing to stop mappers using tags in wrong ways. It should just be flat out impossible. If a field/tag isn't defined in the schema, it could not be used. The range of values should be limited in the same way, too.
Ah, I see where you're going with this. It would slow innovation down to a trickle, though. Anyone who wants to do something that's not yet standardized would have to stash their edits somewhere and wait for official approval. And that would likely take weeks or months at best. Currently, the best idea wins automatically because good ideas are used by both mapping applications and mappers.
Maybe something in the middle, like the current wiki system but with an enforcement rule of whatever is on the wiki. Anyone could freely edit it because it's a wiki (adding allowed options, and removing/changing any that have extremely few uses (zero, or perhaps a few which are then removed)), but at least there is a correct spec.
If people really had to adhere to a strict, slow-moving standard with pre-approval, the spec would also become extremely complex and very difficult to implement, because the spec-makers will want to think of every eventuality. See any other large specification ever, like HTML or something. I can't see that ending well. I think a hybrid system might be better than a strict one, given the choice between those two.
I like thinking about it though! Always good to have hear ideas.
Which is great only if you have all the right tags. A grain bin is not a silo - to a farmer this is an important distinction - the rest of the world cannot tell the difference, both are round towers.
This is an extremely thoughtful and constructive post. Thanks to Serge for making it.
I looked closely at getting deeply involved with OSM a few years back. And bounced off hard because the community is infested with assholes. There's a lot of very rude people who are active on the mailing lists that set the culture. It made me uninterested in trying to work through those cultural problems to improve the technical project. I'm not alone in this feeling.
Thank you for your contributions and writing this. Just spent a week of development time 'fixing' Nomination reverse geo-code issues and creating a pay for RGC backup because we exceeded the 1per sec max and had NO idea when it would turn back on. To boot, no help whatsoever determining why when we used it successfully for over a year. All of a sudden NOPE, we're cut off with foggy reasoning and no explanation of why or when we'll be back up. I've spent several months on maps. All of them. I know them through and through. I could sleep speak API docs and pricing for mapping services. We need OSM. OSM needs many of these 'fixes'. ASAP. No idea who was running things there. You gotta work on your analogies ;), but yeah if a user doesn't know how much the total usage is and can only use a percentage of that unknown total, it's ridiculous. Let's get a premium account going at resonable rates. Same for reverse geocode. No more than a 1ps max. Ok so what if we have a surge. Error 3. Give me a break. And so brashly the docs state this almost warning deterring any user from ever wanting to integrate because there is a big ol' flag saying 'we DO NOT want to scale with you!' At a bare minimum how about you average per second rates over a 10min aggregate. Soooo much room for overhaul and improvement here. This is a big start. Number one on HN! Let's fix/upgrade the antiquated OSM tech, logic and direction. It's a new world and the world needs OSM.
You can install Nominatim om your own hardware and then make as many queries as you want. There are also companies offering hosted Nominatim (e.g. geofabrik), which will come with a limit and a way to see how much you have used.
Previously I really like the flexibility of the tagging system in open street map. I used something like it for a unrelated commercial product and we have found we have to switch away from it. And we had very tight control over the definitions of the tagging we were using.
The tools were difficult, as is mentioned in the main post. That is certainly an inconvenience but I figured we could muddle through that. The bigger problem is that the customers just never got it. They wanted a single, authoritative classification for an object.
Our plan is to move to a system inspired by the model used in BIM for classification. The princicple difference is that in open street map all data identifying an object has to lie within the tags. In the alternate system, a single identifier for the object type is used, and this references external metadata that gives you additional information which does not have to be embedded in the tags on the object. An example of additional information held external to the map and tags is that a bathroom is a type of room. I don't have to include in the map data that the object is both a bathroom and a room.
I should add, the external meta data also gives a strict definition of what types of tags (and maybe values) are allowed on the given type of object. If you need something a little different, you can make a new type.
I am not sure how well this type of system could work in open street map. Is it too rigid? Or would there be too much proliferation of types? Classification is always hard and I am not sure people have solved that problem in any large context.
And of course, changes like this would mean a new database. If people ever do think about making an OSM 2.0 maybe this, or another tagging alternative, is something to consider.
I like the sound of this, like an object graph of data from OSM.
> I am not sure how well this type of system could work in open street map. Is it too rigid? Or would there be too much proliferation of types? Classification is always hard and I am not sure people have solved that problem in any large context.
Ontologies are well-researched. Unfortunately they're not my area of expertise as I turned away from doing a PhD in them and went into business.
I have been playing with a controlled vocabulary since Christmas for a personal project and the complexity of definition has made me think this is why Google et al have gone for a more flexible, machine-learned typing, and why the types for schema.org are so broad.
Having been negative, I think this is exactly what OSM needs to organise the world's objects.
OSM gives people the tools to create their own map rather than offering them a simple, out of the box solution
On this point and the main part of the first argument, I kinda disagree; the OSM project should specifically aim to do that: providing means for one to create a service.
However, providing an actual map service would cost much, much more in bandwidth and I doubt a free software project like this has that kind of funds - actually serving tiles can add up to large costs pretty fast if enough people use the service.
So if you want an actual service, you either do the hosting yourself or go see mapbox/whatever.
Locus (not FOSS) supports vector maps and it's price is way cheaper than an external 1TB drive would cost, just for good map coverage of a few countries.
Are there any sites where you can use it with current data? As far as I know Mapbox only sells it as a service to other companies, they don't have a site where users can browse it. The only site I know that use Mapbox is Strava, and they (or Mapbox) are slow to update the data. I made edits months ago that aren't available yet. Still, the data is much better than Google's for runners and cyclists.
On my Galaxy Note 2 (2012), you're right. I've ordered a new phone yesterday because it's getting quite bad after five years of daily use.
My girlfriend bought a phone for €300, and the one I ordered (now, a year later) for €250 actually has the same chipset. It runs great on that.
I consider ~250eur to be the bottom of the smartphone market unless you're going for a potato to call and text with (not even browse the web). Given that it runs smooth on that... I'm not sure what you're running OsmAnd on with expectations of fast rendering.
> I consider ~250eur to be the bottom of the smartphone market
That's ridiculous statement in itself. The bottom of the market are those <$100 china wonder phones, and you have a selection of low-end, but actually branded, devices in the <150eur category. Surprisingly enough, those <150eur devices look like they might be perfectly usable, with specs comparable to my current daily driver (which is couple of years old). As a random example, Huawei Honor 6A [1] currently retails for 99eur around here
I'm also interested in the development of OSM (being an editor and a data consumer) and I've witnessed first hand some of your issues. For instance, once I've had to create my own "geocoder" for transit fines in São Paulo using OSM data, because no geocoder commercially available would be able to deal with the data.
So, now that we clarified I'm not a troll, nor a bot, I ask you: how should we deal with those issues? Join en masse the OSMF so we can vote for a change? Begin the "New-OSM-But-Better Foundation"?
I also think having layers (particularly, layers vouched by governments) would be awesome. I saw several governmental branches in Brazil wanting to be more integrated with OSM - this would allow them to stay "inside" OSM (and being an accredited citizen), instead of creating their own silos, importing data from time to time.
I can understand the frustration of OSM users that they have to create their own maps, but providing free tile servers would never be possible with the restricted funding for OSM.
FWIW I don't really think any of these problems are "new" and don't think OSM is truly in any more trouble than it's ever been. And as many tensions as there obviously are in OSM, it hasn't stopped us getting pretty damn far. And if anyone thinks that OSM may be stagnating I would suggest getting involved in improving their local area (it's not hard!) and I think they will quickly find out for themselves.
IMHO internet projects, especially free data-centric projects shold just be made as granular as possible. There should be a project to maintain the actual data base and independent projects to display and integrate this data. And as far as I know this is how it actually is in the world of OSM though the actual projects are not being developed and marketed actively enough. I may be wrong - I have never really used OSM (I prefer the proprietary map project native to the country I live in - it does the local job much better than both OSM and Google), just read about it a number of times.
One thing that made me a bit nervous to start editing is the lack of an easy way to roll back changes, at least I couldn't find one using the iD editor. If I make a mistake and save it I need to do new edits to try and fix it, possibly making things worse, instead of just removing the entire changeset. I had some issues with multipolygons and had to edit the same things maybe four or five times before I got it right.
And why isn't the Strava heatmap easier to use? They allow OpenStreetMap to use the heatmap for tracing [1], but doing so in the iD editor requires you to copy a long url from the wiki and use a custom layer. This only shows the heatmap, no aerial photos or other mapping data, and it disappears when zoomed in too much.
Ok, good to know. It seems mostly academic though, since using their iD fork with the slide add-on [0] is a better user experience, and someone from Strava mentions in the Github issue that it's approved internally. Of course I'm no lawyer and I understand that proper licensing is important, I'll just use their fork for now when tracing their heatmap.
I see that you're the iD maintainer, are there any plans on adding the slide functionality? Or b-splines which could get tessellated client side before saving to the database? I can take this opportunity to thank you and the other contributors, I probably wouldn't edit OSM if it weren't for iD.
This was a wonderful post, thanks for the honesty and detail, emacsen.
What I'd like to see as far as a technical overhaul, is reimagining the entire thing as a 3D dataset. Needing to model a paper map is no longer the goal, modeling space in 3 dimentions seems to be the more exciting/useful way forward.
I'd like to have a model of my town in 3D, either at mostly manual low-poly resolution (aka minecraft) thats done by hand, or (though much more data intensive) with point cloud/lidar data.
Web pages can receive the 3d data and render a 2d model, or the server can project 3d onto 2d tiles before delivery.
OSM is too complex for the majority of businesses. I know two that chose to pay Google for their mapping needs rather than jump through OSM's hoops.
Imagine if the only way to get Linux on your computer was to download and compile the whole thing from source yourself, instead of buying or downloading a distro.
If OSM had a mapping solution like Google's, it would thrive. And just for fun, it should undercut Google's pricing.
In that sense, perhaps the linux distro model actually makes more sense than centralizing power into one entity?
Ubuntu, Redhat, and many others produce their own polished, supported products, sell it, and then give back to the central parts of the community in the process.
Mapbox and others certainly seem like they essentially play the part of "OSM Distro" in that regard. And maybe its okay for multiple entities to be able to contribute in different ways.
And kind of like Linux, you have a patchwork of unsatisfying solutions.
Mapbox isn't always cheaper than Google. They have not lowered their tile prices in years, for example. And on geocoding and some other APIs they give out poor results compared to Google. Do they have competition? Cloudmade went in some car-related direction. Mapzen never managed to have a real business plan.
And your "excellent small providers"? Yeah, Andy may be able to create some bespoke solution for you if he has time, and Geofabrik can wrangle some host-it-yourself setup onto AWS, but it's still very true that OSM themselves don't have a good story on a lot of mapping needs. You can either browse their web map with unusable search or download the 38GB database, and that's about it. No wonder many businesses just pay Google rather than dive into the rabbit hole of OSM.
I believe that OSM should deliver data, and not a service because it's extremely data-demanding.
Although I have to say that since OSM offers data, they could make an effort into providing good exports of that data, with better segmentation and choice of format. You should not have to download planet.osm or use some overpass API. The choices are very limited, and offering data through XML is insufficient.
It doesn't entirely make sense to mirror the editing api, unless you want to fork OSM. And I guess a lot of the user data isn't really readily available.
More directly answering your question, most of the configuration of the various servers backing OSM is available here: https://github.com/openstreetmap/chef
I got criticised when I was a young coder for writing long rambling emails and articles. Now I am older I see why. Too much info in the world, I don't have time to read that although I am interested in why this person thinks openstreetmap is in trouble, but man a wall of text. This is why scientific articles have extracts at the top, bulletpoints work well too.
I wouldn't mind seeing a two-tier approach, with authoritative municipal data as a base (see https://openaddresses.io/ for an example of collecting a large number of sources and normalizing it) with user-submitted edits on top.
I personally believe that world needs a "wikipedia" for maps, not another hard to understand tool or technology. Yes. Technology matters of course, but not everyone is an engineer or entrepreneur.
Re: the issues with moderation, I sometimes wonder if Wikipedia were trying to start from scratch in the last few years... would it work? With all of the trolls and bots, easy access it might have died hard and early.
One of the most significant technical problems with OSM is the lack of a review model, that is for a change to the map to be staged and then reviewed before being applied. Not having this functionality caused ripples of problems throughout the system, some of which I'll discuss here.
Moderation is expensive, and Wikipedia got by initially with a pretty minimal degree of it, then ramped up as they got bigger. What if they’d been slammed by modern trolling at the start?
Just to meta-meta this conversation and the article in new and confusing ways, Wikimedia also hosts a map server, using OSM's data, @ https://maps.wikimedia.org/ :)
True. I sort of meant "similar"in a wider sense: large scale, community effort to do something useful. Wikipedia represents a lot about what we that the internet, and internet culture was.^
Remember how big a deal Wikipedia is. A lot of that "human knowledge at your fingertips" is Wikipedia.
Maybe that applies at a bigger level, with "similar"things being implemented as part of something. I suspect that a big part of the reason Wikipedias don't get started now is that they get implemented as startups.
Anything showing promise on that scale is a "unicorn" and in 2018 Jimmy Wales would have had a choice between private/startup and internet-owned. That's a billion dollar choice. For example, stack overflow could have been built like a Wikipedia. If it was done in 1996, that would be a sensible choice. In 2018, a founder would be forgoing an unthinkable sum of money, and also resources to make it happen with.
^I guess all this needs to be caveated with "other than OSS."
It's an interesting question of how much flexibility you give people at the beginning vs as a project matures, but I'm looking at the situation with OSM pragmatically, that is to say "What OSM needs to do now", rather than "What they should have done".
The problem of both bad edits and vandalism are very large (much like Google Mapmaker) and other measures we've put in place, such as changeset comments (which I helped develop) and vandalism tools don't seem to cut it.
What about some kind of automated approach to validation? A major difference between OSM data and something like wikipedia is that, unlike wikipedia, at least some of the OSM data verifiable. A street somewhere, does exist, in a particular spot. There must be some way to croudsource validation.
Ultimately it comes down to the resources to create such a thing, right? There have been some successes with crowdsourced validations such as Facebook's translation efforts (although I think in that case it was merely a component of the process). And it would rely on a volume of edits that OSM may not have. There may be 1000 people ready to verify the location of the Statue of Liberty, but probably not the location of every building in a remote area being mapped for HOT.
Mapbox has built the start of a validation framework, and they have done very cool stuff. Of course then they get criticized for "forking" OSM. There are definitely valid criticisms there, but it seems like there's no way to win.
I didn’t mean to distract from the main point of your piece (which was a good read btw), it just made me curious in general. Maybe the only way OSM can work is by treating moderation/curation as the problem to solve?
It appears to me that there was a golden age of crowdsourcing that lasted from maybe 1999 to 2010.
If you started a project in this time, you had the chance to get lots of quality contributions, for free, from people whose motivation is just to be part of something.
Now, you will get:
- Modern trolling, as you point out. Well-organized armies of trolls, some working for political purposes, some just for the lulz, can subvert anything.
- Disruptive bots (I don't know why this is a change since the golden age, but maybe it's simply that more people are better at programming)
- Spam that's a full-time job to moderate
- Lack of motivation because there are so many other fascinating things on the Internet
- A change in expectations because money has entered the equation -- crowdsourcing is now something that people do as a menial job, and they aren't paid enough to do it well
The sabotage by 4chan of the crowdsourcing team in the Pentagon shredder challenge [1; sorry about the stupid title] makes a good indicator of the end of the crowdsourcing era. That happened in 2011.
I run a project, ConceptNet [2], that has obtained a lot of its data from crowdsourcing. Years ago, I stopped supporting the ability for the public to edit ConceptNet directly. (This does not mean the project shut down. Far from it. It currently produces state-of-the-art NLP models.) Supporting public contributions became obviously not worth the effort. Some projects that started at the right time and maintained excellent communities are still working, though -- Wiktionary would be an example, and it's one of the places that still provides new data to ConceptNet.
that kinda overestimates how long it took wikipedia to get to where it is today. when wikipedia launched, no one knew what it would grow into. why would large amounts of trolls bother harassing wikipedia when in all likelihood it would have just gone nowhere.
Is there a real hurdle in patents on map graphics held by the major mapping companies and national organisations?
Could a goal of being truly open and complete, be incompatible with the interests of privacy, national security even?
If community involvement is lacking, I immediately thought about the cimnunies who have real interest in mapping and particularly free open mapping, such as conservationists, residents of small towns concerned about planning and in my home Sussex county, hikers always fighting losing out to farming interests over ancient byways.
But these constituencies need tools. I would immediately donate to such efforts if I knew they existed. Or a corporate like ACDSystems, who publish Deneba Canvas which while sadly lost adrift far from the software I remember, incorporated superb GIS and technical drawing capabilities. This would be the kind of marketing that traditionally worked very well, tools and support for local news stories, which use to run a long while anyway.
He already commented recently under a repost of his former article. I've pointed out the good things about OSM.
He proceeded with answering to my comment but completely ignoring the content of it while advertising this piece to come. I'm still unable to find those good things in this article which supports my theory of overdrawn and almost(?) total negativity and ignorance of the good aspects of OSM.
Wrong that OSM should be a service provider and not a database - I think it should be a database, and corporations are good for OSM. The post calls for OSM to essentially become Mapbox. It's insanely expensive to be Mapbox, and Mapzen just failed trying to be a less corporate version of Mapbox. And then the usage policy part of the article is moot or at least not that consequential if you don't believe OSM should be a service provider.
Right: Moderation tools need to be created/fixed. New mappers need to be encouraged/onboarded better. Vandalism is a problem. OSMF culture is toxic. Hidden gatekeepers are toxic. Imports need to be supported. Bots need to be enabled, and we need to consider people who contribute code/tech/imports first class OSM contributors and not just glorify the individual mapper.
Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data.
Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.