On the game I'm currently working on, it's built very heavily around Lua. So for the save system, we simply fill a large Lua table, and then write that to disk, as Lua code. The 'save' file then simply becomes a Lua file that can be read directly into Lua.
This is absolutely amazing for debugging purposes. Also you never have to worry about corrupt save files or anything of it's ilk. Development is easier, diagnosing problems is easier, and using a programmatic data structure on the backend means that you can pretty much keep things clean and forward compatible with ease.
(Oh, also being able to debug by altering the save file in any way you want is a godsend).
You probably know this, but remember that storing user data as code is a place where you (general "you") have to think very carefully about security.
Is there any way that arbitrary code in the file could compromise the user's system? If so, does the user know to treat these data files as executables? Is there any way someone untrusted could ever edit the file without the user's knowledge? Even in combination with other programs the user might be running? Are you sure about all of that?
Maybe Lua in particular is sandboxed so that's not a problem (beats me), but in general this is an area where safe high-level languages can all of a sudden turn dangerous. Personally I would rarely find it worth it.
This is a good point, but I feel that discouraging this type of approach is not the way to go.
I apologise in advance for ranting... I hope this is not too off-topic, but instead a "zoom out" on the issue.
This touches on something deep and wrong about how we use computers these days. Computers are really good at being computers, and the amplification of intellectual capabilities they afford is tremendous, but this is reserved for a limited few that were persistent enough and learned enough to rediscover the raw computer buried underneath, and what it can do.
For example, I dream of a world where everything communicates through s-expressions, all code is data and all data is code. Everything understandable all the way down. Imagine what people from all fields could create with this level of plug-ability and inter-operability. We had a whiff of that with the web so far, but it could be so much more powerful, so much simpler, so much more elegant. All the computer science is there, it's just a social problem.
I understand the security issues, but surely limiting the potential of computers is not the solution. There has to be a better way.
Lack of Turing-completeness can be a feature. Take PDF vs PostScript. The latter is Turing-complete and therefore you cannot jump to an arbitrary page or even know how many pages the document has without running the entire thing first.
By limiting expressiveness you also gain static analysis and predictability. It's not about limiting the potential of computers, it's about designing systems that strike the right balance between the power given to the payload and the guarantees offered to the container/receiver.
For example, it is only because JSON is flat data and not executable that web pages can reasonably call JSON APIs from third parties. There really is no "better way" -- if JSON was executable then calling such an API would literally be giving it full control of your app and of the user's computer.
If you have a nice data format like s-exprs, it's a fairly simple matter to just aggressively reject any code/data that can't be proven harmless. For example, if you're loading saved game data, just verify that the table contains only tables with primitive data; if there's anything else, throw an error. Then you can safely execute it in a turing-complete environment and be sure it won't cause problems.
Speaking for myself, in my ideal world this sort of schema-checking and executing is ubiquitous and easy. Obviously that's not the world today. While there are tools for checking JSON schemata there doesn't seem to be a standard format. I wonder how hard it would be to implement a Lua schema-checker.
It's a relatively new data format designed by Rich Hickey that has versioning and backward-compatibility baked in from the start.
EDN stands for "Extensible Data Notation". It has an extensible type system that enables you to define custom types on top of its built-in primitives, and there's no schema.
To define a type, you simply use a custom prefix/tag inline:
While you can register custom handlers for specific tags, properly implemented readers can read unknown types without requiring custom extensions.
The motivating use case behind EDN was enabling the exchange of native data structures between Clojure and ClojureScript, but it's not Clojure specific -- implementations are starting to pop up in a growing number of languages (https://github.com/edn-format/edn/wiki/Implementations).
Here's the InfoQ video and a few threads from when it was announced:
I've looked at EDN a bit, even started a sad little C# parser. I don't see what it has to do with my previous comment, which is all about how schemas are potentially useful. I'm trying to say that after you check the schema, you don't just read the data, you execute it, and that has the effect of applying the configuration or just constructing the object.
>There really is no "better way" -- if JSON was executable then calling such an API would literally be giving it full control of your app and of the user's computer.
Of course there's a "better way": running the code in a sandbox. You could do so using js.js[1], for example. (Of course, replacing a JSON API with sandboxed JS code is likely to be a bad idea. But it is possible.)
You're right inasmuch as I shouldn't have implied that unsandboxed interpretation is the only option.
But my larger point still stands; the fundamental tradeoff is still "power of the payload" vs "guarantees to the container." Even in the case of sandboxed execution, the container loses two important guarantees compared with non-executable data formats like JSON:
1. I can know a priori roughly how much CPU I will spend evaluating this payload.
2. I can know that the payload halts.
This is why, for example, the D language in DTrace is intentionally not Turing-complete.
I agree 100% with you, but #1 isn't completely true. The counterexample is the ZIP bomb (http://en.wikipedia.org/wiki/Zip_bomb) Whenever you unzip anything you got from outside, you should limit the time spent and the amount of memory written.
1. imposing CPU limits incurs an inherent CPU overhead and code complexity.
2. if those limits are hit, you can't tell whether the code just ran too long or whether it was in an infinite loop.
So now if we fully evaluate the options, the choice is between:
1. A purely data language like JSON: simple to implement, fast to parse, decoder can skip over parts it doesn't want, etc.
2. A Turing-complete data format: have to implement sandboxing and CPU limits (both far trickier security attack surfaces), have configure CPU limits, when CPU limits are exceeded the user doesn't know whether the code was in an infinite loop or not, maybe have to re-configure CPU limits.
Sure, sometimes all the work involved in (2) is worth it, that's why we have JavaScript in web browsers after all. But a Turing-complete version of JSON would never have taken off like JSON did for APIs, because it would be far more difficult and perilous to implement.
I have to agree here. General Turing-completeness was known from the beginning to imply undecidable questions -- about it's structure, running time, memory and so on. I don't think this has a place as the 'data'.
Abstractions exist for a reason -- this is analogous to source/channel coding separation or internet layers. They don't have to be that way, but are there for a reason.
Someone could change my opinion, though. Provide me a data format which proves certain things about it's behavior and that would be a nice counterexample.
Join me on my crusade to eliminate the use of 'former' and 'latter' in any writing unless the goal is obfuscation. It's error prone, almost always requires rereading, and never is the clearest choice.
Here's my attempt at a clearer version:
"Take Postscript vs PDF. Postscript is Turing-complete and therefore you cannot jump to an arbitrary page or even know how many pages the document has without running the entire thing first."
Pronouns are fine. Substituting 'the first' and 'the second' would be an improvement. It's specifically 'former' and 'latter' that should be deprecated. I'd be interested in seeing a study comparing readers' comprehensions of the various phrasings. What cost in clarity would you be willing to pay?
Back on topic: The reason for PDF's existence is to be a non-turing complete subset of postscript. Features like direct indexing to a page are why Linux has switched to PDF as the primary interchange format.
In a world where users will willingly enter malicious code into their computers if they believe it will do something they want[1], can there really be a better way?
When it comes right down to it, you can't fully protect people from themselves. Even in 'meat space', which the general population is presumably experienced with, people talk others into doing things that they should not all the time. Anything from social engineering to bog-standard scam artists masquerading as door-to-door salesmen.
But in 'meat space', it is way harder and more expensive to do evil against large numbers of people. For example, phishing as done electronically (throw a very, very wide net, and hope) doesn't make economical sense if one had to do it manually.
Also, if, for example, cars and airplanes and banks and nuclear submarines would accept executable code as input, some people would do damage on a gargantuan scale.
Clearly, being liberal in what you accept must end somewhere. I argue that it should end very, very soon. Even innocuous things such as "let's allow everybody to read the subject of everyone's mail messages", if available at scale and cheaply, would entice criminal behavior, for example by those mining them for information that you are away from home.
Does anybody know how RMS thinks about passwords nowadays?
I'm not saying that scams in "cyberspace" don't present a greater threat than scams in "meatspace". I'm just pointing out that if you cannot protect people from themselves in "meatspace", then doing it in "cyberspace" is futile. You can fight it and cut back on it, but you will never actually win that fight. Technological problems to what is ultimately a sociological problem only go so far, and we should be careful to not obsess over them to a fault.
And yet, in "meatspace", chainsaws come with more safety mechanisms than butter knives because the damage they can do is so much larger. Yes, we can't win that fight; people will die from chainsaw accidents, but I disagree that we shouldn't be more vigilant about chainsaws than about butter knives.
This seems like exactly the problem the parent post is complaining about, though: the people in the limited group the parent talks about aren't the people being tricked by selfxss, it's the people who don't have the technical knowledge to understand what the developer console does and why pasting in random JS might be a bad idea. So the phenomenon of selfxss reinforces the point.
Perhaps modern operating systems (or hardware?) need two modes - "Safe mode", where everything is sanitised, checked, limited and Secure Boot-style verified, and "Open mode" where it's not; where experts and enthusiasts can work without limit and without DRM.
On the other hand, taking the easy way out when this kind of security problem comes up leads to having a machine that's just an appliance and not a computer. If you've closed every local code execution vulnerability, you've probably rendered your system completely non-programmable and erected a monumental barrier to learning how to hack.
Lua sandboxing is relatively straightfoward. You can choose what functinos from the standard library the script you are evaluating will see in its global scope. By passing an empty scope the only thing the evaluated script can do is build tables, concatenate strings, do arithmetic, etc. You only need to worry about DOS due to infinite loops but there are also workarounds for that).
I find it easier to trust Lua than similar facilities in other programming languages because the kernel of the language has a relatively simple semantics, so the TCB of a sandbox is lower, and the source is easier to understand than most other languages.
Note that sandboxing in Lua 5.2 has a still simpler semantics than for Lua 5.1 - few other languages evolve in a way that makes the language easier to trust.
It's the halting problem - all it takes for someone to embed in your data code that loops forever (wait infinite), or recurses (crash), or discover some vulnerability... Not directly related to saving files as lua, but say as bytecode: https://www.youtube.com/watch?v=OSMOTDLrBCQ
Lua can be sandboxed so your data file can't call arbitrary functions (but can still call a controlled subset, e.g. a function called RGB that does r255255+g*255+b so your colors are somewhat human-readable in the file, yet 24-bit integers in memory).
But it's still code, so you can e.g. inject an infinite loop and the loader will hang. (You can protect against this, you can install a debug hook that gets called after every N instructions executed, and kill the loader.)
Aware of all the issues and already have a plan. But Lua generally only has access to the APIs you give it; from game code our Lua VM has no access to the OS at all, just game functions, and those game functions are never system related.
The biggest 'concern' would be save hacking, but at the end of the day that will happen no matter what so it doesn't bother me much.
Preach it. My favorite persistence code: stuff that has nothing to do with SQL/NoSQL.
I leaned heavily on Python's pickle module for serializing a few thousand entities to disk a few years ago. By streaming them to the application at startup time, it remained plenty fast for all datasets it'd encounter. I intended to replace it with SQLite one day, but I never had to. I could just keep them all in memory.
I'd probably choose something a bit safer now, but it was hard to beat the simplicity.
Now that you mention it, I remember running into something like it. It's a big issue. That module had a bunch of scar code to migrate entities as they came up from disk.
Yeah, I wouldn't use pickle for anything where backwards or forwards compatibility is important. It is however very convenient for 'I want to send/store this data for a bit' tasks.
It can only become corrupted by external factors; a lot of games I've worked on, in-game bugs could lead to corrupted saves being written out to disk. Since in this case we are just serializing lua data, unless the serializer itself has a bug, it will always write out correctly, and any issues become issues of game logic rather than anything else.
I don't think it does. I think he meant that if a save became corrupted it wouldn't do so silently, it would violently crash the game because of a syntax error.
I did this with C# in my last game. All the map/object editors output C# code on save, which was then included in the compiled code on the next build. The beauty is that your "data" files get automatically updated when you refactor your regular code! On top of that loading is faster, because you don't need to worry about fetching a file and parsing it, the whole thing is just compiled code embedded in your executable.
That Lua was originally designed as a configuration language becomes really clear when you start doing things like this. Having my code and configuration being separate but equal was really a paradigm shift for me.
Also, the Tiled Map Editor exports directly to Lua.
While one could say that this is about savefiles for games, I would say it's implications could more be about savefiles for software projects. If you are building the game in LUA, of course LUA is going to be the preferred way to save your game in since you are already using LUA objects and interpreting files in that language will be easy to integrate.
If you ever used maven xml configs, java object marshalling or c# xml you would understand the pains of using xml as a file format for software projects and data representation. You have to find a solution that is language agnostic, neither LUA or JSON is.
I did something similar, but used JSON instead (pretty trivial to (de)/serialize LUA tables to JSON. This made it easy to send data to the server, and inspect with standard tools as well.
This sounds a lot like NSCoding for Objective-C (Cocoa).
Though you'd still have to define the types/classes and name for each property you want to save.
But you could technically save it in a big blob, and then read it into memory as you resume.
Could persist to disk as a binary, sql or a plist (xml).
I guess the only downside is, that if you got a lot of composite classes all with their own properties and associations (say a graph), there's a lot of manual work to be done.
I've had to write output save file formats for various projects on several occasions, and it never occurred to me to take this approach.
Thanks for sharing this, it's one of those ideas that (to me) seems so brilliant in its simplicity that I probably would've never thought of it.
Any hiccups in the day-to-day work using this approach? I'm just trying to get a better idea of the workflow since I'm very seriously considering applying it to my next project.
The biggest hiccup is almost a literal one; serializing large lua structures and then writing them to disk can take a lot of time. But this can largely be mitigated by just saving compiled lua instead of text lua.
I had a lot more fun recently playing the free game Boson X for PC (http://www.boson-x.com/) than I would have otherwise because I discovered that the game folder contains editable Lua scripts. The scripts control the game physics, scoring system, controls, level data, and more.
I’ve created mods of the game where you fun faster but gravity is stronger, and where all levels are randomly mixed into one level, and where the dangerous falling platforms also give you energy while you’re on them, and where the sound effects give the player clearer feedback on what they’re doing. And though I could cheat by multiplying my score by 1000 and submitting it online, I actually have been careful to always comment out the high-score saving and submission code in each of my mods.
I like the game much more than if the developers had obfuscated the Lua files so I couldn’t read and edit them.
The save format does not matter at all. It wouldn't matter even if it were an obscure, made-up format. All it would do is slow down 'cheaters' by half an hour.
The only argument against human-editable text files is parsing speed, not security.
Some people do compete for speed or score in single player games. Arcade games have always had scoreboards, modern "arcade-style" games have online ones, and a community can turn any solitary activity into a competitive one:
Speedrunners are an exceptional case, but I think everyone gets a little annoyed when they look at a leaderboard and all the top players have scores of UINT_MAX or times of 0 seconds.
Obviously cheaters will find a way regardless of whether you give them the source code or not, I'm just saying dfc's concern is not totally ridiculous.
I DO get annoyed when I see those scores, but in a lot of cases even having a leaderboard is just something that was introduced in the game just to be more "social" and less because it makes sense in that specific game.
And yes, it's not ridiculous, on the contrary, it's perfectly understandable.
Of course, these kinds of questions depend a lot on the game in question, and I think they don't have a definitive answer :)
oh don't get me wrong, i love speedrunning and trickjumping competitions, except that they always should require the whole replay - and even then you can't be sure if the whole thing was or wasn't TASsed.
Of course, in multi-player competitive games anti-cheating is a pretty big concern, because it works against the purpose of the game: a competition with well defined rules and conditions.
If the core of the game is single-player/non-competitive, why should we be so worried about cheating?
Why not level 7? By 38 the player could be fixing bugged waypoints, profiling out bad O(n!) code, resetting time to get a specific time-based drop, tweaking character attributes to make puzzles/quests easier, etc. The savefile just becomes another interface to play with.
Gives a fresh angle on the 'open world' type game.
There are ways around it but if people want to cheat their own SP experience who am I to stop them? We'll obfuscate a bit to dissuade casual users but I don't know that I've ever encountered a game that didn't have some level of save hacking available.
You could hash the file contents plus a salt that is contained within the application binary. Most people wouldn’t know how to extract the salt from the binary so they could get the hash right. Though I guess if people are determined enough to hack your game, one hacker might just publish the salt or a small program to rehash files for you. If a user has enough time on their hands, there’s nothing you can do to stop them from running your software in a VM with a debugger and finding out its secrets; the best you can do is making that hard enough that people won’t bother.
If you wanted more security, you could keep a secret that you don't include in the saves but do include when you calculate the hash, so that anyone who doesn't have the key is going to get the wrong answer. That's about as far as I'd consider going for relatively trivial data like save games. Though that's, in principle, discoverable if someone's sufficiently interested.
After that point, it becomes much simpler for someone to watch the memory associated with your program and extract/alter the values there. (Programs to do that to games are generally called Trainers.) That's not a complicated thing to do unless someone's tried to stop you doing it.
There are some techniques to provide some degree of security there. Changing where in memory you place your information each time springs to mind, thus making it more difficult for people to find out where the values are and then share their locations. However, even that's not perfect. Depending on how sure you want to be that no-one's going to alter the values, you're potentially looking at requiring very deep knowledge of security there.
After that point the next easiest target may be the program file itself.
That said, if you want to get around that sort of problem and you're really serious about it, then running your encryption in an environment that hostile may be making things more difficult than they need to be. You might use a trusted platform module, to try to make the environment you were in less hostile, if one were present on the user's machine. But, honestly, I'd want the information not to be stored or calculated on the user's machine if it were that valuable. Have the user's end be the input, encrypt their signals with your public key, and do the calculations that you needed to be sure of remotely.
Though then the user has to trust you. I wouldn't usually advocate that my users trust me that much - not unless we were dealing with a situation where the information we were talking about was entangled with others in some way such that a reasonable argument could be made that they didn't own it, and I was just the best common arbiter I could think of.
-she shrugs awkwardly-
You can get yourself into a situation where it's probable that the amount of effort someone would have to invest is vastly greater than the likely value of the information fairly easily. But ultimately it's a question of how expensive you want to make things and what that's worth to you. Against a sufficiently dedicated adversary, with a sufficiently valuable target, there are so many unknowns in computer security that I wouldn't even be sure that storing the data on your server would be sufficient ^^;
What's with all the XML hate? Of course, doing everything in XML is a stupid idea (e.g. XSLT and Ant) and thanks heaven that hype is over.
But if I want something that is able to express data structures customized by myself, usually with hierarchical data that can be verified for validity and syntax (XML Schemas or old-school DTD), what other options are there?
Doing hierarchical data in SQL is a bitch and if you want to transfer it, well good luck with a SQL dump. JSON and other lightweight markup languages fail the verification requirement.
XML is unnecessarily verbose, for the supposed sake of human readability. But used as a serialization format, it isn't really readable or editable by humans (except in the sense that a Turing machine is programmable): remember that the ML in XML stands for "markup language", and SGML, its predecessor, was designed as a way of marking up normal text, not littering data with angular brackets and identifiers. (XML/SGML arguably isn't that hot as a markup language, either.)
If you really need a hierarchical serialization format that is "verified for validity and syntax", the problem is that XML has prevented the adoption of something better (because it was "good enough").
If you don't need that, then XML is overkill and bloat and makes your format less readable than it could be. And you rarely need it, because either your data is computer-generated and -read, so there's little point in putting in extra schema checks, or schema verification is woefully insufficient (because it can't verify the contents of fields, relations between fields, or a ton of other stuff that can accidentally go wrong).
> But if I want something that is able to express data structures customized by myself, usually with hierarchical data that can be verified for validity and syntax (XML Schemas or old-school DTD), what other options are there?
He actually did address my question in a way: "[...] XML has prevented the adoption of something better (because it was "good enough")."
Which IMO is a sensible way looking at it. I too think XML is not perfect but if all the other stuff we're stuck with currently would be as good enough as XML, IT would be a place with less WTFs all around. ;-)
That depends on your specific goals. But essentially, XML schemas are sort of like attribute grammars, except with an unnecessarily convoluted syntax, and yet more limited in their expressiveness than attribute grammars (because whatever constraints you need have to be procrusteanized into XML schemas).
Even if you were to stick with XML semantics as is, you could improve the syntax to be actually readable and eliminate the angle bracket tax [1, 2].
Alternatively Carl Sassenrath was pushing Rebol in the past. See his blog post "Was XML Flawed from the Start?" - http://www.rebol.com/article/0108.html
If used sensibly XML isn't too bad. But there's a whole lot of cruft in the standard that seems to do nothing except make it harder to use. Part of this is a problem with popular libraries rather than inherent to the format, but we judge a thing by its ecosystem rather than in isolation. So: namespaces are a pain, making it much harder than it should be to just make my xpath work. DTDs are annoying, especially when a production system breaks because a remote server that was hosting a DTD goes down so now your parser refuses to load a file. User-defined entities seem pointless, and though most parsers can handle the billion laughs these days it wasn't always so. The handling of text nodes is confusing; whitespace is irrelevant except when it isn't. Specifying the encoding inside the document itself seems wrong, and supporting multiple encodings at all causes trouble (e.g. sometimes it's simply impossible to include one document in another inline).
Is XML schema really so much better than e.g. JSON schema?
To me it feels like there's an impedance mismatch between the kind of structures XML lends itself to and the kind of structures programs are good at dealing with. So for program-to-program communications with a certain level of validation I find Protocol Buffers is a much better fit. Conversely in cases where human readability is really important, XML isn't good enough compared to JSON.
> So: namespaces are a pain, making it much harder than it should be to just make my xpath work.
Namespaces exist to solve a real-world problem that happens in real-world use cases (SVG embedded in HTML, HTML embedded in RSS). While it would be nice to look at things that are complex and say "it would be less complex for these trivial cases without this feature", in reality there are then common use cases that become more complex or even impossible in the general case, which seems like a very short-sighted benefit. Namespace prefixes are really not that difficult to configure, and once configured XPath makes them very easy to use :/.
The biggest caveat with namespaces is that most people have never bothered figuring out how they work. The number of applications I've seen that have hardcoded namespace names instead of looking up the namespace uri for example, is horrifying.
Namespace prefixes are not that difficult to configure once you know about them. But if you're just starting with XML, probably because you need to extract some information from a document you've been sent, you don't want to learn the theory of XML, you want to get the data you need out and get on with adding business value. So you find a tutorial, you write an xpath, and it doesn't work. You try removing the foo: prefixes in your xpath, and it still doesn't work. This is not the experience that a technology should give new users. A default of matching ignoring namespaces would not make anything impossible.
Indeed. XML gets a lot of hate because it's so difficult to use. It would be fine if you could use it without having to care about the 100 features you don't care about and just use the ones you need, but pretty much every library I've seen makes parsing (or generating) a document a huge and complicated task, and most of it is completely irrelevant to the problem I'm trying to solve.
And because of this almost no-one bothers to actually handle it properly so you often can't actually use the advanced features even if you wanted to.
This varies greatly from framework to framework, and language to language. On the JVM at least, the dark machinery that handles the XML is rather rigorously correct. Parsing and generation are trivial, especially using JAXP. You have multiple ways of working with XML (objects, DOM, push, pull).
XML is "good enough" for a lot of cases. There are lots of tools to mess around with it too, which is really quite valuable when you're experimenting with various kinds of data or you're debugging. Being able to extract out stuff you're interested in XML format means you can perform a lot of complex manipulations quite easily.
The issue is probably that 99.999% of all XML use cases don't use (or need) the verification aspect. For all of those, XML is overkill. Besides, surely it would be possible to design a verification layer on top of JSON, for instance - the fact that one does not currently exist does not mean that XML (and abuse of XML!) should not be criticized.
One of the core aspects of XML that is really important is that no typing is inferred by the structure of the file unlike JSON. JSON is by nature tied to the JavaScript type system which is sparse and inaccurate. For example, if you look at the following:
{ "name": "bob", "salary": 1e999 }
Ah crap! Deserializer blew (in most cases silently converting the number to null)
I think it's refreshing to hear someone advocate XML instead of JSON, specifically because you bring up a good point.
The problem I think is that just because XML is human-readable, it's less sufficient as a format that is human-writable (I'm looking at you, Maven!). I believe this is the root cause that many people hate XML, even though it has a very sweet spot in application-to-application communication.
If you take the brackets and the closing tags out (use meaningful space) it's a hell of an improvement[1], . A format I really like (ok it's aimed at html not xml) is the slim templating language[2]. It manages to pack the same information in but is massively more readable.
Yeah this is exactly where my hate towards Maven configuration comes from, but it's more a testimonial of a bad fit for configuration files than critique towards XML. Java enterprise application configuration has the tendency to be very "expert-friendly", and this is where XML got its bad name from.
> Ah crap! Deserializer blew (in most cases silently converting the number to null)
Right -- the parser blew it. That many implementations do this is frustrating (and caused me so many problems that I ended up building my own validator for problems like this: http://mattfenwick.github.io/Miscue-js/).
JSON doesn't set limits on number size. From RFC 4627:
An implementation may set limits on the range of numbers.
It's the implementation's fault if the number is silently converted to null.
I guess we need better implementations!
> JSON is a popular format but it's awful.
If you're willing to take the time to share, I'd love to hear more examples of JSON's problems. I'm collecting examples of problems, which I will then check for in my validator!
If you're looking for examples of problems, RFC7159 (http://rfc7159.net/rfc7159) is a good place to start - just search for 'interop', as suggested by [1]. A quick look at Miscue-js suggests you already check for most of them, but you might still find something new.
Your example doesn't do anything but make XML look as bad as your saying JSON is. Think about it again - do you think your first XML example doesn't ALSO have to be deserialized twice (once into an XML in memory tree, once into a number)? It does. Also, both examples will fail if you try to deserialize either of them into numbers...
Regardless, JSON is so much more readable that I'm very glad it's pushed XML out of the picture for the most part.
XML can be read as a stream and at certain points like after reading an element or attribute, an object can be created on the fly or a property on an object set and the type deserialised at the same time. The types don't have to be native types either; they can be complex types or aggregate types such as any numeric abstraction or date type you desire.
See java.xml.stream (Java) and System.Xml (CLR) for example.
As for readability, some XML is bad which is probably what you've seen but there's plenty that's well designed.
XML is afflicted with piles of criticism which usually comes from poor understanding or looking at machine targeted schemas that humans don't care about.
You'd complain the same if you looked at protobufs over the wire with a hex editor.
What is that massive semantic difference? If you want the number represented by 1e999 as the value for salary, at some point, something has to take "1e999", whether you call it a string or a something-with-no-type, and turn it into a number. Your deserializer has to know to do that in either case.
How does the [deserializer] step in the XML example know to call into [bignum], and why can't the [json reader] in the JSON example have that knowledge in the same fashion?
Because the XML document has a semantic meaning that is specifically designed for this application. It may even have a schema definition document which formally defines what types to expect. JSON, by contrast, has type definitions imposed on it by its nature as JavaScript code.
I've sort of lost track of what this debate is about... Assuming you don't have a schema definition, it seems to me that you can just as easily parse `{ "salary": "1e999" }` with application-encoded semantics as `<salary>1e999</salary>` with (again) application-encoded semantics. Maybe having a formal schema definition is a win, though.
iff you have a schema, and a parser that actually uses it. I've seen a few DTDs but the vast majority of XML documents don't have a schema or even a DTD to follow.
And the vast majority of parsers will not parse anything for you, regardless of schema definitions.
Which effectively puts you in the same place as the JSON string.
Either to author of the serialized data realized that the numbers could overflow a float or didn't. This is independent of serialization format.
In your contrived example, somehow, the user of JSON didn't realize the salary could overflow a float. (OTOH, he succeeded in serializing it, mysteriously.) All the while, the XML user was magically forward thinking and deserialized the value into a big decimal. Your argument simply hinges on making one programmer smarter than the other. If one knows that a value will not fit a float, the memory representation won't be a float and the serialization format won't use a float representation. It has nothing to do with JSON vs XML.
This. Types are a huge pain in JSON, particularly the lack of a good date time type. BSON fixes tips, but only of you're using MongoDB and are willing to give up the "human readable" requirement outside of mongo.
OK, so the provided number format is not sufficient for the kind of numbers he is trying to deal with. So instead you would represent it as a string and handle the encoding/decoding of that number yourself. How is that different from the XML way where there is no provided number format to begin with, and everything is a string?
People seem to prefer JSON, but I don't find it any better to hand-write/hand-edit than XML. If anything it's slightly worse, because it has more syntax edge cases.
And it doesn't support the multitude of accurate numeric types that XML does implicitly. XML data is not just "strings", it's a sequence of characters. The deserializer determines what sort of type it is based on either the structure or the language's capabilities. With XML, you can define these policies. With JSON you're stuck with JavaScript being the semantic standard and type definitions which ties you to floats or numbers inside strings. The latter is criminal.
Edit: clarification as HN won't let me reply any more.
How so? XML by itself only supports strings; any other data types have to be derived from a schema. But you can do the same with any other format that supports strings, including JSON.
But in the design of XML this was already acknowledged.
That's why there is the distinction between well-formed and valid XML documents. Only with valid XML documents there is a schema attached that will describe these nodes with the type attribute. And because it is extendable, these types can be anything but they will be automatically validated by the parser.
JSON OTOH doesn't have this extensibility. There are a couple of predefined types but if you need to go beyond them (and this happens all the time because JSON doesn't even define a date type!) any interpretation is up to the parsing program and this can vary tremendously (again, look at the handling of dates and for example the questions on stackoverflow about them).
To be specific, JSON syntax is a subset of YAML version 1.2.
However, I hate YAML with a passion. It is worse than XML in my books. I can usually read JSON fine. I can also read XML in many cases. For the life of me, I just can't read YAML. It has something to do with "-", line indentation and different ways of writing lists.
Of course, someone will say YAML is technically better ...
Same here, it is very difficult for me to tell levels of nested structures in yaml. Though I'm sure if I sat down and read up on it I could force it into my brain. But shouldn't it be intuitive to read without that?
Python has exactly the same problem -- control-structure nesting quickly gets confusing and hard to read beyond a certain (fairly small) size -- but at least with python, you have the option of splitting off stuff into separate functions to limit the amount of nesting and size of blocks.
It's technically true because YAML includes an alternate "inline style" that lets you write objects in JSON syntax. Therefore any JSON object is a valid YAML object as well. But, not an idiomatically written YAML object, since writing YAML using only inline style is unusual.
Could you provide examples? I'm trying to collect more examples for a JSON validator -- http://mattfenwick.github.io/Miscue-js/ (built during a big project using JSON, after I started running into some issues that I couldn't check using other validators)
I'd love to hear more examples if you're willing to share.
I personally miss having schemas and XSLT in JSON.
> doing everything in XML is a stupid idea (e.g. XSLT and Ant)
XSLT actually made a lot of sense. If everyone writes code to transform format1 to format2 then what you end up with a lot of slightly different transformations. Its main downfall, just like XML itself, was that it was annoying and time consuming to write.
How would you replace all this if you moved away from XML?
> Its main downfall, just like XML itself, was that it was annoying and time consuming to write.
And impossible to debug. Write once, do something else for some weeks, and trying to understand what you were doing at a later point is nearly impossible.
There are schemas in JSON (see, for example, Kwalify) although they are not something that is built into the specification. I don't think the equivalent of XSLT is as necessary when the document readily translates to data structures in a scripting language.
The problem with XSD and DTD is they only offer primitive ways to validate data, and it takes significant effort to validate some data (eg,[ https://stackoverflow.com/questions/3382944 ]). As a result, there have been a bunch of other XML schema validators created to counter these problems, but we should really ask why we need to keep inventing new languages when the existing ones turn out to be insufficient.
If we start out instead with something that's turing complete and simple to begin with (perhaps S-expressions?), we can (often trivially) write our own validators/type-checkers, or any other processing tool to verify the document structure, with few or no constraints, and without requiring the effort and expertise to parse complex syntax.
Unfortunately, XML is way too open-ended for my tastes. You end up getting entire rows of DB content (with full text paragraphs and everything) entirely in one tag, with attributes and values. There are so many options that you typically get a lot of idiot programmers who don't understand the purpose of all the shit in XML, so they fuck up their implementation.
Simply put, XML does not correctly model the data by which we intend to interchange. It was a noble effort, but it didn't come from a place of innovation. It came from corporate needs for standardization.
This may seem innocuous, but XML allows mixing of arrays and objects too liberally, and makes automatic parsing overly complex. At first <customer> appears to be an array of account objects, but wait now that we reach the end we find that <customer> is an object with multiple keys and must create an unnamed array key to hold accounts.
XML is a document markup language, not a data format.
The really annoying issue is as the parent says, that the accounts collection does not have a name. This means there's no canonical mapping for the structure into a programming language object, which necessitates that libraries require annotations or some other side-channel way of specifying how to wrap the accounts into a collection.
In Jaxb e.g., how many times must we add junk like:
@XmlElementWrapper(name = "accounts") ?
In any individual case the workaround is easy, but it's annoying to have to do it repeatedly.
XML really is better as document markup than structured data representation.
People dislike XML because it's way overkill for %99 of people's use cases but it still gets used anyway! Most people who use it should of been using something simpler like JSON to create their configuration file or return their list of strings in some HTTP API. You can have bloody security vulnerabilities with XML, like you had recently with facebook: https://www.facebook.com/BugBounty/posts/778897822124446
The likelihood of a JSON feature biting you in the ass like that is far lower. Don't use XML until you actually need something XML SPECIFICALLY provides.
Also JSON easily translates with easy to work with dictionaries and lists, XML parsers take more code to work with equivalent items.
> But if I want something that is able to express data structures customized by myself, usually with hierarchical data that can be verified for validity and syntax (XML Schemas or old-school DTD), what other options are there?
S-expressions work great. Syntax checking is far simpler, and validity checking is hence something you can roll yourself (and writing an S-expression schema checker ain't tough).
I haven't used it myself, but lisp seems well suited to the task. I've also heard good things about yaml, which is more well-supported by your language of choice.
Because XML solves a 'problem' in the worst possible way. It is not that easy to parse for machines and only the simplest XML files are readable by humans.
Besides, since 1960 or thereabouts we have S-Expressions. The world should just have used that without reinventing the wheel once again.
Firstly some clarification - this appears to just be about the persistence format for his dive log. It was XML, now it's git based with plain text.
As someone who had to manage a system which worked with plain text files structured in a filesystem for a number of years in the 1990s, this is done to death already.
You now end up with the following problems: locking, synchronising filesystem state with the program, inode usage, file handles to manage galore and concurrency. All sorts.
Basically this is a "look I've discovered maildir and stuffed it in a git repo".
Not saying there is a better solution but this isn't a magic bullet. It's just a different set of pain.
> You now end up with the following problems: locking, synchronising filesystem state with the program, inode usage, file handles to manage galore and concurrency. All sorts.
Which is why he's reusing git for resolving those pain points? Well presumably all except "synchronizing filesystem state with the program" -- where he's gone from using some kind of xml parser to marshal xml to objects/structs in ram to using a (simple(r)?) text parser to do the same.
I'm guessing he just writes/reads a full (part) of a log (a branch of the full tree, or whatever is used in the program. Maybe a list anchored at a date?) -- and lets git sort the history/backup thing.
So, yes, it's a different format, but I think the argument you're making is off -- seeing as he already has git for that? It's more like combining Maildir (or mboxes, only commited when valid) and git.
Maybe you want to wait till he release something.
Cause you know, if he took months to get the big picture in mind, I doubt you grasp what he envision just by reading his comment.
So, the solution to the fact that the merging UI is a pile of garbage is HAVE A SINGLE PERSON ALWAYS DO THE MERGE. Excuse me? The whole point of a distributed revision control system is so I don't have to have a single choke point. That's the definition of distributed.
Yeah, the root fault wasn't Git. However, at no point did Git flag that something was going horribly wrong as the repository got corrupted and deleted. Other distributed SCM systems I have used tend to squawk very loudly if something comes off disk wrong.
Maybe the underlying git data structures are fine, but, man, the UI is a pile of crap.
And, I won't even get into rebase, because that seems to be a religious argument.
The issues in the randyfay.com post are due to a misunderstanding when using git as a "centralized" repo like SVN. Git, by design, does not enforce a central repo even if you designate one logically. These issues can be completely avoided if you merge the right way:
Well, that confirms that the "obvious" workflow of "git pull" is dangerous. At least it explains all the spurious merges. Why on earth did it ship with this broken design? Why doesn't git pull do the right thing by default?
I believe you can flip a switch to turn on only allowing FF merges which should alleviate the situation. Certainly FF merges make a lot of things easier and "cleaner".
I guess maybe the reason git doesn't do this by default might be because the idea of rebasing early on (the "omg you're overwriting history in a RCS!!!!!") was a bit taboo and it's taken time for people to get used to the idea. Note that I'm just speculating about that, I did follow the git discussion early on and I know that people then (and still are to some degree) afraid of "rewriting history" (note that I quote that because I don't really see it that way).
I'm not sure I follow. The advice for the "single person always do the merge" is essentially make sure the people doing the merges are experts. These mistakes do not seem like the kind of thing I have heard of experts doing.
Seriously, you can not call yourself a git expert, if you think rebase is a difficult thing to explain.
Might you sometimes make mistakes? Sure. I hardly see this as a systemic thing, though.
The mirror shenanigans I agree suck. Not sure what the real takeaway is there, other than don't rely on mirror as a good form of backup.
So, the solution to the fact that the merging UI is a pile of garbage is HAVE A SINGLE PERSON ALWAYS DO THE MERGE. Excuse me?
That isn't what the post advocates. He says that having a single person approve the pull request is a good idea, but approving the pull isn't the same thing as manually doing a merge. Projects I've worked on required that the submitter merge master into their branch before their PR would be accepted.
I took this to mean that what he is replacing is a single XML file whose content is a tree of element nodes. Every time you have to make a change to that file (changing, removing or adding children nodes within the file) you would have to store a new copy of the file. The most efficient you can get is to store just the text diffs using git or something.
But what he replaces it with is a git object store. Each xml-node becomes a git object. They each point to a parent (just as git commits point to a parent commit).
Now writing to this datastore means adding a new node to the git object database and changing the parent references.
Where git stores commits that are related sequentially in time, this stores nodes in a tree relationship that IS the document.
If he's not talking about this then I'd like to officially take credit for my weird idea right now.
The impression I got was that he was going to store his data in a git object database and that the files would be virtual in there. It would be like the .git directory without the working files on disk. It's all just conjecture until his code his out.
Regardless, I would think that some applications are simple enough (store few enough separate objects in the file system) that the issues you cite are not likely to cause a problem.
Back in the bad old DOS days, instead of creating a file format for saving/loading the configuration of the text editor, I simply wrote out the image in memory of the executable to the executable file. (The configuration was written to static global variables.)
Running the new executable then loaded the new configuration. This worked like a champ, up until the Age of Antivirus Software, which always had much grief over writing to executable files.
It's a trick I learned from the original Fortran version of ADVENT.
Readers may be familiar with the TI-83 programmable graphing calculator's assembly language functionality (especially those who took high school math classes in the mid-to-late 1990's). The TI-83's only user-writable storage was 32K of RAM (there was a small lithium battery to keep it powered when you changed the AAA's; also some of the RAM was used for system stuff so somewhat less than 32K was actually available for user purposes).
You could write hex values in the program text editor, then you could tell the calculator to execute the hex codes as machine code. I understand the previous models, TI-82 and TI-85, were hacked / backdoored to run user-supplied assembly language, so TI responded by including an official entry point and developer documentation for the TI-83.
People later wrote loaders which allowed programs to be stored as binary instead of text (using half the space). Some loaders also had the capability to run binary programs by swapping them into the target address rather than copying them (theoretically a third option would be possible, running programs in-place if they weren't written to depend on their load address, but this wasn't a direction the community went in. gcc users may be familiar with -fPIC which produces code which can run from any address, and this flag is necessary when compiling code for shared libraries.)
This allowed people to create massive 20K+ applications (an RPG called Joltima comes to mind), that used most of the available RAM.
The fact that this loading scheme made static variables permanent was also quite convenient. (And most variables were static; stack-based addressing would be tough because the Z80 only has two index registers, one of which is used -- or perhaps I should say "utterly wasted" -- by the TI-83 OS.)
The next generation, the TI-83+, included I think 256K of flash ROM, and a special silver edition was released which contained 2 MB.
I forgot to mention, on a floppy disk system, saving the configuration in the exe file made for fast loading of the program, since it didn't need to do extra floppy file operations to load the config.
I learned a heckuva lot from reading the ADVENT Fortran source code. I was floored when I figured out how it was saving its configuration - such a brilliant idea. And in DOS it could be implemented in about 5 lines of simple C code. (Code size was critical in the old 64Kb days.)
The other huge thing I learned from ADVENT was polymorphism. The comment in the source code "the troll is a modified dwarf" was an epiphany for me.
With my occasional sysadmin hat on, until a few weeks ago I had the luck to never have had to deal with XML configuration files. Then came Solr and now I know what horror is. (To be clear, Solr itself is great, but those god damn config files...)
What I like is the "I dont start prototyping till I have a good mental picture"
I am currently stuck on a project I want to start becasue I cannot get it to fit right in my (future) head. And I am glad I am not an idiot for not being able to knock out my next great project in between lattes.
(Ok, in direct comparison terms I am an idiot, but at least its not compounded)
"A change in perspective is worth 80 IQ points."
-- Alan Kay
My biggest hurdle solving new problems is divining a unifying, simplifying metaphor. Once you have the right notion, that Eureka! moment, everything falls into place, like magic.
Like how Kepler was able to fully explain Bache's astronomical data once he realized the planets orbits the sun.
Personal example: I used to write print production software. Placing pages onto much larger sheets of paper that get folded and bound into a book. A task called image positioning aka imposition. It took me years to figure out how to model the problem. Key insight was simulating the work backwards, from binding back to the press. Then when I showed the new solution to my coworkers, the response was "Well, duh."
Yeah, I noted that too, also that it took him months to to get his good mental picture. It makes me feel not so bad about spending months trying to get clear on some of my stuff.
The question nobody is asking, but actually should is: I wonder what other good G+ content you are missing?
G+ is largely misunderstood. It is a lousy tool for interaction with people connected to you purely socially. It's a very good way to find and interact with people connected to you by interest.
The really sad thing is that I have tried several times to search for content that I know exists on G+, but I can't find it, even when I knew the author. After the third time failing at this my usage of G+ dropped significantly. Of all of the things that you would think would work search would be at the top... :|
I know this is completely off-topic, and I'll happily be downvoted for it, but why in the world does Google+ capture keyboard shortcuts that are already bound to other well known browser functions? (C-PgUp, C-PgDn, C-w, etc).
Yeah, and I hear some communities (photographers?) have taken G+ as their home. It was tongue-in-cheek and purely from my (PC, desktop, Windows/console game developer) perspective.
I don't quite get Linus' problem with XML for document markup (for anything else - config files, build scripts - sure, XML is horrible). Does anyone know any more details about what his specific gripe is? For me, asciidoc (which looks very similar, conceptually, to markdown) suffers from one huge problem: it's incomplete. Substituting symbols for words results in a more limited vocabulary, if that vocabulary is to remain at all memorable.
Sure, XML can be nasty, but thats very much a function of the care taken to a) format the file sensibly b) use appropriate structure (i.e. be as specific as necessary, and no more).
Document markup is the one place XML is a no-brainer - more specifically, long-form, highly structured documents (i.e., essentially books).
Without it, publishing would be stuck in a morass of nebulous, ill-documented proprietary messes, and a great deal of current learning would be at risk of being lost to posterity. The fact that there are associated open standards such as XSLT with which to transform it is just the icing on the cake as far as publishing is concerned.
This is why there's so much distaste for XML - people try to use it for applications where it isn't ideal (and there are many more of those than there are applications where it is ideal) because they've swallowed someone else's hype, and as a consequence they have a bad time. If not for the unbelievable exaggeration a few years back (I heard people claim without irony that XML - a markup language for god's sake - would literally change the world), the divisiveness wouldn't exist, and it would be a technology used by experts quietly getting on with the jobs it's best for.
>Document markup is the one place XML is a no-brainer
That's only true for minimally formatted documents. For anything that approaches professional typesetting requirements, XML is a nightmare.
By far the biggest problem, it the requirement that inner elements must be closed before outer ones can be. This frequently means that the software must do a huge amount of read-ahead to figure out which aspect of the formatting changes first to make that formatting element innermost.
Sometimes, that's simply not possible to arrange and so you have to close a whole bunch of elements and then reopen all but one of them.
All this because a constraint of the format.
Ideal formats, such as used by typesetting systems that don't use XML, allow you to say: keep this formatting trait on until it's switched off. There is no concept of every element needing to be a subset of its encompassing element.
For those who missed it, here's what Linus wrote in the comments:
"+Aaron Traas no, XML isn't even good for document markup.
Use 'asciidoc' for document markup. Really. It's actually
readable by humans, and easier to parse and way more flexible
than XML.
XML is crap. Really. There are no excuses. XML is nasty to
parse for humans, and it's a disaster to parse even for
computers. There's just no reason for that horrible crap to
exist.
As to JSON, it's certainly a better format than XML both for
humans and computers, but it ends up sharing a lot of the
same issues in the end: putting everything in one file is
just not a good idea. There's a reason people end up using
simple databases for a lot of things.
INI files are fine for simple config stuff. I still think
that "git config" is a good implementation."
Linus' adversion to XML explains also why parsing git's output is so abysmal inconsistent.
Subversion has a really good XML output for its log command which is a joy to use (and that's something to say if you work with XML) whereas with git you always have ugly format options that are most of the time underdocumented.
Just a note, pandoc (implemented in Haskell) makes it almost a joy to work with various "dialects" of ReST/AsciiDoc/Markdown etc. It feels like what python's ReST should have been (at least the last time I looked, it was pretty hard to get different html out of it - even if it is supposed to be extendible). If you have a dependency on python, staying with the python libs are probably best, but if you just want "a document system", I recommend having a look at pandoc.
Actually, i am using sphinx and latexpdf to generate project documentation and it is more then awesome to generate beautiful looking LaTeX documents with graphiz graphs in high-res but only writing rst... :)
I like that the syntax for features is illustrative, so that the raw text representation doesn't end up as a mess of ugly tags. However it looks completely inflexible. It's a mishmash of special cases. How would I implement a new feature without breaking existing implementations? Or without having to write a new parser that in all likelihood will break on some subtle edge case?
At its core XML (if you ignore all the DTD, namespace and entity rubbish) is both simpler and more powerful than this. You have text, tags and attributes. What those tags and attributes mean is up to the application, but at the very least you can be sure that the document can always be reliably parsed into a form you can work with.
That's a wonderful rant - I particularly appreciate the digression into anti-bush rhetoric - but:
1. There's very little detail here; it's a nicely worded, emotionally charged piece that leaves a lot of detail unaddressed, e.g. "'I would like to hear why you think it is so bad, can you be more specific please?' If you really need more information, search the Net, please." That's not very helpful.
2. It argues for 'simpler' markup via the removal of attributes. Where possible, I totally agree, as at least hinted at in my original post. Sometimes, though, this would be impossible or unwieldy (e.g. HREF attribute on an A element).
3. Character entities vs. unicode - totally agree. Wherever possible, I use proper unicode characters rather than ugly character entities in my markup.
4. "But the one thing I would change the most from a markup language ... is to go for a binary representation." Linus would vehemently disagree on this point.
with key words being "Whather what you are really after is foo, bar, or zot, depends on your application.".
His articles on SGML are mandatory reading too.
Several years ago someone posted these links and it opened wonderful world of Lisp to me. Not the language per se (there are many languages) but whole another Universe of how things could be done. I swear I jumped on the chair reading every page of CL standard, how brilliant it is on every level to C. Eventually it led me to rethink attitude to C and Unix in general, core parts of which I despise now.
So here am I returning favor, maybe someone will follow these links too.
What is it with HN commenters and their demented ability to send topics completely of track? I would have thought someone might have examined the code or what Linus is trying to implement and comment about it.
But here we have threads about Lua, why people hate XML and love JSON and all kinds if irrelevant issues which have been well hashed elsewhere ad nauseam. Why not restrict to an analysis of whatever it is Linus developing?
HN is getting truly annoying and sucky, if it isn't so already.
> "I actually want to have a good mental picture of what I'm doing before I start prototyping. And while I had a high-level notion of what I wanted, I didn't have enough of a idea of the details to really start coding."
This I like. The race away from the waterfall straw man has also stripped us of the advantages of BDUF.
While rigid phase-driven project management helps nobody, I think there's still room for speccing as much as we can upfront within iterative processes.
Or you could run to the IDE and start ramming design pattern boilerplate down its throat the second you're out of the first meeting ;)
You should be speccing what you want to achieve: the goals, the why, the impact, the external limitations, measures of success and so forth. This also allows you to describe and plan testing up front. The "how" is best handled in an iterative manner.
A lot of people use AGILE to avoid planning at all, which is a particular destructive anti-pattern, and the exact opposite of what you need.
>The "how" is best handled in an iterative manner.
I think that the first "how" should be planned as much as anything else. I understand how you refactor from v0.0.1 to v5.34.2 iteratively, but I think that getting from vNothing to v0.0.1 is qualitatively different.
If I don't have a complete idea of how my minimally functional thing will work that is small enough that I can completely hold it in my head, and instead just architect by agglutination and test writing, 1) my results are going to be hacky garbage, 2) my first 50 iterations are going to be devoted to replacing it all haphazardly to fix bugs, and 3) the code and interface will become increasingly more complex, harder to work with, and strewn with special cases.
When v0.0.1 is well planned, v2.5.2 may not look anything like the plan anymore, but in my experience it becomes shorter, cleaner, and more correct rather than a giant ball of band-aids propped up with tests.
Personally, once "the goals, the why, the impact, the external limitations" have been defined, we start to do mockups, a programmers "differential", to zero in on a solution that satisfies the demand.
With that on the board/paper/code, we can start to test our assumptions and iterate on the solution. I do not know if that classifies as planning, but it works very well.
I don't really understand what he's talking about here (my ignorance, not his fault.) Is it something like https://camlistore.org/ that is a content addressable (the git part) datastore?
>>So I've been thinking about this for basically months, but the way I work, I actually want to have a good mental picture of what I'm doing before I start prototyping. And while I had a high-level notion of what I wanted, I didn't have enough of a idea of the details to really start coding.
This might be a tangential discussion. Earlier, I used to have a similar approach. Can't code until I have the complete picture. But, it's tough to do in a commercial world and you have deliverables. So, nowadays, I start with what I know and scramble my way until I get a better picture. There are times when that approach works. But, there have been days where I was like - "wish I had spent some more time thinking about this".
I am curious how folks on HN handle this "coding block".
I've got a few strategies that might help, depending on the circumstances.
A notebook: I'll write down some notes and just kind of free write whatever thoughts come to mind. If there's something that I think is important to come back to, I'll draw an empty box in the left margin (to be filled with a check mark later)
Readme: start writing the Readme for the project, even if you're not entirely sure of the details. Include code examples. If you don't like how the API is coming together, change it. It's way less work to modify the API now than it will be later.
Write a test: I don't always unit test, but when I do I test first :). This works well on projects that already have a decent test suite. It's kind of an executable version of the Readme.
Branch and Hack: branches are cheap. Make one and start playing. Don't like how it's turning out? Make a new branch and try again!
Ctrl-Z: maybe the answer won't come to you right away. Let it sit and run in the background for a while and come back to it. If I'm worried about forgetting details, I'll write it down in a notebook first.
This is what Linus does. He has strong opinions and he throws them around. You can't let that get to you. Both XML and JSON are just fine if used properly.
Almost all of Torvalds' "profanity rants" that get passed around are the result of frustration at an existing conversation, and you can find profanity-free comments by him simply by checking out a slightly earlier one.
Did you read the article? It's not really about the filesystem. 1 part your fault for seemingly not reading the article you're commenting about, 1 part the submitter's fault for choosing such a misleading title.
Worked on a project a few years ago where we needed distributed sync capability. Using git (or bazaar or mercurial) was one of the options - store everything in it versus a database. Interesting to see the same thought "coming back".
This is absolutely amazing for debugging purposes. Also you never have to worry about corrupt save files or anything of it's ilk. Development is easier, diagnosing problems is easier, and using a programmatic data structure on the backend means that you can pretty much keep things clean and forward compatible with ease.
(Oh, also being able to debug by altering the save file in any way you want is a godsend).