A lot of people are commenting that SemVer doesn't work, because it's still at the mercy of humans choosing good version numbers.
Elm's package manager, elm-package, actually tries to remove humans from the equation, by automatically choosing the next version number, based on a diff of the API and the exported types of a package: https://github.com/elm-lang/elm-package#publishing-updates
It's not perfect, but it's better than anything else I've seen.
It litterally changes nothing. I've had to fork a lib to bump a dependency version just recently.
The only things that protect elm from knowing dependency hell are the age of the ecosystem and the fact that there are not that many common package publishers outside of the core team.
That’s a pretty neat strategy. Elm really brings a lot of cool (and fun) sugar to dev flows. I’ve been looking for a reason (project) to use it in so I can transition from an enamored fan-person to a true evangelist :)
I'm doing this kind of automation with Maven in Java. There is a plugin (build helper I believe is the name) that gives you properties like "next.release.version", "current.release.version", "next.snapshot.version", etc.
So I've setup an infrastructure where you just click a button and it performs a release with _proper_ version number in accordance with semver, simply does the right thing. Works like a charm.
I don't understand complains about human factor when you eliminate it with ease.
To summarize, any change that offers more (as in, more fields in the output) and/or requires less (more of the original parameters made optional) is non-breaking.
Edit: Of course, this is all under the assumption that the semantics of methods should never change. Instead of changing semantics, he proposes to just create a new method with a new name.
A) you generate an export of the public API (entrypoints, arguments, types) in some stable format.
B) you run the (public-api) testsuite of the previous release against the current release.
- If A yields no changes wrt the previous release, then B should probably succeed as well (barring changed tests). Bump the patchlevel.
- If A has changes but B succeeds, it's a minor version bump.
- If A has changes and B reports failures, it's a major version bump.
Of course, this isn't fool-proof, so you still want some way of overriding the automated version bump. But I think it's reliable enough to at least force developers to think about the changes they're making. Of course, it does rely on having a testsuite, and explicitly marking parts of it as public-api validation.
Probably pretty simple. Destructively making a public API change is breaking. Additively changing an API isn’t. Adding an optional public param or new symbols isn’t.
Harder to detect cases in which the signature stays the same but the expected behavior changes, though (as noted in TFA).
One somewhat subtle implication is that you need to have a clearly defined expected behaviour. If people have to read your implementation to use your API, then your implementation has become the specification and basically every change will be a breaking change.
Basically, if your old unit tests break, or if the post-conditions changed (in case you use design by contract), that's a new major version, otherwise, it's not.
But yeah, there are counter-examples, and anyway a user can expect a behavior to remain even if it is not explicitely specified. For instance, I developed a SAT solver. I could make a non-breaking change that improves the efficiency of the solver in like 99.9% of the cases, because of a cool heuristic. That would be considered a "patch", since I didn't change the API at all, and all unit tests are still working perfectly. But a former user could be pissed off because he's part of the 0.1% and now the tool became just a little too slow for his use. Is that a breaking change or not?
It's simple, everything that has existed should behave exactly as it used to before. In order to ensure this people invented lots so technics - code reviews, exploratory testing, integration tests and so on. It's even boring to discuss it, it's just basics.
Ah, you mean what to bump and when? That's easy, you make a decision to break an API so you need a major version bump. In this case you perform a customized build overriding variables in CI GUI. All other releases are automated minor bumps if on master and patch version bump if it happens on release branch which we keep around for bug fixes. Those bug fixes propagated through release branches up to the master automatically whenever you merge a PR with a bug fix.
I don't know whether parent is doing this, but I used to use the Clirr maven plugin to fail the build if a release had binary-compatibility changes. You could then have a distinct release profile that permits a BC break but increments the major version number.
What about basing the decision on the API unit tests ? Combined with API signatures hashes, maybe there would be enough information to deduce minor changes, bug fixes and major changes
No, it isn't computable (that is, correctly determining one of "these functions behave the same" or "these functions behave differently", and not "unknown") in general, as it is equivalent to the halting problem. Consider these two versions of a function, are they API compatible?
They're only truly API compatible if the program halts, but the program can be arbitrary, so proving that any 'foo' of this style are equivalent is solving the halting problem.
Of course, one can still likely get useful answers of "definitely incompatible" etc., with much more tractable analyses. AIUI, the Elm version ends up just looking at the function signature, and catches things like removing or adding arguments: for appropriately restricted languages, it is likely to even be possible to determine if downstream code will still compile, but that's not a guarantee it will behave correctly.
It's been a while, since I studied this, but linear bounded automata are arguably a better model for the real computers we can actually build and the halting problem is computable for LBAs.
That said, in practice I'm not sure it is too useful (this is a slightly different question to the one originally asked, though). My understanding is the proof of decidability is essentially a pigeonhole principle argument based on LBA having a finite number of states: run the LBA for that many steps, if it hasn't halted, then it has looped. Even if a program is running on a system that only allows programs to use 1GB of memory, there's 2^(2^30*8) (approximately 10^(2.6e9)) states.
Essentially, the problem is not with computers, but with languages.
If your language lets you build a neverending loop/datastructure/etgc, then you cannot prove halting for all the programs that can be made by the language.
Since such languages exist, then by extension, it applies to computers.
I wonder if tighter guarantees can be given if tools can work with constructs like D's contracts. These are extra sections in each function that are intended to check invariants. If these invariants change, then they were either broken and needed fixing or the function had a semantic change.
That's an interesting idea. It seems like it would be a way for tools to flag "this function is likely to have changed in an interesting way", but changing invariants doesn't necessarily mean the function breaks semver.
For one, an invariant might be changed syntactically, without actually changing what it is asserting, reducing to exactly my example above: in its simplest form, the contract could go from in(true) to in(halts("...")).
Secondly, an contract could have been made more permissive, e.g. in(x > 1) becoming in(x > 0), or out(y > 0) becoming out(y > 1). Assuming violating a contract is regarded as a programming error (as in, it isn't considered breaking to go from failing an assertion to not failing), these are also non-breaking changes.
Lastly, changing behaviour doesn't necessarily mean changing invariants/contracts.
The halting problem doesn't matter if your API is required to define timeouts / time guarantees. Thus if a#1.0() returns "hello" after 100ms and a#1.1() returns "hello" after 20s, even though they return the same result, they are deemed functionally different. This follows real-world expectations where performance which suddenly slows to the point of unusability is considered breakage, even if the same result is returned in the end.
In practice your time guarantee isn't going to let you produce a sound and complete static analysis that proves the equivalence of two arbitrary (modulo termination guarantees) functions.
In the real world all programs have a timeout set to the time to the heat death of the universe. This hasn't helped us make sound and complete static analysis.
I think that's essentially the same point as LBA in a sibling comment. The same intractability-in-practice applies to it: even if your timeout is 10 milliseconds, that's still (tens of) millions of operations on most modern CPUs and order of a gigabit written to/read from memory, which is a huge state space.
I think it's more like a lower bound on difficulty. In other words, halting is one of many guarantees that would need to be checked to ensure that an API didn't break, and it is impossible to check, so it cannot be done, irrelevant of the difficulty of the other problems (like timeouts or whatever).
I don't think you understood my comment - you don't need to check for halting if all methods in the API have timeout guarantees. Halting is only a problem if runtime is uncertain. If runtime is certain (again, because of promised performance figures) then it is entirely possible to verify whether an API has been broken or not.
Is it just pedantry? Even in the strongliest of practical strongly typed languages, two functions with the same signatures might fail to return on some inputs in the new version that it didn't in the old.
That's a practical distinction I care about that can't be computed.
If you're narrowly interpreting an API to mean just the function signature, then sure, if your type system is appropriately restricted then it is computable (although many type systems are Turing complete, meaning this won't be computable in all cases, for the same reasons). This is, in fact, something my comment explicitly called out.
Having signature compatibility is an important prerequisite, but I think it's possible worse to have code that compiles but does the wrong thing than to have code that doesn't compile (at least it's obvious to the developer that something needs to be fixed, in the latter case).
I'd imagine it isn't, at least depending on how you define API compatibility, and whether you're only looking at the API interfaces. Imagine two versions of a library that implement the function "add".
Version 1:
add Int -> Int -> Int
add x y = x + y
Version 2:
add Int -> Int -> Int
add x y = x * y
Both versions expose the same API interface, but the functions that conform to that interface are semantically different. A stronger type system could probably differentiate between the two functions, but I doubt you could generally compute whether both functions implement the same behavior.
Perhaps with some sort of functional extensionality you'd be able to compute compatibility perfectly, but I can't imagine that ever being feasible in practice.
That being said, what Elm does offer is still a huge improvement over humans trying to guess whether they made any breaking changes :)
These proofs weren't automatically discovered, though for such simple programs I'd expect an SMT solver to be able to find the proof or a counterexample easily enough.
But even proving they're the same value for all inputs isn't all that helpful, because of lazy Haskell code like:
version 1:
fib :: Int -> Int
fib n = if n < 3 then 1 else fib (n - 2) + fib (n - 1)
version 2:
fib :: Int -> Int
fib n = let fs = 1:1:zipWith (+) fs (tail fs) in fs !! n
They're (provably) the same for all (positive) input values, but if you call version 1 with n greater than, say, 35, you'll be waiting quite a while for an answer, while version 2 will be very snappy for the first few hundred thousand values of n, at least, after which the size of the answer will be a bottleneck.
If a library switched from the latter to the former, it'd have a good chance of breaking code.
While obviously exponential code is obviously exponential, this sort of behavioural change can show up in less obvious ways - a change in memory usage might blow your heap after a supposedly minor version change, eg.
In the end I don't think the value judgement of 'breaking' or even 'significant' is computable, and you'd need to rely on a human doing something that approximates to 'right' for your world view with their version numbering.
I don't think 'does it work exactly the same' isn't necessarily the right question, given there are expected to be bug fixes which may change the behavior in some functions.
Irrelevant. If fixing a bug is a breaking change then it is a breaking change. This is the importance of pre-releases/"nightly" branches so that you don't have a mistake of addNumbers(5,5) returning 25 instead of 10 and not being caught and then needing to increment a major number to fix a typo of × to +.
A bug fix is a change. A breaking change is a change that breaks the API. Doesn't matter if the previous status was the intended one or not.
SemVer not working is almost entirely people simply not following it correctly. The projects that follow it as best as they can have very few mistakes were "woops that was a major change" occurs while projects that use it as a guideline may as well not be using it at all.
Irrelevant?? That is the most relevant question that a developer could ask in this situation.
Often, there are ways to work around bugs in an API. And, just as often, those workarounds will break as soon as the bug is fixed. The API developer is in a particularly poor place to judge whether a bug fix is breaking or not. Sometimes, they can talk to users to see if a change will break their code, but other times they can't. Thus, it comes down to how well a developer can predict whether a fix will break projects in the wild.
Irrelevant?? That is the most relevant question that a developer could ask in this situation.
No, it isn't. The most relevant question is "can somebody be using the current API". It doesn't matter if your current API matches the documentation, what matters is whether your current API is out there for others to build on.
Don't try to use a crystal ball or other form of divination to predict what your downstream users have been doing with the code; you will always lose. Instead, suck it up, acknowledge the mistake, and signal the breaking change by bumping the major version.
Maybe next time your developers will spend more time validating their public contract, so they won't have to endure the embarrassment of a major version bump.
Using that logic, every single bug fix that even remotely changes how the API operates would be considered a major version. What would be the point of semver?
Semver both acknowledges and is predicated upon the fact that every bug fix has a range of possible impacts. At one end, some bug fixes have almost no chance of breaking an application. At the other end, some bug fixes will almost certainly break applications.
This is not divination. It is understanding your application and its users.
Finally, in regards to your last paragraph, I genuinely hope that you don't speak to developers like that. Mistakes will always happen. Have a little bit of respect and treat people with kindness.
>Using that logic, every single bug fix that even remotely changes how the API operates would be considered a major version. What would be the point of semver?
Yes. That's exactly how SemVer works. It signals to people that there was a breaking API change - it doesn't matter if you're on version 10495.0.3 - if you've somehow broken your API 10,494 times you're following SemVer. Although if you're that bad at maintaining a stable API you probably don't have any users you know... using it.
SemVer is predicated upon the fact that other people are using your API. It signals to them whether or not it should be safe to update without breaking their code that relies on your API. If you break your API - you broke their code. That makes you an asshole in their eyes. The entire point of SemVer is to signal that you broke your API so that people relying on it can examine their code for breakage, decide whether or not to update, or start working on modifying their code to match the new API.
If your API is very unstable then people aren't going to rely on it. If it's both unstable and you don't signal breaking changes I'd be amazed if anyone put up with working with it.
If I'm using version 3.0.2 I should be fine using any 3.y.z version. If you release a 4.0.0 then I know I need to investigate and determine whether or not it is safe for me to update to that version. If your API is still unstable and changing on a daily basis - you shouldn't be at 1.0.0 yet. 1.0.0 is meant to signal the "stable API stage" of development.
A bug fix is not a breaking change. You are speaking in tautologies. "A bug fix is a change that is a change to the API". No, bug fix does not change the interface. A bug fix does not change the API. It does not require a semantic version change unless it also changes the expected behavior of the method, in which case it is a change in the documentation, which is part of the application programming interface.
A bug fix is not always a breaking change but a breaking change is always a breaking change. SemVer doesn't care if it is a bug fix or not. The only single relevant detail is if it broke the behavior of the API for any reason. The reason for the breakage is entirely and completely irrelevant because the only relevant detail in SemVer is if it broke the public API. It's a binary "Yes/No" question.
Doesn't matter if it breaks it in a way that you don't expect users to have been using it - you broke it. Doesn't matter if you were just fixing a typo that changed behavior - you broke it. Doesn't matter if you were fixing a logical error - you broke it. Doesn't matter if you fixed the PRNG causing some results to no longer be possible - you broke it. Doesn't matter if you deleted a deprecated endpoint - you broke it. '
SemVer isn't asking why you broke it - it's asking did you break it.
Only as far as the type system supports, but I feel like constraining the minimum version unit to increment is far better than doing nothing. Disallowing incrementing the patch version when the types of existing API changes is great.
True. Besides, my specs are future proof. So I always offer my clients a 120 years "no update needed" guaranty, with very durable titanium punched cards.
This seems like something solved decades ago with c header files, they're easy to do a diff on and the only false negative is from adding a function. Even that would be fine if you weren't export raw structs.
It seems like we gave up simple ways to do stuff like this because we hated header files and moved to tools like java and c# that eschewed them. Then they got reinvented and renamed to interfaces and we've come up with all sorts of other complicated tools (elm-package, mocking, IoC) just to recreate the functionality we lost in header files.
Such as? White space changes can be ignored, comments can be stripped pre diff, both version can be run through the same formatter before hand. What you're then left with are the actual changes.
Anything removed or modified is a breaking change. Anything added to a public struct is a breaking change, which can and arguably should be hidden behind an interface. Adding functions is about all I can think of that's left.
Even if it's not perfect, it's a simple 90% solution with 40 year old tools.
Good points, the second one in particular I don't think could be fixed without a full parser. Function order changes could possibly be worked around by a formatter/linter that can reorder functions, at the risk of creating more issues. The last could be handled be passing the code through the preprocessor (the -E flag in gcc) first.
By this point it's probably gone beyond the "perfect is the enemy of good" threshold though.
I love the MaxInboundMessageSize example. I've run into that many times.
Often there will be a note in the release notes about it, and I know I should read the release notes in detail when I upgrade dependencies, but like many people I don't always. Sometimes it's just laziness or complacency -- especially for "utility" libraries like for compression or encoding -- but other times it's a challenge with release notes:
* Each version has each release note published independently (or worse: only on the Releases tab in GitHub, and you have to click to expand to read each)
* The release notes are really long or dense, and breaking changes are easily missed
There's also worse problems:
* The release notes don't actually call out the breaking change (you have to read each ticket in detail)
* The release notes just say "Bug fixes" or there are no release notes
I think along with the suggestions in this article, library authors should also put effort into making good release notes. This includes realizing sometimes people are using from a couple major versions and/or years ago.
While it is a good example, you could also use the same example and conclude that the problem was inadequate tests. SemVer is great, but you can't count on dependencies that you do not control actually adhering to it, either intentionally or unintentionally.
The only thing that could have prevented something like this for sure was mentioned:
> And while nothing in our early testing sent messages larger than 256k, there were plenty of production instances that did.
To me, this was the clear failure; not the fact that some dependency broke semver. Their production system relied on being able to send messages larger than 256k, and their tests did not.
While it's easy to say, how far do you go? Do you test every bit of every upstream library you use? The ideal is probably yes, but the reality is this rarely happens.
Even with a test, you may not find this. In the IOException example, the author calls out why:
> When we upgraded, all our tests passed (because our test fixtures emulated the old behavior and our network was not unstable enough to trigger bad conditions)
The only way to catch this type of thing is to emulate the entire network side of things, and that's still only as good as your simulation of the real world. Again, reality is even if you test your upstream to this extent, you're probably mocking a bunch of things, and that may mask something in a way you won't see until possibly production use.
> While it's easy to say, how far do you go? Do you test every bit of every upstream library you use? The ideal is probably yes, but the reality is this rarely happens.
It depends. I have a friend who used to work on banking systems. They had full test coverage of every dependency. Even standard lib functions and language features.
One time they found a bug in the md5 implementation in a minor version of a popular database.
> It depends. I have a friend who used to work on banking systems. They had full test coverage of every dependency. Even standard lib functions and language features.
these are not dependencies anymore then, they are part of your code source and should be vendored with it. I don't know what language your friend is using but I'm pretty sure most std libs and languages already have tests with very good coverage.
> One time they found a bug in the md5 implementation in a minor version of a popular database.
Every piece of code can have bugs. 100% code coverage doesn't eliminate bugs, it just says all code path are tested, an algorithm can still be wrong for some values even if 100% code path are tested.
The lesson learned isn’t about code coverage or oaths tested. It’s to not blindly trust 3rd part anything, even “languages already have test with very good coverage” when the stakes are high.
If billions of dollars are riding on your code, you better be damn sure you trust everything it relies on.
Fun side note: every piece of internal code was always developed in parallel to the same spec by 3+ teams so they could cross validate. If all 3 functions don’t return the same value for the same input, every team gets to build it again until all implementations behave the exact same.
Certainly it's fun from the pure engineering perspective but I guess also somewhat tedious.
On the other hand, if a billion dollars depend on your code working or not, or in other cases human lives like in space rockets, you don't get a second chance. If you fuck up, lots of important things get flushed down the toilet, usually including your job.
So you have 3 teams do in parallel to be 99.999999999% certain that it'll work as advertised. It's also sorta why banks are slow to adopt new changes since they want to be sure that whatever is going on, it'll work and not flush down Grandma's rent.
But isn't it impractical to test every feature of every library you are using? In an ideal world you would have everything tested in isolation as well as integration. But in practice there will always be a corner case that remains untested because you don't know all internals of the libraries you use.
If you want to be able to randomly upgrade those dependencies and not have to worry about a breaking change, then yes. Semver is not going to help you there. Server is only going to help you when someone knows they are releasing breaking change. And even then, only if they are nice enough to actually follow the spec.
You don't have to test every bit of every dependency you use, but upgrading them without either carefully reviewing the changes or having tests in place for at least critical functionality is asking for something like this to happen eventually.
God yes! For the apps that I maintain (and which have users outside my team), I enforce high-quality release notes like you describe. Representative example: https://github.com/sapcc/swift-http-import/blob/master/CHANG... (note that this also takes SemVer seriously)
A normal version number MUST take the form X.Y.Z where X, Y, and Z are non-negative integers, and MUST NOT contain leading zeroes. X is the major version, Y is the minor version, and Z is the patch version. Each element MUST increase numerically. For instance: 1.9.0 -> 1.10.0 -> 1.11.0.[1]
> Stop trying to justify your refactoring with the "public but internal" argument. If the language spec says it's public, it's public. Your intentions have nothing to do with it.
This is so wrong. APIs are for people, not tools, so intent is primary. When tools are not expressive enough to capture and enforce intent, you document it, but it's still primary. Someone using a "public" API that clearly says "for internal use only" is no different from a person that uses workarounds like reflection or direct memory access, and there is no obligation to keep things working for the.
> there is no obligation to keep things working for the.
You opened with the correct observation that APIs are mostly for people. Saying there is no obligation here contradicts the expected social norms. And even more importantly, intent does not tightly correspond with reality, and what can happen, tends to happen. The actual code actually existing always has the final say. If you intend to have the best outcome for everyone involve, conform to the unalterable realities as much as possible - if the interface should be public, make it public. If the interface should be private, make it private.
I specifically said "when tools are not expressive enough to capture and enforce intent".
Suppose you're writing a library in Python. Everything in it is public. Even the dunder class members are, because it's just name mangling, and the language spec even documents what exactly it does!
Now, is anyone going to seriously claim that every single identifier in every Python library is part of its public API, and any change that affects it is a breaking change? Because that's certainly not the "expected social norm".
Granted, Python is a somewhat extreme example. But in practice, this also comes up in languages like Java and C#, when dependencies are more intricate than what the access control system in those languages can fully express.
And then there are backdoors:
> What can happen, tends to happen. The actual code actually existing always has the final say.
You can use Reflection to access any private field of any object in Java. There's actual existing code doing that in practice, too. Does it have the final say, and does it mean that internal representation of any Java class in any shipped Java library has to be immutable, so as to not break the API clients?
The language I'm using doesn't let me express the public/private divide I wish to make correctly (e.g. "private" implementation functions for a public C macro.)
The API is 100% intended for internal use only, but someone insists on ignoring that and consuming the private API anyways. Instead of forcing them to write their own headers which silently break at runtime when function signatures change in certain calling conventions which don't check those signatures, I instead allow them to include headers with a few keywords like "private", "internal", "do_not_use", or "i_am_voiding_my_semver_warranty" in the path, perhaps only after they make some similarly scary #define s, so it's at least a build failure.
C++ and C# have the same kind of problem: except for the iffy freind declaration in C++ there is no way in the language to denote that some method is bot meant for use in other modules. C# has the internal scope for each assembly, but this breaks in combination with unit tests placed in seperate testing assemblies.
Generally, proper unit testing is at odds with strict scope restrictions in the tested code. I guess we need more allowances fornunitbtesting at the language level to fix that. E.g. allow testing code to be marked as such and ignore that certain things are declared private, but in turn only allow it to be run in a testing context, but not regular builds, to prevent abuse.
This only covers a small part of the problem. Things that should be private and requite separate yesting are atill required to be more visible than they are supposed to.
This simply cannot work in many cases. It is quite unrealistic to test complex logic that is hidden behind a narrow interface completely. You are hit with the full combinatorial complexity of what is behind that interface, even if might consist of independent parts internally. If you can test these parts independently, the number of required tests is a fraction of what a black box approach requires.
Another situation is checking numerical code for correctness and accuracy. There it is extremely advantageous to have testable small functions that map to individual mathematical expressions. But these are again implementation details that need to be hidden behind interfaces.
Your first statement is exactly what I've arrived at. It's just not avoidable in general.
I have to clarify that I'm not fixated on C#. Sure, you could create a helper assembly in .NET that is a mess of essentially of disembodied functions for computing every slightly more complex function that happens to be in your program. But this breaks OOD.
In C/C++ you can't do quite the same. The best you could do there is break OOD and try to hide these global functions by using private headers (which are ugly in their own ways).
> C# has the internal scope for each assembly, but this breaks in combination with unit tests placed in seperate testing assemblies.
This is a limitation imposed by the IDE, not the language. There's nothing stopping you from compiling code and tests into the same dll that you unit test. Likewise there is no need to separate code from the tests (apart from different files), they can simply not be included in release builds.
Source layout structure does not have to be a 1 to 1 mapping of the output structure.
Not keeping tests separate from the tested code can lead to chaos in the long term. It provides an incentive to blur the lines in inappropriate ways, e.g. by adding helper code for tests to the code under test etc.
In my experience the more things are public the better. Very often a quick workaround turns into a monster bodge because some method is marked strict private instead of protected or public.
So I usually make most stuff public as such, but put internals in a namespace/scope that makes it clear that these are implementation details. Relying on implementation details always carries the risk of breaking when upgrading.
This allows for a lot of flexibility when needed, while also not polluting the "truly public" API.
Making an API public means that people can do whatever they want with it. If you are not sure if you want to allow the API in the future it should not be public. People will always look to do the laziest thing possible which might mean hooking into your "public internal API". Then you will never be able to change it and you will have to maintain it forever.
They can do whatever they want with it but you have no obligation to maintain nor support it if it’s not a documented public API, in my opinion.
It’s a bit like a house on a corner with a big front yard. People may cut through the grass to save time but you can’t blame the homeowner when he finally puts up a fence.
It’s unlikely that allowing the public to short-cut across your yard would create a prescriptive easement, but there are certain circumstances where allowing a party or the public to cross a parcel of land for a sufficiently long period of time does prevent the property owner from erecting a fence.
Although it’s not a hard-and-fast rule, in Ontario certain property owners have paths open to the public most of the time, but close them for at least one day of the year, often Christmas or New Year’s Day. The intent is to prevent access for any one or more continuous years, which in turn prevents an easement from being asserted by any party.
Although it’s far from as simple as, “if you allow the public access to your land for one continuous year, you allow it forver,” a certain folklore around this has arisen, and thus the practice.
Django is my gold standard for this. They have great deprecation policies where they deprecate something in the same release that they add alternatives (allowing for you to fix things up before upgrading Django), they document these changes liberally and offer alternatives, make good use of the warnings system (meaning you can run tests in "deprecated functions not allowed" mode to catch stuff), and generally are careful.
I'm still shocked at the number of projects that make breaking changes without first releasing a "support both versions" release that lets people test their changes easily. Especially frustrating when you have really basic environment variable renames that could support the deprecated name as a one-liner so easily.
SemVer is a social construct, not a contract. It's nice when it applies, but you cannot rely on other developers to adhere to it.
One man's bugfix is another man's breaking change. If product A implements a workaround for a bug in product B, but the bug gets fixed in a patch version, it could break product A's code, so it becomes a breaking change. The only way to anticipate these changes is reading the change logs/release notes, and thorough automated regression testing. (Obviously unfeasible for every dependency.)
Maybe versions should be a single number, like a build number. It just gets tricky when you have multiple versions out there, each requiring patches.
Rich Hickey's take on SemVer makes for a really fantastic talk.
"Change isn't a thing. It's one of two things: Growth or Breakage." Growth means the code requires less, or provides more, or has bug fixes. Breakage is the opposite; the code requires more, provides less, or does something silly like reuse a name for something completely different.
While the distinction of construct vs contract is subtle, improved tooling will eventually elevate the "construct" to a contract.
semver needs to be combined with a package manager and strong version locking semantics for it to be useful.
Both npm and yarn in the node eco-system certainly provide this - with any remaining kinks are being ironed out fast.
Using micro-modules as dependencies is a rather pleasant experience in node/js - especially when the dependencies follow semver. This is even more true of popular modules, where authors take their versioning responsibility seriously.
I regularly use automated version updates (npm-check -u [1]/ npm audit --fix [2]). Coupled with good test coverage of my code, I've been really happy.
I've found that the golden rule of "everyone is lying to you" works well enough. Assume that every change will be a breaking change. Test everything, verify everything, and then continue to test and verify when it's in production.
I've never found standards to be all that standard.
I haven’t figured out how to implement this, but the key observation is that Semver is trying to delineate degrees of substitutability (LSP).
I think the right solution here is to build version numbers off of your black box tests.
The devil is in the details though. What’s a change to the tests mean exactly? Adding new tests is probably a patch release. Modifying tests demands at least a minor version number, but what makes it a major version number? Deleting tests probably qualifies. Removing assertions probably does too.
But now what if the author is bad at tests too? Can I substitute my own? (I could do that right now for regression testing, and in fact I do occasionally for interdepartmental issues).
> Imagine if we had that same attitude with regards to HTTP.
Well, many do. You wouldn't believe the number of broken HTTP clients/servers out there.
The difference is that most HTTP clients/servers have to work with some significant subset of the pre-existing HTTP infrastructure (otherwise why would anyone use them?) and so that constrains their implementation to be "mostly correct". It's literally network effects :).
Libraries on the other hand usually start out with a single consumer and have no such constraints on their implementation so you end up with various levels of ossified brokenness or breaking non-ossification.
SemVer is a starting point. If it's a major release you can prepare yourself mentally for a lot of breaking changes. A point release... probably not
In any case you need to read through the changelog or (absent that) the actual code diff and think about how you use the application. It's not 100%, but no versioning system will be able to identify how you use code and how you expect the semantics of an application to be.
The issue is that because not everyone follows SemVer, you have to assume that no-one follows SemVer and act defensively, otherwise you will have problems.
Ideally, there would be a tool that can inspect your entire codebase and determine if the change is "breaking". This still has issues if the change lies outside of your codebase (perhaps such as changing the configuration of your AWS services).
A problem with SemVer I often see is that it's unclear whether a project adheres to it. You just can't assume every project having x.y.z version numbers uses semantic versioning.
Not necessarily. There are plenty of projects in their infancy that follow semver correctly. I'd argue that a project with a high major number is more likely to be indicative of improper usage.
I don’t think high major version number tells you that. Maybe they left 0.x.y (unstable) too early and were just honest with their early churn since which is as semver as you can get.
But one if the main semver violations I see in the wild is a project slotting major changes into the minor version number because they want to avoid high major version numbers for some reason or have some romantic idea of what a major version bump “should be”.
Build numbers is engineering, semver is marketing.
Private processes vs what you tell the world.
Build numbers unlocks delta debugging achievement. Add 'last known good' and 'found' build numbers to tickets, along with repo steps, then use diff to find bug.
Build numbers also unlocks QA/testing achievement. Add 'found', 'fixed', 'verified' fields to tickets. Now your team is certain when individual changes are ready to merge, ship/deploy.
In programming, type contracts are not. Wouldn't it be nice if we had some sort of infallible static analysis tool that absolutely determines what constitutes a breaking change in your codebase?
This is all great, but I feel like all these problems could be caught just with properly written tests. If your tests correctly cover the API usage of your code, and I mean both your code complying with the intended API and the API complying with the intended usage, then the implementation behind that API should be totally transparent. No need to check versions, release notes, or any of that, just run the API compliance tests on the new version, and if it works then your code should work too.
They built tests that explicitly assumed that the library's interface wasn't going to change:
>When we upgraded, all our tests passed (because our test fixtures emulated the old behavior)
Doing that, upgrading your dependencies and expecting everything to work just because those tests passed? That's naivete.
If they'd built decent integration tests that used the actual library (instead of "assume nothing changes" fixtures) and made more of an effort to simulate realistic scenarios then their tests probably would have flagged up most of the issues they had.
Alas, this seems to be one of the side effects of following the "test pyramid" "best practice".
I wasn't talking about that paragraph, but the following paragraph where they had tests, but they didn't test with large enough packets.
Tests can never cover every scenario. They are very useful, and they catch a lot of unexpected regressions. But they're just a part of the puzzle, not a replacement for good development practices.
Updating a dependency without bothering to read the release notes because you have tests -- maybe naive is the wrong word, maybe hubris fits better.
Tests can't cover every scenario, no, but had they made a bit more of an effort to test realistically then it's absolutely possible that they could have covered every scenario that mattered here.
Over-reliance on unrealistic unit tests (which is likely what led to them not testing large packets) is a pattern I've seen cause issues like this many, many times before.
I upgrade pretty regularly without reading release notes - relying on realistic tests to catch everything. What they do catch is usually not in the slightest bit obvious from release notes (often a regression in the dependency). Call it hubris if you like, but it works for me.
Treating the test suite as your contract and having a well structured + documented test suite also makes SemVer considerations a matter of what tests changed and how did they change (new cases, new features, changes to existing features, removed features, etc.)?
Wouldn't it be easier to write optional variables, allowing you to keep the same method name? This results in cleaner code that doesn't break existing usage.
For example:
func ListItems(query Query, limit int = 0, offset int = 0) Items {
// ....
}
> But for less intrusive changes, I personally feel like you can make some minor SemVer transgressions provided...
Kind of contradicts "Often, it seems that version numbers are incremented by "gut feel" instead of any consistent semantic: "This feels like a minor version update."
> The value of MaxBufferSize was adjusted downward to 2048 because we discovered a buffer overflow in a lower level library for any larger buffer size. See issue #4144
Technically it's a major version bump, as I understand it. But security is important, so what should we do here in addition to writing it down as first sentence on release notes? Perhaps having an excuse to potentially break downstream code in the name of security should be OK and well communicated (i.e in readme)?
It would be helpful to mention that these guidelines -- and semver generally -- don't really apply to applications from what I've read. Versioning applications still feels like a gut feeling.
I'd assume marketing considerations dominate the version number discussion for (consumer) applications--whether you want the release to be seen as the good old whatever you know and trust or as new and improved.
This is something that can I cannot stress the importance of, yet rarely ever receives the recognition that it should. Being able to delicately and compassionstely work on old code, while still bringing valuable updates is hard.
It’s even harder to make your changes look easy and obvious in hindsight.
This is something that most engineering organizations are not equipped to recognize and promote as a virtue - it’s sort of hard to explain as it is. If anything, this patience can be considered an enemy to progress.
Regarding exception handling, letting internal exceptions define external behavior is perhaps a bad idea. The possible exception types can be wide and change over time as new parts or features are added. Example:
// pseudo-code
qry = new query(sql=theSql, dbConfig=DB_FOO);
if (! qry.Execute()) {
errMsg = "Something went wrong during your query. ";
if (qry.errorExceptionName=="DB_Busy") {
errMsg += "The database appears to be busy."; // append more
}
displayAlert(errMsg);
} else {
processQuery(qry.resultRows);
}
Here any fatal errors are caught inside the query object, but details are available if and when you wish to take advantage of them outside the query object. The query object (API) user doesn't have to know all possible exceptions types in order to handle an exception properly (or at least in a good-enough way).
It also bugged me that he was upset when the behavior they were relying on was an internal detail not included in the API's contract.
(Incidentally, the API itself did not change because it was something like func Read(in Reader) error, where error was a parent of all exceptions)
That's not incidental. In my opinion the authors of the API were completely fine changing the internal detail of which specific exception type was thrown because their public API never made a guarantee beyond it being an instance of error.
Future-friendly error-handling can indeed be tricky. I ran into this trying to make a lasting email-sending API. I was hesitant to depend on the API's specific exception types, and so considered mapping them to more general categories, yet still giving details for troubleshooting.
// pseudo-code
err = new Error(hasError=false); // innocent until proven guilty
try {
sendEmail(...);
catch (e in excptn1, excptn6, excptn7) // dummy names
err.hasError=true;
err.recipientProblem=true;
err.errorType = e;
catch (e in excptn2, excptn4, excptn9)
err.hasError=true;
err.contentProblem=true;
err.errorType = e;
catch (else)
err.hasError=true;
err.errorType = e;
}
...
return(err);
One could make an emumerable list of error categories, but in this case I wasn't even sure they were mutually exclusive because it still sends to the rest if one recipient is bad.
What counts a breaking change? I would say that strictly speaking anything could be a breaking change if an user is crazy enough, so increasing major version all the time is not particularly useful - I mean, you could do that, but what is the purpose of semver then. Consider a following function.
f(arg: String): Output
Now, let's say that we change it to be generic. Let's assume that `String` implements `SomeInterface<Output>`.
f<T: SomeInterface<U>, U>(arg: T): U
An user is crazy and passed an empty generic type list and their code broke as there are now two generic arguments instead of none.
f<>("a")
Or for a different example, let's say that you want to introduce a new function, `g`, but the user does the following.
import yourlibrary.*
import otherlibrary.*
An user is using the function `g` from `otherlibrary`, and their code doesn't compile anymore due to an ambiguity.
I would say it's a minor change, but in theory it's possible for it to be a breaking change. I often had situations like this where something could be breaking, but the code had to be really unusual for it to break (and if it would break, it would be a compilation error).
It's a bit funny that the title reads "write nice and stable code" - but it's actually a plea for "proper" versioning.
Starting with the versioning a few examples are given - and I read it as "this is not good versioning" as it continues with the very basic description of how semantic versioning works. Yet all those examples given above are excellent examples of how semantic versioning works at its best!
"""
10. Build metadata MAY be denoted by appending a plus sign (...) Examples: (...) 1.0.0-beta+exp.sha.5114f85.
"""
And these meta data, referencing used library versions and actual hash of the commit used in the build, are given there - and exactly these meta info on used library versions may help a great deal when it comes to checking bug reports as programme behaviour may differ between versions, but work with any.
> To that end, these are safe as part of a feature release:
>
> Adding a field or method to a struct/class/enum/etc.
Um, adding fields is a breaking change if you do a binary distribution.
> Z is the patch version. Changes to this indicate that internal changes were made, but that no changes (even compatible changes) were made to the API.
then, contradicting that
> Mark a thing as deprecated as soon as it is considered deprecated, even if that is a patch or minor release.
Deprecation should be considered a change to the API. I'm not going to thank you for filling up my build logs with warnings when I pull in a patch update of your library.
> Deprecation, after all, is a warning condition, not an error condition.
I try to encourage everyone to define a contract, code to that contract, and update the contract when it is no longer accurate. The exact nature of the contract is contextual, sometimes a schema, sometimes a well documented comment header or perhaps a project README.
Could you describe some situations where adding a field broke stuff? I can only imagine that happening if your code cares about the exact binary representation/layout of a struct - so serialization, FFI or just bit-twiddling.
Serialization is one, but anything involving casting from one struct to another (e.g., if the Berkeley sockets implementation changed the size of struct in_addr), anything requiring careful management of memory alignment (e.g., SSE), etc. could potentially be broken. Never mind the inevitable reckless coding practices that arise in the wild all the time (someone decides to #define a magic number for allocation purposes instead of using sizeof...).
On the other hand, a large number of constants are supposed to be tweakable, to the extent that many are designed to be set at compile time.
Anyway. OP article had good points overall, I'm just not sold on some of the specifics.
fwiw, I maintain a website called smallrepo (https://smallrepo.com). It builds go language code together, and maintains an always buildable commit set as a "super repo". If you sync to smallrepo (rather than using go get), it can shield you from many unexpected build breakages.
I'm maintaining an open source project[0] and I'm struggling with using SemVer because my "app" doesn't have a single API but a few:
At it's core it's a node app. Though I also include a small web server that wraps around it (and a UI frontend).
1. It allows people to write scripts (js) that receives inputs and passes on events based on an API (the strategy API)[1].
2. It has extensive configuration[2] that sometimes changes form (the config API).
3. It talks to a number of external services (crypto exchanges), over a "common" protocol called the "exchange wrapper API"[3] (I am ignoring the version of the exchange API being consumed).
4. The wrapped webserver comes with an API (REST + WS)[4].
5. The "core app" is a chain of plugins, when they change the required config/events also change (usually breaking changes 1 and 4)[5].
I could take all of these components apart (microservice way) and version them separately, but I like the monorepo style I use now where pulling one repo means that everything is working together. Also the fact that (in bug reports) people only have to refer to one version (and when on nightly maybe the git commit if I need more details).
Your project looks cool. Here's some overly harsh criticism from an old dude in no particular order.
The problem I see upon an extremely cursory view of your project is that it's trying very, very hard not to be a sellable product.
I'm a good programmer. Why do I want to learn your API/library instead of calling the exchange API's directly? Is this saving me time? Is it saving me time long term, even when your code changes and breaks things?
If you had to make this into a single web API and sell access to that as your product, what would it look like? There's your versioning and design answer.
Why is the app a "chain of plugins?" If the app breaks when the plugins change then they're not really plugins, are they? Does this design solve a problem or did it just seem like a cool way to do it?
Also, anything that relies heavily on configuration to work is a fundamentally broken design in my opinion. Configuration is global state that is hard to change, and the heavy-handed presence of it in a project is usually an indicator that the abstractions are wrong and most of the code probably relies on some hidden state that's really hard to debug unless you're the code author. If there's a legitimate runtime choice you don't want to make for the user, that's a function parameter. If that looks messy, you probably left too many decisions for the user to make.
An ideal library is stateless so that the user can handle wrapping it with simpler calls and configuration settings. Make building blocks, not skyscrapers.
Sometimes I want to grab all of you young people by the shoulders and shake you until you stop reinventing ever more convoluted ways to do RPC.
Finally, take everything I say with a grain of salt because I'm heavily medicated right now.
Woah great feedback. This is very much appreciated!
I'm not sure if going specifically into all of your points right here is the best way forward. But suffice to say I am very happen to hear them :)
> the abstractions are wrong and most of the code probably relies on some hidden state that's really hard to debug unless you're the code author.
This is very much spot on, definitely something I want to work on.
-----
The main reason that everything is so spread out (plugins, web API, internal API, etc. etc) is because a ton of people are doing different things with it.
99% of the people only touch the basics, and they don't need to touch any config file, they can go through the UI that handles all of it automatically (Gekko is focused on tech savy but not perse professional programmers). They don't know what (my concept of) a plugin is, and they don't care about any API (nor any version for that matter).
It's about the other 1% who are kind of spread out over:
- people who want to hook into certain lifecycles (to push certain data to google spreadsheets[1] for example)
- people who only use subparts of the app, for example to have something that can fetch normalized market data from a number of different websites <- this is a big part of the project, but not the sole, hence it should not dictate versioning.
- Or people who only want to create their own prediction making logic (with AI or whatever) and use the execution logic of my app. <- in the process of pulling this out as a standalone library.
- people building tools on top of the web API that bruteforces a problem space to figure out new solutions[2].
So all the people that care about the versioning (not the 99%) are exactly the hobby DIY hacking people who want to open it up and take it apart. And it feels impossible to steer them into "don't touch this because the interface is not a standardized API".
-----
The main thing I am going to do now is rethink the entire config strategy, because it's a huge mess and I think I am the only one who understands it[3].
This is a great piece. I think my only addition to it is to point out two small insights.
1. SemVer works great in some communities (Ruby) and shit in others (Python). Generally the more computer sciency the community the better I find the SemVer'ing. Python has a bunch of scientists using it, so it's less reliable, even if many of the core libraries follow it pretty well.
2. Apps, plugins, frameworks, and libraries are different things with different SemVer strategies. With a paid app versioning is often a marketing decision. Framework plugins versioning is often a "match the framework to reduce mental burden" decision. Whereas frameworks and libraries I find a much higher adherence to what SemVer strives to do.
Elm's package manager, elm-package, actually tries to remove humans from the equation, by automatically choosing the next version number, based on a diff of the API and the exported types of a package: https://github.com/elm-lang/elm-package#publishing-updates
It's not perfect, but it's better than anything else I've seen.