I've been following Unison since almost the beginning (back in the structure editor days!). It's a very cool project, https://www.unison.cloud/our-approach/ is a great read, and the Unison language (especially their formulation of effect handlers as "abilities") is very cool.
There are two specific things here that make me reluctant to use Unison Cloud in my own work:
1. It doesn't look like there's any FFI or way to shell out to other tools within Unison Cloud. I understand that this is necessary to provide the desired static guarantees, but the lack of an escape hatch for using pre-existing code makes this a really hard sell.
2. Typed storage is excellent! What are its performance characteristics? In my experience, gaining expressiveness in storage systems often requires trading away performance because being able to store more kinds of values means we have fewer invariants to enable performance optimizations. How do migrations work? I've always found online migrations to be a major pain point, especially because data rapidly becomes very heavy. (At a glance, it looks like storage is key-value with some DIY indexing primitives, and I couldn't find anything about migration.)
The approach article asks "Why is it so complicated, anyway?". My guess would be that:
1. For small projects where you can toss together NextJS and SQLite and throw it onto Hetzner, it really _isn't_ that complicated.
2. For large projects where with large amounts of data, high availability requirements, and very large scale, performance and operability matters a lot, and none of these all-in-one systems have yet to demonstrate good performance and operability at scale.
3. There really is not that much demand for projects between these two sizes.
Nice, a structural editor throwback reference! :-) I'll speak to point 1! We aim to add FFI as a fast follow on to the native compilation work that is underway. The work on the JIT compiler opens the door to FFI so that's on the roadmap soon.
Have you ever left a job with a Haskell codebase and then the organization regretted it? Usually niche technology becomes difficult to support and often ported out.
I once built a data import tool using F# (It was a .net shop). I left for another position and my intention was to help them with consulting-style work, but my next position was more demanding than I anticipated and my former company was forced to deal with it.
I consider it one of my larger mistakes. In hindsight the real problem was building the thing in isolation and then dumping it on someone else with minimal training time. I think different tools and languages are fine, but they need a bus factor of 2-3 to prevent this kind of stuff.
Excellent question, and this was a serious concern for us. And we have experience with this! We had one deep Haskell expert on the team who helped build a bunch of core primitives who has since left to another company.
One of the motivations for adoption for us was strong interest on the team. Looking back on our team...
1. We started with me (enthusiast), our expert,and a Go developer with no Haskell experience.
2. Over time, at its peak, the team grew to us 3, Python developer who had dabbled, two Ruby developers who were interested but had no experience, a frontend expert who had dabbled, and another engineer who I didn't work with closely but I think had a background in Java?
3. Our expert left the team before or around the time that the last person (maybe-Java person) joined, I think.
4. On the second Haskell team I spun up, it's me, one person from the old team, a Ruby/JS developer, and a JS developer.
We were able to get all of these people productive in Haskell. It got easier once we had more experience teaching folks. There are a handful of _key differences_ between Haskell and ALGOL-family languages (mostly around evaluation model and effect tracking), and once you nail those down, the rest is pretty smooth sailing, and your SWE experience and intuition begins kicking in again.
Although we miss our expert very much every day (they were a very cool person! They drove a motorcycle!), their departure has not had an outsized impact on our velocity.
A sibling comment recommended a bus factor of 2-3. I think this is roughly correct, although I would think of this as not merely your "don't get hit by a bus" group, but also your core teaching group.
Re: positive and negative - refer to my answer in https://news.ycombinator.com/item?id=36744384 and especially the linked Serokell interview, where we dived into this question specifically.
Re: design workflow - this is not very different from any other code. You have module interfaces, logical subsystems, separately deployable service entrypoints, etc. Our Haskell code tends to take special care to avoid statefulness and globals, but this is not any more of a design burden than writing idiomatic code in any other language (e.g. planning out your classes in Java).
Re: special tricks - we used fused-effects to model our effects, which has been pretty useful (it lets us delay teaching transformers), but this is by no means a secret that we've discovered.
I think the biggest trick is to write simple code. I think when you give a junior engineer a sufficiently powerful hammer (like Haskell and its effect tracking), they will attempt to nail absolutely everything to the wall (e.g. build a taxonomy of every possible effect and every possible exception and write your whole program in this framework for Maximum Compile Time Safety).
Avoid this. It's just not a productive use of time, and the gains to safety often are not worth the effort and velocity and overhead you paid to build the underlying framework. Focus on the high-ROI piece of the program. Write the effects that you would use for testing, do the rest in IO for now, and come back later if you find that you need more granularity in your effects. The important thing is to ship the product.
(I go into much more detail about this in the Serokell interview linked in the other HN comment.)
Hi there. I'm working on my first Haskell app in production; Just wondering..
Do you guys use parsec in production? What is the motivation for having Haskell to be half of the codebase? (Are there some open-source domain-specific libraries that are used intensively? Or do you guys prefer to implement things in-house (on top of libs that are more abstract)?)
And if parsec is used in production, any tips on how to best design/debug parsec parsers? (Thanks!)
Lastly, in your experience, what are some of the best monads / design patterns to improve team productivity? :{
I think we waited too long to ship our initial prototype to customers, but a big part of that was that we were pretty sloppy in our product management at that stage of the company (I think we had closed the A about a year ago, and did not yet have our first dedicated PM), and our technical rewrite was also partially a feature and UX rewrite. It's been several years since, and we have clawed our way out of that hole, and I think our delivery process is actually in a really good place now.
If I were to give advice to people looking at adopting new languages, I would say to get it into production ASAP. I believe the keyword to search for is the "tracer bullet pattern". One of my favorite blog posts on this is https://blog.thepete.net/blog/2019/10/04/hello-production/
I think Haskell and OCaml are popular in finance because they're high-level and familiar to math/quant types (who are broadly--and this is a sweeping generalization--more familiar with expressing algorithms in recursion than in procedural sequences).
I don't have much experience with OCaml, but I can also say that Haskell in particular has incredible support for building very high-level libraries that are both expressive and have strong compile-time guarantees. I've found it's relatively easier for a programming expert to build such a library and a domain expert to consume the library in Haskell than in many other languages I've seen (perhaps Python comes the closest, but that feels to me more like an ecosystem thing than a language thing).
OCaml doesn’t give a specific advantage in trading, it’s just that Jane Street like to be pretentious about most things including programming languages.
Do you mean no specific advantage relative to C++? If so, then wouldn’t it be an advantage that the GC lets you be (nearly) certain that you have no memory bugs?
For us, there were a couple advantages. For context, I work at FOSSA (https://fossa.com/). Our core product solves software supply chain needs in enterprises (around licensing and security), and our core technology is around compiler, build, and source code analysis.
Off the top of my head, 3 advantages stood out:
1. First, if you're not going that far off the beaten low-level path, Haskell has incredible productivity benefits. Effect tracking has enormous benefits for testability and understandability. If you've ever been down a debugging rabbit hole shaped like "there's no way this logging call is sending that API request", then you might be pleasantly surprised to discover that you can statically guarantee that this doesn't occur in Haskell programs! Pattern matching, algebraic data types (sum types!), and typeclass derivation make it much easier to make it impossible to construct invalid representations of data. Other languages are finally picking this up, but their versions of pattern matching often have caveats for backwards-idiom-compatibility. And monads are a very powerful abstraction. It's like being able to write your own semantics for async-await (I've talked more about this before at https://lobste.rs/s/7cllte/monads_part_six_really_what_is_mo...).
2. Haskell was a good domain fit for us. One thing we build is the FOSSA CLI (https://github.com/fossas/fossa-cli/), which runs in customer CI pipelines to analyze their builds. It's a very compiler-shaped problem: shell out to some tools, do a lot of parsing, think very hard, and then spit out a JSON blob to send back to the API. Our first version of this was written in Go. At the time of development, writing correct, testable parsers in Go was like pulling teeth. We have a relatively small headcount-to-product-surface-area ratio, and our team was running up against the overhead of rewriting traverse in Go over and over again (that's a Haskell-flavored joke, but if you've ever been annoyed at writing yet another for-loop in Go, you get it). We decided to hack out a prototype in Haskell, and it turned out to be a good fit.
3. Lastly, the kind of people who wind up working at FOSSA and are interested in the code analysis bits tend to be the same kind of nerds who love Haskell. We had lots of people on our team who were chomping at the bit to try it, so we decided to try it out. I really can't understate how big of a productivity difference it makes when people are working with tools that they actually enjoy rather than are merely forcing themselves to use. It is night and day.
What kind of applications your team are working it?
How's the overall development process felt?
I'd like to know the reason to choose Haskell over other languages.
Re: development process - it's very similar to development in other languages. Write, compile, yell at compiler, push, complain about how slow CI is, deploy. You know, the usual.
I think the most interesting difference is the _onboarding_ curve. Haskell's curve is pretty brutal, although I think most of this is because of bad pedagogy (many monad tutorials are bad, and beginners can't tell) rather than because of intrinsically difficult concepts. Some observations:
1. Empirically, zero to code review is roughly six weeks for a professional industry software engineer. It's not that much longer than other languages we've had to teach. But it _feels_ very brutal because zero to side project is roughly three or four weeks. Contrast this against Go, where zero to side project is about five minutes.
2. Having an experienced Haskell engineer on your team to start with makes a WORLD of a difference. You've gotten a type error - why? Is it because GHC is doing weird backwards type inference stuff again? Or is it because you've misunderstood this fundamental concept? Or is it because you've done a typo, and GHC has inferred a downstream site to be a type error? This sort of thing is very difficult to explain in words and in general, and much easier to pick up through experience and mentorship. If you do not have an experienced Haskeller at your disposal, I would strongly recommend starting with side projects first, and using the Functional Programming Slack (fpslack.com), who are some of the friendliest and most patient folks I've had the pleasure of talking to.
Most tutorials go off the rails because they confuse types supporting this pattern with "being a monad". For example, arrays in JavaScript support the pattern through the `flatMap()` method, but saying "a JavaScript array is a monad" is misleading because most of what people do with arrays are unrelated to this.
As a pattern it is very general. It strings a sequence of functions together, but doesn't care about the semantics of the functions or the types involved, as long as each function just return the same generic type.
But many explanations take the semantics of how some particular types use the pattern and generalize from that. E.g. list and option types are monads, so monads are explained as containers. Or IO and State uses the pattern to represent side effects, so monads are explains as a way to have side effects in Haskell.
This is what leads to the bizarre metaphors, like monads are boxes, monads are spacesuits, monads are train-tracks etc. Each metaphor matches some uses of monads but breaks down on others.
You can just say that a monad is a way to chain operations together by wrapping a value. The consumer usually doesn't need to know the gore-y details of how a monad is implemented, only the purpose of it and how to use it.
A Maybe monad just says the value may be Something(x) or Nothing. You won't know until you run the computation. If you use flatmap and give a function that takes an x and gives a Maybe[x], the monad will first map into Maybe[Maybe[x]] and then flatten into Maybe[x]. The computation has not happened until you execute it and internally all the functions have been composed together.
A List[A] just says, give me a function A->List[B], and I will flatmap (flatten `compose` map) it. So it maps each element into a possibly empty list of Bs, and then flattens it by concatenating them.
You can define your own monads, and as long as they obey the laws of monads, you get a bunch of stuff for free.
You say "by wrapping a value" and then confuse it with the wrapper right away.
No, monads are not wrappers. Some wrappers are monads, but not all.
inb4, monads are not pipes either.
Sure, a monad is a type constructor. The constructed type allows you to compose transformations.
Each monad constructs a type that behaves differently and expects different things, but in general; the monad is defined by a unit/point/return function which brings a value into the monadic context, and a flatMap function aka bind, which further breaks down to “flatten after map”.
So List[_] is a type constructor, given a type T, it produces a List[T], which defines some transformations. Return creates a single element list, and flatmap takes the A->[B] applies it everywhere and then concatenates.
The nice thing about the wrapper analogy is that even though it is technically wrong, it is easier for people to get it because it follows naturally from OOP and it is a sufficiently useful mental model imho.
IMO the spirit of this answer is a big part of the problem. This might not be exactly what the GP was saying but I've found you can quickly get a dev up to speed with a "good enough" sense of what a monad is. One that'd cover a vast majority of their needs, at least in the early going.
But then there are always people who start popping in and pointing out how those definitions aren't quite right. Which is true. But does it matter *for practical purposes* to give a dev a useful for now mental model that they can then use to figure the rest out later? I'd say no.
Then why not just say that a monad (instance) is a type `t` which implements `bind :: t a -> (a -> t b) -> t b`? It's as down to earth and down to the point as it gets, and each time a new dev thinks out a code path which would lead to `t b` they'd remember `bind`. No need for containers metaphors at all.
This. Just point to the definition in your language (NOT the category theory, unless that's what you're coding in).
In Haskell, a Monad is a type class with a method `bind`. In Scala that would be a `flatten` or whatever.
If a person who's asking doesn't yet know what the "type class" is, or how to read signatures, your "monad tutorial" would not make much sense anywat. Guide them to learn the prerequisites first.
What’s a `type`? What is `t a`? How do I get `t a`? How does `t b` become `b` when I bind again? Where is `b` after I bind? What about the reader monad and the state monad? Those have different signatures, so what gives?
For practical purposes you don't have to know what the monad "is". You need some good practical examples how to wield them.
Knowing some prehistory about doesn't affect usage. And metaphors are very tricky and personal.
The personal part is where the "monad tutorials" fall flat. They assume shared context which may or not may not be actually shared by a random reader on the internet. Yes, some monads are really about "wrapping a value" or whatever.
Reading a post using this metaphor when you want such a wrapping and/or deal with chaining wrappers regularly can bootstrap your understanding in no time. But if you're reading "wrapper"-flavored tutorial while dealing with "pipes" then the spell breaks, you end up confused, and another one joins the "monads are uncomprehensible" group.
Teaching is hard enough. Writing good tutorials is even harder. Successfully giving an universally good drive-by explanation is next to zero probability.
I suspect this is a knowledge variant of "XY problem" and you have to establish more context before answering.
And another problem that "what is a monad" is already a meme. Everyone has the burning desire to ask it, but usually there's no practical need for the answer. Without that confusion ensues. Or, even worse, a false understanding gets locked in and starts to proliferate, sustaining the memetic chain reaction.
Instead of rising to the bait and trying to explain monads, let's talk about why it seems to be hard to explain monads.
They're very general. Classes in OOP are general too — surely you can model lots of things with them — but a class always models a category of things with similar functionality, and an instance of that class is one of those things.
Monads are much more flexible than that. You can model nondeterminism with monads, as well as side-effects, exception-based error handling, state, (backtracking) search algorithms, and more. Could all of those things be an instance of the same OOP class? Surely not. Yet in Haskell, 'Monad' is just a "type class" (not dissimilar to an interface / abstract class in the OOP world).
Monad tutorials typically try to do one of the following things:
1. Try to explain the entirety of monads by giving a metaphor (burritos, anyone?) that only works for some of the instantiations of the pattern. It turns out to be hard to find a metaphor that covers all ground that Monad does, unsurprisingly. Personally I think this method is good, as long as you're honest about what it does not give you: an intuition for all Monad instances. It just gives you an intuition about some of them, but that could already be plenty to work with them usefully (and see 3. below).
2. Try to let the reader invent Monad by showing various instantiations and asking the reader to find the pattern. Personally I think this misses the point somewhat; yes, you can see the pattern, but that doesn't give any understanding per se. Why are those things similar, and why does it make sense to abstract over the idea?
3. Not actually explain, but instead say "work with them a bit and you'll understand soon enough". This is the one I'm most in favour of — after giving a special-purpose intuition for the monads you'll be working with as a beginner (IO, perhaps parser combinators, and not much else; perhaps lists (nondeterminism) to jump-start exploration into non-imperative monads).
Note that none of these give a nice and polished answer to what a monad is.
As alluded to above, the closest OOP equivalent to the kind of thing that Monad is, is a design pattern. It's a design pattern that can be encoded into a three-line definition in the language itself, and is super general.
No you don't. You are confusing monads with features which can be implemented using monads. In Haskell monads are used for modeling side effects, but this is not the case in other languages. In Haskell exceptions are implemented using monads, but this is not the case in other languages.
Yes you do. `flatMap`ping an array in javascript is using a monad every bit as much as `join`ing a linked list in Haskell. `Promise`s would be monads were it not for the (inexplicable, terrible) decision to make it impossible to nest them. C#'s `IEnumerable<_>` is not quite a monad, but only because it doesn't have a preferred implementation. Pick one, and it's as monadic as `Maybe` or `Either`. And of course function types are always monads.
What these languages don't have and Haskell does is a way to talk about all monads, all at once. Instead of
replicateM :: forall m a . Monad m => Int -> m a -> m [a]
I find promises and futures in other languages (python, JavaScript, rust) close enough to how monads work to gain a very productive working intuition for them.
You don’t need to explain the monad laws to a beginner. They don’t need to know what bind does. All they need to know is how to use do syntax, and enough of how it works to use IO, Maybe, State, and a few others.
After they’ve been working for a month or so, they’ll “get” monads on a gut level, and the details won’t be so confusing to them.
Yeah, learning by example and by doing is the best way to learn any programming concept. But using misleading and confusing analogies along the way will just make the learning process harder, not easier.
Explaining monads in terms of promises and futures will make a beginner associate them with asynchronous programming, which might make some sense for some monads. But now show such a beginner a simple list comprehension and explain it is also a monad, and I assure you they will be very confused! Explaining concatMap in terms of promises is a very convoluted way to explain something quite simple.
I see what you’re saying. I agree that abstract metaphors confuse people. But clearly, the way we are teaching monads right now isn’t very effective for most people.
Haskell comes from a population that loves to understand things from the ground up, and most people learn by doing.
So, what if, instead of an abstract burrito metaphor, we taught people 4 monads: List, Maybe, State, and IO. We explain how do syntax does “something different” for each of them, and how IO is very similar to promises, but the others aren’t. Then we let them play with those for a few weeks, submit some PRs, let the panic of not understanding subside, and then teach them the theory?
Think about how JavaScript developers learn promises. They don’t understand how they work under the hood at all when they first start using them. Then, when they need to go deeper and look under the hood, that theory is connected to their practice, and makes much more sense to them.
> So, what if, instead of an abstract burrito metaphor, we taught people 4 monads: List, Maybe, State, and IO. We explain how do syntax does “something different” for each of them, and how IO is very similar to promises, but the others aren’t.
A sometimes important difference is that every process in the pipeline is spawned at the start, and may operate on partial input. A function returning Maybe needs to complete before we know whether it returns Just or Nothing, and we can't start the next function until we have the Just in hand, as that's the input to the function.
TL;DR: Monads are Promises and async/await, but generalized to different implementations of `.then`.
You are right to be suspicious; it is _not_ that hard! I think people just get confused and overwhelmed because (1) there are many separate different concepts that are all being introduced at once, and (2) you need to learn a new debugging methodology at the same time. I think the vast majority of this problem is the huge deluge of bad teaching materials (especially monad tutorials written by people who do not use Haskell in production) and the shortage of good teaching materials.
Re: 1 - some separate, orthogonal concepts that I have seen people mix up while trying to teach Haskell:
1. Effect tracking (which is a thing that we _do_ using monads, but is not intrinsically tied to monads - this would be better served by being taught as "dependency injection but stricter")
2. Non-eager evaluation (which is a language runtime evaluation choice that Haskell has made, and which motivates the usage of monads where most languages have function bodies, but is again not intrinsically tied to monads)
3. "Purity" (a more confusing way to explain 1)
4. "Mutable state" (another thing we represent with monads, but again is not intrinsic to monads)
5. Monads the math / category theory concept (don't bother with teaching this - it will not be helpful until you have intuition for writing and operating the programs themselves first, which takes at least a few months to build, and even then it is mostly useful as a source of advanced techniques for library writers rather than application writers)
At work, we've developed our own set of educational materials for teaching folks the language. I'm working on externalizing them in my spare time.
From my personal viewpoint the most beneficial things we did:
1. mostly pair programming, so we all knew the current state of the codebase and the path forward
2. When we didn't understand something, we wrote a separate small prototype of that feature / library / reference to teach ourselves how it worked. ( async, monad transformers, lenses, many reflex features )
3. We paid Obsidian for a one hour a week meeting to help us when we couldn't figure it out by ourselves. ( https://obsidian.systems/ )
4. nix for cross compilation and build / dev automation
5. Several hours a week of explicit teaching each other what we knew. This might should go first in the list, as the culture of being open and teaching was a massive benefit. We had one hour a week where the nixpert taught nix, one hour a week where I taught beginning Haskell, another hour a week where I taught Advanced Haskell (things I was learning that might help). We also had two hours a week where we all got together and worked on the stickiest problem along the path to shipping.
Most software dev jobs I've had want me to "do the thing" and have zero time left over for teaching / training others. I wanted to take the opposite approach and this paid off far more than my already wild hopes.
I would personally keep "pair programming" at the top of the list, this _really_ helps everyone learn.
Even more importantly, it tightens the loop between "write code, review code, change code, merge it finally" since you're basically doing all of that "at once" in the best case.
We used reflex-frp, so our app was a webview that worked on localhost and Android. The docs say it also works on iOS but we don't have an iPhone.
The process was learning Functional Reactive Programming, then learning reflex-frp, then getting a contract with obsidian (creators of reflex) for one hour a week where we could ask questions.
We had a grant requirement to create a phone client for Tahoe-LAFS, a Python application with a bunch of dependencies, including ZFEC, a forward error correction library.
We needed bug for bug compatibility with the Python codebase, so I ran Tahoe on localhost and tested the Haskell client against the Python server. We used servant to build the API, since it builds both client and server side from the same description.
We don't know Kotlin. We both have some prior Java experience, but it's not our preference.
Since the original codebase is in Python, I would have used that if possible, but I spent eight weeks failing to load a shared object into a Python interpreter on Android. That got us to this past January.
I have had a Haskell job before, though I was entirely unfamiliar with Android dev.
In two days I was able to load a shared object into a Haskell interpreter on Android, so we tried that route.
I see, given the pain of using the NDK, and having to make use of JNI for any meanigful Android API, that approach seemed quite strange, when only having to target Android, hence the question.
In functionality, roughly. By LoC, I think the majority of our codebase is still JavaScript/TypeScript. Many of the subsystems still in TS are being rewritten in Haskell (not because we have a "rewrite in Haskell" mandate, but because they are due for a rewrite anyway due to scale and requirement changes, and the team that owns that domain now primarily writes in Haskell).
> Many of the subsystems still in TS are being rewritten in Haskell
Have you tried fp-ts ecosystem? Code written in it looks pretty similar to Haskell, but it's still the same language, and you can gradually adopt it in the codebase instead of re-writing the whole thing.
(I've also sent a question about personal professional development to your email from profile, hope this isn't abuse of your kindness here)
Fp-ts is one of like, 4 things in life I feel the need to shill for. It's docs are a little rough coming into it for the first time, and I think some of gcanti's tutorials are a little to complex. But I've slipped it into 3 or 4 moderate sized projects. Every time someone goes to touch it there's initial confusion, a 5 minute explanation of Either, 5 minutes of Q and A, and then they love it.
The only bad thing about this is, now that I've started writing functional code with it, I don't want to go back. I'm in-between jobs right now, and finding a company which utilizes it is a challenge.
We have, and it's Got The Spirit, but it's still not the same. It's just so much more verbose, so much noisier, and has so much less compiler safety. You don't get things like non-eager evaluation or effect tracking, your stack traces become much messier, and your open source dependencies are still in plain JS.
Overall, we found that fighting against the language this way was the worst of both worlds - you paid a non-trivial cost and did not gain enough benefit. But our original codebase also predated TypeScript (remember TLDRLegal? That's us!), and much of our TS was bolted on after-the-fact on a pretty significant existing codebase that was written in very idiomatic JS. A team with more resources to do technical refactoring (we were quite small and very focused on building product) or starting from scratch may have a different experience.
How are you computing upgrade paths? This seems impossible to do accurately, especially for Ruby, since you can't simulate the user's build's dependency resolution given that Gemfiles are dynamic.
Can you expand a little? Here's some technical background on what we're doing:
We have our own database of every version of every rubygems package alongside its runtime dependencies (like you see at https://rubygems.org/gems/pundit).
Then we parse your Gemfile and Gemfile.lock. We use the Gemfile to figure out gem group and pinned requirements (we run turn your Gemfile into a ruby AST since Gemfiles can be arbitrary ruby code; we use bundler's APIs to parse your Gemfile.lock). This gives us all of the dependencies your rely on.
Then we let you choose one or more package that you want to upgrade and the version you want to target (let's say Rails 7.0.4.3).
Now we have [your dependencies and their current versions], [target rails version], [all of the runtime dependency constraints of these gems]. We run this through a dependency resolution algorithm (pubgrub). If it resolves then you're good to upgrade to that version of Rails without changing anything.
If this fails to resolve, it's because one or more of your current dependencies has a runtime restriction on rails (or another indirect gem being pulled in by the new rails version). This is where the optimization part comes in. The problem becomes "what is the optimal set of versions of all your dependencies that would resolve with the next version of Rails". Currently we solve for this set trying to optimize for the fewest upgrades. As our dataset of breaking changes gets better we'll change that to optimizing for the "lowest effort".
Sure. There are a couple of stumbling blocks here:
> We use the Gemfile to figure out gem group and pinned requirements (we run turn your Gemfile into a ruby AST since Gemfiles can be arbitrary ruby code [...]
Here's an example Gemfile:
if RUBY_VERSION < "3"
gem "minitest", ">= 5.15.0", "< 5.16"
else
gem "minitest", ">= 5.15.0"
end
This makes it impossible to statically determine the direct dependency requirements of a project.
> Now we have [your dependencies and their current versions], [target rails version], [all of the runtime dependency constraints of these gems]. We run this through a dependency resolution algorithm (pubgrub). If it resolves then you're good to upgrade to that version of Rails without changing anything.
>
> If this fails to resolve, it's because one or more of your current dependencies has a runtime restriction on rails (or another indirect gem being pulled in by the new rails version).
This actually isn't that big of a problem for Bundler, which uses pubgrub, which to my understanding is deterministic. A deterministic algorithm means you can actually take requirements and simulate builds. There are two places where I would be hesitant:
1. This determinism only works if you also know what the universe of possible dependencies looks like. In many corporate environments, this is not true! Many corporate environments use private registries that mirror public dependencies, and may have a set of dependencies available that looks different from the public registry, which means your simulated builds will resolve to incorrect dependency versions.
2. As you move to support other languages, many other tools use non-deterministic dependency resolution algorithms. In particular, NPM is famously non-deterministic, which makes it impossible to simulate an NPM build.
---
When you're trying to just determine that there exists a build that resolves properly, these issues aren't particularly painful. At $DAYJOB (and what I suspect you will want in the future), we are often trying to predict the exact build of a user given a new set of dependency requirements (e.g. so we can predict their new vulnerabilities), which means doing accurate build simulations is much more important.
One note - we're not as concerned with predicting exactly the user's build as I think your $DAYJOB might be. We need to scan the space of possible valid resolutions that'd do a big framework upgrade, then pick an optimal (least effort) option from that space and chart your path towards it. Concretely that'll mean opening incremental PRs against your codebase where we provide lockfiles that are individually valid and accumulate in getting your upgrade done.
> where we provide lockfiles that are individually valid
Providing lockfiles is a really interesting idea! That certainly solves the "we need your non-deterministic build tool to reproduce an exact build that we found" problem.
We haven't explored this route yet because a lot of our customers use tools that don't support lockfiles (e.g. Maven - Java in general has a lot of legacy stuff).
Maven POM dependencies act as version pins, until you run versions:resolve-ranges to bump them. The only exceptions would be using a SNAPSHOT version (where each build gets a timestamp and is only cached briefly) or your Maven repo admin replaces the contents of a pre-existing artifact (and you'll see a lot of cached checksum warnings about it).
FOSSA | Software Engineers (Mid, Sr., Staff), PMs (Mid, Sr.) | USA, Canada, Remote (able to work ~US time zone hours)| Full-Time
FOSSA builds developer tools to help engineering teams manage their open source. We help enterprise customers discover legal (licensing and copyright) and security (vulnerabilities) risks in their dependencies, provide tooling for them to catch these issues in CI, and automate the tedium around policy enforcement and report generation. As companies adopt more open source, their engineering teams get bogged down by more distractions around compliance and security. We help automate away those distractions.
We build an open-source CLI tool (https://github.com/fossas/fossa-cli) that integrates with compilers and build systems to extract dependency and build information; a backend distributed system for analyzing dependency metadata; and a web application with a policy, reporting, and enforcement engine.
This is an excellent idea, and looks like an excellent product. As an engineering manager, I've built many of these tools myself before as one-off scripts. I've always wondered whether the market cares enough about this problem for it to be a viable startup.
Let's separate two related but distinct concepts here: (1) effect tracking, and (2) monadic IO.
---
The concept you're asking about is called "effect tracking", and the argument for effect tracking is very similar to the argument for static types.
In a world without static types, functions can be passed values of any type, and can return values of any type. Adding static types makes this code easier to reason about because we can statically enforce (i.e. enforce without needing any runtime checks) that functions only take values of a certain type or return values of a certain type. This makes reasoning about the code easier because it limits the possible things that a function can take or return, so instead of thinking about "what happens if I provide any kind of value to this function?", you can think "I know this function takes an integer, so I only need to worry about understanding how it behaves when provided something that is an integer".
In a world without effect tracking (this is most programming languages today), functions can perform any side effect. Adding effect tracking makes this code easier to reason about because we can statically enforce which side effects a function does. For example, effect tracking makes it possible to express "this function takes a callback function as an argument, but that callback argument is not allowed to print anything". Alternatively, you can also look at a function and know for a fact that might do IO, or that it doesn't do IO. When you're debugging, this helps narrow down the scope of places that could be doing something wrong, because now you'll never have a function that _looks_ like it's doing something innocuous, but is secretly writing a log statement to a file, or pinging an API over the network, or drawing something to your GUI window.
---
Monads are a way to _implement_ effect tracking within an existing type system, rather than inventing a separate, orthogonal "effect tracking system". This allows us to reuse a lot of the existing tools and theories that we understand about types and type theory, and apply them to how we do effect tracking.
Exactly how this implementation works is another topic that gets a bit involved. But note: (1) there are ways to do effect tracking that are _not_ monads, or that do _not_ integrate with the type system that the language uses for values; and (2) monads have applications that are _not_ effect tracking.
That's true - there are many nuanced factors around life expectancy. This is just a quick hack that I threw together, but it's also open source and I'm happy to take pull requests adding this kind of functionality.
Haskell has a lot of problems, but this article is not good, and does not describe them well. It reads like it was written by someone who has tried Haskell, and has sort of written some code in it, but has mostly been put off by bad and confusing tutorials, and has never really grokked the language. Indeed, look at the author's GitHub - of their 50 projects, only 1 is Haskell, and that is a handful of commits before giving up.
My main complaint with Haskell is that it's actually a great language, wrapped up in (1) hilariously beginner-unfriendly syntax and tooling, and (2) a library ecosystem that does not quite understand the value of friendly documentation (although this is actually improving a lot). There are also some weird warts around the language itself, but those are mostly tolerable and not so much worse than the weird warts in all other languages.
My specific complaints about beginner-unfriendliness in syntax and tooling:
- The compiler will infer the types of names as generally as possible, and will run this inference both forwards ("this variable was assigned a string, so must be a string later") and backwards ("this variable is used as a string, and therefore must have been a string earlier") depending on what information is available for inference. This means that type errors are hit-or-miss; about 60% of the time, the type error will point you to your actual mistake, and the rest of the time it will point you to where the type inference algorithm got stuck in unification (this may not be the location you made your actual mistake, because the algorithm will try its best to infer concrete types working under the assumption that you didn't make a mistake, so it sometimes gets stuck trying to unify types at an earlier/later point than your actual mistake).
- The syntax is VERY minimal, and prefers minimalism over readability. In particular, Haskell has no explicit parentheses for function calls, and has implicit currying! This means that it is very easy to accidentally get the arity wrong in a function call (for example, `foo (bar baz) quux` and `foo bar baz quux` mean very different things!), and create an extraordinarily confusing type error (in the best case, an error about applying things to the wrong number of arguments; in the worst case, errors about constructing infinite types).
- This minimal syntax also means that you can sometimes type something that is _almost_ correct, but is actually valid syntax that means something totally different! Fortunately, your almost-correct expression will not type-check (so you won't ever mean something else without getting an error), but it's still a head-scratcher in the beginning to figure out exactly what the compiler is taking your code to mean. In some of these cases, you may even type something that is not syntactically valid vanilla Haskell, but _is_ valid Haskell-with-some-extension, and the compiler will unhelpfully suggest that you turn the extension on even though it may not be what you want (and also provides no explanation of the extension).
- The compiler tells you what expression has a type error, but never actually tells you the exact term that has a type error. You have to sort of guess at this by understanding the rough high-level of how the type inference works. It's something you have to really pick up, and is hilariously unfriendly compared to e.g. `rustc --explain`.
- Print-debugging works differently than what you'd expect. Nobody ever teaches you that `trace` is a thing (seriously, this should be thing #2 that is taught), and nobody ever teaches you "you're getting trace messages in a different order than you expect because Haskell is evaluating expressions with graph reduction, not in the order of the lines of code you wrote").
Really this boils down to: even though Haskell will do a very good job at preventing you from holding it wrong, you really do have to have a decent grasp on what you're doing to write a program. As a complete beginner, you can't just put something you think will mostly work and then rely on simple compiler errors to tell you where to spot fix (unlike e.g. Go, Java, etc.). Fortunately, this gets much better over time - you learn how to interpret compiler errors better as you get a grasp on the language, and you learn tricks for building incrementally correct programs up using `undefined` and friends.
My specific complaints about documentation:
- Contrary to this article, types actually ARE decent documentation in Haskell. The constraints you can do with types in a pure language are much, much stronger than in other languages you've used before. Trust me - this is a Blub paradox thing. It's just a qualitatively different kind of type safety.
- Unfortunately, although types do a great job at preventing me from plugging the pipes together wrong, they still don't tell me anything about the _semantics_ of a function. Lots of older libraries do a very bad job at documenting semantics, but this is getting a lot better.
- Arguments in Haskell functions are not named (or at least, their names are not exposed in documentation). This makes for some confusion sometimes - what am I supposed to pass as arguments to this `foo :: String -> String -> String` function?
- Lastly, a lot of libraries document very academically. Here's an example: https://hackage.haskell.org/package/logict-0.7.0.3/docs/Cont.... The documentation here is written assuming you understand how Logic computation works, provides no examples, and you're supposed to read the module documentation and actually go and _read the paper_ (in fairness, it's pretty short) that they've linked. This is a far cry from the NPM world (which, for all its faults, has really embraced accessibility in documentation), where everybody has a quick start.
Overall though, it's a really good language. Once you get monads (and a HUGE part of this is not the idea of monads, but the really unfriendly syntax and error messages of Haskell; IMO the _real_ lightbulb moment is grokking the difference between data, types, data constructors, type constructors, and typeclasses), a lot of things make a lot more sense. The testing story is amazing, the refactoring experience is amazing, the incremental development and REPL stories are great, and there are some truly astounding abstractions you can use that are just impossible to express in other languages. It's my regular side project driver, and my team maintains a Haskell codebase in production.
Lastly, a word of advice to anyone looking to actually make a good-faith evaluation of Haskell: try it, make something substantive with it, and stick with it until you actually grok it. Ask for help from the FP Slack! Seriously, don't just pick it up, get stumped by error messages, and give up. The tooling is very beginner-unfriendly, but fortunately the humans over at the Functional Programming Slack (https://fpchat-invite.herokuapp.com/) are VERY friendly. As a first project, I'd recommend doing the Advent of Code, or building a small web application (IMO with Scotty, which feels like Express, rather than Yesod, which feels like Rails. Yesod is almost certainly better in production, but has a different and IMO less instructive learning curve). If anyone needs help, follow along with my AoC 2020 (https://github.com/liftM/advent2020), although be aware that some things I do there are a bit fancier than necessary.
I find that articles like this that complain about things like type abstractness or the mathy concept names really are not representative of the actual pain points with Haskell, and are FUD more than useful. Yes, the tooling is beginner-unfriendly. Yes, you will probably need to ask some questions on the FP Slack on your path to learning, things that a friendlier language would have documented. But this mystical aura of "ooh, there are types and they are very abstract!" is not reflective of reality, is not a criticism grounded in the reality of writing Haskell, and IMO is misleading for people genuinely trying to assess the language.
There are two specific things here that make me reluctant to use Unison Cloud in my own work:
1. It doesn't look like there's any FFI or way to shell out to other tools within Unison Cloud. I understand that this is necessary to provide the desired static guarantees, but the lack of an escape hatch for using pre-existing code makes this a really hard sell.
2. Typed storage is excellent! What are its performance characteristics? In my experience, gaining expressiveness in storage systems often requires trading away performance because being able to store more kinds of values means we have fewer invariants to enable performance optimizations. How do migrations work? I've always found online migrations to be a major pain point, especially because data rapidly becomes very heavy. (At a glance, it looks like storage is key-value with some DIY indexing primitives, and I couldn't find anything about migration.)
The approach article asks "Why is it so complicated, anyway?". My guess would be that:
1. For small projects where you can toss together NextJS and SQLite and throw it onto Hetzner, it really _isn't_ that complicated.
2. For large projects where with large amounts of data, high availability requirements, and very large scale, performance and operability matters a lot, and none of these all-in-one systems have yet to demonstrate good performance and operability at scale.
3. There really is not that much demand for projects between these two sizes.