This is a variation on one of my favorite software design principles: Make illegal states unrepresentable. I first learned about it through Scott Wlaschin[1].
IMHO, Poetry is the best we have in the Python dep mgmt space, and it's still endlessly frustrating. It's especially hard to recommend it for newbies looking to get up and running with even a simple ML stack. Check out this thread[1] on the Kafkaesque nightmare that is trying to install PyTorch with Poetry.
Poetry won’t ever be the answer because the Poetry maintainers haven’t shown the maturity to be real leaders in the space. It’s good, as many narrowly opinionated projects are, but ultimately the core maintainers are not interested in use cases they see as outside their vision for the tool. Which means it will never be the one tool to rule them all.
The whole "Python is how science gets done" meme is one of the dumbest things we've allowed to be foisted on otherwise unaware/unsuspecting users (such as the kinds of academics who end up being the victims of the Python ecosystem shitshow). Who knows how many setbacks in science we've suffered, not to mention billions of dollars of productivity lost, sticking to such an unworthy programming system/language/environment. All because, like, colons and significant whitespace make programming so much easier to pick up when you compare it to making someone look at curly braces—which, as we all know is the hardest part of programming.
Once you start to look at scientific packages in other languages, they have the same issues as Python does, because they start to use scientific libraries written in C and Fortran, as rewriting 50+ years of code is actually really hard.
I've used DVC in the past and generally liked its approach. That said, I wholeheartedly agree that it's clunky. It does a lot of things implicitly, which can make it hard to reason about. It was also extremely slow for medium-sized datasets (low 10s of GBs).
In response, I created a command-line tool that addresses these issues[0]. To reduce the comparison to an analogy: Dud : DVC :: Flask : Django. I have a longer comparison in the README[1].
You might be referring to me/Dud[0]. If you are, first off, thanks! I'd love to know more about what development progress you are hoping for. Is there a specific set of features that bar you from using Dud? As far as testing, Dud has a large and growing set of unit and integration tests[1] that are run in Github CI. I'll never have the same resources as Iterative/DVC, but my hope is that being open source will attract collaborators. PRs are always welcome ;)
I very much agree with you about DVC's feature creep. The other issue I have with it is speed. DVC has left me scratching my head at its sluggishness many times. Because of these factors, I've been working on an alternative that focuses on simplicity and speed[0]. My tool is often five to ten times faster than DVC[1]. I'd love to hear what you think.
Thanks! I really like your clear explanation of how Dud differs from DVC (and I prefer your version in all cases).
Would it be possible for Dud to push/pull from a DVC remote and use the DVC shared cache? That would be really useful so I (iconoclastic free software user) could use Dud on my machine/acocunt, but still share data and artifacts with other people (who don't give a shit what tool they use) using DVC on their machines/accounts.
Also: Does Dud support reflinks at all? Or does it only support symlinks?
Unfortunately, there's a few things that currently hinder compatibility with DVC caches. First, Dud uses the Blake3 checksum algorithm, and DVC uses md5. This means the content-addressed caches will have completely different file names. Second, directories are committed to DVC differently than they are in Dud. For directories, not only will the committed file names not match (due to point 1), but the contents will not match either. Both of these things could be addressed, but it would take a lot of effort and would likely cost Dud in terms of its two main goals, speed and simplicity. I'm not opposed to this if we can make it work, though.
Dud currently does not support reflinks, but I think adding reflink support would be fairly straight-forward. Just curious: What filesystem and OS are you using for reflinks?
I'd be happy to chat more about this. Feel free to open GitHub issues for these items. I welcome contributions as well. ;)
I don't know about replacing Make with Docker, but I use the two together to good effect. One of my favorite hacks is adding a 'docker-%' rule in my Makefile to run make commands in a Docker image[1]. It's a bit mind-bending, and there's a few gotchas, but it works surprisingly well for simple rules.
I toyed with a similar setup recently and I had the problem that I, too, would have rules depend on a "docker build"-step (like `docker-image` in your example). Usually `make` would stop building the dependency for non-PHONY targets if it finds the correct file but in this case it obviously cannot find anything.
I tried `touch`ing hidden files for each step and then add those as a dependency but that is not very elegant. Do you have this problem at all?
A thousand times, yes. I've wanted to write this same article. Thanks for saving me the time!
The industry is going to great lengths to avoid writing configuration in any ubiquitous imperative programming language. We're seeing the proliferation of hyper-specialized, clunky declarative languages with sub-par tooling and package ecosystems. In what world are templates acceptable code? I don't mean to pick on anything specific, but this[0] is the most recent example I've come across, and it's far from the most unreadable examples.
[1]: https://fsharpforfunandprofit.com/posts/designing-with-types...