I have the feeling that most developers these days suffer from too much tooling in every area. And would be 2 to 10 times more productive without it.
Most friends that I watch doing their deployment would be way more productive by simply using rsync for file changes and raw database queries when the DB needs a change.
I can see that happening as a result of some other requirement, but I can't work out why you'd purposefully structure a team so that only one team member can push to production. To me, the downsides (single point of failure) outweigh the benefits (ease of deployment.) Is there something I'm missing?
I think the argument isn't so much that "everyone needs to be able to push to production" as much as "more than one person should be able to push to production". Bus factor is an important variable.
Comfy was created because we used environment variables on multiple servers, for many environments (prod, staging, demo, dev, etc), deployed by a few developers.
The purpose of the tool is to help developers share their config and keep sync with the rest of the team.
But I agree, if it's overkill if you don't need it!
How does comfy handle devs who wish to have some personalization of their local config but otherwise get the latest version of the the dev environment?
Comfy is a store of configurations for deployed environment. It's not recommended to store you local config into comfy, but you could.
If you do so, you have a system of tags. A config version can be tagged and the tag moved.
So, the integration environment can retrieve the config with the tag 'integration', say.
And you can retrieve the latest.
Local: comfy get development
Integration: comfy get development -t integration
Change the integration tag to the latest version :
1. Get the latest config hash with `comfy log development`
2. Set the tag integration to the latest `comfy tag move development integration <hash>`
Feature flags are a good example. One developer A enable a feature and deploy its version on integation.
Two days after, developer B deploys its version, but isn't aware that A deployed with the new feature enabled, so the feature is disabled with its deployment on integration.
Ideally, developers should use a centralized configuration management to no suppress each other configs.
That's why we use Jenkins or the Atlassian suite for example. But it's heavy to install and maintain sometimes.
I wanted to post this earlier but hit the posting limit, so, here it is, with apologies for the delay forced upon me:
Tooling is literally just tools. You can have good ones and bad ones and either use them well or use them poorly. A deployment tool like Capistrano or deploybot is a tool, but so is a shell script that runs rsync the the same way each time it's invoked.
An automated deployment process that involves either zero or minimal human work, (e.g. 'click a button' or 'push to branch X in dvcs') means you get repeatability, and with that comes reliability. How many deploys will be messed up if someone is manually running rsync, because they forgot to exclude some files/dirs, or because they had a trailing slash in a path and things got put 1 directory level too deep?
For DB migrations in particular, I think you're alluding to not using a language/framework/model based migration (e.g. `rails db:migrate`) but instead to use just plain SQL. The difference for me here is whether you're suggesting they literally run the SQL required by hand (e.g. remembering/deducing which migration is required themselves, and either running an SQL file or typing the SQL into an interactive SQL session) or just that plain SQL should be used.
I have positive and negative thoughts about framework/model generated schemas. I think having the model system automatically generate/update schemas based on the model's properties, works fine for a dev environment, but for anything like testing/staging/prod environments, I believe the "best" solution is stored SQL files (preferably in the VCS itself) that are applied automatically by a tool. Those SQL files could be generated as an SQL 'diff' after running the Model generator if you wish, or they could just as easily be hand-written queries. Either way they should be reviewed as with the rest of the code anyway, so their origin is less important than their content.
And yes, I am somewhat "biased" in this: I've written a PHP framework that (amongst other things) will perform model-generated schema changes, and I've written a shell tool (written in shell) that will automatically apply DB migrations and rollbacks from a series of `{up,down}.sql` files.
As explained above, I see the former as a development tool, and the latter as a deployment tool.
At my work, we use an internal config store developed by another team for app config management, versioning and audit history.
It’s a bit flaky so we’ve been looking to replace it but don’t want to spend the company’s money building it. We’ve been looking but have not really found anything OSS.
We’re looking for:
- store text/binary documents
- full audit history of document versions
- get/put like api to retrieve/replace documents
- GUI to explore history/documents
- easily deplorable on Windows (ideally without an app server)
Hmm, I’m actually working on a project that’s hopefully not too orthogonal to your need but could overlap on a couple of points above - https://www.deltatrail.io. Basically changelog as a service (which may cover the audit trail history you mention). Our logic is that most shops don’t want to build something like this from scratch. Take a look if you’d like. We are still in beta and experimenting with market so let me know if there is a potential overlap and we can fill in the gaps where needed. Reachable at Ryan at Deltatrail.io.
Vault is for secret management. If you are just looking at a key value based store or a service discovery tool, you might want to have a look at consul by Hashicorp.
I just wish there was more of a standard for for configuration files. Aren't all config files at some level going to be tree like structures? I know it is impossible for everyone to agree on one format, like yaml vs json, vs xml etc. but maybe there could be a common library and standard that could allow each application to load it's configuration from any of these types of files? That way if one organization or distro or whatever wanted to all use Yaml (for example) for all of their config, they could do that.
IMO most configuration should just be environment variables. How you get them there is up to you, but exporting env vars from a config file shouldn't be a super hard task.
Having said that, if your organisation chooses to use YAML, personally a better choice would be to just burn the place to the ground.
Hmm props to getting this out, but my way is has always been "bash is good enough" ja its not perfect a d feature rich but gets you 80% there and most linux based coders knows enough bash. I hate learning a new "language" just to deploy code or servers. Guess im just too old :)
Does it have any features to handle inheritance / composition of configurations?
We usually have a slew of different environments that are mostly the same except for one or two parameters. I would like to find a solution that has the ability to control common parameters in one place without having to code the inheritance into every language / setting where the config is accessed.
In general it seems like simply storing your config in JSON or YAML and putting it in git provides most of the features of many config solutions and the overhead of deploying something separate to handle it rarely justifies the marginal benefit from that.
I didn't want to jump in with this earlier since comfy is the main topic here but https://configrd.io will do the inheritance you are asking about. It's not cli but also has an API, handles secrets and deploys on prem.
thanks! actually looks really cool, love the integrated support for storing encrypted secrets (even if not supporting many providers yet). Will explore it more.
We initially thought to migrate versions from staging to production with a sort of composition, but we didn't implement that because there was a high risk of mixing credentials between environments.
That said, you can easily copy a config from one env to another and update that new config in no time.
comfy get env1 > tmp.json;
comfy setall env2 tmp.json;
comfy set env2 <key> <value>
Most friends that I watch doing their deployment would be way more productive by simply using rsync for file changes and raw database queries when the DB needs a change.