Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Solo-preneurs, how do you DevOps to save time?
356 points by aristofun on Oct 12, 2021 | hide | past | favorite | 317 comments
When you're 1 man show (or tiny team) working on some web oriented application beyond POC level and beyond heroku (for whatever reason) — managing your CI, deployments/rollbacks, DBs etc. looks like a nightmare to me.

Just properly setting up and maintaining a small k8s cluster looks almost like a fulltime job.

I wonder how do your CI, deployment workflows look like?

Any particular tools, practices, guidelines, architecture patterns to reduce pain & timewaste?

How do you CI => deploy/rollback stateless instances? How do you manage your DBs and backups? Etc.




Not a solo founder, but I do a lot of ops professionally. Unless you have a specific reason, I say it’s best to avoid complicated tooling until later.

k8s, ci, etc are all really useful and solve a lot of hard problems, but god are they a bitch to set up and then monitor, fix when something weird happens, secure from nasty people outside, patch when some update becomes necessary, resist the urge to “improve”, etc. This doesn’t include the time it takes to work these tools into your project, like k8s for example really requires you grok its model of how stuff it runs is supposed to look. There’s a reason entire companies exist that simply offer managed versions of these services!

If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running) and honestly it’s pretty fun to just run a command and poof you’re deployed. Then you can just wait until it starts making sense to use fancier tooling (more load, more sites/DCs, more services, more people/teams, etc). Monitoring is always good to start with, for other stuff a good litmus test is that price points for hosted k8s & ops stuff will start to make sense, you’ll be paying in time in any case. Who knows, maybe when that happens something new will pop up that makes even more sense. Until then, reducing the amount of stuff you need to worry about is always a good strategy.


> spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script

I never realised I was using spooky arcane oldhat stuff! I feel wizardly now.

My projects (for small clients and myself) basically use this.

- A "build.sh" script that does a local build of back end and front end

- A "deploy.sh" script that scp's everything to the server (either a digital ocean VPS or an EC2 instance), runs npm install, runs database migrations and restarts with pm2

So running my entire CI pipeline in terminal is: ./build.sh && ./deploy.sh


If you like this style of deployment, and you haven't looked into Ansible yet, give it a try, you may enjoy it. I had similar build processes before using Ansible and I have found it a really positive step up.


My favorite deploy is using a simple `git pull` instead of scp. I also avoid complex build tools when I can. You either need your built files in the repo or you need to avoid them. Either are fine for my small personal projects. The only real exception I make is keeping any necessary secrets out of my repo. Those get dropped on the server manually.

This also solves the roll-back problem mentioned elsewhere. Just checkout the previous version (as long as you don't make any side effects like DB updates that are incompatible).

I admit, this requires relatively simple software.


Why pull, as opposed to push(ing to the server as remote)?

Then you could have any necessary restart or migration or whatever run in a post-receive hook.

I realise that's starting to get back in to tooling and config etc., but it's not that complex, and if you're already using git anyway...


I do the same thing (just a couple os customers, o en Linode per customer). Hopefully I'll need something more complex eventually.


Naive question: what's the correct way of doing this when you need to cp your content to a dir like /var/www/html that your user doesn't own (when logging in as root is prohibited)? My "spooky" (and probably very stupid) method is an Expect script that lets me supply a password to sudo.


Ideally you want some atomic deployment strategy, rather than having to deal with whatever crap might have been in the directory previously. Something like - extract your deployment artefact to a new, unique directory - and then read-only bind-mount that directory as /var/www/html.

> and probably very stupid) method is an Expect script that lets me supply a password to sudo.

The old school sysadmin way of doing this would be to have a dedicated deployment user (ssh pubkey auth only - possibly restricting only specific commands), with a sudoers[.d] configuration to allow that user to run an explicit group commands without a password (NOPASSWD)


Usually these directories are owned by another group (www or www-data) so that your web server program (nginx, apache, etc) can access them without running as root. If you add your user to that group, you should be able to manage files in your web root without sudo. Be careful with permissions, though - you may need to chown to set the group after copying so that the files are readable by your web server.


Login as the user that owns that dir, don't try to hack around the user permissions.

Also you probably don't want your web under /var if its a partition and you fill it with lets say user content then you will lock up your server.


a pattern I have seen before is to clone the branch commit of each release to a folder /var/releases/{release_commit} and then html is not a folder but a soft link so /var/www/html -> /var/releases/latest_release_commit this is useful if you need to revert back quickly. But then releases folder needs to be cleaned up, or you can run out of space.


That's dangerous unless you've properly configured your web server to block access to your .git directory.


Safer/easier to keep the .git somewhere else entirely, rather than in the working directory. Git's got arguments that let you do that. Then you can check out to a dir that doesn't have the .git dir in it at all.

Alternatively, it's pretty easy to use git-archive (optionally, with an exclusions list to keep out files you don't want to serve publicly) to do something similar without ever cloning the remote repo. You can fetch a zip or tar.gz with a given commit's files, straight from a remote repo.


Ansible can do this (you tell it the user to initially log in, and that you need to "become" another user for a task)


This is a great setup if you work with a lot of clients. Every client we work with wants everything to run on their infrastructure that they provide, and then they provide the weirdest jankiest setup you could imagine. Because of this I keep everything basic if it cant run on a fresh linux install or less (looking at you docker) then its going to cause issues down the line.


Basically same here. In cases the deployment has to be zipped and manually deployed to elastic beanstalk. On some EC2 instances for projects running a PHP stack, I don't even bother with this, just SFTP script changes to the server straight out of my IDE. Take care of any rollback needs with local Git versioning.


This is great. Until you need to roll back.


I had a similar setup a couple of years ago, where I had to deploy without downtime. My solution was to simply have the old version running in parallel until I was certain the new version was ok

----------

Static website:

1. Setup NginX using a symlink to the current version

2. Copy the new files to a separate folder

3. Change the symlink to point to the new version

----------

REST service (behind NgninX reverse proxy):

1. Current version runs on port X.

2. Deploy new version and run it on port Y.

3. Update NginX config to use port Y and reload

4. If you need rollback, just reverse step 3.

This can be done using scripts or Ansible too if necessary.


Keep a simple tool like Ansible around for those spooky admin tricks and you can take advantage of the Ansistrano plugins for smooth deploy and rollback (Ruby's great Capistrano tool ported to Ansible).

https://ansistrano.com/

It's pretty fantastic.


This looks awesome. I'm currently in the middle of learning Ansible now for my FjeeBSD jails. Have any other plugins to recommend?


Anything by geerlingguy on Ansible Galaxy.


> Anything by geerlingguy on Ansible Galaxy.

I've been writing some Ansible playbooks recently for the first time in years, and came upon geerlingguy's work. That guy is a powerhouse when it comes to writing Ansible roles/modules!


Why, thanks ;)


    git checkout abcd1234
    ./build.sh && ./deploy.sh
I don't see the issue.


If you have any migration, you probably want to rollback them as well


That's sort of pet peeve of mine: Migration are done separate from code deploys. Version 1 of your code runs on schema version 1. Schema version 2 does not make chances that will break code version 1. Code version 2 can utilize the chances made in schema version 2, but you're still able to rollback the code.

Each schema migration should also come with its own rollback script.

The downside is that you might need three migrations for some operation, but at least you won't break stuff.

The assumption that you can do a schema migration while deploying new code is only valid when you have very limited database sizes. I've seen Flyway migrations break so many times, because developers assumed it was fine to just do complicated migrations on a 200GB database. Or a Django database migration just lock up everything for hours because no one cared to think about the difference between migrating 100MB and 100GB. And I've never seen anyone seriously considering rolling back a Flyway migration.


Agree with this and have practiced and advocated for it. Make the schema changes to support the new feature first, then verify existing software still works. Deploy the schema change. Then develop the new feature, test, and deploy the program. That way you can deploy and rollback without needing to synchronously run a bunch of migration routines.


Why? Checkout what version you want to roll back to, and deploy it.


rollback.sh


This is the way


You're a wizard davedx!


rsync and scp are two of my top favorite commands.


Indeed, for a bunch of hobby things, I use

   rsync -rvzC --no-p --no-g --chmod=ugo=rwX --stats -e "ssh . ${remote}:${path}"
where `${remote}` is defined in `.ssh/config` and `${path}` is in the project config to move files over to a new deployment directory. Then, a quick command over SSH changes the symlink to the site's configuration file and restarts the web server.

The sites are either on Linode or SSDNodes. Has been working for me for decades and don't need to change it for these hobby things. After all, the log files prove that these things get more traffic from exploitation probes than real people ;-)


Also a hobbyist sysadmin at home, but after reading a few comments on the internet, I found that the rsync -z compress flag was bottlenecked on a maxed-out CPU thread of my Raspberry Pi 4 NAS transfers.

Admittedly, these are mostly "local" drive-to-drive transfers over USB rather than server-to-server transfers over house-wide gigabit. But consider trying the transfer without compression.

I'll definitely have to read up on your other flags though, if your rsync works from just one machine rather than both that might solve a minor problem case I have where the destination machine lacks rsync...

https://unix.stackexchange.com/questions/188737/does-compres...


Really good observation. All the settings are from a long time ago when link quality was way worse than what we get these days.

I should review those options, definitely.

> -C, --cvs-exclude auto-ignore files in the same way CVS does

Just goes to show you how long I've been using this stuff :-)

The additional options may not be relevant now that I am just deploying creating a new target directory each time.


Or just deploy it on Heroku and call it a day?


yah, this (or something similar) is the most straightforward solution. the whole point of heroku is to abstract away devops for solo/small dev groups and make deployments “one-click”. it’s unclear why that option was explicitly ruled out here.


Agree with all of this.

If you want an intermediate step you can just make docker images of your application and deploy and run those manually (again with shell scripts).

You can get a long way that way without having to deal with kubernetes.


Yeesh

I cannot imagine not setting up at least a basic CI. Even if you remove all unit/integration test alarms, a CI is still very useful. The 20 minutes of build -> upload -> deploy script -> check prod vs a 1/2 day to set up a CI and forget it.

I'm not sure you save much time but you save a lot of headache


I think you mean automation is very useful. I've worked in a lot of places where we had a single `make deploy` that could compile, test, upload the artifacts, deploy the artifacts to hosts, start the new services, and slowly transition the load balancer to point at the new instances while automatically backing out if error metrics started lighting up. This is just automation, it doesn't have to happen on every commit. Especially when working solo or on a small team, I might not want to maintain a lot of VCS branches and instead, just deploy when I feel it's ready. You have to do all of that stuff anyway, so if you later decide to hook it up to a VCS trigger, you're all set.


I guess that's fair, I'm probably coming into this having only seen it done a certain way and blind to other possibilities.

That being said, it would take a lot to convince me a standard CI/CD pipeline is not worth the investment. My only experience with non CI/CD deployment process was a very manual process at a startup, then when I saw my first CI/CD it felt like magic.


A good old makefile is a CI/CD pipeline. All the new fangled tools are re-inventing a lot of capabilities make already has. I blame the new kids on the block who have only ever known Windows :-)


> If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running)

If you have multiple people on the team that have access to servers, this will shoot you in the foot a year or two later. Good luck replicating that server setup when no one has any clue whatsoever about all the small tweaks that were performed ad-hoc over the years.


Next step up is something like Ansible to manage it all.


Yes. Then you realize that people may still SSH into your servers and manually tweak things when they need the fix "now". So then you start restricting SSH access and forcing everything through Ansible. And now you're deep in deployment best practices again ;)


I think you’re describing the growing pains that justify paying the costs of more complicated tooling. The big difference is that instead of needing to figure it out as a solo founder with bigger problems, in this scenario you get to figure it out as a solo founder with a team and presumably some extra $$.


> Good luck replicating that server setup

It is easy when the server is regularly backuped.


I'm fully on team spooky admin - however on my latest project, I've been loving being on the free tier of tools like vercel, which deploys from git commit, and firebase, which is mostly maintenance free. That really reduces the amount of devops in my day.


At this point CI is pretty easy to setup and monitor and it's integrated with practically every development/deployment tool, so I have to disagree about waiting to implement it.


What do you use for CI?


>you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running)

You would be surprised at how many old enterprises still do this. Sure, it's ugly, but it can be simpler than k8s, and anyone with scripting chops can understand it.


Lots and lots of best practice exists specifically to help teams, and especially teams with some normal amount of turnover.

The problems of a solo dev are very different than a dev on a team. Knowledge silos don't exist. Distributed expertise doesn't exist. There's no one to mentor, no shared vision to maintain, no intertia to combat.

I consult on big complicated team projects. I also manage multiple solo projects.

On solo projects, deployment is a script I run from my dev machine. I'm the only person who deploys; anything else would be solving a problem I don't have.

The only "CI" is running the tests before I run the deployment script. I'm the only one who needs to see the outcome of the tests. Anything more would be solving a problem I don't have.

Architecture is whatever makes the most sense to me personally -- which is often miles away from what I would recommend to a client, who needs something any new hire will recognize and be able to work with.

I pay a service to manage backups so I can walk away from a solo project for months and know it's ticking away.

The point is: solve problems you actually have. Don't try to run a "professional" operation by doing what big teams do all by yourself. Big teams have different problems.


'knowledge silos' in a one person team are real: Me today and me from 2 years ago when I set a thing up are different people, and I don't remember that any more than a team mate in a multi-person team would know it. This is where 'professional' approaches can help considerably: using 'standardized' stuff, instead of bespoking everything (even if bespoking is much easier), or making sure you write a TON of documentation for anything you don't do daily...


I'm a fan of the documenting everything even on stuff I wrote and maintain for myself. Because, like you said, old me was way smarter than current me. When he was writing this code, he had the domain space modeled in his head better than me now, 2 years later trying to fix/maintain something.


Probably the best tool you have is somewhere to write your notes down. What you did, why you did it, how to reproduce it. That saves you time in a year or two when you go 'hmm I did something like this before have to do it again because some CPU ate itself'. My old notes have saved me a lot of re-work over the years. Because I had written down what I did so I do not have to rebuild it. I can just skip along the notes and be most of the way there.

This bit seems to hold true across any size team for me. If I have notes I can spiffy them up and make cheat sheets out of them and bootstrap other developers faster. Or if it is just me, my feeble brain will forget odd details that I needed for something.


Absolutely, extensive notes are a huge part of my practice both on teams and solo [1] -- I'm trying to say "solve the problems you actually have" not "when solo, cowboy-code everything like a madman".

[1] https://sambleckley.com/writing/lab-notebooks.html


Totally agree with this. I do big corp work and the problem space is totally different from what is needed for a single developer.


> I pay a service to manage backups so I can walk away from a solo project for months and know it's ticking away.

Any recommendation of such service?


Borgbase. I'm not affiliated with them, I'm just a happy customer


Pretty much anyone you can pay to host a database, you can pay to back it up. Amazon rds, heroku postgres, google cloud sql... pick the service that works for you, and they have some affordance for backups. Backups are something I expect, rather than something I shop around for.


My experience says to backup to a different vendor.


This is part of my threat model. Is there an easy way to do this with e.g. hosted RDS or block storage device snapshots?


In my case I'm dumping + zipping the entire database at the application level. In my case is as simple as adding a library [1], scheduling the job and transferring to AWS S3 (my main application is on DigitalOcean)

[1] https://github.com/spatie/laravel-backup


Tarsnap


Snapshooter


Rsync.net


I have no idea why there's nothing here saying "don't". Go and get a Heroku account, hook it up to your Github repository's master branch for deploys, use Github Actions for CI, and then get on with life.

Yes, you'll pay more for a Heroku Postgres instance than you would for a VPS on Digital Ocean with Postgres running on it, likewise you'll pay more for Heroku Dynos than another VPS to run your application server. On the other side though, your backup, rollback, deploy, and scaling strategies can all be "Heroku does that", and you can focus on the much more valuable job of building a business.


Heroku is by far the best for this but just to throw in some alternatives:

  * Digital Ocean Apps -- somewhat finicky but works very similar to Heroku, it gets closer to bare metal so I prefer it
  * render.com
  * AWS Elastic Beanstalk -- though setting this up is non-trivial, it is very similar to Heroku with its "set it and forget it"
  * AWS Container Services -- if you're using Docker
  * Google App Engine
  * Supabase -- geared for SPAs but this may be the ticket depending on what you're building, is open source
You may want to use your own VPS but spend for a managed database solution as this has the most potential for catastrophe.


I would advise against Elastic Beanstalk, last time I used it the product had the stink of being on life support - they've not really added any new features to it in several years, and even before that it was incredibly flakey.


A concur. I got burned three times by Elastic Beanstalk in 2019 before switching away.

1. Their server manager process got in an unresponsive state. It wouldn't update my service. It stopped reporting logs from the service that was running. I went to the AWS Loft and consulted with the AWS Support technician. They had me log into the machine with SSH and troubleshoot. I was using Elastic Beanstalk specifically so I would not have to log into machines.

2. The console frequently showed servers as running when they weren't. And vice-versa. I wasted a lot of time trying to figure out why my server didn't start when it actually did start and why my server isn't listening on its port when the console incorrectly showed that it started. Logs were also delayed randomly.

3. The final straw happened on a Saturday. Elastic Beanstalk suddenly started returning errors. I purchased an AWS Support subscription and filed a ticket. It turns out Elastic Beanstalk Team had deployed on a Saturday and then rolled back. Unfortunately, anyone who did API calls through the new API version got new values added to their Elastic Beanstalk API backing database. The rolled-back the server errored when it saw the new values. They refused to just fix the bad data in their database caused by their bad deployment. Instead, they asked me (their customer) to download special management tools and perform manual operations to fix it. And I had to pay them to find out these instructions.

Now I use Heroku.


What additional features are you looking for? I've been using beanstalk for the past few years, and just wrapped up doing a large deployment on it. I've never found beanstalk to be a bottleneck or not have a feature I was looking for.


Recently Beanstalk got upgraded to their "Amazon Linux 2" making it basically pretty much a Heroku clone, you even set up Procfile and services ;-) It still requires a LOT of hacking and insider-knowledge to do some funkier things, but it's a great thing


I use Elastic Beanstalk daily it is maintained and supports everything on AWS, they've added ALB and WAP for example. It runs on Cloud Formation. Never experienced anything "flakey"


Also AWS Copilot if you like CLI based tooling, similar to Elastic Beanstalk but easier (at least if you're comfortable making a Dockerfile)


Was not aware of Copilot! Looks much better than Elastic Beanstalk


Have you ever tried CapRover? https://caprover.com/

Would be curious how it compares to some of these.


Add TinyStacks to this list. They are just getting going, but I got a demo recently - sort of a cross between Vercel and Heroku, but on AWS.


Fly.io is a pretty nice alternative too.


ERR_SSL_PROTOCOL_ERROR when trying to connect


Works now for me (https://fly.io)

Looks interesting


Does SPA stand for Single Page Application?


I think so. You'd use it as the backend for your SPA:

> Create a backend

But you'd probably host your SPA elsewhere, eg Vercel, Netlify, GitHub pages


I really like Heroku, when I work on NodeJS projects.

BUT: In the last year I've had some rather serious downtime issues that were 100% Heroku's fault. (Poor encapsulation of Postgres configuration via the connection string, and they mistakenly claimed that I violated their TOS when I didn't. They apologized and fixed the situation.)

I'd really like to see some good alternatives to Heroku. Not just because of their screwups, but because the languages that I want to work on have limited support in Heroku.


which languages do you want to work on?


C# and Rust.

Edit: Yes, I know there are community supported ways to do this, But I want something that is as slick as the NodeJS ecosystem on Heroku.


This! Plus it includes SSL provisioning, staging environments, background workers etc.

Other competitors like render.com are starting to look like they'll be competitive on features and are way cheaper too.


Yep! Used Heroku at a previous gig (now a mid-size startup), worked great, now on Render.com and it's also great. Also factor in fly.io and railway as quickly-growing platforms in the same PaaS sort of space.


I head up Dev Advocacy at Render (previously at Heroku). Happy to answer any questions about Render.


I'm really only waiting on PITR for postgres before moving all my workloads over to render. Do you know when that will be available?


(Render CEO) We're targeting early next year, but it might be sooner if we can hire faster!


Heroku Postgres – after all those years – still is vulnerable to MITM attacks. You can't reliably verify the authenticity of the connection between Heroku's Dyno and the Postgres database.

They use self-signed certificates on Heroku Postgres servers without a certificate authority present in the chain. No chance to establish reliable peer verification. A shame, I would love to use Heroku more often without compromising security.


Are you actually doing this for a production app with income you rely on? Because Heroku has horrendous uptime these days. Lot's of random emergencies on both worker and web dynos. Don't forget you share routing with thousands of other apps too so expect random hiccups there too. It's easy to give this advice but does not scale well with complexity.


Exactly. And when you are truly "beyond POC level and beyond heroku (for whatever reason)" then of course it is time to hire people.

I see this all the time. Entrepreneurs trying to solve the people problem by avoidance. The problem with the super efficient player is twofold. First, it's not really scaleable. Second, what if the person gets sick or is in vacation?


This is what I've been doing for years, but Heroku Postgres is so bloody expensive, it just doesn't make sense for apps that don't make much money. So I just moved my data to an RDS instance, shared between multiple apps. Cost savings are great, headache savings much less so. I would say if you are going to move off Heroku, do it with the apps/projects where you can f-up, or have downtime without much repercussion. AWS stuff is a deep dark forest, undocumented and confusing. I'd say for anything that is critical, just keep paying Heroku. It isn't worth it.


I'd even go higher up the value chain with something like Vercel or Netlify if your stack fits in their offering. Heroku is a definitely a good option if you need more flexibility. The costs won't be a bottleneck until you actually have enough scale that you can afford DevOps staff.


Vercel has the best dev UX in my opinion, everything just "works" the way you expected it the first time, and if not their documentation is very straight forward.


Until Heroku has a huge downtime (like recently) and you are f*ked. If you have your own VMs, you can quickly move them to a different provider, but it's hard to start figuring out servers when Heroku is down and your business stopped.

That said, I would also recommend Heroku to many.


I can almost guarantee that over a period of years you'll see significantly less downtime that can be attributed to Heroku than you would on VMs that you manage. Even if that isn't the case, if Heroku are responsible for it then you can mostly just kick back and let them fix it, rather than spending 24 hours without sleep migrating VMs between providers.

The context here is also that of a single entrepreneur building a business. At that point in the lifecycle of a product downtime really isn't that big of a deal, customers tend to be pretty understanding so long as you recover.


Anyone who kicks back while their web hosting is down... is a psychopath. That's just not going to happen. Are you accountable to literally nobody? Even as a solopreneur, there are customers/visitors. If my site is down, I am losing money/credibility/visitors/etc. Sure, you will probably get past it, but it means preparing to communicate to those people during/after. It also means understanding what went wrong and what could be done to prevent it. Just kick back and not worry about it sounds like you're doing nothing of consequence to anyone.


Totally agree. There's some impostor syndrome in play here. If you don't need the scale, why waste your time managing a more complex setup? Heroku might be slightly more expensive in the short term but your time is better spent building/selling than future-proofing or just doing it because larger operations have to.


+1. For pre-A series/pre-revenue/pre-1k paying customers and other early stage businesses you can do a lot worse than starting out on Heroku, especially if you don't have sysadmin/cloud/devops expertise (and even if you do, that time could/should be spent on much more important things at this stage).


I scaled to millions of users as a solo founder and still run the whole show myself. AMA if you like.

Here is what works for me:

CI: From the terminal, I run my tests and commit to git.

Deployments: rsync

Rollbacks: Never did one. If something breaks, I fix it and rsync the fix to production.

DB: MariaDB

k8s: I don't use it. Computers are very fast these days. A cheap single VPS will get you a long way.

Nightmare: Not really. I spend about 30 minutes per week on DevOps.


Agreed on rollbacks. I think rollbacks are a thing when you don't know the code, or you're too busy to be able to debug. Like when being on-call for a large codebase.

If you're solo, you probably know enough that fix-forward is the way to go. (This assumes the system isn't critical enough that it needs fixing even if you're asleep, but then you would probably not be solo anyway.)


Not sure if it's a philosophical question but as a solopreneur I have always rolled back since it keeps my commit chain cleaner rather than have previous versions that were buggy. I want to minimize downtime and a rollback is always faster than a fix-forward even when you know, or think you know, the fix.


I do the same, I feel like a simple VPS is a lot easier to manage and less prone to failures than a "cloud" provider. Normally the push-back is "what if it fails, what if you need to scale, etc.", but so far no VPS failed on me (even if it did, you can just use backups and restore with a bit of downtime) and the scaling part can be solved through a good horizontal-scaling server architecture, instead of relying on the cloud to automatically do that for you (which anyway still requires your architecture to support horizontal scaling).


Best comment so far. Thank you for being specific and to the point.


Would you like to share the vps configuration and traffic you handle per month without diving deep into your venture?


What are the most important things you learned and what would be your core advice for someone who wants to run a solo operation? How do you schedule your time to work on your project and how has it changed over time? Thanks in advance!


I would say the most important thing is a combination of two things:

1: Work on something that you yourself or your users want to exist. Don't think about what others might want. Make something that you consider pretty cool shit. Or something your users are asking for. Something where you or your users say "OMG when this is ready I will fricking love using it! I can't wait!".

2: Most founders, me included, often get stifled. They stop putting time into their project because they think it is just a dream. Remember that all amazing things have been built by people just like you. And to keep putting time into building something that you yourself or your users want.


Thank you so much !!! And congratulations on all you have accomplished!


What is the domain of your venture? (assuming it's not something you want to advertise specifically)


I like to keep it off my HN profile. I feel I am more open to share details this way.


Do you mind putting in some contact details (even temporarily) on your profile? I'd like to have a chat with you some time, I was looking through this whole thread for a comment like yours and I resonated a lot with you. (my details are in my profile)

Not immediately, but in the following weeks.

Thanks again for sharing.


Which tech stack are you using?

PD: I follow a similar approach. I am glad you said it.


How about backups? I struggle with it a bit for our home office.


How do you host your DB? Do you use managed DB service?


I run it on my VPS along with the rest of my project.

If I want to set it up on a new server, I start a fresh Debian machine, do "apt install mariadb-server" and I am good to go.

I don't like managed services (like Heroku) because:

1: I cannot easily replicate the setup locally.

2: It creates vendor lock-in which will cause me trouble when the vendors service changes in a way I don't like.

3: I would not be able to put the whole setup of my project in one setup script. It is so nice to just have a single script that contains all the steps that need to be done (Like the "apt install mariadb-server") to run the project on a fresh machine.


If someone wants to know how to host a database for the first time, I have a full chapter on PostgreSQL (and one on Redis) in my new book https://deploymentfromscratch.com/. There is also a demo that sets up an independent database server (which you can treat as your own managed db).

It's easier than you think if you stick to your distributions' packages. Just need to know how to 1, do auth right 2, enable SSL 3, run with SELinux 4, set a different location (for attached storage) 5, managed backups.


Any chance you have an HN discount code? :-)


I want to do Show HN, but probably have to prepare a bigger preview for that. Grab the "ruby30" code from the Reddit announcement: https://www.reddit.com/r/ruby/comments/pg3c9c/i_posted_on_th...


Purchased. Thanks!


Pieter Levels (the guy who made Nomad List) reached $1 000 000 ARR with a single VPS, all his code in an index.php file and using FTP to transfer the latest code to his server...

I think if you’re a solo founder you have to accept on day one you’re at a time disadvantage when it comes to the actual hours you’ll spend on moving the business forward vs admin, ops, legal stuff etc etc and so you have to really fight smart!

I.e K8S? Definitely don’t need it. Complex CI/Deployment workflow? Nope, just the minimal shell scripts to do clean deploys and rollbacks. Pay for a HA RDS so you never need to worry about DB replication etc.

If you just work with the minimal set of tools you’re comfortable with and relentlessly keep things simple it’s possible. I’ve built high availability stuff as a solo founder and still found (at least some...) time to work on sales and the business.


I think you are right. There's a big difference between the technology I would use in a larger organization compared to solo projects.

Unless you build a side project to learn a new technology, you should use something "boring" that you know well.

Here's my personal choices in preferred order:

1. Static web sites - Just upload to S3 behind a CDN and forget about it

2. Server rendered website - PHP. It may not be something you brag about, but in my experience it just "works" and has excellent uptime.

Note: If I had to recommend something to a larger organization, I would probably suggest Next.js to attract developers. PHP is often associated with "old school" tech

3. REST service - Spring Boot. I have so much experience with it and there are so many Spring projects that supports a huge range of technologies from monitoring to data

4. Scripts - Python


This!

Exactly my point that I was making here in this thread: https://news.ycombinator.com/item?id=28840902


What is a HA RDS?


High-availability relational data store.


High Availability Relational Data Store

An example would be AWS' RDS


Because everyone here is hell bent on "spooky arcane oldhat sysadmin deploy techniques", I'll share my setup using k8s.

- CI on github actions

- Project management on post-its (short) and READMEs (long term)

- deployment is done by a github action trigger on the `main` branch

- Hosting is using AKS, GKE, or on-prem k3s on a raspberry pi. Want to restart a service ? just kill the pods.

- Devops took about 2 days of work initially and now shared by every project for less than 30 minutes by project.

- Deloying a test cluster (or a test k3s node) is reproducible on whatever hardware / cloud provider you get, sometimes I often create a full-blown dummy cluster for running test scenario

- Certificate renewal is automatic (cert-manager)

- DNS record creation to services/ingress is automatic (ExternalDNS)

- Authentication using OAuth/OIDC is also set up automatically on whatever identity provider you have

- Database backup is automatic

- Load Balancing is built-in

- Job scheduling is built-in (and -fuck- logs are much more accessible on a failed job container than in a custom cron + sh solution)

- Service discovery is not needed

- Monitoring and alerting is not, but cloud providers often have something for you

Note: this is highly effective because I already had significant K8s experience, so, if you're still learning about what Ingress-Controller to choose for your OIDC proxy, then don't go that route.


Hey, this is pretty cool (I am doing the oldhat stuff, so interesting to hear about something different)

>> Job scheduling is built-in (and -fuck- logs are much more accessible on a failed job container than in a custom cron + sh solution)

Can you expand on this? How do you handle logs?


I don't handle anything, I just output to stdout and I can access any directly from its container. Job or Pod, it's the same. If I had a larger log volume, I would have some kind of log aggregator to send them to ES or cloud provider monitoring stack


> Note: this is highly effective because I already had significant K8s experience, so, if you're still learning about what Ingress-Controller to choose for your OIDC proxy, then don't go that route.

What would you recommend instead ?


Learn about kubernetes for 6 to 9 months, full-time, and come back to it.

You have a lot of freelance work and devops positions that will give you that experience. You shouldn't try to do this alone, you need guidance to understand how this ecosystem works.


How did you set up the DNS part?


Via annotations on my ingresses, and, of course, with this : https://github.com/kubernetes-sigs/external-dns


Use Linux, pretend you’re in the 90s/2000s. If you can’t afford a team, you’re too small to need more.

- Use NFS (EFS), put builds in a /releases folder

- Bash scripts for build, deploy, release

- Bash scripts to wrap apps (write pidfile, start, stop, etc)

- Cron those scripts

- Check crontabs into a repo along with all other config

- Cron a bash script to pull from that repo and update crontabs on every box

- Have a staging environment, deploy stuff there (esp DB migrations) and do smoke tests.

- Have some kind of monitoring (I have a slackbot that sends me messages when things break)

I general I follow the principle of proportional response, and n=2 (or 3 depending on how trivial the task is to automate).

Proportional response means you invest in automation / tooling proportional to the pain you have suffered. Burning a day to build a bulletproof script only makes sense to solve a big problem. For a smaller issue, maybe adding a debug line or writing a line somewhere that you can copy-paste later is good enough.

N=2 means you don’t solve stuff until the second time you need it. This stops you from burning hours building clueless complex garbage to solve unnecessary problems - at least by the time you are automating, you have solved it once or twice by hand and you know where you are going.

Elon’s 5 principles have been very useful to me as a solo dev:

1. Make requirements less stupid

2. Delete the part/requirement

3. Simplify/Improve the design

4. Accelerate cycle time

5. Automate

So in particular you might try 1-4 to see if you can magic problems away before you invest in building devops automation you will then need to maintain.


> CI

GitHub Actions

> deployments/rollbacks

Docker. Scaleway offers a container registry that's ridiculously cheap[1]. Deployments are infrequent and executed manually.

> DBs

Again, Scaleway's managed RDS[2].

Outside these, we have setup Grafana + Loki cloud[3] for monitoring and alerting. They have a generous free plan.

For product analytics that can be derived from our database we've a self hosted instance of Metabase[4].

[1]: https://www.scaleway.com/en/container-registry/

[2]: https://www.scaleway.com/en/database/

[3]: https://grafana.com/

[4]: https://www.metabase.com/

P.S. We were a 1 person company when most of this was setup. We're 3 now, works just as well.


Thanks for being specific.


Ex-Googler here and 20+ year veteran of SF startups.

It sounds like you have a great set of intuitions, and you're right, all that infrastructure is a nightmare to set up and manage.

Step 1: Question every requirement

Step 2: Drop everything that is not critical

Step 3: Profit!

Git hooks and Makefiles are great!

I have an HP workstation I bought from eBay under my desk that has 32GB of RAM, 2x 12 core CPUs and a static IP address. It's probably 8 years old, but fast enough to serve whatever we need for a long while.

The machine (Old Blue), hosts our Git repos, web app, database and integration service (git hook that calls, 'make publish').

We're not serving Google scale traffic, so we don't need Google scale infrastructure.

Keep it simple whenever possible and don't let the modern stack complexity creep in until you absolutely need it.

Even going to the cloud, when you've done it 10 times and know how, is way more work than you need when just starting out.

Take on those costs and complexities only when your traffic requires it, and you may just find out that you never have to pay rent for compute.


I hope you’re backing up customer data if you store any…


Definitely.

In fact, it's super easy to back up as it's in a SQLite database (for now).


How do you backup the SQLite?

I used to rsync them, but when traffic picked up, I started to fear an inconsistent copy (rsync does verify at the end there are no new changes, and restarts - but i don’t know the exact strategy so I am wear if relying on it).

I switched to doing .backup and rsync that instead - but am considering switching to litestream instead.


Some if the comments here are way over the top IMO. Like a full time DevOps person went solo and spent a few weeks setting up their perfect deployment process.

If we're talking a plain saas type deal, I'd keep it simple, elastic bean stalk, or use a heroku or render.com like setup until you grow to the point of hiring a team. If it's just a basic saas, I don't see how a 1 man team could really out grow this setup. I've seen 100 person teams using heroku.

K8s is just way too much work. Even cloud formation is to much for my tiny show.

Use the automated backups setup by your host for your db. If you need to roll back, just redeploy your previous git hash. I typically use GitHub actions to deploy, so rolling back is just a matter of force pushing the prod branch to the sha I want to deploy

Skip micro services, they are too much work for a small time thing, and don't really provide much benefit.


I run https://oikolab.com - it's a weather data service with global, hourly historical weather data from 1950 to 16-day forecast. It launched about a year ago and I think it's probably the largest data offering of its kind.

I don't have dev ops experience (I still don't know much) so a lot of what I did was to exhaust the limits of the setup before implementing any new infrastructure, sort of learning as I go along, with many 'oh, so that's why they do this' moments. I've read about K8 but still can't see when I would need that though.

A lot of what saved effort for me was to use off-the-shelf component whenever I can. I was quoted $10~20k to build a Django website for user & API management to connect to my back-end but it was much easier to use Azure API Management service (~$400/month), which also came with simple, reasonable looking web front-end that I was able to launch within a week.

Before moving things to cloud (Digital Ocean), I've also exhausted what I was able to do with simple NAS servers (overall I processed about 200TB of raw data with ~1TB/day on-going using what I call a 'poor man's HPC' setup) - luckily I'm based out of Hong Kong so had access to fiber-optic internet at home.


Nice, I bookmarked your site. Weather data is surprisingly hard to navigate. Did you source it from NOAA.gov or somewhere else?

I tried to slap together a simple weekend app based on historical weather data by zip code, and never even got to the coding. Just spent most the weekend trying to decipher those API's.


The primary forecast data is from NOAA while the historical data is from ECMWF (sort of the European equivalent of NOAA).

I’ve been told that my API is easy to use - feel free to let me know your thoughts if you get a chance.


I do it KISS. Deployment for me is always a git hook, usually with a private gitea server. So I have versioning and rollbacks available with a cute UI if necessary.

DBs run side by side and are simply backed up regularly with whatever backup solution the VPS offers.

That's it.

Out of school I put way more effort in building my infrastructure but in reality a good VPS, something like Cloudflare and some app based caching can run millions of daily views over 20+ different apps without issues.

Edit:// it's mostly rails apps I host btw. For my wordpress installs i use dedicated VPSes because those scale horrible.


> Out of school I put way more effort in building my infrastructure but in reality a good VPS, something like Cloudflare and some app based caching can run millions of daily views over 20+ different apps without issues.

I read such messages (and also agree to it mostly) and then I encounter my peers wanting active-active clusters with K8s on top or something and I feel I like a dinosaur.

Refreshing to see KISS still being used. Also gitea. What an awesome piece of open source software.


Don't fall victim to resume-driven development; keep on KISSing.


I like this answer, but could you expand a bit on why gitea? Never used it myself, but it looks like self-hosted OSS clone of GitHub? If so, my instinct (in terms of KISS) would be to just use GitHub, so I don’t have to set up and maintain my own thing. What made you decide to go with gitea, out of interest?


GitHub would be the opposite of kiss in my opinion. External dependencies and pushing to remote servers are just pain points.

> git init --bare

On your remote server is all you need to spawn a repo to push to, usually hook examples already included.

The hook can then directly checkout to my live directory, run migrations and restart the server without weird remote automations after every (or specified) push.

Gitea is just a little extra (setup once, touch never thingy) for quick edits and checks (it's a wonderful lightweight piece of software too)


My web app is hosted on a server on Hetzner Cloud. I don't use Docker.

For:

* Database: PostgreSQL installed through apt in the same server: https://github.com/sirodoht/mataroa/blob/master/docs/server-...

* Backups: MinIO-upload to an S3-compatible object storage: https://github.com/sirodoht/mataroa/blob/master/backup-datab...

* CI: Github Actions + sr.ht builds: https://github.com/sirodoht/mataroa/blob/master/.github/work... + https://github.com/sirodoht/mataroa/blob/master/.build.yml

* CD: (not exactly CD but...) ssh + git pull + uWSGI reload: https://github.com/sirodoht/mataroa/blob/master/deploy.sh

* Rollbacks: git revert HEAD + ./deploy.sh

* Architecture pattern: stick to the monolith; avoid to deploy another service at all costs; eg. we need to send multiple emails? not celery, that would mean hosting a redis/rabbitmq. We already have a database so let's use that. We can also use Django management commands and cron: https://github.com/sirodoht/mataroa/blob/5bb46e05524d99c346c... + https://github.com/sirodoht/mataroa/blob/master/main/managem...


I have been working on https://quantale.io since a year and half. It's a web based alternative to Bloomberg Terminal.

I personally manage the CI/CD infra as well as the data infrastructure for my startup.

Here is my setup:

1. Self-hosted Drone.io instance connected to github(the instance is behind a firewall and can only be accessed via a self-hosted VPN. Github IPs are whitelisted so that auto build/deployment can be initiated via github hooks)

I have drone.io config that helps me manange the versioning, deployment to stage/production, run tests, do rollbacks etc.

for eg: merging to master on github auto builds the master branch on drone and deploys to staging along with pulling the latest db backup from S3 and adding to stage for testing

2. Self-hosted production and staging server that hosts my web/django app. It has a local postgres db and is encrypted and backed up on S3 every hour.

3. Self-hosted Elasticsearch cluster using Ansible playbooks with daily snapshot backups to S3.

beyond this, I self-host 3 VPNs to protect my entire infrastructure. read my old HN comment about it: https://news.ycombinator.com/item?id=28671753

All this is one time infrastructure setup and it all has been running smoothly for more than a year without any hiccups.

I would be happy to help you setup a similar setup if you want. hit me up at vikash@quantale.io


> 2. Self-hosted production and staging server that hosts my web/django app. It has a local postgres db and is encrypted and backed up on S3 every hour.

Did you have any instance where you needed to restore the DB / Have you done restoration tests to ensure this is a feasible solution? Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ?

Also, how many days of backup you keep on S3? Considering you create 24 backups a day.


> Did you have any instance where you needed to restore the DB / Have you done restoration tests to ensure this is a feasible solution?

Whenever a build is deployed to stage server, it automatically pulls the latest version backup in last 1 hour and restores it on the stage for testing with the latest data. I have config to do the same restore to the prod if needed.

I have been using the same setup for over 2 years to support an internal web app of a company.

> Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ?

This problem only arises when there is heavy load on the database, specifically high WRITE/UPDATE/DELETE operations. Quantale uses postgres to store only user info and all the other data is stored in Elasticsearch(twitter, reddit, news, SEC, Financial data on over 15000+ stocks). This is intentional because with the scalable server resources, my postgres setup can handle a large number of users before I see that problem. I can always scale up later when needed rather than over-engineering it right now.

>Also, how many days of backup you keep on S3? Considering you create 24 backups a day.

I have 7 days backups on S3 with versioning switched on for 60 days. So at anytime I can access the version backed up in last 60 days.


scottydelta, Thank you for taking the time to give me more context / insight.

Safe to say you have a verified way of backup and restore (with worst case data loss of an 1 hour).

I am not exactly an expert on what stage does Transaction ID Wraparound comes in but still know that as one of the key criticisms of PostgreSQL so wanted to check.

Also, 7 days backup with versioning switched is a smart tactic! Thanks once again for this.


If the hours you spend implementing and maintaining your DevOps exceed the hours of downtime you prevent, you're probably not making good use of your time. The less you build, the less you have to maintain.

I've been running a Python-based, highly custom web store solo since 2007, and it supports multiple people doing fulfillment. I host on Opalstack so as to outsourcing patching, email config, database maintainence, etc. I run directly in a Git repo (actually Hg, it's that old) and releases are "git pull && ./restart.sh". Rollbacks are "git checkout ...".

I've had to migrate/rebuild the VM about every 5 years. Tech changes enough in that time that no automation will still work unmodified. So I just keep good notes about what I did last time, and figure out what the new equivalents are when I finally have to do it again (updating the notes, of course). Database and Conda are easy to port. It's usually DNS and email integrations that are a pain.

As others have said, KISS is key. Industry DevOps is for a work setting with a decent-sized team, where you can afford the overhead of maintaining it all in order to make the overall team more efficient.


Simple. Don’t use K8S. It’s absolutely 100% the wrong way to go for solo devs with no product market fit.

Use Heroku (or any other PaaS) and throw in some serverless stuff here and there.

Again DO NOT geek out on the tech. Geek out on getting customers.


I wrote a series of Medium stories about my experience with tech stacks for solo work - including deployment (start here: https://medium.com/geekculture/surely-building-software-cant...). My needs are slightly different to OP, because I'm not assuming everything will have to scale, so I'm optimising for "scale to zero" (only pay for what I use) and low maintenance... but I'm taking approaches that I know will scale if they have to (scale technically, not scale to have a team working on it).

In short, I lean heavily on serverless and Github Actions.

Each story covers a different pattern in my stack and I apply these principles:

-I only want to pay for what I use -I don’t have a lot of time available for learning or building -I don’t have time for maintenance activities -I’m not a good UI designer or front end engineer


I've been a solo Web/Internet entrepreneur for most of 20+ years, non-stop.

The DevOps side of things is trivial in the beginning and takes very little time compared to actual development.

Ubuntu server, Nginx, Mysql or Postgres, PHP, Go, Redis. Configured reasonably, it's a ridiculously reliable stack where things very rarely go wrong.

I prefer DigitalOcean these days. Takes <15 minutes to configure a base new setup - a template - from scratch, make a few adjustments and double check everything. From there I can pop up a lot of servers as needed. I usually tweak things based on the project, although that doesn't take a ton of time.

And that's it. Back-ups on eg a database focused droplet are automated by DigitalOcean. Occasionally I have specialized back-up needs, and I'll do a bit of custom work for that. Most of this could be offloaded by using DigitalOcean's database service, I just prefer to limit cost by not.

Under no circumstances would I use Kubernetes for a smaller to medium size service.


I have currently 2 projects with some traffic (5k req/day) running in production.

Both have a similar setup:

   - 1 droplet with docker pre-installed from DigitalOcean
   - clone directly the repo from github
   - together with the code, I have a folder with a bunch of docker images (caddy, mysql, php, redis, etc.) that I can easily spin up.
   - for any release, I manually ssh, git pull and run the migrations (if any) and manually rebuild any docker image if needed
   - I have daily jobs that dumps and zip the entire DB to S3
   - if I have some deployment that I know will break any functionality during deployment, I warn the users before and accept that downtime.
   - never had to handle a "hard" rollback till now.
I've planned to change this setup for while, but until now, didn't find any reason that justifies the effort.

I spend 20$ (10$ per droplet) + few cents on S3 per month with them.


Solo founder here. (Typed on my phone)

I’ve been running my small company for 4 years so far. It’s a podcast search engine & api.

Product & infra & Devop & all kinds of manual processes all evolve over the past 4 years. Things were added / improved on demand.

4 years ago: 3 digital ocean instances + ssh to manually deploy code. No users. So no devops.

Now:

- ~20 EC2 instances, provisioned via ansible

- to deploy code: run a bash script on my MacBook, which is actually to run ansible. The script can also rollback to any specific git sha

- use rollbar + Datadog + PagerDuty for monitoring & alerting

- 3 Postgres database EC2 instances (1 master, 2 slaves). Daily cron job to automatically upload db dump to aws s3 and google cloud storage

- no CI. No docker / kubernatte

- a few interesting devop events are sent to slack, so I’m aware of what’s going on for the entire system on my phone. Eg, when a db dump was successfully uploaded to s3, when a Django request is slower than 5 seconds, when an important cron job fails…

Not very sophisticated, right? But in practice, we rarely have outages and spend very little time to do devops.

The most annoying thing was to upgrade Postgres to a new version. We had to prepare for an entire week and minimize downtime (in practice, disable db write for < 1 minute or so).

I’ve got a couple old blog posts on engineering for solo founder:

- https://www.listennotes.com/blog/the-boring-technology-behin...

- https://www.listennotes.com/blog/good-enough-engineering-to-...


Is Datadog worth it? I thought it was too expensive, although they do have a lot of features.


Keep it stupid simple.

Use tech that you understand and minimize the layers of complexity between deploying, starting and troubleshooting an instance.

For me, I like digital ocean droplet(s). Stupid simple IAM. SSH scripts for CI deployments. Everything sits behind a Cloudflare cache.

The backend sends key events like restarts and fatal errors into a dedicated Slack channel. Its my confirmation a deployment worked and first alert if something crashed.

My business partner and I are currently running a $1m/arr business with this setup on a single digital ocean instance, Python3 and Nuxtjs.


100% this! Just curious--what's your business?


We're in the hospitality sector.


Not a solo-preneur, but I’ve had great experiences so far with Dokku, an open-source Heroku clone. I’m running ~40 apps on my single VPS since a couple of years. I use it in combination with Github Actions, Bitbucket Pipelines and Gitlab CI. I still feel very flexible working with Dockerfiles and different cloud services. I’ve only had to update the dokku-letsencrypt plugin a couple of times in the past 5 years. Note that I haven’t had to protect important databases with PII for example.


This seems to (strangely) be an unpopular answer here, but I've spent a weekend to terraform everything. I run a couple of different projects which are yet to pick-up traction, so prototyping quickly is essential.

What I have now is terraform on Github which handles Networking (VPC, subnets), ECS/ECR, ALB for my backend and S3/CloudFront for frontend.

Everything is on AWS because I'm familiar with it, so it's faster. And I compensate for costs with AWS Activate + Free Tier. This combination is usually enough to understand whether the project will get traction or not.

Actually spending little time on IaC was one of the best recent investments. But I guess it holds more value when you have a couple of projects, still exploring and these tasks get repetitive


I spent a lot of time on Terraform + Packer code to set up VPC, RDS, Route53, AWS Cert Manager, ELB, EC2, my API server, S3, CloudFront, and CloudWatch. The setup had a lot of complexity and footguns. Plus there was the risk of accidentally running up a huge AWS bill.

Later, I realized that Heroku + Netlify are good enough. Switching was easy. The hardest part was admitting to myself that I had wasted so much time on AWS. I felt relieved after I deleted the Terraform and Packer code.


AWS bill is a concern. I put alerts on billing but they do come with a delay.

I was thinking Digital Ocean. Or Heroku for the current project, but ended up with a design with quite a few serverless pieces using lambda. Which has a great free tier and is very useful for my use-case.

Would not say that AWS is too much complexity though.. When you go over things at least once and understand how thing fall together


I find this helpful for my side project as well. It's amazing how quickly a) things break down and b) I forget how to set things up and get running again if I have to shelve and restart work on a project.


So I am a solo-preneur and have launched lots of web-apps in my career. And the DevOps hasn't really changed for me is incredibly simple: commit source control and copy the files to the server.

I've done this since writing my first CGI app in 2003. And it hasn't changed other than the commands I call. Back then it was manual through GUIs (visual source safe and FTP) and now it is git and scp, but 80% of the process hasn't changed in almost 2 decades.

If it breaks (which it shouldn't, but inevitably does), I just git checkout the previous commit and scp again, and try and fix the issue.

If I have a "high availability" product (which I hate doing), I will release onto a clone of the production server, then test it, swap to that clone, deploy to the others, swap back and kill the clone. It is a much more involved process that I really don't like doing, so I've stopped making high availability applications.

But that is web, so it's super simple. Also to avoid issues regarding data schemas, I always try to make sure my changes are additive. And on startup, my web server checks the database version then runs any upgrade scripts if needed. If the deploy fails, the db changes stay, and the code just gets updated. This is sloppy, but it's simple.

When a real schema change needs to happen, I abandon the project (just kidding). I usually do it in multiple updates. New table with the schema changes, upgrade script transfers the data, and code to point to new table. Manually verify that all of the data copied. Backup the old table and drop it. Then another update to rename the new table to the old name (if needed) and it's done with minimal disruption in service.


I pay money so I can sleep easier at night!

Envoyer and Forge for PHP stuff.

Render/Heroku for node.js stuff. Netlify for static stuff.

For the more complex bits I have, I use Pulumi to manage everything through IaC. It's still complex but at least it's robust and I can sense-chech changes to infra through a PR to myself.


I have worked mainly on ops/infra professionally the past few years and will strongly with the idea to avoid any complicated tool - i.e Kubernetes nowadays.

I create my small/bootstrap projects the following way:

1- create a free tier AWS account - can work with any VPS/server really, but with AWS you can get it 100% free :)

2- create some Ansible provision/setup/deploy scripts

3- create some bash scripts to wrap this all

Create the Ansible scripts:

1- Provision script, consisting of 1 EC2 instance, 1 RDS and the security groups. Store the IPs/adresses of the newly created instances.

2- Setup script, basic instance config, packages, app/project repo, you name it

3- Deploy script, to run every time I want to update the app, mainly just doing a git pull on the instance and some database backup/migrations (although on RDS backups aren't always necessary)

I get these Ansible scripts wrapped into basic bash scripts, provide my AWS credentials via the AWS cli to keep it safe, few extra creds with Ansible vaults, SSH keys, publish in a Github [private] repo and I'm all set.

It took me a couple of days to get fully operational and I was learning Ansible at the same time so it can really be done with basic features. Now it's done, I can reuse the same skeleton for every new project in a couple of hours!

I find this solution extremely resilient. That's basically free hosting for personal projects. Every year, just need to run the script again when free-tier expire, and can change hosting provider anytime or upgrade the instance type if I decide to get a project publicly released. A small extra Ansible task to write for migrating the data and that's it!


I'm a noob when it comes to devops, so you might not want to do it my way.

I keep it super simple. I have a git repo where I directly commit into master. When I want to take the changes live I ssh onto my VPS, I git pull the changes and restart the website service.

I use an SQLite DB that I automatically copy (backup) to my local machine daily via the Windows equivalent of a cron job. Once it's on my local machine it's also backed up to other places as part of my local backup process.

I run my tests manually when I think I've made significant changes.

Otherwise I just react to errors ASAP (I get an email notification from sentry when a new error occurs)

A lot of this doesnt work with a bigger team of course.

It's a balance between risk and time investment. The simpler your dev ops stuff, the higher your risk of introducing bugs into production. But it saves you a lot of time not dealing with that stuff.


I had a chance to setup several businesses in the past and to setup the foundation - I ended up writing a lot of templates for myself. Nowadays, if I want to setup a SaaS company, I have 90% of the foundational groundwork covered (AWS, Terraform, CI/CD, Github, etc.)

We're building [1] something similar to solve this problem but just on the infrastructure end. I really believe in the power of templates. In our future release, we're planning on launching a "infrastructure" catalog of commonly used setups so that you can just get the foundation in place. Why re-invent the wheel every time when you can just spin up a boilerplate that gets you 90% of the way there?

[1] https://atomizedhq.com


I'm starting to consider this myself and have been looking at https://serverless-stack.com/#guide as a way to prototype and build an MVP. The guide has quite a bit in it that I believe can be repurposed to that end although it doesn't cover backups etc, more integrations with AWS services.


Yup it's meant to be a starting point for projects of all sizes. We built our own stuff using it as well. Feel free to reach out if you have any questions. Or join us on Slack: https://launchpass.com/serverless-stack

Good luck!


I lead a team of 3. We use render.com and have a handful of bare metal servers for CPU-intensive background jobs. The bare metal servers are auto-patched Ubuntu servers. It’s pretty much 0 maintenance. The server setup is automated via a single provisioning bash script. Render.com is quite nice, if a bit limited in its options vs Heroku. Highly recommended.


That's easy, Render.com (it's like if Heroku were designed in 2021, and a lot cheaper). And can handle docker and distributed Elixir out of the box as well as free static sites). Took me just a few hours at most to set up and has been very reliable (I think it's k8s on Google Cloud under the hood but it's all abstracted away).


Like most others said, if its not a painpoint now, just deploy manually and keep it simple. You will save time and money and more importantly, you can concentrate on what is important for early-stage companies: Nailing product-market fit so you can get the sales moving in.

It isn't that hard, for someone with decent experience, to setup TeamCity/Jenkins/Octopus/Whatever but why spend the time? If you change something, you then have to go and spend time changing the automatic deployment.

As someone else said, Devops is about velocity with quality, if velocity is OK and quality won't improve with tooling, don't bother.


My entire stack is in a docker-compose.yml that looks like this:

services:

  certbot 

  haproxy

  db

  api

  client
And gets deployed by a GitLab config file that looks like this:

build:

    stage: build

    script:

        - docker-compose -f docker-compose.yml -f docker-compose.release.yml up -d --remove-orphans --build


Scaled multiple self-hosted products up to tens of millions of hits / month. Scaled several for clients up as far as the hundreds of billions of hits per month. (Yes, hundreds of billions.) Never seen a kubernetes deployment that was (A) necessary or (B) simpler than some alternative.

Unless you're doing something very slow / processor-intensive, you'll probably never even need an autoscaler before you can afford to hire an expert to do it for you. You'd be surprised how far you can get with a small VM running your whole stack.


I only have side-projects, very different from a full-time, revenue earning solo-preneur, but I only use serverless. My preferred stack now is NextJS, hosted on Vercel with FaunaDB. The main reason is to “outsource” devops to them.

I am not sure how it would work on a larger scale, but for my use is perfect.


How much money do you spend on this?


Right now, zero, as I am using only free tiers. But my understanding is that it is expensive when you scale.

Vercel first paid plan is $20/month and FaunaDB $25/month, so my guess is a business with some traction would have to pay at least these $45/month.


My strategy: get really clear on business needs/objectives/internal SLAs for infrastructure. In my experience a lot of time-consuming stuff revolves around uptime, data recovery (RTO, RPO), security, compliance, and scaling. Depending on your business and the stage you’re in, a lot of that may not matter, so don’t build anything you don’t need now.

I’m a solo-founder, with many years working in IT, and I focused on DevOps for part of it; I know what best practice looks like and it would be easy to fall down that rabbit hole doing an unnecessarily complex buildout. Currently I’m doing a small private beta, I’m avoiding 99% of standard practice. I’ve got a single EC2 node running Redis as the only datastore and NodeJS + Nginx + certbot, a cron to do backups to S3. No CI (there are no other devs to integrate code with) I run all tests locally and push/rollback with rsync. All code, assets, and server config (except creds) are in a monorepo.

If the server goes offline, I will have a little downtime, that’s fine. If I run out of memory for Redis (not likely to happen soon), I’ll change to a different datastore or scale up the node. If I lose data, I can restore from S3, and additionally the architecture is such that clients will re-push their latest changes.

Do the bare minimum to support the business, stick with what you know, outsource what you can, and properly value your time.


For my API SaaS I run everything non-critical (marketing website, analytics, customer dashboard) on a single server, either part of a single django monolith or using an official docker image with some command-line arguments. Only the API gets duplicated and load-balanced with Cloudflare.

Dedicated servers, specced to provide lots of capacity for spikes. A 20-node k8s cluster could fit on 2 beefy dedicated servers for about the same cost. Decreased infrastructure redundancy but massively increased operational stability through simplicity.

Everything runs in a docker-compose project: one for the API, another for the everything else monolith. I've worked for a couple of small companies that ran with docker-compose, so have a good sense of the weaknesses and footguns (breaking your firewall, log rotation, handling secrets, etc).

CI is running `make test` on my dev machine. Deployment is `git pull && docker-compose up --build`. Everything sits behind haproxy or nginx which is set to hold and retry requests while the backend is down, so there aren't any failed requests in the few seconds a deployment takes, just increased latency. I only deploy to the API once or so per week, that stability reduces headaches.

DB backups are done with cron: every hour a pgdump is encrypted then uploaded to backblaze. Customer subscription data is mirrored from stripe anyway so an out of date DB backup isn't the end of the world. Error if the backup fails after a few retries.

Sentry on everything for error alerting. All logs go into New Relic.


Don't:

- Manage your own DBs for production. I used to manage my own but now I use IaaS for it. It's worth the extra cost even if your budget is low since it lowers your risk and lets you move faster. AWS Aurora is awesome but expensive. There's a SaaS/IaaS out there for most DBs. And they handle the backups!

- Use Kubernetes. k8s is awesome, I love it, but it is a beast and probably overkill for your solo ops team that is also multitasking.

- Manually SSH / rsync / scp directly onto prod.

Do:

- Version control (Git) everything

- Use containers where it makes sense (I use Docker when I'm not using Serverless). It keeps your dev and prod in sync and helps with immutable deployments (see later points)

- Use Infrastructure as Code

- Use Github actions to do your deploys

- Use immutable deployments. More time to setup but makes rollbacks easier and if you never hot patch production you can't create a unique state that can't be restored.

Finally, (and possibly controversially): use serverless where you can (I personally use AWS Lambda, S3, and Cloudfront).

My personal flow:

- All my infrastructure is in a parameterized CloudFormation template.

- The template is checked into Git with the service code.

- After CI runs the tests, CD builds my serverless functions and uploads the zip.

- Then it runs the cloudformation template.

The same flow also works for Docker except "uploads the zip" becomes "pushes the container"

Using this setup I can tear down and build up my setup in minutes, making it easy (and cheaper) to separate dev and production.


> Don't: manage your own DBs for production. [...]

Why, though?

If you're really on a low budget, the extra $$ for a managed DB doesn't really pay for itself. I'd rather prefer spinning up a container running whatever DB I need and run daily backups to an external vendor.

In my case, the website is free, living on ads revenue, so every penny counts.


Good question and I have an answer for you. It depends on your risk tolerance and resiliency.

If you compare it to a single machine with multiple tenants and no replication, it doesn't make sense. But that comes with risks and given the OPs question in the context of a k8s cluster (which for HA is a minimum of 5 servers), I figured they are prioritizing time savings over money.

Losing customer data, downtime, and data breaches can kill some companies before they even take off.

If your particular business is not sensitive to that than you can run your own database. Or better yet, if you're a content business you may not even need a database at all.

But even so, if you run your own DB you need to worry about backups and such so it may even still be worth the money. And if you are going for High Availability (HA), running a properly replicated cluster can be time consuming.

The cost of a t3.medium RDS instance on AWS for instance is $0.068 / hour compared to a t3.medium instance on EC2 is $0.0416 per hour. That is an extra $19 per month. For my use case, paying an extra $19 a month is well worth the savings in time and the piece of mind I have knowing my backups are running and if the server crashes I can recover quickly. But you must consider use cases and your's might be different.


You bring up good points, fair enough.

I might being naive here, still:

   - losing customer data: the backup must be done frequently enough. So I'm not risking losing data.
   - downtime: I'd expect that RDS could still go south as well, maybe less frequently, but still. In my case I start another VM, run again the container and apply the backup.
   - data breaches: Of course one can misconfigure something here, but not sure how it'd be different using a managed database.
Regarding the cost, I know that there are companies spending hundreds of $$ on their infra, but it just baffles me for most use cases. I still feel that one can get really far with just a budget of 50$/month.


True. You have downtime risk with RDS and managed databases as well. I have taken down RDS instances with a bad query and indexes once or twice.

$50 total would be tough with fully managed AND high availability. I agree. But to digress from the tech side a bit and talk my own experiences, the question I ask myself is: as a solo-prenuer can I use that time to be doing something that will make me more than $X per month? If the answer is "yes, immediately!" I go with the managed server. If the answer is, "it would take me years to break even"... I may roll my own.

To address your points, thought: I think managed databases do help here. Data breaches and outages are usually configuration issues. Since they start with a good known configuration, managed services do help. Also, I see not being able to SSH in as a positive in this scenario.

For example: Say you misconfigured a database an break backups (happens all the time even to good DBs). That is much harder to do on a managed database.

If you're managing just one machine and being down for 10-30 minutes while your backup is being restored is fine, you probably don't gain much by using a managed service. In that case I'd take frequent backups, upload them immediately off the server (probably S3), and containerize the database. In my architectures I always run at least 3 servers.

With most database engines, if you have three servers you need to lose 2 to cause an outage. But a 3 server cluster is much harder to manage manually, tilting the scales more towards a managed database.

Also, to your point, would blow $50 out of the water.

If I can, I use a managed serverless database like DynamoDB (there is also others) that are pay as you go. You can easily run a service on just DynamoDB free tier if your use case is well suited to that kind of key/value stored based DB.

You can also, always start with your own and switch to managed later as you add machines to the cluster.

Just, whichever you choose... but "especially" when rolling your own... make sure you run fire drills. I.e test your backup and restore process regularly.


> Using this setup I can tear down and build up my setup in minutes [...]

Until you decide to move away from AWS. ;-)

Nevermind, ignoring the potential lock-in there are many pros to this setup.


I've been using 10+ years and have definitely considered that a lot over the years. It's a valid point. I used to be concerned with that, but now I think that is not a big issue for two reasons:

1. Vendor lockin when you are a startup should be a low priority in my opinion. Your priority should be moving fast and getting customers.

2. To expand on 1. It's not really vendor lockin, the only things vendor specific are your infrastructure templates and interface code. Rewriting the interface for another cloud provider takes a fraction of the time of rewriting your codebase (source: I've done it many times).

To come full circle to the question, one thing you should probably not do as a solo-prenuer is concern yourself pre-maturely with vendor lockin at the expense of getting a working product.

Edit/Example: The code for interfacing with AWS Aurora is the same as any other MySQL or PostgreSQL code except for authentication. Switching to a non-IaaS database WILL take time but no more time than rolling your own. So your choices are not "lock in" or "freedom".., it's, spend lots of time setting up databases now or later when you have proven your business model (and potentially never)

Edit 2: This is an anecdote certainly but... of the dozens of startups I've worked on and with, some of which have grown pretty large. The only vendor lock-ins I've ever found to be issues have been with software frameworks (especially ERPs). Never infrastructure. I used to be worried about infrastructure lockin, now I prioritize product market fit.


I've been using Beanstalk for my new project, which is like an in-between for full on AWS and Heroku, but if you need you can decompose it into individual AWS components. It's more difficult to get started and there are way less sensible defaults, but I still got everything deployed for my POC in less than 4 hours.

As for DBs / engines and such, I try to go for hosted services as long as they're not stupidly priced. I'm currently using Elastic cloud. If I need a relational DB I'll probably deploy Postgres on AWS and install some scripts to back up the data to an S3 bucket every day or so.

I chose NOT to use k8s or Docker because, like you said, it's basically a fulltime job to maintain. I don't think it's needed at all unless you're a large organization.

So basically: a mixture of roll-your-own, streamlining platforms, and hosted services. Do whatever works and is easiest, and don't worry so much about having ALL the goodies.


I run my own Heroku clone which is then licensed by my other companies.

Also, I review how much time each product / feature consumed each year and then decide what to deprecate. Each April I then have my own little party where I ceremoniously turn off last year's time wasters.

I'd say it all boils down to "know your customer lifetime value"


I use Dokku (https://dokku.com/). It works with GitHub Actions and it's enough for most use cases.


I'm not a solo-preneurs, but I have some open-source projects (mainly a live one, and one that I'm currently developping).

Here are the list of DevOpss flow I've done:

* One project is a website. I have a pipeline, triggered when something is merged to master, that builds Docker images, pushes them to a Dokku instance and deploy them.

* One project is a JS library. I have a pipeline, triggered when a git tag is created, that packages the library and uploads it to npm. I use Netlify to host the documentation website of it.

* For this same project, I'm planning to have a pipeline triggered for each PR to run the linter and tests.

I work as a DevOps/MLOps, and I'm quite comfortable with CI tools in general. I hate doing things manually and I love automating, so it comes quite naturally for me to do it. Once it's set up, it rarely needs to be changed.


I do all like this:

GiHub actions-> docker build->docker push to github package repo

then on vm a 2 line script to docker pull && docker-compose up -d

Database all by hand.

Extremely easy i just push, wait build and restart


I recommend not doing any devops at all if you're a solo founder and trying to get it off the ground. You want to spend time on your product market fit, not fuck around with linux.

New services like render.com make it really easy to get robust, cheap hosting up. Much cheaper than Heroku.


"It depends"...

For simple monoliths it's a non-CI method plus scp, for infrastructure-only it's SaltStack and Terraform. For something where the DevOps part is the deliverable, it can go wild pretty fast: GitLab CI with various CI steps (static analysis, building, unit tests, integration tests, browser tests, blob storage of the result), then CD steps depend on what it is that was built; docker images just get pushed to a registry and another async job listens for pushes and does the deploy in Fargate or Kubernetes. For some jobs with larger needs you get into ArgoCD and Kubernetes resource deployments that way.

Essentially it depends on the zoom level:

Zoomed in to the max: just a build.sh or makefile, optionally called by a CI or IDE of your choice so the building is exactly the same no matter what invoked it.

Zoomed out a little: the results of the build need to end up somewhere, sometimes a step in between like a package registry, docker image registry etc.

Zoomed out to level 3: once the deployment gets actually done, the runtime itself is responsible for checking for database migrations, locks, maintenance mode, so that has to be part of the startup. If it's a multi-instance workload it depends on database migrations if there has to be downtime.

Zoomed out to level 4: if it also needs to deploy 'other things', like rules and records to Cloudflare, resources in AWS, resources in K8S and even update some stuff in buckets or DynamoDB, that requires some orchestration and DSL like Terraform, SaltStack, Ansible and application-specific CD like ArgoCD or Flux or even just the aws cli tools to trigger a refresh.

Customer decides how much money they want to spend and that scopes what zoom level they get. Usually depends on how many developers or ops people they have themselves and how involved they want to be and what continuity or guarantees they are looking for. The only thing that is not optional is that everything has to be in Git. If a customer doesn't want that, they cannot be my customer. I'm done living in the 90's.


As a solo founder, your time is the most precious resource so all your DevOps effort should be minimized to the extent that you can prevent and debug issues. In practice, for me, this meant using tools that I'm already comfortable with (preferably in the same language as my codebase to minimize context switching) and resisting making things overly complicated with too many tools in the chain. For automated deployment, I'm using Fabric (for Python) that allows me to remotely run commands. DB backups are handled by cloud snapshots and rollbacks are ad-hoc based on Git.

This was also how things began at a previous startup I worked at until we grew and we could hire more resources to fix things.


using fabric for deployments (fabric2 kind of a pain but almost got most things ported over) even if our main app is in PHP


I personally like k8s a lot, but my general approach is to fake it until you make it and only add things once it's clear why I need them.

I have a few old boxes that run VMs. Some of these VMs make up a virtual k8s cluster, but almost everything I need runs on bare VMs rather than on k8s.

When I find something particularly annoying to administer, I move it off a bare VM and onto k8s.

The main thing this gives me is a migration path off of "a bunch of custom shell scripts" toward something I believe will be useful in the long run.

A bunch of shell scripts and rsync will work for a long time until they don't. And when they stop working you'll be in for some pain.


Unpopular answer alert, but it worked well for my context for the few years before I hired a team.

I didn’t do CI/CD. I just reviewed my own code the next day, improved it and put it live. As the only dev I knew more or less how my code worked and what risk factors to prepare for in each deploy, and serious, business-impacting production issues were rare.

A simple stack of Meteor on Galaxy + Mongo on Atlas meant all my tools integrated well together and a whole lot of devops things were easy enough that I rarely had to think about them after they were set up once.

I can think of many cases where this would not work but it worked well for my context.


I had a K8s setup running on Google cloud, using their managed database service. This was kind of nice, but expensive, cost ~$100 a month, and I was using very low specs.

I have multiple projects, none of them making money currently, so $100/month for each basic app is not good.

I switched to using Dokku on Vultr, $12 a month. You can easily create Postgres databases and link them to Docker apps. I haven't bothered to setup CD yet but it looks like it should be simple, for now I just push to it when I want to deploy. Liking this setup so far.

For frontend I use Netlify, I use their redirects to proxy the backend.


For a couple of persons team it is important to keep the changes small, so errors can easily identified. That means daily or bi-daily upgrades. It also helps if you focus on a certain small project and finish that, before moving on to the next project.

SVN checkout development, meld into testing. Copy production db into testing. Run a db upgrade script in testing. Test the new function for a day. If all ok stop the production HAproxy at 3am. run production DB backup, Checkout testing into the production server. mount the new dir into the NGINX path. Upgrade+start the DB, start production HAproxy.

Except for the meld and testing part, this is all automated.

A rollback on the app servers is pretty simple, just remount to the previous location. Even though there is a backup of the DB, a rollback there is not really feasible.

Everything is running in LXC containers, which I treat as cattle. Creating new ones is either automatic based on time/demand/failover or a few clicks in a custom web interface. HAproxy automatically pics them all up via DNS.

SVN and production DB are always backed up nightly and copied to a S3 compatible storage. Everything older than 6 months gets deleted. All the free space on the servers is globbed up with glusterfs to give a decent amount of fast,free storage.

Using Bash as scripting language, there is literary no problem in devops which hasn't been solved with bash and the solution is somewhere in the internet.


Here is one particular but typical example - enterprise backend server written by yours truly, PostgreSQL and single page frontend application in Javascript. I do it for my own company and also develop in a same manner for my clients.

1) Server is high performance native C++ application running on rented native hardware on Linux. It processes thousands of requests/s in sustained manner hence no real need to use any of that k8s / docker / etc. There is also separate rented standby server.

2) I maintain a reusable single script that can rebuild complete system from the scratch with a single command. I periodically test it on local VM. It installs all needed compilers, tools, PostgreSQL, restores database from the latest backup, checks out from VCS and builds that C++ server, registers it as a daemon and refreshes Javascript Frontend.

3) I also wrote small server that subscribes to webhooks from VCS and can build artifacts. I only use it to pull and build Javascript part. It could do main server as well but being super cautious I trigger that step manually fingers crossed.

3) for DB I use automatic scheduled backups.

All in all after I've debugged and tested new version locally the release takes seconds. Script is written years ago and does not require much maintenance other then updating artifacts to new versions (replace apt-get install XXX-31 with apt-get install XXX-32) and register and build ZZZ server instead of YYY server. Compared to insanity I saw in some orgs my setups are piece of cake.


I wrote https://provose.com/ so I could quickly set up databases, containers, and filesystems on AWS. I wrote Provose because I needed to deploy a ton of infrastructure to AWS as a solo developer, but I could not afford to use a dramatically more expensive PaaS like Heroku.

Provose is an open source project written in pure Terraform, so all of the high level abstractions are computed locally, and in general do not raise your AWS bill.


I'm a technical co-founder at Northflank https://northflank.com. A platform to deploy microservices, cron jobs and stateful databases from development to production.

I started working on Northflank to make it more simple for developers either solo or in a team to manage and automate DevOps complexity away.

We support CI for GitHub, Bitbucket & Gitlab (SaaS and self-hosted) with either Dockerfiles or Buildpacks. We have a robust and feature rich platform via UI, API and CLI. Out of the box you get end-to-end DevOps: CI/CD/CD, Horizontal and vertical scaling, persistent workloads with managed (MongoDB, Redis, Postgres, MySQL, Minio), backups, restores, real-time observability and metrics, DNS, TLS, mTLS, Teams & RBAC and more…

Let me know what you think of our offering + site!

Platform: https://northflank.com

Application Documentation: https://northflank.com/docs/v1/application/overview

API Documentation: https://northflank.com/docs/v1/api/introduction


From the first glance it's not clear how is it different from heroku and what exact pain points are you trying to solve


Solo founder, sold my last SaaS I had set up using "spooky arcane oldhat sysadmin deploy techniques" h/t to @Fiahil for that shade.

On that it was a Django product with deploy scripts for staging and production, i.e. `./deploy_production` ran:

   #!/bin/bash
   ssh  web@[production ip] << 'ENDSSH'
   cd /home/web/
   ./pg.sh
   ENDSSH
Where `pg.sh` pulled from `master` ran static and migration management commands and restarted the web server.

Last summer I moved away from the spookies and now do all deployment using Github Actions.

Commits to either a feature branch or `develop` run tests and all necessary deployment steps to a staging instance. (my stuff so far does not require k8s) and all containers are put up using docker compose. Commits to `main` (or `master` depending on the project)

I blended a few guides written by Michael Herman to build my GA-based CI, and it took several weeks to work on the scripts and learn Github Actions. But it was time absolutely worth investing because my DevOps is modern enough now.

No more bare metal stuff. No more scripts. Nice visual display of deployment success, "free" VMs for the deployment and CI UI integrated alongside my issues and code all on Github.

Importantly, I still use git flow. I tried moving to trunk based, but the macros inherint in git flow (including the pycharm plugin) make this production / staging // master / devops or feature/ work very well (I'm not making PRs to myself)


Having 90% of the system in a single Cloudformation template worked pretty well for me.

On every stage (dev, test, prod) I can deploy it automatically up to two times (blue and green) by running a simple shell script. So when I update the application I just deploy it another time on prod, test it one last time by adding the IP of the load balancer in my local hosts file and if I'm satisfied with the result switching the DNS entry to the new version using a weighted DNS record with the ability to switch back until I shut down the old version.

Doing anything continuous doesn't feel worth it in such a small setup with one update every 1-3 months.

What I like most about the approach is that I'm free to change any aspect in the main template without any risk to break something in the live application. Only changes to the elements shared between versions need to be handled carefully.

The main template includes VPC, network, fargate service, loadbalancer, firewall, KMS, DNS records, access rights, database tables, queues, monitoring metrics, email alerts

It excludes everything that is shared from update to update which are defined separately and just referenced from the main template such as some database tables, persistent storage container registry, user groups (Only the groups, not the rights assigned to them)


I recently started a small project on Azure for something new, but most of my experience is small companies on AWS which is similar.

If you aren’t already comfortable with K8 I wouldn’t use them, there is a learning curve there for sure and not necessary.

I wouldn’t even worry about containers at all unless you are planning on using a managed container service like elasticbeanstalk.

My CI pipeline is almost nothing, just run tests on main and dev commits. I have no deploy pipeline, I choose to actively deploy.

My deployment workflow is a powershell script since I’m on Azure, if it was AWS and I was using plain EC2 instead of a container service I would probably use ansible to avoid having to repeat commands 2-3 times when I deploy something behind a load balancer, but it would be super minimal.

DB backups are super easy to schedule with RDS or similar offerings. For rollback with database changes I would schedule downtime, let your customers know, replace your page with a maintenance banner, do upgrade, test upgrade then repoint site to your running instance.

Overall I echo what many have said, use familiar tools as much as you can, don’t worry about whether what you are doing will scale to even 2-3 people if it costs significant time. Use managed services where possible to save you time, usually the cost isn’t high vs doing it yourself.


I've used this in the past https://github.com/piyiotisk/fieldbot-server/blob/master/.gi...

Basically, building a docker image and server it using Docker compose. Of course you'll need NGINx as a reverse proxy running on the server.

If I do it again I'll use a service like render. Not worth managing this myself.


For the longest time I have used capistrano to deploy my projects on VPS from Cloud providers. It served me well but wasn't as convenient.

At the start of this year I joined hands with Chris on https://hatchbox.io to work on cost effective deployment service. We are putting together all the deployment best practices we've learned over the years into this product in a cost effective way.


It depends very much what your tech stack looks like, but I use Git, CircleCI and Octopus Deploy to Azure for my stack. It's cheap in terms of maintenance and because I know them all like the back of my hand it's quick and easy for me to set up.

As for rollback, I don't really use rollback. I have a roll-forward strategy. My DB deployments are migrations and so long as there's no data-loss caused by a migration (my migrations are always backwards compatible with the previous version) there's no need to roll back. Azure brings point-in-time restore which can be triggered if necessary.

I think the key is like with anything. There's a learning curve on your toolset. Once you've overcome that hurdle and you know to set up automation for everything from the outset. You build your pipeline shell. You never do anything manually. If you need infrastructure of any form, it's always code-first, included in source control and done the right way. Never do anything manually and rely on your automation to carry you. It seems labour intensive up front, but once you're there, it's sustainable.

It takes discipline not to cut corners. The minute you start to cut corners and get lazy is the path leading to your doom.


Not solo but I head a tiny tech team (4 engineers) so we don't have bandwidth for DevOps stuff. Here's our current state.

We are all in on AWS.

0. Tech stack: NodeJS, React, RDS Aurora, Redis, S3, Route 53, SES, Lambda and a few more.

1. 3 EC2 machines behind an ELB. Deployment is scp + cd <deployment-directory> + ./deploy.sh prod

2. Aurora takes care of everything around DB for us. Backup, read/write replica etc., etc.,

3. No CI; our development cycle isn't rapid enough to warrant investing in CI.

4. We recently moved some of long running tasks to Lambda. There's definitely a big value here so we'll invest further on it.

5. Also, recently, we experimented with an internal service on "ALB + AWS Lambda"; it's a sweet combination so we'll invest further.

6. Frontend is hosted on AWS Amplify. It's an underrated nifty service; highly recommend it.

7. Datadog is for monitoring. It's quite good but a bit expensive.

I can tell you from my experience that CI, auto-rollback etc., are overrated for smallish teams given their development velocity. Even if you have to do once or twice a deployments per week you can do it manually, don't need to wade into the CI territory. And unless you have prior experience steer clear of k8s.


Deployments: GitHub ci + docker stack + docker swarm + docker machine + docker-flow-proxy + docker registry

There are a few rough edges you face once and then forget about but minimal resource impact and it was fairly simple. I have another project that is using hacky scripts and it's a pain when something is off or I need to run something else on top of the machine.

Some stuff is not terribly maintained but everything works.

If you're small, it's definitely not a full time job. I didn't pick k8s because I initially deployed on a single machine and didn't want overhead. Things kept working afterward and I'm reconsidering whether I need k8s at all.

Rollbacks are just deploying an old version + optional down migrations if needed. I try to minimise them and fix the code and redeploy anyway.

For migrations, I guess it's very application specific, but most tools in most languages are fine these days. I use typeorm, a node.js library.

There are some practices you can follow that you probably heard working in some normal company. Eg. I don't break the db structure immediately, I deprecate columns and then delete them after a while.


I'm a solo founder with a product used by 300k+ people. Keep it simple, don't solve problems before they become problems, and keep all config in Git. I would use Heroku, but my usage requirements makes that cost-prohibitive. Heroku is good for low-traffic projects aka Enterprise B2B. I don't use k8s, I just use shell scripts that import each other for setting up new DigitalOcean servers. I separate my shell scripts into 3 categories:

1. common (Google leap second smearing, increase max open connections, misc security and performance changes)

2. infra (install Nginx, install RabbitMQ, install Postgres, install SSDB)

3. product (run common stuff, then infra stuff, then any stuff specific to a single server)

Sometimes it breaks after upgrading Linux, but it's easy to fix. For CI, I use GitHub Actions. Deployments are mostly built with GH Actions. If deployment fails it just keeps the old server code, there is no need for rollbacks I just re-deploy with working code. If deployment passes but there's still a prod bug, I don't rollback. I just commit a revert and deploy that or commit a fix. LetsEncrypt renewal, log aggregation, db backups is all custom with combination of RabbitMQ and CRON + Shell scripts.

I miss out on autoscaling, so someday I'll invest the time to migrate web servers to hosted k8s. With limited time as a solo-founder I have to prioritize for impact meaning incremental improvements usually never get done.

Not having k8s and autoscaling isn't causing any problems, won't increase revenue, and won't save any meaningful expenses so it's currently only an incremental improvement for me.

Finally, I have a small Slack community for bootstrapped founders. Let me know if you want to chat in there about specifics.


I decided to take a few years off work to just build on what I'd like. Perhaps in a startup studio model, so I have a bias for having something that is easily reusable, and that uses tech someone else can pick up and run with easily. I'll probably be in the business of dev/infra tooling.

Currently going with a container image as the minimal deployable unit that gets put on top of a clean up to date OS. For me that's created with a Dockerfile using Alpine image variants. In a way I could see someone's rsync as an ok equivalent, but I'd do versioned symlinked directories so I can easily roll back if necessary if I went with this method. Something like update-alternatives or UIUC Encap/Epk: https://www.ks.uiuc.edu/Development/Computers/docs/sysadmin/.... Anyone remember that? I guess the modern version of Epkg with dependencies these days is https://docs.brew.sh/Homebrew-on-Linux. :-) Or maybe Nixpkgs: https://github.com/NixOS/nixpkgs?

Deployment-wise I've already done the Bash script writing thing to help a friend automate his deployment to EC2 instance. For myself I was going to start using boto3, but just went ahead and learned Terraform instead. So now my scripts are just simple wrappers for Docker/Terraform that build, push, or deploy that work with AWS ECS Fargate or DigitalOcean Kubernetes.

No CI/CD yet. DBs/backups I'll tackle next as I want to make sure I can install or failover to a new datacenter without much difficulty.


I divided my infrastructure into two part.

1. Tricky and unfrequent changes: - mail server, load balancer, database etc: for this I just write bootstrap.sh, rsync to server and run. - It's literally just a for server in `foor bar`; scp ; ssh ` done

2. Deployment/Frequent changes: - docker-compose to spin up everything. It's super nice. Again, the deployment is done with a `rsync` then `docker-compose up -f docker-compose-prod.yml`

Eventually when deployment changes very frequent and need scale/ha I added in Kubernetes. K8S is way easiser to setup than you think and it handle all other suff(load balancer, environment variable etc).

And my deploy now become: `kubectl apply -f`

One trick I used is to use `sed` or `envsubst` to replace the image hash.

For backedup, I again, literally setup cronjob from an external server, `ssh` into database and run `pgdump`.

I also have a nice NFS server to centralize config and sync back to our git repo.

I used this whole setup to operate https://hanami.run an email forwarding service for the first 3 months before I added Kubernetes.


Since all the people in this thread do some kind of personal hosting/DevOps, I am posting one of my recent comments on how to make the security of your infrastructure rock solid:

> I self-host three VPNs to protect my infrastructure heavy startup https://quantale.io

I am a big fan of Pritunl which is opensource and provides network security with a lot of ease. I am in no way affiliated with them, I am just a big fan of Pritunl. I use Pritunl to limit access to servers and web applications for my different teams. For each user, you can generate a profile and assign the servers and port they have access to on the server. For eg:

- Only dev team can access ssh port(22) on stage server and not open to internet.

- Any one in the team can access stage version(port 443) for testing purpose.(Not open to internet)

- Only I can access all ports on all Prod servers(only 443 open to public)

What hackers can't see, they can't attack. Especially the port 22 on your servers should only be accessible to you and not the internet.

I self-host one instance each of OpenVPN and Wireguard with Pi-Hole which is then used to access my Pritunl Server adding extra layer of security.

Each of these 3 servers can be hosted on Hetzner $2/month instance. With a mere $6, you can add an extra layer of security to your infrastructure. Pritunl itself also provides subscription so that is also an option.

https://news.ycombinator.com/item?id=28671753

If you want to discuss more about this or security in general, feel free to reach out to me on my email on profile.


A short list of things that save me time:

Databases:

use AuroraDB on AWS. Pros: no server maintenance, no backups, no maintenance at all really. Automatic snapshopts set to 30 days. I back it up once a week to an S3 bucket for longer term storage. That S3 bucket is "aws s3 sync" to a local drive on a debian machine that has a tape drive in it. I do the tape backups just after the weekly DB backup.

Code deployment: For static assets I use GitHub actions. For the containerised servers I haven't migrated to GitHub actions yet (.NET Core app running in an Ubuntu container) but since the system is usually deployed with the AWS docker tools that should be easy to automate. Lambda functions just use the AWS dotnet command line tools.

This is all much easier when you use a debian machine as your main dev/integration box. You can use WSL2 as a reasonable substitute. The command line AWS and dotnet tools are much nicer in a unix shell as you can do things like use a Makefile or a shell script to capture the more complicated actions. PowerShell is an abomination.


(Profitable Solo founder)

Thankfully lots of tools do most of the heavy lifting. I use k8s, GKE does most of the work for me. It's very nice to have autoscaling for traffic spikes. Same with database (MongoDB Altas), dead simple autoscaling. I would never run my own k8s nor database.

I wrote more details about some of the Ops stuff I do here in a previous similar question: https://news.ycombinator.com/item?id=26204402

Coincidentally I also wrote some architecture notes about a new product last night: https://blog.c0nrad.io/posts/slack-latex/

I think everyone's milage will very, but as general principles, staging is nice, reading docs saves time overall, tests help you sleep at night and make it easier to make changes 6 months in the future, simple health checks (or anything on a critical path) help you catch the real issues that need immediate attention.

Good luck!


I try to use simpler/friendlier tools, so it can be managed with a simple SSH scripts for as long as possible. Use static binaries whenever possible. My typical stack is Go+gRPC+Flutter – so it's really easy to ship it as a single binary (for the web builds). For the DB I also started to use EdgeDB on some projects and it's such a relief in term of migrations.


As a thought exercise, I thought about how I would architect a web application as a 1 man show with a relatively small amount of users.

I tried to optimize for cost and very little devops at all. I still mull about it but so far, here is what my architecture would look like.

- no servers to manage - static website hosted on CDN, and even dynamic user-specific pages would be built statically every 8 minutes - all writes go to an SQS queue - every 8 minutes, a lambda is spun up to batch read the SQS queue and all writes go through a single writer process that writes to a SQLite file on EFS - every 8 minutes, a new static version of the site is built using the above hopefully updated SQLite file

The one thing I hate about this is that I can't in good conscious say to the user that their write request succeeded b/c no state has actually changed on the backend yet. but I might be okay with this tradeoff and just simply showing only them their most recent write.

The other tradeoff is when things go wrong, it'll probably go terribly wrong.


I'm building an open-core data IDE that runs as a desktop application or server. Since it's open source, all tests run on Github Actions for free. This includes basic e2e testing using Selenium on Windows, macOS, and Linux (e.g. [0]); and unit/integration tests.

If it were a private repo I'd still shell out for Github Actions or Circle CI most likely. I'd also consider buying a chunky-enough minipc for ~$500 and an older mac mini and set up runners on them.

For the moment private runners isn't a problem. But soon I'll need to start integration-testing proprietary code paths like querying Oracle or MS SQL Server. In that case I probably need to set up a dedicated box with all the right licenses so I can run CI jobs on it.

[0] https://github.com/multiprocessio/datastation/blob/master/.g...


I have this friend who recently founded a start-up with two other guys. But he basically is the only one doing any DevOps, so we could say he is a one-man show regarding the technology side of things. Two weeks ago we talked about how he is developing his product and he told me he will go full FaaS as much as possible, because it lets him forget about the whole Ops side of things and focus in real product development (what will ultimately bring revenue to the company). I am a big fan of playing as much sysadmin as possible for my own developments, and I like the CI/CD mentality associated with microservices and containers. All I am saying is that that conversation with that guy really opened my eyes to the fact that maybe in some scenarios (specially at the birth of a company) going full serverless might improve the time to market of your product. I am in no way saying that serverless also has a ton of disadvantages, e. g. vendor lock in.


I made my own HTTP app. server and JSON database on top of that.

The server accepts .jars over HTTP with code (and files) so I can hotdeploy while developing on live on the entire cluster in real time. My turnaround is about 1 second.

The JSON database allows for schema-less simplicity, and it has all the features you need like indexes, relations, multi crossref and security (and then some, like global realtime distributed while still being performant) in 2000 lines of code.

I have zero pain developing the most scalable (and energy efficient) backend in the world, yet very few seem to care or use it: https://github.com/tinspin/rupy

It has been proven on a real project with 5 years uptime and 350.000 users: https://store.steampowered.com/app/486310/Meadow/


Does "Book of Travels" run on the same infrastructure?

https://store.steampowered.com/app/1152340/Book_of_Travels/

I know it had some server issues on launch - too popular it seems.


No, nor I or fuse had any involvement with BoT. I wouldn't post here if I was...

BoT did not fail because it had too many players, you can clearly see how many are playing, maybe it has a complex backend... simplicity is your friend!

Meadow for reference launched with ~150 (on T2 Micro at 30%) concurrent players per machine and peaked at 450 (on T2 Medium at 25%). That's before you could buy more credits so we had to upgrade to M4 Large after a while.

And Meadow is an action game, not a point and click!

The rupy chain is solid and proven and has been running Meadow for 5 years without any write downtime except last week when one instance got nuked by AWS or somebody at M&D. That's a pretty good track record of 100% read uptime!

The Meadow launch probably was the only MMO to have a flawless launch with high pressure per server so far, and it did so over HTTP with a JSON file database! :D

It's also the only MMO with one database spanning 3 continents in real-time.


Shameless plug:

Try https://treblle.com/how-it-works

Treblle allows you to test API requests with one click. Because Treblle knows what data was sent and what was returned it can replicate any call that the user has made and quickly allow you to re-run specific requests. Treblle also provides you with a way of running manual tests from our platform.

It also has an auto-generated documentation feature. Treblle can generate the required documentation for each endpoint just after 1 API call. It understands your JSON responses, can detect various authentication methods, group variable URLs in a single endpoint, support different documentation versions, and similar.

https://www.youtube.com/watch?v=3_W7zHrsM7E&ab_channel=Trebl... Here is a quick overview.

Let me know what you think.


Hi OP and everyone else,

DevOps is not a synonym for "operations work" or "server work" or "devs doing operations work".

DevOps literally means "dev teams and ops teams working together". That's all. You don't "do DevOps" unless you are "doing the work of collaborating between two different teams of people".


I replicated Anthony Simon's architecture [0] in a few days and it has worked beautifully so far with a toy project [1]. Probably overkill for OP, but the combination of GH Actions & Flux [2] means that starting (and killing) namespaced projects – with all the cluster benefits of automatic TLS certification, DNS record management, rate limiting, pod restarts, logging, autoscaling, CD, etc. – is a single push to a git repository. Really nice if you're taking a shotgun approach and trying different ideas quickly.

[0] https://anthonynsimon.com/blog/one-man-saas-architecture/

[1] https://www.wyzetickets.com/

[2] https://fluxcd.io/


Don't use them. They require time to maintain and if you are old like me half of them make very little sense anyway.


I run a solo SaaS product with 15,000+ users. I use the "arcane/wizardry" techniques still.

I simply run "make deploy" and a new production build is made and rsync'd to production. Then the script executes a "deploy.sh" on the production server which reloads the system process to pick up the new binary.

Poof.


And if your singular production server catches on fire?


Has been five years and never happened. I don't optimize for problems I don't have.

I do daily backups and have the ability to deploy a "new" server if need be. It's a little bit of manual work but would be trivial.


I agree with most people here. Definitely keep it simple.

With that said, I run a semi resource intensive operation, so I've invested a bit into dev ops to keep our costs down. My setup is currently on AWS, primarily using ECS, RDS, Elasticache. Infra is managed via Terraform.

I felt ECS was a nice balance vs K8s, as it's much simpler to manage, while getting the benefit of maximizing resource utilization.

For CI / deployment, I use Github Actions to build an image and push, then start a rolling refresh to update the containers to the new version. It was pretty easy to setup.

On DBs, RDS handles all the backups and maintenance. For migrations, I use https://github.com/amacneil/dbmate.

Happy to answer any other questions you have, as I've learned a lot through trial and error.


You avoid it as much as possible. When your time is that valuable, lean on PaaS options like Heroku, Fly.io, Gigalixir, etc.

If it’s just you, you can afford to use a simple shell script to run tests locally before deploying or use a Travis/Circle/Cloudbees CI option too.

You have enough to focus on with the application itself.


In what specific ways are you beyond Heroku? I call bullshit on that.

I've got a big client who went from 6 infrastructure engineers to zero. They survived for half a year without any real problem. You either pay a company to manage your infrastructure for you or you pay a person. Devops people are expensive.

Things I've done to automate DevOps Make sure you pay attention to automatic testing and deploys with Ci/CD. I like automated deploys on Gitlab. Trigger staging release on merge into main, and deploy prod on merge from main into a deploy branch. Infrastructure as code with Terraform. K8S is more trouble than it is worth (though it is robust after you invest 3 months into getting it tuned right, although a lot of people make critical mistakes in their config, I've seen a half dozen insane mistakes.)


As others have said, try to avoid any kind of DevOps if you can. But here's my strategy:

Monitoring - Loads of cron jobs pinging slack (monitoring disk, CPU, Network I/O etc), Healthchecks.io, TurboAPI, Nginx Amplify. Mostly built up over time

DB - Done entirely via migrations. Most of the time they are small migrations so I can do it by running the command directly on the instance

Deployments - GitHub actions to build a docker image (there was a prebuilt template). Watchtower then runs on my server to pick up the new image and that's then deployed automatically. There is no rollback system as I've not needed that so far (and 99% of the time you won't).

Backups - DB backup is done via a cron pushing to AWS S3 Glacier. The app code is stored on GitHub already so no need to back that up. Happy to share the script if you like :)


Devops is a culture, not a verb. Success generally being measured by 4 key metrics:

Lead time to change Deployment Frequency Mean time to restore (MTTR) Change Failure Rate.

Though it often is, I do not believe it should be used interchangeably to refer to a suite of tools designed to enable CI/CD and SDLC Lifecycles.


A single VPS server with a GitHub action that builds and syncs code to the server using Git, with a bash one-liner to restart the server. Properly configured this can serve millions of requests.

A cron job which saves a copy of the DB to S3.

That’s it.

If you’re focused on anything more with no employees you’re probably wasting your time.


Sounds like you are overthinking and/or over engineering it. Unless CI or deploy/rollback functionality IS your product then it does not matter. Until you have got traction for your actual product then deployments can have 100 manual steps that you can write down and follow - it does not matter.

I have no idea what you are building or planning it to be but for my projects I have a single VPS with Hetzner per project running linux and hosting the applications. Deployment is done by a powershell script that copies files from my laptop to the server - that's it.

If I were to think about clusters, cloud instances, load balancers and what not then I would be scared away before having delivered anything of value. Cross that bridge when you get there.


I have no DB, static website, deploying in production on a VPS with git (manually) and rsync + Cloudflare. I just don't know what kubernetes is, nor do I encounter any of the problems you mention.

> Just properly setting up and maintaining a small k8s cluster looks almost like a fulltime job.

ok.


> Just properly setting up and maintaining a small k8s cluster looks almost like a fulltime job.

The managed offerings floating around these days (DigitalOcean, AWS, and Google all have them, probably more) are insanely easy to set up and maintain. A managed cluster at DigitalOcean starts at $10/month. I definitely wouldn't recommend standing up your own cluster.

For new projects, I actually find it easier to whip up a deployment.yaml and toss it in my shared k8s cluster (DigitalOcean) than setting up a new VPS where I have to manage updates (including migration when the support window for my OS ends), service restarts, etc.

Github Actions for CI. Makefiles for deployment (`docker build+push` and `kubectl apply`).


I've been using a simple circle CI setup and AWS beanstalk on my side, and for the databases I only look for fully managed. I've had a few issues with beanstalk but it still saved me a lot of time, and it's pretty easy to find AWS credits. It's pretty good also to avoid wasting time on SSL setup as everything is integrated (if you use a load balancer) with Route 53 and ACM. Beanstalk uses application version so it's very easy to rollback to a previous version. I never liked DevOps much so this setup works pretty well for me, and seems quite scalable as well.

If you're not very comfortable with k8s yet I really wouldn't recommend you to setup a cluster for a small project.


I found the HashiStack to be manageable by a single person or very small teams. It's easy to set up and hasn't let us down so far. We get away with using shell scripts to test and build our containers for now. The trickiest part is scaling as soon as it becomes necessary. Our current three-node cluster handles a few hundred requests a second without issues, but I migrated the database three times already, and I'm in the progress of migrating it a forth time to a much bigger machine.

You can find more about our setup and ops here: https://pirsch.io/blog/techstack/


I run three different saas products as a solo. Here’s some things I’ve picked up:

Some things can be simpler when there’s only one of you. Take those opportunities when they arise.

You don’t need remote CI when there’s only one dev. Just make running the test suite part of your deploy script.

Use hosted services. Sure it would be cheaper to rent a server from OVH and do it all yourself. You don’t have time for that. You’re saving money on staff, spend a little on a hosted database, k8s, etc.

I love Google Cloud Run. Love it. Seamless deploys, dead easy rollbacks, inbuilt secret management, etc.

Micro services are about scaling teams, not technology. You’re a team of one, that’s probably the number of services you should have.


I use https://www.hatchbox.io/ , like Heroku (git based workflow), but you can choose what infrastructure you run on.

Hatchbox manages the deployment, backups, rollback etc.


I strip it down to a bare minimum.

Some people probably don't realize that you can have a full-featured git-push deployment including a PostgreSQL database and Redis on few hundred lines of simple Bash code. The only exception to keeping it bare is security (proper SSL, SELinux, etc.).

- 1 cheap VM per project (I prefer Digital Ocean at this time) + snapshots

- No CI (run tests locally), no staging until later

- Little bit of Bash to configure everything, no IaC

- Simple systemd services (+ maybe systemd socket activation)

- git-push deploy... bad release? repush previous version

- Automatic system updates and log rotation

- Few auxiliary scripts for Rails console (./railsc.sh), backups (./backup.sh), etc.

- External error and performance monitoring, but keeping raw logs on server (rarely need them)

- Stable CentOS/Rocky Linux with long time support, rootless access

I teach all of that in my book https://deploymentfromscratch.com/ and I basically run my oldest side-project on the book demo which is similar to this. Some people might not like Bash, but it's surprisingly refreshing, and I keep my script flexible and idempotent:

Set everything up after providing IP address and domain name in settings.sh: $ ./setup.sh

Change the database configuration after changing the config file: $ ./setup.sh -u postgresql

Deploy a new version: $ git push production master:master

Later on I would separate the database and have staging, but it's overkill for my projects right now.

The funny thing is that apart from having CI and staging we could have the same setup at 2 early stage startups I worked on...and it would probably last 3-5 years with maybe an in-place server spec upgrade, no kidding. People really like to overdo operations and you know where overdoing it leads to? Mistakes. You maintain K8s and then forget to do something basic security-wise that almost kill the company (seen).


I'm using Hetzner, Terraform, GitHub Actions, Docker to automate the build and release of an app I'm planning on launching (..eventually). I use Terraform to provision the Hetzner VMs with Docker running on them, in there I have a container with the GitHub Runner that pulls Flutter and creates a nice build environment as well as release env (using Fastlane), then it uses the latest code, compiles, releases to Play store. Works like a charm, initially it was just a Docker container on a Pi, but the main reason I went that route was being able to move it whenever/wherever.


I think the key to getting a good answer here is the “beyond heroku” part. It is hard to make suggestions without knowing why a platform as a service no longer works.

There are so many different options available but they very for different needs.

Regardless, I would assume that you are going to have to pay more for a managed solution that managing everything yourself.

I know the founders at https://releasehub.com/ and it sounds like it might hit that sweet spot for “outgrowing heroku” but “don’t want to do full time devops”.


First you worry about meeting your paying customers need. Then you spin up just enough infrastructure to handle effectively meeting the need to continue getting paid.

You can do that with some shell scripts on an instance and an e-mail service and manually copy the whole mess for backup if it makes sense. The how of implementation is very case specific.

Thinking more on it I'd sum up as "provide a service that doesn't require that you be Google". If you start by specifying an impossible amount of infrastructure maybe there's an issue with management :).


Whenever I do anything on a VPS I religiously write down each step in a README file, so that then I can create bash scripts and automate processes.

The more I do this, the more I learn, the more I automate, the less I worry.


GitLab and Dokku.

You can run almost ANYTHING inside dokku. I use it to run a private/public docker registry, minio (amazon S3 clone), analytics via matomo, and a ton of mini apps that share a bunch of databases. It's incredible.

With dokku, deploying your app is just "git push dokku main"

GitLab plus CI is amazing. And, other than hosting, I don't pay a penny for it. If you self-host, you can put it behind a wireguard VPN. It's incredible. You can run as many runners as you want on your personal laptops, or on the cloud, and bring them up and down when needed.


First of all: no microsrvices, no Kubernetes. The whole business logic is a two binary (server + cli) monolith in a single container image.

For CI GitLab is a bliss. Lots of examples and articles around the web. I made a test-build-dockerize-tag pipeline once and now just taking it from one project to another with minimal changes.

For deploy - Docker Compose is enough. Maybe Docker Swarm some time in the future. Gitlab hosts docker images, rollback is a simple change of container version.

For Postgres backups I am using WAL archive + daily snapshots from replica.


I use Dokku and Caprover, Git hooks and two branches (staging, master) which I think is more than enough for a solopreneur. Dockerized apps are easy to reason about and fix in case of fires.


For my side project I try to keep it as simple as possible.

After I run tests locally, I've got a makefile that deploys to either Google Cloud Run or fly.io.

Database is Cloud SQL which works well enough for me.


Vercel, Porter, Heruko is sort of the simplest way to avoid DevOps. Use their integrated services for data and storage until you get large enough to afford a DevOps team.


I try to keep it as simple/basic as possible. Most of my deployments are Gulp tasks, usually in the form of: build & upload via SFTP.

Whenever there is a task that I have to manually do when building (eg. committing build hashes to a git repo, or manually replacing the version number in the UI), I create a gulp task to do that for me. Not only this saves time when deploying, but it also dramatically reduces the mistakes I can possibly make.


To further add to this, when working at a company we were using a more complex deploying system, but in the end it it was a lot slower and failed a lot more often than the simple tasks that just build and customize locally and upload to the servers.


I might be biased here, as I started as a side project and wanted to learn k8s in the process. It turned out quite funny to set up and I am happy with the result, except for the struggle I had to add SSL. I have 1 frontend, 4 micro-services and 2 postgres. One cluster for test, one for production.

I use Gitlab for CI, commits to master create a new docker image in the Gitlab registry. I manually trigger deployment on k8s with `kubectl rollout restart XXX`


Call me nuts but I run my own startup's DevOps in an extremely minimal, low-tooling fashion:

- Linode VPS

- Nginx

- SSH to deploy code

- Google Cloud hosted (& backed up) DB

Why would I need K8 and some crazy architecture when I'm just trying to get my product to my customers in the fastest / most efficient way possible? I'm not FB or Amazon and if I do get that size, I'll have plenty of time to adapt on the way there

Finally I was going to quote the Pieter levels case but someone else beat me to it


k3s on single VPS. cert-manager and external-dns for maintenance free LE certs.

nginx ingress (I would probably go Contour+Envoy if I was to set it up now).

Kotlin w/Gradle. Build + Deploy single command `gradle deployStage`.

jib for container images, don't need Docker, reproducible super fast builds.

Prometheus + Grafana for metrics and alerting.

Things I like about it:

- JVM is super low maintenance runtime, tune it once and pretty much good to go. Very easy to diagnose memory or performance problems in production. I even have the debugger agent loaded in prod listen on localhost so I can use kubectl port-forward to access it and diagnose bugs that only occur in production or hung threads etc.

- Prometheus seems like overkill but it's just so much less effort because everything already has a prom exporter.

- Using k8s API saves a bunch of time again. No need to screw around with docker-compose or some godawful config management from the 2000s like Ansible (ick).

Things I don't like:

- It's not HA. The DB pods push their WALs to S3 and I have script to spawn a new PG from S3 using PITR. This is how I test new code so it's known working but it still means that if my server goes down it's down until I do something about it.

- The box is a bit snowflaky. I didn't set it up using a script, it was hand configured up until the k8s level.

- I rarely upgrade k3s because I have to do it manually. If I was using EKS or GKE I would probably press the upgrade button more often.

- Grafana is a chore. grafonnet helps (I also use Tanka w/jsonnet for k8s manifests).


ECS gives you pretty much everything K8S gives you, in a fully managed package. Obviously you can't (simply) run ECS in development, but sticking with Docker Compose keeps the semantics close enough, as well as giving you advantage of clean separation from any other code on your machine.

CI/Deployment: hosted CI, merged PRs to main branch auto-deploy (I've used both Codeship and Github Actions with ECS)

DB: managed RDS


Keep in mind you can run EKS in aws to get K8S in a fully managed package as well.


My understanding is that with EKS, you still are responsible for parts of the process that are managed under ECS (for example, firewall and security rules as opposed to leaning on ALB and Target Groups in ECS)


Choose a platform like App Engine. It's auto-scaling. Supports bigtable. And opentelemetry. Manageable from anywhere via Cloud Shell. You can run multiple instances and partition load between them. Even includes a free tier ;)

https://github.com/GoogleCloudPlatform/golang-samples


You did that, as a solo dev with little money, and made a profitable business out of it?


not quite yet ;)

ps some of the google cloud next announcements today around appsheet, appscript, looker, ai vertex notebooks, and "no-code" solutions are starting to look really sophisticated!


Solo founder working on 2 Shopify apps. I tried to keep my setup simple with almost no build process: 1. 1 dev VM, 2 prod VM. ~$30 / m 2. Python / Django apps hosted with docker compose 3. Git pull to prod, just reload gunicorn to deploy most of the time unless I need to add new python libraries 4. Auto backup db daily with a docker container. Weekly backup by digital ocean

No k8s for sure.


I could run things myself on k8s or other self-managed systems but I have better things to spend my time on and it's not my expertise.

I host my apps and databases in Heroku. Yes it's more expensive than alternatives, but it's easy and I trust their database backups.

For everything else, I find managed services.

Basically, I'm a developer and I let other companies handle most of my DevOps for me.


My rule of thumb when working on side projects is to keep it simple:

- use tech-stacks that I know

- even simpler – use node.js / express (i.e. use only one programming language)

- Standard AWS EC2, S3, Postgres instances

- scale vertically before scaling horizontally

- use GitHub Actions as much as possible

- deploys and rollbacks are done manually

- I keep a list of useful CLI commands in Notion

If you're a small team you shouldn't need to think about kubernetes tbh.


I'll get downvoted for this but use Dreamhost (if you're just starting out).

Both their Shared and Dedicated servers are fully managed. You can run PHP, NodeJS, etc on it. They also fully manage MySQL too.

All you have to do is drop your code onto their servers and it just runs. It's also hard to beat their pricing too.


Lots of easy to use CI tools about. Bitbucket, Github both have easy to use config based pipelines/runners. Very little to manage there. Build containers and get somebody else to run them for you. There are lots of options to do that too. There should be very little overhead in managing that.


I prefer Gitlab CI for solo projects, it's super convenient. Just create .gitlab-ci.yml file, attach runners on my pc, laptops and go.

Deploy with scp/sftp.

Also it's very useful to have DEV.md file for each repo. It helps to remember how to build and deploy your project. Documentation for future me :)


Very carefully: Start with a very specific set of steps for security nodes and storing credentials. Then move on to backup and restore. All development and deployment details should have some kind of test to determine if the code or configuration are basically functional.


https://fly.io

- Easy Postgres clusters (but not managed) - Heroku like deploys (but less magic) - Easily geodistributed (deploy near your users) - reasonable pricing to get started (and not too bad beyond that).


I try to avoid any complicated tool and simplify my life with NoOps tools. Using Kubernetes or AWS from scratch is probably going to kill your startup.

In my case, I have tried MetaCall: https://metacall.io


I am a solopreneur running my own and clients' projects and the best way to NOT do devops is using Heroku (for simple/early projects) and/or AWS ElasticBeanstalk (for bigger/complex projects) — nothing beats the simplicity.


I dont think that a hosted k8s with a hosted db on, say DO, would require a full time job? I can imagine it seems that way if one is coming from a large 1k+ employee company though. I always wonder how those companies can utilize so many people.


AWS CDK, and use the library constructs and solutions they provide. It’s like LEGO blocks.


For 1-person projects you just need VPSes, git, and dpkg-buildpackage.

A single makefile command is enough to build .deb files, rsync and install them. Rollbacks are trivial.

Put configuration files as well in git and in the .deb

git tags give you a history of releases and also deployments.


probably deploy infra manually (ec2+rds or vps at random provider)

Deployments can be automated

I usually work with PHP, so I already have playbooks to configure/install php/nginx and others

(just change few variables)

deployment only needs to add a few steps like git checkout ( easy to reuse since you're probably using the same framework across projects)

some examples here

https://github.com/MalpraveCorp/ansible

in specific

https://github.com/MalpraveCorp/ansible/blob/master/mal-be-p...


For the backend of my app[1] I'm mainly using Google Cloud:

* Database: CloudSQL/PostgresQL (with Query Plan for analysis)

* Services: Kubernetes (Services in Go)

* CI: Cloud Build

It's pretty easy to maintain.

[1] https://stockevents.app


I made the decision before starting that a best practices CI/CD setup wasn't a time waster, but an integral part of building robust software.

Deployed into two off-lease dell servers; one for staging and one for production; via docker-compose.


I just use Ansible, Docker, Jenkins, Gitea, NewRelic, rclone, and some custom build services with GoLang for DB backups and maintenance, that’s it.

I’m fairly sure I can maintain up to 50 servers as solo developer with the above mentioned tools.


Digital Ocean App platform is doing most of my work and getting better by the day


Servicing over 1MM API requests / week using API gateway and lambda. It’s been set and forget for 6 years or so. AWS lambda is incredibly maintenance free if your use case fits the lambda constraints.


What about costs?


The whole point of devops or any other devthing is efficiency. If doing some devops stuff will increase your efficiency then do it. But you probably just need to find someone who will pay for something.


I use complex setups of K8S (on Google Cloud, it's simpler there), proper security, terraform etc, because I already did it before, and it as easy for me to just reuse my configs/scripts.


So none of you edits the code directly in the production machine?


For a recent small project, I tried out deta.sh. It takes out the pain of devops entirely, but it does constrain you on the architecture and database side of things.


I'm a javascript developer. My build tool is Meteor and I host on Galaxy. Deployment is one line of code in the terminal. Scaling is handled automatically. :)


this is why everytime i post here on HN that i want to make my toolset smaller, and i’m not interested in learning more tools. i catch a lot of flak, but i’ve been in this boat now for more than 15 years and I can say that if you need to manage everything with 1 or 2 people, you need to approach this differently than what a normal startup does especially if you are not able to raise funding.


I made the decision early on that a best practices CI/CD setup wasn't a time waster, but an integral part of building robust software.


Caprover on a decent VPS like Hetzner.

p.s. previously did quite a bit of AWS, Google, firebase; but the above had been working the best for about a year


I would never switch from a PaaS unless I was expanding the team beyond just myself. Working on DevOps brings 0 value to my users.


Don't.

Use a PaaS like Heroku, hook it up to your GitHub and a CI platform. Get your database hosted on Heroku or somewhere similar - plenty exists for anything you might need (Postgres, MySQL, Redis, Mongo...).

Don't do things that increase your operational overhead like using microservices or running unusual databases for which a good hosted solution doesn't exist. Stay away from anything k8s.

It will be a bit more expensive but well worth the extra money, compared to the time you'd spend on operations.


We use https://porter.run. Pretty great.


I use Convox. It’s great and has been stable the last 5 years for us at many different scale levels.


I wonder why no comments about using docker containers directly or with Swarm


Talking about myself, as a python dev:

- cheap VPS for hosting, postgres for the db, nginx as a proxy, redis for everything else, including caching.

- deploy python projects by packing them as zipapp with shiv, then use fabric to ssh and performs any migrations necessary. No, not even ansible.

- build, lint, format and test, like all automatizable stuff, are made using pydoit. If you are solo, you don't need a CI service. Your laptop is the CI machine.

Scaling is always the same story:

1 - start with the cheapest VPS possible. This will force you to code with some constraints. Not crazy, just enough to balance it out.

2 - once you start seing your load average rising, just move the db on a second cheap VPS. You just tripled the load you can sustain. Not doubled, tripled. Most solo projects will stop here actually.

3 - wait again, collect some perfs data when you see the site starting to be slow. Check for slow queries, code hot paths, etc. Optimize that, add some cache, setup tasks queues. Use cron to purge / regenerate cache if you need to smooth out the curve. Maybe slap some varnish for extra peps, or just cloudflare.

4 - now you got what looks like a real load, and a more realistic code base, so when you peak again, request more perfs for your server, or migrate to another bigger one. You can scale vertically to crazy highs nowadays. Really, really crazy. You can get terabytes of RAM, 64 cores servers, etc. At your current scale, your service should generate enough money to pay 10 times for it anyway. But you probably won't need to. Even cheap servers are beast, look at the current leaseweb offer: https://www.leaseweb.com/dedicated-servers#NL

For €320.09 per month, you get:

    2x 16 cores 2.30GHz
    28GB DDR4
    4x960GB SSD
    30 TB traffic
To put it in context, myspace used to serve all its users with 2 servers only, until they reached 500,000 accounts. Only then it was too much. This was with hardware (and price!) in the years 200x.

5 - you will probably never reach this point. This is the point where kubs, load balancers, sharing, etc. start to be interesting.

How to preserve your data:

- Raid

- Dump the db with a _randomized_ cron

- rsync the dump and all assets. You can do it to another server, or just your laptop at the beginning.

You can get fancy with database replication if you want, or use backups that stream in real time.

But there is one trick: not all data are equal. Identify some data in your db that you can't afford to lose, and make sure this one is saved separately and very regularly while being given priority. There are plenty of data, if you have a hole in it, most people don't care. E.G: if 1 tweet out of million from 10 years ago is missing, do you think it affects the service?

Monitoring:

- Sentry

That's all. It's free (or cheap), and for a small service, you don't need real time. It's ok to be down a few hours once a month for most services at first. I have a service with 700k unique users a day, it still goes down sometimes. It's a blip.

Sometimes, log into the servers, run htop and checks what's up. You can install open telemetry later if you really need to, but for now, even HN hug is not going to kill you.

Summary:

- Modern software and hardware are amazing. Max them out. Horizontal scaling is hard, and expensive.

- You don't need a perfect service. Unless you are handling patient cancer data, that is. Don't worry about perfect uptime, 0 data loss, etc. If you are a solo dev, the cost for that is huge. Just do 97% right⋅

- what worked 20 years ago still work today, and will likely be there tomorrow. And you can move from that to the cloud later. The reverse is not that nice.


People are renting 32 cores for E$320/mo?

...Anybody wanna rent 56 cores and 128GB of RAM for that rate?


If you can give the guarantees and convenience that lease web give, then yes.

Is your bandwidth unmeterred?

Do you have redundancy everywhere, including link and power?

Do you have software to remotely manage the server, including install, reinstall and emergency access?

Can I pay you with the company credit card? Do you provide a billing system with VAT?

If the answers are yes, then congrats, you can open your own hosting business.

But remember next year, the servers will be obsolete.


Just use Heroku until its time to scale


We use an external firm. Works great.


Bash scripts!


I'm afraid after Heroku is not enough for you, if you don't use Kubernetes, you may end up inventing one.

If you need a bit more than Heroku:

1. Try Render[0].

2. If it is not enough, go with any Kubernetes you can afford, any docker registry and Pulumi or Pulumi+Helm[1]. For monitoring you can use either NewRelic[2] or Weave Cloud[3].

3. If you find big cloud kubernetes too expensive, need more bandwidth, or are just willing to try more affordable alternative new players, maybe Civo[4] fits the need.

In my case[5], since I've been bootstrapping for three years, a low-budget setup is key to keeping the lights on. Here are a few constraints I need to satisfy permanently-ish:

- My workload is bandwidth-intensive, so EC2 egress is too expensive for my case and my situation.

- It is not that cheap to run 12 micro-services in any of the existing Heroku-ish platforms, also private networking and routing of my app are a bit above "12 factor" at this point.

- I can't afford the same availability as everyone else: I'm in the business of supporting other apps but I don't want to pass down the cost of AWS, so I can't afford to be "down because everyone else in AWS is down".

- I don't know many things, and until I find PMF, keeping a low-cost setup gives me the resilience that I need as a founder.

Therefore to solve my problem I use an unholy combination of Ansible and Pulumi to manage 3 clusters of Kubernetes for both stateful and stateless workloads.

Why am I doing this to myself? I have mad respect for people who can run things in a VPS, use the hosting daily backup, and don't see the need to apply patches because they're behind Cloudflare; I just can't.

More details:

- Dedicated servers, 32GB RAM, 8 Cores, 500GB SSD(RAID 1), /29 ipv4, 30TB of Bandwidth for $70/mo each. XCP-NG[6] as hypervisor.

- To assign public IP to virtual machines, I run a tiny DHCP ubuntu VM managed by Ansible.

- On top of XCP virtual machines, k3s cluster managed with Ansible.

- To manage XCP and virtual machines, I use Xen-Orchestra[7] managed with Ansible.

- Pulumi+AWS ECR builds and pushes docker images, secret and configuration management using paraphrase secret provider.

- Hashicorp Vault backed by Postgres+AWS KMS managed by Ansible.

- Ansible+Pulumi+Helm to self-host Apache Pulsar with ZFS snapshots, a cron-job and rsync.net alerts. WAL-G+AWS S3 to backup and archive Postgres/TimescaleDB managed with Ansible.

- All logs aggregated by Gelf and Fluentbit into a self-hosted instance of Seq[8] managed with Pulumi+Helm.

- Tailscale between all virtual machines

- DNSimple for domains

- Cloudflare for all things except the self-hosted docker registry.

- Github Actions for CI but not a hard a requirement, I can run ops in my local computer when Github Actions is down.

Cost: $210 /mo.

Estimated cost in AWS: $2,800 /mo

[0] https://render.com

[1] https://newrelic.com/platform

[2] https://www.weave.works/product/cloud/

[3] https://www.pulumi.com/docs/reference/pkg/kubernetes/helm/v3...

[4] https://www.civo.com/pricing

[5] https://widemesh.com

[6] https://xcp-ng.org

[7] https://xen-orchestra.com

[8] https://datalust.co/seq


My solo ops/data stack is:

- docker

- dokku

- sqlite

- litestream


I think this is going to be a pretty contrary comment, let's see how this goes.

I don't know what everyone is doing with kubernetes that takes up so much time and effort, especially in a SME environment.

I'm currently operating three separate kubernetes clusters for three different clients. And this is just me, there are no other tech people involved day-to-day. These clusters require minimal maintenance, just the occasional upgrade.

These clusters are small (1-10 nodes), and are on either GKE or bare metal. The biggest is around 100-200 running pods. I have a CI/CD pipeline setup in GitHub actions for each client. Development & CI uses docker-compose.

I used to do the old-hat 'scp your code to the server' kind of deployment, and this was a lot more pain. Kubernetes is big, but it solves so many problems. Ensuring everything is actually running, networking, SSL certs, persistence, configuration, etc etc.

I've been running this setup for 3+ years now. If I cast my mind back, I think I spent a weekend reading the k8s docs concepts section, then pretty quickly deployed a cluster on GKE. Given it was a team of one, I ditched RBAC which removed some complexity. Getting the CI/CD pipeline setup took some time (maybe a few more days), and I built more tooling as a went.

I really think kubernetes can be really useful for tiny teams, but there is a lot in there that is aimed at big teams. Use the bare minimum possible at first, and grow from there.

---

Now here is a bonus unsolicited contrary comment.

Deploying k8s used to really require a cloud provider, particularly for persistent volumes and load balancing. But I think that has changed over the last few years. OpenEBS and MetalLB look like they are really addressing this.

Also, has anyone noticed that dedicated servers are cheap as chips now? I think the equivalent dedicated server would cost around 10x as much as an AWS instance. (I'm looking at Hetzner in particular, but OVH isn't far behind)

Given what is currently available from the k8s ecosystem, there seems to be a really strong case for deploying on bare metal these days.

For example, at a ballpark you can get a 400GB+/40core cluster for around $300/month (which also includes another 80GB/10cores of ancillary gubbins). The cost for that easily comes in at around $3,000 on AWS (ex any storage).

OpenEBS gives you replicated storage, MetalLB gives you your load balancing, and each instance has a couple of terabytes of NVMe too. And, if you're worried about reliability of physical servers, at that price you could replicate your setup across datacenters at minimal cost.

---

I should add a caveat here. I've been working freelance in tech for 15 years and I am a massive generalist. I do everything from business consulting to design, devops to development, and I also run an ISP so know my way around networking/BGP etc. That being said, setting up a quick cluster on GKE doesn't require many of those skills.

Having written this ambling comment I guess I'll also say that I am am available if anyone wants a cluster setting up. I'm also about to start providing managed k8s services.


When it comes to managing your git repo's support for releases, you might like our alternative to git-flow, which we jokingly call "git-ebb":

https://gitlab.com/northscaler-public/release-management

It's a fairly low-tech set of shell scripts that implement a release management strategy that is based on one release branch per minor version. All it does is manage version strings, release commits, release branches & release tags. You can hook your CI/CD into it whenever you're ready for that.

We've used it to great effect on many client projects.

The workflow is pretty simple for a new release (assume a Node.js project in this example):

0. Your main ("main", "master", "trunk", "dev") branch is where all new features go. Assume our next version is going to be "2.3.0", so the version in the main branch starts out at "2.3.0-pre.0". If you need dev prereleases, issue them any time you'd like with `./release nodejs pre`. This will bump the version to "2.3.0-pre.1", "2.3.0-pre.2", etc each time.

1. Ceremony: decide that you're feature complete for your next release.

2. Use the release script to cut a release candidate ("rc"), say, with `./release nodejs rc`. You'll end up with a new branch of the form vmajor.minor, so v2.3 in this example, and the version in that branch will be 2.3.0-rc.0. Appropriate git tags will also be created. The version in the main branch is bumped to 2.4.0-pre.0, for the next minor release.

3. Test your release candidate, releasing more release candidates to your heart's content with `./release nodejs rc`. Meanwhile, developers can start working on new features off of the main branch.

4. Ceremony: decide you're bug-free enough to perform a "generally available" (GA) release.

5. Perform a GA release with `./release nodejs ga`. This will tag a release commit as "2.3.0", push the tag, then bump the version in the release branch (v2.3) to "2.3.1-rc.0".

6. If you find a bug in production, fix it in the release branch, issue as many RCs as you need until it's fixed, then finally release your patch with `./release nodejs patch`. You'll get a release commit & tag "2.3.1", and the version will be bumped to "2.3.2-rc.0". Lastly, cherry pick, (often, literally "git cherry-pick -x <sha>") the change(s) back to the main branch if the bug still applies there; 99% of the time, it will.

7. Repeat ad nauseum.

This allows you at least manage your versions, branches & git tags in a sane & portable way that's low-tech enough for anyone to work on and understand. It's also got plenty of idiot-proofing in it so that it's hard to shoot yourself in the foot.

Further, it's very customizable. After years of use across lots & lots of projects, we recommend using "dev" as your main branch name and as your main branch's prerelease suffix, and using "qa" as your release branch's prerelease suffix. The defaults are "pre" & "rc", and too many folks are using these scripts nowadays for us to change the defaults.


A lot of good answers here to the actual question in the OP, of "how do you do devops". And while it's good to look at other stacks for inspiration, particularly as a solo/small-team founder, the "correct choice" such as there is one, is in my opinion to do what you can do quickly/efficiently with your current skillset.

I'd say the meta-level answer here should be to timebox your investment based on the tools you know, and an honest assessment of your uptime requirements. Also, know that what feels like a janky solution will usually get you much further than you think it will.

For an initial PoC you should probably not be spending more than an hour or two setting up your infra, so Heroku or a single VM is probably the way to go. But you still _might_ want to start with k8s if it's the tool you know best; you can set up a k8s cluster in GKE with about as many clicks as it takes to set up a single node, and if you have a full "django k8s yaml" or equivalent in your toolkit then you could be up and running in an hour or two from scratch. I wouldn't recommend this to most folks though.

Once you have some traction, beyond PoC stage, then you should be thinking about failure modes, their impact to your business, and how much time it's worth to spend to avoid them. If your users won't notice a few hours a week of downtime, then you don't need a multi-node setup like k8s or nodes behind a LB; a single node should suffice. If you have simple user data (like say comments) where in the worst-case losing everything since last night's backup would not sink your business, then running your DB with no standby replica and nightly backups might suffice. If you're storing financial transactions that's obviously not an option and your first DB with customer data needs replication (or however you want to get to RP0).

Given all that, here are my suggestions for your specific questions:

For CI/CD I'd say don't bother with CD, just set up Github/Gitlab to run your UTs on your PRs and on merges to master. Honestly you can just run UTs manually while building at PoC stage, but when you have paying customers I'd say you should take the 15 mins to wire up simple UTs. Getting a full Continuous Delivery pipeline set up won't save you much time vs. just manually deploying your artifacts, and at the beginning your tests are unlikely to be good enough to continuously deploy safely. At the point where you start having quality issues that your customers are complaining about, you might want to push for CD as a discipline thing; it forces you to write good-enough tests that you're happy to deploy if the tests are green. But it's not mandatory to deploy everything if your tests are good enough to do so.

For rollback on your stateless instances, I like the pattern of having your single-node running your apps in a Docker container (I had one for Django and one for Nginx), that way the upgrade/downgrade is simply `docker pull <newimage>; docker stop <old image>; docker run <new image>`, and rollback is the inverse. But as long as you're just upgrading code and not OS dependencies, then you can do an equivalent operation by swapping out your code artifacts, e.g. checking out the new version of your python app & restarting your WSGI workers.

For running your DB, I see a few folks suggesting running your DB locally on your node -- if you are an experienced DB operator that might be a good option, but if you're not, I would personally recommend using whatever managed SQL offering your cloud provider has; RDS or Cloud SQL will give you a durable instance with backups & easy restore with a few clicks in the UI, which is hard to beat in terms of ROI.

Possibly-controversial recommendation: in general unless you've spent a lot of time working with Terraform/Ansible/etc. I would recommend against using infrastructure-as-code in the early days. It's hard to beat the ROI of a couple clicks in your cloud provider's UI to get your infrastructure set up, you might spend 10-100x more time getting your repeatable build implemented, even if it's just a couple mins vs. a couple hours. Clicking in the UI is not reproducible, but it will get you further than you think. As above though, if you're fluent in these tools and the difference in time-to-provision is very small, then absolutely do this now instead of later. But don't feel like you MUST do it "the right way" from the beginning.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: