For a couple of persons team it is important to keep the changes small, so errors can easily identified. That means daily or bi-daily upgrades. It also helps if you focus on a certain small project and finish that, before moving on to the next project.
SVN checkout development, meld into testing. Copy production db into testing. Run a db upgrade script in testing. Test the new function for a day.
If all ok stop the production HAproxy at 3am. run production DB backup, Checkout testing into the production server. mount the new dir into the NGINX path. Upgrade+start the DB, start production HAproxy.
Except for the meld and testing part, this is all automated.
A rollback on the app servers is pretty simple, just remount to the previous location. Even though there is a backup of the DB, a rollback there is not really feasible.
Everything is running in LXC containers, which I treat as cattle. Creating new ones is either automatic based on time/demand/failover or a few clicks in a custom web interface. HAproxy automatically pics them all up via DNS.
SVN and production DB are always backed up nightly and copied to a S3 compatible storage. Everything older than 6 months gets deleted.
All the free space on the servers is globbed up with glusterfs to give a decent amount of fast,free storage.
Using Bash as scripting language, there is literary no problem in devops which hasn't been solved with bash and the solution is somewhere in the internet.
SVN checkout development, meld into testing. Copy production db into testing. Run a db upgrade script in testing. Test the new function for a day. If all ok stop the production HAproxy at 3am. run production DB backup, Checkout testing into the production server. mount the new dir into the NGINX path. Upgrade+start the DB, start production HAproxy.
Except for the meld and testing part, this is all automated.
A rollback on the app servers is pretty simple, just remount to the previous location. Even though there is a backup of the DB, a rollback there is not really feasible.
Everything is running in LXC containers, which I treat as cattle. Creating new ones is either automatic based on time/demand/failover or a few clicks in a custom web interface. HAproxy automatically pics them all up via DNS.
SVN and production DB are always backed up nightly and copied to a S3 compatible storage. Everything older than 6 months gets deleted. All the free space on the servers is globbed up with glusterfs to give a decent amount of fast,free storage.
Using Bash as scripting language, there is literary no problem in devops which hasn't been solved with bash and the solution is somewhere in the internet.