This is a long-running project that started as the proverbial need to scratch my own itch and then somehow evolved into a full-time job of the past 6 years.
I thought I'd do a Show HN for two reasons.
1. I obviously want to show my baby to those who haven't seen it.
Its primary purpose is to do very fast file replication. If you are familiar with "robocopy /mir" - same idea, but on steroids. Lots of them. It can be used for both mirroring and archiving backups, among other things.
It is light, very small and it is really quite fast. Half of the development time was sunk into the UI/UX design, so there's that too.
Existing version is a result of 5 years of a _very_ careful evolution, focusing more on perfecting existing features rather than adding new ones. Said No to more feature requests than I can remember. The blog captures some of that in a form of development screenshots, sketches and what not [1].
2. Secondly, I wanted to add an anecdotal data point that the desktop software development is still very much an option despite of all the nasty rumors. The demand for well-written Windows software is still there. The biggest takeaway has been that there is LOTS of people, on Windows, that recognize software quality as a feature. They acknowledge and compliment it, and they are actively looking for it.
That's the niche. If you are thinking of trying the Windows ISV path, I'd aim there.
--
By the way of introduction - I'm in my mid 40s. I've been a programmer for my entire life, mostly on the sysdev side of things - firewalls, network stacks, VPNs, etc. - which is one of the reasons I still like things to be as small and as fast as possible. I'm also the original author of Hamachi VPN, there's a chance you might've heard of it.
Any thoughts or comments on the program itself, would appreciate to hear them. If anyone has any questions, I'd be happy to answer them if I can as well.
Thanks for creating Hamachi. I used it for older games that didn't have internet play, but had LAN functionality, with friends online. I never knew it was a solo project before LogmeIn bought it (and made it buggy/unreliable). What a joyfully simple program that was!
Oh wow, I've used both Hamachi and Bvckup, but never realized they were made by the same developer. Two great pieces of software.
We used a couple of alternatives after LogMeIn bought Hamachi and retargeted it at enterprise users, but none of them were ever quite as good. And as games have continued to strip out LAN support and/or gotten better at working through NATs, it's been a while since I needed anything like it.
No longer using Bvckup (my photo library doesn't live on my Windows machine anymore) but it was a nice change of pace to run something where it felt like the developer actually thought about the interface. With most utilities on Windows that's really not the case.
Hamachi had its place back in mid-00's, but I agree that a need for zeroconf LAN bridging has gone down substantially since then.
I do think however that extracting its p2p tunneling bit into a separate service could be a good thing to do even now. Its NAT punching was server-driven, which was (and still is!) a superior way to do it. Better timing, tighter port prediction. Heck, it could even establish _TCP_ tunnels through the simultaneous open trick.
Hey apankrat, thanks for your software! I used it for personal projects and considered it when I was doing full time IT work.
I didn't end up deploying it because for Hyper-V, performance of the differential scans was really poor, and IIRC there were some issues with cluster shared volumes.
Have you considered adding Windows' resilient change tracking support to bvckup2? I think with RCT and hooking into the API for backup quiescence, you could make this an enterprise grade backup product for SOHO IT needs.
Do you remember the details? First time I hear of something like this.
Re: RCT - I read up on it a while back and the overall impression I got was that it was an application-level re-implementation of NTFS write change journaling. So filed it under 'exotics' and moved on.
Skimmed through the specs again now and I think that it's a good feature to have. Basically, RCT gives the program a change list for a VD for free IF it knows how to retrieve it. I made a note, will see what I can do. Thanks for the nudge.
The full scan of say, a production database server's VHD negatively impacted performance to an unacceptable degree. It's something that probably could have been fixed with a more powerful storage subsystem.
RCT not only gives the change list, but I believe it also ensures that on a cluster shared volume (CSV) there is an easy way to safely get the changed blocks. Normally a VSS operation on a cluster shared volume puts the drive into redirected mode, in which all IO has to go through the host that owns the CSV at that time. That also negatively affects performance, effectively limiting throughput to whatever the network speed is instead of the usually much faster SAS bus.
OK, thanks. The second part is something that I, not being a practicing sysadmin, have zero insight into. Very helpful.
Re: full scan issues - that's likely due to bvckup2 being too aggressive with the IO. This is something that's easy to change [1], though I realize it's not a very good excuse and it should not require manual tweaking to begin with.
Just wanted to say I originally heard about Bvckup 2 when some one posted a link to your little IO Monitoring Tool [1]. I love running it now and then, always felt it was the best little tool to use drive usage! I feel that tool almost deserves its own little dedicated page, else I have to keep searching through the blog to find it.
Thanks for making Bvckup, its one of those tools I never knew I needed till I got it.
Ah, yes. Any excuse to do some data vis is worth pursuing. It needs quite a bit of cleanup to be worthy of a separate site though. The UI still needs a magnifying glass on high-DPI monitors and apparently not everyone is OK with that.
I have a question, it might not have one concrete answer.
If my destination doesnt actually have a copy of the data, but only the metadata (onedrive sync with filespace freed up, nasuni with files purged from the cache) can bvckup write only the changed blocks, or for example, would the nasuni need to pull the file from the cloud master back into the cache before it can patch it with the delta? Maybe this is such a tech specific question it doesnt have an answer.
I am not familiar with nasuni, but the general idea is that once bvckup2 has the block hashes for the source file, it will write only modified blocks to the backup copy.
Block hashes are computed either when the file is first copied to destination (by bvckup2) OR when the exact copy already exists at destination and the backup job is set to do "delta pre-comp" [1].
If in your case the latter applies, you may in fact get away with setting up delta copying without pulling any data off the cloud storage.
Nice software. Indie Windows software very much resonates with me. One question. How do you deal with Microsoft's requirement to sign the software? Is it expensive to get past the certification?
You get a code signing cert from any CA that sells them and sign your executables yourself. Certs can be as cheap as $50 a year / $250 of extended validation ones. Stay away from Comodo though, they are staffed by predominantly incompetent idiots and their validation process is a circus and nightmare in equal parts.
If you get an EV cert, it gives you an instant pass through Microsoft's SmartScreen, meaning that the warning the Edge (or what it's called bow) shows upon downloading is less scary.
The whole thing is in C++, but more in C-with-classes style. Very little of meta programming outside of STL, flat class hierarchy, etc.
In terms of dependencies - just the Windows API and a bit of Native API, if available. No WMI, very little of COM (for shell and shadow copying). It does however use quite a few exotics from the Windows API, e.g. SetFileIoOverlappedRange, SetFileCompletionNotificationModes, etc.
It does tree-to-tree comparison for deducing exact changes, and it can also, optionally, use file system change notifications to schedule the actual runs.
I'm going to defer to you on this one, but my hunch is that NTFS being stupid and crusty is a limiting factor? In my experience, it's dog-slow (particularly on small files). Have you tested on ReFS and compared performance?
From what I've seen NTFS is a really well-made file system. There is indeed a fixed per-file overhead that becomes an issue when working through lots of small files, but NTFS handles multi-threaded access very efficiently, so that's a very easy workaround.
It's not NTFS that's slow, it's something in the Windows I/O subsystem common across file systems. And it's not that it's slow on small files (the I/O performance is fine), but that there's a large overhead for repeatedly opening and closing files, especially by paths. Apparently Linux gets around this with some sort of inode cache (hashtable?) whereas Windows doesn't implement this for whatever reasons. (Possibly the filter drive model used to support virus scanning etc. doesn't allow it, though I don't quite understand the details of why.) Not even sure Mac is as fast as Linux in this regard.
On the other hand we have, over the years, seen plenty of fights between application and Linux kernel devs over the degree to which the POSIX spec makes it OK for Linux to discard application data, so perhaps that speed comes at a cost.
I'm trying to wrap my head around the use case here. I love to support well crafted native software so don't take this as a negative question. I often use a tool such as Arq or Veeam to create snapshots in time of a folder or set of folders that then get uploaded to a remote storage location or repository, to recover files or documents from that time. I also use cloud storage providers to keep a copy of data that I am working on in sync between devices.
I can't seem to tell but this product seems to be missing any additional features to really fill a role that I lack. It doesn't appear to make incremental copies of the data set to allow me to roll back in time, it does track deleted items but interim changes are not kept or tracked, so it's really just a capture of the state of the folder at last run time.
It's compared to robocopy but the tool assumes an empty initial destination directory, there's no facility for copying data into a directory with content in it already, so it's can't be used as a general file transfer tool.
It seems the best use case is for say transferring a directory of tarball dumps to a remote location over SMB?
1MB executable and a short, simple EULA. Wish more software came like that.
Given the lack of dependencies, have you considered offering an installer-free download? (The way Mark Russinovich does with SysInternals or Nir Sofer does with NirSoft tools?)
Hi there, you have my upvote.
I have been trying many solutions for a fast deltacopy-based file replication, and yours is really the best.
I am a solopreneur too, living on my product in the last 9 years, somehow i "sensed" this little software gem was the child of a similar mind ;)
All the best!
What is your product, if you don't mind? I'm always eager to hear stories of entrepreneurs that aren't after VC funding or fast growth and are more about the freedom of having your own business and product.
I've used Bvckup 2 for years and found it the most reliable solution for PC-driven backup and replication jobs on Windows. It has a few high-level feature limitations that have bugged me (splitting a large folder backup across multiple smaller drives, for example), but it nails the core competencies better than anything else I've used.
Re: splitting a backup across several drives - this has unique UI requirements that aren't compatible with bvckup2's core Engine/UI model. The engine runs all backups autonomously and the UI merely shows the progress. Conversely, split backups are interactive - eject A, insert B, etc.
Sure, though my use case would look more like treating multiple destination devices as one logical AND, and running the backup only when all are present. I wouldn't need any guarantees w/r/t which files land on which device or the order they're written to, only that if sum(destination device storage) > size of folder to be backed up, everything from the folder makes it onto one of the devices.
"If you are familiar with "robocopy /mir" - same idea, but on steroids. Lots of them."
Do you have any comparisons and/or benchmarks. Doing the quick 10 second scan of your site doesn't list any. So how do I know this is worth $30 for personal use and a whopping $150 for servers when robocopy does just fine and is free for both?
I tried bvckup2 maybe a year or two ago. But I stopped using it because of a couple of license issues (?).
The thing is, my work machine is Windows but my employer doesn't have a good backup solution (other than OneDrive, which I don't like much because of its speed and restrictions). I wasn't sure if I could buy a personal license and use it to backup my work machine at my own cost, since said employer wouldn't pay for this or reimburse the cost. The personal license terms sounds strict to prohibit such use. I'm not sure if that's what was meant though. The new licensing model since June seems to address this, with no distinction on the kind of use.
But the second issue was the cost of the license itself. It just is high for me (due to currency and other issues), and I couldn't find any discount codes either. Now I see that the prices seem to have increased further since June 2019 (so it's beyond my reach). I don't recall the feature set of the personal license from before, but the new Basic version seems pretty basic, and seems to have too many differences with the Pro Workstation version. Narrowing that gap could make for better tiers, IMO.
You are correct that you would've needed an Professional license for your case. Personal licenses were basically a 50% discounted option for backing up one's photos and music stash.
Re: pricing - it is what it is, sorry. We are trying to move towards pro IT use and the pricing reflects that. Basic license is not really meant for home use, but rather for certain very specific replication scenarios. In fact, I am considering removing it altogether, because it only serves to confuse simpler (read, non-IT) home users into thinking that it's meant for them, while it _is_ quite restricted.
Does this support backing up many TB many million file data sets to multiple smaller target disks? I've been looking for a utility which can handle filling a disk with as many complete files as possible, then allowing a new disk to be inserted to continue.
This used to work great with tape utilities, but every hdd based copy/backup utility I've tried assumes your backups will always fit on a single target disk. Most also fail miserably after a few hundred thousand files, so it's nice to see this one supports millions sanely.
That's too bad, the site got my hopes up when I saw the 'unlimited size' capability. It's really limited by the size of the target.
If you have any desire to support this kind of usage, I for one don't actually care what happens after the backup. It would not be an updated backup, just a copy the files to a set of disks fresh every so often.
In addition to the target not being a limiting factor size wise, using independently mountable non-spanned/raided/etc disks can simplify recovery and improve recoverability of files long term. Even if a few disks fail in storage, the files on the others are perfectly accessible without special software or hardware, and if multiple backup disk sets exist, the total number of files lost would likely be very low.
So I've been thinking about this scenario. Tell me this - say you have a mix of files of different sizes. Would you expect the program to try and pack as much files onto each drive (to maximize per-disk use) or copying files in order and switching disks when the next file doesn't fit is OK?
If it's the latter, I think I might be able to cook up something command-line for this sort of case. This is still a very niche scenario, but you aren't the first one to ask about it.
Thanks for at least considering it! The latter would work in my scenario (just copy until the next file doesn't fit, then pause and prompt for a new target). The files I work with are seldom larger than a few hundred MB, and many are only a few dozen KB. Even in the case where a few GB are wasted per disk due to unusually large files showing up, a few GB is nothing compared to the overall size of drives these days. I routinely waste at least that much when doing similar multi-disk backups manually by selecting files which fit. Simplicity and ease of use are big benefits when it comes to backing up hundreds of TBs.
I really like your landing page. FYI: on the feature comparison page, when you expand a row with the '+', the 'More on this...' links are not working yet.
Does the tool replicate NTFS extended attributes, reparse points, short file names, hardlinks, timestamps, and other metadata I can't think of at the moment correctly?
It replicates timestamps, attributes, DACL, SACL, group and owner info, as well as alternate file streams. Replication of some of these is off by default though.
It can follow symlinks, reparse and mount points, but doesn't replicate them.
I see. Note that normal file attributes != extended attributes. Which one are you referring to?
(For symlinks/reparse points, you'd probably have to provide an option on how to replicate them, since there isn't one clear solution. But it's not impossible to do something sane either. Link Shell Extension may have inspiration on various methods you could use.)
Extended attributes have been deprecated by Microsoft quite some time ago in favour of ADS. It's not hard to add support for replicating them, but there's literally no demand for that.
Not sure where you got that idea (link?) but EAs are used in WSL, among other things. Meaning you won't be able to copy WSL file systems properly right now.
As a general rule, you should avoid any information loss regardless of anyone's opinion on the matter. No one is going to think "ah well, Microsoft/etc. doesn't like this feature so it's ok my backup is messed up".
I don't have a link ready, but this was explicitly mentioned in one of TechNet articles on the subject dating back few years.
EA being used for WSL (Windows Subsystem for Linux) - not much of a surprise, really. It is a square peg in a round hole in many aspects. As I said, there's zero demand for replicating EA, WSL or not.
I used this back when I was on Windows - an absolute delight to use. As a developer I hope to someday create something as useful and well-designed as this tool!
I have been using Bvckup 2 to backup primarily to my Synology Nas. It correctly identifies that the destination directory is a SAMBA v3.1 mount, and treats it as NTFS.
I actually love using it for my Synology as it has the ability to do Delta Copying. Some of my folders contain VM's and VeraCrypt drives, and it is very fast when it comes to sending delta copies. It still has to read the file on the destination to determine what to write, but that only becomes apparent on large files. I think you may lose some features since it is a mounted network drive(mainly limitations to SAMBA), but I have had no issues with it.
Scanning remote files is fast, if the contents or timestamps haven't changed I've seen it scan over 600K files in a couple of minuets (duplicacy backup, in the process of moving to a new hard drive and need to store it temporarily on the Nas).
I've been playing with doing file change detection based on NTFS journals. There are some caveats, but it looks very promising. It's O(1) basically instead of O(file-size).
It works really well with Synology boxes. It also has quite a few Synology-specific tweaks, including support for their weird quirk of rounding up file timestamps to the nearest second and then subtracting a millisecond from it.
Hello from a fellow Windows mISV! I'd like to say that there are only two of us remaining, but RSDN forum "Shareware and business" is still alive and kicking.
There are quite a few small ISVs out there. From the top of my head and in areas related to my own - Arq backup, SoftPerfect, Syncovery. I don't think Macrium and Glasswire are terribly big either. Ditto for 2BrightSparks. HexRays. Boatload of smaller shops in the 3D tools/plugins domain. Lots and lots of examples, they are just hard to notice because they look and feel bigger than they actually are.
I love the focus on the UI/UX and performance. Looks like a solid, highly-efficient application, something that is sorely missing (or at least hard to find) on Windows
Separate product seems like it makes a lot of sense. I suspect for most it is an either/or situation and you could keep each tool's user cognitive complexity lower by shedding the unused stuff from the other destination configuration.
Sorry, not a default list. However, it looks like it's in this list which has not been updated for some time. This list blocks websites that use canvas fingerprinting.
Thanks, it's custom. It's simpler than it looks. Except for fade-in/out transitions. It's absolutely ridiculous how much effort it takes to make them flicker-free.
It's an upside down A... or at least that was the idea, which frankly wasn't the very best one. I still wonder how many sales we routinely lose because of the name.
I thought I'd do a Show HN for two reasons.
1. I obviously want to show my baby to those who haven't seen it.
Its primary purpose is to do very fast file replication. If you are familiar with "robocopy /mir" - same idea, but on steroids. Lots of them. It can be used for both mirroring and archiving backups, among other things.
It is light, very small and it is really quite fast. Half of the development time was sunk into the UI/UX design, so there's that too.
Existing version is a result of 5 years of a _very_ careful evolution, focusing more on perfecting existing features rather than adding new ones. Said No to more feature requests than I can remember. The blog captures some of that in a form of development screenshots, sketches and what not [1].
2. Secondly, I wanted to add an anecdotal data point that the desktop software development is still very much an option despite of all the nasty rumors. The demand for well-written Windows software is still there. The biggest takeaway has been that there is LOTS of people, on Windows, that recognize software quality as a feature. They acknowledge and compliment it, and they are actively looking for it.
That's the niche. If you are thinking of trying the Windows ISV path, I'd aim there.
By the way of introduction - I'm in my mid 40s. I've been a programmer for my entire life, mostly on the sysdev side of things - firewalls, network stacks, VPNs, etc. - which is one of the reasons I still like things to be as small and as fast as possible. I'm also the original author of Hamachi VPN, there's a chance you might've heard of it.Any thoughts or comments on the program itself, would appreciate to hear them. If anyone has any questions, I'd be happy to answer them if I can as well.
[1] https://bvckup2.com/wip