I think you really want this to be 2 pieces: something "in the cloud" and a local cache.
When you're accessing from your phone (at least when not on your home WiFi) or a computer that's not at your house, it would access the one in the cloud, which would make it faster and not rely on your home Internet working.
Most importantly, if your house burns down, you're not totally fucked.
So the local device interfaces with TVs and such and keeps you from having to wait for anything to download because it's all cached on there. The cloud piece keeps your data safe and makes it a lot less likely that you won't be able to access your data because your Internet is down.
It's possible that you could use Dropbox as the cloud piece and sync mechanism and put together a special server that can do all of the TV/remote control/stereo integration he wants.
One issue is that even 100GB may not be enough for everyone.
When you're setting up a central storage for all your home content, 100GB is nothing -- you need to start speaking in terms of TB's. So, for now, the best thing is to do local=realtime, and cloud=selective backup/cache.
I've set it up like this using Windows Home Server. I use a few backup services to point it to some of the folders on the server that are important to me (my work folder for instance).
WHS has the ability to stream remotely, and I haven't found any speed issues doing so from my local network. I would like what you're describing, which is local=cache and cloud=realtime, except technology isn't there yet -- disk space in the cloud is still too expensive and ISP <-> home bandwidth still too slow/throttled/capped.
I just upgraded my Dropbox account to 50Gb, and for me, it covers a lot of needs.
I keep all my important files in there (my mp3 collection, projects, documents, etc.). I have Dropbox set up on 3 machines (main machine, family computer and my laptop), so I get automatic syncing of all these files, plus backups of all the files on 3 computers, plus access to all these files in the cloud.
As far as I'm concerned, it's a revolution in how I work. Dropbox is now my guardian angel, silently backing up everything I do.
The only problem with this arrangement is that 50gb is too small for all my files (videos, mostly), and even their 100gb offering is too small for that. I expect this to change in the near future.
What I would really like is some metadata baked into the filesystem layer that allows you to specify backup levels, which vary how often, whether or not, and where things are backed up.
So something like:
1 - always back up to primary backup locations in full.
2 - back up to primary backup locations if it's not too much trouble (some sort of configurable threshold for space)
3 - like 2, but a lower threshold
4 - only back up to secondary backup
5 - do not back up.
I know that there's a lot of configuration that would have to go into this, but I think as data becomes a bigger part of our lives, we need to have a more intentional attitude towards how we store our data, and not just 'let the computer back it up.' Until movies are as easy to store as text is currently, there will always be tradeoffs that we will want to think about.
The beauty of Dropbox is the "it just works" feeling. You put things in a folder, it's synced to all other computers, plus the web. It really is simple in the best sense of the word.
Having said that, the latest beta does allow you to choose to only sync some of the folders. So you can, for example, install Dropbox on your work computer, and choose not to sync your mp3s to it, etc. This should solve a lot of the use cases you have.
The only thing really missing, for me, is the ability to exclude specific files/folders from the Dropbox folder (like, for example, per-computer configuration files). I assume they'll allow that at some point, but for now I have to work my way around it. Still worth it though!
I would love to do this, however, the cost is hard to justify. Though, your post makes me think twice about it. The backup of all my "stuff" would be a welcomed safety net and maybe enough to justify the cost. Thx!
yes. the problem with all online storage services is keeping multiple copies of the data to protect against disk failure. I wish there were a dirt-cheap service which said "we'll shove your files on a drive and keep it online. You'll lose data when the hardware fails; that's your problem". For most of my audio/video files, it'd only be a minor inconvenience to have the online version explode through hardware problems. As it is, I end up paying for reliability that I don't need.
Yes. I do this, although I also use a dedicated HTPC by my TV. A little expensive but gets used all the time and I can stream Netflix, Hulu, watch a movie in almost any format including Blu-Ray.
I've looked into alternatives, (PS3, dedicated media players, etc, etc) but nothing else has everything I wanted.
I've never been too crazy about a device that both (a) stores my media and (b) plays it. Since Hollywood and the scene cannot agree upon formats for any length of time, freezing the state of the industry into a closed box leaves me in fear of not being able to play something in the future.
Specialize. I want a device that stores my media (I already have this), and I can then replace the "display media on television" part with whatever I like. When a device tries to wear too many hats, quality of each hat tends to suffer, in my experience.
(As a totally tangential rant, I've been using Opera recently for no particular reason, and it's been frustrating me how left-behind it's been, even by the tech crowd. The latest version is both faster and more responsive than Chrome, which had been my browser of choice.)
I've had similar feelings. You would think that products that are superior from a technical standpoint would be enough for them to get some traction in the technical community.
I can't quite put my finger on why, but Opera always seems to get ignored.
My longstanding theory is it's totally the name. What do most people think of when they hear the word opera? Lame, annoying, outdated, slow paced, inaccessible, old, boring, etc. I'm hard pressed to think of a worse name for a web browser.
Yep, the problem with their brand is that it's not Google or Apple. If Apple or Google were to release a clone of Opera tomorrow it would have significant higher usage stats and buzz.
Lets face it, as much as we would like to believe we judge technology based on it's merits, the truth of the matter is we're just in High School 2.0 and what really matters is that we're hanging out with the cool kids. In this case Opera isn't a part of the in-crowd, a real shame.
I don't think this is true, at least in the general case. When Chrome first came out (and I was still using Firefox), I downloaded it to test it out, but didn't stick with it, because it wasn't immediately compelling. I came back to it occasionally, and after about 6 months they had turned it into a really great browser, much speedier than Firefox.
If you look at the usage trend during the first few months, you'll notice that it starts high and curves down, then slopes back up again. I think this supports my argument. The reason that it gets a lot of share has little to do with branding so much as the fact that they made it in to a great browser after a lot of hard work on their part.
The litmus test to my observation is that if this is really true, then Opera should pick up a lot of usage share from people like me who switch between browsers based on the latest and greatest. I remember trying out 10.10, and although it was good, it was no where near this good.
Of course, Opera doesn't have that nice bump that Chrome does where Google puts "Hey you! Try Chrome!" on incredibly popular websites, so we'll see.
Google and Apple also have the advantage of being able to promote their respective browsers through cross-marketing. Safari is the default browser for any Apple device, and Chrome is the promoted via other Google properties. Internet Explorer has the obvious Windows advantage. Only Firefox really succeeded in fighting the same fight that Opera was fighting, and it's not surprising that both Chrome and Safari tried to ensure Firefox didn't win by improving upon the browser in one of its biggest weaknesses .... speed.
Ultimately, competition is good for the user, but a nightmare for the web developer. It was hard enough dealing with the multitude of IE versions, but today's developer needs to support the entire cross-section of users on a variety of platforms ... most of which command a non-negligible user-share.
True, but if Opera was better enough, it'd surpass them. The fact is, it's a web browser, and Google can put enough engineers on it to build a better JS engine just because they feel like it. There's not really any room for a killer idea, or at least I can't see any.
I don't think there is a single reason Opera is unpopular. It just doesn't have a single stand-out feature people can associate it with.
Firefox's thing was openness, and an ecosystem of extensions. Chrome's is speed (and openness, and being sponsored by Google). Opera? It's fast, but not as fast as Chrome. Not open. No extensions. Built-in mail client? Gmail is better and easier.
When switching to a new browser, people have to put up with the inconvenience of some sites being broken, in return for some payoff. With Opera I can't see an appealing reason to do this.
Finally, I think Opera has given up on desktop users and concentrated all efforts on sales in the embedded market.
What you're looking for is Windows Home Server. It's amazing. Stores all the media you need, plus has streaming capabilities over the net. Able to expand perpetually -- no need to reformat or anything like that. I have it running with 10TB of space. It has a great front-end UI, and as close to zeroconf as you can get with this type of service. See exampel screenshot: http://ivivo.files.wordpress.com/2008/09/windowshomeserver_2...
EDIT: Forgot to mention one of the biggest pluses. Microsoft pioneered a special file system for WHS called Drive Extender. There are no drive letters, just folders, and it spans across all your hard drives and future hard drives. No more "oh that video is in H:, oops I mean K:", it's just "It's in the video folder." See: http://en.wikipedia.org/wiki/Windows_Home_Server#Drive_Exten...
The best thing is that WHS is tightly integrated with Windows, so it has things like autobackup (selective and not), health monitoring (if any of the PC's hooked up at your home springs an alert, like virus found, or low disk space, etc, it notifies the other computers through dismissable systray popup), central user/password management across all your machines, and other cool things. Many companies have also developed plugins for WHS as well (useful things like PerfectDisk for defragmentation). It also works/integrates with Mac environment.
For playback of media content, I have two HTPCs connect from two different rooms as well as some Xbox 360's (yea, the xboxes just found the folders and was able to play the content. it just works). Also, I installed Air Video to stream videos to ipad/iphone.
And since it needs to be always-on, I hooked up a UPS battery to it. It's headless -- just the machine is needed, which is great for a server in the closet. You'll manage the WHS through remote desktop/web ui. You'll also want to get the Rosewill RSV-S8 -- you will need more and more space! (Some people have gone the "octopus" -- lots and lots of external hard drives hooked up through USB. I initially went this route but the throughput became a bottleneck so I got the Rosewill - See: http://www.newegg.com/product/product.aspx?Item=N82E16816132...).
Some use-cases: I've got private directories for myself, then I've got public directories such as Photos and Videos, with Photos being writable by certain members of the family. This means I can create a folder, say "07.04.10 July 4th in Boston", and each of us can upload our iphone/camera images to that folder. Now we have a global picture folder for the event we shared together. With Videos, since WHS supports Windows Media Center as well as just normal file streaming, I outfitted my WHS box with 1GBit ethernet card and the machine can stream out files to each of the HTPCs and other consuming devices.
Oh yea, the new Windows Home Server release coming out later this year is going to be even better. If you've got an MSDN subscription, download it and mess with it. You won't be disappointed. It's one of the best products Microsoft has put out.
I know WHS is probably easier to set up, but Microsoft did not pioneer the concept of "drive extender". Linux had something called LVM since 1998. According to Wikipedia, IBM AIX had a similar feature in 1989. MS also had similar capabilities in NT for quite some time now (disk volume sets), they just weren't shown to the home users.
Autobackup can be done by rsync, file access by Samba, printing through CUPS. Oh yeah, you can run all of the above on an $50 wifi router with custom firmware. It will also barely use any power and not make any noise.
EDIT: I am not attacking MS, and I don't hate people who don't use Linux. Just pointing out there is a way to achieve this functionality with mostly off-the-shelf software and hardware. Startup idea? Nah, more downvotes :)
Yea, the allure to WHS for me was the zeroconf and that it all just works with all the devices/terminals I already have with minimal effort.
RE: concept, understood it may be used for other OS's. Was pointing out that WHS uses a different file system than normal windows, one built to scale in terms of drives.
I've been tempted by WHS for a while now. I have a few questions for you, since you seem pretty experienced with it. How does the file system deal with failure? What if one of my drives fails? Can I set it up to be as redundant as RAID 5 (i.e. I can lose any one drive and it's okay)? What if my motherboard or OS installation fails? Can I reinstall the operation system and have it recognize my file system (assuming I still have all the drives from it)?
if OS installation fails, you can do a server reinstall and it will preserve your data. Alternatively, you can take out the disk and plug it into another PC and there will be a hidden folder to access the files called \Data.
in terms of drive failure, in my experience it's less about the drive and more about the data you have. so, for instance, if you value your Photos directory like I do, you can select that to be duplicated across multiple drives. This will make sure there are multiple copies of that folder in case of any single drive failure.
I've also got an external RAID that I've connected to my WHS. WHS sees this as one drive, while in reality it's two drives in RAID config.
I've trusted my WHS build with 10TB of very valuable data. So far so good.
I can't seem to find info on whether WHS's filesystem checks for corruption or not. One thing ZFS has going for it is detecting corruption, and when you're storing 10TB of your most precious media to a single machine you really need to trust the filesystem.
Do you know if it verifies data integrity? I have heard about WHS from a few people now and it seems like the best solution. I don't even have a Windows machine so that may sound odd, but afaik there's no Unix-based equivalent available today.
In the current version of WHS (I can't speak for the upcoming "Vail" release) you can think about its redundancy in terms of file-level software RAID 1 without integrity checking.
Pros and cons abound in that setup, of course, but setting aside the lack of integrity checking ala ZFS for a moment, I have personally taken advantage of the file-level software RAID 1 portion when my first WHS server encountered an internal hardware issue and I had to pull its drives (normal NTFS drives full of the files) and bring them online in a replacement WHS server. Each drive was brought online out of the storage pool, its files copied into the pool volumes, and finally the drive was added to the storage pool. It requires an extra drive already in the pool to pull this off, but given that it worked very well, far better than some of the horror stories I've heard about people trying to transplant RAID 5 and 6 setups when other NAS solutions have failed at the server level. A part of me of course wishes for a ZFS-style WHS, and perhaps MSFT will get there some day, but in the meantime I can certainly see how WHS's design decisions were reached.
I haven't experienced much data corruption but it has bitten me here and there. With terabytes on each disk these days I really wish more systems verified our data, even if they can't repair it I want to know when my disks are no longer reliable. Ah well, can't have it all. We'll get there eventually.
For public URL, you can install dropbox on the WHS box or one of the connected PCs, if you'd like.
For movie playback, that's separate from the WHS. You need to think of the WHS as the Server and the playback devices as the Client. In this way, the WHS is very extensible and your playback options are limitless. You can use windows media center on a different PC, or just use VLC or windows media player.
Effectively, WHS is a very "insanely great" home server that stores your files and serves it, allowing you to decide how you want to consume it.
Because I'm a home theater junkie, I did start a side project called http://www.intellimote.com that uses the iPad as a remote to browse and play movies.
My blog post was originally about a device that just worked like I described out of the box with no other requirements.
Preferably it would be a small box like my media playing storage device that I have or like the tonidoplug, which costs $100 and is already set up.
Your WHM solution is super awesome, but it's not something I can buy from the store, plug in, maybe configure a little bit and start using. It's a project. It's something you work on to get it working the way you want it to.
What I'd like to see is somebody create a pre built solution, similar to the tonidoplug, with more media capabilities etc, video outputs etc.
If some startup or company would build that, people would buy that.
Your solution is awesome, I must go ahead and try WHS, I didn't even know that it existed.
It uses Windows Home Server plus their own custom add-ins. They're nice. You get everything I mentioned before, except this is pre built. They have things like integration with Tivo. I decided to build my own since I had spare parts lying around, otherwise I would have bought one of the MediaSmart machines.
Also, when you say something with "more media capabilities", it is doubtful you will be able to surpass a WHS-based system. This is because a system like MediaSmart, out of the box, will stream everything to any PC, any mobile device, any windows media center, etc. They've built it for streaming media, and all kinds of ways to access the content are already in the ecosystem. They have even included a silverlight/flash based streamer if you decide to only use a browser (they transcode on the fly then stream it to your browser). Devices (eg. xbox 360) and even entire OS's are designed to work with WHS (Windows XP, Vista, Windows 7).
100% agree, I have an HP MediaSmart and am very happy. Small, attractive, and very quiet.
Setup was a breeze, plug it in, let it do the windows update thing, while still running pop in additional hard drives (4 bays total), configure them, done.
For video clients I have both the Asus O!Play and the WD HD Live, they both work great but lack search and content download features.
The Acer Aspire Revo running XBMC seems to be a very popular and affordable option for that. The UI is beautiful, downloads movie info and covers from IMDB, very slick.
That was my first reaction, too. Once you add enough features (streaming video, web interface) it stops being a "storage device" and quickly becomes a general purpose computer, which have existed for a while now. It's not a sleek and trendy answer, but it would sure do the trick. And it would be a lot more customizable for whatever additional crazy plans he dreams up for it, while any gadget equivalents quickly fade into obsolescence.
Why are you taking his advice literally? His advice is to a hypothetical startup. The point the author is making that a startup that would make what you describe into a zeroconf plug-it-in-it-works solution would be very successful.
The author also wrote in the context of a problem and under his "solutions out there" he did not list the self-configuring solution as a possibility. Presumably, the author may not be aware of this solution and we are just throwing it out there as a suggestion. As gdl noted, there are advantages to this solution over a plug-it-in-works solution. It's pretty straightforward to configure and set up too (google instructions, download, and tweak some settings); provided you have the hardware lying around.
It seems to be phrased both ways, like "I want something that does these things, and I think there is a market for it". We were mostly addressing the first part.
I agree that a consumer-oriented gadget to do the same thing easily could be neat. I don't know that there'd be that big of a market though. I suspect most of the general population wouldn't see the need, and the very geeky would tend towards existing free methods of doing the same thing (MythTV and the like).
I don't see any really compelling reason to have all that functionality shoved into one device. I like the idea of separating playback/record from storage.
I like the idea even better when the storage is in the cloud (with a possible local cache, would be nice).
Doh... missed the conversation. I am one of the lead developers for the open source ClearOS server/gateway. The software is targeted at small and distributed organizations and you can find the gory details @ http://www.clearfoundation.com/Software/overview.html
We have always talked about adding a new edition for the home market. In fact, it is on the project roadmap for next year. It will be a labor of love of course since it would be difficult to build a business around an open source home server. As already mentioned in this thread, all the pieces already exist -- it's just a matter of putting them together into a coherent package. The "storage device of his dreams" is running in my house.
- file server w/ RAID
- media center with PVR
- backup to the cloud (documents, music and photos -- videos are too large for my crappy DSL line)
- remote access via iPhone or web
Unfortunately, geek credentials are required to get it working and keeping it running. It saved the day when I had to start recording the World Cup overtime on Sunday. Fired up the iPhone on the road and added the new recording -- screenshot @ http://www.mythtv.org/w/images/7/7c/MythWebList.png
I think all of this guy's requirements would be satisfied by something like a Mac Mini with two additions:
- The ability to mirror the Mac Mini in the cloud
- Something like an Airport Express with an HDMI output
Simply leave the Mac Mini on all the time. It becomes the stand alone device, satisfying point 1. You can use the iLife suite and a few other programs (mostly free) to satisfy point 2. It has a remote and has been satisfying all the other points except 4 and 9. The Airport Express+HDMI thing would satisfy point 4. (And Apple TV already does this, but it costs too much to get just for that function.) Being able to mirror it in the cloud would give you point 9. (It's only a matter of time before we see a Time Machine back-end in the cloud.)
Basically it has every need filled. The only exception is the playback I handle through a ps3/apple tv's. Home NAS storage. Easy. Plus it's built on linux, modules are easy and fantastic.
I also like both Thecus and ReadyNAS, with Mac Mini or Xbox 360 as head-ends at the TV sets. Been using these for years. The (add-on|plug-in|module) community for both NAS servers is great.
That said, both Microsoft's Media Center and Home Media Server are also fantastic and extensible.
FreeNAS is awesome and extensible. Out of the box it's quite functional and it's open source.
It's the backbone of my work/home network. All the Linux boxes have it mounted in the home directory, so all the machines with Rhythmbox think it's local. All the windows machines can access it easily.
There are a dozen services you can enable as your needs dictate. Setup is straightforward through a web interface. Extra bonus it runs headless on crap hardware, in my case a 433mghz Emachine.
In my case, this device is a Mac Mini + a Drobo. The Mini is connected to my TV and a receiver and is pretty much always on ... I can access the files pretty easily with MacFuse using SSHFS.
DLink DNS-323 with a sync-to-cloud funplug script would probably be near ideal. I have one of those boxes, and I have to say that it's a pretty awesome little Linux box.
Why does Ctrl-F “crypt” not find a single hit on this web page, neither in the blog post nor the comments (there, or on yc)? Are you guys really not concerned about security at all? I’d demand industry-level encryption on both the storage device and the communication protocols. Stealing the device should leave the thief with nothing but worthless random data, decryption/encryption should completely take place on the clients.
I strongly advise everybody to stay away from Iomega. I bought a 1TB NAS a few years ago when 1 TB was a lot. The web UI, to make shares, was so stupid that it would create a tree, in memory, of all directories and files on the NAS every time you tried to make a new share (this tree was passed as javascript to the web UI). Needless to say, after a few hundred gig the amount of files was so big that it took longer to make the list than the timeout was set, effectively rendering making shares impossible. There were dozens of complaints on the support forum (that's right, support through a forum!) and after a year and a half of no updated I gave up hope. Shoddy material, and that's being generous.
MythTV is awesome...my MythTV box from ~8 years ago was light years ahead in terms of features and UI than my cable companies (Shaw cable in Canada) PVR today.
Does anyone know why the software is so bad on cable boxes? I mean, it is just so incredibly bad, in so many ways, it seems like it has to be deliberate.
consumers are not used to paying for set top boxes. steve jobs stated as much in a recent interview when they were evaluating going after the tv market or the phone market. they chose mobile over tv for that reason.
there are many failed attempts at cracking the living room -- the cable companies make most of the money and they have little incentive to truly innovate the boxes that they are "renting" out to their customers.
to get truly elegant UI's that are powerful enough to do anything useful with all the TV signals (HD, etc) requires significant hardware. when Tivo was ported over to comcast boxes, most of the features that make it tivo were slashed since the boxes comcast gives to customers were significantly cheaper and underpowered.
most cable companies have a monopoly in the physical area they are servicing. where's the incentive to increase cost by upgrading equipment for your customers when the minimal viable product is happily received by the smith household?
Does anyone else find downloading and organizing digital media to be time consuming? If so wouldnt u prefer to just be able to stream everything from the cloud to ur Internet devices (iphone, apple tv, pc, etc..)?
I use to download now i find myself streaming what I want from the cloud. I.E. Im not hyper on quality so listening to music/songs and playlists using youtube on my iphone suits my needs.
When you're accessing from your phone (at least when not on your home WiFi) or a computer that's not at your house, it would access the one in the cloud, which would make it faster and not rely on your home Internet working.
Most importantly, if your house burns down, you're not totally fucked.
So the local device interfaces with TVs and such and keeps you from having to wait for anything to download because it's all cached on there. The cloud piece keeps your data safe and makes it a lot less likely that you won't be able to access your data because your Internet is down.
It's possible that you could use Dropbox as the cloud piece and sync mechanism and put together a special server that can do all of the TV/remote control/stereo integration he wants.
One issue is that even 100GB may not be enough for everyone.