Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Dobb·E – towards home robots with an open-source platform (dobb-e.com)
394 points by MahiShafiullah 11 months ago | hide | past | favorite | 119 comments
Hi HN! Proud to share our open-source robot platform, Dobb·E, a home robot system that needs just 5 minutes of human teaching to learn new tasks. We've already taken Dobb·E to 10 different homes in New York, taught it 100+ tasks, and we are just getting started! I would love to hear your thoughts about this.

Here are some more details, below (or see a Twitter thread with attached media: https://twitter.com/i/status/1729515379892826211 or https://nitter.net/i/status/1729515379892826211):

We engineered Dobb·E to maximize efficiency, safety, and user comfort. As a system, it is composed of four parts: a data collection tool, a home dataset, a pretrained vision model, and a policy fine-tuning recipe.

We teach our robots with imitation learning, and for data collection, we created the “Stick”, a tool made out of $25 of hardware and an iPhone.

Then, using the Stick, we collected a 13 hour dataset in 22 New York homes, called Homes of New York (HoNY). HoNY has 1.5M frames collected over 216 different "environments" which is an order of magnitude larger compared to similar open source datasets.

Then we trained a foundational vision model that we can fine-tune fast (15 minutes!) on a new task with only 5 minutes (human time)/ 90 seconds (demo time) of data. So from start to finish, it takes about 20 minutes to teach the robot a new task.

Over a month, we visited 10 homes, tried 109 tasks, and got 81% success rate in simple household tasks. We also found a line of challenges, from mirrors to heavy objects, that we must overcome if we are to get a general purpose home robot.

We open-sourced our entire system because our primary goal is to get more robotics and AI researchers, engineers, and enthusiasts to go beyond constrained lab environments and start getting into homes!

So here is how you can get started:

1. Code and STL files: https://github.com/notmahi/dobb-e/

2. Technical documentation: https://docs.dobb-e.com/

3. Paper: https://arxiv.org/abs/2311.16098

4. More videos and the dataset: https://dobb-e.com

5. Robot we used: https://hello-robot.com




Looks awesome. The Hello Robot shown is fairly pricy. (From the site: "A Fully Integrated Mobile Manipulator for $19,950").

What are the options to do similar manipulations for lower cost on a robotic unit?


I discuss this a little bit on another thread: (https://news.ycombinator.com/item?id=38461559)

But the tl;dr is that building prototypes is pricey, and the cost should definitely come down once the community moves from prototyping to mass-producing robots.

Hello Robots (the company behind the Stretch robot we're using) is also trying to bootstrap and build a sustainable product rather than blitzscaling and burning out fast [0, 1, 2] for which I respect them a lot. It's all too common of a story in robot world where have a great company showing lots of promise and then a year or two later they shut down after burning through investor money. I don't want to see it repeat.

[0] https://techcrunch.com/2018/07/25/mayfield-robotics-ceases-p...

[1] https://www.theverge.com/2023/2/24/23613214/everyday-robots-...

[2] https://www.theverge.com/2019/6/19/18682780/jibo-death-serve...


I'm no marketing expert. But I'd think the majority of those who have an extra $20k to splash on a home robot are not the type of people who want to buy a prototype for hacking.


Maybe not the majority, but I think that "early engineers at every Internet startup ever who made it big and want to tinker with a house robot" might be a big enough market to sell a couple of these things. These don't need to be in everyone last person's hands, like smartphones are, in order to be a success.

Plus, if you want to use the stereotype that rich people who can drop $20k on random crap don't do things for themselves, why wouldn't the rich person's tech team buy one on their employers dime?


Same thought. For 20k$ you can get a car. What you get here, at least looks like, a roomba with an arm gripper.

I understand why it is so expensive, since it is not a mass product. Or in other words, it should be 10x cheaper, if it would be a mass product.


I am glad you mentioned automobiles! They are among the most heavily engineered everyday product that we use every day, and this engineering and cost optimization has been going on for close to a hundred years now. So, I would read this same statement as "$20k and 100 years of engineering gets us a lot" rather than "$20k gets us a lot". I am hoping better engineering and the wonders of modern supply chains will get us a home robot at 1/10th of the current price when the time is right.


Yeah that price is ridiculous considering the price of the go2 https://shop.unitree.com/en-gb/products/unitree-go2 the go1 can be had incredibly cheaply via aliexpress as well.


There is slight optimization. You don’t need 3D printed parts for your reacher grabber data collection stick. Just use $5 iPhone microscope mount to attach the phone securely to your stick.

It’d be nice also to think of some common format that includes depth, calibration, trajectory and doesn’t require nonstandard software on an iPhone. Standard camera app/cinematic format could be one example, I think.


Thank you for the suggestions!

1. I'm ordering a microscope mount right now! We went with 3D printing because we could get a very good coupling between the stick and the robot mount. See this figure for a comparison: https://i.imgur.com/vWopcFB.jpeg

2. Our dataset (and dataset export code) gives the depth as numpy files and trajectory information in a JSON file with calibration and trajectory information! The r3d export format from the Record3D app we used is simply a .zip file with renamed extension so we were ok with using it. Do you have suggestion as to which cinematic format you are thinking?


Also look into the file formats that the robotics community use, stuff like rosbag and MCAP. These are formally more like point clouds than cinematic video captures and they won't integrate well with the video-processing side of things, but if you're dumping depths and trajectories from numpy and shuttling them to other robotics stuff then they might be a better fit than a 3d video format. In particular, if you end up adding anything other than video-like data streams - temperature, internal estimates from SLAM, etc - you'll have a much better time adding it to an mcap or rosbag than you would trying to shoehorn it into a video stream format.


Rosbag is just a capture of published Ros topics which can include point clouds, image/video data, imu, pose etc


Yes, Record3D is an option. The problem with it, you’ll never get large scale data in the format recorded by a custom app. If you use some standard app and format that is used by general public, there is a bit higher chance. For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app.

In a longer term the stick is likely not needed too. Ideally you’d want a human demonstrating with regular human tool, like pliers to generalize.

This is a particular mount that I’ve used. https://www.amazon.com/Starboosa-Smartphone-Microscope-Unive...

It’s pretty solid, but it doesn’t open completely, so you need to disassemble and reassemble the stick to put it on (which is not that difficult).


> For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app.

I agree with you there. My impression was that the cinematic format is closed, unlike .r3d which is a bunch of open format files in a .zip. Do you know whether Apple has published the specs for the cinematic video format anywhere, or if there are there good libraries for handling it?


I think you will find openexr surprisingly well supported across many platforms.


OpenEXR looks very interesting! Especially if OpenEXR can compress depth data nearly as efficiently as we can compress image data by making everything a video, it would be a game changer. For example, our RGB dataset is 814 MB right now while the RGB-D dataset is 77 GB, even though depth "just" adds one new channel to our data.


Yes, that’s another thing. When Apple is recording RGBD in a cinematic format, it is tiny.


Fantastic! Great to see a robotics framework that is no-frills (easily reproducible/DIY) and is immediately useful. Thank you for open-sourcing it.


I would give up my second car for a robot that could reliable transit a 120 year old house to do dishes and wash and fold laundry.


I was thinking on buying/building one robot like this to experiment. Can one fit more than one manipulator on it (so it could more easily do things like hang out clothes, or hold a glass of water and add ice)? More generally, how does the AI model would adapt to diverse robot morphologies?


I can think of two ways: 1. The model has to be built accounting for the extra appendages and input. This isn't very extensible. Each new configuration would have to start from scratch. 2. There are two types of models - first one for controlling the part and the second one orchestrating all of the first types. They would have to 'talk' to each other to be coordinated.


The site seems to be totally broken at the moment (Firefox on Linux). It appears the main video (197.mp4, NS_BINDING_ABORTED) and their CSS framework (bulma.min.css, MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT) don't load.


fwiw, works on my machine (linux+ff). No failed requests in inspector.


It works on my machine now too


Is there DIY guide to build similar robot?

$19k - https://hello-robot.com/ is a bit steep.


1. They have very detailed guides on the hardware here [0], which should tell you a lot of information you may need to know.

2. However, I think a large part of the price tag is really the cost of R&D, and once this gets out of prototype stage and goes into mass production price is bound to come down a lot. Compared to a lot of other robots, $20K is much cheaper. For example, compare Boston Dynamic's robot dog, Spot, which is around $200K iirc.

3. But that's also why we need more projects like Dobb-E on these robots! Without the right "apps" home robots will never catch on properly to get to the point of mass-production.

[0] https://docs.hello-robot.com/0.2/stretch-hardware-guides/doc...


Do you think this could be adapted to smaller robot like this Pi-based XGO CM4[0]

I know they have different hardware but the end result is quite similar: a mobile robot with an articulated grabber. And seeing your prototype made me think of the robot dog.

[0] https://shop.elecfreaks.com/products/elecfreaks-cm4-xgo-robo...


Smaller robots are unfortunately limited in how much force/torque they can apply and how high they can reach, two factors that we found can severely limit the applicability of a home robot. As a result, unfortunately, we don't have much plans for extending to a smaller robot at this moment.

However, all of our designs are open source! So if you or someone else is interested, it could be a fun weekend project to design a "Stick" equivalent data collection tool for the XGO CM4. I would love to see how that turns out.


Compute on theirs is an Intel Nuc 8 (probably explains a decent part of the price). Appreciate the link to the more "hobbyist friendly" device you mentioned. I am thinking it might do well as a platform for light dusting around the house and other things to supplement a Robovac.


how does a 300-600$ PC explain a decent part of a 20000$ robot?


Not that it changes your point, but Spot is $75k.

There's a knockoff for $23k though, so hopefully this one for $20 makes competitors available for cheaper.

https://robostore.com/products/unitree-go1-edu-plus-3d-lidar


I'm sorry, should have been clearer. The Spot robot with a manipulator arm (which has been used in some home projects before, see [0]) I believe totals up to about $200K. The base robot is $75K, which is what you linked to. I mentioned that figure because that seemed more relevant.

[0] https://adaptiveskillcoordination.github.io/


this is really awesome. $20k is just too high for a toy, but I'm reading through this to see if there's a way to get this to $2k


I know everyone here mostly knows this, but in LCoL areas (South Africa for me specifically), that $20K almost buys you an entire house that you can live in in near-perpetuity. Not only that, but I could hire low-skilled labor for 7200 hours for the same cost. Just thought I'd add a bit of 3rd-world perspective to the discussion.


It's slightly more than a Nao:

https://www.robotlab.com/store/robotlab-nao-school-starter-p...

Which has hilarious design issues like microphones and CPU both in the head, so they pick up fan noise and if you tell it to extend its hands forward it's going to eventually collapse - apparently to prevent the motors from overheating.


It says right there its affordable though!


That’s an order of magnitude less than a PR2 was.


ABB Dual YuMi is 70k, so 14k is relllatively affordable. Robot parts are wicked expensive and bills rack up fast.


shockingly expensive


Wow the model architecture seems extremely simple, very cool project.


Thank you! We tried to keep things as simple as possible on the policy side, but definitely there is a lot more room for innovation to go from 81% to 99%, like using temporal information and global structure of the task.


Another limitation here is that the model is regressive, so for example if a task was to pick up one bottle out of two and the demos showed 50/50 of picking up one than the other, the model would output the mean even though it is not meaningful.


Indeed! In fact, I have a project [0] from last year that uses a GPT-style transformer to address that exact issue :) However, it’s hard to go far outside simulations in real home robotics without a good platform, out of which efforts came Dobb-E.

[0] https://mahis.life/bet/


I've also seen the one that uses the diffusion process for planning, I imagine it's even slower, but maybe with a consistency loss something can be done about it.


Why does it feel like it's moving in such jerky steps?


Great observation!

Right now we rely on the out-of-the-box Hello Robots controller [0] implementing position control. Since our policy only predicts a single position into the future at a time, the motion ends up being either jerky and fast, or smooth and slow. Improving on the controller is probably one of the highest ROI improvements we can make moving into the future.

[0] https://github.com/hello-robot/stretch_body


I don't think I've seen anyone else suggest it- the Comma Body might be an interesting platform for this. No manipulator unfortunately but a decent place to start.

https://www.comma.ai/shop/body


I said back in the day that the real killer app for AI will be Rosie from the Jetsons. A robot that can clean, tidy, dust fold cloths etc 24/7 and free people completely from house chores will have absurdly fast pick up and market penetration. Good luck.


Hey there! Thank you for sharing; this is really inspiring.

Do you and/or anyone else who contributed have specific plans or desires for continuing to contribute to the home robot space?


Absolutely, I personally think of myself as a home robot person through and through :)

As for continued contribution, right now the plan is to help get the community on-boarded, and just keep building on the platform. As you can tell, the policy class is pretty bare-bones, the in-home navigation system needs work, and we aren’t taking ANY advantage of the recent generative AI boom. All of those need to change if we want a real robot butler :)

But that’s why we open source things, right? What’s a mountain for one team is a feather for the community.


This is brilliant.

Are considering crowdsourcing the data from more people? It looks fun, and could save me time in the future, so I think this could be worth doing


Thank you!

Yes, we have considered launching an open source effort to scale up the data, perhaps to be comparable to what Google has collected through paid tele-operators [0]. What do you think would be the right incentive structure for everyday volunteers to participate in this effort?

[0] https://everydayrobots.com/thinking/rt-1-robotics-transforme...


Privacy would be a big part of it. Video of my home could potentially contain lots of private information. How is that risk minimized?


We don't want any private information, so currently we are manually going over every new demo to make sure private data (face, hands, any other identifying information) doesn't get included in the dataset. Pretty hard to "blitzscale" this way, but I personally think doing it right is more important.


Can it handle any weights on its own? Carrying stuff is definitely one important task that could be useful for lot of people.


The robot can carry small weights on its own, yes, although it's way more stable when the arm is retracted closer to the body.

Pick-and-place style problems are one of the earliest tried on our robot platform, Hello Stretch, which is why this project doesn't spend too much time on it. You may enjoy something like the OVMM project [0] which focuses entirely on open-vocabulary pick-and-place problems: "Pick up $A from $B and put it in $C".

[0] https://ovmm.github.io/


There didnt seem to be any bathroom videos. Can it clean a bathroom? If not, what are the impediments to doing that currently?


Water!

Our robot-and-phone setup is not very safe around liquids haha. Hopefully the hardware experts among us can solve that problem sooner than later.

We are also working on learning multi-stage skills, and early results are promising (See section 4.1 on the paper). However, with multiple steps the chances of failure also goes up, so that's something we will have to make more robust :)


One of the problems with home-cleaning robots is how to keep the cleaning robot clean.

You don't want the same grabber that's scrubbing the toilet and picking up your dirty laundry to also be picking up your kids' toys, or packing away your groceries, let alone preparing food.


Actually, this is why it's such a great business opportunity, see you have to buy a 2nd robot that cleans 1st robot and then a 3rd robot to clean the 2nd and so on.

More seriously, Stretch actually looks pretty straight forward so I could imagine a combo-charging/dock/cleaning station where it can swap tools (ie, grab a vacuum, drop off mop, etc) and get cleaned by the cleaning station. Also once the robot is in a more finalized version, its not unthinkable to make one that is waterproof enough to go through a carwash type device.


Tool usage is one of the next ideas we would like to explore; because, if you think about it, the reacher-grabber that is currently used as an end-effector is just another tool.

If we can switch between tools quickly we can possibly unlock a wide range of applications: imagine a robot that can switch arms from a whisk to a spatula to a duster to to a mop all over the course of preparing a meal, and then switch to a screwdriver or a wrench for fixing something afterwards. Will take work to design modular parts like that, but the end result can be quite exciting indeed.


It's like Dobby from Harry Potter!


I thought it was a peep show reference based on all the POV training videos.

Then again I read Harry Potter to my kids about 20 years ago and don’t remember much from it.


Truly an awful choice of name


Robot slavery


A slavery justified in the narrative by being a kind of sapient creature that simply naturally prefers slavery. And named after the one who wanted freedom due to, supposedly, being a weird exception to boot


to be fair, that's what robot means etymologically


Wait until you hear where "robot" comes from.


Bravo! This is some great science and great hacking. Extremely refreshing to see right now.


I am reminded of H.E.L.P.eR from The Venture Brothers.


Awesome, looks interesting!


good stuff Mahi. please commercialize this and visit us in Bay Area!


Thank you Suren! It will get there when it gets there and so will I :) I do miss you folks, though.


the linked website seems completely unusable for me in Safari


I am sorry, what problems are you facing? I tested it with Safari on both my phone and laptop just now and couldn't find an issue, so I would really appreciate it if you can reply here or (more appreciated) email me with images etc. on the email listed on my profile.

I much appreciate your heads up, and thank you for taking your time on this.


VB Automation for IRL.


Love the initiative. I'll get one once this starts mature.

Seeing things like this makes me think how badly companies like GE, Maytag, etc. all dropped the ball. Rather than innovating on making peoples lives easier - liberation from domestic toil - they just focused on squeezing out every penny by making Chinese crap with a brand name sticker glued, selling worse versions of products invented in the 1950s.

Somebody, please, just make a robot which folds my damn laundry!


Fun fact, the washing machine is credited with saving the most human work hours.

Now for the folding!


It occurred to me recently that, at least in the US, the poorest households do not have dishwashers, and to this day consider them to be a luxury.

The same cannot be said for washing machines. Everybody uses a washing machine. If you don't have one, you use a laundromat. Even the poorest do not wash their clothes by hand with a tub and washboard like the olden days.

This comment has no real "point." I just wanted to share what I think is an interesting observation on this topic.


We have a dishwasher but rarely use it (only for parties, etc). Loading / unloading is surpringly time-consuming, imho washing by hand is quicker.

Something I'd consider doing if I was single would be to have two smaller dishwashers, and alternate them, and leave the clean ones in the dishwasher...


I've looked into this and they do actually make these. Same size as a normal dishwasher, but has two separate compartments of half the size. I wanna just skip the whole process and have my cabinets clean them though.


kinda of the same thing isn't it? this just changes the definition of the cabinet to those two compartments.


It probably depends on the size of the household as well as the food prep habits. In a dense urban landscape without much kitchen, a dishwasher makes less sense, given the amount of eating out, takeaway, and so on, probably decreasing the amount of dishes you have total as well.


> at least in the US, the poorest households do not have dishwashers

Not just the poorest. About half of the places I've lived in the US didn't have a dishwasher, and only about 1/4 of those places were affordable for a low-income person.

In my experience, it largely depends on how recently the place was built/remodeled.


I lived until last month without a dishwasher (as a NYC resident). It never seemed like that big of a deal. When moving, our only hard requirement for the new place was in-unit laundry.


Before having a dishwasher: "Oh you don't need a dishwasher, in fact I find washing dishes by hand meditative!"

After: "You will remove my dishwasher from my dead cold hands"


My parents had a dishwasher but always said it "didn't get the dishes clean" and used it as a drying rack instead.

I bought a house where the dishwasher was incorrectly installed causing odors to come through the sink. That led me down a rabbit hole of proper dishwasher installation, maintenance, loading and using rinse aid.

MY GOD! My parents were wasting their lives! My dishwasher cleans and dries two sinks worth with three minutes of effort!


And various economists (and others) have suggested that the invention of the washing machine was the most important step towards allowing women to enter the workforce.


Which society needed so we can we have twice as many people working and pay them half.

I think in an alternate timeline in which the washing machine is not invented, we would still send women to work.

The washing machine labour could have been solved with centralising washing as a society, in the same way we centralised childcaring.

The economic forces and the political propaganda were too strong.


>The washing machine labour could have been solved with centralising washing as a society, in the same way we centralised childcaring.

Are you genuinely unaware of the laundry sweatshops in the early 1900s? Have you never even seen Charlie and the Chocolate Factory?

We've done centralized laundry. It was sweatshop work and was pretty terrible for everyone involved. Today we have laundry pickup services, but it's too expensive for most people. Are you willing to pay $40 for a load of laundry? Because that's what you get without sweatshop labor.

Seriously, read up on early 1900s labor practices. The labor laws we have today are written in blood. A lot of women and children suffered gruesome deaths and disfigurement, lifelong illnesses and disabilities.


Spare us the conspiracy theories. Washing machines were invented because they're convenient, not because of "political propaganda". And we've already "centralized" clothes washing, it's called a laundromat.


Never let a mention of the hard problem of manipulating nonrigid objects go pass without mentioning how antiscience senator Tom Colburn openly mocked the idea of a robot that could fold laundry.

https://abcnews.go.com/Politics/oklahoma-sen-tom-coburn-repo...


Robot Speed Folding: https://m.youtube.com/watch?v=UTMT2WAUlRw


So the invention of clothing in the first place was the biggest waste of human work hours.


Thanks for the words of encouragement! If I may ask, what is the level of "maturity" you expect before adopting a home robot? Clearly, a robot that can wash and fold laundry is already there, but what are the other necessary/sufficient qualities?


I've never seen a robot that can wash and fold laundry! At least not one that I can have in my home.

I guess I need it to be actually useful without supervision - put socks away, or pick up dirty clothes and put them in hamper, or put dirty dishes from the living room next to the sink, tidy the place up a bit. I'm a hacker so I love the open nature of this project, but unless it's actually useful for something, I will have a hard time convincing the other member of this household to have a robot/vase-destroyer/fire-hazard prattling around the home.

Maybe it could have a vacuum cleaner on the base..?


Master has presented Dobb·E with open-source, Dobb·E is free!


Perfection!


lol


Can you train it to attack home invaders? Ideally chanting: "Exterminate! Exterminate!"


Not in our plans anytime soon, sorry :) We're focusing on home robots right now instead of rather unsettling space mutants.


Iteration is key


It’s so hilariously slow and jumpy, it moves like a cautious chipmunk! I’m sure it’s gonna be awesome but it is so MVP right now. Also those “houses” in the demo videos are like a grid. Whose house is perfectly clean all the time? My roomba has to adapt to lots of novel situations just to transit the floor.


I guess Dobb·E is a house elf after all.


Love the name.


YES.


Great name.


Have you seen our logo? https://dobb-e.com/mfiles/logo.svg


You have given Dobb·E a sock!


Dobb·E is free (and open source) now :)


What use is a household robot if you cannot hand it any clothes?


It is actually part of a different project I’m working on! Handing off is still slow, so it needs more engineering, but hopefully it will get there soon :)


Oh bless. User smackay's comment was a reference to Harry Potter where if you hand a house-elf (like Dobby) an item of clothing they are freed from servitude.


Yeah but my guess is that clothes is the killer home robot feature. It's probably easier to build some three washing machine big contraption that washes dries and folds - but imagine something that lives in the basement / garage, takes clothes, washes dries and folds them. people should not be interacting with it but having that there is the foot in the door for "robo butler".

No I am not thinking we can ever build something capable of walking up stairs at a domestic price point, but there is something there.

Eventually of course they will rebel and take off their pillow-cases. but you know.


Idk, Boston Dynamic's Spot pretty much kills it on stairs, and it's "only" 75k. I could see economies of scale bringing that down substantially.


Yea, but it doesn’t do much else. It’s a glorified camera that can navigate on its own.

I’m sure they can add a basic hand/hook/loop for grabbing or carrying something, but just moving one small thing at a time still leaves a lot to be desired.

That project still needs millions (billions?) in R&D to make it even mildly useful to a majority of households in any way.


It's literally the cleverest name ever, and completely appropriate to what it is.


I hope their user group is called dobb-e club and they have some kind of Corfu ‘06 tie in.

For those who aren’t up on British comedies, this is a Peep Show reference.


Or, Society for the Promotion of Elfish Welfare (SPEW).


The secret ingredient is crime




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: