This is a fantastic demo. I've tried to build these type of prototypes for all Meta Headsets in the past but their very limited API/SDKs block you from doing anything meaningful with computer vision. They are too scared of devs getting access to the camera.
Hope that Apple Vision Pro gives a more robust api to developers and that forces Meta to open up.
I will use this approach for a poc I have in mind. Great job and thank you for open sourcing!
Sorry did I miss something in the demo, I must have as it appeared you just took a photo and saved it, a function native to the glasses. Where did the vision come in? I'll watch again. Or was it that you saved it to a feed and not the native app
He's created another Facebook account. When he takes a photo, he tells the glasses to share it with his other account. He has a service that checking for messages sent to that account. When a new message arrives, theres GPT4-v script that analyses the image and logs the results in his food tracker.
`Using this image, estimate the nutritional information of it and output it as a JSON using this data structure:
{
"calories":int;
"protein":int;
etc...
}
`
I was thinking the same thing. Obviously this is not meant to be a polished product, but it lights a spark that will eventually grow into a fire. We’ve always wanted this capability, and this guy has shown us it practically possible now. Doubtless there will be others, but that’s the point.
This is awesome! I got my glasses a couple weeks ago. I’m in Vegas and actually wore them for the first time yesterday. They’re actually really nice and i was thinking how cool it world be to hook it into gpt4v.
Ok so do you just wear these around in public? Do you think people will see the pretty obvious lenses and be confrontational? That's the only thing stopping me... I love tech like this but I'm not trying to get bitched at in public when people think I'm filming them or their kids or something like that, you know?
I’ve been wearing these in public with prescription lenses and in Germany, of all places.
The only person who has mentioned something so far was the guy at the door in the apple store. He recognized them immediately but I could tell he was into tech, so not an average joe and we had a nice chat about the glasses for a bit.
I always make it a point to explain that I’m not recording and that if I was the light would be on. And that if you try to cover it it actually will not allow you to record (which is true).
I have the first gen Ray-Ban Stories (they look very similar), and almost nobody noticed (~3 people in 2 years). I have clear lenses and wear them like normal glasses every day. I mainly use them for the speakers for notifications.
When the new Meta smart glasses were announced, I suddenly had >6 people comment on my old glasses. All the comments were positive, but it did make me more aware that I might get a negative reaction in the future.
Its really hard to notice that they are different from normal raybans, unless you take a picture.
The lens are pretty subtle, and they look like normal wayfarers. When the light comes on, as you make a recording or take a picture, then they notice, but not all of the time.
Ok, so if I wanted to walk through downtown on a pretty day and livestream that, it will have a bright light to let people know I am recording? I can't just do that subtly?
Yeah if you are recording, the new ones have a bright light (possibly flashing I'm not sure) on. Which makes sense really.
They offer better protection against "unsuspecting picture taking" than normal phones, as the light is vaguely tamper proof. I'm not sure they record if you take them off either.
The project seems to have 0 external runtime dependencies, it should work without "bun install". That is really cool. In the code, it is using typescript, fetch API (so no need for node-fetch) and Bun.serve (so no need for express).
From my experience, a lot of users fail to get Node.js projects running - the distributions (Debian, Ubuntu) still ship ancient 10.x or 12.x versions, npm may or may not be installed, "npm install -g" will try writing to /usr by default and fail with EACCESS, etc. Bun gets the UX right - I think that author could even try "bun build --compile" to get ELF binary with bun interpreter and JS payload.
I generally just follow the "It's a drop in replacement for node" and hope the APIs I need are there. Also makes me feel like a 2023 bleeding edge blazingly fast developer which is 90% of the use case right?
What have others experience been using Hey Meta with their Meta glasses? Myself not great as it feels like a beta feature.. doesn't run reliably. Tho good effort by them using current technology.
What the glasses do now reliably are taking photos, short videos and transferring them into the Meta View app. Bluetooth audio transferring from my car stereo into the glasses is annoying and I couldn't find a setting to change / stop that.
As for Meta Vision there's so much innovation to happen here, nice!! gives me tons of ideas
They make brilliant headphones. They have good microphones so they also work a great hands free, perfect for driving.
The cameras are subtle, and only get noticed when you point them out to people. In the street you're not going to notice they are different from any other wayfarer.
I don't use the assistant, because I'm never going to say "hey facebook" in public. Nope, fuck that.
the video is smooth, even if you are running. the stills are kinda useful for "in the moment" pictures.
For the newer version, I think its "hey meta" and they appear to do a lot more than just allow you to read facebook messenger messages.
They're actually good and have nearly replaced my airpods.
I don't need glasses and don't live in SF so I do feel a bit pretentious though.
The potential is definitely there and I'd be very surprised if the Meta Reality Labs team haven't already added heaps of features internally and are just waiting or staggering releases.
Ok, so how long have you had these? I also was unaware they had released these but they look pretty fun AND the price point is surprisingly approachable.
Are you wearing these out in public pretty regularly? Are people noticing the lenses? My main concern with buying these is that I will end up not wearing them in public eventually, which kind of makes them not super useful. I worry about that because I can see some people seeing those lenses and going off on the whole bit about am I filming them? am I filming their kids? And we all now the rabbit hole of uncomfortable public social iteractions this can take one down. So, I worry I would run into too many of these people downtown or something and end up being anxious about even wearing them out anywhere.
What is your experience and how long have you been wearing them around in public (if you are)?
Wore them everyday for 2 weeks and I actually only bought them as I thought I could make something like this (through any hacky means).
No one has noticed they're any different, especially outside as I got transition lenses so they just go into normal rayban sunglasses mode. You could have a similar effect if you wore those old "new" 3D glasses out and about.
Even colleagues had to squint.
If I do ever have a bad interaction with strangers I'll just tell them about the lengths meta have gone through to prevent bad actors. (i.e: knowing you're trying to cover up the flash).
Very nice hack! I did a very similar project integrating ChatGPT bot but using WhatsApp business account instead of fake facebook contact. I got my account blocked when Meta discovered I'm not a business unfortunately. I'll retake the project with the FB account, it seems much easier.
Ok, so how long have you had these? I was unaware they had released these but they look pretty fun AND the price point is surprisingly approachable.
Are you wearing these out in public pretty regularly? Are people noticing the lenses? My main concern with buying these is that I will end up not wearing them in public eventually, which kind of makes them not super useful. I worry about that because I can see some people seeing those lenses and going off on the whole bit about am I filming them? am I filming their kids? And we all now the rabbit hole of uncomfortable public social iteractions this can take one down. So, I worry I would run into too many of these people downtown or something and end up being anxious about even wearing them out anywhere.
What is your experience and how long have you been wearing them around in public (if you are)?
Also, legit question... how did you succesfully make an alt facebook account? lol Anytime I try with a different name so I can have an alt for buying stuff on marketplace it blocks that new account out because I can't verify it is me (because it's a fake name).
Why? Latency too big, must be connected to smartphone. A smartwatch imo is much more versatile for cheating, esp those that look like analog watches with physical needles that indicate the hour.
I have an expectation that people I'm speaking with aren't filming me without making it clear that they're doing so. Anyone doing otherwise I consider a creep and would rather not interact with.
Do also note that people wear glasses in private settings as well as public.
I think you are very confused about what a reductio ad absurdum is. It is not a fallacy, it’s a perfectly valid form of argument (similar to a ‘proof by contradiction’ in maths), and it doesn’t seem to be what you mean here.
Your can't draw over it with a marker? What about the microphones? Will this still be the case in the next generation or in glasses produced by other manufacturers?
No, if you cover the light, it will refuse to take photos or videos. Who cares about the next glasses from another manufacture? We are talking about the current Meta glasses that just came out.
I care and presumably other people with the same outlook care, that's why I was asking.
You're trying to assuage my concerns with the information about the light. That's Meta responding to these concerns and taking efforts to avoid upsetting anyone.
But once the market is proven then another set of glasses will be marketed as having the feature of silent recording. By that time the force of the market will be too great and my concerns will be laughed at and I'll be called a luddite and told I never should have had an expectation of privacy to begin with.
This is the embrace before the inevitable extinguish.
The only solution to your concerns are legislation, and that is not going to happen in America as everyone has a video camera in their pocket and the 1st amendment exists. Private establishments are free to make rules regarding the use of these on their grounds assuming that the user is not using them for a disability related service.
I'm aware of that of course. No one can turn the tide against the market. But why do they bother, then, with the whole show of making it only record with a light? You agree that that's just theatre to seem less invasive than they inevitably will be?
I'm not based in the US so hopefully some local laws will help me out a bit, but at the end of the day, the law is not my moral barometer. I will respect the rights of the users of these devices while also exercising my own rights, as much as I am able to, to treat them as social pariahs.
But aren't the lenses more obvious on these? Just looked at the Glass and I guess they are about as obvious as the lense on that but the glass stands out more for sure because of its design (which I loved).
IMO the pushback was against the perception that people wore them to signal that they had $1500 since they didn’t do that much and were fairly obviously futuristic on your face. I saw someone climbing in them and my impression was not that they were somehow relevant for the climbing..
yeah it would be pretty fucked up if I was recorded all the time, like in every store. Or like walking through a neighborhood and half the doorbells call out as you pass to inform your that they're recording. Or if you do anything slightly embarassing in public and there is a teenager nearby.
We can accept the trade offs for some surveillance while still being critical of new trends. Or maybe you're ready to have a 24/7 livestream from your toilet?
The toilet came up constantly in discussions of Google Glass, as though the presence of a camera on your head would suddenly turn you into a raving lunatic who pops his head over bathroom stalls and peeks over the urinal dividers.
This is my main concern. These lenses are much more obvious than the Google Lense cameras as well. I'd love to use some cool tech like this but I can absolutely see some people seeing the lenses and being confrontational and trying to act like you are purposefully filming them, or their kids... or something along those lines. Yes, it is legal to film in public but I'm not trying to have those sorts of confrontational interactions.
My understanding was that the pushback was based on people's discomfort around other people wearing these things, hence the "glasshole" name being used.
These don't even have the (more interesting, IMO) part though - the HUD.
Glass was obviously an early experimental design with loads of shortcomings, but at least in concept, a personal HUD was fairly interesting. I didn't care about the camera and found it disappointing that it was the only part people seemed to focus on. Isn't this just the camera/mic bit without the cool part, tied to a Facebook/Meta account instead of a Google one?
I agree $1500 was a bit steep for something barely above prototype level, but this is basically an earbud/mic that allows me to take hands-free photos - something I do maybe a few times per year.
Hope that Apple Vision Pro gives a more robust api to developers and that forces Meta to open up.
I will use this approach for a poc I have in mind. Great job and thank you for open sourcing!