Excellent work. Almost forty years ago my first paid job on graduating was clean room reverse engineering a spec of a binary dump of somebody's code. You put me in awe of how well you did this.
As a former chdk user on canon and a Fuji x20 owner I hope somebody carries on your amazing work!
Did you have problems with motivation or feeling that the project was too daunting? Or was it natural? Some projects I wanted to do I just stopped because I wasn't getting anywhere and felt like someone possibly did it better before (though I don't think anybody did something similar to what you did there) and I just quit it. Do you have any method to overcome such dead ends or negative mindsets?
Nah, it is not about motivation, it is always and only about spare time. Family and job obviously have priority.
When I stuck or tired due to lack of progress I just take a break from the project for a day, week or even month. I have noticed when I get back to that fresh, something good happens straight away, new ideas are popping up, things start to spin :)
I am absolutely blown away by the dedication of this person. I can't imagine spending so long working on this.
Somewhat unrelated but I wonder if we will make the move to open hardware like we have done with software. Having access to schematics and firmware source does wonders for the reparability and upgradability of hardware.
This comment is kinda snarky and borderline facetious but man, I can't help but agree. After 10 years hanging around this site if there's anything I've learned it's that I am nowhere fucking near "max geekiness". I couldn't stick with a project like this for 6 hours, let alone 6 years. And yet literally in the last week I've been (gently) ribbed for being nerdy enough to think reading books for pleasure is an acceptable leisure activity.
This is orders of magnitude of geekiness beyond anything I could possibly accomplish, and I mean that in the best possible way. Kudos to the author and thank god for the real nerds.
> This is crazy, how on earth do you have the patience and persistence to work on something like this??? It took five - six years?
Immediate gratification culture (a generational problem?) teaches us that the very short term is the meaningful range in which to work and expect results. But anyone who has become a master at their art or craft knows that 10+ year projects are normal and that the long-term is where mastery resides. In the medium term, one is often on a plateau, barely seeing significant progress. (Just ask the best cellists or figurative fine art painters or marble sculptors.)
There is absolutely no way that immediate gratification culture is a generational problem. I've worked my entire adult life in R&D for medical diagnostic devices and in those forty years nearly all the bad decisions I've been subjected to had quarterly stock results as the eventual root cause (if you dug long enough).
Calling it a generational problem is like blaming Millennials for the participation prizes that they got as kids which the adults invented, created, and awarded.
You just like it, and you keep working on it as time permits and you get unblocked. Which reminds me, I need to write up my own 4-year project (which is nowhere near as cool as this).
Amazing work, clearly a labor of love. You never know how standardized of an approach embedded developers take depending on the constraints on developers and hardware, I’m glad he lucked out with a lot of the binary resources. I’ve bookmarked this, it’s such a delightful resource.
This is part of the reason I can't get in to reverse engineering. It takes an incredible amount of time to get a little done when in the same time you probably could have written your own camera firmware. I do remember seeing an open source camera but it was a really high end one and very expensive.
I really like what they are doing, but they have been working on it for 4 years and have no users. The camera costs 6000 euros and has no internal recording capability.
There exists (or DID exist, before Google snatched up Marc Levoy and made the Pixel cams) open source cameras.
Levoy called the academic exercise he pursued the Frankencamera. (Look it up, a big thing at Stanford). Nokia took him up on it and produced an OTC "smartphone" ... before the iPhone.
i still have a N900 and it works: replaceable batteries do wonders!-) Runs unix etc. Open source camera software.
I remember this, very cool. His team at Google is really at the forefront of computational photography (HDR+, Halide, lightfield photography). Kari Pulli from Nokia also did some very cool stuff on array cameras/computational photography at Light since then.
I think it would actually make sense to build an open source camera on Android. The android camera HAL and camera2 APIs are pretty good (though limited). You get all the UI stuff for free, and a large dev ecosystem. Halide is great for very fast processing. We used them both for the Ubuntu Touch camera app. Though if you really want to push what's possible, access the more low level features would need to be open source too.
Nokia (or HMD Global) are back in the camera phone game again btw - 5 lens android camera coming out this year according to rumours. I think this type of multi-sensor camera really is the way forward on mobile phones. You can keep the lenses thin with much larger sensor size when combined. I really wish they would open source at least some parts of it, but unlikely
Proprietary BLOB's bound the openness of any digital camera. Sensor and chip manufacturing is proprietary and patent encumbered by default. For business agility and cost if for no other reason.
Open source cameras exist. They use chemical processing and have manual controls. The documentation is good and it is even practical to build one's own gear...indeed there is a long tradition. Hacking a Leica film camera just wouldn't be as newsworthy.
In some sense, the low returns smell like an XY problem. To a first approximation, reprogramming a Leica is not going to improve it as a photographic tool. The limitation is the going to be the photographer not the software. For software, there's always post.
The interesting thing about "open-source cameras" in the sense of film is, the film chemistry and film coating methods are very much trade secrets.
Making up developer from base chemicals (metol, phenidone, hydroquinone, and so on) isn't impossible and some people do it so they can "tweak" the process a little bit.
The one thing I've never heard of anyone doing is making up a photographic emulsion from scratch at hobbyist level and coating a film base with it.
I don't disagree. Commercial film stocks tend to be proprietary analogs to BLOB's. Though, for current film stocks, tooling (IDE analogs) for developing those film stocks are available (e.g. C41 and E6). Film stocks requiring proprietary development processes are gone from the market.
There are also well documented photo-chemical substrates that are relatively easy for a radically open source photographer to hack on, e.g. wet plate collodian and the more film like dry plate silver gelatin processes. Not all that practical adapt to a 35mm film camera. But analogously open source jumps through hoops to run on an iPhone.
Oh yes. I've got an A7S2 and while I'm pretty satisfied with it in general (especially after jailbreaking it and killing that annoying 29 minute video limit), there are some things that I really miss, mostly related to networking.
Oh and while one can get a full service manual including block diagrams and wiring specs for it on the Internet, if someone knows how to access its bootloader (probably uboot, it runs Linux in any way, with an Android layer on top for the "apps") so that one can experiment without risking a $2000 brick, that would be really really nice ;)
"My wife and I always wanted a Leica camera and suddenly we realized that if we didn’t buy it now, we will not be able to for a while."
Is this the mindset of "better buy the expensive thing before we can't afford it"? I've seen this expressed before and it's not sound reasoning. If you wouldn't be able to keep that money in a savings account because you know you'd later find other uses of the money to provide better value then you shouldn't be comfortable spending it on a depreciating asset immediately. The only way it makes sense is if you trust your future self's reasoning less than you trust your present self's.
It reminds me of something from Robert Kiyosaki's book "Rich dad, poor dad" (I'm surprised I'd quote that) that basically says that the things you need to do or pay will be done or paid, while the ones that not necessarily will be done or paid might not.
Therefore do the unnecessary ones first because the necessary ones will be done anyways. This way you end up with e.g. both leisure time and work done instead of just work done but no leisure time.
I don’t think that this is about not having the money at a later stage, but about spending that kind of money when you have other responsibilities (kids) being irresponsible.
Right, but it's not like money has an expiration date on it.
If it's going to be irresponsible then, it's probably just as irresponsible now, perhaps not as obviously. ("Better hurry up and make that mistake...")
I bought a $5000 Nikon lens because I learned that there would be a price hike of $900.
Leica equipment gets constant price hikes. The cameras and lenses are great but at this point they are mainly for rich amateurs rather than working professionals.
I also bought my house in 2002 because I had looked at houses in 2000 and 2001 and decided to continue renting. I bought in 2002 because I thought that if I didn't buy now that I would get less value for my money later.
Unfortunately many other vendors encrypts firmware updates, like Panasonic do. I dream to hack Panasonic GH5 and G80 cameras firmware to tune some options.
There is a tool to binary patching Pansonic GH4 and GH2 firmware called Ptool. https://www.personal-view.com/faqs/ptool/ptool-faq
It decrypts encrypted firmware, patch binary and then encrypt again. So you can update firmware via default process.
Honestly speaking, encryption never stopped me. Doesn't matter if it is something simple like M9 xor described in this story or fancy AES/HMAC from Canon or Sony.
It just makes reversing more complicated and invasive. You have to open body, rip off flash or solder wires somewhere to sniff comms. Unlike Apple, camera vendors are not that paranoid and try to keep firmware unencrypted on flash to reduce the boot time. In other words this is the part when your wife can kill you for bricking the camera :)
I couldn't agree more! Even tho I understood the point, a lot of stuff was so foreign but the aricle just drives you and it needs some love not just on techinal part, but also on the writting!
One thing that I learned about reverse engineering is that you can often get very far by just recognizing/guessing what formats and libraries were used by original authors. The article seems to at least partialy confirm this view.
On vocational school we could elect to do long term project instead of practical graduation exam. In my case this involved reverse engineering management protovol used by Merlin Legend PBX in order to port its DOS-based configuration utility (which was in fact emulator of MLX-20L operator phone) to something more modern (and multi-user). One of first things we did was running the binary through strings and ndisasm (I probably still have the hackedup tool to convert MZ EXE to pseudo-COM that could then be read by ndisasm, which was motivated by fact that for various reasons we could not use IDA). What we found out was that it used some weird Unix on DOS emulation layer from AT&T which included Unix-style ncurses and terminal emulation layer.
We tried both to analyze the binary and sniff the communication. At first we thought that disassembling the code would be faster as we had only limited access to the PBX itself and were somehow afraid of bricking the thing. Oneday I just gave up and spent few hours hacking up a way to actually look at the UART data (there were two issues with that: the PBX was somewhat picky about accepted RS232 levels and then slight logistical issue of having preferrably a laptop with two serial ports in early 00's). After we had this ugly mess of wires with four DE9 connectors and active RS232 buffer (powered from adjustable bench supply, needless to say that our advisor was not too thrilled that we decided to connect this thing between somewhat irreplacable PC and still considerably expensive PBX) we found out quite quickly that the actual configuration protocol consisted of XModem for backup/restore, straight ANSI terminal emulation for initial session establishment (and in theory for weird "use PBX as outgoing modem" feature), essentially binary block oriented terminal protocol (think contents of PC text mode framebuffer with one attribute byte for not every character, but block of 8, always sent as whole line) wrapped in weird HDLC subset for the actual interactive configuration and weird handshake reminiscent of OBDII serial protocol to switch between these modes (which probably took the majority of time to reverse engineer).
Interesting aside is that the above mentioned binary block protocol was also used for the UI of almost-ISDN phones that went with the PBX in question. We had access to ISDN protocol analyzer which worked perfectly for normal call flows, but reliably crashed (and not with any kind of meaningful error message, it just overwrote half of its display with random pixels, started ignoring its keypad or otherwise started behaving weirdly in somewhat random manner) any time we did anything more complex. Somehow I think that finding signal that reliably crashes firmware on test equipment which is explicitly designed to debug problems on such interface is achievement in itself :)
A little of topic but the reference as a huge list at the bottom is forgivable for print media, but for interactive media, there is no reason for such inefficient method that requires one to click on the reference only to find yourself skimming through a list trying to remember what the number was and once you have found it, great, now you have to find your way back to where you were reading!
Wikipedia's recent reference links are a great of example of how to do reference right.
In Safari on iOS you can use the back button of the browser to jump back to where you were on the page before you clicked the link to the reference. You should see if the browser you are using does that as well.
Absolutely :) Named anchors in a page have been around for a long time and I recall that going back from a page internal link would bring you to the place in the page that you were for as long as I can remember.
Would be interesting to know which browser was the first to keep track of scroll position for history entries.
Mosaic was before my time but I wouldn’t be completely surprised to learn that even it would store scroll position.
As a former chdk user on canon and a Fuji x20 owner I hope somebody carries on your amazing work!