Hacker News new | past | comments | ask | show | jobs | submit login
Tractor Hacking: Documentary About Farmers Fighting for the Right to Repair (vice.com)
352 points by goodmachine on Feb 15, 2018 | hide | past | favorite | 115 comments



I own a small farm and work in tech. We are struggling with this issue right now as we look at the options for a new tractor vs an older serviceable model. While many of the larger corporate farms are content to buy new equipment every 10 years, those of us at the small end of the scale have begun a push to keep older equipment up and running.

My tractor is from 1996, my hay baler from the early 70s and my hay cutter is even older. These all work just fine even if a little slow. The challenge of not being able to work on my equipment currently outweighs the marginal speed gains I would see on a farm our size. The same goes for nearly all of my neighbors.

The last "new" tractor in the local area was bought a couple years ago by one of the older guys who was buying the last tractor he will ever own and he thought it would be nice to have a top of the line, new, tractor just once.


Yup, that said I think there's enough competition in the < 75hp space that if it really was something that consumers shop against you'd see brands trend that way.

FWIW we've got an '81 ford for similar reasons (also new tractors are effin expensive and hold their value very well).


What do you grow? Hay mainly? Acreage?


We grow hay to feed our beef cattle on about 100 acres +/- a bit. But all around us are plenty of farms growing small farm (under 500 acres) or row crops/beans etc.

We were having this very discussion today when picking up some hay to help us get through till spring with a guy that cuts about 500 acres of hay 3 times a year. He just keeps rebuilding his equipment rather than get sucked into the new stuff he can't work on.

This is becoming another huge issue that affects smaller farms more than it affects larger corporate type farms. When a piece of equipment goes down it's needed back in service in hours to days, not the potential weeks that hauling it to a repair depot can cause.


You weren't kidding about the small farm! The dealers in your area can't do field repairs?


I work in the embedded field and this drives me nuts. I personally feel that companies should be required to go one step further and be compelled to release their embedded keys and source code for any product that they no longer provide complete support for.

The embedded field is strange because the lock-in is generally on the side of the hardware design and tooling. Nobody is going to try to clone a tractor because they have the source code for it. They also need the CAD drawings, machine paths, tooling, supply chain, etc... Yet just about every company in the space feels that their code is absolutely sacred above all else. Just about everything you own running embedded software that has some update functionality has a bootloader containing cryptographic keys such that the firmware patches/images cannot even be disassembled. For a few devices like routers, some toys, a couple handheld radios and the like, people have been able to modify them through exploits that cause firmware dumps from the uC's memory but otherwise it's typically prohibitively time intensive to rewrite new firmware from scratch without having access to the device schematics (and even then). Worse, just about all these devices have some access to JTAG ports such that you could easily program them yourself if you wanted to and had something useful to flash them with. I've run into so many products I use on a regular basis with bugs or simple but badly needed features that I could have easily fixed/hacked in with access to the source code but alas, I cannot.


"Just about everything you own running embedded software that has some update functionality..."

"Worse, just about all these devices have some access to JTAG ports such that you could easily program them yourself if you wanted to and had something useful to flash them with. I've run into so many products I use on a regular basis with bugs or simple but badly needed features that I could have easily fixed/hacked in with access to the source code but alas, I cannot."

IMO, it is a massive waste. Short-sighted, short-term thinking.

There are people who for whatever reason do not want you to have access to the software ("firmware") and to read/fix/improve upon it.

Some of these people comment on HN. They become active during debates over whether users would derive any benefit from being able to read and edit source code.


The problem seems to be that software is viewed as a trade secret.

Big old companies have CEOs and boards that see embedded software in the same way many legislators and judges see it. Firstly they don't really understand it. But in any case it's part of the product and people have no reason to tinker with it. Now software is being used to enforce what manufacturers could previously have only dreamed of: monopolizing repairs.

With hindsight it seems obvious that things converged onto the embedded computers of consumer products: think microwaves and TVs rather than general purpose computers. The problem with tractors is that all the farmers I know have a much better idea of how a tractor works and how to fix it than the average motorist does about their vehicle.

I am reminded of the era when windscreen-mounted GPS navigation systems started becoming mainstream. Most of the models had very similar hardware but the software was all that differed. I never looked into it but I suspect almost zero had source code published.

To the more typical consumer, the unit is as a whole and the software comes with it. If it routinely takes you the wrong way down a one way street or tries to take you down a set of steps, it's defective and needs to be replaced.

To the more technical consumer it's clear that either the software or the data is at fault and either could be fixed with the right resources. I still feel a sadness at this, yet this waste is insignificant compared to a tractor or any other vehicle.


>The problem seems to be that software is viewed as a trade secret.

Because it is? I mean a lot of the automotive embedded industry is moving toward separation of software from hardware (AUTOSAR Runtime Environment), which is not working as well as expected, because embedded and separating software is a bit tricky. But the goal is to have more suppliers for software than for the hardware. The only trade secret these companies will hold is the software IP.

I think farmers should be able to fix their own equipment, but software being a trade secret is a fact, not something only big old stuffy companies see that way.

The other thing is there are many regulations to comply with. Companies don't want to get sued for some hacks by customers.


The only trade secret these companies will hold is the software IP.

Lots of companies in other industries prosper in that situation, without locking everything down. RHT, for instance, has performed really well over the last year. These software subs could likewise go to a service model or literally anything other than "our software must be installed in crippled fashion". It's not as though e.g. Honda is going to decide to "pirate" their shit.


Not only a trade secret, but also a liability. If source code is disclosed, companies can be accused of copyright violation when some code is similar to an older protected program.


There's another angle - vendors also make money from support contracts, and fear that if they gave you the ability to get support from anyone (even yourself!) they'd lose that revenue stream. For more consumer-oriented products, manufacturers don't want to deal with supporting products that have been modified... (well, they don't want to support them unmodified either, often as not).


I believe the change should start from consumers, especially big institutions. For example, a big municipality or government agency may impose limits on buying new software/hardware that isn't open to refactor/repair by their competitors, to avoid lock-in to a single supplier, operate on more open market, and consequently drive the costs down.


I like the idea. I'm worried that this ship has sailed though as it's becoming harder to even get phones without cryptographically verified bootloaders, and god forbid you want to mess with the baseband/modem source. This is only going to get worse too, as software becomes the part that's actually valuable. AI will mean that everything on the processor will be a trade secret x1000 and everything that can possibly be kept from the local processor will be hidden away in the cloud, forever dependent on continued server access.


>I personally feel that companies should be required to go one step further and be compelled to release their embedded keys and source code for any product that they no longer provide complete support for.

The embedded team where I work struggles with the same thing. Frankly this should be included with the first delivery of any industrial system. Under NDA if they absolutely must, though I don't like it.

I also don't like consumer systems being locked up, but at least it's somewhat defensible. But treating farmers as dumb, inbred "hold my beer and watch this firmware hack" hillbillies and not the educated industry professionals that they are is beyond condescending, disrespectful and is completely indefensible.

Exactly who is deemed qualified to hack farm equipment firmware is something for the farming trade associations and their insurance companies to work out, but as far as John Deere and the regulators are concerned, we're talking about professional embedded developers writing code to run on their equipment to support an industrial process, something that is completely uncontroversial in many other fields. That you're not supposed to intentionally or inadvertently convert your combine into a man-eating killbot is surely already covered by insurance policy clauses, labour laws and public safety laws.


I completely agree with you, but I am a little sympathetic to the "bad PR" angle. It's really, really not that hard to imagine a scenario where a well-meaning farmer (or their contractor) loads something onto the embedded processor that does cause a malfunction, and before you know it there's a splashy news story featuring pictures of a big green tractor not living up to its brand.

I work for a robotics company making indoor self-driving vehicles, and we worry about exactly this issue— from an ROI perspective, our product is a slam dunk, so it's safety and reliability are the first and second most significant questions which have to be answered during the sales process. When we invest substantially in associating our brand with those values, we really can't afford the possibility that an unauthorized user modification compromises them in a way that's outside of our control. Hence closed source, FDE, and all the rest of it.


That is a very specific scenario, but agreements can be made to release software in exchange of loss of warranty and liability protection: I give you the code and from that moment (either if you use/modify it or not) you lose all warranties and can't sue me even if the product burns your house and kills all your kittens.


Losing the warranty is a given once it's been modified.

And obviously before electronic control, there were still a hundred ways you could break any given device and have it generate undeserved bad press for the manufacturer.

But I contend that the scope of things that can go wrong when you mess with the software, and the degree of damage that can be done, are both worse than with purely mechanical modifications. There's huge potential for unintended consequences, and bad press puts the original manufacturer in the awkward position of having to disavow their own product, bearing their symbol, because "the user modified it in an unsupported manner."

All this gets even worse if the manufacturer actively supplies source code and an SDK for making such modifications. How do you delineate supported vs. unsupported mods?


you can't release all liability.


I personally feel that companies should be required to go one step further and be compelled to release their embedded keys and source code for any product that they no longer provide complete support for.

Sounds great, but it's something the abandonware[1] scene has been fighting for over the past 20 years, to no avail.

1. https://en.wikipedia.org/wiki/Abandonware


It's a good idea, so people should keep fighting for it, even if it takes more than 20 years.


I think the difference here is that when it comes to embedded software, very expensive capital can be tied up, as is the case in the above video. Not many commercially available software packages cost >$10k and very few people are exposed to that risk. Meanwhile, there are hundreds of millions of cars in the US.


It'll probably take one generation to raise the issue (and find an example for enough people to care), another to debate, and finally a third to agree about the common sense solution.

So give it another 40 years.


"Nobody is going to try to clone a tractor because they have the source code for it."

Thus is the funniest thing I've read in ages. It's also 100% correct and this whole situation shows perhaps clearer than many, how Stallman's warnings were accurate.


While that is true, i think the manufacturers are more afraid of competitors, which could in some way "clone" at least the software.


I'm doubtful that DRM in any space has ever been about competitors.


> Nobody is going to try to clone a tractor because they have the source code for it.

In the automotive industry, people and companies will do this (at the component level). Enabling features via that the OEM has disabled due to its marked segmentation strategy via aftermarket components is done (and would be done to a greater extend if OEMs did not protect the part of their diagnostic stack that handles feature activation).

For example, XCP or UDS security access modules are sometimes distributed by the Tier 1 as a DLL so that even the OEM has no source code for the concrete authentication mechanism that is used.


Not to defend Deere et al., but one argument I've seen bandied about is that the manufacturers consider their software to be their "secret sauce" value add. If machined steel were all there is, there's nothing preventing the Chinese (or whoever happens to be the current outsourcing bogeyman) from producing perfectly good equipment at a fraction of the price. Personally I'm not convinced this is a really sustainable "moat"; surely the Chinese (boo!) can do embedded programming as well.


That's why a lot of people are advocating for a fair law that forces companies to release sources after a reasonable amount of time or when they cease to support the product, so that they can profit for some years from their IP but users aren't screwed when the product becomes obsolete. And of course being repairable and/or upgradeable means that a piece of technology won't be thrown away.


> embedded field [...] companies should be [...] compelled to release their embedded keys and source code

awesome! my botnet malware that exploits and roots your smart meter or set top box can now install new firmware with a permanent backdoor and nothing will complain because all the malware binaries are signed - by the manufacturer's key, no less - so they must be secure, right?


It would be beyond retarded to use the exact same key for every unit. That's not what people are asking for.

People are asking for keys to their exact units so they and only they can sign software for them.


While GP's snark is uncalled for, he is correct as that's exactly how this works. What you're asking for is like saying that a web server would provide a unique HTTPS cert to every distinct visitor.


>What you're asking for is like saying that a web server would provide a unique HTTPS cert to every distinct visitor.

It's not similar at all. The equivalent would not be the certificate but the session key, and you do get your own session key in order to prevent what the person above is describing. My HN session key is useless for decrypting your HN password even if I could intercept the traffic.

There is no technical reason why the devices can't ship with not only the manufacturer's public key, but also a key pair generated for each unit that comes off the assembly line for the customers to use to sign their own firmware images (and if they wish, delete the manufacturer's public key). But they will never be able to sign images for any device but their own because they simply don't have any signing keys that can produce a signature that will be accepted by any device but their own.


As a software engineer with an embedded systems background this kind of OEM behavior driver me up the wall.

I've had OEM's:

- refusing to provide interface specs of a device of which we just ordered hundreds of units. (But hey, here is the binary windows (!) driver for this embedded device)

- refusing to provide the calculation method of a checksum value their device returned.

- Stopping software development (tooling, firmware, drivers) on embedded products, even when there are known issues.

- Dropping support on a device, even when hundreds of them are still being used in the field.

It's not just tractors, its everywhere.

From a business standpoint, it's just so dangerous to work with hardware vendors who don't offer implementation details or source code. If they drop support or stop existing, you're screwed...


As a former farmer (roughly 4000 acres of dryland farming in Canada) and an engineer who also did embedded control system development, I would like to add some of my concerns.

For new ag machinery, embedded control systems are absolutely essential for the machine to do its job. The ECU on the engine is generally reliable and have long life. The other stuff, I'm highly dubious of the long-term maintainability of it. On our farm, we ran a lot of older equipment and did a lot of repairs ourselves. For new equipment, that will be near impossible as the electronic control systems are black boxes. Even the cabling is complicated and is generally not documented. After the machine gets a decade or two old, good luck fixing it.

I suppose you could just drive the machine in the junk pile. However, when a new combine harvester is costing around 3/4 of a million USD, that seems a bit wasteful. I don't know what the solution is though. The farmers buying new equipment don't care so much, their resale value is not suffering too badly. The manufacturers are looking for ways to sell new equipment and adding more features via electronic controls is a relatively cheap way to enhance the product.

Most farmers say they don't want all these new electronics on their machines. When Deere announced their S700 series combines, they had a video bragging about all the new improvements:

https://youtu.be/l5xJOoNyLoU

If you study it, you will notice that most of the new features are due to software changes or to electronic control systems. Little of what the machine actually does to harvest the grain has changed (metal parts, etc). This automation features are nice when they work but are a disaster when something goes wrong. The operator will have trouble to determine what the machine is doing (never mind trying to figure out how to fix it).

However, even though many farmers say they don't want it, the new machines are selling well. So, maybe you could argue the market is working. To me, it feels like there could be some externalization of costs going on. These new machines are very much less valuable as they get older vs the previous era of farm equipment. I guess the same thing has happened to automobiles. You can take a car from the 1940s and fix it up into perfect condition. If you take a 2017 car loaded with electronics, would you have any hope of fixing it to new condition in 50 years from now?


My family doesn't farm but we used to and we still own a business where we use these tractors. The older tractors are much better for us because they're simple and quickly debuggable. I can get our Oliver Rowcrop 77 back up and running in no time almost disregarding what's failed, but the machines with electrical systems period are harder to debug and fix. This has been getting progressively worse as these machines get more sophisticated. Especially now because of the the addition of closed software these tractors aren't going to be around in 10-15 years and it wouldn't surprise me if it gets to the point where the tractor OEMs will start telling farmers just to get a new machine, which is perfectly viable for the huge farms. They just don't seem to give a shit anymore about little farms/businesses or long term reliability.


And it's not just things become unrepairable. Some vehicles, like my mom's previous vehicle, could be repaired by the owner. The problem was that any "serious" repair would require you to disassemble a surprisingly big chunk of the car. For example, every time my mom sent the car to the shop, the dashboard was completely removed. To fix her moonroof, yep, the dash had to be removed. As well as most of the roof.

On top of all that, the manual outright lied about features. It stated that oil could be checked by holding down a button on the instrument cluster, but after trying it then trying all the other buttons, none of them actually did anything outside their normal use (e.g. swapping between total and trip odo). There was no way for the owner to check the oil in this car. Additionally, it is heavily implied that the owner can't even change the oil themselves, though that could be done (with relative difficulty -- the "correct" way was with one of those expensive oil vacuum systems that no one actually owns).

The manual also described Bluetooth features that the car did not have, with no disclaimer like, "This feature may not be available in all models." It also had all the buttons for it and generally looked like it has Bluetooth, but none of the buttons actually did anything.

It's really no wonder she traded her BMW for a Camry.


It took ~30yr for good, fairly cost effective aftermarket, fairly generalized ECUs and transmission controllers for automotive applications to come about.

The same will probably happen to tractors and whatnot. Since the cost side of the equation is far different for business and the problem has already been solved in the automotive space so it may happen quicker once it starts happening.


I do embedded work. Making an open-source, aftermarket electronics for tractors, with standard interfaces.. that sounds fun.


You might like tinkering with something like the Macchina[1].

https://www.macchina.cc/


Couldn't agree more. The industry is so stuck in various dogmas that all of this is totally acceptable in the pursuit of securing source code from presumably China and anyone with a screwdriver.


Right now we have a client that needs an integration, but the company that wrote the software to be integrated will not provide documentation for integration unless we purchase their (otherwise useless to us) software. That is, software that would be tested at the client's location, with their data. In short, it is absolutely not something we need to purchase (or, honestly, even can -- we're a very small company).

So we eventually tracked down someone who had it, and we integrated as best we can.

Basically they were just putting up a wall to people creating integrations, which is the entire method by which their software is used. So... if you can't contract someone to integrate, why would anyone choose that product going forward?

And yet they do.


Are there companies that will do reverse engineering as a service for cases like yours?


No, because they will be sued into oblivion.

Also, you will be sued if you hire one of those companies, as you will be breaking the many NDA's you had to sign to be allowed to use the hardware in the first place.

I've seen this happen, more than once.


I'm not sure, but I reverse engineered quite a few embedded devices for how they communicate. I even wrote a tool to detect which message is where (depending on observable data). And other tools which detect checksums from a host of well know algorithms.


Companies generally don't want you in the software for safety reasons. While true there is intellectual property around the guidance and swath generating algorithms, companies hate lawsuits. Lawsuits tie up resources, can cause stop shipments, generally lost profits, and tarnish the brand. Look at the Chrysler Jeep news...

You hack or change software on a vehicle and cause the auto-guidance to run over and kill someone or you circumvent a safety check in software unbeknownst to you due to your software changes and a spinny sharp thing that should have stopped doesn't and hurts or mames you or calibration changes cause engine to to over-heat and burn up and generally the company is held responsible for it.

As much time is spent testing software as is writing software. All our testing is there to make sure the software works as expected. Millions are spent on test equipment and millions of hours and man hours are spent making sure software works as expected.

I happen to work for a major agricultural company and I'm a lead software architect for their guidance + autonomous vehicles + precision farming embedded devices. We are one of John Deere's major competitors.


This is indeed one of the specific instantiations of the generic single-actor control-delusion that has always been with us, but has been greatly exacerbated by the rise of software. And I've no doubt that it's enough to keep the engineers' cognitive dissonance stoked - "It is difficult to get a man to understand something, when his salary depends upon his not understanding it" plus a bit of ego stroking from the implication that being able to work on this particular code is really special.

But it's utterly fallacious. It's quite clear that if certain changes result in different behavior, then whomever made those specific changes is responsible for the resulting behavior. About the only leg the argument can stand on is if the code is so sloppy that someone attempting to make a simple change tickles a bug somewhere else, but enabling such lack of quality to persist is the exact wrong approach for a safety-critical system!

Furthermore, regardless of its actual merit, this is precisely the kind of Schelling point collusion that is in the collective interest to bust up. "Selling" someone a piece of capital equipment that is deliberately designed to prevent its own servicing is plainly fraudulent.


Except that tractors are more dangerous than your car/appliance/whatever.

You'd be amazed the amount of carnage running a brushhog at 1000rpm will do when it was designed at 540rpm. They will happily throw the 3-4 foot blade right though 1/2in steel up to 300-400ft in whatever direction the wind takes them.

When that process is controlled by software all it takes is flipping the wrong high order bit in the PTO controller. On a manual tractor it's very clear when the PTO lever is in 1000rpm vs 540rpm(and usually on a separate lever from engagement) vs an opaque software stack.


Safety is exactly one of the reasons why you'd want the complexity of software laid bare out in the open, rather than hidden as a black box of unknown size! Your own comment laments the clarity of two entire levers to control PTO versus an "opaque software stack" - why is it a forgone conclusion that that stack must be opaque?!

To address another objection of yours about unknown modifications, this too is an aspect where openness buttresses safety. Rather than blindly trusting that the software has not been modified (because such methods are unknown to you), the proper way to verify integrity is by being able to compare checksums or reflash, knowing there is no hidden state.


"Safety is exactly one of the reasons why you'd want the complexity of software laid bare out in the open"

No, disagree. If I laid out our source code for you, there is no way you would have any way to understand it without all the internal documentation from which the software was architected and developed from. Non computer people always have this misconception that if they see the code, they see everything it's doing. To a degree you can see what code is doing but understanding why certain logic exists is not always clear without supporting documentations.

Additionally, some of our code is 3rd party licensed and that adds an entire new wrinkle to things.

For an over simplified example, our CAN buses have thousands of signals. A single CAN frame may have 20 to 30 internal signals and several signals across many CAN buses are examined to make decisions and I've only scratched the surface here. There is no amount of code study you can do to understand this without some coaching or review of associated internal specifications.

For me or any company to support someone's quest to understand the code would be a very costly endeavor that you're simply not entitled too or nor would it be a reasonable request.


> If I laid out our source code for you, there is no way you would have any way to understand it without all the internal documentation from which the software was architected and developed from. Non computer people always have this misconception that if they see the code, they see everything it's doing

This isn't a forum for "non computer people" - I've been a professional embedded developer, hardware and software.

This stance is wrong and terrifying. The code, safety critical code especially, should be clear and should itself document the quirks of whatever it interacts with. Embedded has a culture of punting on abstractions due to the pervasiveness of cross cutting and global concerns (peripheral allocation, interrupt usage, tight latencies, etc), coupled with traditionally smaller code sizes. But we're no longer talking about devices with little memory or cycle by cycle timing, but 32-bit devices with dynamic allocation performing high level logical processing. The embedded culture of fiddling with stateful hacks, verification primarily through system-level testing, and then zipping up the source tree for revision control needs to be burnt at the stake. Leave the bit banging to single-purpose 8 bit micros at the edge that cannot hide actual complexity - the core is a full computer, and needs a proper type system.

I'm not saying this change is going to happen over night, but we should be aiming for the correct approach. Every time a company says that they cannot release code because it is too complex, wouldn't be understandable, have third party dependencies, etc - what they're really saying is that their code base is a sloppy hack, they have no idea how it works, and they really do not want to own up to this state of affairs.

> For me or any company to support someone's quest to understand the code would be a very costly endeavor that you're simply not entitled too or nor would it be a reasonable request.

It might not be desirable according to a company's business interests, but what is actually unreasonable is to force purchasers and users of heavy equipment to blindly trust the software controlling those devices.


The non-computer people comment refers to the folks in the video; not this forum.

You also seem to be conflating right to repair vs. right to inspect code because you feel you have the right to do so for some reason (safety, don't like how it works, etc...). You don't have that right just like you don't have the right to walk into an assembly plant to see how your vehicle is assembled.

"The code, safety critical code especially, should be clear and should itself document the quirks of whatever it interacts with"

Agree and it does and we are audited by a 3rd party trained to looked at code written according to functional safety guidelines (ISO26262). Companies, like mine, don't have resources to respond to folks untrained on such matters. And for the non-safety aspects of the code, that's our intellectual property and our competitive advantage.

"Embedded has a culture of punting on abstractions due to the pervasiveness of cross cutting and global concerns"

You have data or proof of this or are you just talking from your personal experience? I assure you we take safety very seriously. Multiple millions are spent, as I said above, on testing to make sure code performs as expected; especially when a 3rd party audit and billions are on the line and, more importantly, the safety of our customers. You're inspection of the same code simply can't compete with the level of testing & auditing. You're really kidding yourself if you think otherwise.

"but 32-bit devices with dynamic allocation performing high level logical processing" Certain ASIL ratings dictate thou shall not perform dynamic allocations. You don't appear to be up to speed on functional safety matters.

"unreasonable is to force purchasers and users of heavy equipment to blindly trust the software controlling those devices."

Not true, disagree. It's unreasonable to assume we don't take safety seriously and demand to see our code when you can look at any company's functional safety audit records (they are public). To presume a right to change the code because you're not happy with some aspect of some functionality or you're concerned about safety is not your right and no company has the resources to support these requests from the general customer or public base. Moreover, as I already pointed out, it's our IP.

IP type code in this field is heavily guarded. Swath generation, swath acquisition, path planning are all very interesting algorithms that represent a company's IP.


What is your point? I have a relative that lost his arm to a combine. There was no lawsuit, he did not seek damages. Farmers are well aware of how dangerous this equipment is.


The point is I don't think allowing the modification of software in what is essentially a factory on wheels is a safe thing to do.

When we bought our '81 Ford there was a ton of bodge jobs and half-assed repairs(that helped us talk down the price) that I could easily verify by visual inspection. I knew because I could see them that they weren't inherently dangerous. Software doesn't have the same analogy in this space.


> The point is I don't think allowing the modification of software in what is essentially a factory on wheels is a safe thing to do.

Someone has to do it. What makes John Deere's embedded software engineers more qualified to write software for a factory on wheels than the embedded software engineers working directly for the farms?

Especially when we can't even check the former group's work.


"Someone has to do it. What makes John Deere's embedded software engineers more qualified..."

Simple, John Deere, probably like us, has spent more testing the software than you could ever do on your own.

You can't judge complex software just by looking at it. It must be tested. People who fail to understand this wind up making these uneducated comments. Go study ISO26262 and the MTB calculations required to pass it and maybe you'll change your tune.

Tested software runs on real vehicles and simulators at large companies, as I stated, running into the multi-million dollar price range. There is zero amount of code inspection the general public can do that would give the same assurances you get through testing.

Bottom line: just looking at code is not enough. You must test it. This is unit tests, simulations, field tests, harsh temperature, vibration, power cycle, high side/low side power distribution, code coverage, etc... None of this is possible by you inspecting code.


If this was a consumer product, sure. Tractors aren't consumer products and come with manuals for a reason.

You can't both try and chase the premium for a professional product, while locking down the professionals using it.


In such a scenario, if you were worried that the software was modified, you could flash the ROM with the OEMs software.


not if the modified code also modified the boot loader.


> I knew because I could see them that they weren't inherently dangerous. Software doesn't have the same analogy in this space.

This has a lot more to do with manufacturers unethical approach to closed and obfuscated opaque software than any inherent problem or limitation of the field.

You were able to recognize the problems or lack thereof for your car because a) you could actually see it, and b) you have more than passing familiarity with how cars work. There's no reason both of those can't be true for the vehicle firmware as well.


perhaps, but if you sold the equipment without telling the new owner they were running your code and they got hurt or killed, as I said above, companies don't want tie up their resources fighting these claims or dealing with it in the first place.


there is the doctrine of an 'attractive nuisance' [0] though, which gives precedent for making owners liable for damages caused to third parties by some dangerous object they did not sufficiently prevent access to... that easily hackable by the user marketing checkbox, combined with a sharp and pointy heavy agricultural spinning tool of death might count?

0. https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine


That seems to apply to children.... I mean you should add some safeguards, to prevent the code from being modified by children messing around, but I dont think this would apply to an adult farmer trying to modify his tractor.


Very well said, but I still like the poignant - "All professions are conspiracies against the laity” (George Bernard-Shaw), which is likely based on Adam Smith's earlier writing: "People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices."


>But it's utterly fallacious. It's quite clear that if certain changes result in different behavior, then whomever made those specific changes is responsible for the resulting behavior. About the only leg the argument can stand on is if the code is so sloppy that someone attempting to make a simple change tickles a bug somewhere else, but enabling such lack of quality to persist is the exact wrong approach for a safety-critical system!

If there weren't rules that you have to protect your software from manipulation. But allowing manipulation leads to lawsuits alleging that the manufacturer is still at fault for any manipulations he allowed. So they don't allow any.


Are there any examples happening of this happening in recent years? (80's-now)


" It's quite clear that if certain changes result in different behavior, then whomever made those specific changes is responsible for the resulting behavior."

Umm, well people lie. Or you make the change for yourself and your fellow farmer. Your fellow farmer is killed and his wife sues the tractor company. Do you need more examples or is it clear now?

This is just my opinion. Customers are not entitled to source code or what I call the work product. You're entitled to the end product you purchased, the binary, which the source code is compiled to produce.


> Umm, well people lie

Yes, like the companies that turn their products into black boxes, and rely on tricked human instincts to write off the magnitude of what the "tiny chip" contains.

It's so weird when the spectre of dishonesty is used to justify more secrecy, when sunlight is the best disinfectant!

Does the checksum of the software match an official one that company has released (and not given notice of deprecation)? If not, then how did the image get into this state? That's a clear indicator of responsibility, but requires a paradigm that reifies the fact that software can be changed.

> Your fellow farmer is killed and his wife sues the tractor company

Anybody can sue anybody at any time, regardless of merit. Bringing up this one possibility is fallacious.

> You're entitled to the end product you purchased, the binary

And yet companies constantly refrain that the end user hasn't purchased the binary, but only a license. Needless to say, this is all based on commercial laws (as opposed to manifest natural laws), and are thus straightforward to change when they're no longer sensible. As far as I'm concerned, copyright should require registering the full source code - the output of a compiler is not a novel creative work!

The product the customer purchased is a tractor, part of its implementation being software. As such, the owner has the right to inspect and repair its functionality and parts. Anything less is dishonest and will hopefully not persist very long given the modern prominence of software.


or calibration changes cause engine to to over-heat and burn up and generally the company is held responsible for it

This happens not-too-infrequently in the car modding/tuning world --- the former, that is; to my knowledge no one has blamed the original manufacturer, much less successfully sued, after him/herself modding the ECU of a car and blowing up the engine.

Don't forget that a lot of other safety-critical components of cars (e.g. brakes, tires, etc.) have aftermarket replacements --- many of which actually perform better than stock.

If this "safety" attitude was prevalent since the beginning, we'd have cars that can only use approved fuel, no aftermarket parts would exist at all (their manufacturers being sued out of existence), and it'd be illegal to drive them on roads not approved by the manufacturer.

I will resist the urge to post that famous Benjamin Franklin quote this time.


Tractors & Combines are way more expensive than cars so it's not a fair comparison. You can't use a cherry picker to replace a tractor engine... At least the large kind.

Tractors and Combines use DEF and I hear reports of people defeating it. Our software has checks to look for such activity.


We already have tractors that are required to use only factory oil. The use of any other oil voids the powertrain warranty.


Voiding the warranty is perfectly fine. I don't think it's the burden of Deere to support aftermarket modifications.

The current situation however is more like the tractor bricking itself and never working again if you attempt to open the oil reservoir yourself, without the per-requisite authenticated repairman keys


I won't say the safety argument is meritless. But it is not the whole story.

Companies also use software to exert control over users and extract more money. By moving functionality from hardware into software, they can use both secrecy and laws like DMCA to restrict users' knowledge, ability, and rights to repair and modify their stuff.

The safety argument only makes any sense because so much of the system has moved from hardware into software, and this often makes the product worse, not better (e.g. your Jeep example). Compared to hardware, software is brittle, unreliable, complicated, and exploitable. However, it allows for the feature creep that marketers love and, perhaps as a byproduct or perhaps not, it transfers legal and operational control from the "owner" to the manufacturer.

If you are as committed to user freedom and well-being as you are to safety, then I urge you to modularize your vehicle design so that the safety-critical software you describe, such as auto-guidance, is completely separate from software used to, e.g., diagnose and repair mechanical problems. Then you can keep restrictions around the first kind but not the second. It sounds like, based on frustration with John Deere, you may be rewarded by the marketplace.


I actually agree with this statement and to your credit, a lot of the design work I do follows some of this.

Keep in mind that these are factories on wheels which means not only do we consume a tremendous amount of data, we also generate it. For example, 3D maps are built on the fly to show the customer where product has been sprayed, seeds planted, etc... On these vehicles, it's quite typical to see 1GB Data/hour generated which for an embedded device with no traditional hard drive, that's a lot. Flash memory is susceptible to extreme heat/cold so we have to be creative in this regard too (I can't talk much about this but suffice to say it's challenging).

That being a case of many, there is always a trade-off between performance and what you can reasonable keep separated either within the same PCB or between separate modules.

Thus, as much as I would like to keep separated, you simply can't sometimes. Current state of the art only allows for so much.


That's a poor argument in my opinion. Should my washing machine have no serviceable parts because I might electrocute myself?


No, but both the law and morality are allowed to utilize thresholds for determining acceptable risks; we weigh the risk of someone hurting themselves or others against the benefit of allowing people the ability to modify the hardware. Obviously we will all have different opinions about the relative value of risk vs freedom, but we should be able to settle on some median for how we value each. No matter what, it isn't the case that aggreeing with the idea that some machines are so dangerous we should prevent modification of the software forces us to agree that we are right to prevent modification of software on machines that have any danger at all.


OP was talking about tractors software and machines that can run people over and kill them, not washing machines or serviceable parts.


Shouldn't that liability come with owning the device? I don't understand how we've come to think, as a legal system, that the maker of a tool is responsible for the behavior of modifications made to their tool. Maybe that's not even true, but it seems to be believed so. If it is, I find it preposterous. Perhaps there's some money to follow.

Washing machines can malfunction and cause fires, hypothetically. Just because one device is designed in a way that is particularly dangerous doesn't mean the same reasoning can't be applied to another device that is dangerous to a lesser degree.


Washing machines can malfunction and cause fires, hypothetically.

Not just hypothetically: https://news.ycombinator.com/item?id=12610218

Mind you, those were unmodified so the maker does take the blame; but if the cause is known, modifications can make them safer too.


> modifications can make them safer too.

Very much this! I have a large agricultural tractor that has a transmission controlled by software. While it works great most of the time, there is a bug where it will occasionally not disengage the clutch for about a minute when this condition occurs. One time, due to the tractor showing no signs of going anywhere, I accidentally left the tractor in gear and left the cab. Everything seemed fine until about a minute later when it finally kicked into gear and started moving with me standing beside it.

Fortunately, I happened to be in a low gear and was just creeping along when it started moving. I was able to get back in it and get it stopped. If it had been going faster, who knows how bad the outcome could have been. This is something I would be fixing if I had access to the code.


hmm, sounds suspicious, in that the seat sensor was disabled? Even our vehicles won't move, when in gear if no one is in the seat. Seat senors cut the drive train. Even my 15 year old el-cheap-o Sears garden tractor does the same thing: cuts the engine off if I stand up out of the seat while in gear.


Yup, tractors and most farm equipment are fucking dangerous.

Even the compact/subcompacts regularly maim/kill people. I can totally understand wanting to minimize every possible new vector.


I agree; unlike washing machines, apparently, these tractors are made to be unserviceable.


Anything with mains power can set your house on fire.


Name two suits where someone argued that Alice is liable for what happens when Bob modifies her product.


I don't think your response is fair because even if those suits haven't been filed yet you can't honestly say that you know for certain they won't ever in the future.

The concern is legitimate.


The possibility that a large company might have to fight and win a lawsuit is not a plausible and certainly not an acceptable excuse for destroying the repairability of their products.


it's not and you're right. The right to repair effects agricultural equipment. Hacking the software to add features or disable things is not repairing it!


Yes I believe that is one pillar of the argument. The other is whether or not companies are using reparability (or lack thereof) to increase revenue or protect channels.


The solution to this is obviously simple: just require hacked tractors to remove all branding and void the warranty once it's been hacked. Also include a liability waiver so that if someone tampers with the software or hardware, the company can not be held liable anymore.


I'm "excited" about the day that I'll be forced to choose between getting a life-saving medical device implanted, or instead throwing a hissy fit about getting the source code to something that's going in my body. I hope we can figure this shit out with tractors instead...



I've done contract work for a manufacturer of electronic control systems for excavation and mining equipment. They've aggressively implemented DRM in their new products to support renting time-limited and geofenced access to optional features. Their revenue models for the newest generation of equipment are based in large part on the recurring revenue stream enabled by DRM.


There was an article about these John Deere hacking farmers a year ago, and this is a follow-up. There was good discussion on HN back then: https://news.ycombinator.com/item?id=13925994


A similar situation exists for consumer grade WiFi routers in the US. For about a year now, it's well-nigh impossible to flash routers with DD-WRT, OpenWRT, and friends due to the new FCC conditions manufacturers paid^H^H^H^H lobbied to get put in place.


You've peaked my interest. Has then been new legislation that has enabled router manufacturers lock down their hardware to a point an end-user cannot change it? All three organizations you name are open source and community driven. They cannot possibly write or port their firmware to every platform.


Not legislation, but instead an interpretation of FCC policy. See here[0] and here[1] for more details.

0 - https://www.wired.com/2016/03/way-go-fcc-now-manufacturers-l...

1 - https://hackaday.com/2016/02/26/fcc-locks-down-router-firmwa...


Parent likely refers to the 5ghz dfs rules, see https://arstechnica.com/information-technology/2015/09/fcc-o...


I wish that were the case, but alas it is not. The article you link was unfortunately an optimistic interpretation of what not yet known to come[0].

0 - https://www.wired.com/2016/03/way-go-fcc-now-manufacturers-l...


A history of the FCC router lockdown thing: https://arstechnica.com/information-technology/2016/08/fcc-f...



This story comes up quite often. It seems an opportunity for someone to launch a farm equipment business that makes it's margin on the initial sale instead of the service.

Even if it's just refurbished pre-drm tractors.


This practice is pretty much rampant across all industries ranging from cellphones to luxury cars. One simple answer is big companies just want to extend their profits. Another reasons is they want their distributors or dealerships or middleman to thrive as well without which their business may get stagnant.


Because copyright and software can be used to perform a backdoor attack to seize ownership of physical objects from you. You are looking at a full-on attack on the concept of ownership. You own nothing: It is licensed to you at the whims of the true owners.

Why wouldn't corporations, which are sociopathic "persons" defined exclusively by their avarice not want to do so? It is more profitable to prevent repair. It more profitable to force you into buying more.


This is a good point. In the ag business, the dealers of equipment make little to no money actually selling the equipment. They make their profit on selling parts and doing servicing. So, keeping the necessary software tools locked to authorized dealers would be popular with the dealerships. It ensures you have to go to the dealer and not some 3rd party mechanic shop.


According to the video lawyers from Microsoft and Apple showed up at the legislature to voice their objections to the Right To Repair bill. Interesting that these 2 companies would be so anti-consumer.


What's so interesting about it, are you surprised?

Apple have been fighting the ability to run your own code on the device since day one.

This is part of the business model of the "app store" purchase model and not very surprising.

I'm sure even you have heard that Microsoft have been anti-consumer once or twice over the years.


What if there was a law that any device containing embedded software would have to either be (a) open source, with source code and docs freely available, or (b) closed source, with source code and docs in escrow in the event that the manufacturer ceases support, at which time the code and docs would immediately become open source?



This is only going to get worse. A company called Blue River Technologies, which hopes to automate weeding was acquired by John Deere(the company being discussed in the article). Their software is obviously more complex than simple firmware which can be 'hacked' by the average joe.

If this weeding systems is profitable and becomes popular, these farmers are screwed.

Here's[1] a nice longread on that company. It's a bit fluffy, but I recommend it.

[1]: https://www.bloomberg.com/news/features/2018-01-11/this-army...


Is there any farm equipment (from China perhaps) that isn't DRMed? In addition to discussing the responsibilities of vendors, maybe we can also discuss the responsibility of customers to understand what they're buying.


from China perhaps

In this case their general lack of effort around security (see the whole IoT mess) may be a good thing, as it means any DRM is more hackable.

The flipside of that is the firmware may also be buggier, but then again, you're also more likely to be able to fix those bugs due to the hackability. The old Douglas Adams quote comes to mind: "The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair."


There are no new tractors that aren't DRMed if they don't have electronics. Hell, it's even hard to get service documents. Talking about consumer responsibility is great and all, too, but there is no choice so it doesn't matter.


I'm involved in the design of embedded systems for a relatively consumer market, and I've looked into making my company's designs more accessible to modification/software manipulation. There are a few (sometimes major) costs to allowing the modification of software that people tend to ignore in this discussion:

- Support: If you want to make your product more accessible to repair, you need to provide support for that. This costs developer time (making the code nice enough to be externally visible, maybe putting together an SDK, making sure any actual secret sauce or anti-counterfeiting systems are blocked off), customer support time (you WILL get calls when somebody buys a used system with modified firmware that doesn't work), and increases the hardware costs (want to provide a JTAG port? that's PCB space and BOM cost).

- Brand reflection: We've all seen how much of a ripple the wrong person having a bad experience with a product can have on the market. Say somebody buys a modified Initech Widget (tm) that has a bug or burns some other expensive equipment that it has to work with. That person complains on their friendly local social media network that Initech Widgets have a habit of (say) turning off furnaces in the middle of winter so the user's pipes freeze. Suddenly Initech is dealing with a media storm because they allowed modification of their software, which brings us to...

- Liability. What happens if somebody modifies the software in my product and somebody dies, burns down their house, or (* forbid) makes it possible to cause harmful interference and the FCC finds out? How do I prove to that user's (or the FCC's) lawyers that I shouldn't have to pay the damages? Let's assume that, as in the case of tractors or mobile baseband modems (which seem to get brought up a lot in this case) that software is a FUNDAMENTAL part of ensuring the safety/compliance of the unit.

Every time I've been through that calculus, the answer at the end has been, "I should do everything in my power to prevent the firmware of my devices from being modified".

I can understand and sympathize with the arguments in the other direction, and as a consumer I'm both qualified and interested in modifying the software on my widgets, but it would take a lot of change in the way our society works before most companies will put themselves at risk by supporting or even allowing it for their products.


A low tech alternative, depending on one's needs could be draft horses. My wife's family worked teams of belgians for about 80% of their 400-acre farm.

https://www.motherearthnews.com/homesteading-and-livestock/f...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: