Note that mitmproxy, which requires a Python install, is not necessary to monitor what is being sent out from your computing device.
The same results can be achieved using only socat and the openssl binary.
While I understand the terminology is popular, I would not call this "reverse-engineering"; to me this is simply viewing your own traffic.
I believe users have a right to see the traffic they (or the apps they use) are sending, and for security reasons alone they should monitor what is being sent. https plus third party CA usage complicates such transparency, making proxying techniques necessary.
I've been MITM'ing my connections ever since I discovered the Proxomitron ( http://en.wikipedia.org/wiki/Proxomitron ) - it lets me customise webpages and filter out the crap before it gets to any browser, and works in all the browsers on the system; even the ones embedded in apps. It has a logging feature to show traffic too.
It would be great if a proxy could be run directly on the mobile device, as then it could be used anywhere along with the apps its monitoring.
As you said HTTPS is a bit of a pain since it's actively designed to resist such "attacks", but as long as there's the option to specify your own CAs it will work. Certificate "unpinning" is still a manual process, however...
You just point to your Proxy server under Settings / WiFi. Alternatively you may want to install your proxy SSL CA on your mobile device, so you can see inside HTTPS connections (if supported).
Charles Proxy has some easy-to-use pointers [1][2].
But is there any way to use those to inspect traffic originating from outside of Firefox? With mitmproxy (or other tools, such as Charles/Fiddler/etc) you can run pretty much anything through them.
In this case you get what you pay for.
You get filtering. Local and remote mapping. SSL decryption. Request and response detail views. Breakpoints. The list goes on. Seriously, it's the best $50 I spent as part of my developer toolkit.
I don't know about Kayak's economics, but at least at Hipmunk we pay our data providers per search and it's really quite expensive. If they aren't offering an API anymore, it's probably because it was too pricey to operate.
You could easily cost a travel search company thousands of dollars very very quickly using an API they don't want you to use. I don't know if it's illegal or not, but it's certainly immoral.
How is using the API any different to using the website or app? Sounds like you need to move your access controls one layer down, into the API. If the app does rate limiting, the API should too/instead.
Take the github approach and let open source devs write innovative apps based on your open API!
The problem with any paid API is that the API consumer needs to make money somehow so that he can offer the price of the service.
How will the API consumer make money? I don't think Ads will generate enough revenue -- especially not on mobile.
Kayak will get some revenue from the bookings resulting from the searches, and would need to circle back some of this revenue back to the API consumer. This is technically complicated because it requires a lot of tracking infrastructure.
In the end, a service that would make sense is a lot more complicated than simply having an endpoint and saying it's X dollars per Y calls.
> From that folder, get the mitmproxy-ca-cert.pem file onto your mobile device by emailing it to yourself, for example. Then follow certificate installation steps for iOS or Android.
Does anyone know the possible legal repercussions open-sourcing a web service's API when the company doesn't explicitly grant permission? This is really neat, but could also raise the ire of a service that doesn't offer an API for a reason.
There are two obvious issues, as I see it. #1 is whether someone's liable for poking around inside the Kayak app in the way the author of the post did. #2 is whether it's legal to "open-source" the API by disclosing this information, once obtained.
My quick take is that #1 likely violates Kayak's terms of use for the app, its web site, or both. The TOU for the website has a host of prohibitions, including don't "bypass or circumvent other measures employed to prevent or limit access to Our Website," no deep-linking (such as a link to the API endpoint), and as a catchall also prohibits all inappropriate-as-defined-by-Kayak activity. I haven't read the app TOU, but it would be standard practice to prohibit reverse engineering of the app itself.
So we don't really need to reach point #2, which raises some 1A issues the first doesn't. (Note I'm excluding DMCA and CFAA issues because Kayak isn't our sue-happy friends at the MPAA or RIAA.)
As a practical matter, though, the author of the linked post, Shubhro Saha, appears to be an undergrad, so is probably judgment-proof and not the most likely target of litigation.
Just to add to this: the relevant case law seems to be Bowers v. Baystate Technologies [1], which held that EULA clauses prohibiting reverse engineering are enforceable. I personally think this is a terrible decision, since it also prevents a lot of good uses of reverse engineering (see some examples, like fighting censorship and diagnosing software vulnerabilities, in a recent paper of mine [2]).
DMCA has an exception for "interoperability" to allow reverse-engineering, which AFAIK is meant specifically for this case.
However, I wouldn't consider this any real "reverse-engineering", but just inspecting network traffic --- and it should be absolutely legal to do so when the traffic is generated from an app you legally obtained, running on a device you own.
I'm not sure DMCA applies here, as it only bans tools that circumvent technical measures intended to enforce copyright protections. I don't think that Kayak's data is copyrightable (on the basis of Feist Publications, Inc., v. Rural Telephone Service Co., which says that mere compilations of public information with no creative aspect aren't considered copyrightable), so DMCA shouldn't apply.
Great writeup, there's definitely potential to do something cool, especially if it's possible to get around any tracking and the following price manipulation. I tried installing your demo client, however on running it I got the following error:
root@kayak:~/kayak-mobile-client# python client.py
Departure airport code: LBG
Destination airport code: HAM
Departure date (MM/DD/YY): 12/26/14
Traceback (most recent call last):
File "client.py", line 56, in <module>
searchid = json.loads(r.text)["searchid"]
KeyError: 'searchid'
Have you set up your UUID and HASH according to the post? You'll actually need to run the app through mitmproxy yourself to get your unique credentials. Then, just drop those into the variables at the top of the script.
I just added some error handling to the script to make this a bit more obvious!
A useful technique for sure. The only technique I know to slow this down is to use certificate pinning, but it's probably pointless. Some of your users are probably extremely motivated (like ours [1]) and it's obvious to them that what they are doing is unsupported.
Certificate pinning would help to prevent sniffing traffic with tools like mitmproxy, but once you decompile the binary, you can disable certificate validation.
I'm curious, in your case did you consider preventing using your API by third-party clients?
Interesting! We've used a similar technique to reverse-engineering mobile apps from different banks in order to create a Ruby gem to fetch bank data (balance and transactions).
The same results can be achieved using only socat and the openssl binary.
While I understand the terminology is popular, I would not call this "reverse-engineering"; to me this is simply viewing your own traffic.
I believe users have a right to see the traffic they (or the apps they use) are sending, and for security reasons alone they should monitor what is being sent. https plus third party CA usage complicates such transparency, making proxying techniques necessary.
I wish more users would view their own traffic.
Keep up the good work.