Sunday, 16 June 2013

Transparent Interception of Android HTTPS Traffic

Smoothly intercepting HTTPS traffic of the Android emulator can be a pain in the ass. In particular when you're using a proxy such as Burp, you might not even know that some connections are not being intercepted. During a pentest this is something you absolutely want to avoid. That's why I created a small script to see all the connections the emulator is trying to make (including those that are not shown by your normal proxy).

Preparing Test Environment

It's handy to have a small script which initializes the PATH variable for easy access to all your tools. I use a setenvironment.bat file with the following commands:
@ECHO OFF
SET PATH=%PATH%;C:\Program Files\Java\jdk1.7.0_01\bin
SET PATH=%PATH%;C:\Program Files (x86)\Android\android-sdk\platform-tools
SET PATH=%PATH%;C:\Program Files (x86)\Android\android-sdk\tools
SET PATH=%PATH%;C:\Python27
SET PATH=%PATH%;C:\Mobile-Tools
ECHO === Environment for Andriod Pentesting has been set ===
Modify the paths to match your own environment. You don't have to use that to follow this post, as long as you can execute the command. We begin by starting the emulator1 2 and installing the application we want to test:
emulator @Android4
adb install WhatsApp.apk
Our goal will be to intercept and manipulate the traffic of WhatsApp using Burp.

Adding Burp CA Certificate

First we have to add the CA certificate of Burp as a trusted CA. In this post we will focus on the Burp proxy, though this guide is applicable to other proxy's as well. First we configure Burp to generate CA-signed per-host certificates. In Proxy Listeners select the appropriate entry and click on edit:


Burp now acts as a Certificate Authority (CA) and automatically generates certificates for any domain. We will have the add the Burp CA as a trusted certificate authority on Android. First we need to obtain the public key of the Burp CA. For this we configure firefox to use Burp as a proxy and navigate to a HTTPS website. Now do the following and make sure "PortswiggerCA " is selected in the Certificate Hierarchy:


Save the certificate as PortSwiggerCA.crt and copy it over to the SD card of the emulator:
adb push PortSwiggerCA.crt /sdcard/PortSwiggerCA.crt
In the emulator go to settings, security and then Install from SD card. You will have to set a lock screen PIN in order to add it. This should do the trick! Now run the emulator using the proxy execute:
emulator @Android4 -http-proxy http://localhost:8080
Unfortunately this doesn't work properly. If you visit HTTPS websites it will still complain that an invalid certificate is being used. But there's a simple fix for that :)

Certificate Problems

The reason why the certificates generated by Burp aren't accepted by Android is because they aren't valid yet (at least according to Android). When Burp generates a certificate it marks that it's valid starting precisely this second. Unfortunately Android then thinks the certificate isn't valid yet. This can easily be solved by setting the date of the Android emulator 1 day in the future.

AndroidProxy: Avoiding Additional Certificate Problems

There is an additional annoyance: Burp doesn't show the hostname of the requested site, it shows the IP address. This is quite annoying when viewing the history of all requests made:


When using Burp this is merely an annoyance. But when using a less powerfull proxy this can be a real problem! Behind the scenes the emulator acts as an invisible proxy for all HTTP(S) traffic, forwarding all request to your proxy. The problem is that these requests are of the form "CONNECT <IP>:443". As a result your proxy doesn't know the domain you are requesting. In turn your proxy potentially generates a certificate with as common name the IP address of the server. Unfortunately this will cause an error:


The latest version of Burp solves this by first looking up the hostname corresponding to that IP. But if your proxy doesn't have that capability you have a problem. And we still want a clean history screen in Burp, with the domain names being shown properly. So what we need is something that sits between Android and your actual proxy which rewrites the "CONNECT <ip>" request to "CONNECT <doman>".

To solve this problem I created AndroidProxy. It intercepting all DNS requests and rewrites the "CONNECT <ip>" command to "CONNECT <domain>" before it reaches your proxy. Normal HTTP traffic is left untouched. By intercepting all DNS request a unique IP address can be associated with each domain (even if in reality the domains have the same IP). When an IP address is encountered in a CONNECT method we can lookup the unique IP address and find the corresponding domain name. This process is illustrated below:


Another essential advantage is that AndroidProxy prints all the connection that the device is trying to make. This is important, because some proxy's (such as Burp) don't warn you about HTTPS connections that fail. They silently drop them, and you'll never know a connection was attempted.

You can start AndroidProxy using the following steps:
  1. Start your normal proxy (eg. Burp) listening on port 8080.
  2. Start AndroidProxy: python main.py
  3. Run the emulator: emulator @Avd -http-proxy http://localhost:8007 -dns-server localhost
  4. Remember to change the date of the Android emulator to 1 day in the future!
Future Work

The current setup is not perfect. There are several elements which could be improved:
  1. Find a (cross platform) GUI tool to intercept arbitrary SSL connections (not just HTTPS). TcpCatcher could be a candidiate, unfortunately I was unable to install the CA of TcpCatcher on Android (because the CA certificate was an old version?).
  2. Preferably your normal proxy just does a raw dump of the intercepted SSL connections, even though it's not HTTP. One would need to update Burp to accomplish this.
  3. Or we can extend the AndroidProxy itself to intercept SSL, and write a cross platform GUI on top to easily display individual connections.
I only plan to maintain AndroidProxy, and likely won't be adding new features. Feel free to fork it.

Footnotes
  1. Remember to install Intel HAXM when using the x86 emulator image [More Info]. If not present you get the error "emulator: Open HAX device failed". Download it from the SDK Manager, then go to android-sdk\extras\intel\Hardware_Accelerated_Execution_Manager and launch the installer.
  2. If you get the error "Failed to allocate memory: 8" make sure the emulator gets at most 512MB of memory. For some reason it doesn't start properly when using more memory [More Info].

4 comments:

  1. Hello,

    after starting androidproxy with 'sudo python main.py' I get the error:
    "twisted.internet.error.CannotListenError: Couldn't listen on any:53: [Errno 98] Address already in use."

    Do you have any idea how I can resolve this problem ?

    ReplyDelete
  2. Close the DNS server that is already running.

    ReplyDelete
  3. Thank you, that works.

    ReplyDelete
  4. Hi, The question i have is how can you add your own headers in the Connect request?

    ReplyDelete