Tag Archives: SSL

Key Pinning in Mobile Applications

By and

On Tuesday, October 13, 2015, Hubert Le Van Gong of the PayPal Ecosystem Security team gave a presentation to our developer community on SSL key pinning as it applies to mobile application development. I had a chance to interview him before the presentation to discuss the value and proper methods for incorporating key pinning in Android and IOS app development. Highlights of the interview along with Hubert’s recommended approach for key pinning on each platform are below. For follow up questions please contact Hubert Le Van Gong.

What is key pinning and how does it fit into the overall mobile application development process?

The goal of key pinning is to prevent web browsers or mobile applications from accepting an imposter’s TLS certificate that is used to pose as a legitimate service provider like PayPal. When a client attempts to establish a TLS connection with a server, it has to validate the certificate chain of trust. That chain of trust validation is primordial in PKI (Public Key Infrastructure) and uses cryptography to assess trust in the connection between a client and a server. A certificate chain of trust consists of a certificate issued by a root CA (certificate authority), a sequence of one or more certificates issued by intermediate CAs, and a leaf certificate (the one directly associated to the server). A root CA issues certificates to intermediary CAs and its certificate is the most stable and trusted in the trust chain (browsers and OS come pre-loaded with a long list of root CAs). Intermediate CA certificates exist as a layered defense and are stand-ins for the root certificate. Because an intermediate CA’s certificate is signed by the root cert which is stored offline (or chains back to such intermediate CA certificate), they have the authority to issue leaf certificates. In other words, the trust put in root certificate authorities is propagated down this certificate chain to the leaf certificate which is exposed to TLS clients.

Key pinning is the process of matching value(s) derived from a server’s chain of trust, as seen when the TLS connection is established, with pre-loaded values, derived from the expected chain of trust. If no common values are found, the TLS connection must be immediately terminated. Various approaches existed based on which part of a certificate should be used for generating those hash values, but the Subject Public Key Info, or SPKI is the standard method in the browser world (as per IETF’s RFC 7469, also known as HPKP, published in April this year) and I recommend using the same for applications. By calculating a SHA1 or, preferably, a SHA256 hash of the SPKI and pinning to that value, developers can ensure their apps connect to a server identified by a bona fide certificate and not a fraudulent one. While the fundamentals of key pinning apply to both the web platform realm and the mobile applications world, I’ve focused on the latter in order to provide the most value for the PayPal developer community.

Is this a real threat or an academic theory?

The threat is very real. Key pinning became a popular concept in 2012 after a series of fraudulent certificate-based hacks revealed yet another hole in the PKI infrastructure. The earliest relevant presentation I could find was a 2009 Black Hat presentation titled, New Tricks for Defeating SSL in Practice. Additionally, 2011 was a particularly rough year for CAs. DigiNotar, a Dutch CA suffered a devastating security breach resulting in over 500 fake certificates being issued, including certificates for some major internet domains like Google.com. The certificates were real from the point of view that they seemed to have been legitimately issued and validated properly. However, armed with those fraudulent certificates the attacker could easily perform man-in-the-middle attacks as well as spoof content etc. These are real world examples of large-scale attacks that cost millions to clean up and ultimately have damaged the trust between CAs and their customers. We see key pinning as one of the most promising approaches to address this lack of trust and provide an extra layer of security on top of TLS.

There are several certificates in the trust chain that developers can chose to pin to. Which is the best to pin to?

To clarify, the server is the one in control when it comes to which certificates should be used for pinning. The developers will incorporate whatever values are advertised by the server. Theoretically, it is possible to pin to certificates at all three levels: the end server (AKA leaf) cert, an intermediate CA cert, and the root CA cert. By pinning to two or more of these certificates, you’re ensuring that your site or service doesn’t get “bricked” because the only value you were pinning to has changed. Such change may happen as the result of the lost private key (yes, it can happen) or even a server misconfiguration. In that case, if no other pin is available, the service will be inaccessible to any client during the life of the (bad) pin’s policy. Pinning to a leaf certificate is usually not a good idea because those certificates will change more frequently. Intermediate CA certificates are OK to pin to, but one has to be cautious to avoid cross-signed certificates since they could alter the validation path. Root CA certificate are probably best. Therefore, when pinning to multiple values (e.g. 2), intermediate CA – root CA is a sound approach in my opinion. That said, it is also perfectly OK to only pin to a single certificate as long as that certificate is a root CA. Whatever the number of pin values, a backup pin is an absolute must-have. Note that the existence of a backup pin is a mandate in HPKP (i.e. for the web case) but it is equally important in the mobile app space. In both cases, the key pair corresponding to the backup pin should be kept offline until an issue arises with the primary pin/key.

Are there examples of improper key pinning?

Absolutely. As explained above, pinning to a single leaf (or even intermediate CA) certificate is not recommended. When it comes to what part of a certificate we should pin to, using the whole certificate is problematic because some fields might change. Although pinning solely to the public key of the certificate might seem like a good idea, there are some crypto risks associated to it so, similarly to what is specified in HPKP, applications should pin to the SPKI part of a certificate.

In your presentation, you discuss several methods for key pinning for Android and IOS. Can you give an example of proper pinning for each?

Sure. First, I want to mention a couple best practices to ensure successful pinning. It is important to note that in order to properly pin, applications must include all values in their pinning set. At this time, PayPal uses four values because we’ve recently transitioned from SHA1 root certificates to SHA256 ones and we have the backup values. If we were pinning to two certificates, we would have eight values. Again, the SHA1 or SHA256 sets can be used but SHA256 is recommended since that’s what is used in HPKP and SHA1 is on its way out anyway. Second, try to leverage the includeSubDomain directive as much as possible (as defined in HPKP). Fewer well-executed pins equal less chance of failure. Practice good pin hygiene!

Android Method

The simplest approach is to use a JSEE-based method as shown below. This is the recommended approach for Android. The method’s input arguments are an HTTPS connection and a set of valid pins for the targeted URL.

Android Method for Key Pinning
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
private boolean validatePinning(HttpsURLConnection conn, Set<String> validPins) {
    try {
        Certificate[] certs = conn.getServerCertificates();
        MessageDigest md = MessageDigest.getInstance("SHA-256");
        for (Certificate cert : certs) {
            X509Certificate x509Certificate = (X509Certificate) cert;
            byte[] key = x509Certificate.getPublicKey().getEncoded();
            md.update(key, 0, key.length);
            byte[] hashBytes = md.digest();
            StringBuffer hexHash = new StringBuffer();
            for (int i = 0; i < hashBytes.length; i++) {
                int k = 0xFF & hashBytes[i];
                String tmp = (k<16)? "0" : "";
                tmp += Integer.toHexString(0xFF & hashBytes[i]);
                hexHash.append(tmp);
            }
            if (validPins.contains(hexHash.toString())) {
                return true;
            }
        }
    } catch (Exception e) {
        e.printStackTrace();
        return false;
    }
    return false;
}

The pins are declared as strings. For instance:

Declaring Key Pins

1
2
3
4
private static final Set<String> PINS = new HashSet<String>(Arrays.asList(
        new String[]{
                "996b510ce2380da9c738...87cb13c9ec409941",
                "ba47e83b1ccf0939bb40d2...edf856ba892c06481a"}));

Leveraging the above method, here is an example showing how this can be put to use. The only relevant portion is highlighted below.

Example Using Key Pinning

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
protected String doInBackground(String... urls) {
    try {
        /** Test pinning given the target URL **/
        /** for now use pre-defined endpoint URL instead or urls[0] **/
        Log.i(LOG_TAG, "==> PinningTestTask launched.");
        String dest = defaultEndpoint;
        URL targetURL = new URL(dest);
        HttpsURLConnection targetConnection = (HttpsURLConnection) targetURL.openConnection();
        targetConnection.connect();
        if (validatePinning(targetConnection, PINS)) {
            final String updateText = "Key pinning succeded for: " + dest;
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    textView.setText(updateText);
                }
            });
        } else {
            final String updateText = "Key pinning failed for: " + dest;
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    textView.setText(updateText);
                }
            });
        }
    } catch (Exception e) {
        e.printStackTrace();
        final String updateText = "Key pinning failed for: " + dest + "\n" + e.toString();
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                textView.setText(updateText);
            }
        });
    }
    return null;
}

iOS Method

For iOS, I find DataTheorem’s TrustKit very elegant (developed in collaboration with Yahoo!). It is very easy to integrate with an iOS application: there is no need to update all the classes of the application where https connections are established and instead, there’s at most one API call made during the application’s initialization (because they use a code injection approach). Alternatively (and presumably even easier), it is possible to simply include TrustKit in the manifest file of the bundle and, voilà, key pinning is enabled. For these reasons using TrustKit is our internally recommended approach on iOS. Note that it is currently only available in Objective-C but I have no doubt a Swift version should be coming soon (to meet Apple’s strong push for Swift).

There is a configuration file used to define pinned keys and policies. Interesting detail: in addition to the key hash, the key algorithm must be specified because iOS does not provide developers with an API to easily parse X.509 certificates… So TrustKit needs that info to easily reconstruct the SPKI info. Apparently, most other iOS libraries that provide key pinning only allow pinning to the raw key because it’s easier (even though this is not recommended as explained above).

Another interesting aspect of their approach is that they essentially mimic HPKP functionality (URL for reporting errors, no-enforcement flag) in their configuration file, but from a client app standpoint.

The library supports both application and browser-based (think webview) TLS pinning. It is available on iOS and OS X and was open sourced in August during the BlackHat conference. The implementation directly leverages the low-level SecureTransport framework which is a good thing since SecureTransport seems to be a pretty robust SSL implementation.

The example below shows how TrustKit can be leveraged by simply adding some configuration code to the DidLoad( ) method.

iOS Key Pinning with TrustKit

 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
- (void)viewDidLoad {
    [super viewDidLoad];
    NSDictionary *trustKitConfig;
    trustKitConfig = @{
                       @"api.paypal.com" : @{
                               kTSKPublicKeyAlgorithms : @[kTSKAlgorithmRsa2048],
                               kTSKPublicKeyHashes : @[
                                       @"ukfoOxzPCTm...ft+Fa6iSwGSBo=",
                                       @"mWtRDOI4DanHOM+...MYh8sTyexAmUE="
                                       ],
                               kTSKEnforcePinning : @YES
                               }
                       };
    [TrustKit initializeWithConfiguration
}

Note that the values used above are not direct public key hashes but Base64 encoded values of the SHA256 hashes (following what is defined in HPKP). Upon start, the above code will set up TrustKit and any Https session established with one of the configured URLs (e.g. using NSURLSession) will go through pin validation.

Outbound SSL Performance in Node.js

By

When browsing the internet, we all know that employing encryption via SSL is extremely important. At PayPal, security is our top priority. We use end-to-end encryption, not only for our public website, but for all our internal service calls as well. SSL encrypton will, however, affect Node.js performance at scale. We’ve spent time tuning our outbound connections to try and get the most out of them. This is a list of some of those SSL configuration adjustments that we’ve found can dramatically improve outbound SSL performance.

SSL Ciphers

Out of the box, Node.js SSL uses a very strong set of cipher algorithms. In particular, Diffie Hellman and Elliptical Curve algorithms are hugely expensive and basically cripple Node.js performance when you begin making a lot of outbound SSL calls at default settings. To get an idea about just how slow this can be, here is a CPU sample taken from a service call:

918834.0ms 100.0% 0.0 node (91770)
911376.0ms 99.1% 0.0   start
911376.0ms 99.1% 0.0    node::Start
911363.0ms 99.1% 48.0    uv_run
909839.0ms 99.0% 438.0    uv__io_poll
876570.0ms 95.4% 849.0     uv__stream_io
873590.0ms 95.0% 32.0       node::StreamWrap::OnReadCommon
873373.0ms 95.0% 7.0         node::MakeCallback
873265.0ms 95.0% 15.0         node::MakeDomainCallback
873125.0ms 95.0% 61.0          v8::Function::Call
873049.0ms 95.0% 13364.0       _ZN2v88internalL6InvokeEbNS0
832660.0ms 90.6% 431.0          _ZN2v88internalL21Builtin
821687.0ms 89.4% 39.0            node::crypto::Connection::ClearOut
813884.0ms 88.5% 37.0             ssl23_connect
813562.0ms 88.5% 54.0              ssl3_connect
802651.0ms 87.3% 35.0               ssl3_send_client_key_exchange
417323.0ms 45.4% 7.0                 EC_KEY_generate_key
383185.0ms 41.7% 12.0                ecdh_compute_key
1545.0ms 0.1% 4.0                    tls1_generate_master_secret
123.0ms 0.0% 4.0                     ssl3_do_write
...

Let’s focus on key generation:

802651.0ms 87.3% 35.0 ssl3_send_client_key_exchange
417323.0ms 45.4% 7.0 EC_KEY_generate_key
383185.0ms 41.7% 12.0 ecdh_compute_key

87% of the time for the call is spent in keygen!

These ciphers can be changed to be less compute intensive. This is done in the https (or agent) options. For example:

var agent = new https.Agent({
    "key": key,
    "cert": cert,
    "ciphers": "AES256-GCM-SHA384"
});

The key here is exclusion of the expensive Diffie-Hellman algorithms (i.e. DH, EDH, ECDH). With something like that, we can see a dramatic change in the sample:

...
57945.0ms 32.5% 16.0 ssl3_send_client_key_exchange
28958.0ms 16.2% 9.0 generate_key
26827.0ms 15.0% 2.0 compute_key
...

You can learn more about the cipher strings from openSSL documentation.

SSL Session Resume

If your server supports SSL session resume, then you can pass sessions in the (undocumented as yet) https (or agent) option session. You can also wrap your agent‘s createConnection function:

var createConnection = agent.createConnection;

agent.createConnection = function (options) {
    options.session = session;
    return createConnection.call(agent, options);
};

Session resume will decrease the cost of your connections by performing an abbreviated handshake on connection.

Keep Alive

Enabling keepalive in agent will mitigate SSL handshakes. A keepalive agent, such as agentkeepalive, can ‘fix’ Node’s keepalive troubles but is unnecessary in Node 0.12.

Another thing to keep in mind is agent maxSockets, where high numbers can result in a negative performance impact. Scale your maxSockets based on the volume of outbound connections you are making.

Slab Size

tls.SLAB_BUFFER_SIZE determines the allocation size of the slab buffers used by tls clients (and servers). The size defaults to 10 megabytes.

This allocation will grow your rss and increase garbage collection time. This means performance hits at high volume. Adjusting this to a lower number can improve memory and garbage collection performance. In 0.12, however, slab allocation has been improved and these adjustments are no longer necessary.

Recent SSL Changes in 0.12

Testing out Fedor’s SSL enhancements.

Test Description

Running an http server that acts as a proxy to an SSL server, all running on localhost.

v0.10.22

Running 10s test @ http://127.0.0.1:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 69.38ms 30.43ms 268.56ms 95.24%
Req/Sec 14.95 4.16 20.00 58.65%
3055 requests in 10.01s, 337.12KB read
Requests/sec: 305.28
Transfer/sec: 33.69KB

v0.11.10-pre (build from master)

Running 10s test @ http://127.0.0.1:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 75.87ms 7.10ms 102.87ms 71.55%
Req/Sec 12.77 2.43 19.00 64.17%
2620 requests in 10.01s, 276.33KB read
Requests/sec: 261.86
Transfer/sec: 27.62KB

There isn’t a lot of difference here, but that’s due to the default ciphers, so let’s adjust agent options for ciphers. For example:

var agent = new https.Agent({
    "key": key,
    "cert": cert,
    "ciphers": "AES256-GCM-SHA384"
});

v0.10.22

Running 10s test @ http://localhost:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 59.85ms 6.77ms 95.71ms 77.29%
Req/Sec 16.39 2.36 22.00 61.97%
3339 requests in 10.00s, 368.46KB read
Requests/sec: 333.79
Transfer/sec: 36.83KB

v0.11.10-pre (build from master)

Running 10s test @ http://localhost:3000/
20 threads and 20 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 38.99ms 5.96ms 71.87ms 86.22%
Req/Sec 25.43 5.70 35.00 63.36%
5160 requests in 10.00s, 569.41KB read
Requests/sec: 515.80
Transfer/sec: 56.92KB

As we can see, there is a night and day difference with Fedor’s changes: almost 2x the performance between 0.10 and 0.12!

Wrap Up

One might ask “why not just turn off SSL, then it’s fast!”, and this may be an option for some. Actually, this is typically the answer I get when I ask others how they overcame SSL performance issues. But if anything enterprise SSL requirements will increase rather than decrease; and although a lot has been done to improve SSL in Node.js, performance tuning is still needed. Hopefully some of the above tips will help in tuning for your SSL use case.