Tag Archives: security

DMARC-Related Recommendations Included in NIST Guidance on Trustworthy Email


Another important milestone was recently achieved for Domain-based Message Authentication Reporting and Conformance (DMARC), one of the PayPal Ecosystem Security team’s major undertakings in making the internet a safer, more secure place.

After several years of collaboration with the email security community, the U.S. National Institute of Standards and Technology (NIST) included recommendations for supporting DMARC in NIST’s SP 800-177, Trustworthy Email. SP 800-177 was released in September and is intended to give recommendation and guidelines for enhancing trust in email. While the audience for NIST publications is typically US federal agencies, its guidance does tend to influence other global organizations and industry tides. The recommendations for DMARC in the publication include the following:

  • Security Recommendation 4-11: Sending domain owners who deploy SPF and/or DKIM are recommended to publish a DMARC record signaling to mail receivers the disposition expected for messages purporting to originate from the sender’s domain.
  • Security Recommendation 4-12: Mail receivers who evaluate SPF and DKIM results of received messages are recommended to dispose them in accordance with the sending domain’s published DMARC policy, if any. They are also recommended to initiate failure reports and aggregate reports according to the sending domain’s DMARC policies.

DMARC is a way to make it easier for email senders and receivers to determine whether or not a given message is legitimately from the sender, and what to do if it isn’t. This makes it easier to identify spoofed email messages, a common phishing technique, and reject them. Users often can’t tell a real message from a fake one, and mailbox providers have to make very difficult (and frequently incorrect) choices about which messages to deliver and which might harm users. Senders are also largely unaware of problems with their own email being abused and want feedback from receivers about fraudulent messages. DMARC addresses these issues, helping email senders and receivers work together to better secure email, protecting users and brands from costly abuse.

Our 2015 post on the publication of DMARC within the IETF highlighted a critical step towards making it a widely adopted policy layer that builds on top of Sender Policy Framework (SPF) and Domain Keys Identified Mail (DKIM). In addition to the inclusion by NIST, the U.K.’s Government Digital Service and Germany’s Federal Office for Information Security have incorporated similar DMARC guidance into their recommended email security posture. These global recommendations clearly indicate that DMARC is a required component of effective email security and PayPal is proud to have lead such an important initiative as DMARC that protects not just our company and customers, but anyone that uses email.

[1] https://www.gov.uk/government/publications/email-security-standards/domain-based-message-authentication-reporting-and-conformance-dmarc

[2] https://www.allianz-fuer-cybersicherheit.de/ACS/DE/_/downloads/BSI-CS_098.html

Python by the C side


C shells by the C shoreMahmoud’s note: This will be my last post on the PayPal Engineering blog. If you’ve enjoyed this sort of content subscribe to my blog/pythondoeswhat.com or follow me on Twitter. It’s been fun!

All the world is legacy code, and there is always another, lower layer to peel away. These realities cause developers around the world to go on regular pilgrimage, from the terra firma of Python to the coasts of C. From zlib to SQLite to OpenSSL, whether pursuing speed, efficiency, or features, the waters are powerful, and often choppy. The good news is, when you’re writing Python, C interactions can be a day at the beach.


A brief history

As the name suggests, CPython, the primary implementation of Python used by millions, is written in C. Python core developers embraced and exposed Python’s strong C roots, taking a traditional tack on portability, contrasting with the “write once, debug everywhere” approach popularized elsewhere. The community followed suit with the core developers, developing several methods for linking to C. Years of these interactions have made Python a wonderful environment for interfacing with operating systems, data processing libraries, and everything the C world has to offer.

This has given us a lot of choices, and we’ve tried all of the standouts:

Approach Vintage Representative User Notable Pros Notable Cons
C extension modules 1991 Standard library Extensive documentation and tutorials. Total control. Compilation, portability, reference management. High C knowledge.
SWIG 1996 crfsuite Generate bindings for many languages at once Excessive overhead if Python is the only target.
ctypes 2003 oscrypto No compilation, wide availability Accessing and mutating C structures cumbersome and error prone.
Cython 2007 gevent, kivy Python-like. Highly mature. High performance. Compilation, new syntax and toolchain.
cffi 2013 cryptography, pypy Ease of integration, PyPy compatibility New/High-velocity.

There’s a lot of history and detail that doesn’t fit into a table, but every option falls into one of three categories:

  1. Writing C
  2. Writing code that translates to C
  3. Writing code that calls into libraries that present a C interface

Each has its merits, so we’ll explore each category, then finish with a real, live, worked example.

Writing C

Python’s core developers did it and so can you. Writing C extensions to Python gives an interface that fits like a glove, but also requires knowing, writing, building, and debugging C. The bugs are much more severe, too, as a segmentation fault that kills the whole process is much worse than a Python exception, especially in an asynchronous environment with hundreds of requests being handled within the same process. Not to mention that the glove is also tailored to CPython, and won’t fit quite right, or at all, in other execution environments.

At PayPal, we’ve used C extensions to speed up our service serialization. And while we’ve solved the build and portability issue, we’ve lost track of our share of references and have moved on from writing straight C extensions for new code.

Translating to C

After years of writing C, certain developers decide that they can do better. Some of them are certainly onto something.

Going Cythonic

Cython is a superset of the Python programming language that has been turning type-annotated Python into C extensions for nearly a decade, longer if you count its predecessor, Pyrex. Apart from its maturity, the points that matters to us are:

  • Every Python file is a valid Cython file, enabling incremental, iterative optimization
  • The generated C is highly portable, building on Windows, Mac, and Linux
  • It’s common practice to check in the generated C, meaning that builders don’t need to have Cython installed.

Not to mention that the generated C often makes use of performance tricks that are too tedious or arcane to write by hand, partially motivated by scientific computing’s constant push. And through all that, Cython code maintains a high level of integration with Python itself, right down to the stack trace and line numbers.

PayPal has certainly benefitted from their efforts through high-performance Cython users like gevent, lxml, and NumPy. While our first go with Cython didn’t stick in 2011, since 2015, all native extensions have been written and rewritten to use Cython. It wasn’t always this way however.

A sip, not a SWIG

An early contributor to Python at PayPal got us started using SWIG, the Simplified Wrapper and Interface Generator, to wrap PayPal C++ infrastructure. It served its purpose for a while, but every modification was a slog compared to more Pythonic techniques. It wasn’t long before we decided it wasn’t our cup of tea.

Long ago SWIG may have rivaled extension modules as Python programmers’ method of choice. These days it seems to suit the needs of C library developers looking for a fast and easy way to wrap their C bindings for multiple languages. It also says something that searching for SWIG usage in Python nets as much SWIG replacement libraries as SWIG usage itself.

Calling into C

So far all our examples have involved extra build steps, portability concerns, and quite a bit of writing languages other than Python. Now we’ll dig into some approaches that more closely match Python’s own dynamic nature: ctypes and cffi.

Both ctypes and cffi leverage C’s Foreign Function Interface (FFI), a sort of low-level API that declares callable entrypoints to compiled artifacts like shared objects (.so files) on Linux/FreeBSD/etc. and dynamic-link libraries (.dll files) on Windows. Shared objects take a bit more work to call, so ctypes and cffi both use libffi, a C library that enables dynamic calls into other C libraries.

Shared libraries in C have some gaps that libffi helps fill. A Linux .so, Windows .dll, or OS X .dylib is only going to provide symbols: a mapping from names to memory locations, usually function pointers. Dynamic linkers do not provide any information about how to use these memory locations. When dynamically linking shared libraries to C code, header files provide the function signatures; as long as the shared library and application are ABI compatible, everything works fine. The ABI is defined by the C compiler, and is usually carefully managed so as not to change too often.

However, Python is not a C compiler, so it has no way to properly call into C even with a known memory location and function signature. This is where libffi comes in. If symbols define where to call the API, and header files define what API to call, libffi translates these two pieces of information into how to call the API. Even so, we still need a layer above libffi that translates native Python types to C and vice versa, among other tasks.


ctypes is an early and Pythonic approach to FFI interactions, most notable for its inclusion in the Python standard library.

ctypes works, it works well, and it works across CPython, PyPy, Jython, IronPython, and most any Python runtime worth its salt. Using ctypes, you can access C APIs from pure Python with no external dependencies. This makes it great for scratching that quick C itch, like a Windows API that hasn’t been exposed in the os module. If you have an otherwise small module that just needs to access one or two C functions, ctypes allows you to do so without adding a heavyweight dependency.

For a while, PayPal Python code used ctypes after moving off of SWIG. We found it easier to call into vanilla shared objects built from C++ with an extern C rather than deal with the SWIG toolchain. ctypes is still used incidentally throughout the code for exactly this: unobtrusively calling into certain shared objects that are widely deployed. A great open-source example of this use case is oscrypto, which does exactly this for secure networking. That said, ctypes is not ideal for huge libraries or libraries that change often. Porting signatures from headers to Python code is tedious and error-prone.


cffi, our most modern approach to C integration, comes out of the PyPy project. They were seeking an approach that would lend itself to the optimization potential of PyPy, and they ended up creating a library that fixes many of the pains of ctypes. Rather than handcrafting Python representations of the function signatures, you simply load or paste them in from C header files.

For all its convenience, cffi’s approach has its limits. C is really almost two languages, taking into account preprocessor macros. A macro performs string replacement, which opens a Fun World of Possibilities, as straightforward or as complicated as you can imagine. cffi’s approach is limited around these macros, so applicability will depend on the library with which you are integrating.

On the plus side, cffi does achieve its stated goal of outperforming ctypes under PyPy, while remaining comparable to ctypes under CPython. The project is still quite young, and we are excited to see where it goes next.

A Tale of 3 Integrations: PKCS11

We promised an example, and we almost made it three.

PKCS11 is a cryptography standard for interacting with many hardware and software security systems. The 200-plus-page core specification includes many things, including the official client interface: A large set of C header-style information. There are a variety of pre-existing bindings, but each device has its own vendor-specific quirks, so what are we waiting for?


As stated earlier, ctypes is not great for sprawling interfaces. The drudgery of converting function signatures invites transcription bugs. We somewhat automated it, but the approach was far from perfect.

Our second approach, using cffi, worked well for our first version’s supported feature subset, but unfortunately PKCS11 uses its own CK_DECLARE_FUNCTION macro instead of regular C syntax for defining functions. Therefore, cffi’s approach of skipping #define macros will result in syntactically invalid C code that cannot be parsed. On the other hand, there are other macro symbols which are compiler or operating system intrinsics (e.g. __cplusplus, _WIN32, __linux__). So even if cffi attempted to evaluate every macro, we would immediately runs into problems.

So in short, we’re faced with a hard problem. The PKCS11 standard is a gnarly piece of C. In particular:

  1. Many hundreds of important constant values are created with #define
  2. Macros are defined, then re-defined to something different later on in the same file
  3. pkcs11f.h is included multiple times, even once as the body of a struct

In the end, the solution that worked best was to write up a rigorous parser for the particular conventions used by the slow-moving standard, generate Cython, which generates C, which finally gives us access to the complete client, with the added performance bonus in certain cases. Biting this bullet took all of a day and a half, we’ve been very satisfied with the result, and it’s all thanks to a special trick up our sleeves.

Parsing Expression Grammars

Parsing expression grammars (PEGs) combine the power of a true parser generating an abstract syntax tree, not unlike the one used for Python itself, with the convenience of regular expressions. One might think of PEGs as recursive regular expressions. There are several good libraries for Python, including parsimonious and parsley. We went with the former for its simplicity.

For this application, we defined a two grammars, one for pkcs11f.h and one for pkcs11t.h:


    file = ( comment / func / " " )*
    func = func_hdr func_args
    func_hdr = "CK_PKCS11_FUNCTION_INFO(" name ")"
    func_args = arg_hdr " (" arg* " ); #endif"
    arg_hdr = " #ifdef CK_NEED_ARG_LIST" (" " comment)?
    arg = " " type " " name ","? " " comment
    name = identifier
    type = identifier
    identifier = ~"[A-Z_][A-Z0-9_]*"i
    comment = ~"(/\*.*?\*/)"ms


    file = ( comment / define / typedef / struct_typedef / func_typedef / struct_alias_typedef / ignore )*
    typedef = " typedef" type identifier ";"
    struct_typedef = " typedef struct" identifier " "? "{" (comment / member)* " }" identifier ";"
    struct_alias_typedef = " typedef struct" identifier " CK_PTR"? identifier ";"
    func_typedef = " typedef CK_CALLBACK_FUNCTION(CK_RV," identifier ")(" (identifier identifier ","? comment?)* " );"    member = identifier identifier array_size? ";" comment?
    array_size = "[" ~"[0-9]"+ "]"
    define = "#define" identifier (hexval / decval / " (~0UL)" / identifier / ~" \([A-Z_]*\|0x[0-9]{8}\)" )
    hexval = ~" 0x[A-F0-9]{8}"i
    decval = ~" [0-9]+"
    type = " unsigned char" / " unsigned long int" / " long int" / (identifier " CK_PTR") / identifier
    identifier = " "? ~"[A-Z_][A-Z0-9_]*"i
    comment = " "? ~"(/\*.*?\*/)"ms
    ignore = ( " #ifndef" identifier ) / " #endif" / " "

Short, but dense, in true grammatical style. Looking at the whole program, it’s a straightforward process:

  1. Apply the grammars to the header files to get our abstract syntax tree.
  2. Walk the AST and sift out the semantically important pieces, function signatures in our case.
  3. Generate code from the function signature data structures.

Using only 200 lines of code to bring such a massive standard to bear, along with the portability and performance of Cython, through the power of PEGs ranks as one of the high points of Python in practice at PayPal.

Wrapping up

It’s been a long journey, but we stayed afloat and we’re happy to have made it. To recap:

  • Python and C are hand-in-glove made for one another.
  • Different C integration techniques have their applications, our stances are:
    • ctypes for dynamic calls to small, stable interfaces
    • cffi for dynamic calls to larger interfaces, especially when targeting PyPy
    • Old-fashioned C extensions if you’re already good at them
    • Cython-based C extensions for the rest
    • SWIG pretty much never
  • Parsing Expression Grammars are great!

All of this encapsulates perfectly why we love Python so much. Python is a great starter language, but it also has serious chops as a systems language and ecosystem. That bottom-to-top, rags-to-riches, books-to-bits story is what makes it the ineffable, incomparable language that it is.

C you around!

Kurt and Mahmoud

Security-Related Changes Required to Avoid Service Disruption


REVISED May 12, 2016Please note that some of the deadlines have changed since this post was originally published. This post reflects the latest deadline dates for security updates.

Beginning in early 2016, PayPal will introduce a number of security-related product updates.

These updates are part of an industry-wide initiative to improve security standards. Some of these updates, like the TLS upgrade, are mandated by the PCI Security Council and are required by every website that transmits or processes cardholder data.

Merchants and developers may have to update their integrations in order to be in compliance and ensure that their applications continue to function as expected.

For PayPal customers, these updates include:

TLS 1.2 Upgrade

The most secure protocol for sharing information on the web today is Transport Layer Security (TLS) version 1.2. PayPal is enabling support for TLS 1.2 for all secure connections and in 2016 will start requiring its use. You will need to verify that your environment supports TLS 1.2 and if necessary make appropriate updates. PayPal is updating its services to require TLS v1.2 for all HTTPS connections in June of 2017. After that time, all TLS v1.0 and TLS v1.1 API connections will be refused.

Learn more

IP Address Update for PayPal SFTP

If your integration is set-up to systematically exchange files with PayPal’s Secure FTP Reporting / Batch Servers, please note that the IP addresses for these servers are changing. If your integration is hardcoded to the current IP addresses, you will need to upgrade accordingly. You must act by April 14, 2016.

Learn more

IPN Verification Postback to HTTPS

If you are using PayPal’s Instant Payment Notification (IPN) service, you will need to ensure that HTTPS is used when posting the message back to PayPal for verification. After June of 2017 HTTP postbacks will no longer be supported.

Learn more

Merchant API Credential Upgrade

The certificates issued by PayPal as API Credentials for use with the Classic API are being upgraded to SHA-256 signed 2048-bit certificates. If you currently connect to PayPal using Certificate API Credentials, you will need to have a new certificate issued and start using it for all API requests. Depending on when your certificate expires, you will have to upgrade between Jan 31, 2016 and Jan 1, 2018.

Learn more

SSL Certificate Upgrade

PayPal is in the process of upgrading the SSL certificates used to secure our web sites and API endpoints. These new certificates will be signed using the SHA-256 algorithm and VeriSign’s 2048-bit G5 Root Certificate. You will need to ensure that your environment supports the use of the SHA-256 signing algorithm and discontinue the use of SSL connections that rely on the VeriSign G2 Root Certificate. This action must be taken by June 17, 2016 in order to avoid any disruption of service.

Learn more

PayPal SDK Updates

For developers using one of PayPal’s SDK libraries, depending on the runtime environment and version your application uses, a code change or SDK update may be required in order to enable TLS v1.2. Java 6 u115 b32, Java 7 or Java 8 (preferred) and Android 4.x environments require TLS v1.2 to be enabled in order to maintain functionality.

If you’re using one of PayPal’s REST SDKs, the best practice is to stay up to date with the latest SDK version and latest versions of the code libraries required by the PayPal SDK. PayPal provides server and client SDKs for Java, .Net, PHP, Python, Ruby, Node.js, Android, and iOS. Detailed test instructions can also be found at https://github.com/paypal/TLS-update.

Developers should reference the 2017-2018 Merchant Security Roadmap microsite for all the latest details and up-to-date information regarding these security changes. You can also visit the PayPal Developer site for more information on PayPal security guidelines and best practices.

Adam Colson

Author: Adam Colson

About the author: Adam is a Product Manager at Braintree | PayPal, focusing on the PayPal developer experience since August 2015. When Adam isn’t helping to enable global e-commerce through APIs, he can also be found hacking away at the Internet of Things or tinkering in his garage.

@adamc | LinkedIn

Acceptance of FIDO 2.0 Specifications by the W3C accelerates the movement to end passwords


What if we could eliminate, or at least significantly mitigate the risk of passwords? On February 17, 2016, the World Wide Web Consortium (W3C) announced that the creation of the Web Authentication Working Group to move the world closer to this goal. The mission of the group is to define a client-side API that provides strong authentication functionality to Web Applications. The group’s technical work will be accelerated by the acceptance of the FIDO 2.0 Web APIs. This specification, whose co- authors include PayPal’s Hubert Le Van Gong and Jeff Hodges, help simplify and improve the security of authentication.  As the steward for the Web platform, the W3C is uniquely positioned to focus the attention of Web infrastructure providers and developers on the shortcomings of passwords and the necessity of their replacement.

The FIDO2.0 protocol employs public key cryptography, relying on users’ devices to generate key pairs during a registration process. The user’s device retains the generated private key and delivers the public key to the service provider. The service provider retains this key, associates it with a user’s account, and when a login request is received, issues a challenge that must be signed by the private key holder as a response.

When challenged, the FIDO implementation stack signals the user to authenticate using the mechanism employed at registration time. This might be via PIN, biometric reader, or an alternative modality. A local comparison of the current authentication request is made to the stored registration value. A successful match unlocks the associated private key; the challenge is signed and returned to the service provider.

This approach dramatically alters the economics of attacks on service providers and their password stores. For each service provider that a user interacts with, a unique private/public key pair is generated. Not only does this ensure that service providers are unable to use protocol artifacts to collude in user-unwanted ways, it renders the public key store of little to no value to fraudsters. Attacks at scale through exfiltration of passwords are no longer a viable means of generating revenue – the ultimate goal of fraudsters.

Early version of the FIDO protocols, UAF and U2F, were developed with deployments by PayPal and others. Much has been learned through the process with the Fido 2.0 specifications designed to bring together the best features of U2F and UAF. With the contribution of the Fido 2.0 APIs to the W3C, they will be sedimented into the Web Platform enabling authoring tools, hosting providers, and others to interoperate will a broad range of devices that will support the Fido 2.0 protocols. 

At PayPal, we are committed to a more secure and privacy-respecting web experience for all internet users. We realize that an easy-to-use, secure, and privacy-respecting means of authentication benefits everyone and having the same protections regardless of the site enhances the overall security of the Web. We look forward to actively participating in the W3C Web Authentication Working Group to continue our pursuit of ubiquitous, simple, secure, and privacy-respecting authentication on the Web.


Lessons Learned from the Java Deserialization Bug


(with input from security researcher Mark Litchfield)


At PayPal, the Secure Product LifeCycle (SPLC) is the assurance process to reduce and eliminate security vulnerabilities in our products over time by building repeatable/sustainable proactive security practices embedding them within our product development process.

A key tenet of the SPLC is incorporating the lessons learned from remediating security vulnerabilities back into our processes, tools, and training to keep us on a continuous improvement cycle.

The story behind the Java deserialization vulnerability

The security community has known about deserialization vulnerabilities for a few years but they were considered to be theoretical and hard to exploit. Chris Frohoff (@frohoff) and Gabriel Lawrence (@gebl) shattered that notion back in January of 2015 with their talk at AppSecCali – they also released their payload generator (ysoserial) at the same time.

Unfortunately, their talk did not get enough attention among mainstream technical media. Many called it “the most underrated, under-hyped vulnerability of 2015”. It didn’t stay that way after November 2015, when security researchers at FoxGlove Security published their exploits for many products including WebSphere, JBOSS, WebLogic, etc.

This caught our attention, and we began forking a few work-streams to assess the impact to our application infrastructure.

Overview of the Java Deserialization bug

A custom deserialization method in Apache commons-collections contains a reflection logic that can be exploited to execute arbitrary code. Any Java application that accepts untrusted data to deserialize (aka marshalling or un-pickling) and has commons-collections in its classpath can be exploited to run arbitrary code.

The obvious quick fix was to patch commons-collections jar so that it does not contain the exploitable code. A quick search on our internal code repository showed us how complex this process could be – many different libraries and applications use many different versions of commons-collections and the transitive nature of how this gets invoked made it even more painful.

Also, we quickly realized that this is not just about commons-collections or even Java, but an entire class of vulnerability by itself and could occur to any of our other apps that perform un-pickling of arbitrary object types. We wanted to be surgical about this as patching hundreds of apps and services was truly a gargantuan effort.

Additionally, most financial institutions  have a holiday moratorium that essentially prevents any change or release to the production environment, and PayPal is no exception. We conducted our initial assessment on our core Java frameworks and found that we were not utilizing the vulnerable libraries and therefore had no immediate risk from exploit tools in the wild. We still had to review other applications that were not on our core Java frameworks in addition to our adjacencies.

The Bug Bounty Submission

Mark Litchfield, one of the top security researchers of our Bug Bounty program submitted a Remote Code Execution (RCE) using the above mentioned exploit generator on December 11, 2015. Mark has been an active member of PayPal Bug Bounty community since 2013. He has submitted 17 valid bugs and been featured on our Wall of Fame.

As it turned out, this bug was found in one of our apps that was not on our core Java frameworks. Once we validated the bug, the product development and security teams quickly moved to patch the application in a couple of days despite the holiday moratorium.

Below are Mark’s notes about this submission in his own words:

  • It was a Java deserialization vulnerability
  • Deserialization bugs were (as of a month ago) receiving a large amount of attention as it had been overlooked as being as bad as it was
  • There were nine different attack vectors within the app

While there were nine different instances of the root cause, the underlying vulnerability was just in one place and fixing that took care of fixing all instances.

Remediation – At Scale

The next few days were a whirlwind of activity across many PayPal security teams (AppSec, Incident Response, Forensics, Vulnerability Management, Bug Bounty and other teams) and our product development and infrastructure teams.

Below is a summary of the key learnings that we think would be beneficial to other security practitioners.

When you have multiple application stacks, core and non-core applications, COTS products, subsidiaries, etc. that you think are potentially vulnerable, where do you start?

Let real-world risk drive your priorities. You need to make sure that the critical risk assets get the first attention. Here are the different “phases” of activities that would help bring some structure when driving the remediation of such a large-scale and complex vulnerability.


  • If your organization has a good app inventory, it makes your life easy.
  • If you don’t, then start looking for products that have known exploit code like WebSphere, WebLogic, Jenkins, etc.
  • Validate the manual inventory with automated scans and make sure you have a solid list of servers to be patched.
  • For custom code, use static and dynamic analysis tools to determine the exposure.
  • If you have a centralized code repo, it definitely makes things easy.
  • Make sure you don’t miss one-off apps that are not on your core Java frameworks.
  • Additionally, reach out to the security and product contacts within all of your subsidiaries (if they are not fully integrated) and ask them to repeat the above steps.


Here are a few tools that are good to have in your arsenal:

  • SerializeKiller
  • Burp Extension from DirectDefense
  • Commercial tools that you may already have in-house such as static/dynamic analysis tools and scanners. If your vendors don’t have a rule-pack for scanning, push them to implement one quickly.

Additionally, don’t hesitate to get your hands dirty and write your own custom tools that specifically help you navigate your unique infrastructure.

Monitoring & Forensics

  • Implement monitoring until a full-fledged patch can be applied.
  • Signatures for Java deserialization can be created by analyzing the indicators of the vulnerability.
  • In this case, a rudimentary indicator is to parse network traffic for the hexadecimal string “AC ED 00 05” and in base64 encoded format “rO0”.
  • Whether you use open source or commercial IDS, you should be able to implement a rule to catch this. Refer to the free Snort rules released by Emerging Threats.
  • Given that this vulnerability has been in the wild for a while, do a thorough forensics analysis on all the systems that were potentially exposed.

Short-term Remediation

  • First up, drive the patching of your high-risk Internet facing systems – specifically COTS products that have published exploits.
  • For custom apps that use the Apache commons-collection, you can either remove the vulnerable classes if not used or upgrade to the latest version.
  • If the owners of those systems are different, a lot can be accomplished in parallel.

Long-term Remediation

  • Again, this vulnerability is not limited to commons-collections or Java per se. This is an entire class of vulnerability like XSS. Your approach should be holistic accordingly.
  • Your best bet is to completely turn off object serialization everywhere.
  • If that is not feasible, this post from Terse Systems has good guidance on dealing with custom code.


In closing, we understand that today’s application infrastructure is complex and you don’t own/control all the code that runs in your environment. This specific deserialization vulnerability is much larger than any of us initially anticipated – spanning across open source components, third-party commercial tools and our own custom code. Other organizations should treat this seriously as it can result in remote code execution and implement security controls for long-term remediation, and not stop just at patching the commons-collections library. To summarize:

  • We turned around to quickly patch known vulnerabilities to protect our customers
  • We are investing in tools and technologies that help inventory our applications, libraries and their dependencies in a much faster and smarter way
  • We are streamlining our application upgrade process to be even more agile in responding to such large-scale threats
  • We changed our policy on fixing security bug fixes during moratorium – we now mandate fixing not just P0 and P1 bugs – but also P2 bugs (e.g., internal apps that are not exposed to the Internet).


TLS Version and Cipher Suites Order Matter: Here’s Why.


As with a great many things, when it comes to internet security, the only constant is change. While the framework for secure web communication has been around since the development of SSL in 1994, the specific protocols and ciphers continue to evolve. In order to keep up with the changes, the InfoSec community must continually evaluate new potential threats in the context of security and ongoing usability of older systems. Just as system patches and OS upgrades are regularly released to fix known issues, new and improved protocols and cipher suites are developed that correct inherent flaws and mitigate new threats. However, no matter how well designed they are, vulnerabilities and weaknesses will emerge during the lifespan of any protocol or cipher suite, so PayPal pays close attention to those threats as they could affect both our customers and us.

PayPal recently reevaluated the TLS cipher suites for its www.paypal.com web site after assessing the numerous clients visiting the site (from legacy ones to the most modern configurations). We currently support TLS versions 1.2 to 1.0, prioritized in that order, and our cipher suites selection and prioritization is based on factors such as availability, business needs, security, and compliance requirements. Efficiency is also a factor, as some ciphers can slow down transactions because their algorithms are more demanding. In some cases, a company may opt for increased speed over a more secure connection. Since we are entrusted with your financial transactions and personal information, however, we lean toward security over speed.

In order to set up a secure connection between a server and a client via TLS, both parties must be capable of running the same version of the TLS protocol and have common cipher suites installed. To initiate the process, the client (e.g. a web browser) advertises, to the server, the TLS versions and cipher suites it supports. The server, when deciding on the cipher suite that will be used for the TLS connection, may give the priority to the client’s cipher suites list (picking the first one it also supports) OR it may choose to prioritize its own list (picking the first one in its list that the client supports). At PayPal we prefer the latter process as it allows us to ensure we set up the most secure communication channel possible.

More specifically, PayPal now prioritizes ciphers that leverage Elliptic Curves and the Diffie–Hellman key exchange mechanism over RSA (namely ECDHE based cipher suites). In addition, we prioritized ECDHE because it leverages the concept of forward secrecy. The last E in ECDHE stands for ephemeral. It means that the generated key pair is temporary and a new set will be generated during each handshake. This is an important property that prevents the decryption of previous TLS conversations even in the case of a stolen private (server) key. This makes it the most secure mode of key exchange to date. As more clients are supporting ECDHE, we prioritize it so that we connect at the most secure level possible.

Despite these advances, we do continue allowing older cipher suites to be used when a client can’t negotiate a more secure connection. We do so in order to support users running legacy platforms like Windows XP and IE6. Fortunately, the number of outdated client is dwindling and we’ll soon cut them off entirely. We’re currently ending support for outdated cipher suites such as RC4 that are only connecting at a rate of roughly one in one million; it’s just a matter of time before we make the next cut.

During our routine investigation of our TLS version and cipher suites order we learned a great deal about how many of our customers are currently able to support higher security TLS connections, helping us plan for future upgrades. This research not only helps increase security, but also minimizes disruption in the customer experience.

In the coming weeks we will publish more of our research on how we determined the optimal TLS version and cipher suites ordering. We hope to help others learn from our experience and improve security for everyone in the online ecosystem. Until then, remember to pay with PayPal to stay safe and secure!

PayPal Sponsors First of Its Kind Intel Capture the Flag Contest at DEFCON 23


DEFCON routinely presents the coolest and most thought provoking topics in the hacking community and this year did not disappoint, partially due to the first PayPal-sponsored Intel Capture the Flag (CTF) virtual manhunt contest. IntelCTF events challenge players to utilize their open source intelligence (OSINT) forensic skills in order to identify malicious actors intent on Internet mayhem. Players find strategically placed “flags” that are planted across the Internet as breadcrumbs, allowing them to solve the e-case of whodunit by simply connecting the virtual dots.

This contest, (rated Beginner/Intermediate) which is the first of several that are scheduled for release in the near future, tasked participants with identifying an actor who defaced a rather “popular” webpage. Team participants may use any means necessary to track and identify the perpetrator. It touted 17 flags of increasing difficulty and asked questions such as the timestamp of when certain posts were made, how a website was hacked, the owner of the proxy service the actor was using, and finally, the defacer’s real identity.

Various members of the PayPal Information Security team in Scottsdale, Arizona partnered with several alpha/beta testers to run the 6 hour, 24-team event. The event started a bit slowly as players adjusted to the gaming style but at around the 30-minute mark the competition heated up! The Attribution-Team and Killjoys were neck-in-neck for flags 11 and 12. The Attribution-Team ended up capturing the 13th flag before Killjoys in the final hour. After six long hours of competition, the scoring engine was shut down and the event concluded. The Attribution-Team submitted a write-up detailing their investigative process. It was then reviewed by the IntelCTF team confirming no cheating or flag brute forcing occurred. The IntelCTF team confirmed that the Attribution-Team was the winning team, capturing thirteen out of seventeen flags before the others and winning the $500 USD prize. Below is a snapshot of the top five contenders and the final scoreboard for all teams:

  1. Attribution-Team – 13 flags and $500 USD prize
  2. Killjoys – 13 flags
  3. BAMFBadgers – 12 flags
  4. I tried… but failed – 12 flags
  5. StenoPlasma – 11 flags

Screen Shot 2015-08-19 at 11.48.45 AM

This event was different than past contests because IntelCTF is the first of its kind! Traditional CTFs focus more on WebApp pentesting, reverse-engineering, forensics, and programming challenges where as IntelCTF immerses participants into simulated scenarios of tracking down information about the tools, techniques, capability, and identity of malicious individuals. It generated a lot of interest and excitement and feedback from participants was very positive. Not only did participants thoroughly enjoy the immersive challenge but also asked when IntelCTF would be running the next event and where.

Think you have what it takes to capture the flag? Check out these upcoming IntelCTF challenges and keep your eye out for more PayPal-sponsored information security events.


RFC Issued for DMARC


The PayPal Product and Ecosystem Security Team is pleased to announce that on March 18, 2015, the Internet Engineering Task Force (IETF) published RFC 7489 for Domain-based Message Authentication Reporting and Conformance (DMARC).

Starting in 2009, PayPal began work with several major mailbox providers to explore the effectiveness of combining common email authentication technologies in order to improve their effectiveness. Soon, other companies sharing our need to secure email as a trusted channel of communication joined the collaboration. In 2011, the experimental solution was proven effective at combatting spoofed domain email attacks, those types of attacks where the malicious actor intentionally tries to send email masquerading as coming from another domain. At this point, the group made the DMARC specification available to the public. As a review of the 3rd year since it was made public, we detailed DMARC’s broadening adoption and performance in a post on our PayPal Forward on February 18, 2015.

DMARC is essentially a policy layer that builds on top of Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM). Taken together, SPF (server authorization), DKIM (message integrity), and DMARC form a robust system designed to authenticate email has having been sent from or on behalf of the claimed domain. Further, DMARC enables senders to tell receivers what to do when a message fails to authenticate and requests that receivers send information back to senders about those failures. DMARC’s policies are published in the public DNS and are available for anyone to use. The system removes guesswork from the receiver’s detection of spoofed domain messages, how to handle them, and nearly eliminates the user’s exposure to potentially fraudulent & harmful messages spoofing a protected domain.

While the work to review and update the specification will continue as needed via the IETF DMARC Work Group, the publication of RFC 7489 was a critical step in developing a stable reference point for DMARC’s credibility. Steps like this illustrate our commit to, and the value of, open standards to protect email as a trusted means of communication.

For more technical information: