DMARC-Related Recommendations Included in NIST Guidance on Trustworthy Email

By and

Another important milestone was recently achieved for Domain-based Message Authentication Reporting and Conformance (DMARC), one of the PayPal Ecosystem Security team’s major undertakings in making the internet a safer, more secure place. After several years of collaboration with the email security community, the U.S. National Institute of Standards and Technology (NIST) included recommendations for supporting DMARC in NIST’s SP 800-177, Trustworthy Email. SP 800-177 was released in September and is intended to give recommendation and guidelines for enhancing trust in email. While the audience for NIST publications is typically US federal agencies, its guidance does tend to influence other global… Read more

From Big Data to Fast Data in Four Weeks or How Reactive Programming is Changing the World – Part 2

By

Part 2: Lambda Architecture meets reality Part 1 can be found here. Fast Data Fast forward to December 2015. We have a cross data center Kafka clusters, we have Spark adoption through the roof. All of this, however, was to fix our traditional batch platform. I’m not going to pretend we never thought about real-time stuff. We’d been gearing up toward the Lambda architecture all along, but truly we were not working specifically for the sake of the near real-time analytics. The beauty of our current stack and skill set is that streaming just comes with it. All we needed to do… Read more

Carrier Payments Big Data Pipeline using Apache Storm

By

Carrier payments is a frictionless payment method enabling users to place charges for digital goods directly on their monthly mobile phone bill. There is no account needed, just the phone number. Payment authorization happens by verification of a four digit PIN sent via SMS to a user’s mobile phone. After the successful payment transaction, charges will appears on user’s monthly mobile phone bill. Historically fraud has been handled on the mobile carrier side through various types of spending caps (daily, weekly, monthly, etc.). While these spending caps were able to keep fraud at bay in the early years, as this… Read more

From Big Data to Fast Data in Four Weeks or How Reactive Programming is Changing the World – Part 1

By

Part 1: Reactive Manifesto’s Invisible Hand Let me first setup the context for my story. I’ve been with PayPal for 5-years. I’m an architect. I’m part of the team responsible for PayPal Tracking domain. Tracking is commonly and historically understood as the measurement of customer visits to web pages. With the customer’s permission our platform collects all kinds of signals from PayPal web pages, mobile apps and services, for variety of reasons. Most prominent among them are measuring new product adoptions, A/B testing, and fraud analysis. We collect several terabytes of data on our Hadoop systems every day. This is… Read more

Python by the C side

By

Mahmoud’s note: This will be my last post on the PayPal Engineering blog. If you’ve enjoyed this sort of content subscribe to my blog/pythondoeswhat.com or follow me on Twitter. It’s been fun! All the world is legacy code, and there is always another, lower layer to peel away. These realities cause developers around the world to go on regular pilgrimage, from the terra firma of Python to the coasts of C. From zlib to SQLite to OpenSSL, whether pursuing speed, efficiency, or features, the waters are powerful, and often choppy. The good news is, when you’re writing Python, C interactions… Read more

Spark in Flames – Profiling Spark Applications Using Flame Graphs

By

When your organization runs multiple jobs on a Spark cluster, resource utilization becomes a priority. Ideally, computations receive sufficient resources to complete in an acceptable time and release resources for other work. In order to make sure applications do not waste any resources, we want to profile their threads to try and spot any problematic code. Common profiling methods are difficult to apply to a distributed application running on a cluster. This post suggests an approach to profiling Spark applications. The form of thread profiling used is sampling – capturing stack traces and aggregating these stack traces into meaningful data, in this case displayed… Read more

Python Packaging at PayPal

By

Year after year, Pythonists all over are churning out more code than ever. People are learning, the ecosystem is flourishing, and everything is running smoothly, right up until packaging. Packaging Python is fundamentally un-Pythonic. It can be a tough lesson to learn, but across all environments and applications, there is no one obvious, right way to deploy. Frankly, it’s hard to think of an area where Python’s Zen applies less. At PayPal, we write and deploy our fair share of Python, and we wanted to devote a couple minutes to our story and give credit where credit is due. For… Read more

Powering Transactions Search with Elastic – Learnings from the Field

By

Introduction We see a lot of transactions at PayPal. Millions every day. These transactions originate externally (a customer using PayPal to pay for a purchase on a website) as well as internally, as money moves through our system. Regular reports of these transactions are delivered to merchants in the form of a csv or a pdf file. Merchants use these reports to reconcile their books. Recently, we set out to build a REST API that could return transaction data back to merchants. We also wanted to offer the capability to filter on different criteria such as name, email or transaction… Read more

Interning @ PayPal: Checkout A/B Testing, Developing Features, and Cracking Bugs

By

My internship at PayPal was a great experience. I was given real work that mattered. From day one, I had the opportunity to continuously write, commit, and push production level code that impacted the millions of people who use PayPal Checkout. As a Software Engineering Intern on PayPal’s Checkout Guest and Signup team, I focused on building and iterating A/B tests to improve customers’ experiences and onboard new users. Within a few weeks of joining, I developed a solid enough understanding of our frontend and backend codebase to fix several critical production bugs: everything from updating password tooltip feedback to risk validation fixes.… Read more

Stop by PayPal’s Booth in the Black Hat Career Zone to Talk Security and Learn about REAPER!

By

This year’s Black Hat conference is a big one for PayPal because it is the first time we are attending as a conference sponsor. In 2016 we’ve made a concerted effort to show up at a number of events to discuss security careers and hear about the experiences of you, our peers and colleagues. I will be at the booth along with several other PayPal InfoSec professionals from various security disciplines on August 3rd and 4th. Along with various swag and security puzzles, we want to highlight some interesting academic research conducted this year by our Threat Intelligence team. REAPER… Read more