Monthly Archives: May 2016

Benching Microbenchmarks


In under one week, Statistics for Software flew past 10 Myths for Enterprise Python to become the most visited post in the history of the PayPal Engineering blog. And that’s not counting the Japanese translation. Taken as an indicator of increased interest in software quality, this really floats all boats.

That said, there were enough emails and comments to call for a quick followup about one particularly troubling area.

Statistics for benchmarks

The saying in software goes that there are lies, damned lies, and software benchmarks.

A tachometer (an instrument indicating how hard an engine is working)

Too much software is built without the most basic instrumentation.

Yes, quantiles, histograms, and other fundamentals covered in Statistics for Software can certainly be applied to improve benchmarking. One of the timely inspirations for the post was our experience with a major network appliance vendor selling 5-figure machines, without providing or even measuring latency in quantiles. Just throughput averages.

To fix this, we gave them a Jupyter notebook that drove test traffic, and a second notebook provided the numbers they should have measured. We’ve amalgamated elements of both into a single notebook on PayPal’s Github. Two weeks later they had a new firmware build that sped up our typical traffic’s 99th percentile by two orders of magnitude. Google, Amazon, and their other customers will probably get the fixes in a few weeks, too. Meanwhile, we’re still waiting on our gourmet cheese basket.

Even though our benchmarks were simple, they were specific to the use case, and utilized robust statistics. But even the most robust statistics won’t solve the real problem: systematic overapplication of one or two microbenchmarks across all use cases. We must move forward, to a more modern view.

Performance as a feature

Any framework or application branding itself as performant must include measurement instrumentation as an active interface. One cannot simply benchmark once and claim performance forever.1 Applications vary widely. There is no performance-critical situation where measurement is not also necessary. Instead, we see a glut of microframeworks, throwing out even the most obvious features in the name of speed.

Speed is not a built-in property. Yes, Formula 1 race cars are fast and yes, F1 designers are very focused on weight reduction. But they are not shaving off grams to set weight records. The F1 engineers are making room for more safety, metrics, and alerting. Once upon a time, this was not possible, but technology has come a long way since last century. So it goes with software.

To honestly claim performance on a featuresheet, a modern framework must provide a fast, reliable, and resource-conscious measurement subsystem, as well as a clear API for accessing the measurements. These are good uses of your server cycles. PayPal’s internal Python framework does all of this on top of SuPPort, faststat, and lithoxyl.

Benching the microbenchmark

An ECHO-brand ping pong paddle and ball.

Enough with the games already. They’re noisy and not even that fun.2

Microbenchmarks were already showing signs of fatigue. Strike one was the frequent lack of reproducibility. Strike two came when software authors began gaming the system, changing what was written to beat the benchmark. Now, microbenchmarks have officially struck out. Echos and ping-pongs are worth less than their namesakes.

Standard profiling and optimization techniques, such as those chronicled in Enterprise Software with Python, still have their place for engineering performance. But those measurements are provisional and temporary. Today, we need software that provides idiomatic facilities for live measurement every individual system’s true performance.

  1. I’m not naming names. Yet. You can follow me on Twitter in case that changes. 
  2. Line art by Frank Santoro.

Creating a Subscription Service with PayPal (Part 2 of 2): The Billing Agreement


This is part 2 of creating a subscription service. This second step to creating a subscription for a user is to create and execute a billing agreement, based on an existing activated billing plan. This tutorial assumes that you have already gone through and activated a Billing Plan in part 1, and have an ID for that billing plan to reference in the example.

If you want to jump ahead and just get a complete example for parts 1 & 2, you can get it from the PayPal Developer Github repository.

When you are setting up a billing agreement to create a subscription for a user, you’ll follow 3 steps, which you may find reminiscent to processing a standard PayPal payment:

  1. You create a billing agreement, referencing an underlying billing plan via the ID.
  2. Once created, you redirect the user to PayPal (if paying via PayPal) to confirm the subscription. Once confirmed, PayPal redirects the user back to your site using the redirect provided in the underlying billing plan.
  3. You then execute the billing agreement using a token provided back via the PayPal redirect.

This example is setting up an Express based HTTP server to showcase the billing agreement process.

To start the example, we first need to set up our configuration. We add four requirements, the PayPal SDK, body-parser for handling JSON encoded bodies, http for our simple server integration, and express for the Express framework. We then define our client ID and secret from creating an application, configure the SDK for the sandbox, then configure bodyParser for handling JSON bodies.

var paypal = require('paypal-rest-sdk'),
    bodyParser = require('body-parser'),
    http = require('http'),
    app = require('express')();


  'mode': 'sandbox', //sandbox or live
  'client_id': clientId,
  'client_secret': secret


Next up, we need to create a route to handle the creation of a billing agreement and redirect the user to PayPal to confirm that subscription. We are assuming that a billing plan ID is passed as a query string parameter, such as by loading the following URL with a plan ID from the previous example:


We now need to use that information to create the billing agreement.

app.get('/createagreement', function(req, res){
    var billingPlan = req.query.plan;

    var isoDate = new Date();
    isoDate.setSeconds(isoDate.getSeconds() + 4);
    isoDate.toISOString().slice(0, 19) + 'Z';

    var billingAgreementAttributes = {
        "name": "Standard Membership",
        "description": "Food of the World Club Standard Membership",
        "start_date": isoDate,
        "plan": {
            "id": billingPlan
        "payer": {
            "payment_method": "paypal"
        "shipping_address": {
            "line1": "W 34th St",
            "city": "New York",
            "state": "NY",
            "postal_code": "10001",
            "country_code": "US"

    // Use activated billing plan to create agreement
    paypal.billingAgreement.create(billingAgreementAttributes, function (
        error, billingAgreement){
        if (error) {
            throw error;
        } else {
            //capture HATEOAS links
            var links = {};
                links[linkObj.rel] = {
                    'href': linkObj.href,
                    'method': linkObj.method

            //if redirect url present, redirect user
            if (links.hasOwnProperty('approval_url')){
            } else {
                console.error('no redirect URI present');

We start by extracting the billing plan ID from the query string and create the date when the plan should start.

The next object definition, billingAgreementAttributes, consists of information for the subscription. It contains readable information on the plan, a reference to the billing plan ID, the payment method, and shipping details (if needed for the subscription).

Next, a call to billingAgreement.create(...) is made, passing in the billingAgreementAttributes object we just created. If all is successful, we should have a billing agreement object passed back to us containing details about our newly created subscription. That object also contains a number of HATEOAS links providing us next steps that can be taken on this newly created agreement. The one we care about here is labeled as approval_url.

We loop through all provided links to put them into an easily referenced object. If approval_url is one of those links, we redirect the user to that link, which is PayPal.

At this point the user confirms the subscription on PayPal, and is redirected back to the URL provided in the underlying billing plan. Along with that URL, PayPal will also pass a token along the query string. That token is what we’re going to use to execute (or start) the subscription.

Let’s set up that functionality in the following route.

app.get('/processagreement', function(req, res){
    var token = req.query.token;

    paypal.billingAgreement.execute(token, {}, function (error, 
        billingAgreement) {
        if (error) {
            throw error;
        } else {
            res.send('Billing Agreement Created Successfully');

We extract the token from the query string, then make a call to billingAgreement.execute, passing along that token. If all is successful, we now have a valid subscription for the user. The return object contains information about the active billing agreement.

Lastly, we set up our HTTP server to listen for traffic to our routes.

//create server
http.createServer(app).listen(3000, function () {
   console.log('Server started: Listening on port 3000');

With our billing plan in place, we are able to create a master subscription model for our services. Using that plan, we can then subscribe multiple users via an agreement, which allows us to scale and adjust our plans (and multiple users attached to that plan) very easily.

Creating a Subscription Service with PayPal (Part 1 of 2): The Billing Plan


This is the first of a two part tutorial on creating a subscription model. Over the next two posts, we will cover everything you’ll need to get started with subscriptions using the PayPal REST APIs:

  • Creating billing plans to build a multi-use model for your subscription payments:
  • Subscribing users to those billing plans via a billing agreement.

If you want to jump ahead and just get a complete example for parts 1 & 2, you can get it from the PayPal Developer Github repository.

When creating a subscription for a user, you first need to create and activate a billing plan that a user is then subscribed to using a billing agreement. The complete process for creating a subscription is to:

  • Create a billing plan. This is a reusable model that outlines the details of the subscription.
  • Activate the billing plan.
  • When you want to create a subscription for a user, after the user has chosen that plan, you create a billing agreement using the ID of the billing plan that they should be subscribed to.
  • Once created, you redirect the user to PayPal to confirm the subscription. Once confirmed, the user is redirected back to the merchant’s website.
  • Lastly, you execute the billing agreement to begin the subscription.

Within this example, we’re going to be using the PayPal Node SDK. You can obtain it from NPM using the following command:

npm install paypal-rest-sdk

Within our .js file, we first set up our SDK configuration, which includes adding a requirement for the SDK, defining our client ID and secret from creating our application, and then configuring the SDK for the sandbox environment.

var paypal = require('paypal-rest-sdk');

var clientId = 'YOUR CLIENT ID';
var secret = 'YOUR SECRET';

  'mode': 'sandbox', //sandbox or live
  'client_id': clientId,
  'client_secret': secret

Next, we need to set up two JSON objects. The billingPlanAttribs object contains the information and payment breakdown for the billing plan that we can subscribe users to, and the billingPlanUpdateAttributes object contains values for setting the billing plan to an active state, allowing it to be used.

var billingPlanAttribs = {
    "name": "Food of the World Club Membership: Standard",
    "description": "Monthly plan for getting the t-shirt of the month.",
    "type": "fixed",
    "payment_definitions": [{
        "name": "Standard Plan",
        "type": "REGULAR",
        "frequency_interval": "1",
        "frequency": "MONTH",
        "cycles": "11",
        "amount": {
            "currency": "USD",
            "value": "19.99"
    "merchant_preferences": {
        "setup_fee": {
            "currency": "USD",
            "value": "1"
        "cancel_url": "http://localhost:3000/cancel",
        "return_url": "http://localhost:3000/processagreement",
        "max_fail_attempts": "0",
        "auto_bill_amount": "YES",
        "initial_fail_amount_action": "CONTINUE"

var billingPlanUpdateAttributes = [{
    "op": "replace",
    "path": "/",
    "value": {
        "state": "ACTIVE"

Within the billingPlanAttribs object, there are some relevant pieces of information:

  • name / description / type: Basic visual information to describe the plan, and the type of plan.
  • payment_definitions: Information on how the plan should function and be billed. More details on fields here.
  • merchant_preferences: Additional fee structures, redirect URLs, and settings for the subscription plan. More details on fields here.

With those objects in place, we can now create and activate the billing plan.

paypal.billingPlan.create(billingPlanAttribs, function (error, billingPlan){
    if (error){
        throw error;
    } else {
        // Activate the plan by changing status to Active
        paypal.billingPlan.update(, billingPlanUpdateAttributes, 
            function(error, response){
            if (error) {
                throw error;
            } else {

We call billingPlan.create(...), passing in the billingPlanAttribs object that we just created. If that is successful, the return object will contain information about the billing plan. For the sake of the example, we just need to use the billing plan ID in order to activate the plan for use.

Next, we call billingPlan.update(...), passing in the billing plan ID and the billingPlanUpdateAttributes object we created earlier. If that is successful, our billing plan is now active and ready to use.

In order to create a subscription for a user (or multiple users) on this plan, we’ll need to reference the billing plan id ( above), so store that in a place that can be referenced easily.

In the second subscription step, we need to create a billing agreement based on the plan we just created and execute it to begin processing subscriptions for a user.

Stay tuned for part 2…

Processing a Credit Card Payment with a Vaulted Card


In this tutorial, we’ll be looking at a Node example to show how to store a credit card using the PayPal vault, then reference that stored credit card to process a credit card transaction for a user.

The reason why we would want to use the vault is so that we don’t have to store sensitive credit card information on our own servers. We simply reference the payment method via a provided vault ID, meaning that we don’t have to deal with many PCI compliance regulations with storing the credit cards ourselves.

For the full example code used in the below tutorial, go to the PayPal Developer Github account.

Our first step is to require the packages that we need, and configure our environment.

var paypal = require('paypal-rest-sdk'),
    uuid = require('node-uuid');

var client_id = 'YOUR CLIENT ID';
var secret = 'YOUR SECRET';

  'mode': 'sandbox', //sandbox or live
  'client_id': client_id,
  'client_secret': secret

We add the requirement for the PayPal SDK & a uuid package (more on that in a moment), then set up variables for the client ID and secret that were obtained when creating an application. We then configure our application using these details, and specify the environment that we are working in (live or sandbox). We are using the node-uuid package in order to be able to generate unique UUID’s for the payers when storing the card. An alternative for this is to use your unique customer IDs in place of this functionality. You can install that package via:

npm install node-uuid

Next, we define the credit card JSON object that will be sent to the PayPal vault for storage. It contains information from the card, as well as a unique payer ID that we generate using node-uuid. You should store this unique payer_id in your own database as it will be used when creating a payment with the vaulted card.

var create_card_details = {
    "type": "visa",
    "number": "4417119669820331",
    "expire_month": "11",
    "expire_year": "2018",
    "first_name": "John",
    "last_name": "Doe",
    "payer_id": uuid.v4()

Lastly, we need to store the credit card and process the payment using that card. To vault a credit card, we call credit_card.create(...), passing in the credit_card_details object that we just created. If all goes well, we should have an object returned to us with details about the vaulted card. For the sake of a payment with that card, we only really need two pieces of information: the payer_id that we already stored, and the vault ID, that should also be stored as a reference in our own database.

paypal.credit_card.create(create_card_details, function(error, credit_card){
    } else {
        var card_data = {
            "intent": "sale",
            "payer": {
                "payment_method": "credit_card",
                "funding_instruments": [{
                    "credit_card_token": {
                        "payer_id": credit_card.payer_id
            "transactions": [{
                "amount": {
                    "total": "7.47",
                    "currency": "USD",
                    "details": {
                        "subtotal": "7.41",
                        "tax": "0.03",
                        "shipping": "0.03"
                "description": "This is the payment transaction description." 

        paypal.payment.create(card_data, function(error, payment){
            } else {

In the section following the successful vaulting of the credit card, we then define our card_data object in order to process the payment. The main difference between a regular card payment (where you add in the card details) and processing via a vaulted payment in the structure of the card_data object, is the funding_instruments section, that we define under payer. Instead of defining the credit card information, we instead use the following object that contains the vault ID reference, and the payer ID:

"credit_card_token": {
    "payer_id": credit_card.payer_id

Other than that, we simply set an intent of sale and define our transactions object, which details the payment to be made.

Lastly, we make a request to payment.create(...), passing in the card_data object that we just defined. If all is successful, we have now just vaulted a card and immediately processed a payment with it.

Dates Revised for Security-Related Changes


We’ve revised some of the dates originally published in the Security-Related Changes Required to Avoid Service Disruption post from earlier this year.

The dates have been revised in order to give merchants additional time to plan for any necessary changes and to give developers additional time to update their integrations.

The dates in the original post reflect the current deadlines. In summary, here are the date revisions:

  • The TLS1.2 Upgrade date moves to June 2017.
  • The HTTPS for IPN Postback date moves to June 2017.
  • The SHA-256 Upgrade dates: Merchants must be compliant by September 30, 2016 as we will begin our permanent upgrade of the remaining endpoints in October 2016. Take action by June 17, 2016 to verify if your system is SHA-256 compliant.
  • The SFTP Server move dates remain the same.
  • The API Authentication Credential Certificate dates remain the same.

Developers should continue to reference the 2017-2018 Merchant Security Roadmap microsite for all the latest details and up-to-date information regarding these security changes.

Adam Colson

Author: Adam Colson

About the author: Adam is a Product Manager at Braintree | PayPal, focusing on the PayPal developer experience since August 2015. When Adam isn’t helping to enable global e-commerce through APIs, he can also be found hacking away at the Internet of Things or tinkering in his garage.

@adamc | LinkedIn

squbs: packaging and deployment instructions to run on AWS nodes



This page describes a quick way to package, deploy, and start a squbs application. This guide uses Amazon EC2 as an example, showing how to run a squbs application in a few minutes.

You can leverage either the scala activator template or the java activator template to begin development.


You need to install the following on your build instance

Steps to build:

  • Clone the source code from the git repo to the <project> directory
  • cd <project>
  • Run the sbt build command, including “packArchive”, such as: sbt clean update test packArchive
  • There are two archives created under <project>/target
  • <app>-<version>.tar.gz
  • <app>-<version>.zip


You need to install the following on your running instance

Steps to run:

  • Copy either of the archives to the running instance
  • <app>-<version>.tar.gz
  • <app>-<version>.zip
  • For example, explode the tarball tar zxvf <app>-<version>.tar.gz to the <app>-<version> directory
  • start the application <app>-<version>/bin/run &
  • You can check the admin http://localhost:8080/adm from that instance, or http://<host>:8080/adm


You can terminate the running process, for example, in linux kill $(lsof -ti TCP:8080 | head -1) Since the application registers a shutdown hook with the JVM, it will shutdown gracefully, unless it is abrupt.

Amazon EC2

Log into AWS EC2 and launch an instance

  • You can create from free-tier, if the capacity meet your needs
  • Security group open (inbound) SSH – port 22, Custom TCP Rule – 8080
  • SSH into server (see AWS Console -> Instances -> Actions -> Connect)
  • Execute step Start and Shutdown as described above

Visit us at

Security-Related Changes Required to Avoid Service Disruption


REVISED May 12, 2016Please note that some of the deadlines have changed since this post was originally published. This post reflects the latest deadline dates for security updates.

Beginning in early 2016, PayPal will introduce a number of security-related product updates.

These updates are part of an industry-wide initiative to improve security standards. Some of these updates, like the TLS upgrade, are mandated by the PCI Security Council and are required by every website that transmits or processes cardholder data.

Merchants and developers may have to update their integrations in order to be in compliance and ensure that their applications continue to function as expected.

For PayPal customers, these updates include:

TLS 1.2 Upgrade

The most secure protocol for sharing information on the web today is Transport Layer Security (TLS) version 1.2. PayPal is enabling support for TLS 1.2 for all secure connections and in 2016 will start requiring its use. You will need to verify that your environment supports TLS 1.2 and if necessary make appropriate updates. PayPal is updating its services to require TLS v1.2 for all HTTPS connections in June of 2017. After that time, all TLS v1.0 and TLS v1.1 API connections will be refused.

Learn more

IP Address Update for PayPal SFTP

If your integration is set-up to systematically exchange files with PayPal’s Secure FTP Reporting / Batch Servers, please note that the IP addresses for these servers are changing. If your integration is hardcoded to the current IP addresses, you will need to upgrade accordingly. You must act by April 14, 2016.

Learn more

IPN Verification Postback to HTTPS

If you are using PayPal’s Instant Payment Notification (IPN) service, you will need to ensure that HTTPS is used when posting the message back to PayPal for verification. After June of 2017 HTTP postbacks will no longer be supported.

Learn more

Merchant API Credential Upgrade

The certificates issued by PayPal as API Credentials for use with the Classic API are being upgraded to SHA-256 signed 2048-bit certificates. If you currently connect to PayPal using Certificate API Credentials, you will need to have a new certificate issued and start using it for all API requests. Depending on when your certificate expires, you will have to upgrade between Jan 31, 2016 and Jan 1, 2018.

Learn more

SSL Certificate Upgrade

PayPal is in the process of upgrading the SSL certificates used to secure our web sites and API endpoints. These new certificates will be signed using the SHA-256 algorithm and VeriSign’s 2048-bit G5 Root Certificate. You will need to ensure that your environment supports the use of the SHA-256 signing algorithm and discontinue the use of SSL connections that rely on the VeriSign G2 Root Certificate. This action must be taken by June 17, 2016 in order to avoid any disruption of service.

Learn more

PayPal SDK Updates

For developers using one of PayPal’s SDK libraries, depending on the runtime environment and version your application uses, a code change or SDK update may be required in order to enable TLS v1.2. Java 6 u115 b32, Java 7 or Java 8 (preferred) and Android 4.x environments require TLS v1.2 to be enabled in order to maintain functionality.

If you’re using one of PayPal’s REST SDKs, the best practice is to stay up to date with the latest SDK version and latest versions of the code libraries required by the PayPal SDK. PayPal provides server and client SDKs for Java, .Net, PHP, Python, Ruby, Node.js, Android, and iOS. Detailed test instructions can also be found at

Developers should reference the 2017-2018 Merchant Security Roadmap microsite for all the latest details and up-to-date information regarding these security changes. You can also visit the PayPal Developer site for more information on PayPal security guidelines and best practices.

Adam Colson

Author: Adam Colson

About the author: Adam is a Product Manager at Braintree | PayPal, focusing on the PayPal developer experience since August 2015. When Adam isn’t helping to enable global e-commerce through APIs, he can also be found hacking away at the Internet of Things or tinkering in his garage.

@adamc | LinkedIn

squbs: A New, Reactive Way for PayPal to Build Applications



It is not uncommon for services in PayPal to cover 1000 VMs or more. These services make use of very small VMs and produce very low throughput for each VM. At the same time, the large number of nodes takes a toll on the network and routing infrastructure. Several of these services are interconnected into a complicated mesh, making a user request travel through many network hops. As the number of these services adds up, latency gradually increases and the user experience deteriorates.

While it is good for a service to have a critical mass of VMs spread across many data centers for redundancy, additional VMs beyond the critical mass have diminishing returns. There is an inherent cost to too many services spanning hundreds of VMs, in terms of management and monitoring, ineffective caching, but more importantly in terms of agility. It may take from a few minutes, up to an hour to roll out a new version of the service across 100 VMs. It takes ten times longer to roll out 1000 VMs.

We have been riding the single-thread performance increases of Moore’s law for many decades now, but the trends slowed down since about 2005. Unless there are breakthroughs in alternate technologies like quantum computing, we are at the limit of transistor density and clock speed. This means newer processors do not necessarily make your single-threaded applications run faster any longer. The trend on power savings drove the microprocessor industry to provide more processors at lower clock speeds. This now means applications will have to adapt to making use of more CPUs per VM.

The industry cries “micro-services” out loud. But what are micro-services? When would a service be considered a micro-service? When would it become a nano-service ? And how will engineers and organizations know what the right service boundary is? We heard that micro-services are services that do only one thing. Does this mean if an organization does ten thousand things we need to have ten thousand services? Perhaps. The truth is, services grow organically and are aligned more with organizations than with their function in the grand scheme. A re-org splits an engineering team into two and, quite often, the one service owned by that team is now split into two services. This means even micro-services should be structured to be modular and be able to adapt to organizational re-structuring. In essence, you may want a micro-service to be built upon a multitude of these loosely-coupled nano-services.

Lastly, even our micro-services or nano-services get complicated and hard to maintain. We need modern programming techniques that allow us to quickly build and assemble these services. Ideally, there should be good visibility into that one thing a service does and you should not have to dig into layers and layers of code to figure it out.

These set of problems and requirements drive us to look for a next-generation framework or infrastructure, and include:

  1. Scalable, both horizontally to hundreds of nodes and vertically to very many processors, handling billions of requests per day
  2. Low latency, controllable at a very fine grain
  3. Resilient to failure
  4. Flexibility in adjusting the service boundaries
  5. A programming model AND culture encouraging scalability and simplicity, including clean failure and error handling

Based on these requirements, we quickly narrowed down our options and turned to the Actor Model of Computation for its ultimate scalability. Being a message-based system also allows us to control latency at very good granularity as opposed to deeply stack-based systems. There are two prominent players: Erlang and Akka. Erlang has some very prominent properties, especially the runtime upgradability, and can be a good choice. But PayPal already has a significant investment in the Java Virtual Machine (JVM). Growing a new stack from scratch in this environment is extremely difficult with its many operational hooks and security requirements. Having many of these hooks ready as JVM libraries, for good or worse, does significantly help with cost and time-to-market. Therefore, we decided to use Akka and Spray as the http library as it fully honors the Akka actor and execution model.

The unique mix of functional programming and the actor model in Akka (and definitely Erlang, too) allowed us to write code that is easy to reason about, easy to test, and especially easy to handle errors and failure scenarios when compared to the traditional model used on the JVM. This is a great benefit allowing faster, resilient, and simpler code with streamlined error handling and fewer bugs.

Because Akka and Spray are in the form of libraries and provide the ultimate flexibility to whatever you want to build, it unfortunately allows every service to build their ecosystem from scratch. This would lack standardization and manageability across many of these services we want to grow, causing these services to only become “specialty” services, each with its own way of deployment and management. We looked at Play as an alternative. While it is extremely simple, it did not natively follow the Akka message-based APIs and conventions but rather allow the use of Akka on top of Play.

A Mini-Introduction to squbs

A new stack called “squbs”(spelled in all lower case with the pronunciation rhyming with “cubes”) makes use of the loose coupling already provided by actors. It creates a modular layer (for your nano-services) called “cubes” that are symmetric to other cubes. Unlike libraries with concrete dependencies at the API layers, cubes ride on the actor system and only expose the messaging interface already provided in Akka. The interdependency between cubes are loose and symmetric. It is not hard to see the roots of the name “squbs” from these concepts and properties.

There are only a few principles coming together for designing squbs:

  1. It must be extremely lightweight with no measurable performance deficit over a similar Akka application built from scratch.
  2. New APIs over the Akka APIs are based on absolute necessity. Developers should not need to learn any squbs API or message protocol to build services or applications on squbs. The knowledge base needed to build squbs applications should be the Akka knowledge base which developers can acquire from training, documentation, and forums available on the Internet.
  3. It should be open source from the core up, with hooks for plugging in PayPal operationalization that cannot be open sourced.

With this, squbs became the standard for building Akka-based reactive applications @PayPal.

Culture & Language

Programming to the reactive, functional landscape is very different from the traditional Java programming we have done for the last 20 years. It requires immutability, leaving behind Java Beans and nulls and most mutable APIs in the Java ecosystem, adopting error containers like the Scala Try (a Java version is available in open source), and an ultimate awareness of dangerous function closing-overs and any blocking behavior.

While squbs supports both Scala and Java use cases, Akka APIs are clearly more suitable to their native Scala ecosystem. We also debate whether it is easier for an engineering team with Java background to stay with Java and adopt a culture very different from how we have programmed Java for the last 20 years (empirical programming with pervasive mutability) and try to set our own culture, standards, and guidelines against what Java programmers are used to do, or have them learn Scala and adopt the pre-existing Scala culture well suited for this programming model, with a lot of support libraries that are readily immutable and functional.

The culture adoption is hard to describe. Before teams get their hands dirty, the ultimate answer was almost always Java. It should not be to any surprise as it is their bread and butter language. There is also a resistance from management and architects that do not get their hands dirty. It is the resistance to any change, a resistance to do what you know and not go beyond. Any such change is perceived to introduce a tremendous risk to their projects – in many cases not even knowing they are already trying to create a sub-culture just under the Java brand name – a set of people who speak Java in a very different dialect and will have to maintain this functional, reactive dialect of Java in each of their teams.

For teams and members that actually do get this far and get their sleeves rolled up, they tend to see very quickly their work is much easier when using Akka’s native tongue: Scala. The libraries, the culture, and all they need is readily available. It feels natural, with no tweaks. Also, learning Scala to the point of building reactive services is barely scratching the surface of the ocean depths of Scala.

Programming Patterns

How do you take a programmer who only knows how to write linear code, and make them build high performance, actor-based systems? Since we do not have the luxury to hire the top 5% ubercoders, we have to make sure our developers can be trained to do the job.

Luckily, a vast majority of services do similar things. They receive requests or messages, make database calls to read/write the database, make other service calls, call a rule engine, fetch data from cache, write to cache, all in combination. Here goes our micro-service that does one thing. Some others have one or other forms of stream processing, sometimes ETL.

Because it is useful to create some common patterns that teams can readily adopt, we defined and built our set of programming patterns, which over time, will manifest as application templates. It allows for developers to see the problem and marry it to a well-defined and well-studied pattern ensuring short-term success to their projects.

A common pattern is the “Orchestrator Pattern” which orchestrates requests or messages by talking to a multitude of resources, all asynchronously. These resources may be in the same system, with a manifestation as an actor, another cube which just happens to be on the same address space, or a remote resource altogether.

Another trait we have is graciously called the “Perpetual Stream”. It is no different from just Akka Streams, except that it encourages very long-running streams that will start with the service and stop when the service instance gets stopped. Providing this pattern and utility in squbs allows for streams to hook into the system state and ensure no messages are dropped at shutdown. It provides an additional benefit of modeling the whole flow of the service in a central place, providing a clear oversight and understanding of the service’s functionality.


The adoption of Akka and squbs have already provided very high-scale results. With as little as 8 VMs and 2 vCPU each, applications were able to serve over a billion hits a day. Our systems stay responsive even at 90% CPU, very uncharacteristic for our older architectures. This provides for transaction densities never seen before. Batches or micro-batches do their jobs in one-tenth of the time it took before. With wider adoption, we will see this kind of technology being able to reduce cost and allow for much better organizational growth without growing the compute infrastructure accordingly.

Needless to say, Akka and squbs are still players at the “infrastructure” level. squbs is an open source project by eBay and PayPal. It was designed to be open sourced from the very beginning and is free from pollution and deep library dependencies. We believe the customization hooks allow squbs to fit into any operational environment and would benefit any organization who wants to adopt Akka-based technologies in a larger scale.

Visit us at and leave your comments and questions on the Gitter channel you find on this github site.