Monthly Archives: October 2014

Webhooks for REST APIs launched

By

(By Joe Nash & Alberto López)


PayPal’s latest addition to the roster of products taking advantage of PayPal REST API technology gives developers easier access to notifications of payment activities, such as payment status updates. PayPal webhooks are now available, and not only do they enable similar functionality to that of the IPN available in the Classic APIs, but they provide additional functionality to ease development and create innovative new products.

What is a webhook?

When we discuss webhooks, we refer to three key actors:

  • Webhooks: A webhook takes the form of a POST request to a URL that you set via your PayPal account. When a payment is made, or the status of a payment changes, the POST request is triggered, sending an event. This allows you to see and work with payment updates in real time. Webhooks can be created, updated, listed and deleted all through your PayPal account.
  • Events: An event contains the details about the payment update, including: the time of payment authorization or the amount of a payment or refund. Past events are searchable, and can also be resent in a webhook. There are multiple event types.
  • Event types: Event types can be selected to only cause the triggering of webhooks for specific payment updates and information, allowing you to choose to receive POST requests about the information that is important to you and your product.

Through these three actors, developers can manage operations that happen during the transaction lifecycle. Developers can list all of the event types subscribed to by a given webhook, perform more focused searches such as searching for webhooks by event type, and the API itself can be used to retrieve webhook events.

What operations are available using webhooks?

Webhooks provide some interesting functions for developers:

What are the differences between Webhooks and IPN?

At first glance, the differences between Webhooks and IPN used in the Classic API are subtle. What makes webhooks desirable over IPN is their sheer customisability versus IPN:

  • Whilst IPN did allow for the configuration of different IPN listener’s for different transactions, the IPN listener had to be predefined in the PayPal account. Further, it could only be modified through the PayPal account, and by default all refunds notify to this listener. With Webhooks you will be able to create different webhooks via the API for managing different event types (an authorization created, or voided or refunded), and modify each of them using an API REST call.
  • Webhooks enable the retrieval via API of any webhook event that was unable to be collected in the initial triggering of the webhook. In contrast, IPN events were only accessible via the PayPal account.
  • Webhooks have the important property of real time event notification, whilst with IPN, some delays could be experienced.

It is clear that webhooks give distinct advantages versus IPN; they are more flexible, versatile and can be used in real time. Let’s see how they work.

How do Webhooks work?

To use the webhook API in the Sandbox, first do the following:

  1. Go to the My REST apps page.
  2. Create an application, if you haven’t done so already.
  3. Add a webhook URL (the maximum number of webhooks is 10).
  4. Subscribe to the event types of interest. As we said before, you can also choose to manage webhooks, events, and event types through the Webhooks API.

Getting the bearer token:

Request

curl https://api.sandbox.paypal.com/v1/oauth2/token
\ -H "Accept: application/json" 
\ -H "Accept-Language: en_US" 
\ -u "<Client-Id>:<Secret>" 
\ -d "grant_type=client_credentials”

Response

{ "scope": "https://api.paypal.com/v1/payments/.*
https://api.paypal.com/v1/vault/credit-card/
https://api.paypal.com/v1/vault/credit-card/.*",
"access_token": "EEwJ6tF9x5WCIZDYzyZGaz6Khbw7raYRIBVTTTWxVvgmsG",
"token_type": "Bearer",
"app_id": "APP-6XR95014BA15863X",
"expires_in": 28800
}

Creating the webhook via API

Request

curl -v POST https://api.sandbox.paypal.com/v1/notifications/webhooks \
-H 'Content-Type:application/json' \
-H 'Authorization: 
        Bearer EEwJ6tF9x5WCIZDYzyZGaz6Khbw7raYRIBVTTTWxVvgmsG' \
-d '{
   "url": "http://www.yeowza.com/paypal_webhook",
   "event_types": [
       {
           "name": "PAYMENT.SALE.REFUNDED"
       },
       {
           "name": "PAYMENT.CAPTURE.REFUNDED"
       }
   ]
}'

Response

{
"id":"0E842532SB505363J",
"url":"https://www.mysite.com/paypal_webhook",
"event_types":[
    {
        "name":"PAYMENT.SALE.REFUNDED",
        "description":"A sale payment was refunded"
    },
    {
        "name":"PAYMENT.CAPTURE.REFUNDED",
        "description":"A capture payment was refunded"}
    ],
"links":[
    {
     "href": "https://api.sandbox.paypal.com/v1/notifications/
             webhooks/0E842532SB505363J",
     "rel":"SELF",
     "method":"GET"
    },
    {
     "href": "https://api.sandbox.paypal.com/v1/notifications/
             webhooks/0E842532SB505363J",
     "rel":"UPDATE",
     "method":"PATCH"
    },
    {
     "href": "https://api.sandbox.paypal.com/v1/notifications/
             webhooks/0E842532SB505363J",
     "rel":"DELETE",
     "method":"DELETE"
     }
]
}

After making a payment, you’ll receive in your webhook (https://www.mysite.com/paypal_webhook) an event type with the information of the event type:

{
 "id": "WH-0AF57170N8121644E-6VF27453LH2982421",
 "create_time": “2014-09-25T21:41:28Z",
 "resource_type": "authorization",
 "trigger_event": "PAYMENT.AUTHORIZATION.CREATED",
 "summary": "A payment authorization was created",
 "resource": {
   "id": "0MW879906U0874649",
   "create_time": "2014-09-25T21:39:15Z",
   "update_time": "2014-09-25T21:39:17Z",
   "state": "authorized",
   "amount": {
     "total": “10.00",
     "currency": "USD",
     "details": {
       "subtotal": “10.00"
     }
   },
   "parent_payment": "PAY-0AB48420EF007493KKQVUMBY",
   "valid_until": "2014-07-24T21:39:15Z",
   "links": [
     {
       "href": "https://api.sandbox.paypal.com/v1/payments/
               authorization/0MW879906U0874649",
       "rel": "self",
       "method": "GET"
     },
     {
       "href": "https://api.sandbox.paypal.com/v1/payments/
               authorization/0MW879906U0874649/capture",
       "rel": "capture",
       "method": "POST"
     },
     {
       "href": "https://api.sandbox.paypal.com/v1/payments/
               authorization/0MW879906U0874649/void",
       "rel": "void",
       "method": "POST"
     },
     {
       "href": "https://api.sandbox.paypal.com/v1/payments/
               payment/PAY-0AB48420EF007493KKQVUMBY",
       "rel": "parent_payment",
       "method": "GET"
     }
   ]
 },
 "links": [
   {
     "href": "https://api.sandbox.paypal.com/v1/notfications/
             webhooks-events/WH-0AF57170N8121644E-6VF27453LH2982421",
     "rel": "self",
     "method": "GET"
   },
   {
     "href": "https://api.sandbox.paypal.com/v1/notfications/
             webhooks-events/WH-0AF57170N8121644E-6VF27453LH2982421/
             resend",
     "rel": "resend",
     "method": "POST"
   }
 ]}

In our webhook event, we get all of the information, coupled with useful links, given by the event type. From here, we can perform some operations on the data, update our databases, or launch a process after ensuring the process is done.

Building Data Science at Scale

By

As part of the Boston-based Engineering group, the Data Science team’s charter is to enable science-based personalization and recommendation for PayPal’s global users. As companies of all sizes are starting to leverage their data assets, data science has become indispensable in creating relevant user experience. Helping fulfill PayPal’s mission to build the Web’s most convenient payment solution, the team works with various internal partners and strives to deliver best-in-class data science.

Technology Overview

At the backend of the data science platform reside large-scale machine learning engines that continuously learn and predict, from transactional, behavioral, and other datasets. An example of a question we might try to answer is: if someone just purchased a piece of software, does it increase his likelihood to purchase electronics in near future? It is no wonder that the answer lies in the huge amounts of transaction data, in that what people bought in the past is predictive of what they might consider buying next.

We leverage state-of-the-art machine learning technologies to make such predictions. Machine learning itself has been quickly evolving in recent years. New advances including large-scale matrix factorization, probabilistic behavioral models, and deep learning are no strangers to the team. To make things work at large scale, we leverage Apache Hadoop and its rich ecosystem of tools to process large amounts of data and build data pipelines that are part of the data science platform.

Tackling Data Science

Companies take different approaches in tackling data science. While some companies define a data scientist as someone who performs statistical modeling, we at PayPal Engineering have chosen to take a combined science & engineering approach. Our data scientists are “analytically-minded, statistically and mathematically sophisticated data engineers” [1]. There are of course more science-inclined, and more engineering-inclined individuals on the team, but there is much more of a blend of expertise than a marked distinction between these individuals. This approach to data science allows us to quickly iterate and operationalize high-performing predictive models at scale.

The Venn diagram below, which bears similarity to Conway’s diagram [2], displays the three cornerstones pivotal to the success of the team. Science, which entails machine learning, statistics and analytics, is the methodology by which we generate actionable predictions and insights from very-large datasets. The Engineering component, which includes Apache Hadoop and Spark, makes it possible for us to do science and analytics at scale and deliver results with quick turnaround. Last but not least, I cannot emphasize more the importance of understanding the business for a data scientist. None of the best work in this area that I know of is done in isolation. It is through understanding the problem domain that a data scientist may come up with a better predictive model among other results.

DS-VennDiagram

Scaling Data Science

There are multiple dimensions to scaling data science, which at a minimum involves the team, technology infrastructure, and the operating process. While each of these is worth thoughtful discussion in its own right, I will focus on the operating process since it is critical to how value is delivered. A typical process for the team could break down into these steps:

  • Identify business use case;
  • Understand business logic and KPIs;
  • Identify and capture datasets;
  • Proof of concept and back-test;
  • Operationalize predictive models;
  • Measure lift, optimize and iterate.

Let me explain some of the key elements. First, we see scaling data science as an ongoing collaboration with Product and Business teams. Understanding product and business KPIs and using data science to optimize the same is an essential ingredient of our day-to-day. Second, we follow the best practice in data science, whereby each predictive model is fully back-tested before operationalization. This practice guarantees the effectiveness of the data science platform. Third, the most powerful predictive modeling often requires iterative measurement and optimization. As a concrete example, putting this process into practice along with PayPal Media Network, we were able to achieve excellent results based on:

  • Lookalike modeling: Merchants can reach consumers who look like the merchant’s best existing customers.
  • Purchase intent modeling: Merchants can engage consumers who have a propensity to spend within specific categories.

While it is challenging to crunch the data and create tangible value, it is interesting and rewarding work. I hope to discuss more details of all the fun things we do as data scientists in the future.

References:

[1] http://www.forbes.com/sites/danwoods/2011/10/11/emc-greenplums-steven-hillion-on-what-is-a-data-scientist

[2] http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram