Monthly Archives: March 2015

CSS Essentials


Time and time again, I witness CSS being marginalized because developers think it’s easy. And yes, it is easy! Add a background color, border color, or padding … easy.  Change the width and height of an element … easy. There are a lot of code snippets out there to copy from; Just search for it. But it is also very easy to mess up your work with one change of a property. HTML, JavaScript, and CSS are the trinity for User Interface Engineers, so let’s give CSS the respect it deserves.

I like to classify CSS properties into two categories: embellishment and layout.

  • Embellishment: Gives some visual effect to the element. Some examples of embellishment properties are border, background, padding, and color.
  • Layout: Changes the location of an element. Some of the layout properties are display, z-index, float and position.

If you want to get the biggest bang for your buck, I would suggest for those new to CSS, and even for those who aren’t new to CSS, to truly understand the layout properties and the box model. Know what they mean and what they do like the back of your hand. After all, positioning and visually manipulating elements is the essence of making beautiful and delightful applications. Skim over the embellishment properties. They are quite easy to understand and apply. In this article I’ll be covering the position property.

The ‘Position’ Property

In order to position an element, two things need to be known or specified. 1) the positioning context and 2) the position declaration. Let’s use a real world example to clarify these two things. Say, for instance, I’m giving you directions to my house. “From Main Street make a right at Oak Street.” With this information would you be able to know where to go? If I’m headed northbound on Main Street, then making a right at Oak Street will take me eastbound. Or if I’m headed southbound on Main Street, then making a right at Oak Street will take me westbound. So, which way is the correct way to go? Without any context, you wouldn’t know which way to head on Oak Street. From this example, the positioning context would be, “Head northbound on Main Street,” and the position declaration would be, “Make a right at Oak Street.” Only with these two things do we know exactly where to go.

The position property has five values: static, fixed, relative, absolute, and inherit. In addition to the position property, the top, right, bottom, and left properties are used to specify the offset. Let’s go over each value of the position property.

Position: Static

Positioning context: Closest block ancestor. If there is none, then browser or iframe

The ‘static’ value is the default value of the element. Since the element has not been positioned, it is laid out according to the HTML source order. However, if an element has been positioned, you can use this value to reset it to its original location in the flow.  The top, right, bottom, and left properties don’t apply here since the element has not been positioned. An example is on PayPal’s homepage; The <h1> shows the default value of ‘static’ because it has not been positioned.


Position: Fixed

Positioning context: Browser or iframe

When an element is positioned using the ‘fixed’ value, it is fixed to that spot on the screen or iframe. It will not move if you scroll. The space that the element used to occupy disappears, so it will overlap or be overlapped by another element depending on the value of its z-index property. Also, the width and height of the positioned element will only be as wide or long as its content unless you specify a different width and/or height. An example is the ‘Give us feedback’ button on PayPal’s Shopping page.


Position: Relative

Positioning context: Closest block ancestor. If there is none, then browser or iframe

An element is offset relative to where it would normally appear in the document flow. The space that it used to occupy is preserved (see below.) I added borders around the parent and child element to better illustrate the preservation of the original space. View code for the example here.


I often use position: relative; with no offsets to create a positioning context for a child element that I want to position with the ‘absolute’ value. An example of creating a positioning context using the ‘relative’ value is on PayPal’s new Log In page. Notice that the declaration of the ‘textInput’ class has a position of ‘relative’ with no top, right, bottom, or left offset. I’m using the <div> with the ‘textInput’ class as the positioning context for the children <div>s (see below.)


Position: Absolute

Positioning context: First ancestor that has been positioned with the value ‘relative’, ‘fixed’, or ‘absolute’. If there is none, then browser or iframe

An element is offset relative to its positioning context. The space that it used to take up disappears and its width and height is determined by its content length. So, when you position an element with the ‘absolute’ value, that element is only as wide or tall as its content. See below or view code for example here.


Let’s go back to our log in page example. I used the <div> with the ‘textInput’ class as the positioning context for the children <div>s. I then declared the <div> with class of ‘errorMessage’ to have a position of ‘absolute’, top offset of 1px, left offset of 0, and z-index of 1. I also declared the <div> with class of ‘fieldWrapper’ to have a position of ‘relative’ and z-index of 2. What this does is cause the <div> with class of ‘errorMessage’ to be behind the <div> with class of ‘fieldWrapper’ because it has a lower z-index; effectively hiding the container with the error message behind the text input. When the input has focus, the error message container will slide out from underneath. See below for more details or you can go here to see it in action:


Position: Inherit

Positioning context: Will vary depending on which value the child element inherits from its parent.

A child element can inherit its parent’s position property value using the ‘inherit’ value.


In the above screenshot, I declared the parent element to have a position property value of ‘relative’. I then declared the child element to have a position property value of ‘inherit’. You can see that the calculated value of the child element’s position property is ‘relative’ just like its parent. Now that the child element’s position property is calculated to be ‘relative’, its positioning context is the closest block ancestor. If there is no closest block ancestor, then the positioning context is the browser of iframe. See code here.

Some Useful Applications of the ‘Position’ Property

1. Tooltip using no JavaScript

Screen Shot 2015-03-17 at 11.32.38 PM

This uses the position property with the ‘absolute’ value, the :hover and :focus pseudo classes, and the sibling selector. See code of the no Javascript tooltip here.

2. One element’s height determines another element’s height


To achieve this, I declare that the top and bottom edges of the dependent element to be flushed with the top and bottom edges of the parent container. In other words, I gave the dependent container a top and bottom offset of 0. See code example of one element’s height determining another element’s height here.

3. Two-toned vertical background that spans entire height of window regardless of content length

Screen Shot 2015-03-12 at 10.42.08 PM

Here is the source for the example of a two-toned vertical background. You may ask why didn’t I use a gradient for the background. It is because when the content is extremely long, the gradient will slow down the browser.

4. Create overlapping elements for visual effect

Screen Shot 2015-03-17 at 11.40.56 PM

My friend and colleague, Nat Shanker, came up with this Venn diagram. See his code here.


Being well versed in CSS makes you a better developer. You won’t have to resort to using JavaScript to visually manipulate elements. I have never heard anyone talk about front-end development without mentioning CSS, so please take the time to learn it. It will make your life happier, just like double rainbows and butterflies.

Introducing SuPPort


In our last post, Ten Myths of Enterprise Python, we promised a deeper dive into how our Python Infrastructure works here at PayPal and eBay. There is a problem, though. There are only so many details we can cover, and at the end of the day, it’s just so much better to show than to tell.

support_logoSo without further ado, we’re pleased to introduce SuPPort, an in-development distillation of our PayPal Python Infrastructure.

Started in 2010, Python Infrastructure initially powered PayPal’s internal price-setting interfaces, then grew to support payment orchestration interfaces, and now in 2015 it supports dozens of projects at PayPal and eBay, having handled billions of production-critical requests for a wide range of teams and tiers. So what does it mean to distill this functionality into SuPPort?

SuPPort is an event-driven server framework designed for building scalable and maintainable services and clients. It’s built on top of several open-source technologies, so before we dig into the workings of SuPPort, we ought to showcase its foundations:

Some or all of these may be new to many developers, but all-in-all they comprise a powerful set of functionality. With power comes complexity, and while Python as a language strives for technical convergence, there are many ways to approach the problem of developing, scaling, and maintaining components. SuPPort is one way gevent and the libraries above have been used to build functional services and products with anywhere from 100 requests per day to 100 requests per second and beyond.

Enterprise Ideals, Flexible Features

Many motivations have gone into building up a Python stack at PayPal, but as in any enterprise environment, we continuously aim to achieve the following:

Of course organizations of all sizes want these features as well, but the key difference is that large organizations like PayPal usually end up building more. All while demanding a higher degree of redundancy and risk mitigation from their processes. This often results in great cost in terms of both hardware and developer productivity. Fortunately for us, Python can be very efficient in both respects.

So, let’s take a stroll through a selection of SuPPort’s feature set in the context of these criteria! Note that if you’re unfamiliar with evented programming, nonblocking sockets, and gevent in particular, some of this may seem quite foreign. The gevent tutorial is a good entry point for the intermediate Python programmer, which can be supplemented with this well-illustrated introduction to server architectures.


Python usage here at PayPal has spread to virtually every imaginable use case: administrative interfaces, midtier services, operations automation, developer tools, batch jobs; you name it, Python has filled a gap in that area. This legacy has resulted in a few rather interesting abstractions exposed in SuPPort.


PayPal has hundreds of services across several tiers. Interoperating between these means having to implement over half a dozen network protocols. The BufferedSocket type eliminated our inevitable code duplication, handling a lot of the nitty-gritty of making a socket into a parser-friendly data source, while retaining timeouts for keeping communications responsive. A must-have primitive for any gevent protocol implementer.


Errors happen in live environments. DNS requests fail. Packets are lost. Latency spikes. TCP handshakes are slow. SSL handshakes are slower. Clients rarely handle these problems gracefully. This is why SuPPort includes the ConnectionManager, which provides robust error handling code for all of these cases with consistent logging and monitoring. It also provides a central point of configuration for timeouts and host fallbacks.


As part of a large organization, we can afford to add more machines, and are even required to keep a certain level of redundancy and idle hardware. And while DevOps is catching on in many larger-scale environments, there are many cases in enterprise environments where developers are not allowed to attend to their production code.

SuPPort currently comes with all the same general-purpose introspection capabilities that PayPal Python developers enjoy, meaning that we get you as much structured information about your application as possible without actually requiring login privileges. Of course almost every aspect of this is configurable, to suit a wide variety of environments from development to production.

Context management

Python famously has no global scope: all values are namespaced in module scope. But there are still plenty of aspects of the runtime that are global. Some are out of our control, like the OS-assigned process ID, or the VM-managed garbage collection counters. Others aspects are in our control, and best practice in concurrent programming is to keep these as well-managed as possible.

SuPPort uses a system of Contexts to explicitly manage nonlocal state, eliminating difficult-to-track implicit global state for many core functions. This has the added benefit of creating opportunities to centrally manage and monitor debugging data and statistics (some charts of which are shown below), all made available through the MetaApplication, detailed further down.

Accept SSL Charts

Charting quantiles and recent timings for incoming SSL connections from a remote service.

SuPPort Stats table

Values are also available in table-based and JSON formats, for easy human and machine readability.


While not exclusively a web server framework, SuPPort leverages its strong roots in the web to provide both a web-based user interface and API full of useful runtime information.

As you can see below, there is a lot of information exposed through this default interface. This is partly because of restricted environments not allowing local login on machines, and another part is the relative convenience of a browser for most developers. Not pictured is the feature that the same information is available in JSON format for easy programmatic consumption. Because this application is such a rich source of information, we recommend using SuPPort to run it on a separate port which can be firewalled accordingly, as seen in this example.

MetaApplication Screenshot #1

A screenshot of the MetaApplication, showing load averages and other basic information, as well as subroutes to further info.

MetaApplication Screenshot 2

Another shot of the MetaApplication, showing process and runtime info



At the end of the day, reliability over long periods of time is what earns a stack approval and adoption. At this point, the SuPPort architecture has a billion production requests under its belt here at PayPal, but on the way we put it through the proverbial paces. At various points, we have tested and confirmed several edge behaviors. Here are just a few key characteristics of a well-behaved application:

  • Gracefully sheds traffic under load (no unbounded queues here)
  • Can and has run at 90%+ CPU load for days at a time
  • Is free from framework memory leaks
  • Is robust to memory leakage in user code

To illustrate, a live service handling millions of requests per day had a version of OpenSSL installed which was leaking memory on every handshake. Thanks to preemptive worker cycling on excessive process memory usage, no intervention was required and no customers were impacted. The worker cycling was noted in the logs, the leak was traced to OpenSSL, and operations was notified. The problem was fixed with the next regularly scheduled release rather than being handled as a crisis.

No monkeypatching

One of the first and sometimes only ways that people experience gevent is through monkeypatching. At the top of your main module you issue a call to gevent that automatically swaps out virtually all system libraries with their cooperatively concurrent ones. This sort of magic is relatively rare in Python programming, and rightfully so. Implicit activities like this can have unexpected consequences. SuPPort is a no-monkeypatching approach to gevent. If you want to implement your own network-level code, it is best to use gevent.socket directly. If you want gevent-incompatible libraries to work with gevent, best to use SuPPort’s gevent-based threading capabilities.

Using threads with gevent

“Threads? In my gevent? I thought the whole point of greenlets and gevent was to eliminate evil, evil threads!” –Countless strawmen

Originating in Stackless and ported over in 2004 by Armin Rigo (of PyPy fame), greenlets are mature and powerful concurrency primitives. We wanted to add that power to the process- and thread-based world of POSIX. There’s no point running from standard OS capabilities; threads have their place. Many architectures adopt a thread-per-request or process-per-request model, but the last thing we want is the number of threads going up as load increases. Threads are expensive; each thread adds a bit of contention to the mix, and in many environments the memory overhead alone, typically 4-8MB per thread, presents a problem. At just a few kilobytes apiece, greenlet’s microthreads are three orders of magnitude less costly.

Furthermore, thread usage in our architecture is hardly about parallelism; we use worker processes for that. In the SuPPort world, threads are about preemption. Cooperative greenlets are much more efficient overall, but sometimes you really do need guarantees about responsiveness.

One excellent example of how threads provide this responsiveness is the ThreadQueueServer detailed below. But first, there are two built-in Threadpools with decorators worth highlighting, io_bound and cpu_bound:


This decorator is primarily used to wrap opaque clients built without affordances for cooperative concurrent IO. We use this to wrap cx_Oracle and other C-based clients that are built for thread-based parallelization. Other major use cases for io_bound is when getting input from standard input (stdin) and files.

A rough sketch of what threads inside a worker look like. The outer box is a process, inner boxes are threads/threadpools, and each text label refers to a coroutine/greenlet.

A rough sketch of what threads inside a worker look like. The outer box is a process, inner boxes are threads/threadpools, and each text label refers to a coroutine/greenlet.


The cpu_bound decorator is used to wrap expensive operations that would halt the event loop for too long. We use it to wrap long-running cryptography and serialization tasks, such as decrypting private SSL certificates or loading huge blobs of XML and JSON. Because the majority of use cases’ implementations do not release the Global Interpreter Lock, the cpu_bound ThreadPool is actually just a pool of one thread, to minimize CPU contention from multiple unparallelizable CPU-intensive tasks.

It’s worth noting that some deserialization tasks are not worth the overhead of dispatching to a separate thread. If the data to be deserialized is very short or a result is already cached. For these cases, we have the cpu_bound_if decorator, which conditionally dispatches to the thread, yielding slightly higher responsiveness for low-complexity requests.

Also note that both of these decorators are reentrant, making dispatch idempotent. If you decorate a function that itself eventually calls a decorated function, performance won’t pay the thread dispatch tax twice.


The ThreadQueueServer exists as an enhanced approach to pulling new connections off of a server’s listening socket. It’s SuPPort’s way of incorporating an industry-standard practice, commonly associated with nginx and Apache, into the gevent WSGI server world.

If you’ve read this far into the post, you’re probably familiar with the standard multi-worker preforking server architecture; a parent process opens a listening socket, forks one or more children that inherit the socket, and the kernel manages which worker gets which incoming client connection.

Preforking architecture

Basic preforking architecture. The OS balances traffic between workers, monitored by an arbiter.

The problem with this approach is that it generally results in inefficient distribution of connections, and can lead to some workers being overloaded while others have cycles to spare. Plus, all worker processes are woken up by the kernel in a race to accept a single inbound connection, in what’s commonly referred to as the thundering herd.

The solution implemented here uses a thread that sleeps on accept, removing connections from the kernel’s listen queue as soon as possible, then explicitly pushing accepted connections to the main event loop. The ability to inspect this user-space connection queue enables not only even distribution but also intelligent behavior under high load, such as closing incoming connections when the backlog gets too long. This fail-fast approach prevents the kernel from holding open fully-established connections that cannot be reached in a reasonable amount of time. This backpressure takes the wait out of client failure scenarios leading to a more responsive extrinsic system, as well.

What’s next for SuPPort

The sections above highlight just a small selection of the features already in SuPPort, and there are many more to cover in future posts. In addition to those, we will also be distilling more code from our internal codebase out into the open. Among these we are particularly excited about:

  • Enhanced human-readable structured logging
  • Advanced network security functionality based on OpenSSL
  • Distributed online statistics collection
  • Additional generalizations for TCP client infrastructure

And of course, more tests! As soon as we get a couple more features distilled out, we’ll start porting out more than the skeleton tests we have now. Suffice to say, we’re really looking forward to expanding our set of codified concurrent software learnings, and incorporating as much community feedback as possible, so don’t forget to subscribe to the blog and the SuPPort repo on GitHub.

Mahmoud Hashemi, Kurt Rose, Mark Williams, and Chris Lane