Extending genio

By

I am a great fan of Khan Academy and use it to learn more about economics and astronomy – subjects that I am far removed from professionally but fields that I find fascinating nevertheless. Khan Academy provides a simple API that is RESTful and uses OAuth 1.0 for authentication. I had been meaning to code a simple dashboard that will help me discover new videos and exercises for a while. Documentation is available on their wiki but I was in no mood to read through it – I wanted to hit the IDE right away. That’s where genio came in handy. Khan Academy does not provide a WADL but I noticed that the Khan API explorer used a JSON specification for describing the API. I decided to write a quick and dirty genio parser for the Khan API specification.

Setting up the project

I am a Java / PHP programmer with little Ruby experience but getting started with the genio gem was a breeze. I have Ruby 1.9.3 installed on my machine. First, I created a new folder called khan-parser with this minimal Gemfile.

source 'https://rubygems.org'
gem 'genio'

I have a dependency on the genio gem since I plan to use one of the in-built templates to generate PHP code. I would have listed genio-parser as my dependency if I had to write custom templates in addition. I then ran bundle install to fetch dependencies. If you do not have bundler installed on your system, get it with this simple command gem install bundler

Writing a custom parser

The next step is to write a parser that can understand the custom Khan API specification. The genio-parser uses a specification format agnostic model that is fed into its templates. This allows easy plugging in of multiple specification formats. The first release, in fact, comes with in-built parsers for WSDL, WADL and the Google discovery formats. At a high level, the API model defines

  • DataTypes – Data types as defined by the API domain.
  • Properties – Sub elements under each DataType. A DataType can have optional / mandatory properties and generally has documentation for each property.
  • Operations – Represents a unique API operation and is associated with request and response parameters
  • Services – A grouping of API operations. In the case of RESTful services, each Resource is represented by a DataType that is mapped to a Service. For SOAP services, each portType defined by the WSDL maps to a service.

The Khan parser that I set out to write essentially had to convert the JSON specification into an object tree of these model types. I created a khan-parser.rb file with a KhanParser class that extended Genio::Parser::Format::Base. The class need only implement a single method load that takes the location of the specification file to parse.

module MyApp
  class KhanParser < Genio::Parser::Format::Base

    # Load schema
    def load(filename)
    end

  end
end

RESTful APIs define operations that can be performed on resources and the API url patterns reflect this design. For example, if ‘User’ is a resource in the domain model, all user related operations are made available at, say, /api/v1/user/*. While the Khan API follows this design principle, the specification does not explicitly group operations under resources. I could have parsed the API urls to determine the resource-operation mapping but chose to keep things simple and generate all operations under a single surrogate resource.

In the load function, I parse the Khan API json into a Service object. Notice how I create objects of type Genio::Parser::Types::Service, Genio::Parser::Types::DataType and them to the global object map.

def load(filename)

  # Read the API Specification
  data = JSON.parse(data, :object_class => Genio::Parser::Types::Base, 
                    :max_nesting => 100)
  klass = class_name(filename)

  # Create the meta model tree desribing the API
  service = Genio::Parser::Types::Service.new({});
  service.operations = parse_operations(klass, data)

  # Add the definition to our global definition
  services[klass] = service;

  # Add a dummy resource (data type) with same name 
  # as our service since the Khan API does not 
  # explicitly group operations under resources
  data_types[klass] = 
            Genio::Parser::Types::DataType.new({"properties" => {}});

end

The next step is to parse the API operations in the parseoperations function. parseoperations simply iterates through the API specification and returns an array of Operation objects.

def parse_operations(service_name, data)
  operations = {};
  data.each do |options|
    operations[get_operation_name(options.http_method, 
                                  options.url)] = parse_operation(options)
  end
  return operations
end

Now, let’s look at how the Operation objects are created. The parse_operation function takes a chunk of the Khan API specification JSON that represents an operation as input. It returns a Genio::Parser::Types::Operation object with the following properties set

  • type – The HTTP method for the operation.
  • path – The relative URL path for accessing this resource
  • description – Any human readable desription of the operation that will be used to generate method comments in the generated code.
  • response – The API response type for this operation. Khan Academy’s simply return a JSON payload whose type is not defined in the specification. So, I have chosen to return the JSON response as-is as a string.
  • parameters – An array of query parameters and part parameters.
# Parse operation
def parse_operation(data)
  operation = {
  "type" => data.http_method,
  # genio expects path parameter placeholders to be wrapped between {}, 
  # for .e.g /v1/resource/{resource-id}
  "path" => data.url.gsub("path:","").gsub("", "}"),
  "description" => data.summary 
                    + "n" + data.description + "n", 
                     "response" => "string"
  }

  operation["parameters"] = {};
  data.arguments.each do |options|
    options.type = "string"
    options.location = "path" if options.part_of_url == true
    operation["parameters"][options.name] = options;
  end

  Genio::Parser::Types::Operation.new(operation)
end

And with this, our parser is ready.

Generating files

To generate files using the new parser, I need to tell genio where the parser and the templates are. I created a new file called generator.rb in the khan-generator project. I used thor, a widely used command line toolkit gem so I can pass in command line arguments to the generator. Notice that I have an include on Genio::Template. This allows me to use the built-in “templates/sdk.rest_php.erb” from genio and also provides some command line helpers.

require './khan-parser.rb'
require 'genio'
require 'thor'

class MyTask  :string, :required => true, 
               :desc => "Json schema path"
  class_option :output_path, :type => :string, :default => "output"
  class_option :package, :type => :string, :default => "MyApp", 
               :desc => "Namespace for generated class"

  desc "generate", "Geneate khan PHP SDK"
  def generate
    schema = MyApp::KhanParser.new()
    schema.load(options[:schema].strip)
    schema.data_types.each do|name, data_type|
      render("templates/sdk.rest_php.erb",
        :data_type => data_type,
        :package => options[:package],
        :classname => name,
        :schema => schema,
        :helper => Genio::Helper::PHP,
        :create_file => options[:output_path] + '/' + name + '.php');
    end
  end
end

MyTask.start(ARGV)

I ran ruby generate.rb –output-path=dashboard-app/ –schema=khan.json to test my generator and all’s good.

The dashboard app

Now, on to the dashboard application that will use the generated Khan.php class. Since I intend to use the generated wrapper class with the PayPal Rest SDK, I created a new folder ‘khan-dashboard’. I added a composer.json required by composer, a widely used dependency manager in PHP.

I added a dependency to PayPal’s REST SDK in composer.json and ran composer update --no-dev

{
  "name": "MyApp/khan-dashboard",
  "description": "Khan notifier app",
  "require": {
    "paypal/rest-api-sdk-php" : "*"
  }
}

With this skeletal project ready, I switched back to my ruby project and ran ruby generator.rb generate --schema=khan.json --output-path=../khan-dashboard to generate my service class, Khan.php. With the autogenerated Khan.php ready, now is the time to code my dashboard app. My rudimentary dashboard had this snippet

<?php
require __DIR__ . '/vendor/autoload.php';
require __DIR__ . '/khan.php'; 

# Configure the API call
$apiContext = new PayPalRestApiContext(null);
$apiContext->setConfig(
  array(
  'service.EndPoint' => 'http://www.khanacademy.org'
  )
);

# Call the get_exercises_videos method to get a video listing
$res = MyAppKhan::get_exercises_videos('logarithms_1',
                                        array(), $apiContext);

# Display results
foreach(json_decode($res) as $video) {
  ....
}

With this, my dashboard app is ready. I just ran php app.php to fetch video results using the Khan Academy API. You can find the complete code sample for the Khan custom parser at https://github.com/paypal/genio-sample/tree/master/khan-academy