The EC2 bindings are now in a github repo. It could do with a little code clean up–there’s a bit of unnecessary boilerplate here and there, and at some point I’d like to define some types so users don’t have to worry about mixing up volume ids with image ids and the like. But for now I’m focussing on getting it to properly deal with errors.

So far it can make calls to the EC2 API and parse them. Most of the API actions necessary for spinning up a VM are implemented (except for actually uploading a custom kernel to AWS — a subject for another post, I think).

But now for my Technical Problem. I spent a good part of the weekend thinking about this and it’s time for you to be plagued too. (I kid!)

Basically the compiler and I are currently wrestling over types. It’s making inferences that I don’t think are justified! See, I have functions like these:

val verb : Cohttp.Code.meth -> api_action -> ~parsing_fn:(api_response -> 'a) -> 'a Lwt.t
val get : api_action -> ~parsing_fn:(api_response -> 'a) -> 'a Lwt.t
(* There's also a `post`. Looks a lot like `get` *)

val parse_del_vol_response : api_response -> bool
(* EC2 returns true if the volume was deleted *)
val parse_reg_img_response : api_response -> img_id
(* EC2 returns the id of the image that was registered *)

The compiler sees delete_volume pass to get the function parse_del_vol_response, which returns bool, and goes, oh, yes, of course, ~parsing_fn is actually api_response -> bool even when the .mli file says it’s api_response -> 'a. (This is the unjustified assumption! Actually, well, I guess it is justifiable–it’s just unncessary, isn’t it?)

Then the compiler gets to register_image which uses parse_reg_img_response to parse the API response and return the id of a newly registered image–not a bool. At which point I get chewed out :/

Error: This expression has type api_response -> img_id
       but an expression was expected of type api_response -> bool
       Type img_id is not compatible with type bool 

For the moment I’ve managed to resolve this by explicitly stating that the parsing_fn parameter must be api_response -> 'a. Which is unfortunate because now I have to write

let get api_action (fn: api_response -> 'a) = verb `GET api_action fn

instead of the more beautiful

let get = verb `GET

Admittedly it’s not the end of the world, but surely this error shouldn’t occur at all? I’m somewhat miffed.

AWS Signatures

Hello world. I am long overdue an update–so this is what I’ve been up to:

After the loss of my laptop (wrongful death by proximity to waterbottle) I’ve been installing ocaml on all the computers I can get my hands on…. First on the family desktop. Well ocaml installed alright, but opam install core_extended did not. After rebooting my computer and attempting the install again, I found that opam still had its tenderhooks in my system. But supposedly it’s okay to just delete opam‘s lock files.

But the family desktop did not pan out. I could not even compile the static site in mirage-skeleton. I’ve since commandeered my father’s laptop, where all things ocaml, opam, and mirage are working beautifully. I was able to compile a xen kernel from a static site (both the one in mirage-skeleton and one I’d created myself) and deploy it with amazon ec2. Actually the whole process was far nicer than using wordpress’s web interface to blog.

The unikernel deployment was done using amazon’s command line tools and I’ve begun working on the bindings that will allow me to do this in ocaml — which brings us to what this post is really about. Amazon requires all requests to their API to have a signature, to ensure that a request is valid or was not tampered with. ec2 supports signature versions 2 and 4. Given that the documentation for signature version 2 seemed shorter than that of signature version 4 (it has parts!), I decided to take a look at signature version 2 first.


(An Invalid!) Signature Version 2 Signing Process

To calculate a signature you need a secret key (eg wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY) and a query string (unfortunately I cannot post an example as wordpress does not approve of backslashes).

Now the documentation says,

[...] you calculate the signature by creating a hash-based message authentication code (HMAC) using either the HMAC-SHA1 or HMAC-SHA256 protocols. The HMAC-SHA256 protocol is preferred.
The resulting signature must be base-64 encoded and then URI encoded.

So I set about this task using cryptokit1. The cryptokit library has these two functions

# #require "cryptokit";;
# Cryptokit.transform_string;;
– : transform -> string -> string = <fun>
# Cryptokit.hash_string;;
– : hash -> string -> string = <fun>

After playing around in utop I discovered it also has these

# MAC.hmac_sha256;;
– : string -> hash = <fun>
# Base64.encode_compact;;
– : unit -> transform = <fun>

So naturally I created a hash function

# let my_hash = Cryptokit.hash_string (MAC.hmac_sha256 sample_secret_key);;
val my_hash : string -> string = <fun>

which I then gave the sample query string and piped to a snipped of code meant to do the base-64 encoding:

# my_hash query_string |> transform_string (Cryptokit.Base64.encode_compact ());;
– : string = "754M/4BBDhUk61bZ2zED8QPI2j3+A624SCDGda1Cfio"

Which, if you look at the documentation, is not the expected result! Clearly I’m missing something here. I just don’t yet know what…


(A Successful!) Signature Version 4 Signing Process

But I was able to implement version 4 with little trouble at all.

First things first:

# #require "cryptokit";;
# open Cryptokit;;

# let secret = "AWS4" ^ "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";;
# let date = "20110909";;
# let reg = "us-east-1";;
# let service = "iam";;
# let signing = "aws4_request";;

# let hash_str key str = hash_string (MAC.hmac_sha256 key) str;;

Rather than using a just the query string and secret key, the version 4 signing process uses parameters unique to the request (the date, the region, etc) to create hash functions that result in values that are fed into more hash functions that result in more values. So a secret key (with AWS4 prepended) is used to create a hash function. You use this function to hash the date (eg “20110909”). This result becomes your key for the next hash. And so on.

# let kSecret = secret;;
# let kDate = hash_str kSecret date;;
# let kReg = hash_str kDate reg;;
# let kService = hash_str kReg service;;
# let kSigning = hash_str kService "aws4_request";;

The last key, kSigning is used to hash the string to sign. After hexencoding this result, you should have a signature.

# let signature = hash_str kSigning str_to_sign |> transform_string ( Hexa.encode () );;

The complete example of this process can be found here.


  1. Incidentally I cannot find any up-to-date documentation (online) for cryptokit and am most distressed by this. 

ocaml-github

In preparation for my own project this summer, what follows are some notes on the ocaml-github library.

Dealing with JSON

Requests to/from github’s API are JSON. The JSON parsing/unparsing is done with using Yojson. There is a helpful RWO chapter that demonstrates Yojson use.

Organization of the ocaml-github library

For the most part the different modules (see lib/github.ml(i)) correspond to github API actions eg making pull requests or querying the issues a repository.
Of particular interest is the Monad module. Illuminating comment explains it quite wonderfully, I think :)

(** All API requests are bound through this monad. The [run] function
    will unpack an API response into an Lwt thread that will hold the
    ultimate response. *)
module Monad : sig
  type 'a t
  val bind : 'a t -> ('a -> 'b t) -> 'b t
  val return : 'a -> 'a t
  val run : 'a t -> 'a Lwt.t
  val (>>=) : 'a t -> ('a -> 'b t) -> 'b t
end

The Role of Cohttp

Cohttp makes the actual requests to the github api. I thought these were interesting/important examples of uses of the Cohttp library (called C in the following snippets:

  let get ~user ~pass ~id () =
    let uri = URI.authorization id in
    let headers = C.Header.(add_authorization (init ()) (C.Auth.Basic (user,pass))) in
    API.get ~headers ~uri ~expected_code:`OK (fun body -> return (auth_of_string body))


 (* Add the correct mime-type header *)
  let realize_headers headers = C.Header.add_opt headers "content-type" "application/json"

Follow

Get every new post delivered to your Inbox.