EC2 – Bundling Images

The image bundling process (ec2-bundle-image from the Amazon EC2 AMI Tools) is the first step in creating an amazon machine image (AMI). This process creates an XML manifest and (one or more) image part(s). The image parts are created by compressing, encrypting and then dividing the original image.

For the record, there is an upper limit on the size of an image that can be turned into an AMI (10GB, I believe?)

This is a description of the bundling process as I understand it. Be warned that my understanding is wrong. :( While writing this post I have discovered why my bundled manifests were not working. Sort of.

In the following discussion the file input to the bundling process is mymirage.img.

Input Parameters

required

  • the image! (mymirage.img)
  • user’s private key
  • user’s certificate

optional

  • ec2 certificate

Encrypting the original image

ec2-bundle-image runs this command:

$ openssl sha1 < /tmp/ec2-bundle-image-digest-pipe-10174 & tar -c -h -S --owner 0 --group 0 -C /tmp mymirage.img | tee /tmp/ec2-bundle-image-digest-pipe-10174 | gzip -9 | openssl enc -e -aes-128-cbc -K aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa -iv bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb > ec2_tmp/mymirage.img.tar.gz.enc

The image in tarred, gzipped and then encrypted (AES-CBC with a 128-bit key). The result is stored in mymirage.img.tar.gz.enc. A digest (sha1) of this file is calculated simultaneously. Amazon will attempt to recreate this when launching the VM, terminating the instance if it cannot.

A note on the key & iv

The key and iv are generated using gensymkey

# Load and generate necessary keys.
key = Format::bin2hex( Crypto::gensymkey )
iv = Format::bin2hex( Crypto::gensymkey )

despite the fact that Amazon implements a geniv function. But why?

# Generate an initialization vector suitable use with symmetric cipher.
#
def Crypto.geniv
OpenSSL::Cipher::Cipher.new(SYM_ALG).random_iv
end

#----------------------------------------------------------------------------#

##
# Generate a key suitable for use with a symmetric cipher.
#
def Crypto.gensymkey
OpenSSL::Cipher::Cipher.new(SYM_ALG).random_key
end

The key and iv both are encrypted twice, once with the user’s X.509 certificate and once with amazon’s (the optional ec2 certificate mentioned earlier). These encryptions are included in the manifest.

Dividing the image into parts

The encryption result mymirage.img.tar.gz.enc is divided into parts of 10MB, resulting in the files

$ ls
mymirage.img.part.0
mymirage.img.part.1
# etc

I have yet to implement this. My test mirage images are about 5MB so I’ve just been copying the contents of mymirage.img.tar.gz.enc to a file called mymirage.img.part.0. Which works.

Actually I have tried to implement this. But it is only writing 65536 bytes per part instead of the desired 10MB.

(* split a file into parts of 10MB or less return a list of the names of the files created *)
let split file =
let open Unix in
let chunk_size = 1024 * 1024 * 10 in
let buffer = String.create chunk_size in
let fd_in = openfile file [O_RDONLY] 0 in
let rec copy_loop n ps () =
let part =
let name = Filename.(chop_extension @@ chop_extension @@ name_only file) in
tmp @@ Printf.sprintf "%s.img.part.%i" name n in
let ch_out = openfile part [O_WRONLY; O_CREAT; O_TRUNC] 0o666 in
match read fd_in buffer (n * chunk_size) buffer_size with
| 0 -> close fd_in; ps
| r -> print_endline @@ Printf.sprintf "writing %i bytes" r;
ignore (write ch_out buffer (n * chunk_size) r);
close ch_out;
copy_loop (succ n) (part::ps) () in
copy_loop 0 [] ()

Components of the XML manifest

The xml manifest has this structure:

  • `manifest`
    • `version` a string that identifies the type of manifest
    • `bundler` (optional?) information about who created the manifest
      • `name`
      • `version`
      • `release`
    • `machine_configuration`
      • `architecture` `x86_64` for our purposes
      • `kernel` (optional) a string eg `aki-fc8f11cc`
    • `image`
      • `name` string eg `mymirage.img`
      • `user` 12-digit AWS user id
      • `type` the string “machine” for our purposes
      • `digest` SHA1 digest of the encrypted image (created during the compression & encryption process mentioned above)
      • `size` size of the original image (eg size of `mymirage.img`)
      • `bundled_size` size of the compressed, encrypted image (eg size of `mymirage.img.tar.gz.enc`)
      • `ec2_encrypted_key` RSA public encryption of the key used to encrypt the compressed image. The public key is from the ec2 certificate
      • `user_encrypted_key` RSA public encryption of the key used to encrypt the compressed image. The public key is from the user’s certificate
      • `ec2_encrypted_iv` RSA public encryption of the iv used to encrypt the compressed image. The public key is from the ec2 certificate
      • `user_encrypted_iv` RSA public encryption of the iv used to encrypt the compressed image. The public key is from the user’s certificate
      • `parts` info on the 10MB parts into which the encrypted image was split
        • `part` one or more of these
          • `filename` string eg `mymirage.img.part.0`
          • `digest` SHA1 digest from the contents of the part
    • `signature` SHA1 digest of the XML “ and “ info signed with the user’s private key

A number of these fields are easy to fill (eg size or version). Unfortunately a number of them (eg ec2_encrypted_key) are not easily replicated, making it difficult to pinpoint exactly what is wrong with the manifests I’ve created.

Verifying manifest correctness

ec2-unbundle promises to extract an image given a manifest and private key. Sadly….

$ ec2-unbundle -m mymirage.img.manifest.xml -k onekeytorulethemall.pem --debug
ERROR: padding check failed
#
/usr/local/ec2/ec2-ami-tools-1.5.3/lib/ec2/amitools/unbundle.rb:49:in `private_decrypt'
/usr/local/ec2/ec2-ami-tools-1.5.3/lib/ec2/amitools/unbundle.rb:49:in `unbundle'
/usr/local/ec2/ec2-ami-tools-1.5.3/lib/ec2/amitools/unbundle.rb:100:in `main'
/usr/local/ec2/ec2-ami-tools-1.5.3/lib/ec2/amitools/tool_base.rb:201:in `run'
/usr/local/ec2/ec2-ami-tools-1.5.3/lib/ec2/amitools/unbundle.rb:109:in `'

…so I hardcoded my keys in…

# Extract key and IV from xml manifest
# key = pk.private_decrypt(Format::hex2bin( manifest.ec2_encrypted_key))
# iv = pk.private_decrypt(Format::hex2bin( manifest.user_encrypted_iv))
key = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
iv = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"

… and discovered that my digest is incorrect!

$ ec2-unbundle -m mymirage.img.manifest.xml -k onekeytorulethemall.pem --debug
Pipeline.execute: command = [/bin/bash -c 'openssl sha1 /tmp/image-unbundle-pipeline-pipestatus-020140731-20445-zhcuih & echo ${PIPESTATUS[1]} > /tmp/image-unbundle-pipeline-pipestatus-120140731-20445-y3ajkk & echo ${PIPESTATUS[2]} > /tmp/image-unbundle-pipeline-pipestatus-220140731-20445-13o3moo & echo ${PIPESTATUS[3]} > /tmp/image-unbundle-pipeline-pipestatus-320140731-20445-5l5cs2 & echo ${PIPESTATUS[4]} > /tmp/image-unbundle-pipeline-pipestatus-420140731-20445-vxbp20']
Pipeline.execute: output = [(stdin)= e65de62e203671c803b532f046aa5479277790d6]
ERROR: invalid digest, expected da39a3ee5e6b4b0d3255bfef95601890afd80709 received e65de62e203671c803b532f046aa5479277790d6

Oddly enough re-creating the manifest /can/ produce the digest amazon calculates, but most times will not. I have no idea why this is, but at least I can identify when it is happening. I suspect the problem is more with the cmd being run than with the OCaml code.

(* calculate digest; compress & encrypt image *)
let pipeline ~digest_pipe ~tar ~key ~iv ~encrypted_destination =
let open Unix in
let cmd = Printf.sprintf "openssl sha1 %s"
digest_pipe tar digest_pipe key iv encrypted_destination in
let ic = open_process_in cmd in
let digest = input_line ic in
close_process_in ic;
digest

What remains to be done

Unfortunately I still haven't launched a proper, working instance:

Warning: unable to open an initial console.
Kernel panic - not syncing: No init found. Try passing init= option to kernel.

A diff (thank you internet) on the original image and ec2-unbundled image showed that I hadn’t, say, inadvertently corrupted something during the bundling process. So I’m about 99.997% certain that (but for the digest issue) the issue here is related to how I turned the xen kernel into an image.

EC2 Documentation (warning: pdf link!) says an initrd needs to be generated. I don’t see my script doing that. I have a feeling I’ve accidentally deleted that part (I am prone to these kinds of accidents…). Alas the original script on the mirage wiki is either broken or has mysteriously disappeared. (I’m actually not sure which it is. The first time I checked it out, the page was empty. The second time, literally nothing happened when I clicked the link..)

But anyway, what I have left to do

  • fix (or find) the image creating script
  • consistently calculate the correct digest

(and less pressingly)

  • split compressed/encrypted large images into 10MB pieces

The EC2 bindings are now in a github repo. It could do with a little code clean up–there’s a bit of unnecessary boilerplate here and there, and at some point I’d like to define some types so users don’t have to worry about mixing up volume ids with image ids and the like. But for now I’m focussing on getting it to properly deal with errors.

So far it can make calls to the EC2 API and parse them. Most of the API actions necessary for spinning up a VM are implemented (except for actually uploading a custom kernel to AWS — a subject for another post, I think).

But now for my Technical Problem. I spent a good part of the weekend thinking about this and it’s time for you to be plagued too. (I kid!)

Basically the compiler and I are currently wrestling over types. It’s making inferences that I don’t think are justified! See, I have functions like these:

val verb : Cohttp.Code.meth -> api_action -> ~parsing_fn:(api_response -> 'a) -> 'a Lwt.t
val get : api_action -> ~parsing_fn:(api_response -> 'a) -> 'a Lwt.t
(* There's also a `post`. Looks a lot like `get` *)

val parse_del_vol_response : api_response -> bool
(* EC2 returns true if the volume was deleted *)
val parse_reg_img_response : api_response -> img_id
(* EC2 returns the id of the image that was registered *)

The compiler sees delete_volume pass to get the function parse_del_vol_response, which returns bool, and goes, oh, yes, of course, ~parsing_fn is actually api_response -&gt; bool even when the .mli file says it’s api_response -&gt; 'a. (This is the unjustified assumption! Actually, well, I guess it is justifiable–it’s just unncessary, isn’t it?)

Then the compiler gets to register_image which uses parse_reg_img_response to parse the API response and return the id of a newly registered image–not a bool. At which point I get chewed out :/

Error: This expression has type api_response -> img_id
       but an expression was expected of type api_response -> bool
       Type img_id is not compatible with type bool 

For the moment I’ve managed to resolve this by explicitly stating that the parsing_fn parameter must be api_response -&gt; 'a. Which is unfortunate because now I have to write

let get api_action (fn: api_response -> 'a) = verb `GET api_action fn

instead of the more beautiful

let get = verb `GET

Admittedly it’s not the end of the world, but surely this error shouldn’t occur at all? I’m somewhat miffed.

AWS Signatures

Hello world. I am long overdue an update–so this is what I’ve been up to:

After the loss of my laptop (wrongful death by proximity to waterbottle) I’ve been installing ocaml on all the computers I can get my hands on…. First on the family desktop. Well ocaml installed alright, but opam install core_extended did not. After rebooting my computer and attempting the install again, I found that opam still had its tenderhooks in my system. But supposedly it’s okay to just delete opam‘s lock files.

But the family desktop did not pan out. I could not even compile the static site in mirage-skeleton. I’ve since commandeered my father’s laptop, where all things ocaml, opam, and mirage are working beautifully. I was able to compile a xen kernel from a static site (both the one in mirage-skeleton and one I’d created myself) and deploy it with amazon ec2. Actually the whole process was far nicer than using wordpress’s web interface to blog.

The unikernel deployment was done using amazon’s command line tools and I’ve begun working on the bindings that will allow me to do this in ocaml — which brings us to what this post is really about. Amazon requires all requests to their API to have a signature, to ensure that a request is valid or was not tampered with. ec2 supports signature versions 2 and 4. Given that the documentation for signature version 2 seemed shorter than that of signature version 4 (it has parts!), I decided to take a look at signature version 2 first.


(An Invalid!) Signature Version 2 Signing Process

To calculate a signature you need a secret key (eg wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY) and a query string (unfortunately I cannot post an example as wordpress does not approve of backslashes).

Now the documentation says,

[...] you calculate the signature by creating a hash-based message authentication code (HMAC) using either the HMAC-SHA1 or HMAC-SHA256 protocols. The HMAC-SHA256 protocol is preferred.
The resulting signature must be base-64 encoded and then URI encoded.

So I set about this task using cryptokit1. The cryptokit library has these two functions

# #require "cryptokit";;
# Cryptokit.transform_string;;
– : transform -> string -> string = <fun>
# Cryptokit.hash_string;;
– : hash -> string -> string = <fun>

After playing around in utop I discovered it also has these

# MAC.hmac_sha256;;
– : string -> hash = <fun>
# Base64.encode_compact;;
– : unit -> transform = <fun>

So naturally I created a hash function

# let my_hash = Cryptokit.hash_string (MAC.hmac_sha256 sample_secret_key);;
val my_hash : string -> string = <fun>

which I then gave the sample query string and piped to a snipped of code meant to do the base-64 encoding:

# my_hash query_string |> transform_string (Cryptokit.Base64.encode_compact ());;
– : string = "754M/4BBDhUk61bZ2zED8QPI2j3+A624SCDGda1Cfio"

Which, if you look at the documentation, is not the expected result! Clearly I’m missing something here. I just don’t yet know what…


(A Successful!) Signature Version 4 Signing Process

But I was able to implement version 4 with little trouble at all.

First things first:

# #require "cryptokit";;
# open Cryptokit;;

# let secret = "AWS4" ^ "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";;
# let date = "20110909";;
# let reg = "us-east-1";;
# let service = "iam";;
# let signing = "aws4_request";;

# let hash_str key str = hash_string (MAC.hmac_sha256 key) str;;

Rather than using a just the query string and secret key, the version 4 signing process uses parameters unique to the request (the date, the region, etc) to create hash functions that result in values that are fed into more hash functions that result in more values. So a secret key (with AWS4 prepended) is used to create a hash function. You use this function to hash the date (eg “20110909”). This result becomes your key for the next hash. And so on.

# let kSecret = secret;;
# let kDate = hash_str kSecret date;;
# let kReg = hash_str kDate reg;;
# let kService = hash_str kReg service;;
# let kSigning = hash_str kService "aws4_request";;

The last key, kSigning is used to hash the string to sign. After hexencoding this result, you should have a signature.

# let signature = hash_str kSigning str_to_sign |> transform_string ( Hexa.encode () );;

The complete example of this process can be found here.


  1. Incidentally I cannot find any up-to-date documentation (online) for cryptokit and am most distressed by this. 

ocaml-github

In preparation for my own project this summer, what follows are some notes on the ocaml-github library.

Dealing with JSON

Requests to/from github’s API are JSON. The JSON parsing/unparsing is done with using Yojson. There is a helpful RWO chapter that demonstrates Yojson use.

Organization of the ocaml-github library

For the most part the different modules (see lib/github.ml(i)) correspond to github API actions eg making pull requests or querying the issues a repository.
Of particular interest is the Monad module. Illuminating comment explains it quite wonderfully, I think :)

(** All API requests are bound through this monad. The [run] function
    will unpack an API response into an Lwt thread that will hold the
    ultimate response. *)
module Monad : sig
  type 'a t
  val bind : 'a t -> ('a -> 'b t) -> 'b t
  val return : 'a -> 'a t
  val run : 'a t -> 'a Lwt.t
  val (>>=) : 'a t -> ('a -> 'b t) -> 'b t
end

The Role of Cohttp

Cohttp makes the actual requests to the github api. I thought these were interesting/important examples of uses of the Cohttp library (called C in the following snippets:

  let get ~user ~pass ~id () =
    let uri = URI.authorization id in
    let headers = C.Header.(add_authorization (init ()) (C.Auth.Basic (user,pass))) in
    API.get ~headers ~uri ~expected_code:`OK (fun body -> return (auth_of_string body))


 (* Add the correct mime-type header *)
  let realize_headers headers = C.Header.add_opt headers "content-type" "application/json"

Follow

Get every new post delivered to your Inbox.