Monthly Archives: May 2018

Using AWS Glacier

This is a surprisingly difficult task and infinitely harder than using AWS S3. So why store stuff in Glacier? Because it’s cheap.

Here’s how much you will pay to store a 1TB (1000GB) file today.

Storage Cost per GB Monthly Cost
EBS SSD $0.10/GB $100/month
EBS Snapshot $0.05/GB $50/month
S3 $0.023/GB $23/month
Glacier $0.004/GB $4/month

What else do you need to know about it? Glacier retrieval is not instant. You make a request, it takes awhile to get fulfilled, and then you pick it up when it’s ready. That’s what makes it different from other storage models.
 

Anyways, here are the steps

1) In the AWS Console, go to the Glacier service and create a vault (e.g. my-vault)

2) Make sure you have a user that has permissions to the vault. Here’s a sample policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "glacier:*"
            ],
            "Sid": "Stmt1376667184000",
            "Resource": [
                "arn:aws:glacier:us-east-1:112233445566:vaults/my-vault"
            ],
            "Effect": "Allow"
        }
    ]
}

3) On your machine, ensure you have the awscli

4) Upload your files to the vault using the awscli

aws glacier upload-archive --vault-name my-vault --account-id - --body my-file.zip

Output will look like this:

{
    "checksum": "e5d002bf40...",
    "location": "/112233445566/vaults/my-vault/archives/KYKdL...",
    "archiveId": "KYKdL..."
}

5) To download, you have to make a request. Depending on the Tier service, it could take minutes to hours before your request is fulfilled.

First, create a request.json file like the following:

{
  "Type": "archive-retrieval",
  "ArchiveId": "KYKdL...",
  "Description": "Retrieve archive on 2015-07-17",
  "Tier":"Expedited",
  "SNSTopic":"arn:aws:sns:us-east-1:112233445566:glacier-alert"
}

Type defines the type of job. In this case, you want “archive-retrieval” to retrieve the archived file.

ArchiveId is the archiveId returned in the output when you uploaded the file.

Tier determines how quickly your request is fulfilled. It has several choices that vary in speed and price.
For the most up-to-date tier pricing and speeds, check Data Retrievals section of FAQ
Here’s the pricing and speeds as of this writing

Tier Price Fullfillment
Standard (default) $0.01/GB + $0.05/retrieval 3 – 5 hours
Bulk $0.01/GB + $0.05/retrieval 5 – 12 hours
Expedited $0.03/GB + $0.01/retrieval 1 – 5 minutes

SNSTopic allows you to register for alerts when the request is done.
You can create SNS Topics in the Simple Notification Service tab of the AWS Console and then copy the Topic ARN here.

Next, run this command to initiate the job request

aws glacier initiate-job --vault-name my-vault --account-id - --job-parameters file://request.json

You’ll get a job with a JobID which you’ll need later.

6) You can also check the status of the job like this:

aws glacier describe-job --vault-name my-vault --account-id - --job-id WI6sdXS...

7) Once the job is complete, you can get the output of the job

aws glacier get-job-output --vault-name my-vault --account-id - --job-id WI6sdXS... [MY-OUTPUT-FILE]

4.5) Yes, 4.5 because this should have gone between steps 4 and 5. You can list the files in your vault like so:

aws glacier initiate-job --account-id - --vault-name my-vault --job-parameters '{"Type": "inventory-retrieval"}'

So why didn’t I just tell you about this earlier? Well because, you don’t get back a list of archives after calling this. Notice you “initiate-job” again. Which means, you have to wait for the job to complete (step 5) and then get the output of the job (step 6). So you have to learn steps 5 and 6 before you can do step 4.5.

Clear as mud? Good.

Advertisements
Tagged , ,

Auth0: enriching the id token and access token

With a little auth0 experience under my belt, let’s dive further into a new topic. What are the ID Token and Access Token.

Once you log in, you get back an object that looks like this in angular

{
accessToken: “eyJ0eX…”
, idToken: “eyJ0eXA…”
, idTokenPayload: {
sub: “auth0|5adfa…”
, nickname: “kane”
, name: “ksee@inferlink.com”
, picture: “https://s.gravatar.com/…”
, email: “ksee@inferlink.com”
}
, …
}

At this point, you’ve got the idTokenPayload which stores all the info you would need about the user. But if you need more, you can create additional “claims”.
Reference

To do this, create a new Rule in auth0.com. Start with an empty rule and paste in something like this:

function (user, context, callback) {
  const namespace = 'https://myapp.example.com/';
  context.idToken[namespace + 'favorite_color'] = user.favorite_color;
  context.idToken[namespace + 'preferred_contact'] = user.user_metadata.preferred_contact;
  callback(null, user, context);
}

Now your payload should contain the favorite_color and preferred_contact fields.

But what about the backend? We only send the access Token in the Authorization header and you’ll notice that you don’t have these other neat user fields in there.

You can do the same thing with a Rule. Simply use context.accessToken instead like so:

function (user, context, callback) {
  const namespace = 'https://myapp.example.com/';
  context.accessToken[namespace + 'favorite_color'] = user.favorite_color;
  context.accessToken[namespace + 'preferred_contact'] = user.user_metadata.preferred_contact;
  callback(null, user, context);
}