Monthly Archives: November 2017

Migrating Maven artifacts from Nexus to S3

Awhile ago, I set up a Nexus server to host my company’s private artifacts, both our in-house projects as well as jars that weren’t available on Maven Central. The system standardizes the dependencies we use. This has served us well.

But recently, our physical server has been going down, which means our developers can’t compile. We thought about hosting Nexus on Amazon EC2, which will incur a monthly charge for a 24/7 server that will be used a fraction of that time.

I discovered that we can host our artifacts (privately or publicly) on an Amazon S3 service. S3 is a storage service, basically a data repo for any kind of file. Maven artifacts are basically just files so it’s a perfect match.

There are several maven plugins out there that’ll let you deploy and download artifacts to/from S3. I used spring-projects/aws-maven.

The basic instructions are in the README of that github project, but it is missing a crucial configuration element and doesn’t show how you generate the access keys. I’ll provide all the instructions below for completeness.

Benefits of S3 repo vs Nexus

Before the How, let’s go into the Why.

First is cost. S3 storage is cheaper than a running EC2 instance. Current S3 pricing is at $0.023/GB. Say you’ve got 10GB of artifacts, that’s $0.23/month. Current on-demand EC2 pricing for a medium instance is at $0.0464/hr, so that’s $33.41/month. With reserved instances, you get an average savings of 40%, but that’s still $20.05, or 100x the price of S3 storage.

Second, but perhaps even more important is reliability and availability. Your own EC2 server could get hacked unless you’re diligent about keeping up with security patches. Your server can go down for maintenance or other things you forgot about. Also, you’d have to set up your own backup solution. With S3, I think it’s way less likely it will go down. You don’t have to worry about redundancy or backups. It will just work when you need it.

I should point out that Nexus has some benefits as well. It has a nice web console which has its own access controls and search feature. With S3, you’ll manage access through IAM services but I don’t believe there’s a way to search S3 Buckets. Nexus will also create the appropriate metafiles around each artifact, which I’ll talk about below. S3 is just a plain storage service and knows nothing about maven artifacts.

You’ll have to weigh the pros and cons yourself. If you decide on S3, continue reading…

Set up your S3 Bucket

In your AWS console, select the S3 service.
Select Create bucket and follow the instructions.
Once you have your bucket, create two top-level folders in it called “release” and “snapshot” like this
Screen Shot 2017-11-07 at 4.35.02 PM

Update your maven settings.xml

<settings>
  ...
  <servers>
    ...
    <server>
      <id>aws-release</id>
      <username>0123456789ABCDEFGHIJ</username>
      <password>0123456789abcdefghijklmnopqrstuvwxyzABCD</password>
    </server>
    <server>
      <id>aws-snapshot</id>
      <username>0123456789ABCDEFGHIJ</username>
      <password>0123456789abcdefghijklmnopqrstuvwxyzABCD</password>
    </server>
    ...
  </servers>
  ...
</settings>

The access key should be used to populate the username element, and the secret access key should be used to populate the password element.

Here’s how you get your access key and secret key (currently)

In your AWS console, select the IAM Service.
On the left panel, select Users.
Then select your user from the list.
In the tabs section, select Security credentials
Then click Create access key.
Screen Shot 2017-11-07 at 4.17.56 PM

Hitting that button will pop up a dialog with your Access key ID (for username) and Secret access key (for password).
Screen Shot 2017-11-07 at 4.23.01 PM

Next, set up the repository where artifacts/dependencies are to be downloaded

<settings>
  ...
  <profiles>
    ...
    <profile>
        <id>aws-release-profile</id>
        <activation>
            <activeByDefault>true</activeByDefault>
        </activation>
        <repositories>
            <repository>
                <id>aws-release</id>
                <url>s3://<BUCKET>/release</url>
            </repository>
            <repository>
                <id>aws-snapshot</id>
                <url>s3://<BUCKET>/snapshot</url>
            </repository>
        </repositories>
    </profile>
    ...
  </profiles>
  ...
</settings>

If you have any remnants of your old repository, you can remove or comment them out now. You may find them within the <servers>, <profiles>, or <mirrors> sections

Update your project’s pom.xml

Include the maven plugin that will do the work of communicating with S3, deploying and downloading artifacts.

<project>
  ...
  <build>
    ...
    <extensions>
      ...
      <extension>
        <groupId>org.springframework.build</groupId>
        <artifactId>aws-maven</artifactId>
        <version>5.0.0.RELEASE</version>
      </extension>
      ...
    </extensions>
    ...
  </build>
  ...
</project>

Add this so you can publish your project to S3

<project>
  ...
  <distributionManagement>
    <repository>
      <id>aws-release</id>
      <name>AWS Release Repository</name>
      <url>s3://<BUCKET>/release</url>
    </repository>
    <snapshotRepository>
      <id>aws-snapshot</id>
      <name>AWS Snapshot Repository</name>
      <url>s3://<BUCKET>/snapshot</url>
    </snapshotRepository>
  </distributionManagement>
  ...
</project>

Migrating your repository to S3

To migrate your artifacts to S3, simply copy the files along with its folder structure to S3. You grab the folders and files directly from your existing Nexus repository or from your local repository.

To copy the entire contents of your ~/.m2/repository folder into s3://<BUCKET>/release, go into the release folder and click Upload

When you need to add additional artifacts, simply create the folder structure under s3://<BUCKET>/release and upload the jar into it. This is one drawback compared with Nexus. Nexus will automatically create a bunch of metafiles, like a pom file to describe it and sha1 files to verify its integrity. You won’t have these files so others will get a warning, but you can ignore that. Or you can generate these files yourself.

Clear your local repository

You’ll need to clear out your old repository or maven will complain that the cache is stale. Locate your local repository (on my machine it was under ~/.m2/repository). The just delete it. Or if you’re not comfortable, you can rename or move it until you’re confident that your new setup is working as expected.

That’s it. Now try your mvn compile, package, deploy commands.

Advertisements
Tagged , ,

Deserializing json to java objects (but not all of it)

This post is about deserializing using Java and JAX-RS (Jersey) when you have a piece of json that’s amorphous (not well defined)

You have some json that generally looks like this

{
  id: 1
  config: {
    foo: "lalala"
  }
}

But the “config” is actually amorphous, so it might also look like this

{
  id: 1
  config: {
    bar: 15
  }
}

So you build a REST API like this

@POST
@Path("/api")
public void doStuff(Thing thing) {
  ...
}

And what does “Thing” look like?

class Thing {
  public int id;
  public ??? config;
}

How do you define config? Is it a class with foo or bar?
I needed to keep it as a string, since I’m just storing it in the database.

So let’s write it as

  public String config;

But this won’t work. You’ll likely get an error like this

Can not deserialize instance of java.lang.String out of START_OBJECT token at [Source: org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream@6ed35c92; line: 1, column: 39] (through reference chain: Thing["config"])

You will need to write a Deserializer for it like this

import java.io.IOException;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.TreeNode;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;

public class KeepAsJsonDeserialzier extends JsonDeserializer<String> {

    @Override
    public String deserialize(JsonParser jp, DeserializationContext ctxt)
            throws IOException, JsonProcessingException {

        TreeNode tree = jp.getCodec().readTree(jp);
        return tree.toString();
    }
}

Credit goes to StackOverflow

And then annotate the field or setter method with this

  @JsonDeserialize(using=KeepAsJsonDeserializer.class)
  public String config;
Tagged , , , ,