Tag Archives: java

Deserializing json to java objects (but not all of it)

This post is about deserializing using Java and JAX-RS (Jersey) when you have a piece of json that’s amorphous (not well defined)

You have some json that generally looks like this

  id: 1
  config: {
    foo: "lalala"

But the “config” is actually amorphous, so it might also look like this

  id: 1
  config: {
    bar: 15

So you build a REST API like this

public void doStuff(Thing thing) {

And what does “Thing” look like?

class Thing {
  public int id;
  public ??? config;

How do you define config? Is it a class with foo or bar?
I needed to keep it as a string, since I’m just storing it in the database.

So let’s write it as

  public String config;

But this won’t work. You’ll likely get an error like this

Can not deserialize instance of java.lang.String out of START_OBJECT token at [Source: org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream@6ed35c92; line: 1, column: 39] (through reference chain: Thing["config"])

You will need to write a Deserializer for it like this

import java.io.IOException;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.TreeNode;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;

public class KeepAsJsonDeserialzier extends JsonDeserializer<String> {

    public String deserialize(JsonParser jp, DeserializationContext ctxt)
            throws IOException, JsonProcessingException {

        TreeNode tree = jp.getCodec().readTree(jp);
        return tree.toString();

Credit goes to StackOverflow

And then annotate the field or setter method with this

  public String config;
Tagged , , , ,

Quick Ubuntu 16 Setup with java8, mysql 5.7, tomcat7

sudo apt-get update

# get latest java, which is java8 at time of writing
sudo apt-get install default-jdk

# get latest mysql, which is mysql5.7 at time of writing
sudo apt-get install mysql-server

# get tomcat7
sudo apt-get install tomcat7
Tagged , , ,

Dear Mom, Yours Truly, Program

Have you ever wanted to write an email to your mom… sent by your program? Of course! What respectable programmer hasn’t. Well, this tutorial will show you how to do it from Java.

import javax.mail.Message;
import javax.mail.PasswordAuthentication;
import javax.mail.Session;
import javax.mail.Transport;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeMessage;

  private String SMTP_HOST = "smtp.office365.com";
  private String SMTP_PORT = 587;
  private boolean DEBUG = true;

  private static void sendEmail(String contentType, final String login, final String password,
                                String fromEmail, String replyToEmail, String[] a_to, String a_subject,
                                String a_contents) {
    try {
      Properties props = new Properties();
      props.put("mail.smtp.auth", "true");
      props.put("mail.smtp.starttls.enable", "true");
      props.put("mail.smtp.host", SMTP_HOST);
      props.put("mail.smtp.port", SMTP_PORT);
      Session session = Session.getInstance(props,
          new javax.mail.Authenticator() {
        protected PasswordAuthentication getPasswordAuthentication() {
          return new PasswordAuthentication(login, password);
      if( DEBUG ) session.setDebug(true);

      MimeMessage message = new MimeMessage(session);
      message.setFrom(new InternetAddress(fromEmail));
      for (String toTarget : a_to) {
        message.addRecipient(Message.RecipientType.TO, new InternetAddress(
      message.setFrom(new InternetAddress(fromEmail));
      message.setReplyTo(new InternetAddress[] { new InternetAddress(replyToEmail) });
      message.setContent(a_contents, contentType);
      message.setHeader("Content-Transfer-Encoding", "7bit");

    } catch (Exception e) {
      throw new RuntimeException("Unable to send HTML mail", e);

I won’t explain the code. Just use it. You’re smart. You’ll figure it out.

So let’s say we wanted to send out from a gmail account. Just change SMTP_HOST to “smtp.gmail.com” and we’re good right? No, not quite.

First of all, you need to create an App password as opposed to using your account login password.

Then you’ll test it on your local machine and exclaim “Yay! It works!”. Then you push it to your Amazon server and go to Friday Happy Hour. Then on Sat at 4am, you will get an error that wakes you up in the middle of the night saying Email failed like this

Caused by: javax.mail.MessagingException: [EOF]
        at com.sun.mail.smtp.SMTPTransport.issueCommand(SMTPTransport.java:1481)
        at com.sun.mail.smtp.SMTPTransport.helo(SMTPTransport.java:917)
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:417)
        at javax.mail.Service.connect(Service.java:310)
        at javax.mail.Service.connect(Service.java:169)
        at javax.mail.Service.connect(Service.java:118)
        at javax.mail.Transport.send0(Transport.java:188)
        at javax.mail.Transport.send(Transport.java:118)

Why didn’t it work? Seems Google smtp servers have blocked emails from being sent from AWS machines. Why? Because they hate you, and also probably to prevent spam machines from polluting our internet.

So what can you do to send from a Gmail account? You have to use Google’s own brand of code. It looks the same, but take a close look at the imports.

import com.google.code.javax.mail.Message;
import com.google.code.javax.mail.PasswordAuthentication;
import com.google.code.javax.mail.Session;
import com.google.code.javax.mail.Transport;
import com.google.code.javax.mail.internet.InternetAddress;
import com.google.code.javax.mail.internet.MimeMessage;

  private static void sendEmail(String contentType, final String username, final String password,
                                String fromAddr, String replyToEmail, String[] toAddr, String subj,
                                String txt) {
    Properties props = new Properties();
    props.put("mail.smtp.auth", "true");
    props.put("mail.smtp.starttls.enable", "true");
    props.put("mail.smtp.host", "smtp.gmail.com");
    props.put("mail.smtp.port", "587");
    Session session = Session.getInstance(props,
        new com.google.code.javax.mail.Authenticator() {
          protected PasswordAuthentication getPasswordAuthentication() {
            return new PasswordAuthentication(username, password);
    try {
      Message message = new MimeMessage(session);
      message.setFrom(new InternetAddress(username, fromAddr));
      for (String ta : toAddr) {
      message.setContent(txt, "text/html; charset=utf-8");
    } catch (Exception e) {
      throw new RuntimeException(e);

Ok, now go write your mom an email.

Tagged , , , ,

Html link extractor

Just a handy regex for extracting url and link text from an html page

      String pattern = "<a(.+?)href=\"([^\"]+)\"[^>]*>(.+?)</a>";
      Pattern r = Pattern.compile(pattern);
      Matcher m = r.matcher(table);
      while( m.find() ) {
        String ahreflink = m.group(0);
        String url = m.group(2);
        String linktext = m.group(3);

The ? symbol makes it a lazy match, matching up to the next word “href” or ““

Tagged , , ,

Java is expensive (in terms of memory usage)

I was asked to explain once why my program consumed so much memory when the amount of actual data it loaded was so small. It is often assumed that the culprit is either duplicate data or unused data that the developer carelessly placed in the app.

I asked a coworker to run a simple experiment once. To read a 2GB file into a HashSet. The program required almost 12GB to run. That’s a 6x increase in memory requirement!!!

It turns out that the java vm is very expensive when it comes to storing data.
I found a study by IBM that describes this phenomenon. It’s linked here.

Here are some highlights from the slide deck:
– for a string, 20%-75% of memory footprint is overhead! That means as low as 25% of it is actual data
– an example TreeMap is 82% overhead while an array (that requires binary search) is 2% overhead
– trying to magically solve memory problems without understanding consequences is bad
– upgrading from 32-bit to 64-bit to address more memory increases object memory overheads
– using 64-bit jvm increases memory usage by 40-50% on average compared to 32-bit
– e.g. an 8-byte string requires 64 bytes in 32-bit java5 and 96 bytes in 64-bit java5
– leaving caches unbounded to have higher hit-rate can cause excessive GC that slows performance
– frameworks add additional memory requirements
– none of this even includes duplicate data or unused data

And some lessons learned:
– consider context and usage statistics of collections
– sometimes an IdentityHashMap is better (50% less memory and faster)
– use array/list when number of elements is small
– research found that only 5% of sets had more than a few elements each
– set initial capacity where possible
– trim to size after load
– use weak or soft references so GC can remove unused objects (apache’s ReferenceMap)

Now let’s all be a little more aware of how we’re building our applications so they don’t lead to memory obesity.

Tagged , ,

Finding a needle.class in (multiple) haystack.jar


Sometimes, you get an error like this:

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/collect/Maps
at com.google.gdata.wireformats.AltRegistry.<init>(AltRegistry.java:118)
at com.google.gdata.wireformats.AltRegistry.<init>(AltRegistry.java:100)
at com.google.gdata.client.Service.<clinit>(Service.java:555)
at sample.contacts.ContactsExample.<init>(ContactsExample.java:133)
at sample.contacts.ContactsExample.main(ContactsExample.java:609)

It means you’re missing a class, specifically com.google.common.collect.Maps.

Let’s say you download a package and get a million jars. How do you know if that class is in the jars and which one?

Try this linux script:

for i in *.jar; do jar -tvf "$i" | fgrep Maps; done

(note the quotes need to be regular quotes, and not the stylized ones WordPress imposes on me here)

If it’s found, you can print out all the classes in the jars into a text file and scan through it for the jar that contains your class

for i in *.jar;
  do echo "$i" >> myjarcontents.txt;
  jar -tvf "$i" >> myjarcontents.txt;
Tagged , ,