Category Archives: Coding

Dockerizing a Nodejs app

A quick tutorial on how to dockerize your nodejs app.

I have a nodeapp contained in a folder called myNodeApp. Here’s what the folder structure looks like

Screen Shot 2017-07-26 at 2.35.41 PM

Under myNodeApp, create a file called Dockerfile. Here are the contents of that file

FROM node:latest
WORKDIR /app
COPY . /app
CMD node server.js
EXPOSE 3333

If this directory is a git repo, like mine is, then you will want to create a file called .dockerignore to prevent your git info from being dockerized. Here are the contents of .dockerignore

.git
.gitignore
.DS_Store

My app takes configuration settings so one way to do that is through environment variables which you can pass in when you create the container. You can use the -e flag to pass in environment parameters but I prefer to use a properties file.

So I created a docker.env file with my environment properties.
e.g.

MY_PROPERTY=myValue

Next we create the image. In the myNodeApp folder run:

docker build -t myNodeApp .

Now that you have the docker image, you can instantiate a container like so

docker run -t -p 3333:3333 --env-file ./docker.env myNodeApp

Remember you can start and stop your container with this command:

docker start/stop [container_id]

Angular: Controls between parent and directives

If you need the parent controller to call a directive’s scope function, then this is how you need to set everything up. Stackoverflow

If you need a directive to call the parent’s scope function, then this is how you set them up. Stackoverflow

Tagged

Tomcat ContextPath and Angularjs

An interesting obstacle arose when I tried to use Angularjs (NG) in Tomcat, specifically in regards to the path.

Typically, I’ve been using NG assuming everything starts at the root path. So my index.html is at http://domain.com/index.html.

However, in Tomcat, each webapp has its own starting path. So you may have a weather webapp that is at http://domain.com/weather/ and a clock webapp at http://domain.com/clock/.

How do you configure NG within tomcat?

You could hardcode your starting path. But imagine if you wanted to have two clock webapps in tomcat, renaming one to http://domain.com/clock1/ and the other http://domain.com/clock2.

Now you would have to change all the references of “clock” to “clock1” and “clock2” respectively. And there are many references…

In the index.html, you have to change all your <link …css> as well as all <script …js>. You might think that you can outsmart yourself by changing the index.html to a index.jsp and using <%=request.getContextPath()%>.

But then you still have to change references of the hardcoded path in your .js files, such as your app.js that defines your module and routes.

And don’t even get me started on all your templates that reference .js and .css.

Well, let me jump straight to the short answer.

In your index.html, simply add this within your <head> tag

<head>
    <base href="/clock/" />
</head>

That’s it. Now all your relative paths will have that base href prefixed to them automatically.

So <link href=”mystyle.css”> will automatically fetch from http://domain.com/clock/mystyle.css

<script src=”myscripts.js>” will automatically fetch from http://domain.com/clock/myscripts.js

Even app.js will leverage this base when using relative url’s.
e.g. $routeProvider.when(‘/’, {templateurl: ‘home.html’})
will automatically reference http://domain.com/clock/home.html

This html standard can even be used on jsp pages. Anything that is html will take advantage of it

Tagged , , ,

The power of Shadow DOM

I recently had to inject another site’s webpage into a div in our own site. Back in early 2000, you’d use something like an but this is 2017.The requirements are the following:

  1. Pull the html source of the other site’s webpage
  2. Display it in our own webpage
  3. Make sure all styles are applied
    1. Ensure that styles don’t spill over to our own page’s DOM

Requirement 1: Pull the html source of the other site’s webpage

If you’ve done any web programming in the last decade, you know that simply pulling the html from another site from the frontend javascript is a big no-no, and blocked by cross-site-scripting restrictions. The trick is to have the backend webserver fetch the data and send it to the frontend. In a way, you’re proxying the data. This is a lesson for another time and not the focus of this post.

Requirement 2: Display it in our own webpage

This is actually pretty trivial once your backend webserver sends you the data. If you’re using node, make sure you’re “trusting” the source using $sce.trustAsHtml(htmlString).

Requirement 3: Make sure all styles are applied

But once you do this, you’ll soon realize that while the content is shown, the styling is missing. That’s because the .css files aren’t being fetched, and there’s really no good way of doing that.

You could parse the html and manually fetch the .css. But you’ll run into another problem. That site’s css now conflicts with your own site’s css and you get some weird mixup of styles.

If only there was a way to fetch the css and restrict it to the specific subtree of the DOM…

Shadow DOM

Here’s where this new (yet to be official) W3C standard comes in. As of this writing, Shadow DOM is only supported on Chrome 53.0+ and Safari 10.0+, and Android Webview 53.0+.

The idea is that you can create a subtree within the DOM, but it’s a special shadow subtree. What makes it special is that you can restrict css to just within this DOM.

Why is this such a big deal? Well, it solves the problem at hand. But it also solves a much more general problem.

Have you ever added an id attribute in a page only to find that you or someone else created that same id value on another page? That wasn’t such a big deal back in 2010, but many popular frontend web frameworks (e.g. angularjs, backbone)  consolidate all your pages into one and routes between them. What that means is that id’s can conflict. Even class-based css styles can conflict between two pages (how often have you created a class=”value”, no? just me?)

Anyways, Shadow DOM creates a true separation of DOM subtrees, meaning developers can build webpages as individual modules and not worry about conflicting id’s, classes, and styles. And with websites getting increasingly more complex, a finer more modular approach is needed.

Shadow DOM isn’t official yet, but I would bet my right arm that it will be.

Here’s some angular code to see how simple it is in action.

<div id="shadowSubtree"></div>
...
var shadow = angular.element('#shadowSubtree')[0].attachShadow({mode:'open'});
shadow.innerHTML='<link type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"></link><a class="btn">Bootstrap styled button</a>';

Tagged

When you need things to line up

Suppose you have the following html

<div class="container">
  <div class="thing"></div>
  <div class="thing"></div>
  <div class="thing"></div>
</div>

If you want things to line up horizontally, you could use a tried and true method

.thing {
    float:left;
}

But if the width of the container is not big enough, these things will wrap to the next line.

e.g.

.container {
    width: 50px;
}

Sometimes, you want them to line up and then just cut off when the container ends. For these cases, substitute it with these settings.

.container {
    white-space: nowrap;
}
.thing {
    display: inline-block;
}

If you actually wanted to see the rest of the things, then you could add a horizontal scroll bar

.container {
    overflow-x: scroll;
}
Tagged

Setting up Tomcat for debugging

Short PSA. If you want to set up Tomcat for debugging, set up JPDA.

To do so, simply open the startup.sh file, which is somewhere in TOMCAT/bin folder.

Add these two lines near the top

export JPDA_ADDRESS=7000
export JPDA_TRANSPORT=dt_socket

Then modify the exec line (usually the last line in the file) from this:

exec "$PRGDIR"/"$EXECUTABLE" start "$@"

to this

exec "$PRGDIR"/"$EXECUTABLE" jpda start "$@"
Tagged ,

Upgrading to Jersey 2.x in Tomcat8

Jersey 1.x is still supported. As of this writing, version 1.19.3 was just released on Oct 24th, 2016.

But recently I discovered that Tomcat8 doesn’t play well with Jersey 1.x so we simply have to upgrade to Jersey 2.x. Easy right? (If it were, I wouldn’t be writing this post.)

Jersey 1.x
Let’s review the Jersey 1.x configurations first

As far as maven dependencies, here’s what I used in my pom.xml

<dependency>
  <groupId>com.cedarsoft.rest</groupId>
  <artifactId>jersey</artifactId>
  <version>1.0.0</version>
</dependency>

<dependency>
  <groupId>com.sun.jersey</groupId>
  <artifactId>jersey-json</artifactId>
  <version>1.5</version>
</dependency>

com.cedarsoft.rest:jersey is a bundle that includes these dependencies.

Unfortunately, I could find no such bundle for Jersey 2.x so I had to mix and match until trial-and-error led me to a workable solution. (I save you the trouble.)

Here’s my old web.xml

<web-app version="2.4"
 xmlns="http://java.sun.com/xml/ns/j2ee"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
  <servlet>
    <servlet-name>JerseyREST</servlet-name>
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
    <init-param>
      <param-name>com.sun.jersey.config.property.packages</param-name>
      <param-value>INSERT_MY_PACKAGES_AND_CLASSES_HERE</param-value>
    </init-param>
    <load-on-startup>2</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>JerseyREST</servlet-name>
    <url-pattern>/rest/*</url-pattern>
  </servlet-mapping>
</web-app>

Jersey 2.x
Now here are the changes for Jersey 2.

   <dependency>
     <groupId>javax.servlet</groupId>
     <artifactId>javax.servlet-api</artifactId>
     <version>3.1.0</version>
     <scope>provided</scope>
   </dependency>
   <dependency>
     <groupId>org.glassfish.jersey.containers</groupId>
     <artifactId>jersey-container-servlet-core</artifactId>
     <version>2.13</version>
   </dependency>
   <dependency>
     <groupId>org.glassfish.jersey.containers</groupId>
     <artifactId>jersey-container-servlet</artifactId>
     <version>2.13</version>
   </dependency>
   <dependency>
     <groupId>com.fasterxml.jackson.core</groupId>
     <artifactId>jackson-databind</artifactId>
     <version>2.8.5</version>
   </dependency>
   <dependency>
     <groupId>org.glassfish.jersey.media</groupId>
     <artifactId>jersey-media-json-jackson</artifactId>
     <version>2.13</version>
   </dependency>
   <!-- jersey file upload dependencies -->
   <dependency>
     <groupId>org.glassfish.jersey.media</groupId>
     <artifactId>jersey-media-multipart</artifactId>
     <version>2.13</version>
   </dependency>

org.glassfish.jersey.media:jersey-media-multipart is only required for file upload capability.

Here’s the new web.xml

<web-app version="3.1"
         metadata-complete="false"
         xmlns="http://java.sun.com/xml/ns/j2ee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
                             http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd">
  <servlet>
    <servlet-name>JerseyREST</servlet-name>
    <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
    <init-param>
        <param-name>jersey.config.server.provider.classnames</param-name>
        <param-value>org.glassfish.jersey.media.multipart.MultiPartFeature</param-value>
    </init-param>        
    <init-param>
      <param-name>jersey.config.server.provider.packages</param-name>
      <param-value>
        INSERT_MY_PACKAGES_AND_CLASSES_HERE
      </param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
      <servlet-name>JerseyREST</servlet-name>
      <url-pattern>/rest/*</url-pattern>
  </servlet-mapping>
</web-app>

Notice, the new servlet class.
It’s also important to load the org.glassfish.jersey.media.multipart.MultiPartFeature to support file upload capability.

You might also have to change some of your code.
For example, all your @JsonIgnore annotation classes will have to change from
org.codehaus.jackson.annotate.JsonIgnore
to
import com.fasterxml.jackson.annotation.JsonIgnore

Tagged ,

Quick Ubuntu 16 Setup with java8, mysql 5.7, tomcat7

sudo apt-get update

# get latest java, which is java8 at time of writing
sudo apt-get install default-jdk

# get latest mysql, which is mysql5.7 at time of writing
sudo apt-get install mysql-server

# get tomcat7
sudo apt-get install tomcat7
Tagged , , ,

Meet Watson

I had the opportunity to attend the IBM Watson Conference in San Francisco a few weeks ago. It was an amazing event to showcase the new Watson AI Platform.

IBM Watson is a platform that encompasses many AI and ML (machine learning) services. The so-called Watson Services include API’s to help you understand natural language, convert speech-to-text and text-to-speech, build chatbots, recognize images, perform trade-off analytics and much more.

You can try out demos of some of the services to get a feel of how well they work. Just click on Launch app link next to each Starter Kit.

I believe many of the AI technologies underneath the hood of Watson have been around for many years. I know I have used some of them via different open source toolkits.

However, IBM Watson introduces three game-changing characteristics:

  • Accessibility of knowledge and tools
  • Integration of services
  • Fundamental change in the business model

Accessibility
IBM designed their AI tools in such a way that they are more accessible, not only to non-AI experts, but even to non-developers. Their user interface makes tasks such as training and identifying images very easy to use, without a Phd in computer science  or even programming know-how.

Having a single tool can be useful but is still limiting. More interesting applications will rise from the integration of all your tools. How much can you do with a single screwdriver, right?

Integration
To that effect, Watson supplies you with a workbench of AI tools. The value of the platform is that it can seamlessly integrate and manage all services in one place. There’s a common language between the services so to speak.

Cost Model
Finally, this platform is offered as an on-demand pay-as-you-go service. IBM has historically been known as an enterprise software service. Expect to come with a check with lots of zeros behind it if you want to use something IBM-branded. But Watson has an Amazon Web Services model where you only pay for the service calls and CPU time used on their machines. Instead of millions of dollars to get started, developers can get started for free and small businesses can probably run on a few hundred dollars a month.

Final Thoughts
The knowledge and tools that were once locked in the hands of select companies and experts is now open to the world, and it’s available at a fraction of the cost. Microsoft is moving towards this model and Google has already released free AI tools.  I believe we are entering a new world, where the greatest value will come from those who can blend just the right AI/ML services into the most interesting applications.

As an AI developer, this scares me a bit, but personally, I welcome this new world. It will mean change on my part. I believe my value and that of my company’s will be to offer expert consultation on services like Watson’s and others, to show you the possibilities as well as the boundaries of this new world.

Disclaimer: While I worked for IBM in a previous life, I am not affiliated with them in any way. I am part of a smaller, more agile AI company now that is agnostic to the tools we use. The opinions in this piece are my own, and do not necessarily reflect IBM or my current company’s views.

 

Tagged ,

How to hash mysql varchar/string into different bins

First off, what am I talking about?
Here’s a scenario.

You have a bunch of rows in your table that are URLs. You want to select an evenly distributed random set of them every so often and check if they’re still alive. How do you do this?

You could just take a range.

Get the first 10

SELECT url FROM myTable
ORDER BY url
LIMIT 0, 10

Get the next 10

SELECT url FROM myTable
ORDER BY url
LIMIT 10, 10

And the next 10…

SELECT url FROM myTable
ORDER BY url
LIMIT 20, 10

But what if you wanted to evenly distribute your checks because the URLs from the same site are sequentially ordered.

Well, you could ensure an auto-increment ID and just get chunks of them based on a mod.

So to get every 10th one

SELECT id, url FROM myTable
WHERE id % 10 = 0 

And the next 10th sequence

SELECT id, url FROM myTable
WHERE id % 10 = 1 

And the next 10th sequence

SELECT id, url FROM myTable
WHERE id % 10 = 2 

This is still not that well distributed since you might run into large clusters of site url’s (ie more than 10).
But this also requires a nice integer column value.

Instead, you could also just use the VARCHAR value like the URL itself

So to get the first 10

SELECT id, ... FROM myTable
WHERE CAST(CONV(SUBSTRING(MD5(url), 1, 16), 16, 10) AS SIGNED INTEGER) % 10 = 0

To get the next 10

SELECT id, ... FROM myTable
WHERE CAST(CONV(SUBSTRING(MD5(url), 1, 16), 16, 10) AS SIGNED INTEGER) % 10 = 1

And so on…

Let’s go over what this does.

The MD5() is a hash function that will convert your VARCHAR instead a seemingly random sequence of numbers of letters. It’s not random though. It always converts to the same sequence, but distributes the VARCHAR sequence of characters more uniformly.

The SUBSTRING(…, 1, 16) takes the first 16 digits of the MD5 hash value. I believe this gives you the first 64 bits of it, otherwise there’s a possible overflow error.

The CONV(…, 16, 10) function converts the hash (which is a hex or base-16 value) into a base-10 value.

The CAST(… AS SIGNED INTEGER) function converts it to a signed integer. (If you’re going to read this value into java, you want a signed integer otherwise you’ll get an overflow)

Then simply mod (%) it by the number of bins you want. In my example, I modded it with 10.

Tagged