Anyone who has ever written software at a large scale understands the feeling of building something that doesn’t match what the customer/product owner had in mind. Whether you wasted 1 … Continue reading How a shared understanding leads to faster delivery
If you’ve ever used Amazon’s AWS console you’ll know that it could do with some improvement. It’s easy to get lost in the myriad menus and there are many annoying bugs. It’s also slowwwwww.
I’ve heard that this is no accident – Amazon wants the community to provide the tooling for them. This ties in well with the fact that the API AWS is actually pretty good.
I decided to save myself some time and produce one of these community solutions. I decided to focus on ECS (EC2 container service) as this is the service I interact with the most. ECS is AWS’s way of managing Docker containers running on EC2 instances. Some of the terminology can be a little confusing, but overall it’s a good product.
I built ecs_explorer to cover some of the annoying tasks I do on the console; looking for small pieces of information in a given container/service. For example, I might need the IP address of the container that’s running a certain task. In the console, this can take >20 clicks, in ecs_explorer I can find this in 10 secs.
You can find ecs_explorer on GitHub. The main aim is to get you to the information you need as quickly as possible. It supports navigation of all ECS objects, viewing important details as well as viewing the full JSON if you need more information. One of the key bugbears with the web console, filtering resources, is much improved. Another advantage over the console is that there’s no need to login.
Installation is easy. Once your credentials are available via env variables or a dot file ( see here for more information), it’s just a case of running pip install ecs_explorer. The CLI is now available to run from anywhere with ecs_explorer -h showing the available options.
Give it a try and tell me what you think!
I recently got a new work computer (a 15″ MacBook Pro) and was given the choice of copying across the image from my old one or starting afresh. It’s always good to take stock of what software you are using so I chose the second option. Here’s what I installed:
From my point of view, the most essential piece of software that you can install on MacOS. It’s a package manager that installs command line software as well as GUI applications. It gives you a way to install software in a generic way and from a reputable source, upgrade it when a new version comes out, and a way to keep track of what you’ve already installed. Unless stated, all the following software is installed using it.
The Terminal Client that comes with MacOS is not very good and it doesn’t need to be because ITerm2 exists. It is easy to configure, contains a multitude of shortcuts and helpful features and just gets out of your way.
I spend a lot of time on the command line and I’m not a huge Bash fan (the default shell that is installed in MacOS). Fish feels intuitive, and easy to learn. Function are easy to write, autocompletion works better than you could expect and you can always switch back to bash if you need to run a specific Bash script.
Configuring Fish manually is an intimidating task. Oh-my-fish (named after the Zsh equivalent) gives you a set of packages and themes to install. Themes are the most interesting feature for me. I use the bobthefish theme to display interesting information on my shell prompt. It tells me what directory I’m in, what the return status of the last command was and has git integration too.
IntelliJ is a Java IDE but calling it that does it a disservice. It’s also a terminal client, a database client, and a workflow manager. The Java coding experience is brilliant and it makes huge projects easy to browse. It does have it’s downsides: it eats memory and CPU and doesn’t cope brilliantly with generated source code.
I’ve been using Jenkins 2 more and more in my projects and I’ve found that the Pipeline feature is a great fit for the jobs that I am writing. However, the documentation isn’t fantastic for the older plugins. One example of this is static code analysis tools such as PMD, CheckStyle and FindBugs. In old Jenkins, there would be a screen to configure how the results of these plugins affected the results of the build:
Obviously, in a Pipeline job, you have to configure this via code rather than on a web page, but it’s not documented how (that I could find). I wanted the build to be UNSTABLE if any warnings were generated. After a bit of digging, I figured out it should look like this:
step([$class: 'hudson.plugins.checkstyle.CheckStylePublisher', pattern: '**/target/checkstyle-result.xml', unstableTotalAll:'0'])
PMD and FindBugs are very similar as they are based on the same Publisher class.
step([$class: 'hudson.plugins.checkstyle.CheckStylePublisher', pattern: '**/target/checkstyle-result.xml', unstableTotalAll:'0']) step([$class: 'PmdPublisher', pattern: '**/target/pmd.xml', unstableTotalAll:'0']) step([$class: 'FindBugsPublisher', pattern: '**/findbugsXml.xml', unstableTotalAll:'0'])
The more complicated configuration options that are listed on the web page above can be found in this class. All the static code analysis tools extend HealthAwareRecorder and so the configuration is the same. Use the setters to push configuration via properties in the step definition. For example, if I wanted to set the healthy threshold to 20 and the unhealthy threshold to 100, then by looking at the class, I find
/**
* Returns the healthy threshold, i.e. when health is reported as 100%.
*
* @return the 100% healthiness
*/
@Override
@CheckForNull
public String getHealthy() {
return healthy;
}
/**
* @see {@link #getHealthy()}
*/
@DataBoundSetter
public void setHealthy(final String healthy) {
this.healthy = healthy;
}
/**
* Returns the unhealthy threshold, i.e. when health is reported as 0%.
*
* @return the 0% unhealthiness
*/
@Override
@CheckForNull
public String getUnHealthy() {
return unHealthy;
}
/**
* @see {@link #getUnHealthy()}
*/
@DataBoundSetter
public void setUnHealthy(final String unHealthy) {
this.unHealthy = unHealthy;
}
so I would use
step([$class: 'hudson.plugins.checkstyle.CheckStylePublisher', pattern: '**/target/checkstyle-result.xml', healthy:'20', unHealthy:'100'])
An example of this can be found on my GitHub account here.
Clearly this is more complicated that just filling in the values on the web site, so it will be interesting to see if a snippet generator pops up for this.
Unstructured data is all around us (emails, log files, facebook posts, twitter statuses) yet it’s difficult to analyse. HP has a product that does some common analysis tasks for you. It’s called IDOL (Intelligent Data Operating Layer) and comes with an easy to call web API (IDOL OnDemand https://www.idolondemand.com). In this post I’m going to look at how to call a few of the text analysis endpoints from a Java program.
Companies need to understand how their product is viewed and a useful way of discovering this is analyse posts on social media. We can get a rough measure of whether a user thinks positively or negatively about it by performing sentiment analysis on the post. So, let’s build an application that looks at tweets on a given subject and displays the sentiment along with some other information.
The three endpoints we will use are
The endpoints are called using one http GET with parameters attached. Here’s how we do this in Java.
First we create the URL. Let’s take the sentiment analysis call as an example. The three things that we need to provide (as shown by the documentation) are
We create a request object
public class SentimentAnalysisRequest {
private static final String REQUEST_STRING = "detectsentiment";
private static final String VERSION_STRING = "v1";
private final ProcessRequestType type;
private final String identifier;
private final SentimentLanguage language;
public SentimentAnalysisRequest(ProcessRequestType type, String identifier, SentimentLanguage language) {
this.identifier = identifier;
this.type = type;
this.language = language;
}
public String toUrlComponent(){
return REQUEST_STRING+ "/" + VERSION_STRING + "?" + type.urlSegment + "="+ UrlUtils.urlEncode(identifier)
+ (language==null? "" : "&language="+language.name());
}
public enum SentimentLanguage {
ENG,FRE,SPA,GER,ITA,CHI
}
}
and collapse it to a URL string (toUrlComponent). One thing to notice here is that we need to encode the text to analyse so that we can send it in a URL.
Given text of ‘I like fish for dinner’ the URL ends up looking like this:
The below code shows a standard method for requesting a URL and this is used for all the API endpoints.
private Response executeGet(String url) throws IOException {
HttpURLConnection httpUrlConnection = null;
InputStream inputStream;
try {
httpUrlConnection = (HttpURLConnection) new URL(url).openConnection();
httpUrlConnection.setReadTimeout(20 * 1000);
httpUrlConnection.connect();
inputStream = httpUrlConnection.getResponseCode() != HTTP_OK ? httpUrlConnection.getErrorStream()
: httpUrlConnection.getInputStream();
return new Response(httpUrlConnection.getResponseCode(), fromInputStream(inputStream));
} finally {
closeQuietly(httpUrlConnection);
}
}
The response that we receive is JSON formatted. To do anything useful with this we need to de-serialize it into a Java object. Jackson is a great library for easily doing this — it gets out of your way and doesn’t require mapping files/annotations.
We know what to expect from the response (from the docs) and here is the response for our ‘I like fish for dinner’ example:
{
"positive": [
{
"sentiment": "like",
"topic": "fish for dinner",
"score": 0.7176687736973063,
"original_text": "I like fish for dinner",
"original_length": 22,
"normalized_text": "I like fish for dinner",
"normalized_length": 22
}
],
"negative": [],
"aggregate": {
"sentiment": "positive",
"score": 0.7176687736973063
}
}
Pretty good analysis!
If we provide a java object that closely matches the structure of the JSON then Jackson can do the rest and we don’t have to write much code for the conversion. Here it is:
public class SentimentAnalysisResponse {
public SentimentAnalysisResponse(){}
private List<SentimentDetails> negative;
private List<SentimentDetails> positive;
private SentimentAggregate aggregate;
public SentimentAggregate getAggregate() {
return aggregate;
}
public void setAggregate(SentimentAggregate aggregate) {
this.aggregate = aggregate;
}
public List<SentimentDetails> getNegative() {
return negative;
}
public void setNegative(List<SentimentDetails> negative) {
this.negative = negative;
}
public List<SentimentDetails> getPositive() {
return positive;
}
public void setPositive(List<SentimentDetails> positive) {
this.positive = positive;
}
public static class SentimentAggregate {
private String sentiment;
private Double score;
public String getSentiment() {
return sentiment;
}
public void setSentiment(String sentiment) {
this.sentiment = sentiment;
}
public Double getScore() {
return score;
}
public void setScore(Double score) {
this.score = score;
}
}
public static class SentimentDetails {
public SentimentDetails(){}
public String getSentiment() {
return sentiment;
}
public void setSentiment(String sentiment) {
this.sentiment = sentiment;
}
private String sentiment;
private String topic;
private Double score;
private String normalized_text;
private String original_text;
private Integer original_length;
private Integer normalized_length;
public String getTopic() {
return topic;
}
public void setTopic(String topic) {
this.topic = topic;
}
public Double getScore() {
return score;
}
public void setScore(Double score) {
this.score = score;
}
public String getNormalized_text() {
return normalized_text;
}
public void setNormalized_text(String normalized_text) {
this.normalized_text = normalized_text;
}
public Integer getOriginal_length() {
return original_length;
}
public void setOriginal_length(Integer original_length) {
this.original_length = original_length;
}
public Integer getNormalized_length() {
return normalized_length;
}
public void setNormalized_length(Integer normalized_length) {
this.normalized_length = normalized_length;
}
public String getOriginal_text() {
return original_text;
}
public void setOriginal_text(String original_text) {
this.original_text = original_text;
}
}
}
Note that there is a default constructor (no args) for each object. This is required by Jackson due to the way it constructs objects. Our code to translate the JSON response to this object is
ObjectMapper mapper = new ObjectMapper();
JsonParser parse = new JsonFactory().createParser(response);
SentimentAnalysisResponse resp = mapper.readValue(parse,SentimentAnalysisResponse.class);
It should be clear what we are doing here. Putting it all together looks like this:
public SentimentAnalysisResponse analyseSentimentUsingText(String text, String language) {
SentimentAnalysisRequest.SentimentLanguage lang;
try{
lang = SentimentAnalysisRequest.SentimentLanguage.valueOf(language.toUpperCase());
} catch (IllegalArgumentException e){
lang = null;
}
SentimentAnalysisRequest req = new SentimentAnalysisRequest(
ProcessRequestType.TEXT, text, lang);
String urlComponent = req.toUrlComponent();
String response = null;
try {
response = executeGet(BASE_URL + "sync" + "/" + urlComponent + getApiUrlComponent()).content;
ObjectMapper mapper = new ObjectMapper();
JsonParser parse = new JsonFactory().createParser(response);
SentimentAnalysisResponse resp = mapper.readValue(parse,SentimentAnalysisResponse.class);
return resp;
} catch (JsonMappingException e){
logger.error("error encountered for response " + response + " with text " + text);
return null;
} catch (IOException e) {
logger.error("Exception encountered when trying to analyse sentiment",e);
return null;
}
}
This can be repeated for all of the API endpoints in a similar manner. Once I’ve written the calls for a few more endpoints I’ll publish it as a library.
I’m not going to concentrate on how to get the statuses from Twitter as that is out of scope but you can look at the code for this project <here>. We have a stream of tweets coming in to which we attach extra information from the IDOL API:
These then get sent to a store where they are retrieved when requested from the front-end web page.
The web page is little more than a list of the enriched tweets. It uses JSP to obtain the list then Bootstrap to organise all the elements of the page.
A small bit of JavaScript allows us to see more details about the tweet and we are done! Here is a little GIF of the final version.
The HP IDOL API provides powerful analysis tools in an easy to use format. It can be combined with all sorts of unstructured data sources to create a useful tool in a small amount of time.
To understand why the square in the post before was rendered as a rectangle we need to understand how OpenGL transforms the 3D vertices we specify into 2D window coordinates. This is achieved through a four stage process as shown below.
The section that we need to closely look at is the projection matrix. This is a 4*4 matrix mapping vertices from 
are eventually drawn on screen.
So, we are interested in which section of 
Orthographic projection
An orthographic projection maps a rectangular box to the cube defined above. One property of the orthographic projection is that shapes closer or further away from the ‘camera’ are the drawn as the same size i.e. there is no perspective correlation. I say ‘camera’ with some reserve as there isn’t really a camera in OpenGL. The only sense in which we have a camera is that if we have two opaque objects which are exactly the same except one is at z=-1 and the other is at z=-2, then only the first one will be visible because OpenGL draws the first on top of the second. In this sense the ‘camera’ is pointing down the negative z-axis but still has no real position. We set an orthographic projection using the following command:
void glOrtho(float left, float right, float bottom, float top, float nearZVal, float farZVal );
This type of projection matrix is used for CAD and architectural systems where is is important to be able to compare sizes of objects and in 2D games.
Perspective projection
This performs a mapping from a frustrum to the cube as defined above. This type of projection matrix is used to model the real world as we see it. It has the property that 2 identical objects one close to the origin and one far away appear on the screen as different sizes, the closer the object is, the bigger it seems.
Later in the process this cube is mapped to the 2D window coordinates, so the projection matrix defines what area of the 3D world is eventually visible on screen and more specifically, which vertices that we specify are visible. Our visible area looks like this in the 3D world
Quite simply, any vertex that is in this area after the modelview transformation is drawn on screen. The focus of the frustum (the point where the edges would meet if extended) is always (0,0,0) by definition. The analogy with real-world viewing angles is immediately apparent. If we imagine that our viewer is at (0,0,0) and looking in the direction of the frustum, then we can think of the nearest boundary of the frustum as the computer screen and the far boundary as the point beyond nothing which useful can be seen.
We define a frustum with the following command:
void glFrustum(float left, float right, float bottom, float top, float zNear, float zFar);
where left, right, bottom etc are as definied here. The derivation of the matrix which performs the mapping is derived here.
Both glOrtho and glFrustum can only be called once OpenGL is put in specific mode. It must be told to interpret the following commands in terms of a projective matrix. This command is
glMatrixMode(GL_PROJECTION);
After this command is called, it is good practice to call
glLoadIdentity();
in case there are any commands preloaded into the projection matrix. You should also note that OpenGL takes commands after glMatrixMode accumulatively i.e. if you give lots of commands, you end up with a matrix which is a product of the previous commands.
We return to the original question now. Why is our square drawing incorrectly? It turns out that this problem is due to the size of our window. In our program, we didn’t specify a projection matrix so the default one is used. This is identity matrix, and is equivalent to calling
glOrtho(1,-1,-1,1,-1,1);
So, our square is within the viewing volume and is drawn. It looks exactly as it did before the transformation. Now the square is mapped to the screen, in particular our window. However, our window isn’t square in shape, it is a rectangle. So, our square is squashed vertically to make it fit. This is bad.
How do we solve this? Well, we could change the shape of the window, but that isn’t getting to the root of the problem. The answer is to specify a projection transformation that maps our square to a rectangle stretched in the x-direction by the right amount so that when it is mapped to the screen it appears correctly. The right amount is clearly the aspect ratio of the screen. We will call
glOrtho(aspectRatio,- aspectRatio, -1.0d, 1.0d, 1.0d, -1.0d);
The source code for the fixed square is here.
Let’s draw a shape on our newly created Display. Most of the basic OpenGL commands in LWJGL are contained in the GL11 class. They are all static methods due to the procedural nature of the specification. To create points/lines/shapes we specify a number of vertices wrapped in a context.
Vertices
…or points. Whatever you want to call them. These are simply a location in 3-dimensional space specified by 3 co-ordinates (the familiar x, y and z). Actually OpenGL handles vertices using 4 co-ordinates per point, but more about that later…
A vertex can be recorded using floats or doubles as the coordinates:
glVertex3f(float x, float y, float z) glVertex3d(double x, double y, double z)
Contexts
There are many different contexts to choose from (GL_POINTS, GL_LINES, GL_LINE_LOOP to name a few). The context describes how to associate the vertices that are specified. For example, GL_LINES connects the points with lines, GL_TRIANGLE connects them in triangles etc. The context is specified like so:
glBegin(GL_LINE_LOOP); glVertex3f(-0.5f, -0.5f, 0f); glVertex3f(-0.5f, 0.5f, 0f); glVertex3f(0.5f, 0.5f, 0f); glVertex3f(0.5f, -0.5f, 0f); glEnd();
The specific rules about each context is contained in any good OpenGL reference. A vertex call is meaningless without a context — the vertex cannot be stored and used later so it must always appear along with the information about how it is to be used.
We now have the prerequisite tools to create a square on screen. The code for this example is hosted here. 
Simple! You probably have a lot of questions now:
1) Why has OpenGL rendered a rectangle instead of a square?
2) How did I know which coordinates to use so that the rectangle appears on the screen?
These questions will be answered in my next post.
Let’s get started! To create a window with a canvas to draw on (called a display) we use the following code:
import org.lwjgl.LWJGLException;
import org.lwjgl.opengl.Display;
public class HelloWorld {
public static void main (String args[]){
try {
Display.setTitle("Hello World");
Display.create();
} catch (LWJGLException e) {
e.printStackTrace();
}
}
}
Easy! This creates a window the size of your current desktop with an area where the shapes we will later draw with OpenGL will appear. Unfortunately this window cannot be closed using the traditional ‘x’ button, you must shut down the java process itself. The next thing to do will be to add the close mechanism.
Add the following to the bottom of the main method:
while(!Display.isCloseRequested()){
Thread.sleep(100);
}
Display.destroy();
Again, what this code does is obvious. We can see that the developers of LWJGL have tried to ensure that getting to the stage where we can start drawing shapes is as painless as possible.
OpenGL is an API specification for 2D and 3D graphics. It is implemented by hardware vendors on graphics cards or can be implemented in software. Hardware vendors pick and choose which operations are to be hardware accelerated.
All implementations can be accessed through a library of procedural C functions. More information about OpenGL can be found for free in the following book “The Red Book”. My favourite book on the subject is more mathematical than the previous one and is for sale here.
I like to develop in Java and hence use the JOGL library which translates the C functions into Java functions via the JNI. The functions are not wrapped, appearing as they would in C code. It would be nice to have a library written in an object oriented fashion (as is available for C++ here) but this is not available currently as far as I know. JOGL is available here.
LWJGL stands for the Light-Weight Java Gaming Library and is another library which gives you a lot of useful functions for writing games including windowing tools and precise timers. It is available here.
This is your my first post. My name is Phil and I am a software developer in the telecoms business. I will be using this blog to keep track of my current casual work. I hope you find it useful!
Delivering the requirements 