Java Out of Memory error

java.lang

Class OutOfMemoryError

 

  • All Implemented Interfaces:

 

Serializable

public class OutOfMemoryError

extends VirtualMachineError

Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError  objects may be constructed by the virtual machine as ifsuppression were disabled and/or the stack trace was not writable .

Since:

JDK1.0

See Also:

Serialized Form

 

2 solution of java.lang.OutOfMemoryError in Java

Every one in java development face java.lang.OutOfMemoryError now and then, OutOfMemoryError in Java is one problem which is more due to system’s limitation (memory) rather than due to programming mistakes in most cases though in certain cases you could have memory leak which causing OutOfMemoryError . I have found that even though java.lang.OutOfMemoryError is quite common basic knowledge of its cause and solution is largely unknown among junior developers. In this article we will explore what is java.lang.OutOfMemoryError; Why OutOfMemoryError comes in Java application, different type of OutOfMemoryError and How to fix OutOfMemoryError in Java. This article is purely meant to provide basic knowledge of java.lang.OutMemoryError and won’t discuss profiling in detail.

What is java.lang.OutOfMemoryError in Java

OutOfMemoryError in Java is a subclass of java.lang.VirtualMachineError  and JVM throws java.lang.OutOfMemoryError when it ran out of memory in heap. OutOfMemoryError in Java can come any time in heap mostly while you try to create an object and there is not enough space in heap to allocate that object. javavdoc of OutOfMemoryError is not very informative about this though.

Types of OutOfMemoryError in Java

I have seen mainly two types of OutOfMemoryError in Java:

1) Java.lang.OutOfMemoryError: Java heap space
2) Java.lang.OutOfMemoryError: PermGen space

Though both of them occur because JVM ran out of memory they are quite different to each other and there solutions are independent to each other.

Difference between “java.lang.OutOfMemoryError: Java heap space” and “java.lang.OutOfMemoryError: PermGen space”

If you are familiar with different generations on heap and How garbage collection works in java and aware of new, old and permanent generation of heap space then you would have easily figured out this OutOfMemoryError in Java. Permanent generation of heap is used to store String pool and various Meta data required by JVM related to Class, method and other java primitives. Since in most of JVM default size of Perm Space is around “64MB” you can easily ran out of memory if you have too many classes or huge number of Strings in your project. Important point to remember is that it doesn’t depends on –Xmx value so no matter how big your total heap size you can ran OutOfMemory in perm space. Good think is you can specify size of permanent generationusing JVM options “-XX:PermSize”  and  “-XX:MaxPermSize” based on your project need.

One small thing to remember is that “=” is used to separate parameter and value while specifying size of perm space in heap while “=” is not required while  setting maximum heap size in java, as shown in below example.

export JVM_ARGS=”-Xmx1024m -XX:MaxPermSize=256m”

Another reason of “java.lang.OutOfMemoryError: PermGen ” is memory leak through Classloaders and it’s very often surfaced in WebServer and application server like tomcat, webshere, glassfish or weblogic. In Application server different classloaders are used to load different web application so that you can deploy and undeploy one application without affecting other application on same server, but while undeploying if container some how keeps reference of any class loaded by application class loader than that class and all other related class will not be garbage collected and can quickly fill the PermGen space if you deploy and undeploy your application many times. “java.lang.OutOfMemoryError: PermGen” has been observed many times in tomcat in our last project but solution of this problem are really tricky because first you need to know which class is causing memory leak and then you need to fix that. Another reason of OutOfMemoryError in PermGen space is if any thread started by application doesn’t exit when you undeploy your application.

These are just some example of infamous classloader leaks, anybody who is writing code for loading and unloading classes have to be very careful to avoid this. You can also use visualgc  for monitoring PermGen space, this tool will show graph of PermGen space and you can see how and when Permanent space getting increased. I suggest using this tool before reaching to any conclusion.

Another rather unknown but interesting cause of “java.lang.OutOfMemoryError: PermGen” we found is introduction of JVM options “-Xnoclassgc“. This option sometime used to avoid loading and unloading of classes when there is no further live references of it just to avoid performance hit due to frequent loading and unloading, but using this option is J2EE environment can be very dangerous because many framework e.g. Struts, spring etc uses reflection to create classes and with frequent deployment and undeployment you can easily ran out of space in PermGen  if earlier references was not cleaned up. This instance also points out that some time bad JVM arguments or configuration can cause OutOfMemoryError in Java.

So conclusion is avoid using “-Xnoclassgc  in J2EE environment especially with AppServer.

Tomcat to Solve OutOfMemoryError in PermGen Space

From tomcat > 6.0 onward tomcat provides memory leak detection feature which can detect many common memory leaks on web-app perspective e.g ThreadLocal memory leaks, JDBC driver registration, RMI targes, LogFactory and Thread spawned by web-apps. You can check complete details on htp://wiki.apache.org/tomcat/MemoryLeakProtection you can also detect memory leak by accessing manager application which comes with tomcat, in case you are experiencing memory leak on any java web-app its good idea to run it on tomcat.

How to solve java.lang.OutOfMemoryError: Java heap space
1) Easy way to solve OutOfMemoryError in java is to  increase the maximum heap size by using JVM options “-Xmx512M”, this will immediately solve your OutOfMemoryError. This is my preferred solution when I get OutOfMemoryError in Eclipse, Maven or ANT while building project because based upon size of project you can easily ran out of Memory.here is an example of increasing maximum heap size of JVM, Also its better to keep -Xmx to -Xms ration either 1:1 or 1:1.5 if you are setting heap size in your java application

export JVM_ARGS=”-Xms1024m -Xmx1024m”

2) Second way to resolve OutOfMemoryError in Java is rather hard and  comes when you don’t have much memory and even after increase maximum heap size you are still getting java.lang.OutOfMemoryError, in this case you probably want to profile your application and look for any memory leak. You can use Eclipse Memory Analyzer to examine your heap dump or you can use any profiler like Netbeans or JProbe. This is tough solution and requires some time to analyze and find memory leaks.

How to solve java.lang.OutOfMemoryError: PermGen space

As explained in above paragraph this OutOfMemory error in java comes when Permanent generation of heap filled up. To fix this OutOfMemoryError in Java you need to increase heap size of Perm space by using JVM option   “-XX:MaxPermSize”.  You can also specify initial size of Perm space by using    “-XX:PermSize” and keeping both initial  and maximum Perm Space you can prevent some full garbage collection which may occur when Perm Space gets re-sized. Here is how you can specify initial and maximu Perm size in Java:

export JVM_ARGS=”-XX:PermSize=64M -XX:MaxPermSize=256m”

Some time java.lang.OutOfMemoryError   in Java gets tricky and on those cases profiling remain ultimate solution.Though you have freedom to increase heap size in java, it’s recommended that to follow memory management practices while coding and setting null to any unused references.
That’s all from me on OutOfMemoryError in Java I will try to write more about finding memory leak in java and using profiler in some other post. Please share what is your approach to solve java.lang.OutOfMemoryError in Java.

Important Note: From Tomcat > 6.0 onward tomcat provides memory leak detection feature which can detect many common memory leaks on Java application e.g ThreadLocal memory leaks, JDBC driver registration, RMI targes, LogFactory and Thread spawned by webapps. You can check complete details on htp://wiki.apache.org/tomcat/MemoryLeakProtection. You can also detect memoy leak by accessing manager application which comes with tomcat, in case you are experiencing memory leak on any java webapp its good idea to run it on tomcat to find out reason of OutOfMemoryError in PermGen space.

Tools to investigate and fix OutOfMemoryError in Java

Java.lang.OutOfMemoryError is a kind of error which needs lot of investigation to find out root cause of problem, which object is taking memory, how much memory it is taking or finding dreaded memory leak and you can’t do this without having knowledge of available tools in java space. Here I am listing out some free tools which can be used to analyze heap and will help you to find culprit of OutOfMemoryError

1) Visualgc

Visualgc stands for Visual Garbage Collection Monitoring Tool and you can attach it to your instrumented hostspot JVM. Main strength of visualgc is that it displays all key data graphically including class loader, garbage collection and JVM compiler performance data.

The target JVM is identified by its virtual machine identifier also called as vmid. You can read more about visualgc and vmid options here.

2) Jmap

Jmap is a command line utility comes with JDK6 and allows you to take a memory dump of heap in a file. It’s easy to use as shwon below:

jmap -dump:format=b,file=heapdump 6054

Here file specifies name of memory dump file which is “heapdump” and 6054 is PID of your Java progress. You can find the PDI by using “ps -ef” or windows task manager or by using tool called “jps”(Java Virtual Machine Process Status Tool).

3) Jhat

Jhat was earlier known as hat (heap analyzer tool) but it is now part of JDK6. You can use jhat to analyze heap dump file created by using “jmap“. Jhat is also a command line utility and you can rum it from cmd window as shown below:

jhat -J-Xmx256m heapdump

Here it will analyze memory-dump contained in file “heapdump”. When you start jhat it will read this heap dump file and then start listening on http port, just point your browser into port where jhat is listening by default 7000 and then you can start analyzing objects present in heap dump.

4) Eclipse memory analyzer

Eclipse memory analyzer (MAT) is a tool from eclipse foundation to analyze java heap dump. It helps to find classloader leaks and memory leaks and helps to minimize memory consumption.you can use MAT to analyze heap dump carrying millions of object and it also helps you to extract suspect of memory leak. See here for more information.

Here are some of my other post on Java you may find interesting:

How Hash Map works in Java?

Difference between ClassNotFoundException and NoClassDefFoundError

Difference between String, StringBuffer and StringBuilder in Java

Difference between Comparator and Comparable with Example

10 example of enum in java

What is polymorphism in Java? Method overloading or overriding

Top 20 core java interview question and answers asked in Investment Banks

How to solve java.lang.unsupportedClassVersionError in Java

3 ways to solve java.lang.NoClassDefFoundError in Java

How to solve java.lang.ClassNotFoundException in Java

Read more: http://javarevisited.blogspot.com/2011/09/javalangoutofmemoryerror-permgen-space.html#ixzz3jp1sQVin

java.lang.OutOfMemoryError – How to solve OutOfMemoryError

In this example we will discuss about OutOfMemoryError in Java. This error is thrown by the Java Virtual Machine (JVM) when an object cannot be allocated due to lack of memory space and also, the garbage collector cannot free some space. The OutOfMemoryError objects are created by the JVM when suppression is disabled and/ir the stack trace is not writable.

The OutOfMemoryError extends the VirtualMachineError class, which indicates that the JVM is broken, or it has run out of resources and cannot operate. Furthermore, the the VirtualMachineError extends the  Errorclass, which is used to indicate those serious problems that an application should not catch. A method may not declare such errors in its throwclause, because these errors are abnormal conditions that shall never occur.

Finally, the OutOfMemoryError exists since the 1.0 version of Java.

The Structure of OutOfMemoryError

Constructors

  • OutOfMemoryError()

Creates an instance of the OutOfMemoryError class, setting null  as its message.

  • OutOfMemoryError(String s)

Creates an instance of the OutOfMemoryError class, using the specified string as message. The string argument indicates the name of the class that threw the error.

The OutOfMemoryError in Java

As we have already explained, the OutOfMemoryError indicates that a new object cannot be created, due to lack of available memory space. The following example reproduces this case:

OutOfMemmoryExample.java:

public class OutOfMemoryErrorExample {

public static void main(String[] args) {

Long maxMemory = Runtime.getRuntime().maxMemory();

System.out.println(maxMemory);

 

int[] matrix = new int[(int) (maxMemory + 1)];

for(int i = 0; i < matrix.length; ++i)

matrix[i] = i+1;

}

}

In this example, we retrieve and print the available free memory inside the Java Virtual Machine (JVM), using the methodfreeMemory from the Runtime class. Then, we attempt to create a matrix of integers, whose size is bigger than the available free memory and thus, an OutOfMemoryError is thrown.

A sample execution is shown below:

1841823744

Exception in thread “main” java.lang.OutOfMemoryError: Java heap space

at main.java.OutOfMemoryErrorExample.main(OutOfMemoryErrorExample.java:8)

The java.lang.OutOfMemoryError: Java heap space error is thrown because we tried to allocate, more than the available, memory space. Each Java application can use only a limited amount of memory. This amount is specified at the startup of the JVM, by the following parameters:

  • -Xms <size> : specifies the minimum Java heap size.
  • -Xmx <size> : specifies the minimum Java heap size.

You can find the default values for many parameters of the JVM, by issuing the following command in a Linux or Mac terminal:

$ java -XX:+PrintFlagsFinal -version | grep -iE ‘HeapSize|PermSize|ThreadStackSize’

intx CompilerThreadStackSize                   = 0               {pd product}

uintx ErgoHeapSizeLimit                         = 0               {product}

uintx HeapSizePerGCThread                       = 87241520        {product}

uintx InitialHeapSize                          := 130023424       {product}

uintx LargePageHeapSizeThreshold                = 134217728       {product}

uintx MaxHeapSize                              := 2071986176      {product}

intx ThreadStackSize                           = 1024            {pd product}

intx VMThreadStackSize                         = 1024            {pd product}

java version “1.8.0_11”

Java(TM) SE Runtime Environment (build 1.8.0_11-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)

In Windows, you may use the following command:

java -XX:+PrintFlagsFinal -version | findstr /i “HeapSize PermSize ThreadStackSize”

For a complete list of all the parameters of the JVM, you shall execute the following command:

java -XX:+PrintFlagsFinal -version

More about the OutOfMemoryError in Java

The OutOfMemoryError is thrown due to lack of available memory space. The garbage collector is responsible for detecting objects that are not being used or referenced and collect them. In this way, memory space is being reclaimed for Java applications. However, there are cases where the garbage collector cannot reclaim portions of memory, despite the fact that these objects are no longer accessible by the Java application. These are called memory leaks  and can severely limitate the available memory space.

Memory leaks can be created very easily in Java:

MemoryLeakExample.java:

import java.util.HashMap;

import java.util.Map;

import java.util.Random;

public class MemoryLeakExample {

public static void main(String[] args) {

Random random = new Random();

Map sampleMap = new HashMap();

 

// Loop forever…

while(true) {

// Create and store a random pair.

int randomValue = random.nextInt();

sampleMap.put(randomValue, String.valueOf(randomValue));

}

}

}

In this example, we create and store random pairs in a  HashMap. However, our application only stores values to theHashMap and never retrieves values from it. Thus, memory space is occupied for no reason and the garbage collector cannot reclaim that space, because our application holds a reference to that HashMap. Thus, memory leaks are essential for memory space and must be removed.

Finally, as you can see from here, there are many different causes that can throw an OutOfMemoryError in Java:

  • java.lang.OutOfMemoryError:  Java heap space.
  • java.lang.OutOfMemoryError:  PermGen space.

This error indicates that the permanent generation is full. In Java, the permanent generation is the memory space where class and method objects are stored.

  • java.lang.OutOfMemoryError:  Requested array size exceeds VM limit.

This error indicates that a Java application attempts ti allocate an array, whose size is larger than the heap size. For example:

public class OutOfMemoryErrorVMLimitExample {

public static void main(String[] args) {

int[] matrix = new int[Integer.MAX_VALUE];

 

for(int i = 0; i < matrix.length; ++i)

matrix[i] = i+1;

}

}

 

A sample execution is shown below:

Exception in thread “main” java.lang.OutOfMemoryError: Requested array size exceeds VM limit

at main.java.OutOfMemoryErrorVMLimitExample.main(OutOfMemoryErrorVMLimitExample.java:5)

 

  • java.lang.OutOfMemoryError:  request <size> bytes for <reason>. Out of swap space?

This error indicates that an allocation from the native heap space has failed and also, the heap space is close to exhaustion. Moreover, the message of the error indicates the number of bytes that failed and the reason for the memory request.

  • java.lang.OutOfMemoryError:  <reason> <stack trace> (Native method).

This error indicates that the allocation failure was detected in a JNI or native method instead of the JVM code.

How to deal with the OutOfMemoryError

  • The most obvious solution to this error is to increase the available memory size for the Java Virtual Machine. If your application requires more memory then, you shall grant it to your application.
  • Verify that your application does not store unnecessary information. Store and maintain only those pieces of information required for the proper execution of your Java application.
  • You can use the availabe memory analyzer tools, in order to carefully observe the portions of memory occupied by your application. Examples of such tools are the Eclipse Memory Analyzer and  Java Heap Analysis Tool (jhat).

Download the Eclipse Project

This was a tutorial about OutOfMemoryError in Java.

Download
You can download the full source code of this example here: OutOfMemoryErrorExamples.zip.

 

java.lang.OutOfMemoryError:Java heap space

Java applications are only allowed to use a limited amount of memory. This limit is specified during application startup. To make things more complex, Java memory is separated into two different regions. These regions are called Heap space and Permgen (for Permanent Generation):

The size of those regions is set during the Java Virtual Machine (JVM) launch and can be customized by specifying JVM parameters -Xmx  and -XX:MaxPermSize. If you do not explicitly set the sizes, platform-specific defaults will be used.

The java.lang.OutOfMemoryError: Java heap space error will be triggered when the application attempts to add more data into the heap space area, but there is not enough room for it.

Note that there might be plenty of physical memory available, but thejava.lang.OutOfMemoryError: Java heap space  error is thrown whenever the JVM reaches the heap size limit.

The cause

There most common reason for the java.lang.OutOfMemoryError: Java heap space error is simple – you try to fit an XXL application into an S-sized Java heap space. That is – the application just requires more Java heap space than available to it to operate normally. Other causes for this OutOfMemoryError message are more complex and are caused by a programming error:

  • Spikes in usage/data volume. The application was designed to handle a certain amount of users or a certain amount of data. When the number of users or the volume of data suddenly spikes and crosses that expected threshold, the operation which functioned normally before the spike ceases to operate and triggers the java.lang.OutOfMemoryError: Java heap space error.
  • Memory leaks. A particular type of programming error will lead your application to constantly consume more memory. Every time the leaking functionality of the application is used it leaves some objects behind into the Java heap space. Over time the leaked objects consume all of the available Java heap space and trigger the already familiar java.lang.OutOfMemoryError: Java heap space error.

Solution to the problem Automatic root cause detection of OutOfMemoryErrors TRY PLUMBR

Examples

Trivial example

The first example is truly simple – the following Java code tries to allocate an array of 2M integers. When you compile it and launch with 12MB of Java heap space (java -Xmx12m OOM), it fails with the java.lang.OutOfMemoryError: Java heap space  message. With 13MB Java heap space the program runs just fine.

class OOM {

static final int SIZE=2*1024*1024;

public static void main(String[] a) {

int[] i = new int[SIZE];

}

}

Memory leak example

The second and a more realistic example is of a memory leak. In Java, when developers create and use new objects e.g. new Integer(5) , they don’t have to allocate memory themselves – this is being taken care of by the Java Virtual Machine (JVM). During the life of the application the JVM periodically checks which objects in memory are still being used and which are not. Unused objects can be discarded and the memory reclaimed and reused again. This process is called Garbage Collection. The corresponding module in JVM taking care of the collection is called the Garbage Collector (GC).

Java’s automatic memory management relies on GC to periodically look for unused objects and remove them. Simplifying a bit we can say that a memory leak in Java is a situation where some objects are no longer used by the application but Garbage Collection fails to recognize it. As a result these unused objects remain in Java heap space indefinitely. This pileup will eventually trigger the java.lang.OutOfMemoryError: Java heap space  error.

It is fairly easy to construct a Java program that satisfies the definition of a memory leak:

class KeylessEntry {

 

static class Key {

Integer id;

 

Key(Integer id) {

this.id = id;

}

 

@Override

public int hashCode() {

return id.hashCode();

}

}

 

public static void main(String[] args) {

Map m = new HashMap();

while (true)

for (int i = 0; i < 10000; i++)

if (!m.containsKey(new Key(i)))

m.put(new Key(i), “Number:” + i);

}

}

When you execute the above code above you might expect it to run forever without any problems, assuming that the naive caching solution only expands the underlying Map to 10,000 elements, as beyond that all the keys will already be present in the HashMap. However, in reality the elements will keep being added as the Key class does not contain a proper equals() implementation next to its hashCode().

As a result, over time, with the leaking code constantly used, the “cached” results end up consuming a lot of Java heap space. And when the leaked memory fills all of the available memory in the heap region and Garbage Collection is not able to clean it, thejava.lang.OutOfMemoryError:Java heap space  is thrown.

The solution would be easy – add the implementation for the  equals() method similar to the one below and you will be good to go. But before you manage to find the cause, you will definitely have lose some precious brain cells.

@Override

public boolean equals(Object o) {

boolean response = false;

if (o instanceof Key) {

response = (((Key)o).id).equals(this.id);

}

return response;

}

Solution to the problem No OutOfMemoryError goes unnoticedGET AUTOMATIC ROOT CAUSE DETECTION

The solution

The first solution should be obvious – when your JVM runs out of a particular resource, you should increase the availability of that resource. In our case: when your application does not have enough Java heap space memory to run properly, fixing it is as easy as altering your JVM launch configuration and adding (or increasing if present) the following:

-Xmx1024m

The above configuration would give the application 1024MB of Java heap space. You can use g or G for GB, m or M for MB, k or K for KB. For example all of the following are equivalent to saying that the maximum Java heap space is 1GB:

java -Xmx1073741824 com.mycompany.MyClass

java -Xmx1048576k com.mycompany.MyClass

java -Xmx1024m com.mycompany.MyClass

java -Xmx1g com.mycompany.MyClass

In many cases however, providing more Java heap space is not exactly going to solve the problem. For example, if your application contains a memory leak, adding more heap will just postpone the java.lang.OutOfMemoryError: Java heap space  error. Additionally, increasing the amount of Java heap space also tends to increase the length of GC pauses affecting your application’s throughput or latency.

If you wish to solve the underlying problem with the Java heap space instead of masking the symptoms, you have several tools at your disposal. Debuggers, profilers, heap dump analyzers – the choice is yours. For an automated solution we suggest Plumbr, the Java monitoring solution with root cause detection. Among other performance problems it catches all java.lang.OutOfMemoryErrors and automatically tells you what causes them. A free 14-day trial account includes full root cause detection functionality.

 

Advertisements

Code Review tips

v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}

Normal
0
false

false
false
false

EN-US
X-NONE
X-NONE

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:””;
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:8.0pt;
mso-para-margin-left:0cm;
line-height:107%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-ansi-language:EN-US;
mso-fareast-language:EN-US;}

Code Review, Discussions and Sharing

Top of Form

Bottom of Form

Upsource is great place for your team to communicate with the intent of improving your code base:

Discussions

You can start discussions in code or diff views, with or without creating a formal code review.

When viewing any revision of a file, you have access to all relevant discussions from prior revisions of this file, be it within code reviews or in standalone discussions that weren’t parts of formal code reviews.

Essentially, knowledge about specific files in your code base is accumulated and made available for future reference.

Code reviews

Anyone can request or start post-commit code review on any revision or branch.

If you prefer to review individual revisions, feel free to create code reviews from the revision list.

If you use the increasingly popular workflow of reviewing entire branches, then as soon as you click Create branch review on a branch, Upsource puts all revisions in the branch under a single code review, and makes sure to automatically add any new revisions as they appear. Branch tracking stops when a code review is closed and resumes if it’s reopened later.

The code review process in Upsource is relaxed, and it doesn’t impose any strict workflow:

  • If a developer who made a change wants their change to be reviewed, that’s fine.
  • If someone else on the team wants to raise a concern over a teammate’s change, they can perfectly do that by starting a code review, too.

When starting a code review, you can add one or multiple reviewers, picking the teammate(s) most proficient in the part of code that you’re modifying. Upsource can suggest reviewers to you based on history of files that are being changed, as well code review history.

Reviewers are expected to examine your changes, discuss them if they aren’t clear, and finally accept the changes.

You can also add teammates as watchers in code reviews so that they become aware of important changes in your code base. Watchers aren’t expected to take part in the code review process, but they are kept updated of your project’s status whenever an important change is made.

Developers taking part in a code review can discuss specific lines in the diff view, or add general comments to the review as a whole.

You can add new revisions to a code review if a revision originally submitted for review required further changes stemming from the discussion. Upsource will add new revisions automatically if the subject of your review is the entire branch.

Upsource tracks which revisions were already seen and approved. When new revisions are added to the code review, a reviewer only sees the diff of those revisions, and they don’t have to inspect previous changes once again. To show or hide any revisions within a code review, there’s a revision selector that quickly toggles revisions on and off:

You can access code reviews that you are involved in as an author, reviewer or watcher from your project’s home view. Alternatively, browse all code reviews in the project, or search code reviews by author, reviewer, state, or commit ID:

You can monitor status updates in a project via personalized e-mail notifications. Alternatively, you can track the News feed in the project’s home view, which displays updates such as new comments addressing you, and recently opened and closed code reviews.

If you want to get a high-level overview of code review activities in your project, there’s a set of code review reports that show the share of revisions that are covered with code reviews, how many open and closed code reviews you have in your project, and which project developers are most involved in the code review process.

Code review statistics in Upsource project

Code review in the IDE

We know that most developers are best seated in their IDE and prefer not to switch between tools unless absolutely necessary. Knowing this, we offer a code review plug-in for IntelliJ IDEA and other JetBrains IDEs which allows you to participate in code discussions and manage code reviews from the comfort of your IDE.

The plug-in allows viewing and creating code review comments right from the text editor, and provides a Review tool window that lists code reviews in the current Upsource project and lets you manage them.

Code review plug-in for IntelliJ IDEA: Reviews tool window

In addition, the plug-in integrates into the IDE’s own controls such as Version Control tool window and Commit Changes dialog box.

Upsource integration in IntelliJ IDEA's Git log view

The plug-in works across all IDEs based on the IntelliJ platform, namely:

Code sharing

In Upsource, everything has a URL and can be shared with teammates. This includes code reviews, revision diffs, discussions on code and on revisions, reports or filters applied to commit graphs (for example, all commits by developer X in time span Y).

You can even share custom selections of code in any specific revision:

Share a link to a random selection of code

When you share a URL that Upsource generates for a selection, anyone you share it with can open it and have the selection highlighted like this:

Link to selection as the receiving person sees it

 


 

In addition to being a code review tool, Upsource is also a repository browser. Learn how you can use Upsource to browse and monitor changes in code repositories based on Git, Mercurial, Subversion and/or Perforce.

If you’re a Java developer, you might also be interested to know about additional code insight features in Java projects including code inspections and advanced navigation.


 

Upsource Docs & Demos

Check out Upsource online help to:

If you have a question on using the product that you can’t find an answer to, feel free to contact the Upsource team, and we’ll try our best to help you out.

 

Videos

Watch videos demonstrating how Upsource can be used as a repository browser and code review tool.

Hub, YouTrack and Upsource Integration

Hub, YouTrack and Upsource Integration

12 August 2015

Code review workflow in Upsource

Code review workflow in Upsource

9 June 2015

Upsource plugin for IntelliJ IDEA

Upsource plugin for IntelliJ IDEA

1 June 2015

Upsource plugin for JetBrains IDEs

Upsource plugin for JetBrains IDEs

1 June 2015

JetBrains Upsource Overview

JetBrains Upsource Overview

20 May 2015

What's New in Upsource 2.0

What’s New in Upsource 2.0

20 May 2015

 

View all videos about Upsource and other JetBrains team collaboration tools.


 

Things to check before starting up

Before you proceed to install Upsource, please make sure to…

Check your team’s demands and expectations

  • Upsource is the right tool for you if your development team is looking for ways to browse different revisions of the code base without checking them out to local machines, and to discuss and review changes made in the code base.
  • Upsource only recognises changes that are committed to your repository.
  • Upsource doesn’t support Android Studio projects

Check your hardware and software

  1. Since Upsource is an on-premises application, you should have a server to deploy Upsource to, and the server should:
    • Have 8 GB of RAM or more
    • Run one of the following 64-bit operating systems:
      • Windows Vista or later
      • Mac OS X 10.7 or later
      • Linux (based on our knowledge, any 64-bit distribution should do)
  2. As Upsource is a set of Java applications, it requires Java runtime to be installed. Upsource bundles JDK 1.8 for Windows and Mac OS X and you don’t have to explicitly install JDK if you’re going to run Upsource on one of these operating systems. However if you’re going to install Upsource on Linux, you should first install JRE 1.8 or JDK 1.8 for Linux.
  3. Your fully qualified hostname should be resolvable to your IP address. To check it:
    • On Linux or Mac OS, run the following command:

ping $(hostname -f)

    • On Windows, to get fully qualified hostname, run:

ipconfig /all

then try to ping that name (it should look like mymachine.mydomain.com)

  1. Your development team should use Git, Mercurial, Subversion, or Perforce for version control.
  2. Upsource users should use a modern web browser. Upsource supports Internet Explorer 10+ and recent versions of Firefox, Chrome and Opera.

Things to configure before starting Upsource

When it comes to mainstream installation cases Upsource requires minimal configuring. All you have to do is:

Disable conflicting software (Windows)

If you run Upsource on Windows, please disable all anti-viruses, Windows Defender, and Windows Search service for Upsource root directory, as they may conflict with Upsource processes.

Adjust resource limits (Linux)

If you run Upsource on Linux, insufficient resource limits may result in a number of errors. To prevent that, we recommend to set:

  • maximum open files to 100000
  • memory locking and address space limit to unlimited
  • number of processes to 32768

You can do it by adding the following lines to the /etc/security/limits.conf file:

* – memlock unlimited

* – nofile 100000

* – nproc 32768

* – as unlimited

Other installation options

You can also:

You are now ready to launch Upsource

As soon as you have made all configuration changes that are relevant in your environment, you can proceed to start Upsource.

Proxy configuration

Note: <upsource_home>\directory_name should be read as “open the console and change directory to directory_name under Upsource home directory.”

All commands listed below are Windows commands. If you’re working on a Linux or Mac OS X server, simply replace .bat with .sh.

You can set up Upsource to work behind a reverse proxy server. There are two requirements that your environment should meet to make this possible:

  • Your proxy server must support WebSockets. For example, Nginx supports WebSockets since version 1.3.13.
  • Upsource should be hosted under root URL (/) on your virtual host.

If these requirements are met, start with configuring Upsource to use a base URL (the URL that end users will request for to access your Upsource installation):

<upsource_home>\bin\upsource.bat configure –listen-port 1111 –base-url http://upsource.mydomain.com:2222

where:

  • 1111 is the port number Upsource will listen to
  • http://upsource.mydomain.com is the address of your proxy server
  • and 2222 is the port number your proxy will listen to

Now configure headers in your proxy server, and you’re done. Configuration guidelines for Nginx and Apache HTTP Server are provided below.

Nginx configuration

To ensure support for WebSockets, please use Nginx 1.3.13 or later.

Here’s a sample Nginx header configuration (non SSL):

server {

      listen       2222;

      server_name  localhost;

location  / {

      proxy_set_header X-Forwarded-Host $http_host;

      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_http_version 1.1;

       

      # to proxy WebSockets in nginx

      proxy_set_header Upgrade $http_upgrade;

      proxy_set_header Connection “upgrade”;

      proxy_pass http://upsourcemachine.domain.local:1111/;

      }

}

where:

  • listen 2222 is the port that you have previously specified as a –base-url parameter
  • proxy_pass http://upsourcemachine.domain.local:1111/ is the path to your Upsource machine with the port that you have previously specified using the -–listen-port command

Nginx SSL configuration goes as follows:

  • Configure base url:

<upsource_home>\bin\upsource.bat configure –listen-port 1111 –base-url https://upsource.mydomain.com/

  • Nginx configuration file:

        server {

            listen 443 ssl;

   

             ssl_certificate <path_to_certificate>

             ssl_certificate_key <path_to_key>

   

            server_name  localhost;

   

        location  / {

                proxy_set_header X-Forwarded-Host $http_host;

                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                proxy_set_header X-Forwarded-Proto $scheme;

                proxy_http_version 1.1;

   

                # to proxy WebSockets in nginx

                proxy_set_header Upgrade $http_upgrade;

                proxy_set_header Connection “upgrade”;

                proxy_pass http://upsourcemachine.domain.local:1111/;

                        }

               }

Note: Please refer to the corresponding Nginx documentation pages for a description of server_name, proxy_set_header, proxy_pass.

Apache HTTP server configuration

To ensure support for WebSockets, please use Apache HTTP Server 2.4.10 or later.

Make sure to enable proxy_wstunnel, proxy_http, rewrite modules (and optionally headers if you want to use SSL) using the a2enmod script:

    $ a2enmod headers

    $ a2enmod rewrite

    $ a2enmod proxy_wstunnel

    $ a2enmod proxy_http

Add the following directives to the VirtualHost section of a relevant .conf file:

    RewriteEngine on

    AllowEncodedSlashes on

      

    RewriteCond %{QUERY_STRING} transport=polling

    RewriteRule /(.*)$ http://127.0.0.1:1111/$1 [P]

   

    ProxyRequests off

    ProxyPass /~socket.io/ ws://127.0.0.1:1111/~socket.io/

    ProxyPassReverse /~socket.io/ ws://127.0.0.1:1111/~socket.io/

       

    ProxyPass / http://127.0.0.1:1111/

    ProxyPassReverse / http://127.0.0.1:1111/

where 1111 is the port number you configured Upsource to listen to.

If you want to use SSL, additionally add the following directives to the VirtualHost section:

  RequestHeader set X-Forwarded-Proto “https”

IIS reverse proxy

To use IIS and ARR as a reverse proxy:

  1. Install ARR from here
  2. In IIS Manager, connect to the IIS server – in this case, localhost
  3. Highlight the server in the Connections pane
  4. Double-click URL Rewrite
  5. Click View server variables on the right pane
  6. Add HTTP_X_FORWARDED_HOST, HTTP_X_FORWARDED_SCHEME and HTTP_X_FORWARDED_PROTO to the list
  7. Highlight the server in the Connections pane
  8. Double-click Application Request Routing Cache
  9. Click Server Proxy Settings under the Proxy heading in the Actions pane.
  10. Tick the Enable proxy checkbox and then click Apply. Leave the default values in place.
  11. In the Connections pane, under Sites, highlight Default Web Site
  12. Double-click the URL Rewrite feature, and click Add Rule(s)… in the Actions pane
  13. Add a reverse proxy rule, with server name: localhost:1111 (replace with real location and port of your Upsource service)
  14. Open created rule, check rewrite url, add server variables:
    • set HTTP_X_FORWARDED_HOST to {HTTP_HOST}
    • set HTTP_X_FORWARDED_SCHEME to https (if the IIS site is configured to https, else set to http)
    • set HTTP_X_FORWARDED_PROTO to https (if the IIS site is configured to https, else set to http)
  15. Make sure that Anonymous Authentication is enabled:

a) In the Connections pane, under Sites, highlight Default Web Site

b) Double click Authentication → Select Anonymous → Click Enable in the right pane.

In-cloud installation

Upsource can be easily installed and run in a cloud, using one of the cloud computing services available on the market.

As an example we’ll outline the Upsource installation requirements for Amazon EC2, since it’s one of the most popular cloud solutions at the moment. For easy service setup and trouble-free Upsource installation we encourage you to read EC2 documentation as well.

  1. You can choose any platform: Linux, Mac, or Windows.
  2. The EC2 instance type you select should be M3.large or higher.
  3. When configuring EC2 security group, add a custom rule with the following parameters:
    • Protocol to allow: TCP
    • Port to allow: XXXX (Specify the default Upsource port (80 for Windows, 8080 for Linux and Mac), unless it’s taken or you want to use a different port for other reasons. In this case you’ll have to change it for Upsource as well (Step 4).
  4. Install Upsource, then prior to starting it, configure the base URL (the URL for end users to access your Upsource installation) and port by running the following command:

<upsource_home>\bin\upsource.bat configure –listen-port XXXX –base-url http://upsource.mydomain.com/

where:

    • –listen-port should match the port that you’ve specified for EC2 rule in Step 3. Skip, if you’ve specified the default Upsource port in Step 3.
    • –base-url should match the public DNS name of your EC2 instance.

Note: <upsource_home>\directory_name should be read as “open the console and change directory to directory_name under Upsource home directory.” Above is a Windows command. For Linux or Mac OS X, simply replace .bat with .sh.

  1. You are ready to run Upsource.

Installing Upsource on Linux

To help you install Upsource quickly and easily, let’s take a common Ubuntu installation as an example and go through the entire process.

First of all we need to install Java

1.     Unless you already have one, download JRE 1.8 installation package from the Oracle web-site.

2.     Launch the terminal and create a folder where java binaries will be stored:

sudo mkdir –p /usr/local/java

3.     Go to the folder with the downloaded jre archive (the default location is shown here):

cd /home/user_name/downloads

4.     Copy the archive to your java folder:

sudo cp –r jre-7u25-linux-x64.tar.gz /usr/local/java

5.     Go to the java folder:

cd /usr/local/java

To execute our archive, we need to set permissions.

1.     Change the permissions:

sudo chmod a+x jre.tar.gz

2.     Unpack the archive:

sudo tar xvzf jre.tar.gz

Now we are going to set system variables.

1.     Edit /etc/profile:

sudo gedit /etc/profile

and add the following lines to the bottom of the file:

JRE_HOME=/usr/local/java/jre1.8.0_25 
PATH=$PATH:$JRE_HOME/bin 
export JRE_HOME 
export PATH

Save the file and close it.

2.     Tell your system where JRE is located:

sudo update-alternatives –install “/usr/bin/java” “java” “/usr/local/java/jre1.8.0_25” 1

3.     Make the JRE the default one:

sudo update-alternatives –set java /usr/local/java/jre1.8.0_25/bin/java

4.     Check if java was installed correctly:

sh java -version

Next step is to adjust your system resource limits

Add the following lines to the /etc/security/limits.conf file:

* - memlock unlimited
* - nofile 100000
* - nproc 32768
* - as unlimited

Find a home for your Upsource dist

1.     Download a new Upsource build from our website.

2.     Create a folder for it. We’ll refer to this folder as Upsource home directory:

Sudo mkdir –p /opt/Upsource

3.     Go to the folder with the downloaded Upsource archive (the default location is shown here):

cd /home/user_name/downloads

4.     Copy the archive to your Upsource folder:

cp Upsource.zip /opt

5.     Set permissions:

sudo chmod a+x Upsource.zip

6.     Unpack the archive:

sudo unzip Upsource.zip

7.     Make the Upsource folder writable:

sudo chmod -R a+rwX /opt/Upsource

Now we can start and configure Upsource.

1.     Launch the terminal and go to the bin folder in the Upsource home directory:

cd /opt/Upsource/bin

2.     Run Upsource:

upsource.sh start

When you run Upsource for the first time, it will open Configuration Wizard in your default browser, where you can specify initial settings.

That’s it. As soon as you’re finished, you’ll be taken to Upsource welcome page from where you can proceed to creating your first project.

Starting and stopping Upsource

Note: All commands listed below are Windows commands. If you’re working on a Linux or Mac OS X server, simply replace .bat with .sh. For Mac OS X or Linux, please make sure to start Upsource as a non-root user.

Running Upsource as a background process

To start Upsource, run the following command:

<upsource_home>\bin\upsource.bat start

Running Upsource as a Windows Service

To start Upsource, run the following command:

<upsource_home>\bin\upsource.bat service install /runAsSystem

To start Upsource with another user account, run the following command:

<upsource_home>\bin\upsource.bat service install /user=<user> [/domain=<domain>] /password=<password>

Stopping

To stop Upsource, run the following command:

<upsource_home>\bin\upsource.bat stop

Restarting

To restart Upsource, run the following command:

<upsource_home>\bin\upsource.bat restart

Displaying other commands

To explore other commands that Upsource provides, such as those for running in the current console, configuring, and displaying status, run

<upsource_home>\bin\upsource.bat ?

Things to configure after the first start

When you run Upsource for the first time, it will open Configuration Wizard in your default browser. First you can specify System settings

https://www.jetbrains.com/upsource/help/2.0/assets/setup/upsource_setup1.png

  • Base URL – you might want to change the default value if:
    • your server has several DNS interfaces and you wish to make Upsource accessible by a particular DNS interface;
    • you have set up Upsource to work behind reverse or terminating HTTP proxy (see Proxy configuration) – in this case the new value should match the base URL you’ve set before.
  • Application Listen Port – you can change the default port that Upsource will use for HTTP communications if the suggested port is or expected to be taken.
  • Location of system directories (Under Advanced Settings)

Click Next to proceed to User management settings

https://www.jetbrains.com/upsource/help/2.0/assets/setup/upsource_setup2.png

Here you have to choose to set up Upsource with either build-in Hub or external (standalone) Hub. If you are going to integrate with the YouTrack issue tracker, select External Hub. Otherwise you may choose Build-in Hub.

Please note: Prior to selecting External Hub, you should first install and start your external Hub service.

If Build-in Hub selected:

  • Create admin login and password – specify login name and password for your administrator account. (You’ll be able to reset your password if you ever forget it.)
  • Send usage statistics anonymously – keep this option checked to help us make Upsource better. We never share collected data with any third party.

If External Hub selected:

  • Enter your Hub URL and verify the connection.

Click Next to proceed to Licenses where you can keep the default free license or change it to a different one:

https://www.jetbrains.com/upsource/help/2.0/assets/setup/upsource_setup3.png

Click Finish when you’re done.

Upsource will open its welcome page (http://your-host-name:port_for_upsource/bundle/starting/) which will also be opened upon any consequent startup.


 

What to look for in a Code Review: Upsource Quick Wins

Posted on by Trisha Gee

We’ve had two themes running through the articles on what to look for in a code review:

  1. Lots of what we’re tempted to look for can (and should) be automated
  2. Humans are good at checking things computers can not.

In this final post in the series, we’re going to look at the grey area in between – we’re going to look at how the features in Upsource can make your job as a human code reviewer easier.

As a developer, code reviews can sometimes be a frustrating process – you’re used to all the power of your IDE, yet when you open up a set of changes in your code review tool you can’t leverage any of that. The cognitive load of having to navigate through the code in an unfamiliar way and losing all the context your IDE provides is one of the things that makes developers less keen to perform code reviews.

Let’s look at some of the features that Upsource provides that overcome these issues.

Navigation

It might seem like a trivial thing, but the ability to navigate through the code via the code is something we simply take for granted when we use an IDE like IntelliJ IDEA. Yet the simple Cmd + click feature to navigate to a given class, or to find the declaration of a field, is often missing when we open code in a tool that is not our IDE. Upsource lets you click on a class name, method, field or variable to navigate to the declaration.

Symbol actions

While this is very useful, something I find myself doing a even more in my IDE is pressing Alt + F7 to find usages of a class, method or field. This is especially useful during code review, because if a method or class has been changed in some way, you as the reviewer want to see what the impact of this change is – which means locating all the places it’s used. You can see from the screenshot above that this is easily done in Upsource – clicking on a symbol gives you the option to highlight the usages in the file, or to find usages in the project.

Inspections

Intuitive navigation is great for a reviewer as it lets you browse through the code in a way that’s natural for you, rather than having some arbitrary order imposed on you – it makes it easier to see the context of the changes under review.

But there’s another IDE feature that would be extremely useful during code review – inspections. If you’re already using IntelliJ IDEA, for example, you’re probably used to the IDE giving you pointers on where the code could be simpler, clearer, and generally a bit better. If your code review tool offered the same kind of advice, you could easily check that all new/updated code doesn’t introduce new obvious issues, and possibly even cleans up long-standing problems.

Upsource uses the IntelliJ IDEA inspections – we actually covered how to enable them for Upsource in the last post. There are rather a lot of inspections available in IntelliJ IDEA, so we’re just going to give a taste of what’s possible – we’re going to cover some of the default ones that you may find useful during code review.

Exception Handling Issues

Inspections can catch potential problems around how error conditions are handled. For example, empty catch blocks.

Empty catch block

Solutions
It’s difficult to think of a time when catching and ignoring an Exception is the right thing to do. A code reviewer should be suggesting:

  1. Catching the Exception and wrapping it in a more appropriate one, possibly a RuntimeException, that can be handled at the right level.
  2. Logging the Exception (we also touched on appropriate logging in the last post).
  3. At the very least, documenting why this is OK. If there’s a comment in the catch block, it’s no longer flagged by the inspection.

ExpectedException

Configuring
“Empty ‘catch’ block” is enabled in the default set of inspections.  This and other related inspections can be found in IntelliJ IDEA’s inspections settings under Java > Error Handling.

Probable Bugs

There are a number of inspections available for “probable bugs”. These inspections highlight things that the compiler allows, as they’re valid syntax, but are probably not what the author intended.

String format potential bug

Examples

  • String.format() issues like the one above.
  • Comparing Strings using == not .equals().
  • Querying Collections before putting anything in them (or vice versa).
  • Accessing Collections as if they have elements of a different type (sadly possible due to the way Java implemented generics on collections).

Collections probable bugs

Solution
Not all of these problems are automatically bugs, but they do look suspicious. They’ll usually require you, the code reviewer, to point them out to the author and have a discussion about whether this code is intentional.

Configuring
Inspections to highlight all the potential problems listed are already selected by default. To find more inspections in this category, look under Java > Probable Bugs in the inspections settings.

Code can be simplified

It’s easy as you evolve code to end up with statements and methods that are more complicated than they need to be – it just takes one more bit of boolean logic or an additional if statement. As code reviewers, we’re in a fortunate position of being one step back from the coal-face of the code, so we can call out areas ripe for simplification. Fortunately, we don’t have to do this alone – Upsource shows us some of these things automatically.

Boolean expression can be simplified

Examples

  • Using explicit true and false in a boolean expression (in the example above this is unnecessarily verbose).
  • Boolean expressions that can be simplified, or re-phrased to be simpler to understand.
  • if or while expressions that always evaluate to the same value:

Condition can be simplified

Solutions

  • As with the other examples above, you may simply want to flag them in the code review so the author can use IntelliJ IDEA’s inspections to apply the recommended fix.
  • In some cases, like if statements that can be simplified in equals() methods, the simplified code is not always easier to read. If this is the case, you may want to suggest the code author suppresses the inspection for this code so it is no longer flagged.
  • In other cases, the inspection might be pointing to a different smell.  In the if statement above, the inspection shows this code (which is in a private class) is always called with a particular set of values so this if statement is redundant. It may be viable to remove the statement, but as this specific example is only used in test code it implies there’s a missing test to show what happens when the two objects are equal. The code reviewer should suggest the additional test, or at least have the author document why it wasn’t needed.

Configuring
These types of inspections can be found in Java > Control flow issues and Java > Data flow issues.

Unused Code

Upsource highlights all unused code (classes, methods, fields, parameters, variables) in a grey colour, so you don’t even need to click or hover over the areas to figure out what’s wrong with it – grey should automatically be a smell to a code reviewer.

Unused code

Examples
There are a number of reasons a code review might contain unused code:

  1. It’s an existing class/method/field/variable that has been unused for some time.
  2. It’s an existing class/method/field/variable that is now unused due to the changes introduced in the code review.
  3. It’s new / changed code that is not currently being called from anywhere.

Solutions
As a reviewer, you can check which category the code falls into and suggest steps to take:

  • Delete the unused code. In the case of 1) or 2) above, this should usually be safe at the field/variable level, or private classes and methods. At the class and method level, if these are public they might be used by code outside your project. If you have control over all the code that would call these and you know the code is genuinely unused, you can safely remove them. In case 3) above, it’s possible that some code is work-in-progress, or that the author changed direction during development and needs to clean up left over code – either way, flag the code and check if it can be deleted.
  • Unused code could be a sign the author forgot to wire up some dependencies or call the new features from the appropriate place. If this is the case, the code author will need to fix the unused code by, well, using it.
  • If the code is not safe to delete and is not ready to be used, then “unused code” is at the very least telling you that your test coverage is not sufficient. Methods and classes that are used by other systems, or will be used in the very near future, should have tests that show their expected behaviour. Granted, test coverage can hide genuinely unused code, but it’s better to have code that looks used because it’s tested than have code that is used that is not tested. As the reviewer, you need to flag the lack of tests. For code that existed before this code review, you might want to raise a task/story to create tests for the code rather than to slow down the current feature/bug being worked on with unrelated work. If the unused code is new code, then you can suggest suitable tests. New code that’s untested should not be let off lightly.
  • If you and the code author decide not to address the unused code immediately by deleting it, using it or writing tests for it, then at least document somehow why this code is unused. If there’s a ticket/issue somewhere to address it later, refer to that.

Suppress unused warnings

Gotchas
Inspections are not infallible, hence why they’re useful pointers for reviewers but not a fully automated check with a yes/no answer. Code might be incorrectly flagged as unused if:

  • It’s used via reflection
  • It’s used magically by a framework or code generation
  • You’re writing library code or APIs that are used by other systems

Configuring
These types of inspections can be found in Java > Declaration redundancyJava > Imports and Java > Probable bugs. Or you can search for the string “unused” in the IntelliJ IDEA inspection settings.

And to make it even easier…

The navigation and inspection features are all available in the Upsource application. While it would be great if the app could provide everything we as developers want, sometimes we just feel more comfortable in the IDE.  So that’s why there’s also an Upsource plugin for IntelliJ IDEA and other JetBrains IDEs, so we can do the whole code review from within our IDE.  There’s also a new Open in IDE feature in Upsource 2.5 which, well, lets you open a code review in your IDE.

Conclusion

While many checks can and should be automated, and while humans are required to think about bigger-picture issues like design and “but did it actually fix the problem?”, there’s also a grey area between the two. In this grey area, what we as code reviewers could benefit from is some guidance about code that looks dodgy but might be OK. It seems logical that a code review tool should provide this guidance. Not only this, but we should also expect our code review tool to allow us to navigate through the code as naturally as we would in our IDE.

Upsource aims to make code review not only as painless as possible, but also provide as much help as a tool can, freeing you up to worry about the things that humans are really good at.

Posted in Code review practices | Leave a comment

What to look for in a Code Review: Security

Posted on by Trisha Gee

This is part 6 of 6 posts on what to look for in a code review. See previous posts from the series.

How much work you do building a secure, robust system is like anything else on your project – it depends upon the project itself, where it’s running, who’s using it, what data it has access to, etc. Often, if our team doesn’t have access to security experts, we go too far in one direction or the other: either we don’t pay enough attention to security issues; or we go through some compliance checklist and try to address everything in some 20 page document filled with potential issues.

As usual, this blog post aims to highlight some areas you might like to look at when reviewing code, but mostly it aims to get you having discussions within your team or organisation to figure out what it is you do need to care about in a code review.

Automation is your friend
A surprising number of security checks can be automated, and therefore should not need a human. Security tests don’t necessarily have to be full-blown penetration testing with the whole system up and running, some problems can be found at code-level.

Common problems like SQL Injection or Cross-site Scripting can be found via tools running in your Continuous Integration environment. You can also automate checking for known vulnerabilities in your dependencies via the OWASP Dependency Check tool.

Of course, Upsource also provides numerous security inspections. These can inform a reviewer of potential security problems in the code. For example, this code executes a dynamically generated SQL string, which might be susceptible to SQL Injection:

SQL Warning

Sometimes “It Depends”
While there are checks that you can feel comfortable with a “yes” or “no” answer, sometimes you want a tool to point out potential problems and then have a human make the decision as to whether this needs to be addressed or not. This is an area where Upsource can really shine. Upsource displays code inspections that a reviewer can use to decide if the code needs to be changed or is acceptable under the current situation.

For example, suppose you’re generating a random number. If all your security checks are enabled, you’ll see the following warning in Upsource:

Security Warning - Random Number

The JavaDoc for java.util.Random specifically states “Instances of java.util.Random are not cryptographically secure”. This may be fine for many of the occasions when you need an arbitrary random number.  But if you’re using it for something like session IDs, password reset links, nonces or salts, as a reviewer you might suggest replacing Random with java.util.SecureRandom.

If you and the code author decide that Random is appropriate for this situation, then it’s a good idea to suppress this inspection for this line of code, and document why it’s OK or point to any discussion on the subject – this way future developers looking at the code can understand this is a deliberate decision.

Suppress Warning

So while tools can definitely point you at potential issues, part of your job as a code reviewer is to investigate the results of any automated checks and decide which action to take.

If you are using Upsource to review your code, you can customise your inspection settings, including selecting security settings. Do this by opening your project in IntelliJ IDEA and navigating to the Inspections settings. Select the settings you want and save them to the Project Default profile. Make sure Project_Default.xml is checked in with your project code, and Upsource will use this to determine which inspections to run.

At the time of writing, these are the available security inspections:

Security Inspections

Understand your Dependencies
Let’s move on to other areas that need a human reviewer. One of the areas where security vulnerabilities can creep into your system or code base is via third party libraries. When reviewing code, at the very least you want to check if any new dependencies (e.g. third party libraries) have been introduced. If you aren’t already automating the check for vulnerabilities, you should check for known issues in newly-introduced libraries.

You should also try to minimise the number of versions of each library – not always possible if other dependencies are pulling in additional transitive dependencies. But one of the simplest way to minimise your exposure to security problems in other people’s code (via libraries or services) is to

  • Use a few as sources as possible and understand how trustworthy they are
  • Use the highest quality library you can
  • Track what you use and where, so if new vulnerabilities do become apparent, you can check your exposure.

This means:

  1. Understanding your sources (e.g. maven central or your own repo vs arbitrarily downloaded jar files)
  2. Trying not to use 5 different versions of 3 different logging frameworks (for example)
  3. Being able to view your dependency tree, even if it’s simply through Gradle/Maven

Check if new paths & services need to be authenticated
Whether you’re working on a web application, or providing web services or some other API which requires authentication, when you add a new URI or service, you should ensure that this cannot be accessed without authentication (assuming authentication is a requirement of your system). You may simply need to check that the developer of the code wrote an appropriate test to show that authentication has been applied.

You should also consider that authentication isn’t just for human users with a username and password. Identity might need to be defined for other systems or automated processes accessing your application or services. This may impact your concept of “user” in your system.

Does your data need to be encrypted?
When you’re storing something on disk or sending things over the wire, you need to know whether that data should be encrypted. Obviously passwords should never be in plain text, but there are plenty other times when data needs to be encrypted. If the code under review is sending data on the wire, saving it somewhere, or it is in some way leaving your system, if you don’t know whether it should be encrypted or not, try and locate someone in your organisation who can answer that question.

Are secrets being managed correctly?
Secrets are things like passwords (user passwords, or passwords to databases or other systems), encryption keys, tokens and so forth. These should never be stored in code, or in configuration files that get checked into the source control system. There are other ways of managing these secrets, for example via a secrets server. When reviewing code, make sure these things don’t accidentally sneak into your VCS.

Should the code be logging/auditing behaviour? Is it doing so correctly?
Logging and auditing requirements vary by project, with some systems requiring compliance with stricter rules for logging actions and events than others. If you do have guidelines on what needs logging, when and how, then as a code reviewer you should be checking the submitted code meets these requirements. If you do not have a firm set of rules, consider:

  • Is the code making any data changes (e.g. add/update/remove)? Should it make a note of the change that was made, by whom, and when?
  • Is this code on some performance-critical path? Should it be making a note of start-time and end-time in some sort of performance monitoring system?
  • Is the logging level of any logged messages appropriate? A good rule of thumb is that “ERROR” is likely to cause an alert to go off somewhere, possibly on some poor on-call person’s pager – if you do not need this message to wake someone up at 3am, consider downgrading to “INFO” or “DEBUG”. Messages inside loops, or other places that are likely to be output more than once in a row, probably don’t need to be spamming your production log files, therefore are likely to be “DEBUG” level.

Conclusion
This is just a tiny subset of the sorts of security issues you can be checking in a code review. Security is a very big topic, big enough that your company may hire technical security experts, or at least devote some time or resources to this area. However, like other non-coding activities such as getting to know the business and having a decent grasp of how to test the system, understanding the security requirements of our application, or at least of the feature or defect we’re working on right now, is another facet of our job as a developer.

We can enlist the help of security experts if we have them, for example inviting them to the code review, or inviting them to pair with us while we review. Or if this isn’t an option, we can learn enough about the environment of our system to understand what sort of security requirements we have (internal-facing enterprise apps will have a different profile to customer-facing web applications, for example), so we can get a better understanding of what we should be looking for in a code review.

And like many other things we’re tempted to look for in code reviews, many security checks can also be automated, and should be run in our continuous integration environment. As a team, you need to discuss which things are important to you, whether checking these can be automated, and which things you should be looking for in a code review.

This post has barely scratched the surface of potential issues. We’d love to hear from you in the comments – let us know of other security gotchas we should be looking for in our code.

Posted in Code review practices | Leave a comment

Upsource In Action, Webinar Recording

Posted on by Maria Khalusova

The recording of our September 29th webinar with Trisha Gee, Upsource In Action, is available on JetBrains YouTube channel.

In this webinar Trisha goes over the typical code review workflow in Upsource, then explores Java support features similar to what you have in IntelliJ IDEA, she also shows how you can use Upsource to explore your code bases, and find exactly what you need using powerful search engine. In the end, she demonstrates some of the features that we are working on for the upcoming Upsource 2.5 release.

Below is the timeline of the webinar and select Q&A.

0:22 – Typical code review workflow in Upsource
10:12 – Java support
15:40 – Exploring code bases with Upsource
19:28 – Powerful search
22:35 – New features in upcoming Upsource 2.5

Q: Are there plans to integrate with Atlassian JIRA?
A:Integration with JIRA is already available starting with Upsource 2.0.3.

Q: What is the effect of setting default branch in Upsource?
A: When the default branch is set, the UI becomes a bit smarter when displaying the commits graph, rendering README files, comparing branches, etc.

Q: When is git repository hosting coming?
A: It is planned for Q1 2016.

Q: Is it possible to create a review for a MergeRequest(gitlab)/PullRequest(gihub)?
A: Pull requests support is planned for Q1 2016.

Q: Can I, as a reviewer do the review in IntelliJ? See all the changed files and their diff? And add comments?
A: Yes, you can participate in code review either from Upsource UI, or from IntelliJ IDEA, if you have Upsource plugin installed, and do all the same as in Upsource UI: create a review, view diffs, participate in discussions, leave comments, resolve comments, receive notifications, approve/reject, etc.

Q: Are there any plans in roadmap to integrate with other IDEs beside IntelliJ IDEA?
A: All JetBrains IDEs are supported including PhpStorm, RubyMine, WebStorm, CLion, AppCode, PyCharm.

Trisha_Gee
Trisha has developed Java applications for a range of industries, including finance, manufacturing and non-profit, for companies of all sizes. She has expertise in Java high performance systems, is passionate about enabling developer productivity, and dabbles with Open Source development.

Posted in Webinars | Leave a comment

Upsource 2.5 Early Access is open!

Posted on by Maria Khalusova

It has been slightly more than a year since the first ever Early Access Upsource build has seen the light. A lot has changed since then, two major releases came out, many new features were introduced, and our up-and-coming code review tool has significantly matured. And we are devoted to keep making it better, faster, and, of course, smarter!

Today we are opening the Early Access Program for the next major version – Upsource 2.5, which is planned to be made available within this year. Here’s what we have for you so far.

Enhanced Email Notifications

We no longer send emails as soon as some noteworthy event occurs (comment was left on your review, changes accepted or rejected, etc.). Instead, we bundle them and send a single email so you have less in your inbox to go through. It is also possible to fine-tune the types of notifications you want to receive. Don’t want to get notified when a review is closed by its author? Disable this particular notification on the Settings page! In addition to that, notifications also feature better code highlighting and other design improvements.

email-notification

Reply by email

Another great feature of reworked email notifications you may notice on the screenshot above is the ability to reply to comments via email. Create a mailbox, configure it in Upsource and you’ll be able to participate in ongoing reviews without ever leaving your email client.

Smarter and faster indexer

Upsource 2.5 drastically reduces the time needed to index a newly added project by introducing bulk import of repositories and on-demand indexing of old revisions. While previously a repository with 10,000 commits took hours to index, it now takes only a few short minutes. Indexing has become 10x faster for Git projects.

Thanks to a smarter indexer, we are now able to reindex the project when settings such as project model or character encoding are changed.

Discussion Labels

Labels can now be applied to discussions to denote priority, category, or any other information that you find useful. Some predefined labels are provided (“bug”, “style”, “enhancement”, “help wanted”), others can be created on a per-project basis. As we strive to make our data eminently searchable, you’re able to find discussions marked with a certain label using label: query.

discussionLabels

Branches page

See the activity across branches on the brand-new Branches page. You can quickly search the branches and see which ones are active or stale.

branches_page

Default branch setting was added to the project configuration screen. When the default branch is set, the UI becomes a bit smarter when displaying the commits graph, rendering README files, comparing branches, etc.

Open in IDE

Version 2.0 introduced an IDE plugin that allowed interacting with Upsource from the IDE. In 2.5 we are providing integration in the opposite direction – it is now possible to open a review (or simply a piece of code) in the IDE by clicking a link in the browser so you can quickly resume work on the code under review.

OpenInIDE

Various improvements

  • Sometimes to understand a change in the inline diff we need to see more than the default context. You can now gradually reveal additional lines of context by clicking on cut lines.
  • Reviews can now be removed. Code comments, if any, are preserved as part of corresponding revisions.
  • We now show the number of discussions right in the revisions list as well as in the list of reviews for instant visibility of your colleagues’ activity.
  • Review timeline has a new option of showing only unresolved discussions.
  • When an issue ID or review ID is present in the commit message we automatically attach the revision to the matching review if it’s’ open – which means you no longer have to do it manually. This behavior is always enabled and not configurable, we plan to provide more customisable workflows later on.

This list of features is not final, as we have more great surprises planned for you.
If you want to play with the EAP build, you can download it here. Please remember, that Early Access builds represent work in progress, therefore we recommend installing them on a trial server. See the reasons why.
As usual, we will be happy to hear your feedback!

Posted in EAP | 3 Comments

Live Webinar: Upsource In Action

Posted on by Maria Khalusova

Join us Tuesday, September 29th, 09:00 AM-10:00 AM PDT (16:00 – 17:00 GMT) for our free live webinar with Trisha Gee featuring Upsource, our code review and repository browsing tool, in action.

In this webinar you’ll see how you can have lightweight transparent code reviews without the need to change your existing process. Next, we are going to show you the “superpowers” Upsource gives to Java teams.

In this presentation you will learn how Upsource helps teams with their daily routines, and for dessert we’ll give you a sneak peek of the features we are preparing for the upcoming release.

Space is limited, please register now.

This webinar is geared towards developers of different proficiency regardless of programming language of choice. During the webinar there will be an opportunity to ask questions. The recording will be available after the webinar.

Trisha_Gee
Trisha has developed Java applications for a range of industries, including finance, manufacturing and non-profit, for companies of all sizes. She has expertise in Java high performance systems, is passionate about enabling developer productivity, and dabbles with Open Source development.

Posted in Webinars | Leave a comment

Mysterious “Build System” Setting

Posted on by Maria Khalusova

From time to time we receive questions from our users about a setting called “Build System” under Upsource Properties on Project Setting page:

build-system-setting

The most common ones are: “Why do I even need to specify build system?” and “What does it do?”. Sometimes there’s also confusion about what exactly one needs to specify there. While we do plan to address this ambiguity through UI modifications, we feel it’s a good idea to explain why a code review tool even needs it in the first place and what to do with it.

In short, this setting unlocks all the code insight and intelligence Upsource offers for Java teams. So if you’re using Upsource to perform code reviews on a project written in another language, feel free to ignore it. But if your team uses Java, and you want to make use of IDE-level server side code inspections and navigation, then keep reading:)

Currently 3 types of build systems are supported: IDEA, Maven and Gradle. Let’s take a look at each of them.

IDEA

Upsource has IntelliJ IDEA engine built in to provide the same level of code intelligence as IntelliJ IDEA does. Needless to say, Upsource natively understands IDEA project model, so you only need to let it know where the .idea folder is located.

In some teams each developer configures their IDE to their liking and everybody has different .idea, which they .gitignore, so in the end there’s no .idea folder in the repository. However, the recommended way is to store most of the .idea folder contents in version control, only omitting the user-specific settings, such as .idea/workspace.xml and .idea/shelf – this way Upsource will be able to provide Java code insight for your project.

Maven/Gradle

Things are a bit different for Maven/Gradle project models, as IntelliJ IDEA engine does not understand them natively. To be able to offer code insight for Java projects with these models Upsource first converts specified project model to IDEA format and uses it later on. If your pom.xml doesn’t have unresolved dependencies and builds fine without Upsource, it should convert properly into IDEA format as well: you can always check if all went well in the maven.out/gradle.out file. To find it, open your Upsource project, click Browse code at the top of the revisions list and navigate to /.idea/maven/mvn.out or /.idea/gradle/gradle.out.

Not sure which pom.xml to specify in Upsource properties? You need the one that knows about all the modules you have and you want to have code insight for, i.e. it should be the parent pom. If your project modules are hosted in multiple repositories (e.g. each module in their own repository), you can configure them all in one Upsource project, and keep the parent pom.xml in the first repo.

When it comes to Gradle, the number one troublemakers are Android Studio projects. Unfortunately, Upsource does not support them at the moment. We are looking into it, and by all means feel free to follow/upvote the issue.

There also have been known cases when gradle plugin converted project model, and configured wrong SDK. It’s a known Gradle bug. For the moment, if this is happening, you can find the workaround here. If you plan to work on Gradle projects without a wrapper, please check Upsource documentation for the instructions.

Naturally, we are always happy to answer your questions if you have any.

Posted in Feature | Leave a comment

Upsource 2.0.4 bug-fix update

Posted on by Maria Khalusova

Hi everyone!

Previous Upsource update featured YouTrack integration via external Hub, and some of our users have experienced issues with the integration. We apologise for inconvenience this may have caused and we would like to thank you for your patience and for reporting to us the problems you have encountered.
We have addressed the issues and today we have made available a bug-fix update. Please, upgrade your Hub as well as soon as the new Hub version is available.

Take a look at the complete list of fixes here, and download the build.

We recommend upgrading your instance to the latest version: for the upgrade instructions, please refer to Upsource documentation. Don’t forget to backup your current instance before upgrading.

Posted in Update | Leave a comment

What to look for in a Code Review: SOLID Principles

Posted on by Trisha Gee

This is part 5 of 6 posts on what to look for in a code review. See other posts from the series.

In today’s post we’ll look more closely at the design of the code itself, specifically checking to see if it follows good practice Object Oriented Design.  As with all the other areas we’ve covered, not all teams will prioritise this as the highest value area to check, but if you are trying to follow SOLID Principles, or trying to move your code in that direction, here are some pointers that might help.

What is SOLID?

The SOLID Principles are five core principles of Object Oriented design and programming. The purpose of this post is not to educate you on what these principles are or go into depth about why you might follow them, but instead to point those performing code reviews to code smells that might be a result of not following these principles.

SOLID stands for:

Single Responsibility Principle (SRP)

There should never be more than one reason for a class to change.

This can sometimes be hard to spot from a single code review. By definition, the author is (or should be) applying a single reason to change the code base – a bug fix, a new feature, a focussed refactoring.

You want to look at which methods in a class are likely to change at the same time, and which clusters of methods are unlikely to ever be changed by a change to the other methods. For example:

Single Responsibility Principle

This side-by-side diff from Upsource shows that a new piece of functionality has been added to TweetMonitor, the ability to draw the top ten Tweeters in a leaderboard on some sort of user interface. While this seems reasonable because it uses the data being gathered by the onMessage method, there are indications that this violates SRP. The onMessage and getTweetMessageFromFullTweet methods are both about receiving and parsing a Twitter message, whereas draw is all about reorganising that data for displaying on a UI.

The reviewer should flag these two responsibilities, and then work out with the author a better way of separating these features: perhaps by moving the Twitter string parsing into a different class; or by creating a different class that’s responsible for rendering the leaderboard.

Open-Closed Principle (OCP)

Software entities should be open for extension, but closed for modification.

As a reviewer, you might see indications that this principle is being violated if you see a series of if statements checking for things of a particular type:

Open-Closed Principle

If you were reviewing the code above, it should be clear to you that when a new Event type is added into the system, the creator of the new event type is probably going to have to add another else to this method to deal with the new event type.

It would be better to use polymorphism to remove this if:

Open-Closed Principle

Open-Closed Principle

As always, there’s more than one solution to this problem, but the key will be removing the complex if/else and the instanceof checks.

Liskov Substitution Principle (LSP)

Functions that use references to base classes must be able to use objects of derived classes without knowing it.

One easy way to spot violations of this principle is to look for explicit casting. If you have to cast a object to some type, you are not using the base class without knowledge of the derived classes.

More subtle violations can be found when checking:

Imagine, for example, we have an abstract Order with a number of subclasses – BookOrder, ElectronicsOrder and so on. The placeOrder method could take a Warehouse, and could use this to change the inventory levels of the physical items in the warehouse:

Liskov Substitution Principle

Now imagine we introduce the idea of electronic gift cards, which simply add balance to a wallet but do not require physical inventory. If implemented as a GiftCardOrder, the placeOrder method would not have to use the warehouse parameter:

CR5-LSP1

This might seem like a logical use of inheritance, but in fact you could argue that code that uses GiftCardOrder could expect similar behaviour from this class as the other classes, i.e. you could expect this to pass for all subtypes:

CR5-LSP2

But this will not pass, as GiftCardOrders have a different type of order behaviour. If you’re reviewing this sort of code, question the use of inheritance here – maybe the order behaviour can be plugged in using composition instead of inheritance.

Interface Segregation Principle (ISP)

Many client specific interfaces are better than one general purpose interface.

Some code that violates this principle will be easy to identify due to having interfaces with a lot of methods on.  This principle compliments SRP, as you may see that an interface with many methods is actually responsible for more than one area of functionality.

But sometimes even an interface with just two methods could be split into two interfaces:

CR5-LSP3

In this example, given that there are times when the decode method might not be needed, and also that a codec can probably be treated as either an encoder or a decoder depending upon where it’s used, it may be better to split the SimpleCodec interface into an Encoder and a Decoder. Some classes may choose to implement both, but it will not be necessary for implementations to override methods they do not need, or for classes that only need an Encoder to be aware that their Encoder instance also implements decode.

Dependency Inversion Principle (DIP)

Depend upon Abstractions. Do not depend upon concretions.

While it may be tempting to look for simple cases that violate this, like liberal use of the new keyword (instead of using Dependency Injection or factories, for example) and overfamiliarity with your collection types (e.g. declaring ArrayList variables or parameters instead of List), as a reviewer you should be looking to make sure the code author has used or created the correct abstractions in the code under review.

For example, service-level code that uses a direct connection to a database to read and write data:

CR5-DIP1

This code is dependent on a lot of specific implementation details: JDBC as a connection to a (relational) database; database-specific SQL; knowledge of the database structure; and so on. This does belong somewhere in your system, but not here where there are other methods that don’t need to know about databases. Better to extract a DAO or use the Repository pattern, and inject the DAO or repository into this service.

Summary

Some code smells that might indicate one or more of the SOLID Principles have been violated:

  • Long if/else statements
  • Casting to a subtype
  • Many public methods
  • Implemented methods that throw UnsupportedOperationException

As with all design questions, finding a balance between following these principles and knowingly bending the rules is down to your team’s preferences. But if you see complex code in a code review, you might find that applying one of these principles will provide a simpler, more understandable, solution.

Posted in Code review practices | 4 Comments

What to look for in a Code Review: Data Structures

Posted on by Trisha Gee

This is part 4 of 6 posts on what to look for in a code review. See previous posts from the series.

Data structures are a fundamental part of programming – so much so it’s actually one of the areas that’s consistently taught in Computer Science courses. And yet it’s surprisingly easy to misuse them or select the wrong one. In this post, we’re going to guide you, the code reviewer, on what to look out for – we’re going to look at examples of code and talk about “smells” that might indicate the wrong data structure was chosen or that it’s being used in an incorrect fashion.

Lists

Probably the most common choice for a data structure. Because it is the most common choice, it’s sometimes used in situations it shouldn’t be.

Anti-Pattern: Too Much Searching Iterating over a listIterating over a list is not, in itself, a bad thing of course. But if iteration is required for a very common operation (like the example above of finding a customer by ID), there might be a better data structure to use. In our case, because we always want to find a particular item by ID, it might be better to create a map of ID to Customer.

Remember that in Java 8, and languages which support more expressive searches, this might not be as obvious as a for-loop, but the problem still remains.

Iterating over a list using Java 8

Anti-Pattern: Frequent Reordering

List Sorting

Lists are great if you want to stick to their default order, but if as a reviewer you see code that’s re-sorting the list, question whether a list is the correct choice. In the code above, on line 16 the twitterUsers list is always re-sorted before being returned. Once again, Java 8 makes this operation look so easy it might be tempting to ignore the signs:

List sorting in Java 8

Upsource shows the pre-Java-8 method of sorting (lines 16-20, in red) replaced with the Streams solution (line 16, in green)

In this case, given that a TwitterUser is unique and it looks like you want a collection that’s sorted by default, you probably want something like a TreeSet.

Use a sorted set instead of a list

Upsource’s side-by-side diff with the change

Maps

A versatile data structure that provide O(1) access to individual elements, if you’ve picked the right key.

Anti-Pattern: Map as global constant

The map is such a good general purpose data structure that it can be tempting to use globally accessible maps to let any class get to the data.

CR4MapGlobal

In the above code, the author has chosen to simply expose the CUSTOMERS map as a global constant. The CustomerUpdateService therefore uses this map directly when adding or updating customers. This might not seem too terrible, since the CustomerUpdateService is responsible for add and update operations, and these have to ultimately change the map. The issue comes when other classes, particularly those from other parts of the system, need access to the data.

CR4MapAbuse

Here, the order service is aware of the data structure used to store customers. In fact, in the code above, the author has made an error – they don’t check to see if the customer is null, so line 12 could cause a NullPointerException. As the reviewer of this code, you’ll want to suggest hiding this data structure away and providing suitable access methods. That will make these other classes easier to understand, and hide any complexity of managing the map in the CustomerRepository, where it belongs. In addition, if later you change the customers data structure, or you move to using a distributed cache or some other technology, the changes associated with that will be restricted to the CustomerRepository class and not ripple throughout the system. This is the principle of Information Hiding.

CR4MapHidden

Although the updated code isn’t much shorter, you have standardised and centralised core functions – for example, you know that getting a customer who doesn’t exist is always going to give you an Exception. Or you can choose to have this method return the new Optional type.

Note that this is exactly the sort of issue that should be found during a code review – hiding global constants is hard to do once their use has propagated throughout the system, but it’s easy to catch this when they’re first introduced.

Other Anti-Patterns: Iteration & Reordering

As with lists, if a code author has introduced a lot of sorting of, or iterating over, a map, you might want to suggest an alternative data structure.

Java-specific things to be aware of

In Java, map behaviour usually relies on your implementation of equals and hashCode for the key and the value. As a reviewer, you should check these methods on the key and value classes to ensure you’re getting the expected behaviour.

Java 8 has added a number of very useful methods to the Map interface. The getOrDefault method, for example, could simplify the CustomerRepository code at line 11 in the example above.

Sets

An often-underused data structure, its strength is that is does not contain duplicate elements.

Anti-pattern: Sometimes you really do want duplicates

Let’s assume you had a user class that used a set to track which website they had visited. Now, the new feature is to return the most recently visited of these websites.

CR4SetMisuse

The author of this code has changed the initial set that tracks the sites a user has visited from HashSet to LinkedHashSet – this latter implementation preserves insertion order, so now our set tracks every URI in the order in which they were visited.

There are a number of signs in this code that this is wrong though. Firstly, the author has had to do a costly full iteration of the whole set to reach the last element (lines 13-15) – sets are not designed for accessing elements by position, something that lists are perfect for. Secondly, because sets do not contain duplicate values, if the last page they visited had been visited previously, it will not be in the last position in the set. Instead, it will be where it was first added to the set.

In this case, a list, a stack (see below), or even just a single field, might give us better access to the last page visited.

Java-specific things to be aware of

Because one of the key operations of a set is contains, as a reviewer you should be checking the implementation of equals on the type contained in the set.

Stacks

Stacks are a favourite of Computer Science classes, and yet in the real world are often overlooked – in Java, maybe this is because Stack is an extension of Vector and therefore somewhat old-fashioned. Rather than going into a lot of detail here I’ll just cover key points:

  • Stacks support LIFO, and should ideally be used with push/pop operations, it’s not really for iterating over.
  • The class you want for a stack implementation in Java (since version 1.6) is Deque. This can act as both a queue and a stack, so reviewers should check that deques are used in a consistent fashion in the code.

Queues

Another CS favourite. Queues are often spoken about in relation to concurrency (indeed, most of the Java implementations live in java.util.concurrent), as it’s a common way to pass data between threads or modules.

  • Queues are FIFO data structures, generally working well when you want to add elements to the tail of the queue, or remove things from the front of the queue. If you’re reviewing code that shows iteration over a queue (in particularly accessing elements in the middle of the queue), question if this is the correct data type.
  • Queues can be bounded or unbounded. Unbounded queues could potentially grow forever, so if reviewing code with this type of data structure, check out the earlier post on performance. Bounded queues can come with their own problems too – when reviewing code, you should look for the conditions under which the queue might become full, and ask what happens to the system under these circumstances.

A general note for Java developers

As a reviewer, you should be aware not only of the characteristics of general data structures, but you should also be aware of the strengths and weaknesses of each of the implementations all of which are documented in the Javadoc:

If you’re using Java 8, remember that many of the collections classes have new methods. As a reviewer you should be aware of these – you should be able to suggest places where the new methods can be used in place of older, more complex code.

Why select the right data structure?

We’ve spent this blog post looking at data structures – how to tell if the code under review might be using the wrong data structures, and some pointers for the pros and cons of various data structure so not only can you, as the reviewer, identify when they might be being used incorrectly, but you can also suggest better alternatives. But let’s talk about why using the right data structure is important.

Performance

If you’ve studied data structures in computer science, you’ll often learn about the performance implications of picking one over another. Indeed, we even mentioned “Big O Notation” in this blog post to highlight some of the strengths of particular structures. Using the right data structure in your code can definitely help performance, but this is not the only reason to pick the right tool for the job.

Stating Expected Behaviour

Developers who come to the code later, or who use any API exposed by your system, will make certain assumptions based on data structures. If the data returned from a method call is in a list, a developer will assume it is ordered in some fashion. If data is stored in a map, a developer can assume that there is a frequent need to look up individual elements by the key. If data is in a set, then a developer can assume it’s intentional that this only stores an element once and not multiple times. It’s a good idea to work within these assumptions rather than break them.

Reducing Complexity

The overall goal of any developer, and especially of a reviewer, should be to ensure the code does what it’s supposed to do with the minimal amount of complexity – this makes the code easier to read, easier to reason about, and easier to change and maintain in the future. In some of the anti-patterns above, for example the misuse of Set, we can see that picking the wrong data structure forced the author to write a lot more code. Selecting the right data structure should, generally, simplify the code.

In Summary

Picking the right data structure is not simply about gaining performance or looking clever in front of your peers. It also leads to more understandable, maintainable code. Common signs that the code author has picked the wrong data structure:

  • Lots of iterating over the data structure to find some value or values
  • Frequent reordering of the data
  • Not using methods that provide key features – e.g. push or pop on a stack
  • Complex code either reading from or writing to the data structure

In addition, exposing the details of selected data structures, either by providing global access to the structure itself, or by tightly coupling your class’s interface to the operation of an underlying data structure, leads to a brittleness of design, and will be hard to undo later. It’s better to catch these problems early on, for example during a code review, than incur avoidable technical debt.

Posted in Code review practices | 5 Comments

Malicious code reviews: 11 tips to piss the whole team off

Posted on by Maria Khalusova

Disclaimer: Please do not take this post seriously, and happy Friday!

Your team has decided to adopt code review practice, and your perfect code now will be judged by someone else. Well, how dare they think they can find any issues with your brilliant code? On top of that, you need to spend your valuable time reviewing someone else’s code? That does it! Fight back the tyranny by following these simple rules:
If your code is about to be reviewed

  1. Commit all the features and bug-fixes you’ve worked on this week in one go, and ask for a review. Let them see how much work you’ve done.
  2. Invite the whole team to review your code. Everybody has to know how great your features are.
  3. Be original. Do not use spell checker, do not use static code analysis. Do, however, use ReSharper or IntelliJ IDEA to reformat code to your style, ignoring the company’s standards.
  4. Introduce a tricky bug on purpose. See if they’re smart enough to find it.

If you are reviewing someone else’s code

  1. Take your time. Got a code review assigned to you? Take your time, it’s not important. No, really, have a cup of tea, answer all unanswered emails, tweet something, see what you friends have been up to on facebook, go home. Why should reading someone’s code be more important than what you want to do? Feel free to ignore notifications for a week or two: you are busy.
  2. Take your time doing review. They wanted you to find issues? Don’t close the review until you find at least a dozen, even if it takes months.
  3. See a bug? Perfect! First, question the author’s intelligence, then, when your superiority is well established, demand a fix.
  4. Found an elegant solution to a problem? Don’t tell anyone, especially the code author, that you’re impressed. Keep it cool, act like you’ve always knew how to do things best.
  5. Embrace your inner grammar nazi: Focus on the spelling, it’s the most important thing! How can we misspell things and call ourselves professionals?
  6. Count all the spaces. Point out every bad indentation.
  7. Never compromise. Even if another solution offered by your teammate is better than yours. Stand your ground.

Good luck!

 

Eclipse IDE optimization

7 Tips to Speed Up Eclipse

About a month ago, I blogged about my love/hate relationship with Eclipse. I was asked by a few people to share my tips on how I was able to speed it up so here we go… As a side note, this article is not about comparing IDEs, please refrain from displaying your hate for the IDE or your preference for another…  This post is just about optimizations that help Eclipse run faster for those who use it. I’ve described each tip for Windows, Linux and MacOS users. Once you have gone through all the optimization tips, Eclipse should start within 10 seconds and run much smoother than before.

[edit]: most of those tips will speed up your Eclipse experience, not just the startup time. Classes will open faster, jumping from one definition to another will be faster. Viewing method javadocs will be faster… Unfortunately, none of that can be timed precisely so there is no specific benchmark about the actual speed gains for each tip.

Note: I assume that you already have an SSD and at least 8Gb of RAM. If not, look no further for performance boosts…

Tip 1 : Always run the latest JDK and Eclipse.

More often than not, every new JDK and Eclipse version includes fixes and optimizations from previous versions. Make sure that you are using the 64 bits version of Eclipse and of the Oracle JDK. For any web development, you will want to use Eclipse for Java EE and not Eclipse for Java.

Oracle JDK : http://www.oracle.com/technetwork/java/javase/downloads
Eclipse : https://eclipse.org/downloads/

Place Eclipse and your code on your SSD.

Time to launch Eclipse.

Tip 2 : Tune Eclipse Preferences

  • General > Startup and Shutdown : remove all plugins activated on startup
  • General > Editors > Text Editors > Spelling : Disable spell checking
  • General > Validation > Suspend all
  • Window > Customize Perspective > Remove stuff you don’t use or want (shortcut keys are your friends), same for Menu Visibility (how many times have you printed a source file…)
  • Install/Update > Automatic Updates > Uncheck “Automatically find new updates”
  • General > Appearance > Uncheck Enable Animations
  • Stay with the default theme. Unfortunately, anything else makes it really laggy and slow.

I personally turn off autosuggestions so that they don’t slow down my typing. Instead, I manually trigger the auto suggestions using Ctrl+Space when needed. This is done through:

  • Java > Editor > Content Assist > disable Enable Auto Activation. Advanced > Remove all unwanted kinds

Tip 3 : Keep your JDK on a RAM Disk

A RAM disk is a virtual disk or hard drive that is stored into the computer’s memory. It boosts the I/O performance on anything that is on it. Creating a RAM disk uses your RAM even though it just appears as a drive on the computer. Because the memory used will be reserved for the disk and become unavailable for other programs, we will only be putting the JDK on there. 300MB will be enough.

Warning: Don’t put anything permanent on the RAM Disk we will create as it will be erased/recreated at each reboot.

Step by Step for Linux users:

Everything is described here already

Step by Step for Mac Users:

The RAM Disk is created with the diskutil tool:

  1. Create a new batch file, for example: ~/tools/batch/ramdisk.sh
    Replace x, y, and z with the JDK version you have installed on disk:

#!/bin/bash

diskutil erasevolume HFS+ ‘JDK RAMDISK’ `hdiutil attach -nomount ram://614400`

cp -r  /Library/Java/JavaVirtualMachines/jdk1.x.y_z.jdk /Volumes/JDKRAMDISK

(Note that diskutil expects the number of 512 bytes sectors to be allocated: for 300MB, 300 * 1024 ^ 2 / 512 = 614400)

  1. Save your file and make it executable:

chmod 755 ~/tools/batch/ramdisk.sh

Now run ramdisk.sh to create the Ram Disk:

$ ramdisk.sh

Started erase on disk4

Unmounting disk

Erasing

Initialized /dev/rdisk4 as a 300 MB case-insensitive HFS Plus volume

Mounting disk

Finished erase on disk4 JDKRAMDISK

You should now see in Finder the new device called JDKRAMDISK containing your JDK. You can remove the RAM Disk and free its memory by clicking on the Eject button:

Note that if you follow this tip, you will always need to have the RAM Disk in order to launch Eclipse (otherwise you will see the error “A Java Runtime … must be available in order to run Eclipse”). You can configure your system to run the ramdisk.sh script automatically upon startup using Automator or using a launchtl daemon.

Step by Step for Windows Users:

  1. Download and install the utility called imdisk
  2. Create a new batch file, for example: C:/tools/batch/ramdisk.bat
    Replace x, y, and z with the JDK version you have installed on disk:

@echo Placing JDK on Virtual Disk N:/

@echo off

sc config imdisk start= auto

net start imdisk

imdisk -a -t vm -s 300m -m n:

format n: /q /Y

call xcopy C:\<path_jdk>\jdk1.x.y_z N:\jdk1.x.y_z\ /S /E /Y /Q

label n: JDK RAMDISK

By running ramdisk.bat, you will have created a new disk N: labeled “JDK RAMDISK” that will contain your JDK.

  1. Make sure the file is run as an Administrator. Right click on the file, go into Properties and check Run as Administrator.

Note that if you follow this tip, you will always need to have the RAM Disk in order to launch Eclipse (otherwise you will see the error “A Java Runtime … must be available in order to run Eclipse”). You can configure your system to run the ramdisk.bat script automatically by placing the script into your Startup folder.

In order for Tip 3 to work, you will need to add the -vm setting in eclipse.ini (see next section)

Tip 4 : Tweak your eclipse.ini

This is one of the most confusing areas of Eclipse optimizations. There are thousands of online articles preaching different configurations… I’ll just describe the way I tweaked my options and add more to the existing confusion

Location your eclipse.ini file:
Windows/Linux: located in $ECLIPSE_HOME
MacOS: located in $ECLIPSE_HOME/Eclipse.app/Contents/MacOS

Understanding what’s going on…

Eclipse.ini contains 2 types of properties: properties relative to the Eclipse application and properties relative to the JVM. Those options are different depending upon your version of JDK or Eclipse. Here is the most up-to-date list I was able to find online.

To understand those options require just a bit of vocabulary on Oracle’s JVM. Basically, the JVM memory is divided into multiple memory pools where objects reside with time:

  • The Eden Space (heap) provides the memory for most initial objects. The Garbage Collector passes often through this space containing objects of “young generation”. It removes any objects that hasn’t been used for a while.
  • The Survivor Space (heap) contains the objects that have not been destroyed after 2-3 passes of the GC in the Eden Space. They are still part of the “young generation” but have moved to a much safer place where they have less chances of being destroyed: the Garbage Collector passes much less often there (it assumes from past experience that objects are used more frequently).
  • The Tenured Space (heap) contains the objects that have been in the Survivor Space for a while.
  • The Permanent Generation (non-heap) contains all the metadata about of the JVM, such as the class properties, methods, enums, etc. Because that data can be shared accross multiple JVMs, the permanent generation has read-only and read-write areas.
  • The Code Cache (non-heap) provides the memory used for compiling and storing code.

In case you are interested, Oracle wrote a great article about garbage collection tuning, detailing all those spaces and roles

The sizes for all those memory pools can be tweaked in eclipse.ini. I have 16Gb of RAM, but these settings will work fine with 8Gb of RAM.

use the JDK stored on the RAM Disk (use the version according to what you did in Step 3):

-vm

/Volumes/JDKRAMDISK/jdk1.x.y_z.jdk/Contents/Home/

-vm

N:/jdk1.x.y_z/bin

disable bytecode verification (risky)

-Xverify:none

This basically skips the verification of class files (described in http://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.10), meaning that the JVM won’t be able to detect the authenticity of the class files you are using. This poses a security threat if the compiled files have been altered. Use at your own risk. (I use that option for hobby projects but not at work)

turn on performance compiler optimizations

-XX:+AggressiveOpts

increase permanent generation space (where new objects are allocated)

-XX:PermSize=512m

-XX:MaxPermSize=512m

increase min and max heap sizes (which includes young and tenured generations)

-Xms2048m

-Xmx2048m

increase heap size for the young generation

-Xmn512m

set stack size for each thread

-Xss2m

tweak garbage collection

-XX:+UseParallelOldGC

And finally, here are some other options that you may see online… Personally I didn’t notice much of a boost by using any of them so I’m just stating them for info. You can look at the doc reference to know what they correspond to and then play with them:

-XX:MaxGCPauseMillis=10

-XX:+UseG1GC

-XX:CompileThreshold=5

-XX:MaxGCPauseMillis=10

-XX:MaxHeapFreeRatio=70

-XX:+CMSIncrementalPacing

-XX:+UseFastAccessorMethods

-server

Finally, remove all the duplicates as well as launcher.XXMaxPermSize which is useless with the option XX:MaxPermSize.

Tip 5: Get your anti-virus outta here

If you have an anti-virus, make sure that it doesn’t interfere with your code folders. Add the JDK folder, the Eclipse folder, your .m2/jar repositories, and your code folders to the antivirus whitelist.

Tip 6: Keep SVN and GIT out of Eclipse

This is really a matter of personal preference. Some will like the Eclipse integration with the collaborative team tools. Personnally I find them slow, I’d rather have Eclipse concentrate on the dev task than try to do too many things at once… I’m also a sucker for the SVN/GIT CLIs… Anyways,  I removed the integration with Eclipse and found it much more responsive.

Tip 7: Use your keyboard

One of the perks of Eclipse is the trumendous amount of keyboard shortcuts it offers. You can remap them if you are used to other schemes. I usually remap my debugging keys for ex. so that they match the ones in VIsual Studio &  Chrome Dev Env. Take some time to learn them. The less your hands leave your keyboard, the faster your Eclipse experience will be!

I won’t go into details about which shortcuts to know but you can easily find that information online. Just thinking about it, a few must-know shortcuts come to mind:

Ctrl+Shift+R : jump to resource

Ctrl+Shift+T : jump to class

Ctrl+. : jump to next error

Ctrl+Shift+G : search for references

Ctrl+Shift+P : select matching bracket

Alt+Arrows : go forward / backwards

Ctrl+Space : autocomplete

Ctrl+Shift+F : format source

Ctrl+Shift+O : organize imports

Ctrl+D : delete line

etc.

This pretty much wraps it up! Without comparing Eclipse to any other tool out there, I find that it can be a very powerful and fast tool to write java code with.

If you have any other tips, I’d love to hear about them!

 

Specifying Tomcat ports in IntelliJ IDEA to use multiple tomcat servers

Source from: http://www.altuure.com/2008/11/12/tomcat-port-configuration/

Edit the server.xml:

<Server port="${tomcat.init.port}1" shutdown="SHUTDOWN">

<Service name="Catalina">
<Connector port="${tomcat.init.port}2" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="${tomcat.init.port}3"/>
<Connector port="${tomcat.init.port}3" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"/>
<Connector port="${tomcat.init.port}4" protocol="AJP/1.3" redirectPort="${tomcat.init.port}3"/>

</Service>
</Server>

Run it like this:

CATALINA_OPTS=-Dtomcat.init.port=908
 ./startup.sh

so you will have

SHUTDOWN:9081
 HTTP:9082
 HTTPS:9083
 AJP:9084