Friday, September 17, 2010

Is there a silver bullet to improve intellij indexing performance?

Its known that Intellij indexing is I/O intensive. Upgrading to a faster storage device like solid-state device seems to be a silver bullet that solves Intellij indexing performance issues. Here are few reasons why this is not the case and why upgrading to SSD doesnt necessarily solve indexing problems:

Write performance of SSD is quite different from read performance. For random accesses of read operations, SSD performs much better than typical hard-disk. Where as, performance of SSD and hard-disk are almost similar in-case of sequential read operations. Problematic area with SSD is write operations. SSD performs writes at 'erase block' level. It merges write data with data in erase block, basically write operations are at erase block level. Erase blocks will be 1-4MB in size. As a result write performance of SSD is worse compared to typical hard-disks. To make things worse, existing file systems are not optim ized for SSDs. There are few file systems, like ZFS (for solaris) which is optimized for SSD. Here is example to illustrate the effect of write operations on performance of SSD: Intel X25 flash SSD input/output operations per second (IOPS) drops from ~2000 IOPS in pure read mode to ~700 IOPS in 80/20 read/write on 8k blocks. A typical hard-disk can perform ~200 IOPS. (Refer to the following articles for further information related to SSD.)

Just like input/output has been neglected in compute architecture; it has been neglected in profiling world also. Very few tools are available for input/output profiling of a process. These tools are rudimentary and some collect information from /proc/<pid>/io. These tools does not provide any information on number of read operations, write operations performed by a process. Read/write data volume or, read/write rates is not recommended way of measuring I/O performance. Refer to these articles to know why.

If we know the mix of read/write operations performed by Intellij, then we could compute effective performance improvement/degradation because of SSD. As we dont know mix of read/write operations, only way to measure the improvement of SSD is by running build/indexing etc with SSD and without SSD. The amount of time taken to build community edition index remained almost same with and without SSD. Here are index times for community edition on 8GB machine, 64-bit JVM and using Intellij 10.5

Storage Device
Num Files
Scanning Files Time
Indexing Time
HardDisk drive
Index times of Intellij community edition code using SSD, normal hard-disk. (Avg of 10 runs)

To summarize, SSD will be a silver bullet that solves intellij indexing problems only if the underlying filesystem is optimized for SSDs (like ZFS of solaris), number read operations performed is lot more than number of write operations. Till then, we need to find alternate solutions to fix intellij indexing performance issues.

Some more observations and notes related I/O profiling:

/proc/<pid>/io contains information about I/O done by the process. Here is amount of I/O performed by Intellij (during rebuild of Intellij community edition): (Refer to proc help to know the terminology used in io file)

rchar: 4263207814
wchar: 1363709925
syscr: 4788271
syscw: 1342145
read_bytes: 1504063488
write_bytes: 3310067712
cancelled_write_bytes: 405561344

read_bytes represents number of bytes this process really caused to be read from storage (~1.5GB). write_bytes represents number of bytes this process really caused to be written to storage (~3 GB).
wchar - The number of bytes which this process has caused to be written is less than write_bytes -

I/O Profiling Tools:
collectl can be used to monitor I/O on a system. This has capabilities similar to top command.

1 comment:

  1. My guess is there is very little use of the hdd apart from the initial read. 4 GB of RAM is probably more than enough to store the intelliJ binary and any runtime files(indexes) that it needs. The only other writes will be done async in the background to maintain consistency between the persistent store and your RAM.

    If that's not the case, you can always use tmpfs to force the OS to store your intelliJ files in the RAM always.