[jbosstools-issues] [JBoss JIRA] (ERT-550) Benchmark old and new JDT Indexers

[Roland Grunberg (JIRA)]

    [ https://issues.jboss.org/browse/ERT-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494887#comment-13494887 ] 

Roland Grunberg edited comment on ERT-550 at 11/28/17 1:18 PM:
---------------------------------------------------------------

Commented on jdt-core-dev asking for some clarification and it would seem that the new indexer is not intended to outperform the old one "across the board".
I am finding it difficult to produce cases where the new one outperforms the old one so I may have to keep trying later on.

Methadology :
See https://github.com/rgrunber/eclipse.jdt.core/tree/index_performance (should be top-most commit)

We take a set of jars, and split them (roughly) evenly among the number of projects to exist in the workspace. When then perform a type hierarchy search on java.lang.Object from one of the projects. The results are calculated from an average of 5 runs.

There's also testing of the actual time it takes to build the indexes but these seem pretty consistent across all scenarios. The new indexer is usually 2x as large and takes 3-4x as long to build.


Results so far for search indexing :

1 workspace project / 1500 jars / 43000 classes
Old : 5s
New : 22s

1 workspace project / 1 jar / 18000 classes
Old : 1s
New : 3s

1 workspace project / 1 jar / 24000 classes
Old : 2s
New : 7s


1 workspace project / 1500 jars / 75000 classes
Old : 7s
New : 34s

10 workspace project / 1500 jars / 75000 classes
Old : 4s
New : 7s

100 workspace project / 1500 jars / 75000 classes
Old : 3s
New : 6s

1 workspace project / 4000 jars / 120000 classes
Old : 44s
New : 320s

10 workspace project / 4000 jars / 120000 classes
Old : 9s
New : 15s

100 workspace project / 4000 jars / 120000 classes
Old : 8s
New : 12s

The results do seem to indicate that the new indexer works much faster when the number of projects increases, but its improvements never seem to do better than the old indexer.

Notes / Definitions :
jars - referring to .jar files, so N jars is exactly that many jars on the filesystem
classes - this is the number of classes detected by the indexer, and doesn't correspond to total number of unique classes of all jars


was (Author: rgrunber):
Commented on jdt-core-dev asking for some clarification and it would seem that the new indexer is not intended to outperform the old one "across the board".
I am finding it difficult to produce cases where the new one outperforms the old one so I may have to keep trying later on.

Methadology :
See https://github.com/rgrunber/eclipse.jdt.core/tree/index_performance (should be top-most commit)

We take a set of jars, and split them (roughly) evenly among the number of projects to exist in the workspace. When then perform a type hierarchy search on java.lang.Object from one of the projects. The results are calculated from an average of 10 runs.

There's also testing of the actual time it takes to build the indexes but these seem pretty consistent across all scenarios. The new indexer is usually 2x as large and takes 3-4x as long to build.


Results so far for search indexing :

1 workspace project / 1500 jars / 43000 classes
Old : 5s
New : 22s

1 workspace project / 1 jar / 18000 classes
Old : 1s
New : 3s

1 workspace project / 1 jar / 24000 classes
Old : 2s
New : 7s


1 workspace project / 1500 jars / 75000 classes
Old : 7s
New : 34s

10 workspace project / 1500 jars / 75000 classes
Old : 4s
New : 7s

100 workspace project / 1500 jars / 75000 classes
Old : 3s
New : 6s

1 workspace project / 4000 jars / 120000 classes
Old : 44s
New : 320s

10 workspace project / 4000 jars / 120000 classes
Old : 9s
New : 15s

100 workspace project / 4000 jars / 120000 classes
Old : 8s
New : 12s

The results do seem to indicate that the new indexer works much faster when the number of projects increases, but its improvements never seem to do better than the old indexer.

Notes / Definitions :
jars - referring to .jar files, so N jars is exactly that many jars on the filesystem
classes - this is the number of classes detected by the indexer, and doesn't correspond to total number of unique classes of all jars

> Benchmark old and new JDT Indexers
> ----------------------------------
>
>                 Key: ERT-550
>                 URL: https://issues.jboss.org/browse/ERT-550
>             Project: Eclipse Release Train
>          Issue Type: Task
>            Reporter: Roland Grunberg
>            Assignee: Roland Grunberg
>
> Benchmark the old and new JDT Indexers to determine which should be favoured in various conditions.



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)