[teiid-issues] [JBoss JIRA] (TEIID-4208) Increase plan parallelism

[Steven Hawkins (JIRA)]

    [ https://issues.jboss.org/browse/TEIID-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471689#comment-13471689 ] 

Steven Hawkins commented on TEIID-4208:
---------------------------------------

Taking another look at this we have:

1. Algorithmic parallelism - namely parallelSort in the sortutility. This has no effect for small row counts < 8k - which can also occur in memory constrained situations.  It also has a significant performance degradation the more the input is in reverse order.  But it is straight-forward to include.  We could consider a hint or a system flag to disable if needed.
2. Targeted multi-threading / blocking - adding blocking tasks to SortUtility and/or temp/system tuplesources (similar to datatiertuplesource).  This allows those to proceed while other plan work in the event of a union/join continues.  However just as before what I see is of limited benefit performance-wise and introduces performance degradation with increased client load - even just 2 concurrent user queries.  It looks like this should only be introduced in parallel plans and only for large temp table access or window function project nodes (both of which have non-blocking sortutility usage).
3. Increased parallelism at branching points, such as unions and non-dependent joins.  This actually results in real performance gains, but as mentioned before requires more thread safety for things like the CommandContext.

I have the changes for #1 ready and will focus on #3 at least for unions.

> Increase plan parallelism
> -------------------------
>
>                 Key: TEIID-4208
>                 URL: https://issues.jboss.org/browse/TEIID-4208
>             Project: Teiid
>          Issue Type: Sub-task
>          Components: Query Engine
>            Reporter: Steven Hawkins
>            Assignee: Steven Hawkins
>             Fix For: 10.0
>
>
> Increased parallelism within the plan, which includes parallelization of source queries via partitioning beyond multi-source and parallelizing the execution of union branches.



--
This message was sent by Atlassian JIRA
(v7.2.3#72005)