But the execution order of @NotNull, @Size and @CheckCase annotations can not be defined

That's correct. Currently there is no way in Java to define order of annotations on the same level (like here on the @interface type). The Java reflection API does not guarantee that annotations are returned in any specific order. To create any order we would need a solution for HV-706.

and sometimes it breaks validation by throwing exception

I am not quite sure what you mean with that. Can you explain this a bit more? Which exception(s) are you talking about?

Child constraints can be sorted by groups such top level constraints.

What do you mean? Do you want to use groups to create some sort of order? How would that look like?

Note, the Bean Validation is also very clear regarding groups and composed constraints:

Groups from the main constraint annotation are inherited by the composing annotations. Any groups definition on a composing annotation is ignored.

So unless you are having some concrete plan in mind, I would consider this a duplicate of HV-706. WDYT?