Chalst: as far as I know that is one of the objections many people have to working in haskell, or any language with a particularly "high level" semantic model sufficiently divorced from machine-parts. A correct and performant implementation of the language requires a large and complex runtime, often with a heavy set of automatic services, auxiliary data structures, and nontrivial compiler activity. This forces the programmer to give up a degree of control and predictability, and sets up a general performance tax / performance ceiling for the whole program.

It's rather the same objection asm programmers make when choosing against C or C++. The comparison extends, in fact, to the counter-arguments made by the high-level system defenders: that the C compiler (or JVM runtime as the case may be) is capable of automatic optimizations far beyond the "intention and control" of the lower-level hacker.

Strangely, media codecs and arithmetic libraries still get some of their cores written in asm, and OS kernels, graphics libraries, network stacks, servers, games and desktop applications still get written in C. I think a bit of the "automatic-optimization better than any human" story is overreaching, doesn't happen as often as the defenders wish, or enough to make up the difference for the systemic taxes.

The OP's notion that he'll someday be able to "choose" between LLVM and a JVM as backend is meaningless, alas, apples-to-oranges. LLVM is a lower-level component (compiler backend); you could implement a JVM using LLVM, but the complexity of a JVM comes from the abstract semantics required by the Java language spec (which includes a VM spec), not any particular implementation substrate.