Performance Issue with the Tuple Join

-

Comments

Post a Comment

Popular posts from this blog

Java Hotspot Compiler Brief

-
Image
1)       Other name: JIT Compiler, Dynamic Compiler 2)       How a java program runs in JVM? -           javac: translate java code to byte code -           JVM Interpreter runs instruction by instruction -           In JVM, there is a lightweight profiler detects common block of instructions running (for example loop) and try to optimize the execution ON-FLY, immediate speed up. For example: we run 5*5*5 for 1000 time, it gets faster after some runs. Why do I care?       You may run bench-marking test. The result will come out funny and you will wonder why.
Read more

Spark 2.2 Count Table Multiple Time Issue

-
Problem Spark 2.2 Count Table Multiple Time returns the same value regardless of table actual state Description There is table A, initially have 100 records Loop: (1) int countBefore = Spark table A count (2) //Remove 2 records from table A (3) int countAfter = Spark table A count * Note that (1) is tried with both session.sql(select count(*)) and session.sql(select *).count() Behavior 1st iteration: countBefore = 100, countAfter = 98 2nd iteration: countBefore = 100, countAfter = 96 Explanation and Solution Look like the count is cached for optimization, although the variable countBefore declared inside the loop iteration. The scope concept does not hold. Solution is session.catalog.clearCache() in the beginning of any iteration. The  other attempt using session.catalog.refreshTable(tableA) does not solve the problem.
Read more