Spark 2.2 Count Table Multiple Time Issue

-
Problem
Spark 2.2 Count Table Multiple Time returns the same value regardless of table actual state

Description
There is table A, initially have 100 records

Loop:
(1) int countBefore = Spark table A count
(2) //Remove 2 records from table A
(3) int countAfter = Spark table A count

* Note that (1) is tried with both session.sql(select count(*)) and session.sql(select *).count()

Behavior
1st iteration: countBefore = 100, countAfter = 98
2nd iteration: countBefore = 100, countAfter = 96

Explanation and Solution
Look like the count is cached for optimization, although the variable countBefore declared inside the loop iteration. The scope concept does not hold.
Solution is session.catalog.clearCache() in the beginning of any iteration.
The  other attempt using session.catalog.refreshTable(tableA) does not solve the problem.

Comments

Post a Comment

Popular posts from this blog

test

-
import com.azure.identity.DefaultAzureCredential; import com.azure.identity.DefaultAzureCredentialBuilder; import com.azure.core.credential.TokenCredential; import com.azure.core.util.Configuration; public class AzureAccessTokenExample {     public static void main(String[] args) {         try {             // Load Azure App Configuration             Configuration configuration = Configuration.getGlobalConfiguration();             // Set the Azure AD authority host (optional, can be omitted for default)             String azureAuthorityHost = configuration.get("AZURE_AUTHORITY_HOST");                          // Create a DefaultAzureCredential using the Azure Identity library             TokenCredential credential = new DefaultAzureCredentialBuilder()                     .authorityHost(azureAuthorityHost) // Optional: Set the authority host                     .build();             // Specify the scope (optional, can be omitted for default)             String scope = "ht
Read more

Java Concurrency: Final

-
Image
What does it mean by FINAL? (Definition below only applicable to JDK6+) 1) Address of enclosing object not allowed to escape util final variable for initialized and change made to memory done 2) JVM can execute special CPU caching instruction 3) All change in constructor to final variable, even reference will be pushed to memory
Read more

Java Object Size and Overhead

-
Let's look at the details few details of object header and calculate the memory size an object occupies inside JVM Heap. Each Object contains following information. • The Object Header. • The memory for primitive types. 
 • The memory for reference types. 
 • Offset / alignment -  in fact, these are a few unused bytes that are placed after the data object itself. This is done in order that an address in memory was always a multiple of machine word, to speed up the memory read + reduce the number of bits for a pointer to an object. It is also worth noting that in java a size of any object is multiple of 8 bytes! •   Object Header :             In case of 32-bit system, the header size is 8 bytes, in the case of 64-bit system, respectively is 16 bytes. It contains following information. 1   Hash Code - 2   Garbage Collection Information  -  each java object contains the information needed for the memory management. 3   Type Information Block Pointe
Read more