Dudes Study Note

Problem Spark 2.2 Count Table Multiple Time returns the same value regardless of table actual state Description There is table A, initially have 100 records Loop: (1) int countBefore = Spark table A count (2) //Remove 2 records from table A (3) int countAfter = Spark table A count * Note that (1) is tried with both session.sql(select count(*)) and session.sql(select *).count() Behavior 1st iteration: countBefore = 100, countAfter = 98 2nd iteration: countBefore = 100, countAfter = 96 Explanation and Solution Look like the count is cached for optimization, although the variable countBefore declared inside the loop iteration. The scope concept does not hold. Solution is session.catalog.clearCache() in the beginning of any iteration. The  other attempt using session.catalog.refreshTable(tableA) does not solve the problem.

 Problem Description In Jan 2018, this is what I was trying to do with Spark 1.6 There is an external Hive ORC table, partitioned on region code and country code. I read the whole table to a Spark dataFrame.  I took some partitions, let say regionCode=01 and countryCode=01, make change to it to make it regionCode=02 and countryCode=02 I wrote the new change into a tem file. A new change means all records which used to have regionCode=01 and countryCode=01 and now have regionCode=02 and countryCode=02. The write finished successfully and verified after writting. Then I deleted the old partition regionCode=01 and countryCode=01. By "delete", I mean call Hive ALTER TABLE DROP PARTITION(...) and deleted files and folders of that partition in HDFS. Now I tried to read the tem file and write it back to Hive table using dataFrame.write().  Error! Saying regionCode=01 and countryCode=01 folder was not found.  Yes because I removed it from HDFS. But I did als...