2018 Concurrency Training Note
2018 Concurrency Training Node
- Hotspot Compiler Brief
- Java Synchronization: Keyword Synchronize
- Java Synchronization: The Lock
- Java Synchronization: The Reader
- Volatile
Popular posts from this blog
Spark 1.6: The table metadata problem in Spark 1.6
Problem Description In Jan 2018, this is what I was trying to do with Spark 1.6 There is an external Hive ORC table, partitioned on region code and country code. I read the whole table to a Spark dataFrame. I took some partitions, let say regionCode=01 and countryCode=01, make change to it to make it regionCode=02 and countryCode=02 I wrote the new change into a tem file. A new change means all records which used to have regionCode=01 and countryCode=01 and now have regionCode=02 and countryCode=02. The write finished successfully and verified after writting. Then I deleted the old partition regionCode=01 and countryCode=01. By "delete", I mean call Hive ALTER TABLE DROP PARTITION(...) and deleted files and folders of that partition in HDFS. Now I tried to read the tem file and write it back to Hive table using dataFrame.write(). Error! Saying regionCode=01 and countryCode=01 folder was not found. Yes because I removed it from HDFS. But I did als...
Comments
Post a Comment