Thursday, 23 June 2016

Bigdata Eco Systems 00 - Hadoop Eco System EDW



Hadoop Eco System  EDW 




Data Analysis 01 - Hive , Spark and impala -Hive MetaStore




Hive Metastore 

Data Analysis(CCA175)

Use Data Definition Language (DDL) to create tables in the Hive metastore for use by Hive and Impala.

Hive Meta-store Architecture.

 

 

 

 

 

 

Multiple Tools are used to process data once data is in place.

1. Hive ,  2. Spark(Hive ) and 3. impala(Cloudera product) are used to process same data and DB structure.
To check list of components 
hadoop fs -ls /user/cloudera

> hive (Check  hive)
> mysql -u root -p (check mysql)

Metastore Conf details:
core-site.xml conatins name node info:
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://quickstart.cloudera:8020</value>
  </property>

> cd /etc/hive/conf (Press Enter)
> view hive-site.xml
important items
1) metastore-warehouse-dir (Default location hive reads data)
<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
  </property>
2) Hive uses metastore as mysql( Configuration of mysql as metadat serevr you can folow below blog).

http://apachehive.blogspot.in/

CHD5.6 show below metastore info:


 Hive and Create Table Internals:

1) hive> create table cca_demo(t int, s string);
2) insert data into table 
insert into table cca_demo values(1,'testing')
3) check file created in hive warehouse dir
hadoop fs -ls /user/hive/warehouse/cca_demo/00000_0
4)  Verify the content of file
hadoop fs -cat /user/hive/warehouse/cca_demo/00000_0
Result: Record rows display
5) Verify same data in hive tables.
hive> select * from cca_demo;
6) check metastore entries in mysql tables(TBLS)
select * from TBLS;
Result: a table entry should be exists.
7) check description of table in hive.
describe formatted cca_demo;
Result: display schema information details of table.
-----------------------------------------------------------------

Spark-shell
start  > spark-shel -- master local 

> show table;
Result: list cca_demo table

impala-shell
>impala-shell
impala> show tables
 Result: List cca_demo table 

* same data should display ussing al tools.