hadoop - When to use Hcatalog and what are its benefits -


i'm new hcatlog (hcat), know in usecases/scenario's use hcat, benefits of making use of hcat, there performance improvement can gain hcatlog. can 1 provide information on when use hcatlog

apache hcatalog table , storage management layer hadoop enables users different data processing tools – apache pig, apache map/reduce, , apache hive – more read , write data on grid.

hcatalog creates table abstraction layer on data stored on hdfs cluster. table abstraction layer presents data in familiar relational format , makes easier read , write data using familiar query language concepts.

hcatalog data structures defined using hive's data definition language (ddl) , hive metastore stores hcatalog data structures. using command-line interface (cli), users can create, alter, , drop tables. tables organized databases or placed in default database if none defined table. once tables created, can explore metadata of tables using commands such show table , describe table. hcatalog commands same hive's ddl commands.

hcatalog’s ensures users need not worry or in format data stored. hcatalog displays data rcfile format, text files, or sequence files in tabular view. provides rest apis external systems can access these tables’ metadata.

hcatalog opens hive metadata other map/reduce tools. every map/reduce tools has own notion hdfs data (example pig sees hdfs data set of files, hive sees tables) hcatalog supported map/reduce tools not need care data stored, in format , storage location.

  1. it assist integration other tools , supplies read , write interfaces pig, hive , map/reduce.
  2. it provide shared schema , data types hadoop tools.you not have explicitly type data structures in each program.
  3. it expose information rest interface external data access.
  4. it integrates sqoop, tool designed transfer data , forth between hadoop , relational databases such sql server , oracle
  5. it provide apis , webservice wrapper accessing metadata in hive metastore.
  6. hcatalog exposes rest interface can create custom tools , applications interact hadoop data structures.

this allows use right tool right job. example, can load data hadoop using hcatalog, perform etl on data using pig, , aggregate data using hive. after processing, send data data warehouse housed in sql server using sqoop. can automate process using oozie.

how works:

  1. pig- hcatloader , hcatstore interface
  2. map/reduce- hcatinputformat , hcatoutputformat interface
  3. hive- no interface necessary. direct access metadata

references:

microsoft big data solution

http://hortonworks.com/hadoop/hcatalog/

answer question:

as described earlier hcatalog provides shared schema , data types hadoop tools simplifies work during data processing. if have created table using hcatalog, can directly access hive table through pig or map/reduce (you cannot access hive table through pig or map reduce).you don't need create schema every tool.

if working shared data can used multiple users(some team using hive, team using pig, team using map/reduce) hcatalog useful need table access data processing.

it not replacement of tool facility provide single access many tools.

performance depends on hadoop cluster. should performance benchmarking in hadoop cluster major performance.


Comments

Post a Comment

Popular posts from this blog

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more

How to access a php class file from PHPFox framework into javascript code written in simple HTML file? -

i've website developed using phpfox v3.0.7 . now i'm trying implement 'sse(server sent events)' 1 of php files. reference i'm giving below necessary code part of file. file titled process.class.php , present @ location /var/www/module/notification/include/service/process.class.php present on server having ip http://34.124.40.142/ . <?php /** * [phpfox_header] */ header('content-type: text/event-stream'); header('cache-control: no-cache'); defined('phpfox') or exit('no dice!'); /** * * * @copyright [phpfox_copyright] * @author raymond benc * @package phpfox_service * @version $id: service.class.php 67 2009-01-20 11:32:45z raymond_benc $ */ class notification_service_process extends phpfox_service { /** * class constructor */ public function __construct() { $this->_stable = phpfox::gett('notification'); } public function add($...
Read more

java - Null response to php query in android, even though php works properly -

Image
table i'm trying retrieve above data using 2 php files below. want pass email parameter , return bets user = email. work fine when tested advanced rest client, problem not php. however, when try through android 2 java classes below, null pointer exception (logcat below) , json object returned null. i'm not sure error is, think i'm not passing parameter php , that's why it's producing null response. logcat 05-31 15:10:50.227 26074-26074/com.example.albert.betterapp e/androidruntime﹕ fatal exception: main process: com.example.albert.betterapp, pid: 26074 java.lang.runtimeexception: unable start activity componentinfo{com.example.albert.betterapp/com.example.albert.betterapp.displayallbets}: java.lang.nullpointerexception @ android.app.activitythread.performlaunchactivity(activitythread.java:2292) @ android.app.activitythread.handlelaunchactivity(activitythread.java:2350) @ android.app.activitythread.acce...
Read more