Posts Tagged bigdata

(fixed) sqoop import from Oracle to Hive throw Heap size error

If you use CDH and Sqoop. you probably met following issue:

15/08/07 15:21:55 INFO manager.SqlManager: Using default fetchSize of 1000
15/08/07 15:21:55 INFO tool.CodeGenTool: Beginning code generation
15/08/07 15:21:56 INFO manager.OracleManager: Time zone has been set to GMT
15/08/07 15:21:56 INFO manager.SqlManager: Executing SQL statement: select xxx where (1 = 0)
Exception in thread “main” java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Array.newArray(Native Method)
at java.lang.reflect.Array.newInstance(Array.java:70)
at oracle.jdbc.driver.BufferCache.get(BufferCache.java:226)
at oracle.jdbc.driver.PhysicalConnection.getCharBuffer(PhysicalConnection.java:7698)
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:1013)
at oracle.jdbc.driver.T4CTTIdcb.receiveCommon(T4CTTIdcb.java:277)

This related with HDFS client HEAP settings. you can fix it by increase HEAP size for HDFS client.

from CDH:
Client Java Heap Size in Bytes.
Read the rest of this entry »

No Comments

Hive Issue: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

2015-06-25 11:16:52,554 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(623)) – 0: get_table : db=lntest tbl=tmp_download
2015-06-25 11:16:52,554 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) – ugi=abcuser ip=unknown-ip-addr cmd=get_table : db=lntest tbl=
tmp_download
2015-06-25 11:16:52,555 ERROR [main]: bonecp.ConnectionHandle (ConnectionHandle.java:markPossiblyBroken(388)) – Database access problem. Killing off this connection and all r
emaining connections in the connection pool. SQL State = 08S01
2015-06-25 11:16:52,557 ERROR [main]: bonecp.BoneCP (BoneCP.java:destroyConnection(221)) – Error in attempting to close connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
at sun.reflect.GeneratedConstructorAccessor114.newInstance(Unknown Source)

Problem related with MySQL for Hive settings:
mysql> show global variables like ‘%time%’ ;
Read the rest of this entry »

No Comments