spark 连接 Kerberos hbase 使用小记
使用spark连接Kerberos认证的hbase
背景：使用一个spark集群(集群上有hbase 等大数据组件)，去连接另一个集群上的 Kerberos 认证的 hbase 集群。
改写mlsql 连接hbase的项目。使用 yarn-client 模式。将 krb5.conf 和 wc1-ods.keytab 文件分发到所有集群节点的同一路径下。 使用spark newAPIHadoopRDD的方式去读 最好重写 TableInputFormat 的方法 在里面加入Kerberos认证
报错
20/10/19 16:08:24 ERROR utils.hbase_KerberorsJavaUtil: Get HBaseAuthentication Failed
java.io.IOException: Login failure for ods from keytab /home/yqq/wc1/wc1-ods.keytab: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name [email protected]:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to [email protected]
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1052)
at org.apache.spark.sql.execution.datasources.utils.hbase_KerberorsJavaUtil.getHBaseAuthentication(hbase_KerberorsJavaUtil.java:44)
at org.apache.spark.sql.execution.datasources.hbase.HBaseConfBuilder$.buildKerberos(HBaseConfBuilder.scala:147)
at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(DefaultSource.scala:210)
at org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(DefaultSource.scala:54)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at streaming.core.datasource.impl.MLSQLHbase.load(MLSQLHbase.scala:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at tech.mlsql.dsl.adaptor.LoadPRocessing$$anonfun$parse$2.apply(LoadAdaptor.scala:114)
at tech.mlsql.dsl.adaptor.LoadPRocessing$$anonfun$parse$2.apply(LoadAdaptor.scala:112)
at scala.Option.map(Option.scala:146)
at tech.mlsql.dsl.adaptor.LoadPRocessing.parse(LoadAdaptor.scala:112)
at tech.mlsql.dsl.adaptor.LoadAdaptor.parse(LoadAdaptor.scala:82)
at streaming.dsl.ScriptSQLExecListener.exitSql(ScriptSQLExec.scala:289)
at streaming.dsl.parser.DSLSQLParser$SqlContext.exitRule(DSLSQLParser.java:296)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:47)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:30)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at streaming.dsl.ScriptSQLExec$._parse(ScriptSQLExec.scala:155)
at streaming.dsl.ScriptSQLExec$.parse(ScriptSQLExec.scala:142)
at streaming.rest.RestController$$anonfun$query$1$2.apply$mcV$sp(RestController.scala:140)
at tech.mlsql.job.JobManager$.run(JobManager.scala:73)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name [email protected]: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to [email protected]
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:588)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1043)
... 45 more
Caused by: java.lang.IllegalArgumentException: Illegal principal name [email protected]: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to [email protected]
at org.apache.hadoop.security.User.<init>(User.java:50)
at org.apache.hadoop.security.User.<init>(User.java:43)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:215)
... 57 more
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to [email protected]
at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:400)
at org.apache.hadoop.security.User.<init>(User.java:48)
... 59 more
大数据Kerberos认证报No rules applied to
throw new IOException("Login failure for " + user + " from keytab " + path+ ": " + le, le);
排除 user 和 路径的问题
/**
* Get the translation of the principal name into an operating system
* user name.
* @return the short name
* @throws IOException throws if something is wrong with the rules
*/
public String getShortName() throws IOException {
String[] params;
if (hostName == null) {
// if it is already simple, just return it
if (realm == null) {
return serviceName;
}
params = new String[]{realm, serviceName};
} else {
params = new String[]{realm, serviceName, hostName};
}
for(Rule r: rules) {
String result = r.apply(params);
if (result != null) {
return result;
}
}
throw new NoMatchingRule("No rules applied to " + toString());
}
根据传入的 principal 遍历所有的规则未能返回一个result，getShortName抛出这个错误。在使用时只会连接一个Kerberos 认证的集群。
hconf.set("hadoop.security.auth_to_local", "RULE:[1:$1]\n" +
"RULE:[2:$1]\n" +
"DEFAULT");
//上面是 hadoop.security.auth_to_local 的默认配置规则。使用下面的方法 可以由 传入的 priinciple
val arry = Array[String]("hbase/[email protected]")
HadoopKerberosName.main(arry)
//Name: hbase/[email protected] to hbase
//Name: [email protected] to ods
kerberos主体的hadoop转换规则
Principal可以理解为用户或服务的名字，全集群唯一，由三部分组成：username(or servicename)/[email protected]，例如：nn/[email protected]，zelda1为集群中的一台机器；或admin/[email protected]，管理员账户。
<property>
<name>hadoop.security.auth_to_local</name>
<value>
RULE:[2:$1@$0]([nd][email protected])s/.*/dtdream/
DEFAULT
</value>
</property>
RULE:[1:$1@$0]([email protected])s/.*/hdfs/
RULE:[2:$1@$0]([email protected])s/.*/ods/
ps.
在使用spark standalone 模式(Kerberos认证hbase 同环境的spark集群) 往hdfs写数据时 报过No rules applied to 错误。未配置规则，默认规则也未生效,[email protected] 没有映射到hdfs的用户名 加入规则后能成功映射到hdfs的用户名ods生效。
hbase/[email protected]
hbase:username or servicename 服务名。
_HOST:机器名 instance。
WC1.HBASE.COM:域 realm
$0: 转换的域名 $1表示 第一个组件 ，$2表示用户名中的第二个组件 （没看懂）
//修改规则
hconf.set("hadoop.security.auth_to_local", "RULE:[1:$1@$0]([email protected])s/.*/asianfo/");
如果直接加 ods 会报没有该用户
20/10/20 14:34:45 WARN security.ShellBasedUnixGroupsMapping: unable to return groups for user ods
PartialGroupNameException The user name 'ods' is not found. id: ods: no such user
id: ods: no such user
没报错时的日志
20/10/26 22:47:04 ERROR utils.MyTableInputFormat: get Kerberos realm: null
20/10/26 22:47:04 ERROR utils.MyTableInputFormat: username: ods
20/10/26 22:47:04 ERROR utils.MyTableInputFormat: keytabFile: /home/yqq/wc1/wc1-ods.keytab
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: ------Start Get HBaseAuthentication-----
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: master_principal: hbase/[email protected]
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: regionserver_principal: hbase/[email protected]
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: ------dev_yx.keytab path is---/home/yqq/wc1/wc1-ods.keytab
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: get Kerberos realm: null
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: System get Kerberos realm: WC1.HBASE.COM
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: System get Kerberos kdc: wc1.server.ambari
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: ===========loginUserFromKeytab username ods
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: ------HadoopKerberosName.getRules-----RULE:[1:$1@$0]([email protected])s/.*/asianfo/
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: UserGroupInformation.getLoginUser0: yqq (auth:SIMPLE)
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: UserGroupInformation.getCurrentUser0: yqq (auth:SIMPLE)
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: UserGroupInformation.getLoginUser1: [email protected] (auth:KERBEROS)
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: UserGroupInformation.getCurrentUser1: [email protected] (auth:KERBEROS)
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: ------Get HBaseAuthentication Successed-----
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: =====put the logined userinfomation to user====
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: UserGroupInformation.getLoginUser: [email protected] (auth:KERBEROS)
20/10/26 22:47:04 ERROR utils.hbase_KerberorsJavaUtil: UserGroupInformation.getCurrentUser: [email protected] (auth:KERBEROS)
2
Caused by: Can't get Kerberos realm
Can’t get Kerberos realm
没能正常加载conf 文件。
在启动的 spark submit 语句加入
--conf "spark.driver.extraJavaOptions"="-Djava.security.krb5.conf=/home/yqq/wc1/krb5.conf" \
--conf "spark.executor.extraJavaOptions"="-Djava.security.krb5.conf=/home/yqq/wc1/krb5.conf" \
或者在代码里加入
System.setProperty("java.security.krb5.conf","/home/yqq/wc1/krb5.conf")
// //[email protected]
// System.setProperty("java.security.krb5.realm","WC1.HBASE.COM")
//
// System.setProperty("java.security.krb5.kdc","wc1.server.ambari")
sun.security.krb5.Config.refresh();
因为重写了 TableInputFormat 加入了Kerberos认证
System.setProperty("java.security.krb5.conf", "/home/yqq/wc1/krb5.conf");
System.setProperty("java.security.krb5.realm","WC1.HBASE.COM");
System.setProperty("java.security.krb5.kdc","wc1.server.ambari");
LOG.error("get Kerberos realm: "+getProperty("java.security.krb5.realm"));
try {
sun.security.krb5.Config.refresh();
} catch (KrbException e) {
e.printStackTrace();
}
3
20/10/26 22:48:11 WARN hdfs.LeaseRenewer: Failed to renew lease for [DFSClient_NONMAPREDUCE_2029880377_1] for 68 seconds. Will retry shortly ...
java.io.IOException: Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.; Host Details : local host is: "
YHBB01/10.161.75.84"; destination host is: "sta1":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy12.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:590)
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy13.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:942)
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525)
at org.apache.hadoop.ipc.Client.call(Client.java:1448)
... 16 more
网上资料说1 Server端 要求 simple 的验证方式。但是只有 Kerberos 验证的方式。
2 本地搭建Kerberos 客户端删除 提交 spark 任务时需要删掉下面的配置文件。
3 经过排查 在Kerberos认证时在代码里配置的默认 ipc.client.fallback-to-simple-auth-allowed 未生效。(CDH管理的集群)。在hdfs配置文件里重新配置 后无此报错。
4
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=61, exceptions:
Wed Oct 28 14:55:51 CST 2020, null, java.net.SocketTimeoutException: callTimeout=540000, callDuration=680718: row 'td_b_payment_deposit,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740
, hostname=wc1.slave3.ambari,16020,1555077306296, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:286)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:231)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:88)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
at org.apache.spark.sql.execution.datasources.utils.MyTableInputFormat.getSplits(MyTableInputFormat.java:252)
at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:130)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
... 72 more
Caused by: java.net.SocketTimeoutException: callTimeout=540000, callDuration=680718: row 'td_b_payment_deposit,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=wc1.slave3.amb
ari,16020,1555077306296, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on wc1.slave3.ambari,16020,1567683889060
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3273)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3250)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2964)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3289)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:402)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:203)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:381)
报错猜测是 连接hbase的meta表失败 region获取失败
NotServingRegionException 异常猜测是 region正在分裂；region数据有损坏。报错多指向是hbase的问题。
后查看spark executors 日志 。因为访问的是另一个集群的hbase，hosts文件只在本机增加了，在spark集群的executer 节点 telnet hbase地址端口，重连报连接被拒绝，没有报hostname not found ，疑惑，怀疑是其他节点的hosts文件未修改导致。修改hosts文件没有权限 该问题未解决。
修改 hosts 文件。 HBase 设置
5
在本地跑测试代码 和在节点使用local模式 验证 可以通过Kerberos认证。在使用 yarn client时报错。查看driver端日志 Kerberos 认证通过。executor 端报错。spark ui界面 stage 看到报错日志
20/11/06 10:11:59 ERROR utils.MyTableInputFormat_ugi: ------Start MyTableInputFortmatUGI-----
20/11/06 10:11:59 ERROR utils.MyTableInputFormat_ugi: get Kerberos realm: WC1.HBASE.COM
20/11/06 10:11:59 ERROR utils.MyTableInputFormat_ugi: get Kerberos realm: /data/v01/wc1/krb5.conf
20/11/06 10:11:59 ERROR utils.MyTableInputFormat_ugi: HBase client scan came from : file:/data/v01/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/jars/hbase-client-1.2.0-cdh5.10.0.jar!/org/apache/hadoop/hbase/client/Scan.class
20/11/06 10:11:59 ERROR utils.MyTableInputFormat_ugi: username: ods
20/11/06 10:11:59 ERROR utils.MyTableInputFormat_ugi: keytabFile: /data/v01/wc1/wc1-ods.keytab
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: =====put the logined userinfomationUGI to user====
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: ------Start Get HBaseAuthenticationUGI-----
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: master_principal: hbase/[email protected]
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: regionserver_principal: hbase/[email protected]
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: ------dev_yx.keytab path is---/data/v01/wc1/krb5.conf
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: get Kerberos realm: null
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: System get Kerberos realm: WC1.HBASE.COM
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: System get Kerberos kdc: wc1.server.ambari
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: ===========loginUserFromKeytab username ods
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: ------HadoopKerberosName.getRules-----RULE:[1:$1@$0]([email protected])s/.*/yqq/
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: UserGroupInformation.getLoginUser0: yarn (auth:SIMPLE)
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: UserGroupInformation.getCurrentUser0: yqq (auth:SIMPLE)
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: /data/v01/wc1/wc1-ods.keytab file:false
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: /data/v01/wc1/wc1-ods.keytab canRead file:false
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: Get HBaseAuthentication Failed
java.io.IOException: Login failure for ods from keytab /data/v01/wc1/wc1-ods.keytab
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1261)
... 32 more
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: UserGroupInformation.getLoginUser: yarn (auth:SIMPLE)
20/11/06 10:11:59 ERROR utils.hbase_KerberorsJavaUtil_ugi: UserGroupInformation.getCurrentUser: yqq (auth:SIMPLE)
20/11/06 10:11:59 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)Lorg/apache/hadoop/hbase/client/Scan;
at org.apache.spark.sql.execution.datasources.utils.MyTableInputFormat_ugi.setConf(MyTableInputFormat_ugi.java:189)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:189)
driver 端日志如下
一 executor Kerberos验证失败
二 executor 端hbase-client jar包冲突
java.net.URL res = MyTableInputFormat.class.getClassLoader().getResource("org/apache/hadoop/hbase/client/Scan.class");
System.out.println("HBase client scan came from " + res.getPath());
LOG.error("HBase client scan came from : "+ res.getPath());
在代码中判断是否可读 keytab文件 和 使用的hbase-client的版本 打印可知。
在使用yarn client时 把当前用户切换成yarn用户 无法访问 keytab 文件报错。executor 端使用的jar包是cdh版本 没有使用上传的 开源版本。
Spark连接需Kerberos认证的HBase
我重写了doAs方法传出connection 在使用yarn-client还是会报错。将keytab文件和jar包赋权为777后执行成功。
在启动命令中加入 解决 hbase-clientjar包问题
--conf "spark.driver.extraClassPath=/data/v01/mlsql_1.6/updown/jar/*" \
--conf "spark.executor.extraClassPath=/data/v01/mlsql_1.6/updown/jar/*" \
spark程序jar与spark lib jar冲突，加载顺序
spark on yarn运行产生jar包冲突问题
spark-submit参数说明–on YARN
Spark on Yarn运行时加载的jar包
spark.executor.userClassPathFirst 和spark.driver.userClassPathFirst 参数时会报错 可能是spark 1.3之前可以使用
Spark2中操作HBase的异常：java.lang.NoSuchMethodError:
定位 是哪个jar包冲突导致
Hadoop认证Kerberos–UserGroupInformation.doAs
doAs 不会因为切换用户导致验证失败
spark on yarn模式下扫描带有kerberos的hbase
没有用到 值得参照
改写 UserGroupInformation.doAs 参照
[kerberos下JAVA代码操作hbase的方式（客户端方式，应用程序方式）]
优雅解决Spark Application jar包冲突问题
使用打jar包的方式解决jar包冲突
收藏的链接
Spark部署模式详解（Local，Standlone，Yarn）
有kerberos认证hbase在spark环境下的使用
重写了 TableInputFormat的方法
![【Spark Mllib】K-均值聚类——电影类型K-均值聚类数据特征提取[图]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)