天天看點

[HBase]abort引發的死鎖

今天在HBase群裡碰到一個用戶端死鎖問題,堆棧如下

Java stack information for the threads listed above:
===================================================
"KVQueueService-Handler-2-EventThread":
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
        - waiting to lock <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:605)
        - locked <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
        at org.apache.hadoop.hbase.zookee
per.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
"TableDataQueueService-Handler-4":
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:600)
        - waiting to lock <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
        - locked <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
        at org.
apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:830)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:942)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.j
ava:942)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1520)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1376)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:936)
        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:792)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:775)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
        at com.alibaba.alimonitor.store.hbase.TableDataDaoService.writeData(TableDataDaoService.java:111)
        at com.alibaba.alimonitor.store.service.Tab
leDataQueueService.handle(TableDataQueueService.java:28)
        at com.alibaba.alimonitor.store.service.TableDataQueueService.handle(TableDataQueueService.java:14)
        at com.alibaba.alimonitor.store.service.AbstractQueueService$HandleThread.run(AbstractQueueService.java:115)

Found 1 deadlock.
           

 可見發生死鎖的業務線程和zookeeper的eventThread線程。EventThread是zookeeper用來通知用戶端watcher的線程,它會回調使用者注冊的watcher。

在這裡,首先是eventThread處理connectionEvent的Expired事件

case Expired:
        if (ZKUtil.isSecureZooKeeper(this.conf)) {
          // We consider Expired equivalent to AuthFailed for this
          // connection. Authentication is never going to complete. The
          // client should proceed to do cleanup.
          saslLatch.countDown();
        }
        String msg = prefix(this.identifier + " received expired from " +
          "ZooKeeper, aborting");
        // TODO: One thought is to add call to ZooKeeperListener so say,
        // ZooKeeperNodeTracker can zero out its data values.
        if (this.abortable != null) this.abortable.abort(msg,
            new KeeperException.SessionExpiredException());
        break;
    }
           

 這裡會abort調用,其實是調用的HConnectionImplementation的abort方法

public void abort(final String msg, Throwable t) {
      if (t instanceof KeeperException) {
        LOG.info("This client just lost it's session with ZooKeeper, will"
            + " automatically reconnect when needed.");
        if (t instanceof KeeperException.SessionExpiredException) {
          LOG.info("ZK session expired. This disconnect could have been" +
              " caused by a network partition or a long-running GC pause," +
              " either way it's recommended that you verify your environment.");
          resetZooKeeperTrackers();
        }
        return;
      }
      if (t != null) LOG.fatal(msg, t);
      else LOG.fatal(msg);
      this.aborted = true;
      close();
    }
           

 再看resetZooKeeperTrackers方法,是一個同步方法,EventThread在這裡拿到HConnectionImplementation對象鎖

private synchronized void resetZooKeeperTrackers() {
      if (masterAddressTracker != null) {
        masterAddressTracker.stop();
        masterAddressTracker = null;
      }
      if (rootRegionTracker != null) {
        rootRegionTracker.stop();
        rootRegionTracker = null;
      }
      clusterId = null;
      if (zooKeeper != null) {
        zooKeeper.close();
        zooKeeper = null;
      }
    }
           

 這個時候業務線程開始region location,進入RootRegionTrack的waitRootRegionLocation

public ServerName waitRootRegionLocation(long timeout)
  throws InterruptedException {
    if (false == checkIfBaseNodeAvailable()) {
      String errorMsg = "Check the value configured in 'zookeeper.znode.parent'. "
          + "There could be a mismatch with the one configured in the master.";
      LOG.error(errorMsg);
      throw new IllegalArgumentException(errorMsg);
    }
    return dataToServerName(super.blockUntilAvailable(timeout, true));
  }
           

 業務線程進入blockUntilAvailable,拿到父類ZookeeperNodeTracker的對象鎖

public synchronized byte [] blockUntilAvailable(long timeout, boolean refresh)
  throws InterruptedException {
    if (timeout < 0) throw new IllegalArgumentException();
    boolean notimeout = timeout == 0;
    long startTime = System.currentTimeMillis();
    long remaining = timeout;
    if (refresh) {
      try {
	//這裡正好抛出異常
        this.data = ZKUtil.getDataAndWatch(watcher, node);
      } catch(KeeperException e) {
        abortable.abort("Unexpected exception handling blockUntilAvailable", e);
      }
    }
    while (!this.stopped && (notimeout || remaining > 0) && this.data == null) {
      // We expect a notification; but we wait with a
      //  a timeout to lower the impact of a race condition if any
      wait(100);
      remaining = timeout - (System.currentTimeMillis() - startTime);
    }
    return this.data;
  }
           

 由于這裡zookeeper調用正好抛出異常導緻業務線程去abort,這個時候因為EventThread已經先abort了,是以業務線程需要等待HConnectionImplementation對象鎖的釋放。而這個時候EventThread繼續執行,resetZooKeeperTrackers中繼續調用 rootRegionTracker.stop()方法

public synchronized void stop() {
    this.stopped = true;
    notifyAll();
  }
           

 這個方法需要拿到父類ZookeeperNodeTracker的對象鎖,而此時這個鎖已經被業務線程拿到了,這就發生了死鎖。根本原因abort操作沒有做并發控制。最新版本的hbase已經修複了這個bug,修複後的abort方法,增加了resetting同步狀态,後續abort請求直接return了。

public void abort(final String msg, Throwable t) {
      if (t instanceof KeeperException) {
        LOG.info("This client just lost it's session with ZooKeeper, will"
            + " automatically reconnect when needed.");
        if (t instanceof KeeperException.SessionExpiredException) {
          LOG.info("ZK session expired. This disconnect could have been" +
              " caused by a network partition or a long-running GC pause," +
              " either way it's recommended that you verify your environment.");
          //同步狀态
          synchronized (resetLock) {
            if (resetting) return;
            this.resetting = true;
          }
          resetZooKeeperTrackers();
          this.resetting = false;
        }
        return;
      }
      if (t != null) LOG.fatal(msg, t);
      else LOG.fatal(msg);
      this.aborted = true;
      close();
    }
           

 官方issue位址 https://issues.apache.org/jira/browse/HBASE-7259

繼續閱讀