重寫 Nacos 服務發現：多個伺服器如何跨命名空間，通路公共服務？

一、問題背景

在開發某個公共應用時，筆者發現該公共應用的資料是所有測試環境（假設存在 dev/dev2/dev3）通用的。

這就意味着隻需部署一個應用，就能滿足所有測試環境的需求；也意味着所有測試環境都需要調用該公共應用，而不同測試環境的應用注冊在不同的 Nacos 命名空間。

二、兩種解決方案

如果所有測試環境都需要調用該公共應用，有兩種可行的方案。第一種，将該公共服務同時注冊到不同的測試環境所對應的命名空間中。

第二種，将公共應用注冊到單獨的命名空間，不同的測試環境能夠跨命名空間通路該應用。

三、詳細的問題解決過程

先行交代筆者的版本号配置。Nacos 用戶端版本号為 NACOS 1.4.1；Java 項目的 Nacos 版本号如下。

最初想法是将該公共應用同時注冊到多個命名空間下。在查找資料的過程中，團隊成員在 GitHub 上發現了一篇類似問題的部落格分享：Registration Center: Can services in different namespaces be called from each other? #1176。

01 注冊多個命名空間

從該部落格中，我們看到其他程式員朋友也遇到了類似的公共服務的需求。在本篇文章中，筆者将進一步分享實作思路以及示例代碼。

說明：以下代碼内容來自使用者 chuntaojun 的分享。

shareNamespace={namespaceId[:group]},{namespaceId[:group]} 
           

@RunWith(SpringRunner.class)
@SpringBootTest(classes = NamingApp.class, properties = {"server.servlet.context-path=/nacos"},
    webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class SelectServiceInShareNamespace_ITCase {

    private NamingService naming1;
    private NamingService naming2;
    @LocalServerPort
    private int port;
    @Before
    public void init() throws Exception{
        NamingBase.prepareServer(port);
        if (naming1 == null) {
            Properties properties = new Properties();
            properties.setProperty(PropertyKeyConst.SERVER_ADDR, "127.0.0.1"+":"+port);
            properties.setProperty(PropertyKeyConst.SHARE_NAMESPACE, "57425802-3058-4507-9a73-3229b9f00a36");
            naming1 = NamingFactory.createNamingService(properties);

            Properties properties2 = new Properties();
            properties2.setProperty(PropertyKeyConst.SERVER_ADDR, "127.0.0.1"+":"+port);
            properties2.setProperty(PropertyKeyConst.NAMESPACE, "57425802-3058-4507-9a73-3229b9f00a36");
            naming2 = NamingFactory.createNamingService(properties2);
        }
        while (true) {
            if (!"UP".equals(naming1.getServerStatus())) {
                Thread.sleep(1000L);
                continue;
            }
            break;
        }
    }

    @Test
    public void testSelectInstanceInShareNamespaceNoGroup() throws NacosException, InterruptedException {
        String service1 = randomDomainName();
        String service2 = randomDomainName();
        naming1.registerInstance(service1, "127.0.0.1", 90);
        naming2.registerInstance(service2, "127.0.0.2", 90);

        Thread.sleep(1000);

        List<Instance> instances = naming1.getAllInstances(service2);
        Assert.assertEquals(1, instances.size());
        Assert.assertEquals(service2, NamingUtils.getServiceName(instances.get(0).getServiceName()));
    }

    @Test
    public void testSelectInstanceInShareNamespaceWithGroup() throws NacosException, InterruptedException {
        String service1 = randomDomainName();
        String service2 = randomDomainName();
        naming2.registerInstance(service1, groupName, "127.0.0.1", 90);
        naming3.registerInstance(service2, "127.0.0.2", 90);

        Thread.sleep(1000);

        List<Instance> instances = naming3.getAllInstances(service1);
        Assert.assertEquals(1, instances.size());
        Assert.assertEquals(service1, NamingUtils.getServiceName(instances.get(0).getServiceName()));
        Assert.assertEquals(groupName, NamingUtils.getServiceName(NamingUtils.getGroupName(instances.get(0).getServiceName())));
    }

}
           

進一步考慮後發現該解決方案可能不太契合目前遇到的問題。公司目前的開發測試環境有很多個，并且不确定以後會不會繼續增加。

如果每增加一個環境，都需要修改一次公共服務的配置，并且重新開機一次公共服務，着實太麻煩了。倒不如反其道而行，讓其他的伺服器實作跨命名空間通路公共服務。

02 跨命名空間通路

針對實際問題查找資料時，我們找到了類似的參考分享《重寫 Nacos 服務發現邏輯動态修改遠端服務IP位址》。

跟着部落格思路看代碼，筆者了解到服務發現的主要相關類是 NacosNamingService, NacosDiscoveryProperties, NacosDiscoveryAutoConfiguration。

然後，筆者将部落格的示例代碼複制過來，試着進行如下調試：

@Slf4j
@Configuration
@ConditionalOnNacosDiscoveryEnabled
@ConditionalOnProperty(
        name = {"spring.profiles.active"},
        havingValue = "dev"
)
@AutoConfigureBefore({NacosDiscoveryClientAutoConfiguration.class})
public class DevEnvironmentNacosDiscoveryClient {

    @Bean
    @ConditionalOnMissingBean
    public NacosDiscoveryProperties nacosProperties() {
        return new DevEnvironmentNacosDiscoveryProperties();
    }

    static class DevEnvironmentNacosDiscoveryProperties extends NacosDiscoveryProperties {

        private NamingService namingService;

        @Override
        public NamingService namingServiceInstance() {
            if (null != this.namingService) {
                return this.namingService;
            } else {
                Properties properties = new Properties();
                properties.put("serverAddr", super.getServerAddr());
                properties.put("namespace", super.getNamespace());
                properties.put("com.alibaba.nacos.naming.log.filename", super.getLogName());
                if (super.getEndpoint().contains(":")) {
                    int index = super.getEndpoint().indexOf(":");
                    properties.put("endpoint", super.getEndpoint().substring(0, index));
                    properties.put("endpointPort", super.getEndpoint().substring(index + 1));
                } else {
                    properties.put("endpoint", super.getEndpoint());
                }

                properties.put("accessKey", super.getAccessKey());
                properties.put("secretKey", super.getSecretKey());
                properties.put("clusterName", super.getClusterName());
                properties.put("namingLoadCacheAtStart", super.getNamingLoadCacheAtStart());

                try {
                    this.namingService = new DevEnvironmentNacosNamingService(properties);
                } catch (Exception var3) {
                    log.error("create naming service error!properties={},e=,", this, var3);
                    return null;
                }

                return this.namingService;
            }
        }

    }

    static class DevEnvironmentNacosNamingService extends NacosNamingService {

        public DevEnvironmentNacosNamingService(Properties properties) {
            super(properties);
        }

        @Override
        public List<Instance> selectInstances(String serviceName, List<String> clusters, boolean healthy) throws NacosException {
            List<Instance> instances = super.selectInstances(serviceName, clusters, healthy);
            instances.stream().forEach(instance -> instance.setIp("10.101.232.24"));
            return instances;
        }
    }

}
           

調試後發現部落格提供的代碼并不能滿足筆者的需求，還得進一步深入探索。

但幸運的是，調試過程發現 Nacos 服務發現的關鍵類是 com.alibaba.cloud.nacos.discovery.NacosServiceDiscovery，其中的關鍵方法是 getInstances() 和 getServices()，即「傳回指定服務 ID 的所有服務執行個體」和「擷取所有服務的名稱」。

也就是說，對 getInstances() 方法進行重寫肯定能實作本次目标——跨命名空間通路公共服務。

/**
 * Return all instances for the given service.
 * @param serviceId id of service
 * @return list of instances
 * @throws NacosException nacosException
 */
public List<ServiceInstance> getInstances(String serviceId) throws NacosException {
        String group = discoveryProperties.getGroup();
        List<Instance> instances = discoveryProperties.namingServiceInstance()
                        .selectInstances(serviceId, group, true);
        return hostToServiceInstanceList(instances, serviceId);
}

/**
 * Return the names of all services.
 * @return list of service names
 * @throws NacosException nacosException
 */
public List<String> getServices() throws NacosException {
        String group = discoveryProperties.getGroup();
        ListView<String> services = discoveryProperties.namingServiceInstance()
                        .getServicesOfServer(1, Integer.MAX_VALUE, group);
        return services.getData();
}
           

03 最終解決思路及代碼示例

具體的解決方案思路大緻如下：

1. 生成一個共享配置類NacosShareProperties，用來配置共享公共服務的 namespace 和 group；
2. 重寫配置類 NacosDiscoveryProperties （新：NacosDiscoveryPropertiesV2），将新增的共享配置類作為屬性放進該配置類，後續會用到；
3. 重寫服務發現類 NacosServiceDiscovery （新：NacosServiceDiscoveryV2），這是最關鍵的邏輯；
4. 重寫自動配置類 NacosDiscoveryAutoConfiguration，将自定義相關類比 Nacos 原生類更早的注入容器。

最終代碼中用到了一些工具類，可以自行補充完整。

/**
 * <pre>
 *  @description: 共享nacos屬性
 *  @author: rookie0peng
 *  @date: 2022/8/29 15:22
 *  </pre>
 */
@Configuration
@ConfigurationProperties(prefix = "nacos.share")
public class NacosShareProperties {

    private final Map<String, Set<String>> NAMESPACE_TO_GROUP_NAME_MAP = new ConcurrentHashMap<>();

    /**
     * 共享nacos實體清單
     */
    private List<NacosShareEntity> entities;

    public List<NacosShareEntity> getEntities() {
        return entities;
    }

    public void setEntities(List<NacosShareEntity> entities) {
        this.entities = entities;
    }

    public Map<String, Set<String>> getNamespaceGroupMap() {
        safeStream(entities).filter(entity -> nonNull(entity) && nonNull(entity.getNamespace()))
                .forEach(entity -> {
                    Set<String> groupNames = NAMESPACE_TO_GROUP_NAME_MAP.computeIfAbsent(entity.getNamespace(), k -> new HashSet<>());
                    if (nonNull(entity.getGroupNames()))
                        groupNames.addAll(entity.getGroupNames());
                });
        return new HashMap<>(NAMESPACE_TO_GROUP_NAME_MAP);
    }

    @Override
    public String toString() {
        return "NacosShareProperties{" +
                "entities=" + entities +
                '}';
    }

    /**
     * 共享nacos實體
     */
    public static final class NacosShareEntity {

        /**
         * 命名空間
         */
        private String namespace;

        /**
         * 分組
         */
        private List<String> groupNames;

        public String getNamespace() {
            return namespace;
        }

        public void setNamespace(String namespace) {
            this.namespace = namespace;
        }

        public List<String> getGroupNames() {
            return groupNames;
        }

        public void setGroupNames(List<String> groupNames) {
            this.groupNames = groupNames;
        }

        @Override
        public String toString() {
            return "NacosShareEntity{" +
                    "namespace='" + namespace + '\'' +
                    ", groupNames=" + groupNames +
                    '}';
        }
    }
}
           

/**
 * @description: naocs服務發現屬性重寫
 * @author: rookie0peng
 * @date: 2022/8/30 1:19
 */
public class NacosDiscoveryPropertiesV2 extends NacosDiscoveryProperties {

    private static final Logger log = LoggerFactory.getLogger(NacosDiscoveryPropertiesV2.class);

    private final NacosShareProperties nacosShareProperties;

    private static final Map<String, NamingService> NAMESPACE_TO_NAMING_SERVICE_MAP = new ConcurrentHashMap<>();

    public NacosDiscoveryPropertiesV2(NacosShareProperties nacosShareProperties) {
        super();
        this.nacosShareProperties = nacosShareProperties;
    }

    public Map<String, NamingService> shareNamingServiceInstances() {
        if (!NAMESPACE_TO_NAMING_SERVICE_MAP.isEmpty()) {
            return new HashMap<>(NAMESPACE_TO_NAMING_SERVICE_MAP);
        }
        List<NacosShareProperties.NacosShareEntity> entities = Optional.ofNullable(nacosShareProperties)
                .map(NacosShareProperties::getEntities).orElse(Collections.emptyList());
        entities.stream().filter(entity -> nonNull(entity) && nonNull(entity.getNamespace()))
                .filter(PredicateUtil.distinctByKey(NacosShareProperties.NacosShareEntity::getNamespace))
                .forEach(entity -> {
                    try {
                        NamingService namingService = NacosFactory.createNamingService(getNacosProperties(entity.getNamespace()));
                        if (namingService != null) {
                            NAMESPACE_TO_NAMING_SERVICE_MAP.put(entity.getNamespace(), namingService);
                        }
                    } catch (Exception e) {
                        log.error("create naming service error! properties={}, e=", this, e);
                    }
                });
        return new HashMap<>(NAMESPACE_TO_NAMING_SERVICE_MAP);
    }

    private Properties getNacosProperties(String namespace) {
        Properties properties = new Properties();
        properties.put(SERVER_ADDR, getServerAddr());
        properties.put(USERNAME, Objects.toString(getUsername(), ""));
        properties.put(PASSWORD, Objects.toString(getPassword(), ""));
        properties.put(NAMESPACE, namespace);
        properties.put(UtilAndComs.NACOS_NAMING_LOG_NAME, getLogName());
        String endpoint = getEndpoint();
        if (endpoint.contains(":")) {
            int index = endpoint.indexOf(":");
            properties.put(ENDPOINT, endpoint.substring(0, index));
            properties.put(ENDPOINT_PORT, endpoint.substring(index + 1));
        }
        else {
            properties.put(ENDPOINT, endpoint);
        }

        properties.put(ACCESS_KEY, getAccessKey());
        properties.put(SECRET_KEY, getSecretKey());
        properties.put(CLUSTER_NAME, getClusterName());
        properties.put(NAMING_LOAD_CACHE_AT_START, getNamingLoadCacheAtStart());

//        enrichNacosDiscoveryProperties(properties);
        return properties;
    }
}
           

/**
 * @description: naocs服務發現重寫
 * @author: rookie0peng
 * @date: 2022/8/30 1:10
 */
public class NacosServiceDiscoveryV2 extends NacosServiceDiscovery {

    private final NacosDiscoveryPropertiesV2 discoveryProperties;

    private final NacosShareProperties nacosShareProperties;

    private final NacosServiceManager nacosServiceManager;

    public NacosServiceDiscoveryV2(NacosDiscoveryPropertiesV2 discoveryProperties, NacosShareProperties nacosShareProperties, NacosServiceManager nacosServiceManager) {
        super(discoveryProperties, nacosServiceManager);
        this.discoveryProperties = discoveryProperties;
        this.nacosShareProperties = nacosShareProperties;
        this.nacosServiceManager = nacosServiceManager;
    }

    /**
     * Return all instances for the given service.
     * @param serviceId id of service
     * @return list of instances
     * @throws NacosException nacosException
     */
    public List<ServiceInstance> getInstances(String serviceId) throws NacosException {
        String group = discoveryProperties.getGroup();
        List<Instance> instances = discoveryProperties.namingServiceInstance()
                .selectInstances(serviceId, group, true);
        if (isEmpty(instances)) {
            Map<String, Set<String>> namespaceGroupMap = nacosShareProperties.getNamespaceGroupMap();
            Map<String, NamingService> namespace2NamingServiceMap = discoveryProperties.shareNamingServiceInstances();
            for (Map.Entry<String, NamingService> entry : namespace2NamingServiceMap.entrySet()) {
                String namespace;
                NamingService namingService;
                if (isNull(namespace = entry.getKey()) || isNull(namingService = entry.getValue()))
                    continue;
                Set<String> groupNames = namespaceGroupMap.get(namespace);
                List<Instance> shareInstances;
                if (isEmpty(groupNames)) {
                    shareInstances = namingService.selectInstances(serviceId, group, true);
                    if (nonEmpty(shareInstances))
                        break;
                } else {
                    shareInstances = new ArrayList<>();
                    for (String groupName : groupNames) {
                        List<Instance> subShareInstances = namingService.selectInstances(serviceId, groupName, true);
                        if (nonEmpty(subShareInstances)) {
                            shareInstances.addAll(subShareInstances);
                        }
                    }
                }
                if (nonEmpty(shareInstances)) {
                    instances = shareInstances;
                    break;
                }
            }
        }
        return hostToServiceInstanceList(instances, serviceId);
    }

    /**
     * Return the names of all services.
     * @return list of service names
     * @throws NacosException nacosException
     */
    public List<String> getServices() throws NacosException {
        String group = discoveryProperties.getGroup();
        ListView<String> services = discoveryProperties.namingServiceInstance()
                .getServicesOfServer(1, Integer.MAX_VALUE, group);
        return services.getData();
    }

    public static List<ServiceInstance> hostToServiceInstanceList(
            List<Instance> instances, String serviceId) {
        List<ServiceInstance> result = new ArrayList<>(instances.size());
        for (Instance instance : instances) {
            ServiceInstance serviceInstance = hostToServiceInstance(instance, serviceId);
            if (serviceInstance != null) {
                result.add(serviceInstance);
            }
        }
        return result;
    }

    public static ServiceInstance hostToServiceInstance(Instance instance,
                                                        String serviceId) {
        if (instance == null || !instance.isEnabled() || !instance.isHealthy()) {
            return null;
        }
        NacosServiceInstance nacosServiceInstance = new NacosServiceInstance();
        nacosServiceInstance.setHost(instance.getIp());
        nacosServiceInstance.setPort(instance.getPort());
        nacosServiceInstance.setServiceId(serviceId);

        Map<String, String> metadata = new HashMap<>();
        metadata.put("nacos.instanceId", instance.getInstanceId());
        metadata.put("nacos.weight", instance.getWeight() + "");
        metadata.put("nacos.healthy", instance.isHealthy() + "");
        metadata.put("nacos.cluster", instance.getClusterName() + "");
        metadata.putAll(instance.getMetadata());
        nacosServiceInstance.setMetadata(metadata);

        if (metadata.containsKey("secure")) {
            boolean secure = Boolean.parseBoolean(metadata.get("secure"));
            nacosServiceInstance.setSecure(secure);
        }
        return nacosServiceInstance;
    }

    private NamingService namingService() {
        return nacosServiceManager
                .getNamingService(discoveryProperties.getNacosProperties());
    }
}
           

/**
 * @description: 重寫nacos服務發現的自動配置
 * @author: rookie0peng
 * @date: 2022/8/30 1:08
 */
@Configuration(proxyBeanMethods = false)
@ConditionalOnDiscoveryEnabled
@ConditionalOnNacosDiscoveryEnabled
@AutoConfigureBefore({NacosDiscoveryAutoConfiguration.class})
public class NacosDiscoveryAutoConfigurationV2 {

    @Bean
    @ConditionalOnMissingBean
    public NacosDiscoveryPropertiesV2 nacosProperties(NacosShareProperties nacosShareProperties) {
        return new NacosDiscoveryPropertiesV2(nacosShareProperties);
    }

    @Bean
    @ConditionalOnMissingBean
    public NacosServiceDiscovery nacosServiceDiscovery(
            NacosDiscoveryPropertiesV2 discoveryPropertiesV2, NacosShareProperties nacosShareProperties, NacosServiceManager nacosServiceManager
    ) {
        return new NacosServiceDiscoveryV2(discoveryPropertiesV2, nacosShareProperties, nacosServiceManager);
    }
}

           

本以為問題到這就結束了，但最後自測時發現程式根本不走 Nacos 的服務發現邏輯，而是執行 Ribbon 的負載均衡邏輯com.netflix.loadbalancer.AbstractLoadBalancerRule。

不過實作類是 com.alibaba.cloud.nacos.ribbon.NacosRule，繼續基于 NacosRule 重寫負載均衡。

/**
 * @description: 共享nacos命名空間規則
 * @author: rookie0peng
 * @date: 2022/8/31 2:04
 */
public class ShareNacosNamespaceRule extends AbstractLoadBalancerRule {

    private static final Logger LOGGER = LoggerFactory.getLogger(ShareNacosNamespaceRule.class);

    @Autowired
    private NacosDiscoveryPropertiesV2 nacosDiscoveryPropertiesV2;
    @Autowired
    private NacosShareProperties nacosShareProperties;

    /**
     * 重寫choose方法
     *
     * @param key
     * @return
     */
    @SneakyThrows
    @Override
    public Server choose(Object key) {
        try {
            String clusterName = this.nacosDiscoveryPropertiesV2.getClusterName();
            DynamicServerListLoadBalancer loadBalancer = (DynamicServerListLoadBalancer) getLoadBalancer();
            String name = loadBalancer.getName();

            NamingService namingService = nacosDiscoveryPropertiesV2
                    .namingServiceInstance();
            List<Instance> instances = namingService.selectInstances(name, true);
            if (CollectionUtils.isEmpty(instances)) {
                LOGGER.warn("no instance in service {}, then to get share service's instance", name);
                List<Instance> shareNamingService = this.getShareNamingService(name);
                if (nonEmpty(shareNamingService))
                    instances = shareNamingService;
                else
                    return null;
            }
            List<Instance> instancesToChoose = instances;
            if (org.apache.commons.lang3.StringUtils.isNotBlank(clusterName)) {
                List<Instance> sameClusterInstances = instances.stream()
                        .filter(instance -> Objects.equals(clusterName,
                                instance.getClusterName()))
                        .collect(Collectors.toList());
                if (!CollectionUtils.isEmpty(sameClusterInstances)) {
                    instancesToChoose = sameClusterInstances;
                }
                else {
                    LOGGER.warn(
                            "A cross-cluster call occurs，name = {}, clusterName = {}, instance = {}",
                            name, clusterName, instances);
                }
            }

            Instance instance = ExtendBalancer.getHostByRandomWeight2(instancesToChoose);

            return new NacosServer(instance);
        }
        catch (Exception e) {
            LOGGER.warn("NacosRule error", e);
            return null;
        }
    }


    @Override
    public void initWithNiwsConfig(IClientConfig iClientConfig) {

    }

    private List<Instance> getShareNamingService(String serviceId) throws NacosException {
        List<Instance> instances = Collections.emptyList();
        Map<String, Set<String>> namespaceGroupMap = nacosShareProperties.getNamespaceGroupMap();
        Map<String, NamingService> namespace2NamingServiceMap = nacosDiscoveryPropertiesV2.shareNamingServiceInstances();
        for (Map.Entry<String, NamingService> entry : namespace2NamingServiceMap.entrySet()) {
            String namespace;
            NamingService namingService;
            if (isNull(namespace = entry.getKey()) || isNull(namingService = entry.getValue()))
                continue;
            Set<String> groupNames = namespaceGroupMap.get(namespace);
            List<Instance> shareInstances;
            if (isEmpty(groupNames)) {
                shareInstances = namingService.selectInstances(serviceId, true);
                if (nonEmpty(shareInstances))
                    break;
            } else {
                shareInstances = new ArrayList<>();
                for (String groupName : groupNames) {
                    List<Instance> subShareInstances = namingService.selectInstances(serviceId, groupName, true);
                    if (nonEmpty(subShareInstances)) {
                        shareInstances.addAll(subShareInstances);
                    }
                }
            }
            if (nonEmpty(shareInstances)) {
                instances = shareInstances;
                break;
            }
        }
        return instances;
    }
}
           

至此問題得以解決。

在 Nacos 上配置好共享 namespace 和 group 後，就能夠進行跨命名空間通路了。

# nacos共享命名空間配置 示例
nacos.share.entities[0].namespace=e6ed2017-3ed6-4d9b-824a-db626424fc7b
nacos.share.entities[0].groupNames[0]=DEFAULT_GROUP
# 指定服務使用共享的負載均衡規則，service-id是注冊到nacos上的服務id，ShareNacosNamespaceRule需要寫全限定名
service-id.ribbon.NFLoadBalancerRuleClassName=***.***.***.ShareNacosNamespaceRule

           

注意：如果 Java 項目的 nacos discovery 版本用的是 2021.1，則不需要重寫 Ribbon 的負載均衡類，因為該版本的 Nacos 不依賴 Ribbon。

2.2.1.RELEASE 版本的 nacos discovery 依賴 Ribbon.

2021.1 版本的 nacos discovery 不依賴 Ribbon。

四、總結

為了達到共享命名空間的預期，構思、查找資料、實作邏輯、調試，前後一共花費 4 天時間。成就感滿滿的同時，筆者也發現該功能仍存在共享服務緩存等可優化空間，留待後續實作。