現象
- 現象是請求中的中文儲存到資料庫後會亂碼。
- 亂碼的字元并不是什麼特殊字元。
- 删除了亂碼字元前面的字元後，亂碼的地方會向後偏移。
調查過程
第一反應是資料庫字段的字元集設定導緻的，但修改成 utf8mb4 字元集後問題依舊。
通過本地調試發現，直接請求接口的字元串并沒有亂碼。
通過測試環境日志發現，Controller 接收到的參數中字元串已經亂碼了。
測試環境和開發環境的差別是其請求是通過網關轉發的。
調查網關後，發現其中有一個 Filter 曾對請求内容進行了轉碼處理。具體代碼如下：
java複制代碼import java.nio.charset.StandardCharsets;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.GlobalFilter;
import org.springframework.core.Ordered;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferUtils;
import org.springframework.core.io.buffer.NettyDataBufferFactory;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.server.reactive.ServerHttpRequest;
import org.springframework.http.server.reactive.ServerHttpRequestDecorator;
import org.springframework.stereotype.Component;
import org.springframework.web.server.ServerWebExchange;
import io.netty.buffer.ByteBufAllocator;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
/**
* 跨站腳本過濾器
*/
@Component
@ConditionalOnProperty(value = "security.xss.enabled", havingValue = "true")
public class XssFilter implements GlobalFilter, Ordered
{
// 跨站腳本的 xss 配置，nacos自行添加
@Autowired
private XssProperties xss;
@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain)
{
ServerHttpRequest request = exchange.getRequest();
// GET DELETE 不過濾
HttpMethod method = request.getMethod();
if (method == null || method.matches("GET") || method.matches("DELETE"))
{
return chain.filter(exchange);
}
// 非json類型，不過濾
if (!isJsonRequest(exchange))
{
return chain.filter(exchange);
}
// excludeUrls 不過濾
String url = request.getURI().getPath();
if (StringUtils.matches(url, xss.getExcludeUrls()))
{
return chain.filter(exchange);
}
ServerHttpRequestDecorator httpRequestDecorator = requestDecorator(exchange);
return chain.filter(exchange.mutate().request(httpRequestDecorator).build());
}
private ServerHttpRequestDecorator requestDecorator(ServerWebExchange exchange)
{
ServerHttpRequestDecorator serverHttpRequestDecorator = new ServerHttpRequestDecorator(exchange.getRequest())
{
@Override
public Flux<DataBuffer> getBody()
{
Flux<DataBuffer> body = super.getBody();
return body.map(dataBuffer -> {
byte[] content = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(content);
DataBufferUtils.release(dataBuffer);
String bodyStr = new String(content, StandardCharsets.UTF_8);
// 防xss攻擊過濾
bodyStr = EscapeUtil.clean(bodyStr);
// 轉成位元組
byte[] bytes = bodyStr.getBytes();
NettyDataBufferFactory nettyDataBufferFactory = new NettyDataBufferFactory(ByteBufAllocator.DEFAULT);
DataBuffer buffer = nettyDataBufferFactory.allocateBuffer(bytes.length);
buffer.write(bytes);
return buffer;
});
}
@Override
public HttpHeaders getHeaders()
{
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.putAll(super.getHeaders());
// 由于修改了請求體的body，導緻content-length長度不确定，是以需要删除原先的content-length
httpHeaders.remove(HttpHeaders.CONTENT_LENGTH);
httpHeaders.set(HttpHeaders.TRANSFER_ENCODING, "chunked");
return httpHeaders;
}
};
return serverHttpRequestDecorator;
}
/**
* 是否是Json請求
*
* @param
*/
public boolean isJsonRequest(ServerWebExchange exchange)
{
String header = exchange.getRequest().getHeaders().getFirst(HttpHeaders.CONTENT_TYPE);
return StringUtils.startsWithIgnoreCase(header, MediaType.APPLICATION_JSON_VALUE);
}
@Override
public int getOrder()
{
return -100;
}
}
本地調試網關服務時發現出問題的是 getBody() 方法。 當請求body較長時，byte[] content = new byte[dataBuffer.readableByteCount()]; 變量中擷取到的并不是全部的請求體，而隻是其中的部分内容。 這就解釋了為什麼會固定在一定長度的時候會亂碼。
java複制代碼@Override
public Flux<DataBuffer> getBody()
{
Flux<DataBuffer> body = super.getBody();
return body.map(dataBuffer -> {
byte[] content = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(content);
DataBufferUtils.release(dataBuffer);
String bodyStr = new String(content, StandardCharsets.UTF_8);
// 防xss攻擊過濾
bodyStr = EscapeUtil.clean(bodyStr);
// 轉成位元組
byte[] bytes = bodyStr.getBytes();
NettyDataBufferFactory nettyDataBufferFactory = new NettyDataBufferFactory(ByteBufAllocator.DEFAULT);
DataBuffer buffer = nettyDataBufferFactory.allocateBuffer(bytes.length);
buffer.write(bytes);
return buffer;
});
}
在這篇部落格中部落客遇到了同樣的問題，并給出了一種解決方案。
java複制代碼.route("route1",
r -> r.method(HttpMethod.POST)
.and().readBody(String.class, requestBody -> {
// 這裡不對body做判斷處理
return true;
}).and().path(SERVICE1)
.filters(f -> {
f.filters(list);
return f;
}).uri(URI));
但是這種方法需要修改代碼，而現在項目中網關的路由配置一般都是放在配置中心上的。 檢視 readBody() 方法的代碼，發現這裡指定的是一個 Predicate （斷言）。
參考這篇部落格可以通過修改網關配置檔案實作相同的效果。
修改後網關路由配置如下：
yaml複制代碼spring:
cloud:
gateway:
routes:
- id: some-system
uri: lb://some-system
filters:
- StripPrefix=1
predicates:
- Path=/some-system/**
- name: ReadBodyPredicateFactory
args:
inClass: '#{T(String)}'
之後啟動網關服務時報了如下錯誤：
Unable to find RoutePredicateFactory with name ReadBodyPredicateFactory
完整錯誤消息堆棧如下：
java複制代碼14:38:40.771 [boundedElastic-27] ERROR o.s.c.g.r.CachingRouteLocator - [handleRefreshError,94] - Refresh routes error !!!
java.lang.IllegalArgumentException: Unable to find RoutePredicateFactory with name ReadBodyPredicateFactory
at org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator.lookup(RouteDefinitionRouteLocator.java:203)
at org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator.combinePredicates(RouteDefinitionRouteLocator.java:192)
at org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator.convertToRoute(RouteDefinitionRouteLocator.java:116)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmitScalar(FluxFlatMap.java:488)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:421)
at reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.drain(FluxMergeSequential.java:432)
at reactor.core.publisher.FluxMergeSequential$MergeSequentialMain.innerComplete(FluxMergeSequential.java:328)
at reactor.core.publisher.FluxMergeSequential$MergeSequentialInner.onComplete(FluxMergeSequential.java:584)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:166)
at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:269)
at reactor.core.publisher.FluxFilter$FilterConditionalSubscriber.onComplete(FluxFilter.java:300)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:846)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:608)
at reactor.core.publisher.FluxFlatMap$FlatMapMain.innerComplete(FluxFlatMap.java:894)
at reactor.core.publisher.FluxFlatMap$FlatMapInner.onComplete(FluxFlatMap.java:997)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1799)
at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:128)
at org.springframework.cloud.commons.publisher.FluxFirstNonEmptyEmitting$FirstNonEmptyEmittingSubscriber.onComplete(FluxFirstNonEmptyEmitting.java:325)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.onComplete(FluxSubscribeOn.java:166)
at reactor.core.publisher.FluxIterable$IterableSubscription.fastPath(FluxIterable.java:360)
at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.requestUpstream(FluxSubscribeOn.java:131)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.onSubscribe(FluxSubscribeOn.java:124)
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:164)
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
at reactor.core.publisher.Flux.subscribe(Flux.java:8402)
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:200)
at reactor.core.publisher.MonoFlatMapMany.subscribeOrReturn(MonoFlatMapMany.java:49)
at reactor.core.publisher.FluxFromMonoOperator.subscribe(FluxFromMonoOperator.java:76)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at java.base/java.util.concurrent.FutureTask.run$$capture(FutureTask.java:264)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
報錯的方法源碼如下：
java複制代碼@SuppressWarnings("unchecked")
private AsyncPredicate<ServerWebExchange> lookup(RouteDefinition route, PredicateDefinition predicate) {
RoutePredicateFactory<Object> factory = this.predicates.get(predicate.getName());
if (factory == null) {
throw new IllegalArgumentException("Unable to find RoutePredicateFactory with name " + predicate.getName());
}
if (logger.isDebugEnabled()) {
logger.debug("RouteDefinition " + route.getId() + " applying " + predicate.getArgs() + " to "
+ predicate.getName());
}
// @formatter:off
Object config = this.configurationService.with(factory)
.name(predicate.getName())
.properties(predicate.getArgs())
.eventFunction((bound, properties) -> new PredicateArgsEvent(
RouteDefinitionRouteLocator.this, route.getId(), properties))
.bind();
// @formatter:on
return factory.applyAsync(config);
}
Debug 發現 this.predicates 的内容如下：
java複制代碼predicates = {LinkedHashMap@11025} size = 13
"After" -> {AfterRoutePredicateFactory@14744} "[AfterRoutePredicateFactory@7ea8224b configClass = AfterRoutePredicateFactory.Config]"
"Before" -> {BeforeRoutePredicateFactory@14746} "[BeforeRoutePredicateFactory@5a010eec configClass = BeforeRoutePredicateFactory.Config]"
"Between" -> {BetweenRoutePredicateFactory@14748} "[BetweenRoutePredicateFactory@623ded82 configClass = BetweenRoutePredicateFactory.Config]"
"Cookie" -> {CookieRoutePredicateFactory@14750} "[CookieRoutePredicateFactory@180e33b0 configClass = CookieRoutePredicateFactory.Config]"
"Header" -> {HeaderRoutePredicateFactory@14752} "[HeaderRoutePredicateFactory@270be080 configClass = HeaderRoutePredicateFactory.Config]"
"Host" -> {HostRoutePredicateFactory@14754} "[HostRoutePredicateFactory@752ffce3 configClass = HostRoutePredicateFactory.Config]"
"Method" -> {MethodRoutePredicateFactory@14756} "[MethodRoutePredicateFactory@78f35e39 configClass = MethodRoutePredicateFactory.Config]"
"Path" -> {PathRoutePredicateFactory@14758} "[PathRoutePredicateFactory@11896124 configClass = PathRoutePredicateFactory.Config]"
"Query" -> {QueryRoutePredicateFactory@14760} "[QueryRoutePredicateFactory@69fe8c75 configClass = QueryRoutePredicateFactory.Config]"
"ReadBody" -> {ReadBodyRoutePredicateFactory@14762} "[ReadBodyRoutePredicateFactory@633cad4d configClass = ReadBodyRoutePredicateFactory.Config]"
"RemoteAddr" -> {RemoteAddrRoutePredicateFactory@14764} "[RemoteAddrRoutePredicateFactory@15c3585 configClass = RemoteAddrRoutePredicateFactory.Config]"
"Weight" -> {WeightRoutePredicateFactory@14766} "[WeightRoutePredicateFactory@5b86f4cb configClass = WeightConfig]"
"CloudFoundryRouteService" -> {CloudFoundryRouteServiceRoutePredicateFactory@14768} "[CloudFoundryRouteServiceRoutePredicateFactory@468646ea configClass = Object]"
也就是說配置檔案中 predicate.name 應該為 ReadBody 而不是 ReadBodyPredicateFactory。
this.predicates 指派相關的代碼如下，從中可以看到在 NameUtils.normalizeRoutePredicateName 方法中去除了字尾 RoutePredicateFactory 。
估計這是由于項目中使用的 Gateway 版本比較新（spring-cloud-gateway-server:3.0.3）導緻的。
RouteDefinitionRouteLocator.java
java複制代碼private void initFactories(List<RoutePredicateFactory> predicates) {
predicates.forEach(factory -> {
String key = factory.name();
if (this.predicates.containsKey(key)) {
this.logger.warn("A RoutePredicateFactory named " + key + " already exists, class: "
+ this.predicates.get(key) + ". It will be overwritten.");
}
this.predicates.put(key, factory);
if (logger.isInfoEnabled()) {
logger.info("Loaded RoutePredicateFactory [" + key + "]");
}
});
}
RoutePredicateFactory.java
java複制代碼default String name() {
return NameUtils.normalizeRoutePredicateName(getClass());
}
NameUtils.java
java複制代碼public static String normalizeRoutePredicateName(Class<? extends RoutePredicateFactory> clazz) {
return removeGarbage(clazz.getSimpleName().replace(RoutePredicateFactory.class.getSimpleName(), ""));
}
private static String removeGarbage(String s) {
int garbageIdx = s.indexOf("$Mockito");
if (garbageIdx > 0) {
return s.substring(0, garbageIdx);
}
return s;
}
初次之外還需要配置一個 Predicate 類型的 Bean 并 自定義個 RoutePredicateFactory (從 ReadBodyRoutePredicateFactory 複制修改）。 否則會出現 NPE 和 404 NOT FOUND 錯誤。（解決方法參考自前面提到的部落格及另一篇部落格）
NPE 錯誤如下：
java複制代碼16:13:00.409 [reactor-http-epoll-3] ERROR o.s.c.g.h.RoutePredicateHandlerMapping - [lambda$null$3,132] - Error applying predicate for route: launch-tencent
java.lang.NullPointerException: null
at org.springframework.cloud.gateway.handler.predicate.ReadBodyRoutePredicateFactory$1.lambda$null$1(ReadBodyRoutePredicateFactory.java:96)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
報錯的代碼是最後一行 config.getPredicate().test(objectValue) ，由于沒有配置 config.getPredicate() 導緻了 NPE 。
java複制代碼return ServerWebExchangeUtils.cacheRequestBodyAndRequest(exchange,
(serverHttpRequest) -> ServerRequest
.create(exchange.mutate().request(serverHttpRequest).build(), messageReaders)
.bodyToMono(inClass).doOnNext(objectValue -> exchange.getAttributes()
.put(CACHE_REQUEST_BODY_OBJECT_KEY, objectValue))
.map(objectValue -> config.getPredicate().test(objectValue)));
添加一個自定義的斷言 alwaysTruePredicate ，并在配置檔案的 args 中指定使用這個斷言 predicate: '#{@alwaysTruePredicate}' 。
java複制代碼@Bean
public Predicate alwaysTruePredicate(){
return new Predicate() {
@Override
public boolean test(Object o) {
return true;
}
};
}
404 NOT FOUND 錯誤需要在 ReadBodyRoutePredicateFactory 的基礎上，在 .map(objectValue -> config.getPredicate().test(objectValue)) 的後面加一段 .thenReturn(true) 處理，使其永遠傳回 true 。
對策
- 添加工廠類 GwReadBodyRoutePredicateFactory
- java複制代碼
- import lombok.extern.slf4j.Slf4j; import org.reactivestreams.Publisher; import org.springframework.cloud.gateway.handler.AsyncPredicate; import org.springframework.cloud.gateway.handler.predicate.AbstractRoutePredicateFactory; import org.springframework.cloud.gateway.support.ServerWebExchangeUtils; import org.springframework.http.codec.HttpMessageReader; import org.springframework.stereotype.Component; import org.springframework.web.reactive.function.server.HandlerStrategies; import org.springframework.web.reactive.function.server.ServerRequest; import org.springframework.web.server.ServerWebExchange; import reactor.core.publisher.Mono; import java.util.List; import java.util.Map; import java.util.function.Predicate; /** * Predicate that reads the body and applies a user provided predicate to run on the body. * The body is cached in memory so that possible subsequent calls to the predicate do not * need to deserialize again. */ @Component @Slf4j public class GwReadBodyRoutePredicateFactory extends AbstractRoutePredicateFactory<GwReadBodyRoutePredicateFactory.Config> { private static final String TEST_ATTRIBUTE = "read_body_predicate_test_attribute"; private static final String CACHE_REQUEST_BODY_OBJECT_KEY = "cachedRequestBodyObject"; private final List<HttpMessageReader<?>> messageReaders; public GwReadBodyRoutePredicateFactory() { super(GwReadBodyRoutePredicateFactory.Config.class); this.messageReaders = HandlerStrategies.withDefaults().messageReaders(); } public GwReadBodyRoutePredicateFactory(List<HttpMessageReader<?>> messageReaders) { super(GwReadBodyRoutePredicateFactory.Config.class); this.messageReaders = messageReaders; } @Override @SuppressWarnings("unchecked") public AsyncPredicate<ServerWebExchange> applyAsync( GwReadBodyRoutePredicateFactory.Config config ) { return new AsyncPredicate<ServerWebExchange>() { @Override public Publisher<Boolean> apply(ServerWebExchange exchange) { Class inClass = config.getInClass(); Object cachedBody = exchange.getAttribute(CACHE_REQUEST_BODY_OBJECT_KEY); Mono<?> modifiedBody; // We can only read the body from the request once, once that happens if // we try to read the body again an exception will be thrown. The below // if/else caches the body object as a request attribute in the // ServerWebExchange so if this filter is run more than once (due to more // than one route using it) we do not try to read the request body // multiple times if (cachedBody != null) { try { boolean test = config.predicate.test(cachedBody); exchange.getAttributes().put(TEST_ATTRIBUTE, test); return Mono.just(test); } catch (ClassCastException e) { if (log.isDebugEnabled()) { log.debug("Predicate test failed because class in predicate " + "does not match the cached body object", e); } } return Mono.just(false); } else { return ServerWebExchangeUtils.cacheRequestBodyAndRequest(exchange, (serverHttpRequest) -> ServerRequest.create( exchange.mutate().request(serverHttpRequest).build(), messageReaders) .bodyToMono(inClass) .doOnNext(objectValue -> exchange.getAttributes() .put(CACHE_REQUEST_BODY_OBJECT_KEY, objectValue)) .map(objectValue -> config.getPredicate().test(objectValue)) .thenReturn(true)); } } @Override public String toString() { return String.format("ReadBody: %s", config.getInClass()); } }; } @Override @SuppressWarnings("unchecked") public Predicate<ServerWebExchange> apply( GwReadBodyRoutePredicateFactory.Config config ) { throw new UnsupportedOperationException("GwReadBodyPredicateFactory is only async."); } public static class Config { private Class inClass; private Predicate predicate; private Map<String, Object> hints; public Class getInClass() { return inClass; } public GwReadBodyRoutePredicateFactory.Config setInClass(Class inClass) { this.inClass = inClass; return this; } public Predicate getPredicate() { return predicate; } public GwReadBodyRoutePredicateFactory.Config setPredicate(Predicate predicate) { this.predicate = predicate; return this; } public <T> GwReadBodyRoutePredicateFactory.Config setPredicate(Class<T> inClass, Predicate<T> predicate) { setInClass(inClass); this.predicate = predicate; return this; } public Map<String, Object> getHints() { return hints; } public GwReadBodyRoutePredicateFactory.Config setHints(Map<String, Object> hints) { this.hints = hints; return this; } } }
- 添加一個 GwReadBody 用的斷言 Bean alwaysTruePredicate 。
- java複制代碼
- @Bean public Predicate alwaysTruePredicate(){ return new Predicate() { @Override public boolean test(Object o) { return true; } }; }
- 在網關的路由配置中添加 GwReadBody 斷言配置，将接收類型指定為 String，并将 predicate 參數指定上一步中配置的 alwaysTruePredicate 。
- yaml複制代碼
- spring: cloud: gateway: routes: - id: some-system uri: lb://some-system filters: - StripPrefix=1 predicates: - Path=/some-system/** - name: GwReadBody args: inClass: '#{T(String)}' predicate: '#{@alwaysTruePredicate}'
- 重新開機網關服務後，再次在 XssFilter 中通過 getBody() 擷取請求内容時，獲得的已經是完整的請求體了。
作者：佳佳_
連結：https://juejin.cn/post/7237424021766914106
![MyBatis-Plus 之AR模式[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)