相关文章推荐

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset #8917 ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset #8917 myifeng opened this issue Nov 25, 2021 · 17 comments

Describe the bug

I get a reset error when I use keycloak to search for users.
This error does not appear every time, only occasionally, but I don't know how to fix it.
Please help~

private boolean verifyAccountFromKeycloak(String account, String email) {
        final UsersResource usersResource = keycloak.realm(currentRealm).users();
        if (CollectionUtils.isNotEmpty(usersResource.search(account, true))) {
            return false;
        List<UserRepresentation> users2 = usersResource.search(null, null, null, email, 0, 20);
        return CollectionUtils.isEmpty(users2) || !users2.stream().anyMatch(s -> email.equals(s.getEmail()));

Version

15.0.2

Expected behavior

No errors or exceptions

Actual behavior

javax.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:328)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:443)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:149)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
        at com.sun.proxy.$Proxy196.refreshToken(Unknown Source)
        at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:111)
        at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:72)
        at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:65)
        at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:579)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:440)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:149)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
        at com.sun.proxy.$Proxy249.search(Unknown Source)
        at com.commercial.provider.service.impl.UserServiceImpl.lambda$verifyAccountFromKeycloak$0(UserServiceImpl.java:133)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        ... 1 common frames omitted
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute$original$3AyxwFtD(InternalHttpClient.java:185)
        at org.apache.http.impl.client.InternalHttpClient.doExecute$original$3AyxwFtD$accessor$QdjClIoY(InternalHttpClient.java)
        at org.apache.http.impl.client.InternalHttpClient$auxiliary$wWrcgHZe.call(Unknown Source)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:323)
        ... 18 common frames omitted

How to Reproduce?

Sorry,This error does not appear every time, only occasionally.

Anything else?

Spring Boot: 2.3.11.RELEASE
JDK Image: openjdk:8u212-jre-slim

@stianst In the error log I saw jbossTokenManager.refreshTokenBearerAuthFilter.filter .Is this related to token expiration?

I highly suspect that there is an error in'TokenManager.refreshToken(TokenManager.java:111)':
currentToken = tokenService.refreshToken(config.getRealm(), form.asMap());

An unknown error occurred when the client requested the token, which may be on server or during network transmission.

After the token expired, when the client tried to obtain the token again, the error occurred?

The following is how I use it:

KeycloakClientConfig.java

@RefreshScope
@Configuration
public class KeycloakClientConfig {
    @Value("${project.keycloak.server-url}")
    private String serverUrl;
    @Value("${project.keycloak.realm}")
    private String realm;
    @Value("${project.keycloak.client-id}")
    private String clientId;
    @Value("${project.keycloak.username}")
    private String username;
    @Value("${project.keycloak.password}")
    private String password;
    @Bean
    public Keycloak keycloak() {
        return KeycloakBuilder.builder()
                .serverUrl(serverUrl)
                .realm(realm)
                .clientId(clientId)
                .username(username)
                .password(password)
                .build();

UserServiceImpl.java

@Resource
    Keycloak keycloak;
    @Value("${project.keycloak.current-realm}")
    private String currentRealm;
    private boolean verifyAccountFromKeycloak(String account, String email) {
        final UsersResource usersResource = keycloak.realm(currentRealm).users();
        if (CollectionUtils.isNotEmpty(usersResource.search(account, true))) {
            return false;
        List<UserRepresentation> users2 = usersResource.search(null, null, null, email, 0, 20);
        return CollectionUtils.isEmpty(users2) || !users2.stream().anyMatch(s -> email.equals(s.getEmail()));
          

Thanks for taking the time to submit this issue. However, we were unable to reproduce the mentioned steps. We strongly recommend that people upgrade to the latest releases of Keycloak. The WildFly distribution was discontinued and no longer supported by our team.

If you can reproduce the issue using the latest releases of Keycloak, please reopen this issue, including a reproducer.

I have the same issue, but possibly found the workaround (it is under tests now). KeycloakBuilder has a method: .resteasyClient(..) to provide custom implementation of javax.ws.rs.client.Client and that implementation could have some settings about connection/read timeouts, evicting idle connections policy and retry handler implementation. Spring configuration for Keycloak bean could look like the following:

    @Bean
    public Client customizedResteasyClient() {
        CloseableHttpClient httpClient = createHttpClient();
        ApacheHttpClient43Engine engine = new ApacheHttpClient43Engine(httpClient);
        return ((ResteasyClientBuilder) ClientBuilder.newBuilder())
                .httpEngine(engine)
                .connectTimeout(10000, TimeUnit.MILLISECONDS)
                .readTimeout(7000, TimeUnit.MILLISECONDS)
                .connectionTTL(-1, TimeUnit.MILLISECONDS)
                .disableTrustManager()
                .build();
    private CloseableHttpClient createHttpClient() {
        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        cm.setValidateAfterInactivity(1000);
        cm.setMaxTotal(200);
        cm.setDefaultMaxPerRoute(20);
        return HttpClients.custom()
                .setConnectionManager(cm)
                .evictExpiredConnections()
                .evictIdleConnections(10000, TimeUnit.MILLISECONDS)
                .setRetryHandler(DefaultHttpRequestRetryHandler.INSTANCE)
                .build();
    @Bean
     public Keycloak customizedKeycloakAdminClient() {
        return KeycloakBuilder.builder()
                .grantType(OAuth2Constants.CLIENT_CREDENTIALS)
                .realm("admin")
                .clientId("clientId")
                .clientSecret("clientSecret")
                .serverUrl("http://127.0.0.1:8080")
                .resteasyClient(customizedResteasyClient())
                .build();

I think that pooling settings like retry handler and idle connections eviction mechanism could cause that sockets used to connect through HTTP protocol should be tested before use and connections closed by some external mechanisms like firewall would be rejected by KeycloakAdmin client.

We have the problem reported from the client's environment - random java.net.SocketException: Connection reset while executing get user operation (in most cases). Our scenario was" create a user (through Keycloak API) and generate some OTP for it (OTP generation code was not related to the Keycloak API, but we needed to get user Id from Keycloak to put it as a reference in our database). Keycloak and the service that invoked the API were deployed as separate pod's in the same Kubernetes namespace - the client's TEST/UAT env and our DEV environments were pretty the same (client followed our instructions to prepare env).

Of course as we could not reproduce the issue, we tried to seek for the solution in a blind manner - we tried to use the latest possible version of keycloak-admin-client library, but it did not help. So we thought that we should have as wide control over the HTTP client as possible and prepared the workaround described above. Our first idea was to turn on some DEBUG logs and observe how the connection pool or timeouts behave on client's env, especially when the problem occurs again. But it did not occur... and the client closed the issue in Jira :) As client swears that he did not change enything in the environment, we suppose that some idle connection eviction/validation mechanims resolved the problem.

@tramiaczek thanks for clarification. Do I understand correctly that when passing the custom client the error did not occur again?

Yes, it currently works almost 2 months and the error was not reopened by the client.

I've implemented the solution (here's the Clojure version). Where we first got roughly 30 RESTEASY004655 error messages a day, we now average 3 per day. Mainly coming from the health check that calls the service every 15 seconds thus 5760 times per day. So tenfold decrease in error messages, but there are still some. Maybe can use some further tuning, but already great reduction. Thanks a lot for this improvement @tramiaczek.

I was able to reliable reproduce the issue using the latest keycloak and keycloak-admin-client (each 22.0.5).

Reproducer setup

I am running on Ubuntu 20.04 (WSL2 on Windows) and OpenJDK 17.

  • Start a keycloak server: docker run -p 8081:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:22.0.5 start-dev
  • Have tcpkill send RST packets to connections on port 8081 (you may need to sudo apt install dsniff): sudo tcpkill -1 -i lo port 8081
  • This is to simulate a firewall or other network infrastructure closing the connection. In our case we suspect a firewall to kill connections that are open for too long (10+ hours)
  • Set up a java (e.g. maven) project that uses the keycloak-admin-client and run this reproducer:
    public class App {
        public static void main(String[] args) throws InterruptedException {
            Keycloak kc = KeycloakBuilder.builder()
                    .serverUrl("http://localhost:8081")
                    .realm("master")
                    .username("admin")
                    .password("admin")
                    .clientId("admin-cli")
                    .build();
            while (true) {
                tryTalkToKeycloakWithExpiredToken(kc);
                Thread.sleep(100);
        private static void tryTalkToKeycloakWithExpiredToken(Keycloak kc) {
            // force the next request to refresh the token
            kc.tokenManager().invalidate(kc.tokenManager().getAccessToken().getToken());
            kc.realm("master").getAdminEvents().size(); // trigger any request
    

    The reproducer may take a few iterations until it hits this exception and exits:

    Exception in thread "main" jakarta.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset
    	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:361)
    	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:427)
    	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:134)
    	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:103)
    	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:61)
    	at jdk.proxy2/jdk.proxy2.$Proxy16.refreshToken(Unknown Source)
    	at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:121)
    	at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:77)
    	at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:70)
    	at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
    	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:644)
    	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:424)
    	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:134)
    	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:103)
    	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:61)
    	at jdk.proxy2/jdk.proxy2.$Proxy29.getAdminEvents(Unknown Source)
    	at org.example.App.tryTalkToKeycloakWithExpiredToken(App.java:33)
    	at org.example.App.main(App.java:25)
    Caused by: java.net.SocketException: Connection reset
    	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
    	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
    	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
    	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
    	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
    	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
    	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:344)
    	... 17 more
    

    Some iterations may fully succeed, some may only print this message:

    Nov 07, 2023 12:51:58 PM org.apache.http.impl.execchain.RetryExec execute
    INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8081: Connection reset by peer
    Nov 07, 2023 12:51:58 PM org.apache.http.impl.execchain.RetryExec execute
    INFO: Retrying request to {}->http://localhost:8081
    

    I believe the differences are due to tcpkill working on a best-effort and timing based approach to inject RST packets, which may arrive at the application sooner or later. The desicionmaking whether a request is retried lives inside the DefaultHttpRequestRetryHandler. I observed that retryRequest (the method that decides whether to retry a request) ends with this snippet of code:

            if (!clientContext.isRequestSent() || this.requestSentRetryEnabled) {
                // Retry if the request has not been sent fully or
                // if it's OK to retry methods that have been sent
                return true;
            // otherwise do not retry
            return false;

    The code always reaches that statement. I saw that this.requestSentRetryEnabled is always false, and the deciding factor is clientContext.isRequestSent(). My conclusion is that the POST request to refresh the token has been fully sent before a RST packet was received, resulting in the retry mechanism not retrying the request because it might have been processed by the server and can not be assumed to be idempotent.

    Possible solutions

    Assuming the troublemaker is some network component closing idle connections, the problem can be avoided by auto-closing idle connections and/or testing pooled connections before use. This is what @tramiaczek suggested in his previous comment #8917 (comment)

    A minimal configuration to enable auto-closing of connections may look like this:

    Keycloak kc = KeycloakBuilder.builder()
            .serverUrl("http://localhost:8081")
            .realm("master")
            .username("admin")
            .password("admin")
            .clientId("admin-cli")
            .resteasyClient(((ResteasyClientBuilder) ClientBuilder.newBuilder())
                    .connectionPoolSize(3)
                    .connectionTTL(10, TimeUnit.SECONDS)
                    .build())
            .build();

    For this specific usecase, namely the keycloak-admin-client transparently refreshing its token, it might be useful for it to automatically retry this too, since other connection failures are typically retried as well. Though I am unsure how this would be done.

    Alternatively, application-side retry mechanisms may be considered. Given the keycloak admin client may throw connection errors at any time anyway, applications should probably be robust against this.

  •  
    推荐文章