So, we are now experiencing this problem - Alfresco is very slow during working hours and finally crashes. Our setup is following:
Virtual server for applications (alfresco and share): 4 x 2.40 GHz, 9.5 GB RAM;
Database server: 4 x 2.40 GHz, 5GB RAM;
OS: Centos 6;
Alfresco version: Alfresco 4.2.e + Alvex Enterprise 2.1.2 plugin;
Users: <500;
Active users: 10-20;
Problem we are having
Alfresco documentation says, that with such server parameters we should be able to run server with 500 users or 50 concurrent.
During working hours, load average of server increases to ~30, CPU runs 100% and system becomes very slow. After running slow system crashed and starts throwing errors. I attach screenshot of server load:
Those spikes in load average looks strange for me. When I am using the system when there is no load, I see 10-40% CPU utilization for 1 or 2 seconds after each click I make (home page, repository, my tasks, site page). If I press my completed tasks, there is 40% CPU utilization for about 2-3 seconds (I have only 135 tasks completed, other business users has a lot more tasks completed).
Some quotes from logs:
2016-02-11 09:17:16,038 INFO [webscripts.connector.RemoteClient][http-apr-8080-exec-314] Error status 408 Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.springframework.extensions.webscripts.connector.RemoteClient.service(RemoteClient.java:1054)
at org.springframework.extensions.webscripts.connector.RemoteClient.service(RemoteClient.java:829)
at org.springframework.extensions.webscripts.connector.RemoteClient.call(RemoteClient.java:577)
at org.springframework.extensions.webscripts.connector.RemoteClient.call(RemoteClient.java:511)
at org.springframework.extensions.webscripts.connector.HttpConnector.call(HttpConnector.java:70)
at org.springframework.extensions.webscripts.RequestCachingConnector.call(RequestCachingConnector.java:90)
at org.springframework.extensions.webscripts.connector.AbstractConnector.call(AbstractConnector.java:116)
at org.springframework.extensions.webscripts.connector.AuthenticatingConnector.call(AuthenticatingConnector.java:92)
at org.alfresco.web.scripts.UserPreferences.getValue(UserPreferences.java:66)
at sun.reflect.GeneratedMethodAccessor1018.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
at org.mozilla.javascript.JavaMembers.get(JavaMembers.java:117)
at org.mozilla.javascript.NativeJavaObject.get(NativeJavaObject.java:113)
at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1375)
at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1364)
at org.mozilla.javascript.gen.c5._c32(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:1859)
at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108)
at org.mozilla.javascript.gen.c5._c0(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:1862)
at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
at org.mozilla.javascript.gen.c5.exec(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScriptImpl(JSScriptProcessor.java:318)
at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScript(JSScriptProcessor.java:192)
at org.springframework.extensions.webscripts.AbstractWebScript.executeScript(AbstractWebScript.java:1305)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:86)
at org.springframework.extensions.webscripts.PresentationContainer.executeScript(PresentationContainer.java:70)
at org.springframework.extensions.webscripts.LocalWebScriptRuntimeContainer.executeScript(LocalWebScriptRuntimeContainer.java:236)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
at org.springframework.extensions.webscripts.WebScriptProcessor.executeBody(WebScriptProcessor.java:310)
at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57)
at org.springframework.extensions.surf.render.RenderService.process(RenderService.java:599)
at org.springframework.extensions.surf.render.RenderService.renderSubComponent(RenderService.java:505)
at org.springframework.extensions.surf.render.RenderService.processComponent(RenderService.java:419)
at org.springframework.extensions.surf.render.RenderService.renderComponent(RenderService.java:942)
at org.springframework.extensions.surf.render.RenderService.renderRegionComponents(RenderService.java:900)
at org.springframework.extensions.surf.render.bean.RegionRenderer.body(RegionRenderer.java:89)
at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
at org.springframework.extensions.surf.render.RenderService.renderRegion(RenderService.java:851)
at org.springframework.extensions.directives.RegionDirectiveData.render(RegionDirectiveData.java:91)
at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408)
at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169)
at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137)
at freemarker.core.Environment.visit(Environment.java:274)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.Environment$3.render(Environment.java:246)
at org.springframework.extensions.surf.extensibility.impl.DefaultExtensibilityDirectiveData.render(DefaultExtensibilityDirectiveData.java:119)
at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408)
at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169)
at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137)
at freemarker.core.Environment.visit(Environment.java:274)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.Environment.visit(Environment.java:406)
at freemarker.core.BodyInstruction.accept(BodyInstruction.java:93)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.Macro$Context.runMacro(Macro.java:172)
at freemarker.core.Environment.visit(Environment.java:614)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:106)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.Environment.process(Environment.java:199)
at org.springframework.extensions.webscripts.processor.FTLTemplateProcessor.process(FTLTemplateProcessor.java:171)
at org.springframework.extensions.webscripts.WebTemplateProcessor.executeBody(WebTemplateProcessor.java:438)
at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57)
at org.springframework.extensions.surf.render.RenderService.processTemplate(RenderService.java:721)
at org.springframework.extensions.surf.render.bean.TemplateInstanceRenderer.body(TemplateInstanceRenderer.java:140)
at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
at org.springframework.extensions.surf.render.bean.PageRenderer.body(PageRenderer.java:85)
at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
at org.springframework.extensions.surf.render.RenderService.renderPage(RenderService.java:762)
at org.springframework.extensions.surf.mvc.PageView.dispatchPage(PageView.java:411)
at org.springframework.extensions.surf.mvc.PageView.renderView(PageView.java:306)
at org.springframework.extensions.surf.mvc.AbstractWebFrameworkView.renderMergedOutputModel(AbstractWebFrameworkView.java:316)
at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.alfresco.web.site.servlet.MTAuthenticationFilter.doFilter(MTAuthenticationFilter.java:74)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.alfresco.web.site.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:168)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.alfresco.web.site.servlet.CSRFFilter.doFilter(CSRFFilter.java:313)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:378)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1810)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
2016-02-11 09:22:34,322 ERROR [extensions.webscripts.AbstractRuntime][http-apr-8080-exec-325] Exception from executeScript - redirecting to status template error: 01110055 Failed to execute script 'classpath*:alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js': 01110054 TypeError: Cannot read property "data" from null(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37) org.springframework.extensions.webscripts.WebScriptException: 01110055 Failed to execute script 'classpath*:alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js': 01110054 TypeError: Cannot read property "data" from null(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37) at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScript(JSScriptProcessor.java:200) at org.springframework.extensions.webscripts.AbstractWebScript.executeScript(AbstractWebScript.java:1305) at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:86) at org.springframework.extensions.webscripts.PresentationContainer.executeScript(PresentationContainer.java:70) at org.springframework.extensions.webscripts.LocalWebScriptRuntimeContainer.executeScript(LocalWebScriptRuntimeContainer.java:236) at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378) at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209) at org.springframework.extensions.webscripts.WebScriptProcessor.executeBody(WebScriptProcessor.java:310) at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57) at org.springframework.extensions.surf.render.RenderService.process(RenderService.java:599) at org.springframework.extensions.surf.render.RenderService.renderSubComponent(RenderService.java:505) at org.springframework.extensions.surf.render.RenderService.processComponent(RenderService.java:419) at org.springframework.extensions.surf.render.RenderService.renderComponent(RenderService.java:942) at org.springframework.extensions.surf.render.RenderService.renderRegionComponents(RenderService.java:900) at org.springframework.extensions.surf.render.bean.RegionRenderer.body(RegionRenderer.java:89) at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77) at org.springframework.extensions.surf.render.RenderService.renderRegion(RenderService.java:851) at org.springframework.extensions.directives.RegionDirectiveData.render(RegionDirectiveData.java:91) at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408) at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169) at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137) at freemarker.core.Environment.visit(Environment.java:274) at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.MixedContent.accept(MixedContent.java:92) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.Environment$3.render(Environment.java:246) at org.springframework.extensions.surf.extensibility.impl.DefaultExtensibilityDirectiveData.render(DefaultExtensibilityDirectiveData.java:119) at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408) at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169) at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137) at freemarker.core.Environment.visit(Environment.java:274) at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.MixedContent.accept(MixedContent.java:92) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.Environment.visit(Environment.java:406) at freemarker.core.BodyInstruction.accept(BodyInstruction.java:93) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.MixedContent.accept(MixedContent.java:92) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.Macro$Context.runMacro(Macro.java:172) at freemarker.core.Environment.visit(Environment.java:614) at freemarker.core.UnifiedCall.accept(UnifiedCall.java:106) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.MixedContent.accept(MixedContent.java:92) at freemarker.core.Environment.visit(Environment.java:221) at freemarker.core.Environment.process(Environment.java:199) at org.springframework.extensions.webscripts.processor.FTLTemplateProcessor.process(FTLTemplateProcessor.java:171) at org.springframework.extensions.webscripts.WebTemplateProcessor.executeBody(WebTemplateProcessor.java:438) at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57) at org.springframework.extensions.surf.render.RenderService.processTemplate(RenderService.java:721) at org.springframework.extensions.surf.render.bean.TemplateInstanceRenderer.body(TemplateInstanceRenderer.java:140) at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77) at org.springframework.extensions.surf.render.bean.PageRenderer.body(PageRenderer.java:85) at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77) at org.springframework.extensions.surf.render.RenderService.renderPage(RenderService.java:762) at org.springframework.extensions.surf.mvc.PageView.dispatchPage(PageView.java:411) at org.springframework.extensions.surf.mvc.PageView.renderView(PageView.java:306) at org.springframework.extensions.surf.mvc.AbstractWebFrameworkView.renderMergedOutputModel(AbstractWebFrameworkView.java:316) at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250) at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:621) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339) at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.alfresco.web.site.servlet.MTAuthenticationFilter.doFilter(MTAuthenticationFilter.java:74) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.alfresco.web.site.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:168) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.alfresco.web.site.servlet.CSRFFilter.doFilter(CSRFFilter.java:313) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:378) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1810) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.springframework.extensions.webscripts.WebScriptException: 01110054 TypeError: Cannot read property "data" from null(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37) at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScriptImpl(JSScriptProcessor.java:328) at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScript(JSScriptProcessor.java:192) ... 106 more Caused by: org.mozilla.javascript.EcmaError: TypeError: Cannot read property "data" from null(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37) at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3350) at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3340) at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3356) at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3375) at org.mozilla.javascript.ScriptRuntime.undefReadError(ScriptRuntime.java:3388) at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1362) at org.mozilla.javascript.gen.c5._c2(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:37) at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js) at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108) at org.mozilla.javascript.gen.c5._c3(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:41) at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js) at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108) at org.mozilla.javascript.gen.c5._c33(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:1871) at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js) at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108) at org.mozilla.javascript.gen.c5._c0(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:2093) at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834) at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js) at org.mozilla.javascript.gen.c5.exec(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js) at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScriptImpl(JSScriptProcessor.java:318) ... 107 more
After Alfresco restart everything works fine until again load average reaches value of ~30.
Solutions we have tried
We have tried several solutions, which were not very successful:
1. Different servers for alfresco application and database (worked for couple months);
2. Tuning JVM, but no results;
Current JVM settings: JAVA_OPTS="-XX:MaxPermSize=256m -Xss1024K -Xms1G -Xmx8G -XX:+DisableExplicitGC -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true -server"
Do you have any other ideas which would be worth trying? Maybe you have any ideas how to identify which user actions are causing these spikes in load average?
