Switching to read-only mode!!!

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Switching to read-only mode!!!

nsincaglia
I am running eXist-db 3.0 RC1 with a Linux AWS OS on a production server. Recently, I have been having to needing to restart it manually once a day or it will eventually switch to read-only on me. I have looked through wrapper.log and exist.log for hints of why this might be happening. I have chased down a few theories and made some adjustments to my code but so far, the behavior has not changed. I was hoping that if I posted the error messages I am seeing in exist.log, someone might be able to give me a hint as to why this is happening. I can see that eXist-db is complaining that there are too many files open in jetty/tmp. Is there anything that might be causing this? Does anyone have any suggestions on how I might identify the root cause of this issue? I am hoping someone with more knowledge about eXist-db might be able to suggest a direction in which I should look for answers. I don’t have any clue at the moment of what is causing this issue.

Nick
 
2017-06-07 18:32:01,747 [DefaultQuartzScheduler_Worker-1] INFO  (NativeBroker.java [sync]:3594) - Memory: 4,700,160K total; 5,592,576K max; 1,221,496K free
2017-06-07 18:32:01,747 [DefaultQuartzScheduler_Worker-1] INFO  (BTree.java [printStatistics]:2540) - dom.dbx INDEX Buffers occupation : 100% (64 out of 64) Cache efficiency : 100%
2017-06-07 18:32:01,747 [DefaultQuartzScheduler_Worker-1] INFO  (DOMFile.java [printStatistics]:1358) - dom.dbx DATA Buffers occupation : 100% (256 out of 256) Cache efficiency : 26%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO  (BTree.java [printStatistics]:2540) - collections.dbx INDEX Buffers occupation : 72% (104 out of 144) Cache efficiency : 99%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO  (BFile.java [printStatistics]:433) - collections.dbx DATA Buffers occupation : 88% (266 out of 303) Cache efficiency : 98%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO  (BTree.java [printStatistics]:2540) - values.dbx INDEX Buffers occupation : 2% (1 out of 64) Cache efficiency : 100%
2017-06-07 18:32:01,748 [DefaultQuartzScheduler_Worker-1] INFO  (BFile.java [printStatistics]:433) - values.dbx DATA Buffers occupation : 0% (0 out of 64) Cache efficiency : N/A
2017-06-07 18:32:24,247 [DefaultQuartzScheduler_Worker-2] INFO  (BrokerPool.java [setReadOnly]:1333) - Switching to read-only mode!!!
2017-06-07 18:32:25,349 [DefaultQuartzScheduler_Worker-3] ERROR (FilterInputStreamCacheFactory.java [instantiate]:81) - null
java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedConstructorAccessor118.newInstance(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_65]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_65]
        at org.exist.util.io.FilterInputStreamCacheFactory.instantiate(FilterInputStreamCacheFactory.java:72) [exist.jar:?]
        at org.exist.util.io.FilterInputStreamCacheFactory.getCacheInstance(FilterInputStreamCacheFactory.java:60) [exist.jar:?]
        at org.exist.xquery.value.BinaryValueFromInputStream.<init>(BinaryValueFromInputStream.java:33) [exist.jar:?]
        at org.exist.xquery.value.BinaryValueFromInputStream.getInstance(BinaryValueFromInputStream.java:51) [exist.jar:?]
        at org.expath.httpclient.model.exist.EXistResult.add(EXistResult.java:97) [exist-expath-20130805.jar:?]
        at org.expath.httpclient.impl.BinaryResponseBody.<init>(BinaryResponseBody.java:45) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at org.expath.httpclient.impl.BodyFactory.makeResponsePart(BodyFactory.java:105) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at org.expath.httpclient.impl.BodyFactory.makeResponseBody(BodyFactory.java:85) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at org.expath.httpclient.impl.HttpRequestImpl.send(HttpRequestImpl.java:63) [http-client-java-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at org.expath.exist.SendRequestFunction.sendOnce(SendRequestFunction.java:178) [exist-expath-20130805.jar:?]
        at org.expath.exist.SendRequestFunction.sendRequest(SendRequestFunction.java:151) [exist-expath-20130805.jar:?]
        at org.expath.exist.SendRequestFunction.eval(SendRequestFunction.java:133) [exist-expath-20130805.jar:?]
        at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
        at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.ForExpr.processItem(ForExpr.java:238) [exist.jar:?]
        at org.exist.xquery.ForExpr.eval(ForExpr.java:178) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
        at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
        at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
        at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:209) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.execute(Eval.java:442) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.doEval(Eval.java:396) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.eval(Eval.java:225) [exist.jar:?]
        at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
        at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
        at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
        at org.exist.xquery.TryCatchExpression.eval(TryCatchExpression.java:127) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:209) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.execute(Eval.java:442) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.doEval(Eval.java:396) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.eval(Eval.java:225) [exist.jar:?]
        at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
        at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.FunctionCall$DeferredFunctionCallImpl.execute(FunctionCall.java:427) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.realize(DeferredFunctionCall.java:57) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:198) [exist.jar:?]
        at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:76) [exist.jar:?]
        at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:200) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:195) [exist.jar:?]
        at org.exist.scheduler.UserXQueryJob.execute(UserXQueryJob.java:217) [exist.jar:?]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.1.jar:?]
Caused by: java.nio.file.FileSystemException: ../../../tools/jetty/tmp/_mmtfm_3c7e7f9b-9900-4444-9f03-a9d850829b0c1632133065744495093/mmtf_14968603453487918292215225476280.tmp: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:1.8.0_65]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_65]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_65]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:1.8.0_65]
        at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_65]
        at java.nio.file.Files.createFile(Files.java:632) ~[?:1.8.0_65]
        at java.nio.file.TempFileHelper.create(TempFileHelper.java:138) ~[?:1.8.0_65]
        at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161) ~[?:1.8.0_65]
        at java.nio.file.Files.createTempFile(Files.java:852) ~[?:1.8.0_65]
        at org.exist.util.io.TemporaryFileManager.getTemporaryFile(TemporaryFileManager.java:95) ~[exist.jar:?]
        at org.exist.util.io.FileFilterInputStreamCache.<init>(FileFilterInputStreamCache.java:61) ~[exist.jar:?]
        at org.exist.util.io.FileFilterInputStreamCache.<init>(FileFilterInputStreamCache.java:55) ~[exist.jar:?]
        ... 145 more
2017-06-07 18:32:25,494 [DefaultQuartzScheduler_Worker-3] ERROR (XMLDBCreateCollection.java [evalWithCollection]:87) - Unable to create new collection 20170607
org.xmldb.api.base.XMLDBException: database is read-only
        at org.exist.xmldb.function.LocalXmldbFunction.apply(LocalXmldbFunction.java:48) ~[exist.jar:?]
        at org.exist.xmldb.AbstractLocal.withDb(AbstractLocal.java:193) ~[exist.jar:?]
        at org.exist.xmldb.LocalCollectionManagementService.createCollection(LocalCollectionManagementService.java:81) ~[exist.jar:?]
        at org.exist.xmldb.LocalCollectionManagementService.createCollection(LocalCollectionManagementService.java:71) ~[exist.jar:?]
        at org.exist.xmldb.LocalCollectionManagementService.createCollection(LocalCollectionManagementService.java:60) ~[exist.jar:?]
        at org.exist.xquery.functions.xmldb.XMLDBAbstractCollectionManipulator.createCollection(XMLDBAbstractCollectionManipulator.java:199) ~[exist.jar:?]
        at org.exist.xquery.functions.xmldb.XMLDBAbstractCollectionManipulator.createCollectionPath(XMLDBAbstractCollectionManipulator.java:209) ~[exist.jar:?]
        at org.exist.xquery.functions.xmldb.XMLDBCreateCollection.evalWithCollection(XMLDBCreateCollection.java:79) [exist.jar:?]
        at org.exist.xquery.functions.xmldb.XMLDBAbstractCollectionManipulator.eval(XMLDBAbstractCollectionManipulator.java:174) [exist.jar:?]
        at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
        at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:102) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:74) [exist.jar:?]
        at org.exist.xquery.Atomize.eval(Atomize.java:67) [exist.jar:?]
        at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
        at org.exist.xquery.TryCatchExpression.eval(TryCatchExpression.java:127) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:65) [exist.jar:?]
        at org.exist.xquery.ForExpr.processItem(ForExpr.java:238) [exist.jar:?]
        at org.exist.xquery.ForExpr.eval(ForExpr.java:178) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.ConditionalExpression.eval(ConditionalExpression.java:100) [exist.jar:?]
        at org.exist.xquery.DebuggableExpression.eval(DebuggableExpression.java:58) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:209) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.execute(Eval.java:442) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.doEval(Eval.java:396) [exist.jar:?]
        at org.exist.xquery.functions.util.Eval.eval(Eval.java:225) [exist.jar:?]
        at org.exist.xquery.BasicFunction.eval(BasicFunction.java:70) [exist.jar:?]
        at org.exist.xquery.InternalFunctionCall.eval(InternalFunctionCall.java:56) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.UserDefinedFunction.eval(UserDefinedFunction.java:155) [exist.jar:?]
        at org.exist.xquery.FunctionCall$DeferredFunctionCallImpl.execute(FunctionCall.java:427) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.realize(DeferredFunctionCall.java:57) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:198) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
        at org.exist.xquery.DeferredFunctionCall.isEmpty(DeferredFunctionCall.java:199) [exist.jar:?]
        at org.exist.xquery.DynamicCardinalityCheck.eval(DynamicCardinalityCheck.java:76) [exist.jar:?]
        at org.exist.xquery.DynamicTypeCheck.eval(DynamicTypeCheck.java:61) [exist.jar:?]
        at org.exist.xquery.FunctionCall.evalFunction(FunctionCall.java:330) [exist.jar:?]
        at org.exist.xquery.FunctionCall.eval(FunctionCall.java:234) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:99) [exist.jar:?]
        at org.exist.xquery.LetExpr.eval(LetExpr.java:111) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.PathExpr.eval(PathExpr.java:267) [exist.jar:?]
        at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:280) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:200) [exist.jar:?]
        at org.exist.xquery.XQuery.execute(XQuery.java:195) [exist.jar:?]
        at org.exist.scheduler.UserXQueryJob.execute(UserXQueryJob.java:217) [exist.jar:?]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.1.jar:?]
Caused by: org.exist.security.PermissionDeniedException: database is read-only
        at org.exist.storage.NativeBroker.getOrCreateCollectionExplicit(NativeBroker.java:738) ~[exist.jar:?]
        at org.exist.storage.NativeBroker.getOrCreateCollection(NativeBroker.java:664) ~[exist.jar:?]
        at org.exist.xmldb.LocalCollectionManagementService.lambda$createCollection$234(LocalCollectionManagementService.java:83) ~[exist.jar:?]
        at org.exist.xmldb.function.LocalXmldbFunction.apply(LocalXmldbFunction.java:46) ~[exist.jar:?]
        ... 88 more
2017-06-07 18:34:04,247 [DefaultQuartzScheduler_Worker-4] FATAL (Journal.java [clearBackupFiles]:381) - Could not clear journal backup files
java.nio.file.FileSystemException: /mnt/data/exist-3.0-RC/tools/wrapper/bin/../../../webapp/WEB-INF/data/fs.journal: Too many open files


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: Switching to read-only mode!!!

Joe Wicentowski
Hi Nick,

You might find the troubleshooting techniques described in this thread
from exist-open from Oct. 2015 useful:

  http://markmail.org/message/cuwcfbnh6hg5vvwj

See also this article (mentioned in the same thread):

  https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors.html

One cause of exceeding the limit on open files can be failing to close
HTTP connections made via eXist's native httpclient or EXPath HTTP
Client modules.  Can you check your HTTP requests and make sure you're
including the "Connection: Close" header?

Joe

> I am running eXist-db 3.0 RC1 with a Linux AWS OS on a production server. Recently, I have been having to needing to restart it manually once a day or it will eventually switch to read-only on me... I can see that eXist-db is complaining that there are too many files open in jetty/tmp.
> ...
> 2017-06-07 18:34:04,247 [DefaultQuartzScheduler_Worker-4] FATAL (Journal.java [clearBackupFiles]:381) - Could not clear journal backup files
> java.nio.file.FileSystemException: /mnt/data/exist-3.0-RC/tools/wrapper/bin/../../../webapp/WEB-INF/data/fs.journal: Too many open files

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: Switching to read-only mode!!!

nsincaglia
Joe,

Thanks for the suggestion. We are using the EXPath HTTP Client Module but we are not including the Connection:close header. That is something I will implement and I will report back if it corrects the issue.

Nick
Reply | Threaded
Open this post in threaded view
|

Re: Switching to read-only mode!!!

nsincaglia
I just wanted to report back on this issue. We added the Connection: Close header to all of our Expath Http-Client requests and after monitoring it for several days now, it appears to have corrected the issue, which we are very happy about. Thanks for the suggestion!!!

What I find puzzling is why this is not documented anywhere. I looked through the EXPath HTTP-Client Module documentation and I don't see any references at all to the Connection:Close header.

I am also trying to understand why the default is to keep the connection open. How would one even use a previous http request in XQuery? There is no handle or reference one can use, I don't think. I am wondering why the default is to not close the connection?

Does anyone have any insights on this?

Nick

Reply | Threaded
Open this post in threaded view
|

Re: Switching to read-only mode!!!

Adam Retter
Would anyone be interested in volunteering to add a PR to the documentation repo to mention this?

On 20 June 2017 at 11:46, nsincaglia <[hidden email]> wrote:
I just wanted to report back on this issue. We added the Connection: Close
header to all of our Expath Http-Client requests and after monitoring it for
several days now, it appears to have corrected the issue, which we are very
happy about. Thanks for the suggestion!!!

What I find puzzling is why this is not documented anywhere. I looked
through the EXPath HTTP-Client Module documentation and I don't see any
references at all to the Connection:Close header.

I am also trying to understand why the default is to keep the connection
open. How would one even use a previous http request in XQuery? There is no
handle or reference one can use, I don't think. I am wondering why the
default is to not close the connection?

Does anyone have any insights on this?

Nick





--
View this message in context: http://exist.2174344.n4.nabble.com/Switching-to-read-only-mode-tp4672288p4672338.html
Sent from the exist-open mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open