Quantcast

IO exception: temporary file can't be found

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

IO exception: temporary file can't be found

ron.vandenbranden
Hi,

I'm deploying an eXist-3.0RC1 WAR build inside Tomcat-7.0.52 on an
Ubuntu-12.0.4 box. I'm using this setup to serve a webapp which serves a
corpus of PDF files whose text contents are being extracted and indexed
for searching. Since new files are being added when they're ready for
publication, a trigger has been set up that for each PDF file that's
being uploaded, performing following tasks:
     -extract the text contents to an XHTML file via the
contentextraction module
     -further processes that file with XSLT to filter out unwanted noise
from the index
     -store the resulting index-ready XHTML file in the database

In principle, this works fine. Only, every so often, after uploading a
bunch of PDF files with the Java Admin client, the webapp is throwing
errors for a number of pages. For example, a call to the dashboard returns:

   <exception>
     <path>/db/apps/dashboard/modules/view.xql</path>
     <message>exerr:ERROR An IO exception occurred:
       /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
       (No such file or directory) [at line 36, column 17]
     </message>
   </exception>

Also, pages in my app that make use of request:get-data() fail with a
similar error:

   <exception>
     <path>/db/apps/tw-ovl/tw-ovl_site/xquery/get-data.xq</path>
     <message>exerr:ERROR An IO exception occurred:
       /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
       (No such file or directory) [at line 3, column 18]
     </message>
   </exception>

Subsequent requests to these pages slightly reduce the error message to:

   <exception>
     <path>/db/apps/dashboard/modules/view.xql</path>
     <message>exerr:ERROR An IO exception occurred: No such file or directory [at line 36, column 17]</message>
   </exception>

Indeed, I see no *.tmp file of that name at said location; only one
other temporary folder (_mmtfm_610866a9-02d7-44fd-a1e1-c65662bb8196),
which is empty.

The only solution is to restart Tomcat, after which everything runs fine
again, until another PDF upload triggers the next error.

Since handling of PDF files is somehow involved, I don't know if this
might be related to the settings for binary-manager in conf.xml:

   <binary-manager>
     <cache class="org.exist.util.io.FileFilterInputStreamCache"/>
   </binary-manager>

I'll add what seems a relevant snippet of the exist.log file, which I
think displays the error occurring when uploading a PDF file.

I'm aware lots of variables are involved, but does anyone have a clue
what might cause these temporary file locations or their references to
get lost, and how to avoid this?

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

ioexception.log (14K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

Jonathan Rowell

Hi Ron,


have a look and see if there are a lot of open files (LOF command?)

I have this issue on Windows after using binary-doc()


Jonathan




From: Ron Van den Branden <[hidden email]>
Sent: Wednesday, March 22, 2017 10:37 AM
To: eXist open
Subject: [Exist-open] IO exception: temporary file can't be found
 
Hi,

I'm deploying an eXist-3.0RC1 WAR build inside Tomcat-7.0.52 on an
Ubuntu-12.0.4 box. I'm using this setup to serve a webapp which serves a
corpus of PDF files whose text contents are being extracted and indexed
for searching. Since new files are being added when they're ready for
publication, a trigger has been set up that for each PDF file that's
being uploaded, performing following tasks:
     -extract the text contents to an XHTML file via the
contentextraction module
     -further processes that file with XSLT to filter out unwanted noise
from the index
     -store the resulting index-ready XHTML file in the database

In principle, this works fine. Only, every so often, after uploading a
bunch of PDF files with the Java Admin client, the webapp is throwing
errors for a number of pages. For example, a call to the dashboard returns:

   <exception>
     <path>/db/apps/dashboard/modules/view.xql</path>
     <message>exerr:ERROR An IO exception occurred:
       /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
       (No such file or directory) [at line 36, column 17]
     </message>
   </exception>

Also, pages in my app that make use of request:get-data() fail with a
similar error:

   <exception>
     <path>/db/apps/tw-ovl/tw-ovl_site/xquery/get-data.xq</path>
     <message>exerr:ERROR An IO exception occurred:
       /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
       (No such file or directory) [at line 3, column 18]
     </message>
   </exception>

Subsequent requests to these pages slightly reduce the error message to:

   <exception>
     <path>/db/apps/dashboard/modules/view.xql</path>
     <message>exerr:ERROR An IO exception occurred: No such file or directory [at line 36, column 17]</message>
   </exception>

Indeed, I see no *.tmp file of that name at said location; only one
other temporary folder (_mmtfm_610866a9-02d7-44fd-a1e1-c65662bb8196),
which is empty.

The only solution is to restart Tomcat, after which everything runs fine
again, until another PDF upload triggers the next error.

Since handling of PDF files is somehow involved, I don't know if this
might be related to the settings for binary-manager in conf.xml:

   <binary-manager>
     <cache class="org.exist.util.io.FileFilterInputStreamCache"/>
   </binary-manager>

I'll add what seems a relevant snippet of the exist.log file, which I
think displays the error occurring when uploading a PDF file.

I'm aware lots of variables are involved, but does anyone have a clue
what might cause these temporary file locations or their references to
get lost, and how to avoid this?

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

ron.vandenbranden

Hi Jonathan,


From past experience (http://markmail.org/message/r6tjidvwahwi5i5e) I had remembered that Windows would always be a bit trickier w.r.t. temporary files management. So I'm surprised to see this on Linux, and hence also my suspicion that the Binary Manager may be involved.


On 22/03/2017 12:45, Jonathan Rowell wrote:

have a look and see if there are a lot of open files (LOF command?)



Thanks for your suggestion. I'm not sure this is what you mean: "lsof | wc -l" returns a number around 7000. No clue if that is much or little?

The most concrete clues I have are that binary files and memory (the XSLT processing triggered after each upload) are involved, and that the entire webapp is affected.

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

Jonathan Rowell

Hi Ron,


I wrote "LOF" with a question mark because I can't remember how ListOpenFiles works on Linux, since I program exclusively on Windows. I think wc is word count and assuming that lsof returns just the path and which does not contain spaces, it would mean you have 7000 open files. That strikes me as rather a lot, whether that is enough to cause file store problems I don't know. But in the old days when I worked on SCO Unix that would have brought the house down.


regards


Jonathan





From: Ron Van den Branden <[hidden email]>
Sent: Wednesday, March 22, 2017 12:38 PM
To: Jonathan Rowell; eXist open
Subject: Re: [Exist-open] IO exception: temporary file can't be found
 

Hi Jonathan,


From past experience (http://markmail.org/message/r6tjidvwahwi5i5e) I had remembered that Windows would always be a bit trickier w.r.t. temporary files management. So I'm surprised to see this on Linux, and hence also my suspicion that the Binary Manager may be involved.


On 22/03/2017 12:45, Jonathan Rowell wrote:

have a look and see if there are a lot of open files (LOF command?)



Thanks for your suggestion. I'm not sure this is what you mean: "lsof | wc -l" returns a number around 7000. No clue if that is much or little?

The most concrete clues I have are that binary files and memory (the XSLT processing triggered after each upload) are involved, and that the entire webapp is affected.

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

ron.vandenbranden

Hi Jonathan,


Well, yes, this is the number of open files on the entire server (serving a couple of websites and Plesk a.s.o.), so I don't know how relevant this is. If I trim down the list to this specific webapp, I only get a list of 29 files.


Best,


Ron


On 22/03/2017 14:14, Jonathan Rowell wrote:

Hi Ron,


I wrote "LOF" with a question mark because I can't remember how ListOpenFiles works on Linux, since I program exclusively on Windows. I think wc is word count and assuming that lsof returns just the path and which does not contain spaces, it would mean you have 7000 open files. That strikes me as rather a lot, whether that is enough to cause file store problems I don't know. But in the old days when I worked on SCO Unix that would have brought the house down.


regards


Jonathan





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

ron.vandenbranden
In reply to this post by ron.vandenbranden
Sigh,

This is killing my app to the point that it's becoming completely
unstable: the error can occur at any time.

Is anyone else seeing this, or know a solution/workaround?

Ron


On 22/03/2017 11:37, Ron Van den Branden wrote:

> Hi,
>
> I'm deploying an eXist-3.0RC1 WAR build inside Tomcat-7.0.52 on an
> Ubuntu-12.0.4 box. I'm using this setup to serve a webapp which serves
> a corpus of PDF files whose text contents are being extracted and
> indexed for searching. Since new files are being added when they're
> ready for publication, a trigger has been set up that for each PDF
> file that's being uploaded, performing following tasks:
>     -extract the text contents to an XHTML file via the
> contentextraction module
>     -further processes that file with XSLT to filter out unwanted
> noise from the index
>     -store the resulting index-ready XHTML file in the database
>
> In principle, this works fine. Only, every so often, after uploading a
> bunch of PDF files with the Java Admin client, the webapp is throwing
> errors for a number of pages. For example, a call to the dashboard
> returns:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred:
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 36, column 17]
>     </message>
>   </exception>
>
> Also, pages in my app that make use of request:get-data() fail with a
> similar error:
>
>   <exception>
> <path>/db/apps/tw-ovl/tw-ovl_site/xquery/get-data.xq</path>
>     <message>exerr:ERROR An IO exception occurred:
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 3, column 18]
>     </message>
>   </exception>
>
> Subsequent requests to these pages slightly reduce the error message to:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred: No such file or
> directory [at line 36, column 17]</message>
>   </exception>
>
> Indeed, I see no *.tmp file of that name at said location; only one
> other temporary folder (_mmtfm_610866a9-02d7-44fd-a1e1-c65662bb8196),
> which is empty.
>
> The only solution is to restart Tomcat, after which everything runs
> fine again, until another PDF upload triggers the next error.
>
> Since handling of PDF files is somehow involved, I don't know if this
> might be related to the settings for binary-manager in conf.xml:
>
>   <binary-manager>
>     <cache class="org.exist.util.io.FileFilterInputStreamCache"/>
>   </binary-manager>
>
> I'll add what seems a relevant snippet of the exist.log file, which I
> think displays the error occurring when uploading a PDF file.
>
> I'm aware lots of variables are involved, but does anyone have a clue
> what might cause these temporary file locations or their references to
> get lost, and how to avoid this?
>
> Best,
>
> Ron


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

Joe Wicentowski
Ron,

Ugh, sorry to hear you're facing such a serious problem.  In the past the lsof (unix file limit) issue has hit me a couple of times.  7,000 open file handles sounds very high to me, and this could certainly be a cause of system seizures, though I'm not an expert.  Let me paste in some info from past troubleshooting sessions with various gurus:

1. You can usually see the limits *for the current user* by running:

  ulimit -n

On most unix systems the limits can be configured in /etc/security/limits.conf.  There can be soft and hard limits.  

I'd suggest figuring out what limits are in effect on your user account, the account running exist / your other apps, and make a note of this.

2. By grepping through the results of lsof you can analyze what service is using up the available file handles:

  lsof
  sudo lsof
  sudo lsof | grep TCP
  sudo lsof | grep exist | wc -l

Tracking the growth in these results after a restart can tell you what's using file handles, the rate of growth of usage, and correlations between usage and instability.  

I'd suggest finding out what's eating up the file handles and proceeding from there.  Ideally this will help narrow down the # of variables you are looking at.

Joe


On Wed, Mar 22, 2017 at 8:07 PM, Ron Van den Branden <[hidden email]> wrote:
Sigh,

This is killing my app to the point that it's becoming completely
unstable: the error can occur at any time.

Is anyone else seeing this, or know a solution/workaround?

Ron


On 22/03/2017 11:37, Ron Van den Branden wrote:
> Hi,
>
> I'm deploying an eXist-3.0RC1 WAR build inside Tomcat-7.0.52 on an
> Ubuntu-12.0.4 box. I'm using this setup to serve a webapp which serves
> a corpus of PDF files whose text contents are being extracted and
> indexed for searching. Since new files are being added when they're
> ready for publication, a trigger has been set up that for each PDF
> file that's being uploaded, performing following tasks:
>     -extract the text contents to an XHTML file via the
> contentextraction module
>     -further processes that file with XSLT to filter out unwanted
> noise from the index
>     -store the resulting index-ready XHTML file in the database
>
> In principle, this works fine. Only, every so often, after uploading a
> bunch of PDF files with the Java Admin client, the webapp is throwing
> errors for a number of pages. For example, a call to the dashboard
> returns:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred:
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 36, column 17]
>     </message>
>   </exception>
>
> Also, pages in my app that make use of request:get-data() fail with a
> similar error:
>
>   <exception>
> <path>/db/apps/tw-ovl/tw-ovl_site/xquery/get-data.xq</path>
>     <message>exerr:ERROR An IO exception occurred:
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 3, column 18]
>     </message>
>   </exception>
>
> Subsequent requests to these pages slightly reduce the error message to:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred: No such file or
> directory [at line 36, column 17]</message>
>   </exception>
>
> Indeed, I see no *.tmp file of that name at said location; only one
> other temporary folder (_mmtfm_610866a9-02d7-44fd-a1e1-c65662bb8196),
> which is empty.
>
> The only solution is to restart Tomcat, after which everything runs
> fine again, until another PDF upload triggers the next error.
>
> Since handling of PDF files is somehow involved, I don't know if this
> might be related to the settings for binary-manager in conf.xml:
>
>   <binary-manager>
>     <cache class="org.exist.util.io.FileFilterInputStreamCache"/>
>   </binary-manager>
>
> I'll add what seems a relevant snippet of the exist.log file, which I
> think displays the error occurring when uploading a PDF file.
>
> I'm aware lots of variables are involved, but does anyone have a clue
> what might cause these temporary file locations or their references to
> get lost, and how to avoid this?
>
> Best,
>
> Ron


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

Joe Wicentowski
Also, Ron, sorry if I've missed it, but which edition and version of Java are you using?

On Thu, Mar 23, 2017 at 12:38 AM, Joe Wicentowski <[hidden email]> wrote:
Ron,

Ugh, sorry to hear you're facing such a serious problem.  In the past the lsof (unix file limit) issue has hit me a couple of times.  7,000 open file handles sounds very high to me, and this could certainly be a cause of system seizures, though I'm not an expert.  Let me paste in some info from past troubleshooting sessions with various gurus:

1. You can usually see the limits *for the current user* by running:

  ulimit -n

On most unix systems the limits can be configured in /etc/security/limits.conf.  There can be soft and hard limits.  

I'd suggest figuring out what limits are in effect on your user account, the account running exist / your other apps, and make a note of this.

2. By grepping through the results of lsof you can analyze what service is using up the available file handles:

  lsof
  sudo lsof
  sudo lsof | grep TCP
  sudo lsof | grep exist | wc -l

Tracking the growth in these results after a restart can tell you what's using file handles, the rate of growth of usage, and correlations between usage and instability.  

I'd suggest finding out what's eating up the file handles and proceeding from there.  Ideally this will help narrow down the # of variables you are looking at.

Joe


On Wed, Mar 22, 2017 at 8:07 PM, Ron Van den Branden <[hidden email]> wrote:
Sigh,

This is killing my app to the point that it's becoming completely
unstable: the error can occur at any time.

Is anyone else seeing this, or know a solution/workaround?

Ron


On 22/03/2017 11:37, Ron Van den Branden wrote:
> Hi,
>
> I'm deploying an eXist-3.0RC1 WAR build inside Tomcat-7.0.52 on an
> Ubuntu-12.0.4 box. I'm using this setup to serve a webapp which serves
> a corpus of PDF files whose text contents are being extracted and
> indexed for searching. Since new files are being added when they're
> ready for publication, a trigger has been set up that for each PDF
> file that's being uploaded, performing following tasks:
>     -extract the text contents to an XHTML file via the
> contentextraction module
>     -further processes that file with XSLT to filter out unwanted
> noise from the index
>     -store the resulting index-ready XHTML file in the database
>
> In principle, this works fine. Only, every so often, after uploading a
> bunch of PDF files with the Java Admin client, the webapp is throwing
> errors for a number of pages. For example, a call to the dashboard
> returns:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred:
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 36, column 17]
>     </message>
>   </exception>
>
> Also, pages in my app that make use of request:get-data() fail with a
> similar error:
>
>   <exception>
> <path>/db/apps/tw-ovl/tw-ovl_site/xquery/get-data.xq</path>
>     <message>exerr:ERROR An IO exception occurred:
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 3, column 18]
>     </message>
>   </exception>
>
> Subsequent requests to these pages slightly reduce the error message to:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred: No such file or
> directory [at line 36, column 17]</message>
>   </exception>
>
> Indeed, I see no *.tmp file of that name at said location; only one
> other temporary folder (_mmtfm_610866a9-02d7-44fd-a1e1-c65662bb8196),
> which is empty.
>
> The only solution is to restart Tomcat, after which everything runs
> fine again, until another PDF upload triggers the next error.
>
> Since handling of PDF files is somehow involved, I don't know if this
> might be related to the settings for binary-manager in conf.xml:
>
>   <binary-manager>
>     <cache class="org.exist.util.io.FileFilterInputStreamCache"/>
>   </binary-manager>
>
> I'll add what seems a relevant snippet of the exist.log file, which I
> think displays the error occurring when uploading a PDF file.
>
> I'm aware lots of variables are involved, but does anyone have a clue
> what might cause these temporary file locations or their references to
> get lost, and how to avoid this?
>
> Best,
>
> Ron


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

ron.vandenbranden
Hi Joe,

On 23/03/2017 5:54, Joe Wicentowski wrote:
> Also, Ron, sorry if I've missed it, but which edition and version of
> Java are you using?

I'm running Oracle Java Development Kit 1.8.0_60:

   java version "1.8.0_60"
   Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
   Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

Thanks for the Linux pointers; I'm a novice on that side of the sysadmin
planet, and it will take some time (and a deep breath) to dive into
those issues. Fortunately, the front side of the affected app keeps
running (if I work around methods like request:get-data()), only the
administrative apps like dashboard and eXide become inaccessible until
restart.

Still, it strikes me that:
     -only this webapp is affected; for the other eXist webapps running
in the same Tomcat server, everything (including dashboard + eXide)
keeps working fine
     -binary (PDF) files are involved, which rings alarms of previous issues
     -it /seems/, though I'm not entirely sure yet, that the issue is
not so much triggered by retrieving the PDF files from the db, but
rather by manipulating them (uploading and content extraction)
So I'm still willing to believe this might be an eXist issue in the
field of binary file management. I'll definitely try the other settings
of binary-manager in conf.xml (viz.
org.exist.util.io.MemoryMappedFileFilterInputStreamCache and
org.exist.util.io.MemoryFilterInputStreamCache), and see if that has any
effect.

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

ron.vandenbranden
Weird,

I've just installed eXist-3.1.1 on my local machine (Windows-7 Pro,
64bit), was testing some (super-lightweight) things with some test apps
on my file system, and suddenly, when accessing
http://localhost:8080/exist/apps/dashboard/index.html:

     <exception>
       <path>/db/apps/dashboard/modules/view.xql</path>
       <message>exerr:ERROR An IO exception occurred: Could not load cache for class: org.exist.util.io.FileFilterInputStreamCache [at line 36, column 17]</message>
     </exception>

To avoid confusion: while I'm hitting the same issue, the execution
context is totally unrelated: different machines, different OS,
different eXist versions, and different operations (this time, no binary
files were involved at all). I think the only common factor is the use
of request:get-data() in the dashboard and eXide apps (which method is
explicitly mentioned in the comment section for the binary-manager in
conf.xml:
https://github.com/eXist-db/exist/blob/develop/conf.xml.tmpl#L366)...

Don't know what to think, and haven't found a way to reproduce this,
perhaps others are encountering similar issues?

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

Joe Wicentowski
In reply to this post by ron.vandenbranden
Hi Ron,

> So I'm still willing to believe this might be an eXist issue in the field of binary file management. I'll definitely try the other settings of binary-manager in conf.xml (viz. org.exist.util.io.MemoryMappedFileFilterInputStreamCache and org.exist.util.io.MemoryFilterInputStreamCache), and see if that has any effect.

It certainly could be.  On history.state.gov, we're not doing much
uploading (except via WebDAV) or content extraction, so we may not be
encountering what you are.  But we are working with the util module's
functions for dealing with binary docs, converting between binary and
string, etc., when handling binary/text files or HTTP requests for
binary/text resources.  For reference, we're using the default
binary-manager in conf.xml, and that's on Amazon Linux (a CentOS
variant as I recall).  And we haven't had any lsof issues in a long
time.

Joe

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

Adam Retter
In reply to this post by ron.vandenbranden
> , a trigger has been set up that for each PDF file that's being
> uploaded, performing following tasks:
>     -extract the text contents to an XHTML file via the contentextraction
> module
>     -further processes that file with XSLT to filter out unwanted noise from
> the index
>     -store the resulting index-ready XHTML file in the database
>
> In principle, this works fine. Only, every so often, after uploading a bunch
> of PDF files with the Java Admin client, the webapp is throwing errors for a
> number of pages. For example, a call to the dashboard returns:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred:
>
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 36, column 17]
>     </message>
>   </exception>
>
> Also, pages in my app that make use of request:get-data() fail with a
> similar error:
>
>   <exception>
>     <path>/db/apps/tw-ovl/tw-ovl_site/xquery/get-data.xq</path>
>     <message>exerr:ERROR An IO exception occurred:
>
> /vhosts/existapps/temp/_mmtfm_9af18d53-c0a0-4676-8051-adab514290ec/mmtf_14900938406111947230095861080508.tmp
>       (No such file or directory) [at line 3, column 18]
>     </message>
>   </exception>
>
> Subsequent requests to these pages slightly reduce the error message to:
>
>   <exception>
>     <path>/db/apps/dashboard/modules/view.xql</path>
>     <message>exerr:ERROR An IO exception occurred: No such file or directory
> [at line 36, column 17]</message>
>   </exception>
>
> Indeed, I see no *.tmp file of that name at said location; only one other
> temporary folder (_mmtfm_610866a9-02d7-44fd-a1e1-c65662bb8196), which is
> empty.

Ron,

The binary file manager tries to make it efficient to reuse the
content of binary files.

When you upload a file via HTTP, Java presents you with an
InputStream, that input stream can only be read once (because it might
be backed by a network socket for example).

The BinaryFileManager attempts to transparently make InputStreams look
like they can be read more than once. It does that by caching the
contents to a temporary file on disk when the InputStream is first
read. When you attempt to read it a second time, the BinaryFileManager
reads the contents from the temporary file instead of the network
socket (or whatever the original transient source was). This is
tricky, as we have to know when we are finished with a BinaryFile so
that we can release its contents.

I see that you are using 3.0RC1, there were some fixes in 3.0 final
around binary files and also a leak of cached binary file chunks was
fixed in 3.1.1.
I don't know if it will fix the problem you describe, but could you
test again against 3.1.1? Then we will at least have a solid base line
to proceed from.


--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

ron.vandenbranden
Hi Adam,

Thanks for the explanation.

On 25/03/2017 15:06, Adam Retter wrote:
> I see that you are using 3.0RC1, there were some fixes in 3.0 final
> around binary files and also a leak of cached binary file chunks was
> fixed in 3.1.1.
> I don't know if it will fix the problem you describe, but could you
> test again against 3.1.1? Then we will at least have a solid base line
> to proceed from.

Ok, I'll set up a test app running eXist-3.1.1 (for the production app
I'm sticking to 3.0RC1 until some other issues with eXist-3.1.1 have
been resolved) and see what happens.

For the production webapp, I'll try and see if other binary file manager
settings have any effect.

I'm hoping to be able to report back soon,

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

ron.vandenbranden
In reply to this post by Adam Retter
Hi again,

On 25/03/2017 15:06, Adam Retter wrote:
> I see that you are using 3.0RC1, there were some fixes in 3.0 final
> around binary files and also a leak of cached binary file chunks was
> fixed in 3.1.1.
> I don't know if it will fix the problem you describe, but could you
> test again against 3.1.1? Then we will at least have a solid base line
> to proceed from.
>

I don't see much difference with 3.1.1: this version, too, starts to
throw the IO exception. Unfortunately, I haven't been able to isolate an
exact trigger: sometimes it happens right after start-up; sometimes it
takes some time before the error comes up. I'm not even sure anymore
whether retrieval of binary files is a necessary error condition; I do
notice, however, that anything using request:get-data() is affected
(hence, the dashboard and eXide webapps).

I'm fully aware it may be a complicating factor that I have a single
Tomcat container serving multiple eXist webapps. Contrary to what I said
earlier, it seems that once one webapp is hitting the error, all of them
are.

Yet, it seems that changing the class for binary manager in conf.xml to:

   <binary-manager>
     <cache class="org.exist.util.io.MemoryMappedFileFilterInputStreamCache"/>
   </binary-manager>

...seems to avoid the error (for eXist versions from 3.0 onwards). Which
is quite a relief ;-).

So, although the exact circumstances are unclear, there seems to be an
indication that org.exist.util.io.FileFilterInputStreamCache is somehow
'vulnerable' to a condition triggering an IO exception. I hope this can
contribute at least something...

Many thanks for all helpful replies I've received!

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IO exception: temporary file can't be found

Adam Retter
> I don't see much difference with 3.1.1: this version, too, starts to throw
> the IO exception. Unfortunately, I haven't been able to isolate an exact
> trigger: sometimes it happens right after start-up; sometimes it takes some
> time before the error comes up. I'm not even sure anymore whether retrieval
> of binary files is a necessary error condition; I do notice, however, that
> anything using request:get-data() is affected (hence, the dashboard and
> eXide webapps).
>
> I'm fully aware it may be a complicating factor that I have a single Tomcat
> container serving multiple eXist webapps. Contrary to what I said earlier,
> it seems that once one webapp is hitting the error, all of them are.

The TemporaryFileManager is a Java Singleton, which means that there
will only be one of these per JVM. So running multiple eXist's in the
same JVM (i.e. Tomcat) will cause them to all use a single
TemporaryFileManager. So if you get a problem, they will all see the
problem.


> Yet, it seems that changing the class for binary manager in conf.xml to:
>
>   <binary-manager>
>     <cache
> class="org.exist.util.io.MemoryMappedFileFilterInputStreamCache"/>
>   </binary-manager>
>
> ...seems to avoid the error (for eXist versions from 3.0 onwards). Which is
> quite a relief ;-).

I would be careful here. In the past issues were reported around
MemoryMappedFileFilterInputStreamCache, which is why
FileFilterInputStreamCache is now the default in conf.xml.
MemoryMappedFileFilterInputStreamCache used to be the default and
theoretically offers better performance.


> So, although the exact circumstances are unclear, there seems to be an
> indication that org.exist.util.io.FileFilterInputStreamCache is somehow
> 'vulnerable' to a condition triggering an IO exception. I hope this can
> contribute at least something...

I would really love to get a mechanism for reproducing your issues, so
that we could solve them...


--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Loading...