length limit on constructed sequences?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

length limit on constructed sequences?

ron.vandenbranden
Hi all,

Is there a length limit on sequences eXist can construct? For both
eXist-1.4.x and eXist-trunk, I notice a stack overflow error when the
length of a constructed sequence grows too large. For example, a query like:

let $a := concat('(',
   string-join(
     for $i in 1 to 1000
     return concat('"',$i,'"')
   , ',')
, ')')

return util:eval($a)


This logs following error when run in the Java Client (eXist-trunk):

2011-01-31 11:31:38,864 [eXistThread-25] DEBUG (RpcConnection.java [handleException]:113) -
java.lang.StackOverflowError
     at java.util.ArrayList.get(ArrayList.java:322)
     at java.util.AbstractList$Itr.next(AbstractList.java:345)
     at org.exist.xquery.PathExpr.getDependencies(PathExpr.java:392)
     at org.exist.xquery.PathExpr.getDependencies(PathExpr.java:393)
     at org.exist.xquery.PathExpr.getDependencies(PathExpr.java:393)
     at org.exist.xquery.PathExpr.getDependencies(PathExpr.java:393)
     at org.exist.xquery.PathExpr.getDependencies(PathExpr.java:393)
     at org.exist.xquery.PathExpr.getDependencies(PathExpr.java:393)
     at org.exist.xquery.PathExpr.eval(PathExpr.java:214)
     at org.exist.xquery.SequenceConstructor.eval(SequenceConstructor.java:81)
     at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:59)
     ...


Some observations:
     -the size limit approximates 1000 items, though it differs between
1.4.x (error from 936 items) and trunk (error from 924 items). I'm not
too sure, however, if there are small differences between XQuery runs
(so maybe it's memory/system -related?)
     -the error seems to be triggered by the number of items in the
sequence, rather than their individual length: no difference if the
sequence items are longer or shorter strings
     -the error seems limited to sequence construction, as eXist clearly
can handle larger sequences that have been computed (through node
retrieval, e.g.)

Kind regards,

Ron

--
Ron Van den Branden
Wetenschappelijk attaché / Senior Researcher

Centrum voor Teksteditie en Bronnenstudie - CTB (KANTL)
Centre for Scholarly Editing and Document Studies
Koninklijke Academie voor Nederlandse Taal- en Letterkunde
Royal Academy of Dutch Language and Literature
Koningstraat 18 / b-9000 Gent / Belgium
tel: +32 9 265 93 51 / fax: +32 9 265 93 49
E-mail : [hidden email]
www.kantl.be/ctb


------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: length limit on constructed sequences?

Andrew Welch
> java.lang.StackOverflowError

>From a purely Java perspective, you can increase the memory available
for the stack using the -Xss switch.  For example,

-Xss1m




--
Andrew Welch
http://andrewjwelch.com

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: length limit on constructed sequences?

Wolfgang Meier-2

The problem here is how the xquery parser deals with the sequence constructor: it recursively translates the expression into nested pairs. We have to add a normalization step to flatten this. I'll put it on my todo.

Wolfgang


------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: length limit on constructed sequences?

Wolfgang Meier-2
> The problem here is how the xquery parser deals with the sequence
> constructor: it recursively translates the expression into nested pairs. We
> have to add a normalization step to flatten this. I'll put it on my todo.

Apart from causing stack overflows, sequence constructors did consume
too much memory. I fixed this in trunk yesterday. There should be no
limit to the length of the sequence now.

Wolfgang

------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: length limit on constructed sequences?

Stefan Majewski
On 01/03/11 16:32, Wolfgang Meier wrote:
> Apart from causing stack overflows, sequence constructors did consume
> too much memory. I fixed this in trunk yesterday. There should be no
> limit to the length of the sequence now.

that's great news! thank you. We have been struggling with similar
stack-overflows when processing sequences of tokens for linguistic
analysis for quite a while, now.

cheers and thanks,
Stefan

--
| Stefan Majewski  | Department of English, University of Vienna     |
| VOICE Corpus     | Spitalgasse 2-4, Universitätscampus AAKH, Hof 8 |
|                  | A-1090 Vienna                                   |
| Research Ass.(IT)| Phone: +43 1 4277 424 46                        |


------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open