Quantcast

performance of teipublisher

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

performance of teipublisher

Eduard Drenth
Dear all,

I notice that searching for a string in 1 document of 1mb using teipublisher takes 12 seconds to show me a kwic result.

The problem is not the database, the index query returns very fast.

Profiling statistics of the functions show me (sorry for the layout):

templates:call /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36] 49 12.71
templates:process /home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9] 3 12.71
templates:call-by-introspection /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28] 64 12.707
templates:call-with-args /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13] 65 12.678
templates:apply /db/apps/tdb/modules/view.xql [49:5] 1 12.617
pages:app-root /db/apps/tdb/modules/view.xql [217:18] 1 12.611
templates:process /db/apps/tdb/modules/lib/pages.xql [482:9] 1 12.611
intl:translate /db/apps/tdb/modules/view.xql [221:18] 1 12.608
templates:process /db/apps/tdb/modules/lib/i18n-templates.xql [33:13] 2 12.585
templates:process-output /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9] 64 12.584
templates:process-output /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9] 35 12.54
search:show-hits /db/apps/tdb/modules/view.xql [223:18] 1 12.528
kwic:get-summary /db/apps/tdb/modules/lib/search.xql [221:13] 5 12.318
kwic:get-summary jar:file:/home/eduard/eXist-db/exist.jar!/or[...]kwic.xql [208:9] 5 12.318
kwic:truncate-following jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [250:32] 5 6.34
kwic:truncate-previous jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [244:27] 5 5.978
util:expand /db/apps/tdb/modules/lib/search.xql [200:22] 1 0.207

>From this I cannot identify the problem. Can this somehw be optimized?

Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: performance of teipublisher

Eduard Drenth
Dear all,

Does anyone else experience performance issues in teipublisher search? A while ago I send the mail below.

We want to go into production with teipublisher in the comming period, but hesitate because of performance.

I notice that searching in 1 document of 1mb using teipublisher takes 12 seconds to show me the kwic result page.

The problem is not the database, the index query returns very fast.

Profiling statistics of the functions show me the figures below (sorry for the layout, last column is the seconds):

templates:call  /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]       49      12.71
templates:process       /home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]       3       12.71
templates:call-by-introspection /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]       64      12.707
templates:call-with-args        /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]       65      12.678
templates:apply /db/apps/tdb/modules/view.xql [49:5]    1       12.617
pages:app-root  /db/apps/tdb/modules/view.xql [217:18]  1       12.611
templates:process       /db/apps/tdb/modules/lib/pages.xql [482:9]      1       12.611
intl:translate  /db/apps/tdb/modules/view.xql [221:18]  1       12.608
templates:process       /db/apps/tdb/modules/lib/i18n-templates.xql [33:13]     2       12.585
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]       64      12.584
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]       35      12.54
search:show-hits        /db/apps/tdb/modules/view.xql [223:18]  1       12.528
kwic:get-summary        /db/apps/tdb/modules/lib/search.xql [221:13]    5       12.318
kwic:get-summary        jar:file:/home/eduard/eXist-db/exist.jar!/or[...]kwic.xql [208:9]       5       12.318
kwic:truncate-following jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [250:32]       5       6.34
kwic:truncate-previous  jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [244:27]       5       5.978
util:expand     /db/apps/tdb/modules/lib/search.xql [200:22]    1       0.207

>From this I cannot identify the problem. Can this somehow be optimized?

Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: performance of teipublisher

Andreas Wagner

Hi Eduard, hi list,

I am also looking forward to seeing how other people use the profiling logs.

From my - limited - understanding the functions kwic:get-summary and kwic:truncate-* could make a difference. Is it possible that, just in order to test this, you run the same search without the kwic functions?

On the other hand, I noticed that as the very last function listed, there is an util:eval which has caused me quite some headaches in the past.
@Everyone: the times listed in the log, are they describing when the function is entered or when it's done?

Best,

Andreas

Von meinem Smartphone gesendet

--
Dr. Andreas Wagner
Project "The School of Salamanca"
Academy of Sciences and Literature, Mainz
and Institute of Philosophy
Goethe University Frankfurt
http://salamanca.adwmainz.de

IGF HP 25 / R 2.455
Norbert-Wollheim-Platz 1
60629 Frankfurt/Main

Tel +49 69/798-32774
Fax +49 69/798-32794



---- Eduard Drenth schrieb ----

Dear all,

Does anyone else experience performance issues in teipublisher search? A while ago I send the mail below.

We want to go into production with teipublisher in the comming period, but hesitate because of performance.

I notice that searching in 1 document of 1mb using teipublisher takes 12 seconds to show me the kwic result page.

The problem is not the database, the index query returns very fast.

Profiling statistics of the functions show me the figures below (sorry for the layout, last column is the seconds):

templates:call  /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]       49      <a href="tel:12.71">12.71
templates:process       /home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]       3       <a href="tel:12.71">12.71
templates:call-by-introspection /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]       64      <a href="tel:12.707">12.707
templates:call-with-args        /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]       65      <a href="tel:12.678">12.678
templates:apply /db/apps/tdb/modules/view.xql [49:5]    1       <a href="tel:12.617">12.617
pages:app-root  /db/apps/tdb/modules/view.xql [217:18]  1       <a href="tel:12.611">12.611
templates:process       /db/apps/tdb/modules/lib/pages.xql [482:9]      1       <a href="tel:12.611">12.611
intl:translate  /db/apps/tdb/modules/view.xql [221:18]  1       <a href="tel:12.608">12.608
templates:process       /db/apps/tdb/modules/lib/i18n-templates.xql [33:13]     2       <a href="tel:12.585">12.585
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]       64      <a href="tel:12.584">12.584
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]       35      <a href="tel:12.54">12.54
search:show-hits        /db/apps/tdb/modules/view.xql [223:18]  1       <a href="tel:12.528">12.528
kwic:get-summary        /db/apps/tdb/modules/lib/search.xql [221:13]    5       <a href="tel:12.318">12.318
kwic:get-summary        jar:file:/home/eduard/eXist-db/exist.jar!/or[...]kwic.xql [208:9]       5       <a href="tel:12.318">12.318
kwic:truncate-following jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [250:32]       5       6.34
kwic:truncate-previous  jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [244:27]       5       <a href="tel:5.978">5.978
util:expand     /db/apps/tdb/modules/lib/search.xql [200:22]    1       <a href="tel:0.207">0.207

>From this I cannot identify the problem. Can this somehow be optimized?

Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
<a href="tel:+31 58 234 30 47">+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: performance of teipublisher

Eduard Drenth

Dear all


It looks like kwic:get-summary is expensive, especially with large documents, see tests below. Doubling the amount of memory doesn't make a difference.


figures when I let show-hits in search.xql return $expanded (browser processing is the bottleneck now):


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]312.108templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]22.108templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]462.107templates:apply/db/apps/tdb/modules/view.xql [49:5]12.06templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]471.987pages:app-root/db/apps/tdb/modules/view.xql [217:18]11.95templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]11.822intl:translate/db/apps/tdb/modules/view.xql [221:18]11.818templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]21.292templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.677templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.551i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.47i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.443search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.306util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.3exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.219collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.149collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.149templates:apply/db/apps/monex/modules/view.xql [43:5]10.049i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.027



figures when I let show-hits in search.xql return <span>test</span>:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.894templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.894templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.892templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.879templates:apply/db/apps/tdb/modules/view.xql [49:5]10.847pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.841templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.841intl:translate/db/apps/tdb/modules/view.xql [221:18]10.838i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.456i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.432templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.424templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.368templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.346search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.333util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.327exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.216collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.146collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.145templates:apply/db/apps/monex/modules/view.xql [43:5]10.047i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.024


figures with original search.xql, but large document deleted:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.583templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.583templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.582templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.564templates:apply/db/apps/tdb/modules/view.xql [49:5]10.525pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.52templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.52intl:translate/db/apps/tdb/modules/view.xql [221:18]10.516i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.452i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.427exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.211collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.144collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.144templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.105templates:apply/db/apps/monex/modules/view.xql [43:5]10.058templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.056templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.031i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.025templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [271:17]210.022search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.018



Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Andreas Wagner <[hidden email]>
Sent: Friday, March 31, 2017 8:00 AM
To: eXist DB ML; Eduard Drenth
Subject: AW: Re: [Exist-open] performance of teipublisher
 

Hi Eduard, hi list,

I am also looking forward to seeing how other people use the profiling logs.

From my - limited - understanding the functions kwic:get-summary and kwic:truncate-* could make a difference. Is it possible that, just in order to test this, you run the same search without the kwic functions?

On the other hand, I noticed that as the very last function listed, there is an util:eval which has caused me quite some headaches in the past.
@Everyone: the times listed in the log, are they describing when the function is entered or when it's done?

Best,

Andreas

Von meinem Smartphone gesendet

--
Dr. Andreas Wagner
Project "The School of Salamanca"
Academy of Sciences and Literature, Mainz
and Institute of Philosophy
Goethe University Frankfurt
http://salamanca.adwmainz.de

IGF HP 25 / R 2.455
Norbert-Wollheim-Platz 1
60629 Frankfurt/Main

Tel +49 69/798-32774
Fax +49 69/798-32794



---- Eduard Drenth schrieb ----

Dear all,

Does anyone else experience performance issues in teipublisher search? A while ago I send the mail below.

We want to go into production with teipublisher in the comming period, but hesitate because of performance.

I notice that searching in 1 document of 1mb using teipublisher takes 12 seconds to show me the kwic result page.

The problem is not the database, the index query returns very fast.

Profiling statistics of the functions show me the figures below (sorry for the layout, last column is the seconds):

templates:call  /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]       49      <a href="tel:12.71">12.71
templates:process       /home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]       3       <a href="tel:12.71">12.71
templates:call-by-introspection /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]       64      <a href="tel:12.707">12.707
templates:call-with-args        /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]       65      <a href="tel:12.678">12.678
templates:apply /db/apps/tdb/modules/view.xql [49:5]    1       <a href="tel:12.617">12.617
pages:app-root  /db/apps/tdb/modules/view.xql [217:18]  1       <a href="tel:12.611">12.611
templates:process       /db/apps/tdb/modules/lib/pages.xql [482:9]      1       <a href="tel:12.611">12.611
intl:translate  /db/apps/tdb/modules/view.xql [221:18]  1       <a href="tel:12.608">12.608
templates:process       /db/apps/tdb/modules/lib/i18n-templates.xql [33:13]     2       <a href="tel:12.585">12.585
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]       64      <a href="tel:12.584">12.584
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]       35      <a href="tel:12.54">12.54
search:show-hits        /db/apps/tdb/modules/view.xql [223:18]  1       <a href="tel:12.528">12.528
kwic:get-summary        /db/apps/tdb/modules/lib/search.xql [221:13]    5       <a href="tel:12.318">12.318
kwic:get-summary        jar:file:/home/eduard/eXist-db/exist.jar!/or[...]kwic.xql [208:9]       5       <a href="tel:12.318">12.318
kwic:truncate-following jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [250:32]       5       6.34
kwic:truncate-previous  jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [244:27]       5       <a href="tel:5.978">5.978
util:expand     /db/apps/tdb/modules/lib/search.xql [200:22]    1       <a href="tel:0.207">0.207

>From this I cannot identify the problem. Can this somehow be optimized?

Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
<a href="tel:&#43;31 58 234 30 47">+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: performance of teipublisher

Eduard Drenth

complementary: caching has a big influence!


When I refresh (F5) to search again it takes just as long as a fresh search.


When I search again via the search box it takes less then half a second.


But of course this only means caching works like a charm


Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Eduard Drenth <[hidden email]>
Sent: Friday, March 31, 2017 9:36 AM
To: Andreas Wagner; eXist DB ML
Subject: Re: [Exist-open] performance of teipublisher
 

Dear all


It looks like kwic:get-summary is expensive, especially with large documents, see tests below. Doubling the amount of memory doesn't make a difference.


figures when I let show-hits in search.xql return $expanded (browser processing is the bottleneck now):


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]312.108templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]22.108templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]462.107templates:apply/db/apps/tdb/modules/view.xql [49:5]12.06templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]471.987pages:app-root/db/apps/tdb/modules/view.xql [217:18]11.95templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]11.822intl:translate/db/apps/tdb/modules/view.xql [221:18]11.818templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]21.292templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.677templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.551i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.47i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.443search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.306util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.3exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.219collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.149collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.149templates:apply/db/apps/monex/modules/view.xql [43:5]10.049i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.027



figures when I let show-hits in search.xql return <span>test</span>:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.894templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.894templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.892templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.879templates:apply/db/apps/tdb/modules/view.xql [49:5]10.847pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.841templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.841intl:translate/db/apps/tdb/modules/view.xql [221:18]10.838i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.456i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.432templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.424templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.368templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.346search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.333util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.327exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.216collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.146collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.145templates:apply/db/apps/monex/modules/view.xql [43:5]10.047i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.024


figures with original search.xql, but large document deleted:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.583templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.583templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.582templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.564templates:apply/db/apps/tdb/modules/view.xql [49:5]10.525pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.52templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.52intl:translate/db/apps/tdb/modules/view.xql [221:18]10.516i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.452i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.427exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.211collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.144collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.144templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.105templates:apply/db/apps/monex/modules/view.xql [43:5]10.058templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.056templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.031i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.025templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [271:17]210.022search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.018



Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Andreas Wagner <[hidden email]>
Sent: Friday, March 31, 2017 8:00 AM
To: eXist DB ML; Eduard Drenth
Subject: AW: Re: [Exist-open] performance of teipublisher
 

Hi Eduard, hi list,

I am also looking forward to seeing how other people use the profiling logs.

From my - limited - understanding the functions kwic:get-summary and kwic:truncate-* could make a difference. Is it possible that, just in order to test this, you run the same search without the kwic functions?

On the other hand, I noticed that as the very last function listed, there is an util:eval which has caused me quite some headaches in the past.
@Everyone: the times listed in the log, are they describing when the function is entered or when it's done?

Best,

Andreas

Von meinem Smartphone gesendet

--
Dr. Andreas Wagner
Project "The School of Salamanca"
Academy of Sciences and Literature, Mainz
and Institute of Philosophy
Goethe University Frankfurt
http://salamanca.adwmainz.de

IGF HP 25 / R 2.455
Norbert-Wollheim-Platz 1
60629 Frankfurt/Main

Tel +49 69/798-32774
Fax +49 69/798-32794



---- Eduard Drenth schrieb ----

Dear all,

Does anyone else experience performance issues in teipublisher search? A while ago I send the mail below.

We want to go into production with teipublisher in the comming period, but hesitate because of performance.

I notice that searching in 1 document of 1mb using teipublisher takes 12 seconds to show me the kwic result page.

The problem is not the database, the index query returns very fast.

Profiling statistics of the functions show me the figures below (sorry for the layout, last column is the seconds):

templates:call  /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]       49      <a href="tel:12.71">12.71
templates:process       /home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]       3       <a href="tel:12.71">12.71
templates:call-by-introspection /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]       64      <a href="tel:12.707">12.707
templates:call-with-args        /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]       65      <a href="tel:12.678">12.678
templates:apply /db/apps/tdb/modules/view.xql [49:5]    1       <a href="tel:12.617">12.617
pages:app-root  /db/apps/tdb/modules/view.xql [217:18]  1       <a href="tel:12.611">12.611
templates:process       /db/apps/tdb/modules/lib/pages.xql [482:9]      1       <a href="tel:12.611">12.611
intl:translate  /db/apps/tdb/modules/view.xql [221:18]  1       <a href="tel:12.608">12.608
templates:process       /db/apps/tdb/modules/lib/i18n-templates.xql [33:13]     2       <a href="tel:12.585">12.585
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]       64      <a href="tel:12.584">12.584
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]       35      <a href="tel:12.54">12.54
search:show-hits        /db/apps/tdb/modules/view.xql [223:18]  1       <a href="tel:12.528">12.528
kwic:get-summary        /db/apps/tdb/modules/lib/search.xql [221:13]    5       <a href="tel:12.318">12.318
kwic:get-summary        jar:file:/home/eduard/eXist-db/exist.jar!/or[...]kwic.xql [208:9]       5       <a href="tel:12.318">12.318
kwic:truncate-following jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [250:32]       5       6.34
kwic:truncate-previous  jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [244:27]       5       <a href="tel:5.978">5.978
util:expand     /db/apps/tdb/modules/lib/search.xql [200:22]    1       <a href="tel:0.207">0.207

>From this I cannot identify the problem. Can this somehow be optimized?

Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
<a href="tel:&#43;31 58 234 30 47">+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: performance of teipublisher

Eduard Drenth

Optimization in kwic.xql for large contexts?


These two lines might be very expensive for large contexts (did not test):

let $next := $node/following::text()[1] intersect $root//text()

let $next := $node/preceding::text()[1] intersect $root//text()

Maybe all text() calls can be made only once and passed in as arguments when recursing?


Maybe the length of the result of the text() calls can be limited because we already know how far we will look back and forward.


What do you think?


Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Eduard Drenth <[hidden email]>
Sent: Friday, March 31, 2017 9:49 AM
To: Andreas Wagner; eXist DB ML
Subject: Re: [Exist-open] performance of teipublisher
 

complementary: caching has a big influence!


When I refresh (F5) to search again it takes just as long as a fresh search.


When I search again via the search box it takes less then half a second.


But of course this only means caching works like a charm


Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Eduard Drenth <[hidden email]>
Sent: Friday, March 31, 2017 9:36 AM
To: Andreas Wagner; eXist DB ML
Subject: Re: [Exist-open] performance of teipublisher
 

Dear all


It looks like kwic:get-summary is expensive, especially with large documents, see tests below. Doubling the amount of memory doesn't make a difference.


figures when I let show-hits in search.xql return $expanded (browser processing is the bottleneck now):


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]312.108templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]22.108templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]462.107templates:apply/db/apps/tdb/modules/view.xql [49:5]12.06templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]471.987pages:app-root/db/apps/tdb/modules/view.xql [217:18]11.95templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]11.822intl:translate/db/apps/tdb/modules/view.xql [221:18]11.818templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]21.292templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.677templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.551i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.47i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.443search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.306util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.3exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.219collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.149collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.149templates:apply/db/apps/monex/modules/view.xql [43:5]10.049i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.027



figures when I let show-hits in search.xql return <span>test</span>:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.894templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.894templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.892templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.879templates:apply/db/apps/tdb/modules/view.xql [49:5]10.847pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.841templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.841intl:translate/db/apps/tdb/modules/view.xql [221:18]10.838i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.456i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.432templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.424templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.368templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.346search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.333util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.327exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.216collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.146collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.145templates:apply/db/apps/monex/modules/view.xql [43:5]10.047i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.024


figures with original search.xql, but large document deleted:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.583templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.583templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.582templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.564templates:apply/db/apps/tdb/modules/view.xql [49:5]10.525pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.52templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.52intl:translate/db/apps/tdb/modules/view.xql [221:18]10.516i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.452i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.427exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.211collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.144collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.144templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.105templates:apply/db/apps/monex/modules/view.xql [43:5]10.058templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.056templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.031i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.025templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [271:17]210.022search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.018



Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Andreas Wagner <[hidden email]>
Sent: Friday, March 31, 2017 8:00 AM
To: eXist DB ML; Eduard Drenth
Subject: AW: Re: [Exist-open] performance of teipublisher
 

Hi Eduard, hi list,

I am also looking forward to seeing how other people use the profiling logs.

From my - limited - understanding the functions kwic:get-summary and kwic:truncate-* could make a difference. Is it possible that, just in order to test this, you run the same search without the kwic functions?

On the other hand, I noticed that as the very last function listed, there is an util:eval which has caused me quite some headaches in the past.
@Everyone: the times listed in the log, are they describing when the function is entered or when it's done?

Best,

Andreas

Von meinem Smartphone gesendet

--
Dr. Andreas Wagner
Project "The School of Salamanca"
Academy of Sciences and Literature, Mainz
and Institute of Philosophy
Goethe University Frankfurt
http://salamanca.adwmainz.de

IGF HP 25 / R 2.455
Norbert-Wollheim-Platz 1
60629 Frankfurt/Main

Tel +49 69/798-32774
Fax +49 69/798-32794



---- Eduard Drenth schrieb ----

Dear all,

Does anyone else experience performance issues in teipublisher search? A while ago I send the mail below.

We want to go into production with teipublisher in the comming period, but hesitate because of performance.

I notice that searching in 1 document of 1mb using teipublisher takes 12 seconds to show me the kwic result page.

The problem is not the database, the index query returns very fast.

Profiling statistics of the functions show me the figures below (sorry for the layout, last column is the seconds):

templates:call  /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]       49      <a href="tel:12.71">12.71
templates:process       /home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]       3       <a href="tel:12.71">12.71
templates:call-by-introspection /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]       64      <a href="tel:12.707">12.707
templates:call-with-args        /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]       65      <a href="tel:12.678">12.678
templates:apply /db/apps/tdb/modules/view.xql [49:5]    1       <a href="tel:12.617">12.617
pages:app-root  /db/apps/tdb/modules/view.xql [217:18]  1       <a href="tel:12.611">12.611
templates:process       /db/apps/tdb/modules/lib/pages.xql [482:9]      1       <a href="tel:12.611">12.611
intl:translate  /db/apps/tdb/modules/view.xql [221:18]  1       <a href="tel:12.608">12.608
templates:process       /db/apps/tdb/modules/lib/i18n-templates.xql [33:13]     2       <a href="tel:12.585">12.585
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]       64      <a href="tel:12.584">12.584
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]       35      <a href="tel:12.54">12.54
search:show-hits        /db/apps/tdb/modules/view.xql [223:18]  1       <a href="tel:12.528">12.528
kwic:get-summary        /db/apps/tdb/modules/lib/search.xql [221:13]    5       <a href="tel:12.318">12.318
kwic:get-summary        jar:file:/home/eduard/eXist-db/exist.jar!/or[...]kwic.xql [208:9]       5       <a href="tel:12.318">12.318
kwic:truncate-following jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [250:32]       5       6.34
kwic:truncate-previous  jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [244:27]       5       <a href="tel:5.978">5.978
util:expand     /db/apps/tdb/modules/lib/search.xql [200:22]    1       <a href="tel:0.207">0.207

>From this I cannot identify the problem. Can this somehow be optimized?

Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
<a href="tel:&#43;31 58 234 30 47">+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: performance of teipublisher

Eduard Drenth

tested with my own poor mans kwic, 5 times as fast. It isn't a generic solution, but may be of use:


declare function kwic:get-summary($root as node(), $node as element(), 
$config as element(config)?,
$callback as (function(text(), xs:string) as text()?)?
) as element() {
let $table := $config/@table = ('yes', 'true')
let $debug-text-nodes := $config/@debug-text-nodes = ('yes', 'true')

let $prevTrunc := $node/preceding::text()[fn:position() < 10]
let $followingTrunc := $node/following::text()[fn:position() < 10]
return
if (not($table)) then
<p>
<span class="previous">...{$prevTrunc}</span>
{
if ($config/@link) then
<a class="hi" href="{$config/@link}">{$node/text()}</a>
else
<span class="hi">{kwic:output-text($debug-text-nodes, $node/text())}</span>
}
<span class="following">{$followingTrunc}...</span>
</p>
else
<tr>
<td class="previous">...{$prevTrunc}</td>
<td class="hi">
{
if ($config/@link) then
<a href="{$config/@link}">{$node/text()}</a>
else
kwic:output-text($debug-text-nodes, $node/text())
}
</td>
<td class="following">{$followingTrunc}...</td>
</tr>
};




Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Eduard Drenth <[hidden email]>
Sent: Friday, March 31, 2017 10:40 AM
To: Andreas Wagner; eXist DB ML
Subject: Re: [Exist-open] performance of teipublisher
 

Optimization in kwic.xql for large contexts?


These two lines might be very expensive for large contexts (did not test):

let $next := $node/following::text()[1] intersect $root//text()

let $next := $node/preceding::text()[1] intersect $root//text()

Maybe all text() calls can be made only once and passed in as arguments when recursing?


Maybe the length of the result of the text() calls can be limited because we already know how far we will look back and forward.


What do you think?


Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Eduard Drenth <[hidden email]>
Sent: Friday, March 31, 2017 9:49 AM
To: Andreas Wagner; eXist DB ML
Subject: Re: [Exist-open] performance of teipublisher
 

complementary: caching has a big influence!


When I refresh (F5) to search again it takes just as long as a fresh search.


When I search again via the search box it takes less then half a second.


But of course this only means caching works like a charm


Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Eduard Drenth <[hidden email]>
Sent: Friday, March 31, 2017 9:36 AM
To: Andreas Wagner; eXist DB ML
Subject: Re: [Exist-open] performance of teipublisher
 

Dear all


It looks like kwic:get-summary is expensive, especially with large documents, see tests below. Doubling the amount of memory doesn't make a difference.


figures when I let show-hits in search.xql return $expanded (browser processing is the bottleneck now):


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]312.108templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]22.108templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]462.107templates:apply/db/apps/tdb/modules/view.xql [49:5]12.06templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]471.987pages:app-root/db/apps/tdb/modules/view.xql [217:18]11.95templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]11.822intl:translate/db/apps/tdb/modules/view.xql [221:18]11.818templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]21.292templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.677templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.551i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.47i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.443search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.306util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.3exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.219collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.149collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.149templates:apply/db/apps/monex/modules/view.xql [43:5]10.049i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.027



figures when I let show-hits in search.xql return <span>test</span>:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.894templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.894templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.892templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.879templates:apply/db/apps/tdb/modules/view.xql [49:5]10.847pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.841templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.841intl:translate/db/apps/tdb/modules/view.xql [221:18]10.838i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.456i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.432templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.424templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.368templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.346search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.333util:expand/db/apps/tdb/modules/lib/search.xql [200:22]30.327exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.216collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.146collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.145templates:apply/db/apps/monex/modules/view.xql [43:5]10.047i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.024


figures with original search.xql, but large document deleted:


templates:call/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]310.583templates:process/home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]20.583templates:call-by-introspection/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]460.582templates:call-with-args/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]470.564templates:apply/db/apps/tdb/modules/view.xql [49:5]10.525pages:app-root/db/apps/tdb/modules/view.xql [217:18]10.52templates:process/db/apps/tdb/modules/lib/pages.xql [482:9]10.52intl:translate/db/apps/tdb/modules/view.xql [221:18]10.516i18n:process/db/apps/tdb/modules/lib/i18n-templates.xql [29:9]20.452i18n:getLanguageCollection/db/apps/tdb/modules/lib/i18n.xql [44:35]160.427exists/db/apps/tdb/modules/lib/i18n.xql [180:9]160.211collection/db/apps/tdb/modules/lib/i18n.xql [181:9]160.144collection/db/apps/tdb/modules/lib/i18n.xql [180:16]160.144templates:process/db/apps/tdb/modules/lib/i18n-templates.xql [33:13]20.105templates:apply/db/apps/monex/modules/view.xql [43:5]10.058templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]460.056templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]250.031i18n:process/db/apps/tdb/modules/lib/i18n.xql [46:13]160.025templates:process-output/home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [271:17]210.022search:show-hits/db/apps/tdb/modules/view.xql [223:18]10.018



Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Andreas Wagner <[hidden email]>
Sent: Friday, March 31, 2017 8:00 AM
To: eXist DB ML; Eduard Drenth
Subject: AW: Re: [Exist-open] performance of teipublisher
 

Hi Eduard, hi list,

I am also looking forward to seeing how other people use the profiling logs.

From my - limited - understanding the functions kwic:get-summary and kwic:truncate-* could make a difference. Is it possible that, just in order to test this, you run the same search without the kwic functions?

On the other hand, I noticed that as the very last function listed, there is an util:eval which has caused me quite some headaches in the past.
@Everyone: the times listed in the log, are they describing when the function is entered or when it's done?

Best,

Andreas

Von meinem Smartphone gesendet

--
Dr. Andreas Wagner
Project "The School of Salamanca"
Academy of Sciences and Literature, Mainz
and Institute of Philosophy
Goethe University Frankfurt
http://salamanca.adwmainz.de

IGF HP 25 / R 2.455
Norbert-Wollheim-Platz 1
60629 Frankfurt/Main

Tel +49 69/798-32774
Fax +49 69/798-32794



---- Eduard Drenth schrieb ----

Dear all,

Does anyone else experience performance issues in teipublisher search? A while ago I send the mail below.

We want to go into production with teipublisher in the comming period, but hesitate because of performance.

I notice that searching in 1 document of 1mb using teipublisher takes 12 seconds to show me the kwic result page.

The problem is not the database, the index query returns very fast.

Profiling statistics of the functions show me the figures below (sorry for the layout, last column is the seconds):

templates:call  /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [135:36]       49      <a href="tel:12.71">12.71
templates:process       /home/eduard/eXist-db/webapp/WEB-INF/dat[...]templates.xql [88:9]       3       <a href="tel:12.71">12.71
templates:call-by-introspection /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [187:28]       64      <a href="tel:12.707">12.707
templates:call-with-args        /home/eduard/eXist-db/webapp/WEB-INF/d[...]templates.xql [208:13]       65      <a href="tel:12.678">12.678
templates:apply /db/apps/tdb/modules/view.xql [49:5]    1       <a href="tel:12.617">12.617
pages:app-root  /db/apps/tdb/modules/view.xql [217:18]  1       <a href="tel:12.611">12.611
templates:process       /db/apps/tdb/modules/lib/pages.xql [482:9]      1       <a href="tel:12.611">12.611
intl:translate  /db/apps/tdb/modules/view.xql [221:18]  1       <a href="tel:12.608">12.608
templates:process       /db/apps/tdb/modules/lib/i18n-templates.xql [33:13]     2       <a href="tel:12.585">12.585
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [205:9]       64      <a href="tel:12.584">12.584
templates:process-output        /home/eduard/eXist-db/webapp/WEB-INF/da[...]templates.xql [273:9]       35      <a href="tel:12.54">12.54
search:show-hits        /db/apps/tdb/modules/view.xql [223:18]  1       <a href="tel:12.528">12.528
kwic:get-summary        /db/apps/tdb/modules/lib/search.xql [221:13]    5       <a href="tel:12.318">12.318
kwic:get-summary        jar:file:/home/eduard/eXist-db/exist.jar!/or[...]kwic.xql [208:9]       5       <a href="tel:12.318">12.318
kwic:truncate-following jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [250:32]       5       6.34
kwic:truncate-previous  jar:file:/home/eduard/eXist-db/exist.jar!/o[...]kwic.xql [244:27]       5       <a href="tel:5.978">5.978
util:expand     /db/apps/tdb/modules/lib/search.xql [200:22]    1       <a href="tel:0.207">0.207

>From this I cannot identify the problem. Can this somehow be optimized?

Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
<a href="tel:&#43;31 58 234 30 47">+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Loading...