pagination

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

pagination

Eduard Drenth
Dear all,

Does exist-db implement or promote pagination techniques or best practices?

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Dannes Wessels-3
Context please?

Regards

Dannes

--
www.exist-db.org

> On 6 Jun 2017, at 10:17, Eduard Drenth <[hidden email]> wrote:
>
> Does exist-db implement or promote pagination techniques or best practices?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Eduard Drenth
Hi,

It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.

- large collections
- many xquery queries with lots of results
- pagination for performance and stability

input: limit and offset
output: limit, offset, total

I hope this is context enough? Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Dannes Wessels <[hidden email]> on behalf of Dannes Wessels <[hidden email]>
Sent: Tuesday, June 6, 2017 12:12 PM
To: Eduard Drenth
Cc: [hidden email]
Subject: Re: [Exist-open] pagination

Context please?

Regards

Dannes

--
www.exist-db.org

> On 6 Jun 2017, at 10:17, Eduard Drenth <[hidden email]> wrote:
>
> Does exist-db implement or promote pagination techniques or best practices?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Joe Wicentowski
In reply to this post by Eduard Drenth
Hi Eduard,

> Does exist-db implement or promote pagination techniques or best practices?

Sure, many eXist apps implement pagination of query results.  Could
you be more specific about what you have in mind?

Joe

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Eduard Drenth
Hi,

Suppose I have a query yielding lots of results and I do this:

mylib:somequery($query)[$start to $end]

I want to prevent that each time first the query will be executed, then the result set will be built and after this it will be filtered.

Is there a way to achieve this, or is the 'engine' smart enough to optimize?

A seperate subject is that in the response I would like to see a total (which may imply executing an extra count query).

In SQL you have LIMIT and OFFSET. Totals in SQL is often achieved by executing a count() query.

Regards,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Joe Wicentowski <[hidden email]>
Sent: Tuesday, June 6, 2017 3:59 PM
To: Eduard Drenth
Cc: [hidden email]
Subject: Re: [Exist-open] pagination

Hi Eduard,

> Does exist-db implement or promote pagination techniques or best practices?

Sure, many eXist apps implement pagination of query results.  Could
you be more specific about what you have in mind?

Joe

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Joe Wicentowski
Hi Eduard,

eXide stores the results of the most recently executed query in a
request attribute.  For the source code showing this, see
https://github.com/wolfgangmm/eXide/blob/develop/controller.xql#L164-L230
and https://github.com/wolfgangmm/eXide/blob/develop/modules/session.xql#L38-L102.
The relevant functions are session:set-attribute() and
session:get-attribute(); see
http://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/session#get-attribute.1
and http://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/session#set-attribute.2.
eXide uses URL rewriting and MVC pipeline architecture to facilitate
storage/retrieval of these with the set-attribute action; see
http://exist-db.org/exist/apps/doc/urlrewrite.xml#D1.2.8.

Also you might find the cache module useful.  It's not enabled by
default, but you can uncomment this line in your conf.xml file to
enable it: https://github.com/eXist-db/exist/blob/develop/conf.xml.tmpl#L915.

Joe

On Tue, Jun 6, 2017 at 11:00 AM, Eduard Drenth
<[hidden email]> wrote:

> Hi,
>
> Suppose I have a query yielding lots of results and I do this:
>
> mylib:somequery($query)[$start to $end]
>
> I want to prevent that each time first the query will be executed, then the result set will be built and after this it will be filtered.
>
> Is there a way to achieve this, or is the 'engine' smart enough to optimize?
>
> A seperate subject is that in the response I would like to see a total (which may imply executing an extra count query).
>
> In SQL you have LIMIT and OFFSET. Totals in SQL is often achieved by executing a count() query.
>
> Regards,
>
> Eduard Drenth, Software Architekt
>
> [hidden email]
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> +31 58 234 30 47
>
> gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43
>
> ________________________________________
> From: Joe Wicentowski <[hidden email]>
> Sent: Tuesday, June 6, 2017 3:59 PM
> To: Eduard Drenth
> Cc: [hidden email]
> Subject: Re: [Exist-open] pagination
>
> Hi Eduard,
>
>> Does exist-db implement or promote pagination techniques or best practices?
>
> Sure, many eXist apps implement pagination of query results.  Could
> you be more specific about what you have in mind?
>
> Joe

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Loren Cahlander
In reply to this post by Eduard Drenth
You can search the XQuery WikiBook for examples on pagination:



On Jun 6, 2017, at 6:23 AM, Eduard Drenth <[hidden email]> wrote:

Hi,

It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.

- large collections
- many xquery queries with lots of results
- pagination for performance and stability

input: limit and offset
output: limit, offset, total

I hope this is context enough? Bye,

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Dannes Wessels <[hidden email]> on behalf of Dannes Wessels <[hidden email]>
Sent: Tuesday, June 6, 2017 12:12 PM
To: Eduard Drenth
Cc: [hidden email]
Subject: Re: [Exist-open] pagination

Context please?

Regards

Dannes

--
www.exist-db.org

On 6 Jun 2017, at 10:17, Eduard Drenth <[hidden email]> wrote:

Does exist-db implement or promote pagination techniques or best practices?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Wolfgang Meier-2
In reply to this post by Eduard Drenth
> It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.

As a rule of thumb, paginate immediately after executing a query to build the result sequence. Do not apply any post-processing, but extract the items to be shown on the current page, using e.g. fn:subsequence($result, $offset, $max), store the rest into the HTTP session with session:set-attribute and call post-processing to format the output only on the subsequence. Background: within eXist, references to nodes stored inside the database do not consume a lot of memory as they are just pointers. However, if you post-process and format those results, you create in-memory XML fragments which do consume memory.

Cheers,

Wolfgang
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Eduard Drenth
Thanks, good to know about the pointers also. Keeping the results in session means that the first call may take a while, subsequent calls directly address the results in the session. Am I right?

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Wolfgang Meier <[hidden email]>
Sent: Wednesday, June 7, 2017 3:33 PM
To: Eduard Drenth
Cc: Dannes Wessels; [hidden email]
Subject: Re: [Exist-open] pagination

> It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.

As a rule of thumb, paginate immediately after executing a query to build the result sequence. Do not apply any post-processing, but extract the items to be shown on the current page, using e.g. fn:subsequence($result, $offset, $max), store the rest into the HTTP session with session:set-attribute and call post-processing to format the output only on the subsequence. Background: within eXist, references to nodes stored inside the database do not consume a lot of memory as they are just pointers. However, if you post-process and format those results, you create in-memory XML fragments which do consume memory.

Cheers,

Wolfgang

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: pagination

Wolfgang Meier-2
> Thanks, good to know about the pointers also. Keeping the results in session means that the first call may take a while, subsequent calls directly address the results in the session. Am I right?

Yes, that’s right.

Wolfgang
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open