Performance problems in eXist-db

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Performance problems in eXist-db

Felix Maas
Hello eXist community,
 
I've got dramatic problems with the performance in eXist-db.
My databased web application I created with eXist should manage projects. There are not millions of projects but 10000 or 20000 of them which should be stored, read and editted by the user (each project has about 100 tags). I wouldn't call that a much amount.
Well at the moment 400 projects are stored in the database (size of the data is about 2MB) and the performance is so bad that you cannot really use it. To read all the data into a table it takes about 5 or 6 seconds. But that is not the big problem. If I try to create a new project there is an error that it takes too much time to process the request. I find that very confusing because I do not even read the whole data, I just want to store a new xml file into the database and I cannot understand why that creates performance problems.
Also confusing to me is the fact that when I try to open the collection with the 400 projects in the Java Admin Client it also takes a lot of time. Actually when I navigate to a collection and I do not open one of the files no content of the data is read in. I guess it is just an overview about the data like e.g. in the windows explorer? So why does eXist have so much problems to read data which are not bigger than 2MB?
Also tried indexing (do not really know if I did it the right way) but without success.
 
Best regards,
Felix

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance problems in eXist-db

Adam Retter
Felix,

I am afraid you are going to have to help us to help you.

All you have really told us is that:

1) The application you have built is slow. 
2) You have 2MB of data, but you want to scale to ~ 50MB or 100MB of data.
3) Listing Collections in the Java Admin client can be slow.
4) You have tried indexing but without success.

You are going to have to be a lot more specific with your problems. eXist can certainly handle 100MB of data without blinking, in fact we have had reports of systems running with hundreds of gigabytes of data.

Listing the contents of a database which has hundreds of Collections using the query browser in the Java Admin Client, is known to be slow I am afraid as it tries to gather a list of all collections in the database up-front. You might have a better experience with eXide or the Atom Editor (https://github.com/wolfgangmm/atom-existdb).

You need to delve into the queries you have written to understand what is slow, and ask us about improving specific queries. This usually involves some rewriting of the queries to optimize them and also correct configuration and use of indexes.

You have not told us anything about what your queries are, which parts of them are slow, or what the problems were that you experienced with indexing. If you want to look into the performance of your queries and index use, then the Monex application is a good start as it allows you to profile such things.



On 18 July 2017 at 09:45, Felix Maas <[hidden email]> wrote:
Hello eXist community,
 
I've got dramatic problems with the performance in eXist-db.
My databased web application I created with eXist should manage projects. There are not millions of projects but 10000 or 20000 of them which should be stored, read and editted by the user (each project has about 100 tags). I wouldn't call that a much amount.
Well at the moment 400 projects are stored in the database (size of the data is about 2MB) and the performance is so bad that you cannot really use it. To read all the data into a table it takes about 5 or 6 seconds. But that is not the big problem. If I try to create a new project there is an error that it takes too much time to process the request. I find that very confusing because I do not even read the whole data, I just want to store a new xml file into the database and I cannot understand why that creates performance problems.
Also confusing to me is the fact that when I try to open the collection with the 400 projects in the Java Admin Client it also takes a lot of time. Actually when I navigate to a collection and I do not open one of the files no content of the data is read in. I guess it is just an overview about the data like e.g. in the windows explorer? So why does eXist have so much problems to read data which are not bigger than 2MB?
Also tried indexing (do not really know if I did it the right way) but without success.
 
Best regards,
Felix

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open




--
Adam Retter

eXist Core Developer
{ United Kingdom / United States }
[hidden email]

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Performance problems in eXist-db

محمد جواد حسنی
Hi Felix,
I use exist-db to implement a system with multi service and concurrent users. It's good if tune perfectly.
My experience about exist-db tuning abstracted as below:

1. fn:Collection speed reduced by number of documents in that collection. For optimizing, you should manage your collections hierarchy.
2. Manage users permission to limit access not necessary collection for any group of users. This reduced the number of documents.
3. Do not use fn:collection iterative. you can use let statement to put the result of fn:collection in a variable and query on it.
4. Use cache module to store the result of fn:collection temporarily. The querying on cached fragments is very quick.
5. Optimize your document structure according to query optimization rules.
6.Use range index surely.

I hope this suggestion be useful.

Regards.

On Tue, Jul 18, 2017 at 7:05 PM Adam Retter <[hidden email]> wrote:
Felix,

I am afraid you are going to have to help us to help you.

All you have really told us is that:

1) The application you have built is slow. 
2) You have 2MB of data, but you want to scale to ~ 50MB or 100MB of data.
3) Listing Collections in the Java Admin client can be slow.
4) You have tried indexing but without success.

You are going to have to be a lot more specific with your problems. eXist can certainly handle 100MB of data without blinking, in fact we have had reports of systems running with hundreds of gigabytes of data.

Listing the contents of a database which has hundreds of Collections using the query browser in the Java Admin Client, is known to be slow I am afraid as it tries to gather a list of all collections in the database up-front. You might have a better experience with eXide or the Atom Editor (https://github.com/wolfgangmm/atom-existdb).

You need to delve into the queries you have written to understand what is slow, and ask us about improving specific queries. This usually involves some rewriting of the queries to optimize them and also correct configuration and use of indexes.

You have not told us anything about what your queries are, which parts of them are slow, or what the problems were that you experienced with indexing. If you want to look into the performance of your queries and index use, then the Monex application is a good start as it allows you to profile such things.



On 18 July 2017 at 09:45, Felix Maas <[hidden email]> wrote:
Hello eXist community,
 
I've got dramatic problems with the performance in eXist-db.
My databased web application I created with eXist should manage projects. There are not millions of projects but 10000 or 20000 of them which should be stored, read and editted by the user (each project has about 100 tags). I wouldn't call that a much amount.
Well at the moment 400 projects are stored in the database (size of the data is about 2MB) and the performance is so bad that you cannot really use it. To read all the data into a table it takes about 5 or 6 seconds. But that is not the big problem. If I try to create a new project there is an error that it takes too much time to process the request. I find that very confusing because I do not even read the whole data, I just want to store a new xml file into the database and I cannot understand why that creates performance problems.
Also confusing to me is the fact that when I try to open the collection with the 400 projects in the Java Admin Client it also takes a lot of time. Actually when I navigate to a collection and I do not open one of the files no content of the data is read in. I guess it is just an overview about the data like e.g. in the windows explorer? So why does eXist have so much problems to read data which are not bigger than 2MB?
Also tried indexing (do not really know if I did it the right way) but without success.
 
Best regards,
Felix

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open




--
Adam Retter

eXist Core Developer
{ United Kingdom / United States }
[hidden email]
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Best way to use Monex profiler? [Was: Performance problems in eXist-db]

Tony Graham
In reply to this post by Adam Retter
On 18/07/2017 15:32, Adam Retter wrote:
...
> If you want to look into the performance of your queries and
> index use, then the Monex application is a good start as it allows you
> to profile such things.

What is the recommended practice for using the Monex profiler?

It's fine for tracing a single action, but it doesn't have any 'memory',
so it's hard to track how the timing changes over time as you tweak the
code and the indexes. It's also not simple to profile the same action
multiple times to get the average timing.

I've tried screenshots of successive runs and I've tried pasting the
text of the 'Functions' tab into successive worksheets in a spreadsheet,
but I don't have an easy way to track performance changes in response to
my changes.

I think that what I really want is a way to name individual profiling
runs and have them stored in the database so that I can later make
nice-looking tables and charts to visualise how things have changed.

Regards,


Tony Graham.
--
Senior Architect
XML Division
Antenna House, Inc.
----
Skerries, Ireland
[hidden email]

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to use Monex profiler? [Was: Performance problems in eXist-db]

Wolfgang Meier-2
Dear Tony,

> I've tried screenshots of successive runs and I've tried pasting the
> text of the 'Functions' tab into successive worksheets in a spreadsheet,
> but I don't have an easy way to track performance changes in response to
> my changes.

Yes, this is true and I agree it would be helpful if you could save the current profiling results. If you look at the code, the profiling within monex is all based on XQuery functions, which are available outside monex as well. For example, see

https://github.com/wolfgangmm/monex/blob/master/modules/app.xql#L183

I guess the easiest approach would be to just save the XML output of system:trace() for future comparisons. You could do this via eXide as the results will be the same as in your running monex.

For sure I’d be happy if you would be willing to contribute something along those lines.

Wolfgang
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Loading...