Quantcast

concurrency?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

concurrency?

Eduard Drenth
Dear all,

Perhaps I missed it, but does exist provide mechanisms for concurrency and transactions?

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Eduard Drenth
I did find http://exist.2174344.n4.nabble.com/Concurrent-writes-in-exist-db-td4662147.html on concurrency

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Eduard Drenth
Sent: Tuesday, March 28, 2017 8:04 AM
To: [hidden email]
Subject: concurrency?

Dear all,

Perhaps I missed it, but does exist provide mechanisms for concurrency and transactions?

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Adam Retter
In reply to this post by Eduard Drenth
Hi Eduard,

Internally eXist uses a WAL (Write Ahead Log) to ensure consistency.
There is a transaction mechanism around that, but they are not
transactions as you might think of them, they are more like
checkpoints that are written to the WAL.

For isolation, eXist uses Reentrant Locking.
Each Collection has its own Read/Write Lock, but only allows a single
reader or writer thread.
Each Document has its own Read/Write Lock, and allows multiple reader
threads or a single writer thread; Waiting Writers are prioritised
over readers.
The isolation level in eXist is rather weak, it exhibits phenomena
such as dirty-reads, non-repeatable reads and phantom reads. The
equivalent ANSI SQL isolation level is probably READ UNCOMMITTED.

Concurrency typically follows the locking scheme above and will depend
on the contention of Collections and Documents across threads of your
application. That being said, there are some bottle necks in eXist,
particularly around the Collection Cache and Collection Configuration
documents which reduce throughput to exclusive single threaded (for a
particular Collection or Document) in some areas.

I have a redesign of eXist's locking and concurrency mechanisms in the
wings that should greatly improve multi-threaded performance. However
it is not yet ready, I hope to send a PR in the next couple of weeks.

On 28 March 2017 at 02:04, Eduard Drenth <[hidden email]> wrote:

> Dear all,
>
> Perhaps I missed it, but does exist provide mechanisms for concurrency and transactions?
>
> Eduard Drenth, Software Architekt
>
> [hidden email]
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> +31 58 234 30 47
>
> gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Eduard Drenth
Thanks for ths! Good to hear about the upcoming improvements. Will you be using Spring? I am not a particular fan of spring, but declarative, container managed transaction is simple and powerfull.

One question: will the current mechanism prevent multiple users from editing the same document simultaneously, also when using different editing environments (eXide, oxygen, webdav)?

I am asking all of these question because we are seriously considering using existdb/teipublisher for some 8000 manuscripts in the Frisian language. The number of manuscripts will grow. Therefore I am extensively testing and experimenting.

In production I opt for a corpus query environment for researchers, probably developed in Java using xqj, multiple existdb instances and connection pooling. This environment will be a CLARIN service and should also respect juridical aspects. This environment is the most exciting, also because it will as a more generic approach offer an alternative for https://github.com/meertensinstituut/mtas and https://github.com/INL/BlackLab.

Next to this I aim for a teipublisher environment on the same material for the wider public.

Editing will be done in a separate environment, probably oxygen.


Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Adam Retter <[hidden email]>
Sent: Tuesday, March 28, 2017 5:25 PM
To: Eduard Drenth
Cc: [hidden email]
Subject: Re: [Exist-open] concurrency?

Hi Eduard,

Internally eXist uses a WAL (Write Ahead Log) to ensure consistency.
There is a transaction mechanism around that, but they are not
transactions as you might think of them, they are more like
checkpoints that are written to the WAL.

For isolation, eXist uses Reentrant Locking.
Each Collection has its own Read/Write Lock, but only allows a single
reader or writer thread.
Each Document has its own Read/Write Lock, and allows multiple reader
threads or a single writer thread; Waiting Writers are prioritised
over readers.
The isolation level in eXist is rather weak, it exhibits phenomena
such as dirty-reads, non-repeatable reads and phantom reads. The
equivalent ANSI SQL isolation level is probably READ UNCOMMITTED.

Concurrency typically follows the locking scheme above and will depend
on the contention of Collections and Documents across threads of your
application. That being said, there are some bottle necks in eXist,
particularly around the Collection Cache and Collection Configuration
documents which reduce throughput to exclusive single threaded (for a
particular Collection or Document) in some areas.

I have a redesign of eXist's locking and concurrency mechanisms in the
wings that should greatly improve multi-threaded performance. However
it is not yet ready, I hope to send a PR in the next couple of weeks.

On 28 March 2017 at 02:04, Eduard Drenth <[hidden email]> wrote:

> Dear all,
>
> Perhaps I missed it, but does exist provide mechanisms for concurrency and transactions?
>
> Eduard Drenth, Software Architekt
>
> [hidden email]
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> +31 58 234 30 47
>
> gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Dannes Wessels-3
Hi,

On Wed, Mar 29, 2017 at 8:22 AM, Eduard Drenth <[hidden email]> wrote:
One question: will the current mechanism prevent multiple users from editing the same document simultaneously, also when using different editing environments (eXide, oxygen, webdav)?

yes....


cheers

D.
 
--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Adam Retter
In reply to this post by Eduard Drenth
> Thanks for ths! Good to hear about the upcoming improvements. Will you be using Spring? I am not a particular fan of spring, but declarative, container managed transaction is simple and powerfull.
>

No, we won't be using Spring.


> One question: will the current mechanism prevent multiple users from editing the same document simultaneously, also when using different editing environments (eXide, oxygen, webdav)?
>

No.
Database locks are not held during the editing process as we have no
way to know what Oxygen is really doing over WebDAV. You can lock a
document over webdav, but that is only checked against other WebDAV
clients AFAIK.

Database locks are enforced at save/update time, so last writer wins
as far as I can see.

Dannes can perhaps comment further?


> In production I opt for a corpus query environment for researchers, probably developed in Java using xqj, multiple existdb instances and connection pooling. This environment will be a CLARIN service and should also respect juridical aspects. This environment is the most exciting, also because it will as a more generic approach offer an alternative for https://github.com/meertensinstituut/mtas and https://github.com/INL/BlackLab.
>

Just a warning that XQJ is somewhat limited as it only allows you to
work with XQuery. It does not give you access to managing documents or
database administration.



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Eduard Drenth
Thanks for the elaboration. So we'll have to be carefull with the editing process, that's ok I guess and there's the versioning plugin that may be of help.

I am aware of the limitations of xqj, but I think xquery is all I want to do in the research environment.

Eduard Drenth, Software Architekt

[hidden email]

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Adam Retter <[hidden email]>
Sent: Wednesday, March 29, 2017 5:09 PM
To: Eduard Drenth
Cc: [hidden email]
Subject: Re: [Exist-open] concurrency?

> Thanks for ths! Good to hear about the upcoming improvements. Will you be using Spring? I am not a particular fan of spring, but declarative, container managed transaction is simple and powerfull.
>

No, we won't be using Spring.


> One question: will the current mechanism prevent multiple users from editing the same document simultaneously, also when using different editing environments (eXide, oxygen, webdav)?
>

No.
Database locks are not held during the editing process as we have no
way to know what Oxygen is really doing over WebDAV. You can lock a
document over webdav, but that is only checked against other WebDAV
clients AFAIK.

Database locks are enforced at save/update time, so last writer wins
as far as I can see.

Dannes can perhaps comment further?


> In production I opt for a corpus query environment for researchers, probably developed in Java using xqj, multiple existdb instances and connection pooling. This environment will be a CLARIN service and should also respect juridical aspects. This environment is the most exciting, also because it will as a more generic approach offer an alternative for https://github.com/meertensinstituut/mtas and https://github.com/INL/BlackLab.
>

Just a warning that XQJ is somewhat limited as it only allows you to
work with XQuery. It does not give you access to managing documents or
database administration.



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Dannes Wessels-3
In reply to this post by Adam Retter

On 29 Mar 2017, at 17:09 , Adam Retter <[hidden email]> wrote:

Database locks are not held during the editing process as we have no
way to know what Oxygen is really doing over WebDAV. You can lock a
document over webdav, but that is only checked against other WebDAV
clients AFAIK.

Database locks are enforced at save/update time, so last writer wins
as far as I can see.

Dannes can perhaps comment further?


Under the hood a webdav Lock is in essence registering a lock-token plus a ‘normal’ lock in the database. If a ‘normal’ lock is in place, webdav can not lock the resource , will not generate the lock-token and will report accordingly.

cheers

Dannes

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Eduard Drenth

Thanks, what does this mean functionally? That users can edit the same document at the same time? Last one that saves the document wins?


Or does your comment mean otherwise?


And I presume there is no optimistic locking by means of a incremental versioning number?


Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Dannes Wessels <[hidden email]> on behalf of Dannes Wessels <[hidden email]>
Sent: Wednesday, March 29, 2017 8:17 PM
To: Adam Retter
Cc: Eduard Drenth; [hidden email]
Subject: Re: [Exist-open] concurrency?
 

On 29 Mar 2017, at 17:09 , Adam Retter <[hidden email]> wrote:

Database locks are not held during the editing process as we have no
way to know what Oxygen is really doing over WebDAV. You can lock a
document over webdav, but that is only checked against other WebDAV
clients AFAIK.

Database locks are enforced at save/update time, so last writer wins
as far as I can see.

Dannes can perhaps comment further?


Under the hood a webdav Lock is in essence registering a lock-token plus a ‘normal’ lock in the database. If a ‘normal’ lock is in place, webdav can not lock the resource , will not generate the lock-token and will report accordingly.

cheers

Dannes

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Dannes Wessels-3
hi,

On 29 Mar 2017, at 20:57 , Eduard Drenth <[hidden email]> wrote:

Thanks, what does this mean functionally? That users can edit the same document at the same time? Last one that saves the document wins?

Or does your comment mean otherwise?

And I presume there is no optimistic locking by means of a incremental versioning number?


erhm… by definition the intention of a lock is to block modification by others; like “write exclusivity". I have no idea how you could read different from my answer….. so no.

cheers

Dannes

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Eduard Drenth

Sorry, but I was confused because Adam writes:


No.
Database locks are not held during the editing process as we have no
way to know what Oxygen is really doing over WebDAV. You can lock a
document over webdav, but that is only checked against other WebDAV
clients AFAIK.

Database locks are enforced at save/update time, so last writer wins
as far as I can see.


In answer to this question:


One question: will the current mechanism prevent multiple users from editing the same document simultaneously, also when using different editing environments (eXide, oxygen, webdav)?


You both say that when a user is editing a document over webdav an exclusive lock is placed that prevents others from editing. Adam says the database lock is only held shortly during save, so my conclusion now is that editing via eXide is still possible when a webdav lock is in place. Am I right?


Optimistic locking using an incremental version (like hibernate does) is I think a nice and cheap mechanism to prevent 'dirty' data.


Eduard Drenth, Software Architekt


[hidden email]


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47


gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43




From: Dannes Wessels <[hidden email]> on behalf of Dannes Wessels <[hidden email]>
Sent: Wednesday, March 29, 2017 9:53 PM
To: Eduard Drenth
Cc: Adam Retter; [hidden email]
Subject: Re: [Exist-open] concurrency?
 
hi,

On 29 Mar 2017, at 20:57 , Eduard Drenth <[hidden email]> wrote:

Thanks, what does this mean functionally? That users can edit the same document at the same time? Last one that saves the document wins?

Or does your comment mean otherwise?

And I presume there is no optimistic locking by means of a incremental versioning number?


erhm… by definition the intention of a lock is to block modification by others; like “write exclusivity". I have no idea how you could read different from my answer….. so no.

cheers

Dannes

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Dannes Wessels-3
Hi,

On 30 Mar 2017, at 6:50 , Eduard Drenth <[hidden email]> wrote:

Sorry, but I was confused because Adam writes:

No.
Database locks are not held during the editing process as we have no
way to know what Oxygen is really doing over WebDAV. You can lock a
document over webdav, but that is only checked against other WebDAV
clients AFAIK.

Database locks are enforced at save/update time, so last writer wins
as far as I can see.

In answer to this question:

One question: will the current mechanism prevent multiple users from editing the same document simultaneously, also when using different editing environments (eXide, oxygen, webdav)?

You both say that when a user is editing a document over webdav an exclusive lock is placed that prevents others from editing. Adam says the database lock is only held shortly during save, so my conclusion now is that editing via eXide is still possible when a webdav lock is in place. Am I right?


AH ok. I think Adam is not completely right here.

If we look at the relevant webDav code (github) I see the following:



inputToken.setOwner(subject.getName());
inputToken.createOpaqueLockToken();
//inputToken.setTimeOut(inputToken.getTimeOut());
inputToken.setTimeOut(LockToken.LOCK_TIMEOUT_INFINITE);
// Update document
document.getMetadata().setLockToken(inputToken);
document.setUserLock(subject);

the first half is about the webDAV code registering a webdav lock, the second one actually places a User lock on the resource itself. it is the same User-level lock as created by the util:lock() function (github). With this lock gets exclusive (write) access to a resource.

Maybe Adam an I are talking about different Lock levels…..



Optimistic locking using an incremental version (like hibernate does) is I think a nice and cheap mechanism to prevent 'dirty' data.

sure but that is an other domain and software product.

cheers

Dannes



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: concurrency?

Adam Retter
AH ok. I think Adam is not completely right here.

If we look at the relevant webDav code (github) I see the following:



inputToken.setOwner(subject.getName());
inputToken.createOpaqueLockToken();
//inputToken.setTimeOut(inputToken.getTimeOut());
inputToken.setTimeOut(LockToken.LOCK_TIMEOUT_INFINITE);
// Update document
document.getMetadata().setLockToken(inputToken);
document.setUserLock(subject);

the first half is about the webDAV code registering a webdav lock, the second one actually places a User lock on the resource itself. it is the same User-level lock as created by the util:lock() function (github). With this lock gets exclusive (write) access to a resource.


User Level locks are not "exclusive" locks as such, there is no Mutex or CAS. A User Level lock just records in the metadata of a document the userId of a user that has "user level locked" the document.

User Level locks are only used in WebDAV, and XML:DB API code, and only for a few operations such as "lock", "hasLock" and "unlock". As such they have no impact in the actual core of the database when it comes to contention on resources.

For example, most of the XQuery document management functions talk directly to the core of the database, and so will not respect these "user level locks" that are used in the WebDAV API.

The Document and Collection locks in the core of the database are those that control the real concurrent access to the resources, and those are the ones that I was previously talking about.


Maybe Adam an I are talking about different Lock levels…..

It seems so.
 

--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Loading...