executing system commands

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

executing system commands

Immanuel Normann
Hi,

exit seems to be very open to integrate custom java-modules for specific purposes and to run them as xquery functions via the import mechanism.

However, Java is not always everyone's friend for every purpose. In my case, for instance, I am much more productive in any other programming languages than Java. So I was very happy to find at least the option to run system commands within exist with process:execute via

import module namespace process="http://exist-db.org/xquery/process" at "java:org.exist.xquery.modules.process.ProcessModule";

I wonder though, why this process module is not documented in http://exist-db.org/exist/apps/fundocs/browse.html . It seems as if this is a secret functionality that should not be propagated. For me, though, it is a welcome option to escape the Java world (after all this was one central reason for me to use exist as a webapp-platform that frees me from Java coding).

Yet, there is one drawback with the ProcessModule in xquery: process:execute is only available to users with dba role. I understand that not every user should be able to execute any system command. However, it should be possible for the admin to somehow configure or declare particular system commands as safe. I mean after all a custom Java-module can be as dangerous as any dangerous as any other executable running as system command.

In summary, I would like to enable particular users to execute a custom defined list of executables on the system within exist. Is this possible?

Immanuel

------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: executing system commands

Joe Wicentowski
Hi Immanuel,

As to your question about why this module doesn't appear on eXist-db.org's function documentation, one limitation of the "fundocs" application is that modules appear only if they are enabled on the server.  Thus, the process module won't show up on exist-db.org's copy of the function documentation if that module is disabled.  It would be great if there were a way for exist-db.org (or even local installations) to show documentation for all modules, even if it doesn't have these enabled.  

For exposing dba-only functions to non-dba users, I think the best practice is to use setuid/setgid so that an .xq file executes as a user, but give certain users/groups the ability to execute this query. 

Joe

On Wed, Sep 9, 2015 at 10:33 AM, Immanuel Normann <[hidden email]> wrote:
Hi,

exit seems to be very open to integrate custom java-modules for specific purposes and to run them as xquery functions via the import mechanism.

However, Java is not always everyone's friend for every purpose. In my case, for instance, I am much more productive in any other programming languages than Java. So I was very happy to find at least the option to run system commands within exist with process:execute via

import module namespace process="http://exist-db.org/xquery/process" at "java:org.exist.xquery.modules.process.ProcessModule";

I wonder though, why this process module is not documented in http://exist-db.org/exist/apps/fundocs/browse.html . It seems as if this is a secret functionality that should not be propagated. For me, though, it is a welcome option to escape the Java world (after all this was one central reason for me to use exist as a webapp-platform that frees me from Java coding).

Yet, there is one drawback with the ProcessModule in xquery: process:execute is only available to users with dba role. I understand that not every user should be able to execute any system command. However, it should be possible for the admin to somehow configure or declare particular system commands as safe. I mean after all a custom Java-module can be as dangerous as any dangerous as any other executable running as system command.

In summary, I would like to enable particular users to execute a custom defined list of executables on the system within exist. Is this possible?

Immanuel

------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: executing system commands

Immanuel Normann
Thanks Joe!
Actually, I heard about setuid/setgid but couldn't find any docu on that either. Do you have a conrete pointer to learn more about it? Thanks in advance!

2015-09-09 16:53 GMT+02:00 Joe Wicentowski <[hidden email]>:
Hi Immanuel,

As to your question about why this module doesn't appear on eXist-db.org's function documentation, one limitation of the "fundocs" application is that modules appear only if they are enabled on the server.  Thus, the process module won't show up on exist-db.org's copy of the function documentation if that module is disabled.  It would be great if there were a way for exist-db.org (or even local installations) to show documentation for all modules, even if it doesn't have these enabled.  

For exposing dba-only functions to non-dba users, I think the best practice is to use setuid/setgid so that an .xq file executes as a user, but give certain users/groups the ability to execute this query. 

Joe

On Wed, Sep 9, 2015 at 10:33 AM, Immanuel Normann <[hidden email]> wrote:
Hi,

exit seems to be very open to integrate custom java-modules for specific purposes and to run them as xquery functions via the import mechanism.

However, Java is not always everyone's friend for every purpose. In my case, for instance, I am much more productive in any other programming languages than Java. So I was very happy to find at least the option to run system commands within exist with process:execute via

import module namespace process="http://exist-db.org/xquery/process" at "java:org.exist.xquery.modules.process.ProcessModule";

I wonder though, why this process module is not documented in http://exist-db.org/exist/apps/fundocs/browse.html . It seems as if this is a secret functionality that should not be propagated. For me, though, it is a welcome option to escape the Java world (after all this was one central reason for me to use exist as a webapp-platform that frees me from Java coding).

Yet, there is one drawback with the ProcessModule in xquery: process:execute is only available to users with dba role. I understand that not every user should be able to execute any system command. However, it should be possible for the admin to somehow configure or declare particular system commands as safe. I mean after all a custom Java-module can be as dangerous as any dangerous as any other executable running as system command.

In summary, I would like to enable particular users to execute a custom defined list of executables on the system within exist. Is this possible?

Immanuel

------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open




------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: executing system commands

Joe Wicentowski
I seem to recall having seen documentation on this, but a quick search of exist-db.org documentation just now seems not to turn up any information about this feature.  I also thought the eXist book covered setGid and setUid in chapter 8 on security, but I just checked and found it's not there.  Perhaps the feature was added after the book had gone to press?

The best starting point, then, might be Adam's presentation from XML Prague 2014 on this, especially slides 17-32 (PDF) and 14:25 (YouTube):


I've attached a screenshot of the java admin client showing where you can set the uid and gid bits.  I'll also excerpt from an email thread I had with Adam on this below.  I hope this info helps!

Joe


Me:
------
My immediate goal for using setUid and setGid is to apply it to the
twitter polling scheduled job on history.state.gov.  Can I talk
through how to do this with you?

Right now I use system:as-user('admin'...) to authorize xmldb:store()
to store the newly fetched tweets.  I currently have three groups on
history.state.gov: admin, guest, and cms-edit -- the latter being the
group assigned to ordinary users of the "CMS" section of the site (so
they can log into history.state.gov/cms), but in practice the only
group that has any real write privileges on the site is admin.

From what I understood from your talk, to use setUid and setGid for a
twitter polling scheduled job, I'd do something like this:

1. Create a user and group, say "twitter-user" and "twitter-group"

2. Create a data collection, say "twitter-data", owned by this user
and group, where the tweets will be saved.

3. Set twitter polling query, say "poll-twitter.xq", owned by the
twitter user and group, where guest has execute privileges, and with
the setGid bit set.  This way, the job scheduler (which runs as guest)
can execute the query, but the query runs as if the twitter user was
running it.  No xmldb:login() or system:as-user() is needed.  The
xmldb:store() operation will work without authentication as long as
the destination is owned by the twitter group.

Adam:
--------
Everything you have said sounds correct. However you need not create
both a twitter-user and twitter-group if you do not wish to. As you
are using the setGid bit, you can get away with just the
twitter-group. In that way the owner user of the script can have
permission to modify the script i.e. the write bit, but you do not
grant guest users (through the twitter-group setGid) write access to
modify the script.

So your poll-twitter.xq may have the owner joew:twitter-group and the
permissions mode 02755, i.e. setGid, rwx for joew, r-x for
twitter-group, and r-x for everyone else (guest included).

Your collection twitter-data, then may have the owner
joew:twitter-group and the permissions 0755 or whatever you wish. You
can also use setGid on the Collection to preserve the group, i.e.
twitter-group on new resources created in the collection.

On Wed, Sep 9, 2015 at 11:15 AM, Immanuel Normann <[hidden email]> wrote:
Thanks Joe!
Actually, I heard about setuid/setgid but couldn't find any docu on that either. Do you have a conrete pointer to learn more about it? Thanks in advance!

2015-09-09 16:53 GMT+02:00 Joe Wicentowski <[hidden email]>:
Hi Immanuel,

As to your question about why this module doesn't appear on eXist-db.org's function documentation, one limitation of the "fundocs" application is that modules appear only if they are enabled on the server.  Thus, the process module won't show up on exist-db.org's copy of the function documentation if that module is disabled.  It would be great if there were a way for exist-db.org (or even local installations) to show documentation for all modules, even if it doesn't have these enabled.  

For exposing dba-only functions to non-dba users, I think the best practice is to use setuid/setgid so that an .xq file executes as a user, but give certain users/groups the ability to execute this query. 

Joe

On Wed, Sep 9, 2015 at 10:33 AM, Immanuel Normann <[hidden email]> wrote:
Hi,

exit seems to be very open to integrate custom java-modules for specific purposes and to run them as xquery functions via the import mechanism.

However, Java is not always everyone's friend for every purpose. In my case, for instance, I am much more productive in any other programming languages than Java. So I was very happy to find at least the option to run system commands within exist with process:execute via

import module namespace process="http://exist-db.org/xquery/process" at "java:org.exist.xquery.modules.process.ProcessModule";

I wonder though, why this process module is not documented in http://exist-db.org/exist/apps/fundocs/browse.html . It seems as if this is a secret functionality that should not be propagated. For me, though, it is a welcome option to escape the Java world (after all this was one central reason for me to use exist as a webapp-platform that frees me from Java coding).

Yet, there is one drawback with the ProcessModule in xquery: process:execute is only available to users with dba role. I understand that not every user should be able to execute any system command. However, it should be possible for the admin to somehow configure or declare particular system commands as safe. I mean after all a custom Java-module can be as dangerous as any dangerous as any other executable running as system command.

In summary, I would like to enable particular users to execute a custom defined list of executables on the system within exist. Is this possible?

Immanuel

------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open





------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

setuid-setgid-in-existdb-java-admin-client.png (61K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: executing system commands

wsalesky
This may also be helpful: 

-Winona

On Wed, Sep 9, 2015 at 11:43 AM, Joe Wicentowski <[hidden email]> wrote:
I seem to recall having seen documentation on this, but a quick search of exist-db.org documentation just now seems not to turn up any information about this feature.  I also thought the eXist book covered setGid and setUid in chapter 8 on security, but I just checked and found it's not there.  Perhaps the feature was added after the book had gone to press?

The best starting point, then, might be Adam's presentation from XML Prague 2014 on this, especially slides 17-32 (PDF) and 14:25 (YouTube):


I've attached a screenshot of the java admin client showing where you can set the uid and gid bits.  I'll also excerpt from an email thread I had with Adam on this below.  I hope this info helps!

Joe


Me:
------
My immediate goal for using setUid and setGid is to apply it to the
twitter polling scheduled job on history.state.gov.  Can I talk
through how to do this with you?

Right now I use system:as-user('admin'...) to authorize xmldb:store()
to store the newly fetched tweets.  I currently have three groups on
history.state.gov: admin, guest, and cms-edit -- the latter being the
group assigned to ordinary users of the "CMS" section of the site (so
they can log into history.state.gov/cms), but in practice the only
group that has any real write privileges on the site is admin.

From what I understood from your talk, to use setUid and setGid for a
twitter polling scheduled job, I'd do something like this:

1. Create a user and group, say "twitter-user" and "twitter-group"

2. Create a data collection, say "twitter-data", owned by this user
and group, where the tweets will be saved.

3. Set twitter polling query, say "poll-twitter.xq", owned by the
twitter user and group, where guest has execute privileges, and with
the setGid bit set.  This way, the job scheduler (which runs as guest)
can execute the query, but the query runs as if the twitter user was
running it.  No xmldb:login() or system:as-user() is needed.  The
xmldb:store() operation will work without authentication as long as
the destination is owned by the twitter group.

Adam:
--------
Everything you have said sounds correct. However you need not create
both a twitter-user and twitter-group if you do not wish to. As you
are using the setGid bit, you can get away with just the
twitter-group. In that way the owner user of the script can have
permission to modify the script i.e. the write bit, but you do not
grant guest users (through the twitter-group setGid) write access to
modify the script.

So your poll-twitter.xq may have the owner joew:twitter-group and the
permissions mode 02755, i.e. setGid, rwx for joew, r-x for
twitter-group, and r-x for everyone else (guest included).

Your collection twitter-data, then may have the owner
joew:twitter-group and the permissions 0755 or whatever you wish. You
can also use setGid on the Collection to preserve the group, i.e.
twitter-group on new resources created in the collection.

On Wed, Sep 9, 2015 at 11:15 AM, Immanuel Normann <[hidden email]> wrote:
Thanks Joe!
Actually, I heard about setuid/setgid but couldn't find any docu on that either. Do you have a conrete pointer to learn more about it? Thanks in advance!

2015-09-09 16:53 GMT+02:00 Joe Wicentowski <[hidden email]>:
Hi Immanuel,

As to your question about why this module doesn't appear on eXist-db.org's function documentation, one limitation of the "fundocs" application is that modules appear only if they are enabled on the server.  Thus, the process module won't show up on exist-db.org's copy of the function documentation if that module is disabled.  It would be great if there were a way for exist-db.org (or even local installations) to show documentation for all modules, even if it doesn't have these enabled.  

For exposing dba-only functions to non-dba users, I think the best practice is to use setuid/setgid so that an .xq file executes as a user, but give certain users/groups the ability to execute this query. 

Joe

On Wed, Sep 9, 2015 at 10:33 AM, Immanuel Normann <[hidden email]> wrote:
Hi,

exit seems to be very open to integrate custom java-modules for specific purposes and to run them as xquery functions via the import mechanism.

However, Java is not always everyone's friend for every purpose. In my case, for instance, I am much more productive in any other programming languages than Java. So I was very happy to find at least the option to run system commands within exist with process:execute via

import module namespace process="http://exist-db.org/xquery/process" at "java:org.exist.xquery.modules.process.ProcessModule";

I wonder though, why this process module is not documented in http://exist-db.org/exist/apps/fundocs/browse.html . It seems as if this is a secret functionality that should not be propagated. For me, though, it is a welcome option to escape the Java world (after all this was one central reason for me to use exist as a webapp-platform that frees me from Java coding).

Yet, there is one drawback with the ProcessModule in xquery: process:execute is only available to users with dba role. I understand that not every user should be able to execute any system command. However, it should be possible for the admin to somehow configure or declare particular system commands as safe. I mean after all a custom Java-module can be as dangerous as any dangerous as any other executable running as system command.

In summary, I would like to enable particular users to execute a custom defined list of executables on the system within exist. Is this possible?

Immanuel

------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open





------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
wit
Reply | Threaded
Open this post in threaded view
|

HTML serializing

wit
Hi all,
I am using the eXist serailizer to render poems in HTML  which are saved
in TEI format.
http://digi.ub.uni-heidelberg.de/diglit/cpg389/0018#tab_text_tei

I have had the issue, that  in stable 2.2 version, I was having spaces
rendered  (in some cases) if, there was a space in XML file.

But this seems to be fixed in the trunk.

I am interested, on which basis the serializer is written, e.g. if there
is a W3C document, which defines the rules ?
The background for the question is that TEI seems to allow explicit
blanks and I want to generate a XSD for the  TEI-Subset which we are using.


Thanks for any help
Dulip






------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: HTML serializing

Adam Retter
Dulip,

If you are coming via XQuery then the serialization is defined here -
http://www.w3.org/TR/xslt-xquery-serialization/

On 10 September 2015 at 09:04, withanage <[hidden email]> wrote:

> Hi all,
> I am using the eXist serailizer to render poems in HTML  which are saved
> in TEI format.
> http://digi.ub.uni-heidelberg.de/diglit/cpg389/0018#tab_text_tei
>
> I have had the issue, that  in stable 2.2 version, I was having spaces
> rendered  (in some cases) if, there was a space in XML file.
>
> But this seems to be fixed in the trunk.
>
> I am interested, on which basis the serializer is written, e.g. if there
> is a W3C document, which defines the rules ?
> The background for the question is that TEI seems to allow explicit
> blanks and I want to generate a XSD for the  TEI-Subset which we are using.
>
>
> Thanks for any help
> Dulip
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
> Get real-time metrics from all of your servers, apps and tools
> in one place.
> SourceForge users - Click here to start your Free Trial of Datadog now!
> http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open