get parameter _query = open proxy

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

get parameter _query = open proxy

Mathias Göbel
dear list,

i used the _query parameter to write a proxy with just a few basic
functions:
?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

unfortunately it seems that many eXist-db instances are not secured
appropriately so you can send this query string to *any* public endpoint
(html pages, public REST) of eXist and abuse the instance as an open
proxy (upgrading to ssl when available).

i am fine with a Apache rewrite rule

        RewriteEngine on
        RewriteCond %{QUERY_STRING} _query=
        RewriteRule (.*) $1? [R=permanent]

but i also like the idea offering RESTXQ to the public to query the
database as it offers a handy tool with capabilities exceeding a
(preconfigured) search function on a website.
to avoid an open proxy eXist should not preform any request to the
internet, but respond to. there are still some issues to think about
when offering an »public xquery processor« but a RESTXQ endpoint to my
data would be great!

are there any solutions available?

Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
but it disables public request with ?_query completely.

finally i like to encourage anyone to test own instances against this
issue and avoid offering open proxies as they are a risk for the
internet at large.

best,
mathias

--

Mathias Göbel
Research and Development

Georg-August-Universität Göttingen
State and University Library Göttingen
D-37070 Göttingen

Papendiek 14 (hist. building, room 2.408)
+49 551 39-20184 (Tel.)

[hidden email]
http://www.sub.uni-goettingen.de

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

wshager
Thanks for this warning. I might add that eXide is an obvious open backdoor in this respect too.

2017-03-11 10:05 GMT+01:00 Mathias Göbel <[hidden email]>:
dear list,

i used the _query parameter to write a proxy with just a few basic
functions:
?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

unfortunately it seems that many eXist-db instances are not secured
appropriately so you can send this query string to *any* public endpoint
(html pages, public REST) of eXist and abuse the instance as an open
proxy (upgrading to ssl when available).

i am fine with a Apache rewrite rule

        RewriteEngine on
        RewriteCond %{QUERY_STRING} _query=
        RewriteRule (.*) $1? [R=permanent]

but i also like the idea offering RESTXQ to the public to query the
database as it offers a handy tool with capabilities exceeding a
(preconfigured) search function on a website.
to avoid an open proxy eXist should not preform any request to the
internet, but respond to. there are still some issues to think about
when offering an »public xquery processor« but a RESTXQ endpoint to my
data would be great!

are there any solutions available?

Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
but it disables public request with ?_query completely.

finally i like to encourage anyone to test own instances against this
issue and avoid offering open proxies as they are a risk for the
internet at large.

best,
mathias

--

Mathias Göbel
Research and Development

Georg-August-Universität Göttingen
State and University Library Göttingen
D-37070 Göttingen

Papendiek 14 (hist. building, room 2.408)
+49 551 39-20184 (Tel.)

[hidden email]
http://www.sub.uni-goettingen.de

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



--

W.S. Hager
Lagua Web Solutions
http://lagua.nl


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

wshager
However, there's also security by obscurity when using eXist.

2017-03-12 19:34 GMT+01:00 W.S. Hager <[hidden email]>:
Thanks for this warning. I might add that eXide is an obvious open backdoor in this respect too.

2017-03-11 10:05 GMT+01:00 Mathias Göbel <[hidden email]>:
dear list,

i used the _query parameter to write a proxy with just a few basic
functions:
?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

unfortunately it seems that many eXist-db instances are not secured
appropriately so you can send this query string to *any* public endpoint
(html pages, public REST) of eXist and abuse the instance as an open
proxy (upgrading to ssl when available).

i am fine with a Apache rewrite rule

        RewriteEngine on
        RewriteCond %{QUERY_STRING} _query=
        RewriteRule (.*) $1? [R=permanent]

but i also like the idea offering RESTXQ to the public to query the
database as it offers a handy tool with capabilities exceeding a
(preconfigured) search function on a website.
to avoid an open proxy eXist should not preform any request to the
internet, but respond to. there are still some issues to think about
when offering an »public xquery processor« but a RESTXQ endpoint to my
data would be great!

are there any solutions available?

Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
but it disables public request with ?_query completely.

finally i like to encourage anyone to test own instances against this
issue and avoid offering open proxies as they are a risk for the
internet at large.

best,
mathias

--

Mathias Göbel
Research and Development

Georg-August-Universität Göttingen
State and University Library Göttingen
D-37070 Göttingen

Papendiek 14 (hist. building, room 2.408)
<a href="tel:+49%20551%203920184" value="+495513920184" target="_blank">+49 551 39-20184 (Tel.)

[hidden email]
http://www.sub.uni-goettingen.de

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



--

W.S. Hager
Lagua Web Solutions
http://lagua.nl




--

W.S. Hager
Lagua Web Solutions
http://lagua.nl


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Martin Holmes
In reply to this post by Mathias Göbel
This is definitely an issue. I notice that there's a setting to turn off
XSL processing over rest:

"Setting _xsl to no disables any stylesheet processing"
http://exist-db.org/exist/apps/doc/devguide_rest.xml

Is there a similar setting we could use to turn off _query? If not,
should there be? I'm not sure whether there are any internal processes
that would be affected by turning off access to _query on the rest
interface.

Cheers,
Martin

On 2017-03-11 01:05 AM, Mathias Göbel wrote:

> dear list,
>
> i used the _query parameter to write a proxy with just a few basic
> functions:
> ?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)
>
> unfortunately it seems that many eXist-db instances are not secured
> appropriately so you can send this query string to *any* public endpoint
> (html pages, public REST) of eXist and abuse the instance as an open
> proxy (upgrading to ssl when available).
>
> i am fine with a Apache rewrite rule
>
>         RewriteEngine on
>         RewriteCond %{QUERY_STRING} _query=
>         RewriteRule (.*) $1? [R=permanent]
>
> but i also like the idea offering RESTXQ to the public to query the
> database as it offers a handy tool with capabilities exceeding a
> (preconfigured) search function on a website.
> to avoid an open proxy eXist should not preform any request to the
> internet, but respond to. there are still some issues to think about
> when offering an »public xquery processor« but a RESTXQ endpoint to my
> data would be great!
>
> are there any solutions available?
>
> Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
> but it disables public request with ?_query completely.
>
> finally i like to encourage anyone to test own instances against this
> issue and avoid offering open proxies as they are a risk for the
> internet at large.
>
> best,
> mathias
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Omar Siam-2
In reply to this post by Mathias Göbel
Dear members of this list!

This issue was reported to one of our apps running on exist-db so I'd
like to add my thoughts:

* The only solution is to hide the "REST-API" on any public facing
server and then, if you must, do a explicit redirect in a controller.xql
to expose the "REST-API" at some path you set.

* The default setting for the "REST-API" is a trap for everyone who is
not a java servlet specialist like me for example. I use exist-db. I
usually don't care about the intricacies of it's java foundation

Why isn't it enough to just be careful if the "REST-API" is not hidden?

Because in my opinion you have absolutely no chance of getting this safe!

The logic as explained here:
https://github.com/eXist-db/documentation/issues/98 is: if "REST-API" is
not hidden then *every* path in /exist/apps that has no controller.xql
will be seved by the "REST-API". Let me repeat that: every path that is
not served by controller.xql. The point is: Every path also means *every
non-existing path*.

Examples (from a site that maybe is some throw away example instance):

* http://localhost:8080/exist/apps/homepage/index.html?_query=1 (the
file is no controller.xql, but hey maybe there is no controller.xql in
http://localhost:8080/exist/apps/homepage)

* http://localhost:8080/exist/apps/fundocs/index.html?_query=1 (as far
as I know there is a controller.xql in
http://localhost:8080/exist/apps/fundocs and so
http://localhost:8080/exist/apps/fundocs/?_query=1 actually does not
"work")  and then of course very hard to catch: all of the paths that
*do not exist*

*
http://localhost:8080/exist/apps/does_not_matter/what/path/even_if_it_does_not_exist.badending?_query=xquery%20version%20%221.0%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

Because of the last feature our idea of reverse proxying right up to or
into a particular exist-db app (like internal-sever/exist/apps/funcdocs/
-> funcdocs.example.org/) exposed the "REST-API". That is very obvious
to everyone. Isn't it?

So no chance to stop this DDOS service using controller.xql. Also
imagine what can be done using a combination of util:eval ant
httpclient:get.

When it comes to public facing services then most developers nowadays
try to do defensive programming. Better you don't expose a funtionality
that can bite you.

Therefore I strongly advocate for the exist team to make a prominent
notice of this flaw that exists at least since exist 2.2 and urge people
to change their settings *now*. Furthermore please don't distribute
exist setup in this manner. People should better know that the
"REST-API" needs to be explicitly enabled because of its implications.

Perhaps there is a chance to check if a path actually exists and/or is
no file before happily invoking the "REST-API". At least in /exist/apps
I see no point in _querying an html file.

Also perhaps the non obvious concept of /exist/apps == /exist/rest if
there is no controller.xql should be reconsidered.

Best regards

Omar Siam


Am 11.03.2017 um 10:05 schrieb Mathias Göbel:

> dear list,
>
> i used the _query parameter to write a proxy with just a few basic
> functions:
> ?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)
>
> unfortunately it seems that many eXist-db instances are not secured
> appropriately so you can send this query string to *any* public endpoint
> (html pages, public REST) of eXist and abuse the instance as an open
> proxy (upgrading to ssl when available).
>
> i am fine with a Apache rewrite rule
>
>          RewriteEngine on
>          RewriteCond %{QUERY_STRING} _query=
>          RewriteRule (.*) $1? [R=permanent]
>
> but i also like the idea offering RESTXQ to the public to query the
> database as it offers a handy tool with capabilities exceeding a
> (preconfigured) search function on a website.
> to avoid an open proxy eXist should not preform any request to the
> internet, but respond to. there are still some issues to think about
> when offering an »public xquery processor« but a RESTXQ endpoint to my
> data would be great!
>
> are there any solutions available?
>
> Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
> but it disables public request with ?_query completely.
>
> finally i like to encourage anyone to test own instances against this
> issue and avoid offering open proxies as they are a risk for the
> internet at large.
>
> best,
> mathias
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Alister Pillow-3
Hi Omar,
I have tested this issue on my demo server running eXist 2.2x and indeed it is a problem.

A possible solution is the use of the Security Constraints provided by the Java Authentication and Authorization Service (JAAS).

This is a fragment from my web-app/WEB-INF/web.xml.

<security-constraint>
<web-resource-collection>
            <web-resource-name>Administration
            </web-resource-name>
            <url-pattern>/apps/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
<security-constraint>
<web-resource-collection>
            <web-resource-name>
                Pekoe Administration
            </web-resource-name>
            <url-pattern>/restxq/pekoe/admin/*</url-pattern>
            <url-pattern>/apps/*</url-pattern> <!— ***************** This blocks any unauthorised access to /exist/apps and redirects to a login form —>
        </web-resource-collection>
        <auth-constraint>
            <description>
                Let only administrators use this app
            </description>
            <!-- Role name can be ** to allow any authenticated user - regardless of role.
                Or, can be * to allow any user who has the top-level role-name (pekoe-users below).
            -->
            <role-name>dba</role-name>
        </auth-constraint>
    </security-constraint>
            <role-name>dba</role-name>
        </auth-constraint>
    </security-constraint>

Today I added the url-pattern /apps/* and tested - I was directed to a login page when running your example. (The base of all url-patterns is /exist)
<a href="https://demo.pekoe.io/exist/apps/does_not_matter/what/path/even_if_it_does_not_exist.badending?_query=xquery version &quot;1.0&quot;;response:stream-binary( xs:base64Binary( data(httpclient:get(xs:anyURI(&quot;http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg&quot;), false(), ()" class="">https://demo.pekoe.io/exist/apps/does_not_matter/what/path/even_if_it_does_not_exist.badending?_query=xquery%20version%20%221.0%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

Once logged-in I was able to access the Dashboard and eXide as usual. 

I’ve been using the JAAS service in production (eXist-db v2.2x) on “demo" and “live" for over 18 months without any problem. I’ll be testing the /apps/* pattern more extensively before applying it to “live” - today if possible.

Using JAAS requires a modification to src/org/exist/security/internal/EXistDBLoginModule.java which provides the User’s Groups as Roles, so all external users of my server must belong to the “pekoe-users” group. 
**** I tested this last year in a pre-release version of eXist 3 ****
(I’ve been hoping for a good reason to suggest it to the group - thank you!)

Regards,
Alister

On 14 Mar 2017, at 9:01 pm, Omar Siam <[hidden email]> wrote:

Dear members of this list!

This issue was reported to one of our apps running on exist-db so I'd 
like to add my thoughts:

* The only solution is to hide the "REST-API" on any public facing 
server and then, if you must, do a explicit redirect in a controller.xql 
to expose the "REST-API" at some path you set.

* The default setting for the "REST-API" is a trap for everyone who is 
not a java servlet specialist like me for example. I use exist-db. I 
usually don't care about the intricacies of it's java foundation

Why isn't it enough to just be careful if the "REST-API" is not hidden?

Because in my opinion you have absolutely no chance of getting this safe!

The logic as explained here: 
https://github.com/eXist-db/documentation/issues/98 is: if "REST-API" is 
not hidden then *every* path in /exist/apps that has no controller.xql 
will be seved by the "REST-API". Let me repeat that: every path that is 
not served by controller.xql. The point is: Every path also means *every 
non-existing path*.

Examples (from a site that maybe is some throw away example instance):

* http://localhost:8080/exist/apps/homepage/index.html?_query=1 (the 
file is no controller.xql, but hey maybe there is no controller.xql in 
http://localhost:8080/exist/apps/homepage)

* http://localhost:8080/exist/apps/fundocs/index.html?_query=1 (as far 
as I know there is a controller.xql in 
http://localhost:8080/exist/apps/fundocs and so 
http://localhost:8080/exist/apps/fundocs/?_query=1 actually does not 
"work")  and then of course very hard to catch: all of the paths that 
*do not exist*

* 
http://localhost:8080/exist/apps/does_not_matter/what/path/even_if_it_does_not_exist.badending?_query=xquery%20version%20%221.0%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

Because of the last feature our idea of reverse proxying right up to or 
into a particular exist-db app (like internal-sever/exist/apps/funcdocs/ 
-> funcdocs.example.org/) exposed the "REST-API". That is very obvious 
to everyone. Isn't it?

So no chance to stop this DDOS service using controller.xql. Also 
imagine what can be done using a combination of util:eval ant 
httpclient:get.

When it comes to public facing services then most developers nowadays 
try to do defensive programming. Better you don't expose a funtionality 
that can bite you.

Therefore I strongly advocate for the exist team to make a prominent 
notice of this flaw that exists at least since exist 2.2 and urge people 
to change their settings *now*. Furthermore please don't distribute 
exist setup in this manner. People should better know that the 
"REST-API" needs to be explicitly enabled because of its implications.

Perhaps there is a chance to check if a path actually exists and/or is 
no file before happily invoking the "REST-API". At least in /exist/apps 
I see no point in _querying an html file.

Also perhaps the non obvious concept of /exist/apps == /exist/rest if 
there is no controller.xql should be reconsidered.

Best regards

Omar Siam


Am 11.03.2017 um 10:05 schrieb Mathias Göbel:
dear list,

i used the _query parameter to write a proxy with just a few basic
functions:
?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

unfortunately it seems that many eXist-db instances are not secured
appropriately so you can send this query string to *any* public endpoint
(html pages, public REST) of eXist and abuse the instance as an open
proxy (upgrading to ssl when available).

i am fine with a Apache rewrite rule

        RewriteEngine on
        RewriteCond %{QUERY_STRING} _query=
        RewriteRule (.*) $1? [R=permanent]

but i also like the idea offering RESTXQ to the public to query the
database as it offers a handy tool with capabilities exceeding a
(preconfigured) search function on a website.
to avoid an open proxy eXist should not preform any request to the
internet, but respond to. there are still some issues to think about
when offering an »public xquery processor« but a RESTXQ endpoint to my
data would be great!

are there any solutions available?

Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
but it disables public request with ?_query completely.

finally i like to encourage anyone to test own instances against this
issue and avoid offering open proxies as they are a risk for the
internet at large.

best,
mathias



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Omar Siam-2

Hi Alister,

That's an interesting solution. As far as I understand your url-pattern it is useful if you don't want to serve things to the general public (we do that). Perhaps with a more elaborate pattern that would also be a solution for sites that want to provide public services. Some pattern that just asks for authentication if you supply a get parameter starting with an _.

Nevertheless: This is one solution based on java intricacies I don't really want to busy myself with. Exist-db to some extend makes you think that you can provide similar authentication services using an XQuery app. That is definitly untrue as of now. I would much prefer the exist-db setup jar and the git code to be more secure and less feature rich.

Best regards

Omar Siam


Am 14.03.2017 um 21:58 schrieb Alister Pillow:
Hi Omar,
I have tested this issue on my demo server running eXist 2.2x and indeed it is a problem.

A possible solution is the use of the Security Constraints provided by the Java Authentication and Authorization Service (JAAS).

This is a fragment from my web-app/WEB-INF/web.xml.

<security-constraint>
<web-resource-collection>
            <web-resource-name>Administration
            </web-resource-name>
            <url-pattern>/apps/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
<security-constraint>
<web-resource-collection>
            <web-resource-name>
                Pekoe Administration
            </web-resource-name>
            <url-pattern>/restxq/pekoe/admin/*</url-pattern>
            <url-pattern>/apps/*</url-pattern> <!— ***************** This blocks any unauthorised access to /exist/apps and redirects to a login form —>
        </web-resource-collection>
        <auth-constraint>
            <description>
                Let only administrators use this app
            </description>
            <!-- Role name can be ** to allow any authenticated user - regardless of role.
                Or, can be * to allow any user who has the top-level role-name (pekoe-users below).
            -->
            <role-name>dba</role-name>
        </auth-constraint>
    </security-constraint>
            <role-name>dba</role-name>
        </auth-constraint>
    </security-constraint>

Today I added the url-pattern /apps/* and tested - I was directed to a login page when running your example. (The base of all url-patterns is /exist)

Once logged-in I was able to access the Dashboard and eXide as usual. 

I’ve been using the JAAS service in production (eXist-db v2.2x) on “demo" and “live" for over 18 months without any problem. I’ll be testing the /apps/* pattern more extensively before applying it to “live” - today if possible.

Using JAAS requires a modification to src/org/exist/security/internal/EXistDBLoginModule.java which provides the User’s Groups as Roles, so all external users of my server must belong to the “pekoe-users” group. 
**** I tested this last year in a pre-release version of eXist 3 ****
(I’ve been hoping for a good reason to suggest it to the group - thank you!)

Regards,
Alister

On 14 Mar 2017, at 9:01 pm, Omar Siam <[hidden email]> wrote:

Dear members of this list!

This issue was reported to one of our apps running on exist-db so I'd 
like to add my thoughts:

* The only solution is to hide the "REST-API" on any public facing 
server and then, if you must, do a explicit redirect in a controller.xql 
to expose the "REST-API" at some path you set.

* The default setting for the "REST-API" is a trap for everyone who is 
not a java servlet specialist like me for example. I use exist-db. I 
usually don't care about the intricacies of it's java foundation

Why isn't it enough to just be careful if the "REST-API" is not hidden?

Because in my opinion you have absolutely no chance of getting this safe!

The logic as explained here: 
https://github.com/eXist-db/documentation/issues/98 is: if "REST-API" is 
not hidden then *every* path in /exist/apps that has no controller.xql 
will be seved by the "REST-API". Let me repeat that: every path that is 
not served by controller.xql. The point is: Every path also means *every 
non-existing path*.

Examples (from a site that maybe is some throw away example instance):

* http://localhost:8080/exist/apps/homepage/index.html?_query=1 (the 
file is no controller.xql, but hey maybe there is no controller.xql in 
http://localhost:8080/exist/apps/homepage)

* http://localhost:8080/exist/apps/fundocs/index.html?_query=1 (as far 
as I know there is a controller.xql in 
http://localhost:8080/exist/apps/fundocs and so 
http://localhost:8080/exist/apps/fundocs/?_query=1 actually does not 
"work")  and then of course very hard to catch: all of the paths that 
*do not exist*

* 
http://localhost:8080/exist/apps/does_not_matter/what/path/even_if_it_does_not_exist.badending?_query=xquery%20version%20%221.0%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

Because of the last feature our idea of reverse proxying right up to or 
into a particular exist-db app (like internal-sever/exist/apps/funcdocs/ 
-> funcdocs.example.org/) exposed the "REST-API". That is very obvious 
to everyone. Isn't it?

So no chance to stop this DDOS service using controller.xql. Also 
imagine what can be done using a combination of util:eval ant 
httpclient:get.

When it comes to public facing services then most developers nowadays 
try to do defensive programming. Better you don't expose a funtionality 
that can bite you.

Therefore I strongly advocate for the exist team to make a prominent 
notice of this flaw that exists at least since exist 2.2 and urge people 
to change their settings *now*. Furthermore please don't distribute 
exist setup in this manner. People should better know that the 
"REST-API" needs to be explicitly enabled because of its implications.

Perhaps there is a chance to check if a path actually exists and/or is 
no file before happily invoking the "REST-API". At least in /exist/apps 
I see no point in _querying an html file.

Also perhaps the non obvious concept of /exist/apps == /exist/rest if 
there is no controller.xql should be reconsidered.

Best regards

Omar Siam


Am 11.03.2017 um 10:05 schrieb Mathias Göbel:
dear list,

i used the _query parameter to write a proxy with just a few basic
functions:
?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)

unfortunately it seems that many eXist-db instances are not secured
appropriately so you can send this query string to *any* public endpoint
(html pages, public REST) of eXist and abuse the instance as an open
proxy (upgrading to ssl when available).

i am fine with a Apache rewrite rule

        RewriteEngine on
        RewriteCond %{QUERY_STRING} _query=
        RewriteRule (.*) $1? [R=permanent]

but i also like the idea offering RESTXQ to the public to query the
database as it offers a handy tool with capabilities exceeding a
(preconfigured) search function on a website.
to avoid an open proxy eXist should not preform any request to the
internet, but respond to. there are still some issues to think about
when offering an »public xquery processor« but a RESTXQ endpoint to my
data would be great!

are there any solutions available?

Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
but it disables public request with ?_query completely.

finally i like to encourage anyone to test own instances against this
issue and avoid offering open proxies as they are a risk for the
internet at large.

best,
mathias



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

ron.vandenbranden
In reply to this post by Mathias Göbel
Hi,

Many thanks for signaling this problem (which it definitely is!), Omar,
and thanks for your suggestion, Alister.

On 14/03/2017 21:59, Alister Pillow wrote:
> A possible solution is the use of the Security Constraints provided by
> the Java Authentication and Authorization Service (JAAS).

I've been trying to follow the suggestions in web.xml
(https://github.com/eXist-db/exist/blob/develop/webapp/WEB-INF/web.xml.tmpl#L87-L101): 


>         <!--
>             If parameter "hidden" is set to true, direct access to the
>             REST interface will be denied. Only requests coming from the
>             URL rewriting will be processed.
>
>             Rationale: on a production server, you may want to store the
>             application within the database, but without allowing users
>             to browse the database contents. By setting hidden=true, you
>             can expose defined URLs to the outside world via URL rewriting
>             while hiding the rest of the database.
>         -->
When setting "hidden=true", this indeed seems to disable access via the
REST interface:
       -the '_query=' request parameter isn't executed anymore
       -http://localhost:8080/exist/rest/db/apps now returns a HTTP 403
"Forbidden" error

Yet, I'm trying to get the URL rewriting part working. I've been trying
this rule in a controller.xql:

     <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
       <forward url="{$exist:root}/rest/db/apps"/>
     </dispatch>

When placed in a 'controlrest' app, this will just internally forward
any requests starting with http://localhost:8080/exist/controlrest/ to
http://localhost:8080/exist/rest/db/apps/. This works fine when
"hidden=false".

Yet, when "hidden=true", http://localhost:8080/exist/controlrest/ *also*
returns an HTTP 403 "Forbidden" error, so this internal forwarding
doesn't seem to work. Or is this not what is meant with "expose defined
URLs to the outside world via URL rewriting"?

Could anyone please clarify how that should be done via URL rewriting?

Best,

Ron




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

controller.xql (130 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Omar Siam-2
Hi Ron,

Sorry, I think I can not be of any help with this, I don't use the
"REST-API". I just cited that comment you also found:

https://github.com/eXist-db/exist/blob/develop/webapp/WEB-INF/web.xml.tmpl#L87-L101

We decided to use restxq instead when we need REST. It seems to us that
the restxq entpoint is able to return an error if the URL is not
registered so I prefer that solution.

Best regards
Omar Siam

Am 15.03.2017 um 13:50 schrieb Ron Van den Branden:

>
>>             If parameter "hidden" is set to true, direct access to the
>>             REST interface will be denied. Only requests coming from the
>>             URL rewriting will be processed.
>
>
> Yet, when "hidden=true", http://localhost:8080/exist/controlrest/ 
> *also* returns an HTTP 403 "Forbidden" error, so this internal
> forwarding doesn't seem to work. Or is this not what is meant with
> "expose defined URLs to the outside world via URL rewriting"?
>
> Could anyone please clarify how that should be done via URL rewriting?
>
> Best,
>
> Ron


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Joe Wicentowski
In reply to this post by ron.vandenbranden
Hi Ron,

In your controller.xql, $exist:root resolves to the root of the
relevant controller hierarchy configured in controller-config.xml.
Could you paste in the relevant root entry you have for your
"controlrest" controller hierarchy?

Joe

>    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>      <forward url="{$exist:root}/rest/db/apps"/>
>    </dispatch>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

wshager
Guys, you're making this way too complex. The problem is that httpclient let's you do stuff outside the domain. This should be stopped short by requiring the URL be inside the domain, or additional settings are required, e.g. CORS or dba rights.

Op 15 mrt. 2017 5:29 p.m. schreef "Joe Wicentowski" <[hidden email]>:
Hi Ron,

In your controller.xql, $exist:root resolves to the root of the
relevant controller hierarchy configured in controller-config.xml.
Could you paste in the relevant root entry you have for your
"controlrest" controller hierarchy?

Joe

>    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>      <forward url="{$exist:root}/rest/db/apps"/>
>    </dispatch>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Mathias Göbel

Making httpclient namepsace functions available for dba only is a good idea. on the other side: try

doc("https://en.wikipedia.org/wiki/Main_Page")

Best,
Mathias

On 15.03.2017 17:58, W.S. Hager wrote:
Guys, you're making this way too complex. The problem is that httpclient let's you do stuff outside the domain. This should be stopped short by requiring the URL be inside the domain, or additional settings are required, e.g. CORS or dba rights.

Op 15 mrt. 2017 5:29 p.m. schreef "Joe Wicentowski" <[hidden email]>:
Hi Ron,

In your controller.xql, $exist:root resolves to the root of the
relevant controller hierarchy configured in controller-config.xml.
Could you paste in the relevant root entry you have for your
"controlrest" controller hierarchy?

Joe

>    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>      <forward url="{$exist:root}/rest/db/apps"/>
>    </dispatch>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

--
Mathias Göbel
Abt. Forschung & Entwicklung

Georg-August-Universität Göttingen
Niedersächsische Staats- und Universitätsbibliothek Göttingen
D-37070 Göttingen

Papendiek 14 (hist. Gebäude, Raum 2.408)
+49 551 39-20184 (Tel.)
+49 551 39-33856 (Fax.)

[hidden email]
http://www.sub.uni-goettingen.de

--

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

wshager
I know, same story there. fn:doc is essentially just a client request + parse.

2017-03-15 21:37 GMT+01:00 Mathias Göbel <[hidden email]>:

Making httpclient namepsace functions available for dba only is a good idea. on the other side: try

doc("https://en.wikipedia.org/wiki/Main_Page")

Best,
Mathias

On 15.03.2017 17:58, W.S. Hager wrote:
Guys, you're making this way too complex. The problem is that httpclient let's you do stuff outside the domain. This should be stopped short by requiring the URL be inside the domain, or additional settings are required, e.g. CORS or dba rights.

Op 15 mrt. 2017 5:29 p.m. schreef "Joe Wicentowski" <[hidden email]>:
Hi Ron,

In your controller.xql, $exist:root resolves to the root of the
relevant controller hierarchy configured in controller-config.xml.
Could you paste in the relevant root entry you have for your
"controlrest" controller hierarchy?

Joe

>    <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>      <forward url="{$exist:root}/rest/db/apps"/>
>    </dispatch>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

--
Mathias Göbel
Abt. Forschung & Entwicklung

Georg-August-Universität Göttingen
Niedersächsische Staats- und Universitätsbibliothek Göttingen
D-37070 Göttingen

Papendiek 14 (hist. Gebäude, Raum 2.408)
<a href="tel:+49%20551%203920184" value="+495513920184" target="_blank">+49 551 39-20184 (Tel.)
<a href="tel:+49%20551%203933856" value="+495513933856" target="_blank">+49 551 39-33856 (Fax.)

[hidden email]
http://www.sub.uni-goettingen.de

--

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open




--

W.S. Hager
Lagua Web Solutions
http://lagua.nl


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

ron.vandenbranden
In reply to this post by Joe Wicentowski
Hi Joe,

I'm not sure how to tell; is this the info you need?
     -controller.xql lives on the file system* in
${exist.home}/webapps/controlrest/
     -I don't know what $exist:root is resolving to exactly, but it
works when I'm setting "hidden=false", so I guess it is resolving
correctly. Setting "hidden=true" triggers the 403 error, so it seems to
be resolving correctly, too, only it obviously is not getting past the
"hidden=true" setting.

* = Unfortunately, I've just discovered that this example behaves
differently on the file system vs stored in a /db/apps/controlrest/
collection, blast. So, sorry, on top of the problem I'm trying to
address, I feel I'm heading for a fresh crash against the mysteries that
still surround the different execution contexts for URL rewriting (file
system vs inside the database).

I could be completely on the wrong track, so perhaps I should just state
my problem from scratch:
     -I'm very worried about the _query vulnerability I'm exposing my
webapps to
     -I want to solve this ASAP
     -I try to follow the advice inside web.xml, viz. to hide the REST
interface from the outside world
     -I'm looking for a way to address resources stored in the database
from a controller.xql script, regardless of where it lives (file system
or inside the database)

With an example:
     -I have a data resource in my database at
/db/apps/data/my-app/some-data.xml
     -when entering a URL, e.g.
         -http://localhost:8080/exist/my-app/my-data-please.xml (when
the app is stored on the file system), or
         -http://localhost:8080/exist/apps/my-app/my-data-please.xml 
(when the app is stored in the database)
      ...I want a rule in the controller.xql file catch this request and
retrieve the file located at /db/apps/data/my-app/some-data.xml
      -since the app has to run in different contexts (file system and
inside the database), I should find a way to retrieve that file from the
database from these different contexts

If anyone has an idea, I'd be very grateful!

Best,

Ron

On 15/03/2017 17:27, Joe Wicentowski wrote:

> Hi Ron,
>
> In your controller.xql, $exist:root resolves to the root of the
> relevant controller hierarchy configured in controller-config.xml.
> Could you paste in the relevant root entry you have for your
> "controlrest" controller hierarchy?
>
> Joe
>
>>     <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>>       <forward url="{$exist:root}/rest/db/apps"/>
>>     </dispatch>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

ron.vandenbranden
Hi Joe

Ah, sorry, I only now fully understood your question:

On 15/03/2017 17:27, Joe Wicentowski wrote:

> In your controller.xql, $exist:root resolves to the root of the
> relevant controller hierarchy configured in controller-config.xml.
> Could you paste in the relevant root entry you have for your
> "controlrest" controller hierarchy?

I think these are the relevant entries (just the default settings):

   <root pattern="/apps" path="xmldb:exist:///db/apps"/>
   <root pattern=".*" path="/"/>

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Jonathan Rowell
In reply to this post by ron.vandenbranden

Hi all,


My twopence worth


>>I feel I'm heading for a fresh crash against the mysteries that 
still surround the different execution contexts for URL rewriting (file 
system vs inside the database).


said Ron.


I quite agree. I think the best way is to front end the system with Apache (using simple rewrite) or, if you want to control access at a very fine level, with Node/Express/Passport.


Run eXist on 8081, Apache on 8080, deny 8081 from outside and send a 40x to any 8080 URL with _query in it.


Jonathan

From: Ron Van den Branden <[hidden email]>
Sent: Wednesday, March 15, 2017 10:11 PM
To: Joe Wicentowski
Cc: [hidden email]
Subject: Re: [Exist-open] get parameter _query = open proxy
 
Hi Joe,

I'm not sure how to tell; is this the info you need?
     -controller.xql lives on the file system* in
${exist.home}/webapps/controlrest/
     -I don't know what $exist:root is resolving to exactly, but it
works when I'm setting "hidden=false", so I guess it is resolving
correctly. Setting "hidden=true" triggers the 403 error, so it seems to
be resolving correctly, too, only it obviously is not getting past the
"hidden=true" setting.

* = Unfortunately, I've just discovered that this example behaves
differently on the file system vs stored in a /db/apps/controlrest/
collection, blast. So, sorry, on top of the problem I'm trying to
address, I feel I'm heading for a fresh crash against the mysteries that
still surround the different execution contexts for URL rewriting (file
system vs inside the database).

I could be completely on the wrong track, so perhaps I should just state
my problem from scratch:
     -I'm very worried about the _query vulnerability I'm exposing my
webapps to
     -I want to solve this ASAP
     -I try to follow the advice inside web.xml, viz. to hide the REST
interface from the outside world
     -I'm looking for a way to address resources stored in the database
from a controller.xql script, regardless of where it lives (file system
or inside the database)

With an example:
     -I have a data resource in my database at
/db/apps/data/my-app/some-data.xml
     -when entering a URL, e.g.
         -http://localhost:8080/exist/my-app/my-data-please.xml (when
the app is stored on the file system), or
         -http://localhost:8080/exist/apps/my-app/my-data-please.xml
(when the app is stored in the database)
      ...I want a rule in the controller.xql file catch this request and
retrieve the file located at /db/apps/data/my-app/some-data.xml
      -since the app has to run in different contexts (file system and
inside the database), I should find a way to retrieve that file from the
database from these different contexts

If anyone has an idea, I'd be very grateful!

Best,

Ron

On 15/03/2017 17:27, Joe Wicentowski wrote:
> Hi Ron,
>
> In your controller.xql, $exist:root resolves to the root of the
> relevant controller hierarchy configured in controller-config.xml.
> Could you paste in the relevant root entry you have for your
> "controlrest" controller hierarchy?
>
> Joe
>
>>     <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
>>       <forward url="{$exist:root}/rest/db/apps"/>
>>     </dispatch>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
lists.sourceforge.net
Your email address: Your name (optional): You may enter a privacy password below. This provides only mild security, but should prevent others from messing with ...



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

ron.vandenbranden

Hi,


On 16/03/2017 13:47, Jonathan Rowell wrote:

I quite agree. I think the best way is to front end the system with Apache (using simple rewrite) or, if you want to control access at a very fine level, with Node/Express/Passport.



That's what I've done, as a first measure: I've left the REST service accessible in web.xml (hidden=false) but since I'm using Apache as reverse proxy, I've added a rewrite rule there that filters out "_query" key/value pairs from the URL parameters and rewrites the URL:
  RewriteCond %{QUERY_STRING}  (.*)(?:^|&)_query=(?:[^&]*)((?:&|$).*)
  RewriteCond %1%2 (^|&)([^&].*|$)
  RewriteRule ^(.*)$ $1?%2 [R=permanent]
So the requests that get to eXist will never contain the _query parameter.

But, still, I'm hoping to find a way to fix this within eXist. Ideally by finding out how the db can be accessed from the controller on a file system (which is my main problem, I think); otherwise it would be a useful feature if users could disable _query processing for the REST servlet.

Best,

Ron

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

ron.vandenbranden

Ok,


I think I've finally found a working solution for accessing database resources that allows me to switch REST to "hidden", in order to avoid the risks attached to execution of query code in the _query request parameter.


Background: my apps need to be deployable in different contexts: on the file system, and in the database. I'm using the default root in controller-config.xml:

  <root pattern="/apps" path="xmldb:exist:///db/apps"/>
  <root pattern=".*" path="/"/>
For the most part, I'm accessing the database resources via collection() and doc() functions inside XQuery scripts, so no problem so far. Yet, for some cases, I need to access the database resources via HTTP requests. When REST is enabled, that's easy enough via URLs like e.g. http://localhost:8080/exist/rest/db/apps/dashboard/repo.xml. However, when REST is "hidden" by setting "hidden=true" in web.xml, this URL returns a HTTP 403 "Forbidden" error.

Hence, I've been looking for a way to construct URLs that could internally be forwarded to resources in the database, via controller.xql, and seem to have found it. For the sake of this example, suppose a webapp is stored either:
    -in an ${exist.home}/webapps/rest-in-peace folder on the file system, or
    -in a /db/apps/rest-in-peace collection in the database
In order to keep this example self-contained, let's assume that the data I want to retrieve is located at /db/apps/dashboard/repo.xml (which is installed by default). Now, as already mentioned, http://localhost:8080/exist/rest/db/apps/dashboard/repo.xml won't work with REST disabled, so the problem I have to solve is finding a way to still retrieve the data I need for my webapp via URL.

This simple controller.xql example illustrates how such a URL mapping can be defined:
    if (matches($exist:path, '^/get-my-data.xml$')) then
      <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
        <forward url="/apps/dashboard/repo.xml" absolute="yes"/>
      </dispatch>
    else 
      <dispatch xmlns="http://exist.sourceforge.net/NS/exist">
        <ignore/>
      </dispatch>
This works when the app is deployed in both contexts:
    -on the file system: http://localhost:8080/exist/rest-in-peace/get-my-data.xml
    -in the database: http://localhost:8080/exist/apps/rest-in-peace/get-my-data.xml

I'm astonished how easy this looks, and apologies if I'm stating the obvious here, but I hope it can help anyone else struggling with this security issue.

Best,

Ron

On 16/03/2017 14:06, Ron Van den Branden wrote:

Hi,


On 16/03/2017 13:47, Jonathan Rowell wrote:

I quite agree. I think the best way is to front end the system with Apache (using simple rewrite) or, if you want to control access at a very fine level, with Node/Express/Passport.



That's what I've done, as a first measure: I've left the REST service accessible in web.xml (hidden=false) but since I'm using Apache as reverse proxy, I've added a rewrite rule there that filters out "_query" key/value pairs from the URL parameters and rewrites the URL:
  RewriteCond %{QUERY_STRING}  (.*)(?:^|&)_query=(?:[^&]*)((?:&|$).*)
  RewriteCond %1%2 (^|&)([^&].*|$)
  RewriteRule ^(.*)$ $1?%2 [R=permanent]
So the requests that get to eXist will never contain the _query parameter.

But, still, I'm hoping to find a way to fix this within eXist. Ideally by finding out how the db can be accessed from the controller on a file system (which is my main problem, I think); otherwise it would be a useful feature if users could disable _query processing for the REST servlet.

Best,

Ron


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

controller.xql (322 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: get parameter _query = open proxy

Adam Retter
In reply to this post by Mathias Göbel
I have opened a PR here - https://github.com/eXist-db/exist/pull/1419

This provides two options for disabling XQuery and XUpdate submission,
see conf.xml.tmpl in the PR for the details.

On 11 March 2017 at 04:05, Mathias Göbel <[hidden email]> wrote:

> dear list,
>
> i used the _query parameter to write a proxy with just a few basic
> functions:
> ?_query=xquery%20version%20%223.1%22;response:stream-binary(%20xs:base64Binary(%20data(httpclient:get(xs:anyURI(%22http://24.media.tumblr.com/tumblr_lt8vrdas9o1qb8xalo1_400.jpg%22),%20false(),%20())//httpclient:body))%20,%20%22image/jpg%22)
>
> unfortunately it seems that many eXist-db instances are not secured
> appropriately so you can send this query string to *any* public endpoint
> (html pages, public REST) of eXist and abuse the instance as an open
> proxy (upgrading to ssl when available).
>
> i am fine with a Apache rewrite rule
>
>         RewriteEngine on
>         RewriteCond %{QUERY_STRING} _query=
>         RewriteRule (.*) $1? [R=permanent]
>
> but i also like the idea offering RESTXQ to the public to query the
> database as it offers a handy tool with capabilities exceeding a
> (preconfigured) search function on a website.
> to avoid an open proxy eXist should not preform any request to the
> internet, but respond to. there are still some issues to think about
> when offering an »public xquery processor« but a RESTXQ endpoint to my
> data would be great!
>
> are there any solutions available?
>
> Joe pointed to $EXIST_HOME/webapp/WEB-INF/web.xml hidden=true parameter
> but it disables public request with ?_query completely.
>
> finally i like to encourage anyone to test own instances against this
> issue and avoid offering open proxies as they are a risk for the
> internet at large.
>
> best,
> mathias
>
> --
>
> Mathias Göbel
> Research and Development
>
> Georg-August-Universität Göttingen
> State and University Library Göttingen
> D-37070 Göttingen
>
> Papendiek 14 (hist. building, room 2.408)
> +49 551 39-20184 (Tel.)
>
> [hidden email]
> http://www.sub.uni-goettingen.de
>
> ------------------------------------------------------------------------------
> Announcing the Oxford Dictionaries API! The API offers world-renowned
> dictionary content that is easy and intuitive to access. Sign up for an
> account today to start using our lexical data to power your apps and
> projects. Get started today and enter our developer competition.
> http://sdm.link/oxford
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Loading...