Too many CLOSE_WAIT status in jetty

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Too many CLOSE_WAIT status in jetty

Masoud Vakili
Hi all,
I had configured eXist-db to be behind of an other web server, at the first, I configured it to proxying eXist-db behind an Apache web server; all of things were good until I understood that the performance has decreased very much, though the memory and cpu usage were normal! by netstat command I noticed that too many connections are in CLOSE_WAIT status and not terminated!! I searched very much and imagined the bug is for apache! therefore I change Apache with Nginx web server! but the problem not solved!! and after 3 or 4 days, too many connections will be stays on CLOSE_WAIT still! the solution is restart the eXist process (java)! (right now, 214 connections are in close_wait status!)
so, I guessing the problem may be from jetty or jvm, therefore I searched around this issue and found following link (and some other links):
http://stackoverflow.com/questions/23644090/too-many-close-wait-in-jetty-9-0-3
http://stackoverflow.com/questions/28875406/why-am-i-seeing-lots-of-sockets-in-close-wait-status-when-webservice-stops-worki
...
now, i want to help for solve this issue! what is the solution?
(in the first link, writer is mentioned to use closeIdleConnections(0) function; is it helpful? Where is it being?)

Thanks

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: Too many CLOSE_WAIT status in jetty

Masoud Vakili
eXist-db version is 2.2
java version is oracle java 1.7
OS is Centos 7


On Tue, Feb 16, 2016 at 12:27 AM, Masoud Vakili <[hidden email]> wrote:
Hi all,
I had configured eXist-db to be behind of an other web server, at the first, I configured it to proxying eXist-db behind an Apache web server; all of things were good until I understood that the performance has decreased very much, though the memory and cpu usage were normal! by netstat command I noticed that too many connections are in CLOSE_WAIT status and not terminated!! I searched very much and imagined the bug is for apache! therefore I change Apache with Nginx web server! but the problem not solved!! and after 3 or 4 days, too many connections will be stays on CLOSE_WAIT still! the solution is restart the eXist process (java)! (right now, 214 connections are in close_wait status!)
so, I guessing the problem may be from jetty or jvm, therefore I searched around this issue and found following link (and some other links):
http://stackoverflow.com/questions/23644090/too-many-close-wait-in-jetty-9-0-3
http://stackoverflow.com/questions/28875406/why-am-i-seeing-lots-of-sockets-in-close-wait-status-when-webservice-stops-worki
...
now, i want to help for solve this issue! what is the solution?
(in the first link, writer is mentioned to use closeIdleConnections(0) function; is it helpful? Where is it being?)

Thanks


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: Too many CLOSE_WAIT status in jetty

Peter Stadler
In reply to this post by Masoud Vakili
Hi Masoud,

I was experiencing this sort of error in connection with the EXPath HTTPClient, see http://exist.2174344.n4.nabble.com/EXPath-HTTPClient-not-closing-connections-td4661258.html

Hope it helps?!
Best
Peter

> Am 15.02.2016 um 21:57 schrieb Masoud Vakili <[hidden email]>:
>
> Hi all,
> I had configured eXist-db to be behind of an other web server, at the first, I configured it to proxying eXist-db behind an Apache web server; all of things were good until I understood that the performance has decreased very much, though the memory and cpu usage were normal! by netstat command I noticed that too many connections are in CLOSE_WAIT status and not terminated!! I searched very much and imagined the bug is for apache! therefore I change Apache with Nginx web server! but the problem not solved!! and after 3 or 4 days, too many connections will be stays on CLOSE_WAIT still! the solution is restart the eXist process (java)! (right now, 214 connections are in close_wait status!)
> so, I guessing the problem may be from jetty or jvm, therefore I searched around this issue and found following link (and some other links):
> http://stackoverflow.com/questions/23644090/too-many-close-wait-in-jetty-9-0-3
> http://stackoverflow.com/questions/28875406/why-am-i-seeing-lots-of-sockets-in-close-wait-status-when-webservice-stops-worki
> ...
> now, i want to help for solve this issue! what is the solution?
> (in the first link, writer is mentioned to use closeIdleConnections(0) function; is it helpful? Where is it being?)
>
> Thanks
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open
--
Peter Stadler
Carl-Maria-von-Weber-Gesamtausgabe
Arbeitsstelle Detmold
Hornsche Str. 39
D-32756 Detmold
Tel. +49 5231 975-676
Fax: +49 5231 975-668
stadler at weber-gesamtausgabe.de
www.weber-gesamtausgabe.de


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Too many CLOSE_WAIT status in jetty

Masoud Vakili
Hi Peter Stadler and thanks for reply.
I am testing <http:header name="Connection" value="close"/> in my httpclient requests for one week. then will report the result.

Best
Masoud

On Tue, Feb 16, 2016 at 11:59 AM, Peter Stadler <[hidden email]> wrote:
Hi Masoud,

I was experiencing this sort of error in connection with the EXPath HTTPClient, see http://exist.2174344.n4.nabble.com/EXPath-HTTPClient-not-closing-connections-td4661258.html

Hope it helps?!
Best
Peter

> Am 15.02.2016 um 21:57 schrieb Masoud Vakili <[hidden email]>:
>
> Hi all,
> I had configured eXist-db to be behind of an other web server, at the first, I configured it to proxying eXist-db behind an Apache web server; all of things were good until I understood that the performance has decreased very much, though the memory and cpu usage were normal! by netstat command I noticed that too many connections are in CLOSE_WAIT status and not terminated!! I searched very much and imagined the bug is for apache! therefore I change Apache with Nginx web server! but the problem not solved!! and after 3 or 4 days, too many connections will be stays on CLOSE_WAIT still! the solution is restart the eXist process (java)! (right now, 214 connections are in close_wait status!)
> so, I guessing the problem may be from jetty or jvm, therefore I searched around this issue and found following link (and some other links):
> http://stackoverflow.com/questions/23644090/too-many-close-wait-in-jetty-9-0-3
> http://stackoverflow.com/questions/28875406/why-am-i-seeing-lots-of-sockets-in-close-wait-status-when-webservice-stops-worki
> ...
> now, i want to help for solve this issue! what is the solution?
> (in the first link, writer is mentioned to use closeIdleConnections(0) function; is it helpful? Where is it being?)
>
> Thanks
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open

--
Peter Stadler
Carl-Maria-von-Weber-Gesamtausgabe
Arbeitsstelle Detmold
Hornsche Str. 39
D-32756 Detmold
Tel. +49 5231 975-676
Fax: +49 5231 975-668
stadler at weber-gesamtausgabe.de
www.weber-gesamtausgabe.de



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: Too many CLOSE_WAIT status in jetty

Mathias Göbel
I think this is a bug in one of the recent RCs. I encountered this issue and went a step back to 2.2 - with exactly the same code this problem did not occur since then. There were some reports on "too many open files" - which is the error message jetty thtows when ulimit is reached...
it seems to me, that the set-header with connection "close" do not help at the moment.

best,
mathias


On 02/16/2016 02:45 PM, Masoud Vakili wrote:
Hi Peter Stadler and thanks for reply.
I am testing <http:header name="Connection" value="close"/> in my httpclient requests for one week. then will report the result.

Best
Masoud

On Tue, Feb 16, 2016 at 11:59 AM, Peter Stadler <[hidden email]> wrote:
Hi Masoud,

I was experiencing this sort of error in connection with the EXPath HTTPClient, see http://exist.2174344.n4.nabble.com/EXPath-HTTPClient-not-closing-connections-td4661258.html

Hope it helps?!
Best
Peter

> Am 15.02.2016 um 21:57 schrieb Masoud Vakili <[hidden email]>:
>
> Hi all,
> I had configured eXist-db to be behind of an other web server, at the first, I configured it to proxying eXist-db behind an Apache web server; all of things were good until I understood that the performance has decreased very much, though the memory and cpu usage were normal! by netstat command I noticed that too many connections are in CLOSE_WAIT status and not terminated!! I searched very much and imagined the bug is for apache! therefore I change Apache with Nginx web server! but the problem not solved!! and after 3 or 4 days, too many connections will be stays on CLOSE_WAIT still! the solution is restart the eXist process (java)! (right now, 214 connections are in close_wait status!)
> so, I guessing the problem may be from jetty or jvm, therefore I searched around this issue and found following link (and some other links):
> http://stackoverflow.com/questions/23644090/too-many-close-wait-in-jetty-9-0-3
> http://stackoverflow.com/questions/28875406/why-am-i-seeing-lots-of-sockets-in-close-wait-status-when-webservice-stops-worki
> ...
> now, i want to help for solve this issue! what is the solution?
> (in the first link, writer is mentioned to use closeIdleConnections(0) function; is it helpful? Where is it being?)
>
> Thanks
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open

--
Peter Stadler
Carl-Maria-von-Weber-Gesamtausgabe
Arbeitsstelle Detmold
Hornsche Str. 39
D-32756 Detmold
Tel. +49 5231 975-676
Fax: +49 5231 975-668
stadler at weber-gesamtausgabe.de
www.weber-gesamtausgabe.de




------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140


_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Mathias Göbel
Abt. Forschung & Entwicklung

Georg-August-Universität Göttingen
Niedersächsische Staats- und Universitätsbibliothek Göttingen
D-37070 Göttingen

Papendiek 14 (hist. Gebäude, Raum 2.211)
+49 551 39-12238 (Tel.)
+49 551 39-33856 (Fax.)

[hidden email]
http://www.sub.uni-goettingen.de

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: Too many CLOSE_WAIT status in jetty

Masoud Vakili
Mathias:
I think this is a bug in one of the recent RCs. I encountered this issue and went a step back to 2.2 ... 
but I have this problem with eXist-db 2.2 and not recent RCs.
However, I know that set-header with connection "close" is not the main and true solution! but i don't have any idea!!! unfortunately, I have this problem for many month (with both apache and nginx, too) and it takes me very time for research and test. I think the problem most likely is originated from jetty or jvm!

On Tue, Feb 16, 2016 at 6:09 PM, Mathias Göbel <[hidden email]> wrote:
I think this is a bug in one of the recent RCs. I encountered this issue and went a step back to 2.2 - with exactly the same code this problem did not occur since then. There were some reports on "too many open files" - which is the error message jetty thtows when ulimit is reached...
it seems to me, that the set-header with connection "close" do not help at the moment.

best,
mathias


On 02/16/2016 02:45 PM, Masoud Vakili wrote:
Hi Peter Stadler and thanks for reply.
I am testing <http:header name="Connection" value="close"/> in my httpclient requests for one week. then will report the result.

Best
Masoud

On Tue, Feb 16, 2016 at 11:59 AM, Peter Stadler <[hidden email]> wrote:
Hi Masoud,

I was experiencing this sort of error in connection with the EXPath HTTPClient, see http://exist.2174344.n4.nabble.com/EXPath-HTTPClient-not-closing-connections-td4661258.html

Hope it helps?!
Best
Peter

> Am 15.02.2016 um 21:57 schrieb Masoud Vakili <[hidden email][hidden email]>:
>
> Hi all,
> I had configured eXist-db to be behind of an other web server, at the first, I configured it to proxying eXist-db behind an Apache web server; all of things were good until I understood that the performance has decreased very much, though the memory and cpu usage were normal! by netstat command I noticed that too many connections are in CLOSE_WAIT status and not terminated!! I searched very much and imagined the bug is for apache! therefore I change Apache with Nginx web server! but the problem not solved!! and after 3 or 4 days, too many connections will be stays on CLOSE_WAIT still! the solution is restart the eXist process (java)! (right now, 214 connections are in close_wait status!)
> so, I guessing the problem may be from jetty or jvm, therefore I searched around this issue and found following link (and some other links):
> http://stackoverflow.com/questions/23644090/too-many-close-wait-in-jetty-9-0-3
> http://stackoverflow.com/questions/28875406/why-am-i-seeing-lots-of-sockets-in-close-wait-status-when-webservice-stops-worki
> ...
> now, i want to help for solve this issue! what is the solution?
> (in the first link, writer is mentioned to use closeIdleConnections(0) function; is it helpful? Where is it being?)
>
> Thanks
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open

--
Peter Stadler
Carl-Maria-von-Weber-Gesamtausgabe
Arbeitsstelle Detmold
Hornsche Str. 39
D-32756 Detmold
Tel. +49 5231 975-676
Fax: +49 5231 975-668
stadler at weber-gesamtausgabe.de
www.weber-gesamtausgabe.de




------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140


_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Mathias Göbel
Abt. Forschung & Entwicklung

Georg-August-Universität Göttingen
Niedersächsische Staats- und Universitätsbibliothek Göttingen
D-37070 Göttingen

Papendiek 14 (hist. Gebäude, Raum 2.211)
+49 551 39-12238 (Tel.)
+49 551 39-33856 (Fax.)

[hidden email]
http://www.sub.uni-goettingen.de

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: Too many CLOSE_WAIT status in jetty

Mathias Göbel
OK, so for 2.2 all requests are getting closed with the correct header.

good luck,
mathias


On 02/16/2016 04:42 PM, Masoud Vakili wrote:
but I have this problem with eXist-db 2.2 and not recent RCs.

-- 
Mathias Göbel
Abt. Forschung & Entwicklung

Georg-August-Universität Göttingen
Niedersächsische Staats- und Universitätsbibliothek Göttingen
D-37070 Göttingen

Papendiek 14 (hist. Gebäude, Raum 2.211)
+49 551 39-12238 (Tel.)
+49 551 39-33856 (Fax.)

[hidden email]
http://www.sub.uni-goettingen.de

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open