hmac signature for AWS S3

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: hmac signature for AWS S3

nsincaglia
Hi Alister,
I was just looking up an old question I once asked on the forum because I was going to suggest you write a recursive function. 


If you read through this thread, you will see that Peter Stadler commented that he experienced the same “too many open files” error. He says:
"One issue I faced with the httpclient module once was that the connections stayed in CLOSE_WAIT which resulted in „too many open files“. You can check for open connections with lsof -i but if that was the issue you should see the „too many open files“ in the log files …”

I believe Peter was using the httpclient, not the Expath http client, so I am not sure if your issue is the same. Perhaps his comments may get you looking in the right direction. I just did a search on “CLOSE_WAIT” and there are quite a few entries in the forum. 


Nick

 
On Feb 10, 2017, at 8:29 PM, Alister Pillow <[hidden email]> wrote:

eXist-db 2.2x, 10GB available memory.
Uploading binary files to AWS S3 using EXPath HTTP client.

And I’ve just hit this issue - it’s not a Memory problem, it’s Too Many Open Files. 
On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:

Since you mention that you are working with binary data, I feel compelled to also mention that Adam also identified and fixed a bug in eXist-db where if you were working with binary files, eXist-db would not release the memory those binary files require, even after you were done performing your operation on it. 

Once I have removed these files, it will be much easier to begin the upgrade-to-3 process.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: hmac signature for AWS S3

Joe Wicentowski
Are you closing the connection?

  <hc:request href="{$url}" method="get">
    <hc:header name="Connection" value="close"/>
  </hc:request>



On Sat, Feb 11, 2017 at 11:50 AM, Nick Sincaglia <[hidden email]> wrote:

> Hi Alister,
> I was just looking up an old question I once asked on the forum because I
> was going to suggest you write a recursive function.
>
> http://exist.2174344.n4.nabble.com/Recursive-function-question-tc4663361.html#a4663370
>
> If you read through this thread, you will see that Peter Stadler commented
> that he experienced the same “too many open files” error. He says:
> "One issue I faced with the httpclient module once was that the connections
> stayed in CLOSE_WAIT which resulted in „too many open files“. You can check
> for open connections with lsof -i but if that was the issue you should see
> the „too many open files“ in the log files …”
>
> I believe Peter was using the httpclient, not the Expath http client, so I
> am not sure if your issue is the same. Perhaps his comments may get you
> looking in the right direction. I just did a search on “CLOSE_WAIT” and
> there are quite a few entries in the forum.
>
> http://exist.2174344.n4.nabble.com/template/NamlServlet.jtp?macro=search_page&node=2174344&query=CLOSE_WAIT&n=2174344
>
> Nick
>
>
>
> On Feb 10, 2017, at 8:29 PM, Alister Pillow <[hidden email]> wrote:
>
> eXist-db 2.2x, 10GB available memory.
> Uploading binary files to AWS S3 using EXPath HTTP client.
>
> And I’ve just hit this issue - it’s not a Memory problem, it’s Too Many Open
> Files.
>
> On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:
>
> Since you mention that you are working with binary data, I feel compelled to
> also mention that Adam also identified and fixed a bug in eXist-db where if
> you were working with binary files, eXist-db would not release the memory
> those binary files require, even after you were done performing your
> operation on it.
>
>
> Once I have removed these files, it will be much easier to begin the
> upgrade-to-3 process.
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: hmac signature for AWS S3

Michael Westbay-2
Joe has the right answer. I had this problem until I insert the Connection-close header into all of my HTTP requests.

It seems that the connection is keep-open by default, but never reused or properly closed when it goes out of scope. I have a feeling that fixing this "bug" is more complicated the closing it after it's gone out of scope.

This work around works.


2017-02-12 4:26 GMT+09:00 Joe Wicentowski <[hidden email]>:
Are you closing the connection?

  <hc:request href="{$url}" method="get">
    <hc:header name="Connection" value="close"/>
  </hc:request>



On Sat, Feb 11, 2017 at 11:50 AM, Nick Sincaglia <[hidden email]> wrote:
> Hi Alister,
> I was just looking up an old question I once asked on the forum because I
> was going to suggest you write a recursive function.
>
> http://exist.2174344.n4.nabble.com/Recursive-function-question-tc4663361.html#a4663370
>
> If you read through this thread, you will see that Peter Stadler commented
> that he experienced the same “too many open files” error. He says:
> "One issue I faced with the httpclient module once was that the connections
> stayed in CLOSE_WAIT which resulted in „too many open files“. You can check
> for open connections with lsof -i but if that was the issue you should see
> the „too many open files“ in the log files …”
>
> I believe Peter was using the httpclient, not the Expath http client, so I
> am not sure if your issue is the same. Perhaps his comments may get you
> looking in the right direction. I just did a search on “CLOSE_WAIT” and
> there are quite a few entries in the forum.
>
> http://exist.2174344.n4.nabble.com/template/NamlServlet.jtp?macro=search_page&node=2174344&query=CLOSE_WAIT&n=2174344
>
> Nick
>
>
>
> On Feb 10, 2017, at 8:29 PM, Alister Pillow <[hidden email]> wrote:
>
> eXist-db 2.2x, 10GB available memory.
> Uploading binary files to AWS S3 using EXPath HTTP client.
>
> And I’ve just hit this issue - it’s not a Memory problem, it’s Too Many Open
> Files.
>
> On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:
>
> Since you mention that you are working with binary data, I feel compelled to
> also mention that Adam also identified and fixed a bug in eXist-db where if
> you were working with binary files, eXist-db would not release the memory
> those binary files require, even after you were done performing your
> operation on it.
>
>
> Once I have removed these files, it will be much easier to begin the
> upgrade-to-3 process.
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



--
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|

Re: hmac signature for AWS S3

Alister Pillow-3
Brilliant - I’ll try that. Thanks to all.

Yesterday I finished transferring some 35,000 files out of eXist-db and into AWS S3. From now on, when the client uploads a PDF or similar “associated file”, it will go straight from their browser into their S3 bucket. But they won’t know the difference between it being in S3 or in the /db.


On 12 Feb 2017, at 11:07 am, Michael Westbay <[hidden email]> wrote:

Joe has the right answer. I had this problem until I insert the Connection-close header into all of my HTTP requests.

It seems that the connection is keep-open by default, but never reused or properly closed when it goes out of scope. I have a feeling that fixing this "bug" is more complicated the closing it after it's gone out of scope.

This work around works.


2017-02-12 4:26 GMT+09:00 Joe Wicentowski <[hidden email]>:
Are you closing the connection?

  <hc:request href="{$url}" method="get">
    <hc:header name="Connection" value="close"/>
  </hc:request>



On Sat, Feb 11, 2017 at 11:50 AM, Nick Sincaglia <[hidden email]> wrote:
> Hi Alister,
> I was just looking up an old question I once asked on the forum because I
> was going to suggest you write a recursive function.
>
> http://exist.2174344.n4.nabble.com/Recursive-function-question-tc4663361.html#a4663370
>
> If you read through this thread, you will see that Peter Stadler commented
> that he experienced the same “too many open files” error. He says:
> "One issue I faced with the httpclient module once was that the connections
> stayed in CLOSE_WAIT which resulted in „too many open files“. You can check
> for open connections with lsof -i but if that was the issue you should see
> the „too many open files“ in the log files …”
>
> I believe Peter was using the httpclient, not the Expath http client, so I
> am not sure if your issue is the same. Perhaps his comments may get you
> looking in the right direction. I just did a search on “CLOSE_WAIT” and
> there are quite a few entries in the forum.
>
> http://exist.2174344.n4.nabble.com/template/NamlServlet.jtp?macro=search_page&node=2174344&query=CLOSE_WAIT&n=2174344
>
> Nick
>
>
>
> On Feb 10, 2017, at 8:29 PM, Alister Pillow <[hidden email]> wrote:
>
> eXist-db 2.2x, 10GB available memory.
> Uploading binary files to AWS S3 using EXPath HTTP client.
>
> And I’ve just hit this issue - it’s not a Memory problem, it’s Too Many Open
> Files.
>
> On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:
>
> Since you mention that you are working with binary data, I feel compelled to
> also mention that Adam also identified and fixed a bug in eXist-db where if
> you were working with binary files, eXist-db would not release the memory
> those binary files require, even after you were done performing your
> operation on it.
>
>
> Once I have removed these files, it will be much easier to begin the
> upgrade-to-3 process.
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open



--
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
12