Quantcast

hmac signature for AWS S3

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hmac signature for AWS S3

Alister Pillow-3
Hi,
Using eXist 2.2 with the EXPath Cryptographic Module v 0.3.4

I’m trying to build a File Upload form for AWS S3. The docs start here:
http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html

In particular, a Signature must be created using the HMAC function, which is provided by the Expath Crpyto module.
This page provides examples of creating the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
It provides test data showing the values produced for each step of the signature.

The crypto:hmac function correctly produces the example hex value for the first step ($kDate), but I’m unable to use the result in the second step.
The instructions are very clear that binary results must be passed to each step. I suspect that this is not happening.
Can anyone see what might be wrong?


This is how I’m constructing the signature:


xquery version "3.1";
import module namespace crypto = "http://expath.org/ns/crypto";

declare variable $local:check := map {
    'kSecret'  := '41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559',
    'kDate'    := '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d',
    'kRegion'  := '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c',
    'kService' := 'f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa',
    'kSigning' := 'f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d'
};

declare function local:hmac($key, $data) {
    local:hmac($key, $data,'base64')
};
declare function local:hmac($key, $data,$encoding) {
    crypto:hmac($data, $key, 'HMAC-SHA-256',$encoding)
};

(: only the last calculation is required - the intermediate steps allow me to check the hex output :)
let $kDate := local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215", "hex")
let $kRegion := local:hmac(local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215"),"us-east-1","hex")
let $kService := local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam")
let $kSigning := local:hmac(local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam"),"aws4_request","hex")

return ($kRegion eq $local:check?kRegion)
(:return ($kDate eq $local:check?kDate):)


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Adam Retter
Just to add that I think that Nick Sincaglia has the same problem, he
posted here too if you check the archive. I don't think it is solved
yet.
I mentioned to Claudius, that I suspect it is a problem internally
with the crypto module using the wrong type of InputStream base64
encoded vs unencoded.

Cheers Adam.

On 21 June 2016 at 00:38, Alister Pillow <[hidden email]> wrote:

> Hi,
> Using eXist 2.2 with the EXPath Cryptographic Module v 0.3.4
>
> I’m trying to build a File Upload form for AWS S3. The docs start here:
> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html
>
> In particular, a Signature must be created using the HMAC function, which is provided by the Expath Crpyto module.
> This page provides examples of creating the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
> It provides test data showing the values produced for each step of the signature.
>
> The crypto:hmac function correctly produces the example hex value for the first step ($kDate), but I’m unable to use the result in the second step.
> The instructions are very clear that binary results must be passed to each step. I suspect that this is not happening.
> Can anyone see what might be wrong?
>
>
> This is how I’m constructing the signature:
>
>
> xquery version "3.1";
> import module namespace crypto = "http://expath.org/ns/crypto";
>
> declare variable $local:check := map {
>     'kSecret'  := '41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559',
>     'kDate'    := '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d',
>     'kRegion'  := '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c',
>     'kService' := 'f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa',
>     'kSigning' := 'f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d'
> };
>
> declare function local:hmac($key, $data) {
>     local:hmac($key, $data,'base64')
> };
> declare function local:hmac($key, $data,$encoding) {
>     crypto:hmac($data, $key, 'HMAC-SHA-256',$encoding)
> };
>
> (: only the last calculation is required - the intermediate steps allow me to check the hex output :)
> let $kDate := local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215", "hex")
> let $kRegion := local:hmac(local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215"),"us-east-1","hex")
> let $kService := local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam")
> let $kSigning := local:hmac(local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam"),"aws4_request","hex")
>
> return ($kRegion eq $local:check?kRegion)
> (:return ($kDate eq $local:check?kDate):)
>
>
> ------------------------------------------------------------------------------
> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
> present their vision of the future. This family event has something for
> everyone, including kids. Get more information and register today.
> http://sdm.link/attshape
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open



--
Adam Retter

eXist Developer
{ United Kingdom }
[hidden email]
irc://irc.freenode.net/existdb

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Alister Pillow-3
Hi Claudius,

I’m not sure if it’s relevant but the AWS example java code for this process very explicitly calls data.getBytes(“UTF-8”) in the mac.doFinal() call when creating the hash. The matching function in the crypto-java-lib Hmac.java does not do this.

Compare the use of the mac.doFinal function calls here
http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
mac.doFinal(data.getBytes("UTF8"));

and here
https://github.com/claudius108/crypto-java-lib/blob/master/src/main/java/ro/kuberam/libs/java/crypto/digest/Hmac.java
mac.doFinal(data);

I have tested the AWS example version and it produces the desired result. Clutching at straws of course.

Regards,
Alister

> On 21 Jun 2016, at 7:38 PM, Adam Retter <[hidden email]> wrote:
>
> Just to add that I think that Nick Sincaglia has the same problem, he
> posted here too if you check the archive. I don't think it is solved
> yet.
> I mentioned to Claudius, that I suspect it is a problem internally
> with the crypto module using the wrong type of InputStream base64
> encoded vs unencoded.
>
> Cheers Adam.
>
> On 21 June 2016 at 00:38, Alister Pillow <[hidden email]> wrote:
>> Hi,
>> Using eXist 2.2 with the EXPath Cryptographic Module v 0.3.4
>>
>> I’m trying to build a File Upload form for AWS S3. The docs start here:
>> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html
>>
>> In particular, a Signature must be created using the HMAC function, which is provided by the Expath Crpyto module.
>> This page provides examples of creating the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
>> It provides test data showing the values produced for each step of the signature.
>>
>> The crypto:hmac function correctly produces the example hex value for the first step ($kDate), but I’m unable to use the result in the second step.
>> The instructions are very clear that binary results must be passed to each step. I suspect that this is not happening.
>> Can anyone see what might be wrong?
>>
>>
>> This is how I’m constructing the signature:
>>
>>
>> xquery version "3.1";
>> import module namespace crypto = "http://expath.org/ns/crypto";
>>
>> declare variable $local:check := map {
>>    'kSecret'  := '41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559',
>>    'kDate'    := '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d',
>>    'kRegion'  := '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c',
>>    'kService' := 'f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa',
>>    'kSigning' := 'f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d'
>> };
>>
>> declare function local:hmac($key, $data) {
>>    local:hmac($key, $data,'base64')
>> };
>> declare function local:hmac($key, $data,$encoding) {
>>    crypto:hmac($data, $key, 'HMAC-SHA-256',$encoding)
>> };
>>
>> (: only the last calculation is required - the intermediate steps allow me to check the hex output :)
>> let $kDate := local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215", "hex")
>> let $kRegion := local:hmac(local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215"),"us-east-1","hex")
>> let $kService := local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam")
>> let $kSigning := local:hmac(local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam"),"aws4_request","hex")
>>
>> return ($kRegion eq $local:check?kRegion)
>> (:return ($kDate eq $local:check?kDate):)
>>
>>
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> http://sdm.link/attshape
>> _______________________________________________
>> Exist-open mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> [hidden email]
> irc://irc.freenode.net/existdb


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Claudius Teodorescu
Hi,


I will look again at this asap.

Anyway, at the java level, the needed string for AWS access is generated correctly.

I will test with your suggestion, too.


Thanks,
Claudius

On Tue, Jun 21, 2016 at 6:08 PM, Alister Pillow <[hidden email]> wrote:
Hi Claudius,

I’m not sure if it’s relevant but the AWS example java code for this process very explicitly calls data.getBytes(“UTF-8”) in the mac.doFinal() call when creating the hash. The matching function in the crypto-java-lib Hmac.java does not do this.

Compare the use of the mac.doFinal function calls here
http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
mac.doFinal(data.getBytes("UTF8"));

and here
https://github.com/claudius108/crypto-java-lib/blob/master/src/main/java/ro/kuberam/libs/java/crypto/digest/Hmac.java
mac.doFinal(data);

I have tested the AWS example version and it produces the desired result. Clutching at straws of course.

Regards,
Alister

> On 21 Jun 2016, at 7:38 PM, Adam Retter <[hidden email]> wrote:
>
> Just to add that I think that Nick Sincaglia has the same problem, he
> posted here too if you check the archive. I don't think it is solved
> yet.
> I mentioned to Claudius, that I suspect it is a problem internally
> with the crypto module using the wrong type of InputStream base64
> encoded vs unencoded.
>
> Cheers Adam.
>
> On 21 June 2016 at 00:38, Alister Pillow <[hidden email]> wrote:
>> Hi,
>> Using eXist 2.2 with the EXPath Cryptographic Module v 0.3.4
>>
>> I’m trying to build a File Upload form for AWS S3. The docs start here:
>> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html
>>
>> In particular, a Signature must be created using the HMAC function, which is provided by the Expath Crpyto module.
>> This page provides examples of creating the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
>> It provides test data showing the values produced for each step of the signature.
>>
>> The crypto:hmac function correctly produces the example hex value for the first step ($kDate), but I’m unable to use the result in the second step.
>> The instructions are very clear that binary results must be passed to each step. I suspect that this is not happening.
>> Can anyone see what might be wrong?
>>
>>
>> This is how I’m constructing the signature:
>>
>>
>> xquery version "3.1";
>> import module namespace crypto = "http://expath.org/ns/crypto";
>>
>> declare variable $local:check := map {
>>    'kSecret'  := '41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559',
>>    'kDate'    := '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d',
>>    'kRegion'  := '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c',
>>    'kService' := 'f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa',
>>    'kSigning' := 'f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d'
>> };
>>
>> declare function local:hmac($key, $data) {
>>    local:hmac($key, $data,'base64')
>> };
>> declare function local:hmac($key, $data,$encoding) {
>>    crypto:hmac($data, $key, 'HMAC-SHA-256',$encoding)
>> };
>>
>> (: only the last calculation is required - the intermediate steps allow me to check the hex output :)
>> let $kDate := local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215", "hex")
>> let $kRegion := local:hmac(local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215"),"us-east-1","hex")
>> let $kService := local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam")
>> let $kSigning := local:hmac(local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam"),"aws4_request","hex")
>>
>> return ($kRegion eq $local:check?kRegion)
>> (:return ($kDate eq $local:check?kDate):)
>>
>>
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> http://sdm.link/attshape
>> _______________________________________________
>> Exist-open mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> [hidden email]
> irc://irc.freenode.net/existdb




--

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Claudius Teodorescu
In reply to this post by Alister Pillow-3
Hi,

The signature of the hmac function is hmac(byte[] data, byte[] secretKey, String algorithm, String format), where the data's bytes are generated by using StandardCharsets.UTF_8 everywhere this function is called.

On the other hand, the eXist implementation for this function, when detects that the input data is base64binary, decodes the data and generates the HMAC result afterwards. See below:

case Type.BASE64_BINARY:
   processedData = Base64.getDecoder().decode(data.getStringValue().getBytes(StandardCharsets.UTF_8));
break;


Some time ago I have got an idea on how to test successive usage of base64binary data in eXist, usage similar to what you are asking for, but I forgot it. :)

I will try to remember it, and set a unit test, to see how it works.


Claudius
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

nsincaglia
In reply to this post by Claudius Teodorescu
Hi Claudius,

The way we were testing the crypto module was, we wrote the below Xquery script that recreates the steps that Amazon outlines in their documentation: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-other

The result at the end of the script allows you to compare the calculated result with the result that Amazon says is correct.

Does this help?


xquery version "3.0";

import module namespace http = "http://expath.org/ns/http-client";
import module namespace crypto = "http://expath.org/ns/crypto";
import module namespace xqjson = "http://xqilla.sourceforge.net/lib/xqjson";

let $region := "us-east-1"
let $service := 'iam'
let $date-YYYYMMDD := '20120215'
let $s3-secret-key := 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
let $kSecret := concat("AWS4", $s3-secret-key)
let $kDate := crypto:hmac($date-YYYYMMDD, $kSecret, "HmacSha256")
let $kRegion := crypto:hmac($region, $kDate, "HmacSha256")
let $kService := crypto:hmac($service, $kRegion, "HmacSha256")
let $kSigning := crypto:hmac("aws4_request", $kService, "HmacSha256")

let $log := util:log-system-out("----------")
let $log := util:log-system-out($kSecret)
let $log := util:log-system-out($kDate)
let $log := util:log-system-out($kRegion)
let $log := util:log-system-out($kService)
let $log := util:log-system-out($kSigning)

return
    <results>
        <kSecret>{$kSecret}</kSecret>
        <kSecret-hex>41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559</kSecret-hex>
        <kDate>{$kDate}</kDate>
        <kDate-hex>969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d</kDate-hex>
        <kRegion>{$kRegion}</kRegion>
        <kRegion-hex>69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c</kRegion-hex>
        <kService>{$kService}</kService>
        <kService-hex>f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa</kService-hex>
        <kSigning>{$kSigning}</kSigning>
        <kSigning-hex>f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d</kSigning-hex>
    </results>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

nsincaglia
In reply to this post by Claudius Teodorescu
Claudius / Alister / Adam:
Has a determination been made yet on the root of the problem with eXist-db and the crypto library? I am asking because, there are now certain AWS regions which only support version 4 of their API security (http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html). Is it still unclear if the problem is with the crypto library or eXist-db? Or have we determined that yet?

Nick
 
On Jun 22, 2016, at 4:30 AM, Claudius Teodorescu <[hidden email]> wrote:

Hi,


I will look again at this asap.

Anyway, at the java level, the needed string for AWS access is generated correctly.

I will test with your suggestion, too.


Thanks,
Claudius

On Tue, Jun 21, 2016 at 6:08 PM, Alister Pillow <[hidden email]> wrote:
Hi Claudius,

I’m not sure if it’s relevant but the AWS example java code for this process very explicitly calls data.getBytes(“UTF-8”) in the mac.doFinal() call when creating the hash. The matching function in the crypto-java-lib Hmac.java does not do this.

Compare the use of the mac.doFinal function calls here
http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
mac.doFinal(data.getBytes("UTF8"));

and here
https://github.com/claudius108/crypto-java-lib/blob/master/src/main/java/ro/kuberam/libs/java/crypto/digest/Hmac.java
mac.doFinal(data);

I have tested the AWS example version and it produces the desired result. Clutching at straws of course.

Regards,
Alister

> On 21 Jun 2016, at 7:38 PM, Adam Retter <[hidden email]> wrote:
>
> Just to add that I think that Nick Sincaglia has the same problem, he
> posted here too if you check the archive. I don't think it is solved
> yet.
> I mentioned to Claudius, that I suspect it is a problem internally
> with the crypto module using the wrong type of InputStream base64
> encoded vs unencoded.
>
> Cheers Adam.
>
> On 21 June 2016 at 00:38, Alister Pillow <[hidden email]> wrote:
>> Hi,
>> Using eXist 2.2 with the EXPath Cryptographic Module v 0.3.4
>>
>> I’m trying to build a File Upload form for AWS S3. The docs start here:
>> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html
>>
>> In particular, a Signature must be created using the HMAC function, which is provided by the Expath Crpyto module.
>> This page provides examples of creating the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
>> It provides test data showing the values produced for each step of the signature.
>>
>> The crypto:hmac function correctly produces the example hex value for the first step ($kDate), but I’m unable to use the result in the second step.
>> The instructions are very clear that binary results must be passed to each step. I suspect that this is not happening.
>> Can anyone see what might be wrong?
>>
>>
>> This is how I’m constructing the signature:
>>
>>
>> xquery version "3.1";
>> import module namespace crypto = "http://expath.org/ns/crypto";
>>
>> declare variable $local:check := map {
>>    'kSecret'  := '41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559',
>>    'kDate'    := '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d',
>>    'kRegion'  := '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c',
>>    'kService' := 'f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa',
>>    'kSigning' := 'f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d'
>> };
>>
>> declare function local:hmac($key, $data) {
>>    local:hmac($key, $data,'base64')
>> };
>> declare function local:hmac($key, $data,$encoding) {
>>    crypto:hmac($data, $key, 'HMAC-SHA-256',$encoding)
>> };
>>
>> (: only the last calculation is required - the intermediate steps allow me to check the hex output :)
>> let $kDate := local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215", "hex")
>> let $kRegion := local:hmac(local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215"),"us-east-1","hex")
>> let $kService := local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam")
>> let $kSigning := local:hmac(local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam"),"aws4_request","hex")
>>
>> return ($kRegion eq $local:check?kRegion)
>> (:return ($kDate eq $local:check?kDate):)
>>
>>
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> http://sdm.link/attshape
>> _______________________________________________
>> Exist-open mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> [hidden email]
> irc://irc.freenode.net/existdb




--



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Alister Pillow-3
Hi Nick,

I think there is a problem with the way the result is returned from the Java module to eXist. 

I wrote a Java module to create the signature for an Upload Using Post

The module provides this function
s3m:shmac($iso-short-now, $secret-key, $region, $encoded-policy)

Happy to share with anyone - but the code is unpolished.

Regards,
Alister


TLDR;

My use-case:
Clients are treating the db like their own “cloud” server, uploading various files associated (mostly PDFs) with the “jobs” they are processing. This module allows me to create an upload-to-S3 solution and store a Table of Contents in each collection where this happens. The items from the TOC are added to the collection contents when a listing is produced (see attached PNG - shows a PDF and PNG which have been uploaded into S3). 

On 9 Nov. 2016, at 3:39 pm, Nick Sincaglia <[hidden email]> wrote:

Claudius / Alister / Adam:
Has a determination been made yet on the root of the problem with eXist-db and the crypto library? I am asking because, there are now certain AWS regions which only support version 4 of their API security (http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html). Is it still unclear if the problem is with the crypto library or eXist-db? Or have we determined that yet?

Nick
 
On Jun 22, 2016, at 4:30 AM, Claudius Teodorescu <[hidden email]> wrote:

Hi,


I will look again at this asap.

Anyway, at the java level, the needed string for AWS access is generated correctly.

I will test with your suggestion, too.


Thanks,
Claudius

On Tue, Jun 21, 2016 at 6:08 PM, Alister Pillow <[hidden email]> wrote:
Hi Claudius,

I’m not sure if it’s relevant but the AWS example java code for this process very explicitly calls data.getBytes(“UTF-8”) in the mac.doFinal() call when creating the hash. The matching function in the crypto-java-lib Hmac.java does not do this.

Compare the use of the mac.doFinal function calls here
http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
mac.doFinal(data.getBytes("UTF8"));

and here
https://github.com/claudius108/crypto-java-lib/blob/master/src/main/java/ro/kuberam/libs/java/crypto/digest/Hmac.java
mac.doFinal(data);

I have tested the AWS example version and it produces the desired result. Clutching at straws of course.

Regards,
Alister

> On 21 Jun 2016, at 7:38 PM, Adam Retter <[hidden email]> wrote:
>
> Just to add that I think that Nick Sincaglia has the same problem, he
> posted here too if you check the archive. I don't think it is solved
> yet.
> I mentioned to Claudius, that I suspect it is a problem internally
> with the crypto module using the wrong type of InputStream base64
> encoded vs unencoded.
>
> Cheers Adam.
>
> On 21 June 2016 at 00:38, Alister Pillow <[hidden email]> wrote:
>> Hi,
>> Using eXist 2.2 with the EXPath Cryptographic Module v 0.3.4
>>
>> I’m trying to build a File Upload form for AWS S3. The docs start here:
>> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html
>>
>> In particular, a Signature must be created using the HMAC function, which is provided by the Expath Crpyto module.
>> This page provides examples of creating the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
>> It provides test data showing the values produced for each step of the signature.
>>
>> The crypto:hmac function correctly produces the example hex value for the first step ($kDate), but I’m unable to use the result in the second step.
>> The instructions are very clear that binary results must be passed to each step. I suspect that this is not happening.
>> Can anyone see what might be wrong?
>>
>>
>> This is how I’m constructing the signature:
>>
>>
>> xquery version "3.1";
>> import module namespace crypto = "http://expath.org/ns/crypto";
>>
>> declare variable $local:check := map {
>>    'kSecret'  := '41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559',
>>    'kDate'    := '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d',
>>    'kRegion'  := '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c',
>>    'kService' := 'f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa',
>>    'kSigning' := 'f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d'
>> };
>>
>> declare function local:hmac($key, $data) {
>>    local:hmac($key, $data,'base64')
>> };
>> declare function local:hmac($key, $data,$encoding) {
>>    crypto:hmac($data, $key, 'HMAC-SHA-256',$encoding)
>> };
>>
>> (: only the last calculation is required - the intermediate steps allow me to check the hex output :)
>> let $kDate := local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215", "hex")
>> let $kRegion := local:hmac(local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215"),"us-east-1","hex")
>> let $kService := local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam")
>> let $kSigning := local:hmac(local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam"),"aws4_request","hex")
>>
>> return ($kRegion eq $local:check?kRegion)
>> (:return ($kDate eq $local:check?kDate):)
>>
>>
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> http://sdm.link/attshape
>> _______________________________________________
>> Exist-open mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> [hidden email]
> irc://irc.freenode.net/existdb




--




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Claudius Teodorescu
Hi,

As I said some time ago, I made a small unit test:
xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto";

let $input := "20120215"
let $key := "AWS4wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
let $date := crypto:hmac($input, $key, 'HMAC-SHA-256', "base64")

return ($date instance of xs:base64Binary)

This unit test returns false(), even the crypto:hmac() function generates its result by using the following java code:
result = BinaryValueFromInputStream.getInstance(context, new Base64BinaryValueType(),
new ByteArrayInputStream(hmacResult.getBytes(StandardCharsets.UTF_8)));

On the other hand, the script
(util:string-to-binary("aaa") instance of xs:base64Binary)
returns true(), while the result in java is generated with BinaryValueFromInputStream.getInstance(context, new Base64BinaryValueType(), new ByteArrayInputStream(str.getBytes(encoding)));


So, no differences. Maybe the eXist masters can have an insight on this ...


Claudius
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Claudius Teodorescu
In reply to this post by Alister Pillow-3
Hi,

I amswered on the eXist mailing list.


Claudius

On Wed, Nov 9, 2016 at 10:39 AM, Alister Pillow <[hidden email]> wrote:
Hi Nick,

I think there is a problem with the way the result is returned from the Java module to eXist. 

I wrote a Java module to create the signature for an Upload Using Post

The module provides this function
s3m:shmac($iso-short-now, $secret-key, $region, $encoded-policy)

Happy to share with anyone - but the code is unpolished.

Regards,
Alister


TLDR;

My use-case:
Clients are treating the db like their own “cloud” server, uploading various files associated (mostly PDFs) with the “jobs” they are processing. This module allows me to create an upload-to-S3 solution and store a Table of Contents in each collection where this happens. The items from the TOC are added to the collection contents when a listing is produced (see attached PNG - shows a PDF and PNG which have been uploaded into S3). 

On 9 Nov. 2016, at 3:39 pm, Nick Sincaglia <[hidden email]> wrote:

Claudius / Alister / Adam:
Has a determination been made yet on the root of the problem with eXist-db and the crypto library? I am asking because, there are now certain AWS regions which only support version 4 of their API security (http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html). Is it still unclear if the problem is with the crypto library or eXist-db? Or have we determined that yet?

Nick
 
On Jun 22, 2016, at 4:30 AM, Claudius Teodorescu <[hidden email]> wrote:

Hi,


I will look again at this asap.

Anyway, at the java level, the needed string for AWS access is generated correctly.

I will test with your suggestion, too.


Thanks,
Claudius

On Tue, Jun 21, 2016 at 6:08 PM, Alister Pillow <[hidden email]> wrote:
Hi Claudius,

I’m not sure if it’s relevant but the AWS example java code for this process very explicitly calls data.getBytes(“UTF-8”) in the mac.doFinal() call when creating the hash. The matching function in the crypto-java-lib Hmac.java does not do this.

Compare the use of the mac.doFinal function calls here
http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
mac.doFinal(data.getBytes("UTF8"));

and here
https://github.com/claudius108/crypto-java-lib/blob/master/src/main/java/ro/kuberam/libs/java/crypto/digest/Hmac.java
mac.doFinal(data);

I have tested the AWS example version and it produces the desired result. Clutching at straws of course.

Regards,
Alister

> On 21 Jun 2016, at 7:38 PM, Adam Retter <[hidden email]> wrote:
>
> Just to add that I think that Nick Sincaglia has the same problem, he
> posted here too if you check the archive. I don't think it is solved
> yet.
> I mentioned to Claudius, that I suspect it is a problem internally
> with the crypto module using the wrong type of InputStream base64
> encoded vs unencoded.
>
> Cheers Adam.
>
> On 21 June 2016 at 00:38, Alister Pillow <[hidden email]> wrote:
>> Hi,
>> Using eXist 2.2 with the EXPath Cryptographic Module v 0.3.4
>>
>> I’m trying to build a File Upload form for AWS S3. The docs start here:
>> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTForms.html
>>
>> In particular, a Signature must be created using the HMAC function, which is provided by the Expath Crpyto module.
>> This page provides examples of creating the signature: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
>> It provides test data showing the values produced for each step of the signature.
>>
>> The crypto:hmac function correctly produces the example hex value for the first step ($kDate), but I’m unable to use the result in the second step.
>> The instructions are very clear that binary results must be passed to each step. I suspect that this is not happening.
>> Can anyone see what might be wrong?
>>
>>
>> This is how I’m constructing the signature:
>>
>>
>> xquery version "3.1";
>> import module namespace crypto = "http://expath.org/ns/crypto";
>>
>> declare variable $local:check := map {
>>    'kSecret'  := '41575334774a616c725855746e46454d492f4b374d44454e472b62507852666943594558414d504c454b4559',
>>    'kDate'    := '969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d',
>>    'kRegion'  := '69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c',
>>    'kService' := 'f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa',
>>    'kSigning' := 'f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d'
>> };
>>
>> declare function local:hmac($key, $data) {
>>    local:hmac($key, $data,'base64')
>> };
>> declare function local:hmac($key, $data,$encoding) {
>>    crypto:hmac($data, $key, 'HMAC-SHA-256',$encoding)
>> };
>>
>> (: only the last calculation is required - the intermediate steps allow me to check the hex output :)
>> let $kDate := local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215", "hex")
>> let $kRegion := local:hmac(local:hmac("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20120215"),"us-east-1","hex")
>> let $kService := local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam")
>> let $kSigning := local:hmac(local:hmac(local:hmac(local:hmac(util:string-to-binary("AWS4" || "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY","UTF-8"), "20120215" ),"us-east-1"),"iam"),"aws4_request","hex")
>>
>> return ($kRegion eq $local:check?kRegion)
>> (:return ($kDate eq $local:check?kDate):)
>>
>>
>> ------------------------------------------------------------------------------
>> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
>> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
>> present their vision of the future. This family event has something for
>> everyone, including kids. Get more information and register today.
>> http://sdm.link/attshape
>> _______________________________________________
>> Exist-open mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
>
> --
> Adam Retter
>
> eXist Developer
> { United Kingdom }
> [hidden email]
> irc://irc.freenode.net/existdb




--






--

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

nsincaglia
In reply to this post by Claudius Teodorescu
All:

I think I know what the problem is here. I think the problem is not being caused by a bug in either the Crypto module nor its interaction with eXist-db. I think the issue is being cause by the fact that the Crypto module's secret key input variable must be encoded as a UTF-8 string. The reason this is a problem is because the conversion from base64 string to UTF-8 strings, in certain circumstances, results in corrupted a corrupted secret key variable.

I found this article which explains the problem in detail.
http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/

In order for the Crypto Module to be able to support Amazon S3 V4 signatures, a new function call needs to be added to the Crypto Module, which will accept base64 string encoded secret keys as an input, without the need to first convert it to UTF-8. Because it is in this conversion, which the secret key is getting corrupted.

Let me know if this does not make sense. I have software examples I can provide, but the above mentioned article explains the issue pretty well.

Nick
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Alister Pillow-3
Thanks Nick, appreciate your follow up.


> On 13 Dec. 2016, at 4:54 am, nsincaglia <[hidden email]> wrote:
>
> All:
>
> I think I know what the problem is here. I think the problem is not being
> caused by a bug in either the Crypto module nor its interaction with
> eXist-db. I think the issue is being cause by the fact that the Crypto
> module's secret key input variable must be encoded as a UTF-8 string. The
> reason this is a problem is because the conversion from base64 string to
> UTF-8 strings, in certain circumstances, results in corrupted a corrupted
> secret key variable.
>
> I found this article which explains the problem in detail.
> http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/
>
> In order for the Crypto Module to be able to support Amazon S3 V4
> signatures, a new function call needs to be added to the Crypto Module,
> which will accept base64 string encoded secret keys as an input, without the
> need to first convert it to UTF-8. Because it is in this conversion, which
> the secret key is getting corrupted.
>
> Let me know if this does not make sense. I have software examples I can
> provide, but the above mentioned article explains the issue pretty well.
>
> Nick
>
>
>
>
> --
> View this message in context: http://exist.2174344.n4.nabble.com/hmac-signature-for-AWS-S3-tp4670354p4671042.html
> Sent from the exist-open mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Exist-open mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/exist-open


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

nsincaglia
All,

I am very happy to report that a new Expath Crypto module (version 0.3.5) was recently posted by Claudius, which supports Amazon’s S3 V4 REST web service API’s algorithm for creating signatures. Yeah! 

But wait there is more….. :-)

While we were working with Claudius on the Amazon S3 V4 testing, we also had a need to connect to the Azure Cloud Storage Service REST API as well. From reading through this API documentation, we discovered that the Azure Storage Service API added a new signature requirement that was different from either the Amazon’s V2 and V4 API signatures. So, in addition, this new Expath Crypto library can also support the Azure Cloud Storage REST API signature. Double Yeah!

With support for these 3 different signature algorithms, I believe this new version of the Expath Crypto library will be able to support a number of encryption scenarios. Let me try to explain the differences between the individual signature algorithms and how the new capabilities within the Expath Crypto module is now able to support these different requirements. 

In the case of the Amazon S3 V2 API signature algorithm, the secret key used to generate the signature is a UTF-8 string and the output of the hmac() function is a base64 encoded string. The algorithm only requires a single call to the hmac() function. The below snippet of code illustrates how to use the Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 2

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto"; 

let $secret-key := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
let $nl := '&#10;' (: the newline character :)
let $string-to-sign := concat("GET", $nl, $nl, $nl, "Wed, 28 Mar 2007 01:49:49 +0000", $nl, "/dictionary/fran%C3%A7ais/pr%c3%a9f%c3%a8re")
let $signature := crypto:hmac($string-to-sign, util:string-to-binary($secret-key), "HMAC-SHA-1", "base64")

return
    <results>
        <calculated>{$signature}</calculated>
        <expected-value>DNEZGsoieTZ92F3bUfSPQcbGmlM=</expected-value>
    </results>

In the case of the Amazon S3 V4 API signature algorithm, the secret key used to create the signature is again, a UTF-8 string. However, this signature calculation algorithm requires 4 calls to the hmac() function, where the output of the each hmac() function is the input of the next hmac() function call. In this case the first call to the hmac() function receives a UTF-8 secret key as an input, but every subsequent call to the hmac() function must accept a base64 encoded string as an input because that is what the hmac() function returns. In the previous version of the Expath crypto module, this scenario was not supported. In the new version it is.

The below snippet of code illustrates how to use the new Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 4

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto";

let $region := 'us-east-1'
let $service := 'iam'
let $date-YYYYMMDD := '20120215'
let $s3-secret-key := 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
let $kDate := crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256')
let $kRegion := crypto:hmac($region, $kDate, 'HMAC-SHA-256')
let $kService := crypto:hmac($service, $kRegion, 'HMAC-SHA-256')
let $kSigning := crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')

return 
    <results>
        <kDate-calculated>{crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256','hex')}</kDate-calculated>
        <kDate-expected>969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d</kDate-expected>
        <kRegion-calculated>{crypto:hmac($region, $kDate, 'HMAC-SHA-256','hex')}</kRegion-calculated>
        <kRegion-expected>69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c</kRegion-expected>
        <kService-calculated>{crypto:hmac($service, $kRegion, 'HMAC-SHA-256','hex')}</kService-calculated>
        <kService-expected>f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa</kService-expected>
        <kSigning-calculated>{crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')}</kSigning-calculated>
        <kSigning-expected>f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d</kSigning-expected>
    </results>


In the case of the Azure Storage Service API signature algorithm, the secret key used to create the signature is a base64 encoded string. This base64 encoded secret key string must converted directly into a byte array inside the hmac() function before the encryption algorithm can be applied. This was also something the previous version of the Expath Crypto module was not able to support but which the new version can. In order to support this, Claudius needed to create the ExPath Datatypes Module which is needed to work in conjunction with the Expath Crypto Module.  Below is a snippet of code to illustrate how the new Expath Crypto module and Expath Datatypes Module must be used together to generate the correct signature for the Azure Storage Service API.

Microsoft Azure Storage Service API

xquery version "3.0";
import module namespace crypto = "http://expath.org/ns/crypto"; 
import module namespace datatypes = "http://expath.org/ns/datatypes";

let $storage-key := "RzpkpwCtKeD25zjx+19YD0kWT0vwOIsA7yk6hnJco0jnMBM7J8tTrppqOJZ7wB0KSO6B2oKtkOlzUN/LRkpY2Q=="
let $storage-account-name := "d11portalvhds2g6lndzyd0ej"
let $date := "Wed, 30 Nov 2016 22:18:58 GMT"
let $nl := '&#10;'
let $string-to-sign := concat('GET', $nl,
                            $nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl, 
                            'x-ms-date:', $date, $nl,
                            'x-ms-version:2014-02-14', $nl,
                            '/', $storage-account-name, '/', $nl,
                            'comp:list') 
                            
let $hmac := crypto:hmac($string-to-sign, datatypes:base64Binary-to-byte($storage-key), "HMAC-SHA-256", "base64")
return
    <results>
        <calculation>{$hmac}</calculation>
        <expected-answer>pSSVTYF4pErpH2DOi8Wzvj0B1nru/sM4pLJHimoMZMg=</expected-answer>
    </results>

We have tested this new Expath Crypto module thoroughly for several weeks now and I am happy to report that it is working very well for us!

A big "Thank You" to Claudius for the work he put into this and working with us to test all of these different scenarios. Encryption can be a very difficult and confusing area to work in. 

And while I am at it, a big “Thank You” to the core developers of eXist-db as well. eXist-db is really an amazing project with a strong supportive community. I am looking forward to the official 3.0 release!

Nick
  
On Dec 13, 2016, at 6:17 AM, Alister Pillow <[hidden email]> wrote:

Thanks Nick, appreciate your follow up.


On 13 Dec. 2016, at 4:54 am, nsincaglia <[hidden email]> wrote:

All:

I think I know what the problem is here. I think the problem is not being
caused by a bug in either the Crypto module nor its interaction with
eXist-db. I think the issue is being cause by the fact that the Crypto
module's secret key input variable must be encoded as a UTF-8 string. The
reason this is a problem is because the conversion from base64 string to
UTF-8 strings, in certain circumstances, results in corrupted a corrupted
secret key variable.

I found this article which explains the problem in detail.
http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/

In order for the Crypto Module to be able to support Amazon S3 V4
signatures, a new function call needs to be added to the Crypto Module,
which will accept base64 string encoded secret keys as an input, without the
need to first convert it to UTF-8. Because it is in this conversion, which
the secret key is getting corrupted.

Let me know if this does not make sense. I have software examples I can
provide, but the above mentioned article explains the issue pretty well.

Nick




--
View this message in context: http://exist.2174344.n4.nabble.com/hmac-signature-for-AWS-S3-tp4670354p4671042.html
Sent from the exist-open mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Alister Pillow-3
Hi Nick,

Excellent! Would this be for eXist-db v3+ only?

How are you interacting with S3? Are you using EXPath HTTP? 

I’ve recently written a module for creating the ASW S4 V4 signature which works with eXist v2.2x. It can create signed HTML Forms (for upload to S3 from a browser) and URLs (for download from S3), AND until just now (while trying to describe the problem) I was unable to send directly from eXist using the EXPath client. But now - thanks to your unwitting help :) I have found the magic incantation that makes the send-request work with AWS S3. 

tldr;

My users have been filling the database with binary files and the weekly backup weighs 12GB zipped. I need to offload their files into their own AWS bucket. Now I can do this. 

Happy to share for those wanting this in 2.2 (assuming Claudius’ work is v3 only).

Regards,
Alister



On 29 Jan 2017, at 4:59 am, Nick Sincaglia <[hidden email]> wrote:

All,

I am very happy to report that a new Expath Crypto module (version 0.3.5) was recently posted by Claudius, which supports Amazon’s S3 V4 REST web service API’s algorithm for creating signatures. Yeah! 

But wait there is more….. :-)

While we were working with Claudius on the Amazon S3 V4 testing, we also had a need to connect to the Azure Cloud Storage Service REST API as well. From reading through this API documentation, we discovered that the Azure Storage Service API added a new signature requirement that was different from either the Amazon’s V2 and V4 API signatures. So, in addition, this new Expath Crypto library can also support the Azure Cloud Storage REST API signature. Double Yeah!

With support for these 3 different signature algorithms, I believe this new version of the Expath Crypto library will be able to support a number of encryption scenarios. Let me try to explain the differences between the individual signature algorithms and how the new capabilities within the Expath Crypto module is now able to support these different requirements. 

In the case of the Amazon S3 V2 API signature algorithm, the secret key used to generate the signature is a UTF-8 string and the output of the hmac() function is a base64 encoded string. The algorithm only requires a single call to the hmac() function. The below snippet of code illustrates how to use the Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 2

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto"; 

let $secret-key := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
let $nl := '&#10;' (: the newline character :)
let $string-to-sign := concat("GET", $nl, $nl, $nl, "Wed, 28 Mar 2007 01:49:49 +0000", $nl, "/dictionary/fran%C3%A7ais/pr%c3%a9f%c3%a8re")
let $signature := crypto:hmac($string-to-sign, util:string-to-binary($secret-key), "HMAC-SHA-1", "base64")

return
    <results>
        <calculated>{$signature}</calculated>
        <expected-value>DNEZGsoieTZ92F3bUfSPQcbGmlM=</expected-value>
    </results>

In the case of the Amazon S3 V4 API signature algorithm, the secret key used to create the signature is again, a UTF-8 string. However, this signature calculation algorithm requires 4 calls to the hmac() function, where the output of the each hmac() function is the input of the next hmac() function call. In this case the first call to the hmac() function receives a UTF-8 secret key as an input, but every subsequent call to the hmac() function must accept a base64 encoded string as an input because that is what the hmac() function returns. In the previous version of the Expath crypto module, this scenario was not supported. In the new version it is.

The below snippet of code illustrates how to use the new Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 4

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto";

let $region := 'us-east-1'
let $service := 'iam'
let $date-YYYYMMDD := '20120215'
let $s3-secret-key := 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
let $kDate := crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256')
let $kRegion := crypto:hmac($region, $kDate, 'HMAC-SHA-256')
let $kService := crypto:hmac($service, $kRegion, 'HMAC-SHA-256')
let $kSigning := crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')

return 
    <results>
        <kDate-calculated>{crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256','hex')}</kDate-calculated>
        <kDate-expected>969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d</kDate-expected>
        <kRegion-calculated>{crypto:hmac($region, $kDate, 'HMAC-SHA-256','hex')}</kRegion-calculated>
        <kRegion-expected>69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c</kRegion-expected>
        <kService-calculated>{crypto:hmac($service, $kRegion, 'HMAC-SHA-256','hex')}</kService-calculated>
        <kService-expected>f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa</kService-expected>
        <kSigning-calculated>{crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')}</kSigning-calculated>
        <kSigning-expected>f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d</kSigning-expected>
    </results>


In the case of the Azure Storage Service API signature algorithm, the secret key used to create the signature is a base64 encoded string. This base64 encoded secret key string must converted directly into a byte array inside the hmac() function before the encryption algorithm can be applied. This was also something the previous version of the Expath Crypto module was not able to support but which the new version can. In order to support this, Claudius needed to create the ExPath Datatypes Module which is needed to work in conjunction with the Expath Crypto Module.  Below is a snippet of code to illustrate how the new Expath Crypto module and Expath Datatypes Module must be used together to generate the correct signature for the Azure Storage Service API.

Microsoft Azure Storage Service API

xquery version "3.0";
import module namespace crypto = "http://expath.org/ns/crypto"; 
import module namespace datatypes = "http://expath.org/ns/datatypes";

let $storage-key := "RzpkpwCtKeD25zjx+19YD0kWT0vwOIsA7yk6hnJco0jnMBM7J8tTrppqOJZ7wB0KSO6B2oKtkOlzUN/LRkpY2Q=="
let $storage-account-name := "d11portalvhds2g6lndzyd0ej"
let $date := "Wed, 30 Nov 2016 22:18:58 GMT"
let $nl := '&#10;'
let $string-to-sign := concat('GET', $nl,
                            $nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl, 
                            'x-ms-date:', $date, $nl,
                            'x-ms-version:2014-02-14', $nl,
                            '/', $storage-account-name, '/', $nl,
                            'comp:list') 
                            
let $hmac := crypto:hmac($string-to-sign, datatypes:base64Binary-to-byte($storage-key), "HMAC-SHA-256", "base64")
return
    <results>
        <calculation>{$hmac}</calculation>
        <expected-answer>pSSVTYF4pErpH2DOi8Wzvj0B1nru/sM4pLJHimoMZMg=</expected-answer>
    </results>

We have tested this new Expath Crypto module thoroughly for several weeks now and I am happy to report that it is working very well for us!

A big "Thank You" to Claudius for the work he put into this and working with us to test all of these different scenarios. Encryption can be a very difficult and confusing area to work in. 

And while I am at it, a big “Thank You” to the core developers of eXist-db as well. eXist-db is really an amazing project with a strong supportive community. I am looking forward to the official 3.0 release!

Nick
  
On Dec 13, 2016, at 6:17 AM, Alister Pillow <[hidden email]> wrote:

Thanks Nick, appreciate your follow up.


On 13 Dec. 2016, at 4:54 am, nsincaglia <[hidden email]> wrote:

All:

I think I know what the problem is here. I think the problem is not being
caused by a bug in either the Crypto module nor its interaction with
eXist-db. I think the issue is being cause by the fact that the Crypto
module's secret key input variable must be encoded as a UTF-8 string. The
reason this is a problem is because the conversion from base64 string to
UTF-8 strings, in certain circumstances, results in corrupted a corrupted
secret key variable.

I found this article which explains the problem in detail.
http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/

In order for the Crypto Module to be able to support Amazon S3 V4
signatures, a new function call needs to be added to the Crypto Module,
which will accept base64 string encoded secret keys as an input, without the
need to first convert it to UTF-8. Because it is in this conversion, which
the secret key is getting corrupted.

Let me know if this does not make sense. I have software examples I can
provide, but the above mentioned article explains the issue pretty well.

Nick




--
View this message in context: http://exist.2174344.n4.nabble.com/hmac-signature-for-AWS-S3-tp4670354p4671042.html
Sent from the exist-open mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Claudius Teodorescu
Dear Alister,

If I remember well, the package for AbstractInternalModule, which is extended by the crypto module, is different for eXist 2, so I decided at certain moment to support only eXist 3+.


Claudius
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

nsincaglia
In reply to this post by Alister Pillow-3
Alister,

The new Crypto module is for eXist-db v3.0 due to the fact that the structure of the XAR files has changed between the 2.2 and 3.0 versions. 

In regards to interacting with these various web service APIs using the EXpath HTTP module. I know that when we were using eXist-db 2.2, there were some issues with that module which only allowed us to PUT/POST XML files. We were unable to transfer binary files using this module. We sought out some help from Adam Retter to try to determine what was wrong. He determined that it was an issue with that module Expath HTTP module and coordinated with Florent Georges the original author of that module and some others to make the needed changes to get uploads of binary files using regular and multi-part requests. Those improvements are now incorporated into eXist-db v3.0.

Since you mention that you are working with binary data, I feel compelled to also mention that Adam also identified and fixed a bug in eXist-db where if you were working with binary files, eXist-db would not release the memory those binary files require, even after you were done performing your operation on it. This is another reason to switch to version 3.0. I know experienced this issue because we have processes which we run which grab a set of binary files, one at a time, in a loop and if the cumulative size of those binary files exceeded the RAM we allocated to our JVM, the system would crash. It is my understanding that this issue has been eliminated in the new version.

Nick
 
On Jan 29, 2017, at 5:56 PM, Alister Pillow <[hidden email]> wrote:

Hi Nick,

Excellent! Would this be for eXist-db v3+ only?

How are you interacting with S3? Are you using EXPath HTTP? 

I’ve recently written a module for creating the ASW S4 V4 signature which works with eXist v2.2x. It can create signed HTML Forms (for upload to S3 from a browser) and URLs (for download from S3), AND until just now (while trying to describe the problem) I was unable to send directly from eXist using the EXPath client. But now - thanks to your unwitting help :) I have found the magic incantation that makes the send-request work with AWS S3. 

tldr;

My users have been filling the database with binary files and the weekly backup weighs 12GB zipped. I need to offload their files into their own AWS bucket. Now I can do this. 

Happy to share for those wanting this in 2.2 (assuming Claudius’ work is v3 only).

Regards,
Alister



On 29 Jan 2017, at 4:59 am, Nick Sincaglia <[hidden email]> wrote:

All,

I am very happy to report that a new Expath Crypto module (version 0.3.5) was recently posted by Claudius, which supports Amazon’s S3 V4 REST web service API’s algorithm for creating signatures. Yeah! 

But wait there is more….. :-)

While we were working with Claudius on the Amazon S3 V4 testing, we also had a need to connect to the Azure Cloud Storage Service REST API as well. From reading through this API documentation, we discovered that the Azure Storage Service API added a new signature requirement that was different from either the Amazon’s V2 and V4 API signatures. So, in addition, this new Expath Crypto library can also support the Azure Cloud Storage REST API signature. Double Yeah!

With support for these 3 different signature algorithms, I believe this new version of the Expath Crypto library will be able to support a number of encryption scenarios. Let me try to explain the differences between the individual signature algorithms and how the new capabilities within the Expath Crypto module is now able to support these different requirements. 

In the case of the Amazon S3 V2 API signature algorithm, the secret key used to generate the signature is a UTF-8 string and the output of the hmac() function is a base64 encoded string. The algorithm only requires a single call to the hmac() function. The below snippet of code illustrates how to use the Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 2

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto"; 

let $secret-key := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
let $nl := '&#10;' (: the newline character :)
let $string-to-sign := concat("GET", $nl, $nl, $nl, "Wed, 28 Mar 2007 01:49:49 +0000", $nl, "/dictionary/fran%C3%A7ais/pr%c3%a9f%c3%a8re")
let $signature := crypto:hmac($string-to-sign, util:string-to-binary($secret-key), "HMAC-SHA-1", "base64")

return
    <results>
        <calculated>{$signature}</calculated>
        <expected-value>DNEZGsoieTZ92F3bUfSPQcbGmlM=</expected-value>
    </results>

In the case of the Amazon S3 V4 API signature algorithm, the secret key used to create the signature is again, a UTF-8 string. However, this signature calculation algorithm requires 4 calls to the hmac() function, where the output of the each hmac() function is the input of the next hmac() function call. In this case the first call to the hmac() function receives a UTF-8 secret key as an input, but every subsequent call to the hmac() function must accept a base64 encoded string as an input because that is what the hmac() function returns. In the previous version of the Expath crypto module, this scenario was not supported. In the new version it is.

The below snippet of code illustrates how to use the new Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 4

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto";

let $region := 'us-east-1'
let $service := 'iam'
let $date-YYYYMMDD := '20120215'
let $s3-secret-key := 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
let $kDate := crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256')
let $kRegion := crypto:hmac($region, $kDate, 'HMAC-SHA-256')
let $kService := crypto:hmac($service, $kRegion, 'HMAC-SHA-256')
let $kSigning := crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')

return 
    <results>
        <kDate-calculated>{crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256','hex')}</kDate-calculated>
        <kDate-expected>969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d</kDate-expected>
        <kRegion-calculated>{crypto:hmac($region, $kDate, 'HMAC-SHA-256','hex')}</kRegion-calculated>
        <kRegion-expected>69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c</kRegion-expected>
        <kService-calculated>{crypto:hmac($service, $kRegion, 'HMAC-SHA-256','hex')}</kService-calculated>
        <kService-expected>f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa</kService-expected>
        <kSigning-calculated>{crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')}</kSigning-calculated>
        <kSigning-expected>f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d</kSigning-expected>
    </results>


In the case of the Azure Storage Service API signature algorithm, the secret key used to create the signature is a base64 encoded string. This base64 encoded secret key string must converted directly into a byte array inside the hmac() function before the encryption algorithm can be applied. This was also something the previous version of the Expath Crypto module was not able to support but which the new version can. In order to support this, Claudius needed to create the ExPath Datatypes Module which is needed to work in conjunction with the Expath Crypto Module.  Below is a snippet of code to illustrate how the new Expath Crypto module and Expath Datatypes Module must be used together to generate the correct signature for the Azure Storage Service API.

Microsoft Azure Storage Service API

xquery version "3.0";
import module namespace crypto = "http://expath.org/ns/crypto"; 
import module namespace datatypes = "http://expath.org/ns/datatypes";

let $storage-key := "RzpkpwCtKeD25zjx+19YD0kWT0vwOIsA7yk6hnJco0jnMBM7J8tTrppqOJZ7wB0KSO6B2oKtkOlzUN/LRkpY2Q=="
let $storage-account-name := "d11portalvhds2g6lndzyd0ej"
let $date := "Wed, 30 Nov 2016 22:18:58 GMT"
let $nl := '&#10;'
let $string-to-sign := concat('GET', $nl,
                            $nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl, 
                            'x-ms-date:', $date, $nl,
                            'x-ms-version:2014-02-14', $nl,
                            '/', $storage-account-name, '/', $nl,
                            'comp:list') 
                            
let $hmac := crypto:hmac($string-to-sign, datatypes:base64Binary-to-byte($storage-key), "HMAC-SHA-256", "base64")
return
    <results>
        <calculation>{$hmac}</calculation>
        <expected-answer>pSSVTYF4pErpH2DOi8Wzvj0B1nru/sM4pLJHimoMZMg=</expected-answer>
    </results>

We have tested this new Expath Crypto module thoroughly for several weeks now and I am happy to report that it is working very well for us!

A big "Thank You" to Claudius for the work he put into this and working with us to test all of these different scenarios. Encryption can be a very difficult and confusing area to work in. 

And while I am at it, a big “Thank You” to the core developers of eXist-db as well. eXist-db is really an amazing project with a strong supportive community. I am looking forward to the official 3.0 release!

Nick
  
On Dec 13, 2016, at 6:17 AM, Alister Pillow <[hidden email]> wrote:

Thanks Nick, appreciate your follow up.


On 13 Dec. 2016, at 4:54 am, nsincaglia <[hidden email]> wrote:

All:

I think I know what the problem is here. I think the problem is not being
caused by a bug in either the Crypto module nor its interaction with
eXist-db. I think the issue is being cause by the fact that the Crypto
module's secret key input variable must be encoded as a UTF-8 string. The
reason this is a problem is because the conversion from base64 string to
UTF-8 strings, in certain circumstances, results in corrupted a corrupted
secret key variable.

I found this article which explains the problem in detail.
http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/

In order for the Crypto Module to be able to support Amazon S3 V4
signatures, a new function call needs to be added to the Crypto Module,
which will accept base64 string encoded secret keys as an input, without the
need to first convert it to UTF-8. Because it is in this conversion, which
the secret key is getting corrupted.

Let me know if this does not make sense. I have software examples I can
provide, but the above mentioned article explains the issue pretty well.

Nick




--
View this message in context: http://exist.2174344.n4.nabble.com/hmac-signature-for-AWS-S3-tp4670354p4671042.html
Sent from the exist-open mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open







Nick Sincaglia
President/Founder
NueMeta LLC
Digital Media & Technology
Phone: +1-630-303-7035
Skype: nsincaglia





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Alister Pillow-3
Thanks Claudius and Nick for your replies. I think we’re all looking forward to v3.0, but I can’t upgrade yet.

Nick, do you remember whether you were using the binaries via the src attribute in the http body?
<http:body media-type=“binary” src=“…

or directly embedded
<http:body media-type=“binary”>{util:binary-doc(‘/db/path/to/doc’)}</http:body>

I was unable to POST binary files until yesterday when I tried adding the http-version to the request:
<http:request method=“post” http-version=“1.0”>…

The file issue is a great concern (after my joy at getting this “move-to-s3” to work yesterday). Are the files closed when the query terminates? Is there any way to determine whether the files are still open? If the files are closed when the query ends, I can use the scheduler to “move-to-s3” in small batches.

I think I encountered this problem when I tried to delete a file after uploading it to S3. I decided that the “delete” process could be run separately.

Thanks again,
Alister


On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:

Alister,

The new Crypto module is for eXist-db v3.0 due to the fact that the structure of the XAR files has changed between the 2.2 and 3.0 versions. 

In regards to interacting with these various web service APIs using the EXpath HTTP module. I know that when we were using eXist-db 2.2, there were some issues with that module which only allowed us to PUT/POST XML files. We were unable to transfer binary files using this module. We sought out some help from Adam Retter to try to determine what was wrong. He determined that it was an issue with that module Expath HTTP module and coordinated with Florent Georges the original author of that module and some others to make the needed changes to get uploads of binary files using regular and multi-part requests. Those improvements are now incorporated into eXist-db v3.0.

Since you mention that you are working with binary data, I feel compelled to also mention that Adam also identified and fixed a bug in eXist-db where if you were working with binary files, eXist-db would not release the memory those binary files require, even after you were done performing your operation on it. This is another reason to switch to version 3.0. I know experienced this issue because we have processes which we run which grab a set of binary files, one at a time, in a loop and if the cumulative size of those binary files exceeded the RAM we allocated to our JVM, the system would crash. It is my understanding that this issue has been eliminated in the new version.

Nick
 
On Jan 29, 2017, at 5:56 PM, Alister Pillow <[hidden email]> wrote:

Hi Nick,

Excellent! Would this be for eXist-db v3+ only?

How are you interacting with S3? Are you using EXPath HTTP? 

I’ve recently written a module for creating the ASW S4 V4 signature which works with eXist v2.2x. It can create signed HTML Forms (for upload to S3 from a browser) and URLs (for download from S3), AND until just now (while trying to describe the problem) I was unable to send directly from eXist using the EXPath client. But now - thanks to your unwitting help :) I have found the magic incantation that makes the send-request work with AWS S3. 

tldr;

My users have been filling the database with binary files and the weekly backup weighs 12GB zipped. I need to offload their files into their own AWS bucket. Now I can do this. 

Happy to share for those wanting this in 2.2 (assuming Claudius’ work is v3 only).

Regards,
Alister



On 29 Jan 2017, at 4:59 am, Nick Sincaglia <[hidden email]> wrote:

All,

I am very happy to report that a new Expath Crypto module (version 0.3.5) was recently posted by Claudius, which supports Amazon’s S3 V4 REST web service API’s algorithm for creating signatures. Yeah! 

But wait there is more….. :-)

While we were working with Claudius on the Amazon S3 V4 testing, we also had a need to connect to the Azure Cloud Storage Service REST API as well. From reading through this API documentation, we discovered that the Azure Storage Service API added a new signature requirement that was different from either the Amazon’s V2 and V4 API signatures. So, in addition, this new Expath Crypto library can also support the Azure Cloud Storage REST API signature. Double Yeah!

With support for these 3 different signature algorithms, I believe this new version of the Expath Crypto library will be able to support a number of encryption scenarios. Let me try to explain the differences between the individual signature algorithms and how the new capabilities within the Expath Crypto module is now able to support these different requirements. 

In the case of the Amazon S3 V2 API signature algorithm, the secret key used to generate the signature is a UTF-8 string and the output of the hmac() function is a base64 encoded string. The algorithm only requires a single call to the hmac() function. The below snippet of code illustrates how to use the Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 2

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto"; 

let $secret-key := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
let $nl := '&#10;' (: the newline character :)
let $string-to-sign := concat("GET", $nl, $nl, $nl, "Wed, 28 Mar 2007 01:49:49 +0000", $nl, "/dictionary/fran%C3%A7ais/pr%c3%a9f%c3%a8re")
let $signature := crypto:hmac($string-to-sign, util:string-to-binary($secret-key), "HMAC-SHA-1", "base64")

return
    <results>
        <calculated>{$signature}</calculated>
        <expected-value>DNEZGsoieTZ92F3bUfSPQcbGmlM=</expected-value>
    </results>

In the case of the Amazon S3 V4 API signature algorithm, the secret key used to create the signature is again, a UTF-8 string. However, this signature calculation algorithm requires 4 calls to the hmac() function, where the output of the each hmac() function is the input of the next hmac() function call. In this case the first call to the hmac() function receives a UTF-8 secret key as an input, but every subsequent call to the hmac() function must accept a base64 encoded string as an input because that is what the hmac() function returns. In the previous version of the Expath crypto module, this scenario was not supported. In the new version it is.

The below snippet of code illustrates how to use the new Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 4

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto";

let $region := 'us-east-1'
let $service := 'iam'
let $date-YYYYMMDD := '20120215'
let $s3-secret-key := 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
let $kDate := crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256')
let $kRegion := crypto:hmac($region, $kDate, 'HMAC-SHA-256')
let $kService := crypto:hmac($service, $kRegion, 'HMAC-SHA-256')
let $kSigning := crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')

return 
    <results>
        <kDate-calculated>{crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256','hex')}</kDate-calculated>
        <kDate-expected>969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d</kDate-expected>
        <kRegion-calculated>{crypto:hmac($region, $kDate, 'HMAC-SHA-256','hex')}</kRegion-calculated>
        <kRegion-expected>69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c</kRegion-expected>
        <kService-calculated>{crypto:hmac($service, $kRegion, 'HMAC-SHA-256','hex')}</kService-calculated>
        <kService-expected>f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa</kService-expected>
        <kSigning-calculated>{crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')}</kSigning-calculated>
        <kSigning-expected>f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d</kSigning-expected>
    </results>


In the case of the Azure Storage Service API signature algorithm, the secret key used to create the signature is a base64 encoded string. This base64 encoded secret key string must converted directly into a byte array inside the hmac() function before the encryption algorithm can be applied. This was also something the previous version of the Expath Crypto module was not able to support but which the new version can. In order to support this, Claudius needed to create the ExPath Datatypes Module which is needed to work in conjunction with the Expath Crypto Module.  Below is a snippet of code to illustrate how the new Expath Crypto module and Expath Datatypes Module must be used together to generate the correct signature for the Azure Storage Service API.

Microsoft Azure Storage Service API

xquery version "3.0";
import module namespace crypto = "http://expath.org/ns/crypto"; 
import module namespace datatypes = "http://expath.org/ns/datatypes";

let $storage-key := "RzpkpwCtKeD25zjx+19YD0kWT0vwOIsA7yk6hnJco0jnMBM7J8tTrppqOJZ7wB0KSO6B2oKtkOlzUN/LRkpY2Q=="
let $storage-account-name := "d11portalvhds2g6lndzyd0ej"
let $date := "Wed, 30 Nov 2016 22:18:58 GMT"
let $nl := '&#10;'
let $string-to-sign := concat('GET', $nl,
                            $nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl, 
                            'x-ms-date:', $date, $nl,
                            'x-ms-version:2014-02-14', $nl,
                            '/', $storage-account-name, '/', $nl,
                            'comp:list') 
                            
let $hmac := crypto:hmac($string-to-sign, datatypes:base64Binary-to-byte($storage-key), "HMAC-SHA-256", "base64")
return
    <results>
        <calculation>{$hmac}</calculation>
        <expected-answer>pSSVTYF4pErpH2DOi8Wzvj0B1nru/sM4pLJHimoMZMg=</expected-answer>
    </results>

We have tested this new Expath Crypto module thoroughly for several weeks now and I am happy to report that it is working very well for us!

A big "Thank You" to Claudius for the work he put into this and working with us to test all of these different scenarios. Encryption can be a very difficult and confusing area to work in. 

And while I am at it, a big “Thank You” to the core developers of eXist-db as well. eXist-db is really an amazing project with a strong supportive community. I am looking forward to the official 3.0 release!

Nick
  
On Dec 13, 2016, at 6:17 AM, Alister Pillow <[hidden email]> wrote:

Thanks Nick, appreciate your follow up.


On 13 Dec. 2016, at 4:54 am, nsincaglia <[hidden email]> wrote:

All:

I think I know what the problem is here. I think the problem is not being
caused by a bug in either the Crypto module nor its interaction with
eXist-db. I think the issue is being cause by the fact that the Crypto
module's secret key input variable must be encoded as a UTF-8 string. The
reason this is a problem is because the conversion from base64 string to
UTF-8 strings, in certain circumstances, results in corrupted a corrupted
secret key variable.

I found this article which explains the problem in detail.
http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/

In order for the Crypto Module to be able to support Amazon S3 V4
signatures, a new function call needs to be added to the Crypto Module,
which will accept base64 string encoded secret keys as an input, without the
need to first convert it to UTF-8. Because it is in this conversion, which
the secret key is getting corrupted.

Let me know if this does not make sense. I have software examples I can
provide, but the above mentioned article explains the issue pretty well.

Nick




--
View this message in context: http://exist.2174344.n4.nabble.com/hmac-signature-for-AWS-S3-tp4670354p4671042.html
Sent from the exist-open mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open







Nick Sincaglia
President/Founder
NueMeta LLC
Digital Media & Technology
Phone: +1-630-303-7035
Skype: nsincaglia






------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

nsincaglia
Hi Alister,
Here is the code we use to upload binary files to S3. We have this code inside a function which we pass in a variable $input node which is a small XML structure with a number of our settings to connect to our S3 bucket.

    let $connection-string := $input//connection-string/text()
    let $s3-access-key := $input//connection-login/text()
    let $s3-secret-key := $input//connection-passcode/text() 
    let $file-path := $input//file-path/text()
    
    let $nl := '&#10;' (: the newline character :)
    let $date := utilities:http-date()
    let $bucket-name := substring-after(substring-before($connection-string, concat('.', $amazon-s3-v3:s3-site)), 'https://')
    let $file-upload-url := concat($connection-string, '/', $file-path) 
    let $content-type := 'application/octet-stream'
    let $string-to-sign := concat('POST', $nl, $nl, $nl, $nl, 'x-amz-date:', $date, $nl, '/', $bucket-name, '/', $file-path)

    let $signature := crypto:hmac($string-to-sign, util:string-to-binary($s3-secret-key), "HMAC-SHA-1", "base64")
    let $headers := (<http:header name="Date" value="{$date}" />, <http:header name="Authorization" value="AWS {$s3-access-key}:{$signature}" />)

    let $request := 
        <http:request method="POST" http-version="1.0">
            <http:header name="Authorization" value="AWS {$s3-access-key}:{$signature}" />
            <http:header name="x-amz-date" value="{$date}" />
            <http:multipart media-type="multipart/form-data" boundary="9431149156168">
            
                <http:header name="Content-Disposition" value="form-data; name=&quot;key&quot;"/>
         <http:body media-type="text/plain" method="text">{functx:substring-after-last($file-path, '/')}</http:body>    

         <http:header name="Content-Disposition" value="form-data; name=&quot;file&quot;; filename=&quot;audio-file-upload.wav&quot;"/>
         <http:header name="Content-Type" value="audio/wav"/>
         <http:body media-type="audio/wav" method="binary"/>
        
         </http:multipart>
        </http:request>

    let $response := http:send-request($request, $file-upload-url, $binary-file-data)
    
    return
        <results>{$response}</results>

Let me know if you have any questions.

Nick

On Jan 30, 2017, at 4:20 PM, Alister Pillow <[hidden email]> wrote:

Thanks Claudius and Nick for your replies. I think we’re all looking forward to v3.0, but I can’t upgrade yet.

Nick, do you remember whether you were using the binaries via the src attribute in the http body?
<http:body media-type=“binary” src=“…

or directly embedded
<http:body media-type=“binary”>{util:binary-doc(‘/db/path/to/doc’)}</http:body>

I was unable to POST binary files until yesterday when I tried adding the http-version to the request:
<http:request method=“post” http-version=“1.0”>…

The file issue is a great concern (after my joy at getting this “move-to-s3” to work yesterday). Are the files closed when the query terminates? Is there any way to determine whether the files are still open? If the files are closed when the query ends, I can use the scheduler to “move-to-s3” in small batches.

I think I encountered this problem when I tried to delete a file after uploading it to S3. I decided that the “delete” process could be run separately.

Thanks again,
Alister


On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:

Alister,

The new Crypto module is for eXist-db v3.0 due to the fact that the structure of the XAR files has changed between the 2.2 and 3.0 versions. 

In regards to interacting with these various web service APIs using the EXpath HTTP module. I know that when we were using eXist-db 2.2, there were some issues with that module which only allowed us to PUT/POST XML files. We were unable to transfer binary files using this module. We sought out some help from Adam Retter to try to determine what was wrong. He determined that it was an issue with that module Expath HTTP module and coordinated with Florent Georges the original author of that module and some others to make the needed changes to get uploads of binary files using regular and multi-part requests. Those improvements are now incorporated into eXist-db v3.0.

Since you mention that you are working with binary data, I feel compelled to also mention that Adam also identified and fixed a bug in eXist-db where if you were working with binary files, eXist-db would not release the memory those binary files require, even after you were done performing your operation on it. This is another reason to switch to version 3.0. I know experienced this issue because we have processes which we run which grab a set of binary files, one at a time, in a loop and if the cumulative size of those binary files exceeded the RAM we allocated to our JVM, the system would crash. It is my understanding that this issue has been eliminated in the new version.

Nick
 
On Jan 29, 2017, at 5:56 PM, Alister Pillow <[hidden email]> wrote:

Hi Nick,

Excellent! Would this be for eXist-db v3+ only?

How are you interacting with S3? Are you using EXPath HTTP? 

I’ve recently written a module for creating the ASW S4 V4 signature which works with eXist v2.2x. It can create signed HTML Forms (for upload to S3 from a browser) and URLs (for download from S3), AND until just now (while trying to describe the problem) I was unable to send directly from eXist using the EXPath client. But now - thanks to your unwitting help :) I have found the magic incantation that makes the send-request work with AWS S3. 

tldr;

My users have been filling the database with binary files and the weekly backup weighs 12GB zipped. I need to offload their files into their own AWS bucket. Now I can do this. 

Happy to share for those wanting this in 2.2 (assuming Claudius’ work is v3 only).

Regards,
Alister



On 29 Jan 2017, at 4:59 am, Nick Sincaglia <[hidden email]> wrote:

All,

I am very happy to report that a new Expath Crypto module (version 0.3.5) was recently posted by Claudius, which supports Amazon’s S3 V4 REST web service API’s algorithm for creating signatures. Yeah! 

But wait there is more….. :-)

While we were working with Claudius on the Amazon S3 V4 testing, we also had a need to connect to the Azure Cloud Storage Service REST API as well. From reading through this API documentation, we discovered that the Azure Storage Service API added a new signature requirement that was different from either the Amazon’s V2 and V4 API signatures. So, in addition, this new Expath Crypto library can also support the Azure Cloud Storage REST API signature. Double Yeah!

With support for these 3 different signature algorithms, I believe this new version of the Expath Crypto library will be able to support a number of encryption scenarios. Let me try to explain the differences between the individual signature algorithms and how the new capabilities within the Expath Crypto module is now able to support these different requirements. 

In the case of the Amazon S3 V2 API signature algorithm, the secret key used to generate the signature is a UTF-8 string and the output of the hmac() function is a base64 encoded string. The algorithm only requires a single call to the hmac() function. The below snippet of code illustrates how to use the Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 2

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto"; 

let $secret-key := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
let $nl := '&#10;' (: the newline character :)
let $string-to-sign := concat("GET", $nl, $nl, $nl, "Wed, 28 Mar 2007 01:49:49 +0000", $nl, "/dictionary/fran%C3%A7ais/pr%c3%a9f%c3%a8re")
let $signature := crypto:hmac($string-to-sign, util:string-to-binary($secret-key), "HMAC-SHA-1", "base64")

return
    <results>
        <calculated>{$signature}</calculated>
        <expected-value>DNEZGsoieTZ92F3bUfSPQcbGmlM=</expected-value>
    </results>

In the case of the Amazon S3 V4 API signature algorithm, the secret key used to create the signature is again, a UTF-8 string. However, this signature calculation algorithm requires 4 calls to the hmac() function, where the output of the each hmac() function is the input of the next hmac() function call. In this case the first call to the hmac() function receives a UTF-8 secret key as an input, but every subsequent call to the hmac() function must accept a base64 encoded string as an input because that is what the hmac() function returns. In the previous version of the Expath crypto module, this scenario was not supported. In the new version it is.

The below snippet of code illustrates how to use the new Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 4

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto";

let $region := 'us-east-1'
let $service := 'iam'
let $date-YYYYMMDD := '20120215'
let $s3-secret-key := 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
let $kDate := crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256')
let $kRegion := crypto:hmac($region, $kDate, 'HMAC-SHA-256')
let $kService := crypto:hmac($service, $kRegion, 'HMAC-SHA-256')
let $kSigning := crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')

return 
    <results>
        <kDate-calculated>{crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256','hex')}</kDate-calculated>
        <kDate-expected>969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d</kDate-expected>
        <kRegion-calculated>{crypto:hmac($region, $kDate, 'HMAC-SHA-256','hex')}</kRegion-calculated>
        <kRegion-expected>69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c</kRegion-expected>
        <kService-calculated>{crypto:hmac($service, $kRegion, 'HMAC-SHA-256','hex')}</kService-calculated>
        <kService-expected>f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa</kService-expected>
        <kSigning-calculated>{crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')}</kSigning-calculated>
        <kSigning-expected>f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d</kSigning-expected>
    </results>


In the case of the Azure Storage Service API signature algorithm, the secret key used to create the signature is a base64 encoded string. This base64 encoded secret key string must converted directly into a byte array inside the hmac() function before the encryption algorithm can be applied. This was also something the previous version of the Expath Crypto module was not able to support but which the new version can. In order to support this, Claudius needed to create the ExPath Datatypes Module which is needed to work in conjunction with the Expath Crypto Module.  Below is a snippet of code to illustrate how the new Expath Crypto module and Expath Datatypes Module must be used together to generate the correct signature for the Azure Storage Service API.

Microsoft Azure Storage Service API

xquery version "3.0";
import module namespace crypto = "http://expath.org/ns/crypto"; 
import module namespace datatypes = "http://expath.org/ns/datatypes";

let $storage-key := "RzpkpwCtKeD25zjx+19YD0kWT0vwOIsA7yk6hnJco0jnMBM7J8tTrppqOJZ7wB0KSO6B2oKtkOlzUN/LRkpY2Q=="
let $storage-account-name := "d11portalvhds2g6lndzyd0ej"
let $date := "Wed, 30 Nov 2016 22:18:58 GMT"
let $nl := '&#10;'
let $string-to-sign := concat('GET', $nl,
                            $nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl, 
                            'x-ms-date:', $date, $nl,
                            'x-ms-version:2014-02-14', $nl,
                            '/', $storage-account-name, '/', $nl,
                            'comp:list') 
                            
let $hmac := crypto:hmac($string-to-sign, datatypes:base64Binary-to-byte($storage-key), "HMAC-SHA-256", "base64")
return
    <results>
        <calculation>{$hmac}</calculation>
        <expected-answer>pSSVTYF4pErpH2DOi8Wzvj0B1nru/sM4pLJHimoMZMg=</expected-answer>
    </results>

We have tested this new Expath Crypto module thoroughly for several weeks now and I am happy to report that it is working very well for us!

A big "Thank You" to Claudius for the work he put into this and working with us to test all of these different scenarios. Encryption can be a very difficult and confusing area to work in. 

And while I am at it, a big “Thank You” to the core developers of eXist-db as well. eXist-db is really an amazing project with a strong supportive community. I am looking forward to the official 3.0 release!

Nick
  
On Dec 13, 2016, at 6:17 AM, Alister Pillow <[hidden email]> wrote:

Thanks Nick, appreciate your follow up.


On 13 Dec. 2016, at 4:54 am, nsincaglia <[hidden email]> wrote:

All:

I think I know what the problem is here. I think the problem is not being
caused by a bug in either the Crypto module nor its interaction with
eXist-db. I think the issue is being cause by the fact that the Crypto
module's secret key input variable must be encoded as a UTF-8 string. The
reason this is a problem is because the conversion from base64 string to
UTF-8 strings, in certain circumstances, results in corrupted a corrupted
secret key variable.

I found this article which explains the problem in detail.
http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/

In order for the Crypto Module to be able to support Amazon S3 V4
signatures, a new function call needs to be added to the Crypto Module,
which will accept base64 string encoded secret keys as an input, without the
need to first convert it to UTF-8. Because it is in this conversion, which
the secret key is getting corrupted.

Let me know if this does not make sense. I have software examples I can
provide, but the above mentioned article explains the issue pretty well.

Nick




--
View this message in context: http://exist.2174344.n4.nabble.com/hmac-signature-for-AWS-S3-tp4670354p4671042.html
Sent from the exist-open mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open







Nick Sincaglia
President/Founder
NueMeta LLC
Digital Media & Technology
Phone: +1-630-303-7035
Skype: nsincaglia








Nick Sincaglia
President/Founder
NueMeta LLC
Digital Media & Technology
Phone: +1-630-303-7035
Skype: nsincaglia





------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Memory use in eXide (eXist-db 2.2x)

Alister Pillow-3
In reply to this post by nsincaglia
Hi 

My key observation is that memory use when running a script via the scheduler is much less, and relatively constant, than when running the same script in eXide. In this particular process, I’m using the EXPath HTTP client to POST binary documents to AWS S3.

tl;dr;

To reduce the “weight" of the db, I’m moving a Client's binary files (PDFs, ODTs etc) into their own S3 bucket. This is going well and I’ve already reduced the db backup (zip) by 2GB - over 16,000 files. Being concerned about memory use, I am performing this transfer in batches of 10 “job” collections at a time - about 100-200 files. 

Initially, while running these batches manually from eXide, the memory use gradually increased to the point where I chose to restart the db rather than risk anything. But now, I’m running the script from a scheduled-xquery job, and the memory use is staying low.

The simple conclusion is that eXide has an impact on memory use (or is not freeing up memory). I guess it's also possible that the reported values for memory use are distorted in that environment.

I’ve been recording the memory use for each batch using the script from the eXist book (p 393 in Chapter 15, Understanding Memory Use) - just in case this is useful.

Regards,
Alister

(PS Hope you’re having fun in Prague)

On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:

Alister,

The new Crypto module is for eXist-db v3.0 due to the fact that the structure of the XAR files has changed between the 2.2 and 3.0 versions. 

In regards to interacting with these various web service APIs using the EXpath HTTP module. I know that when we were using eXist-db 2.2, there were some issues with that module which only allowed us to PUT/POST XML files. We were unable to transfer binary files using this module. We sought out some help from Adam Retter to try to determine what was wrong. He determined that it was an issue with that module Expath HTTP module and coordinated with Florent Georges the original author of that module and some others to make the needed changes to get uploads of binary files using regular and multi-part requests. Those improvements are now incorporated into eXist-db v3.0.

Since you mention that you are working with binary data, I feel compelled to also mention that Adam also identified and fixed a bug in eXist-db where if you were working with binary files, eXist-db would not release the memory those binary files require, even after you were done performing your operation on it. This is another reason to switch to version 3.0. I know experienced this issue because we have processes which we run which grab a set of binary files, one at a time, in a loop and if the cumulative size of those binary files exceeded the RAM we allocated to our JVM, the system would crash. It is my understanding that this issue has been eliminated in the new version.

Nick
 
On Jan 29, 2017, at 5:56 PM, Alister Pillow <[hidden email]> wrote:

Hi Nick,

Excellent! Would this be for eXist-db v3+ only?

How are you interacting with S3? Are you using EXPath HTTP? 

I’ve recently written a module for creating the ASW S4 V4 signature which works with eXist v2.2x. It can create signed HTML Forms (for upload to S3 from a browser) and URLs (for download from S3), AND until just now (while trying to describe the problem) I was unable to send directly from eXist using the EXPath client. But now - thanks to your unwitting help :) I have found the magic incantation that makes the send-request work with AWS S3. 

tldr;

My users have been filling the database with binary files and the weekly backup weighs 12GB zipped. I need to offload their files into their own AWS bucket. Now I can do this. 

Happy to share for those wanting this in 2.2 (assuming Claudius’ work is v3 only).

Regards,
Alister



On 29 Jan 2017, at 4:59 am, Nick Sincaglia <[hidden email]> wrote:

All,

I am very happy to report that a new Expath Crypto module (version 0.3.5) was recently posted by Claudius, which supports Amazon’s S3 V4 REST web service API’s algorithm for creating signatures. Yeah! 

But wait there is more….. :-)

While we were working with Claudius on the Amazon S3 V4 testing, we also had a need to connect to the Azure Cloud Storage Service REST API as well. From reading through this API documentation, we discovered that the Azure Storage Service API added a new signature requirement that was different from either the Amazon’s V2 and V4 API signatures. So, in addition, this new Expath Crypto library can also support the Azure Cloud Storage REST API signature. Double Yeah!

With support for these 3 different signature algorithms, I believe this new version of the Expath Crypto library will be able to support a number of encryption scenarios. Let me try to explain the differences between the individual signature algorithms and how the new capabilities within the Expath Crypto module is now able to support these different requirements. 

In the case of the Amazon S3 V2 API signature algorithm, the secret key used to generate the signature is a UTF-8 string and the output of the hmac() function is a base64 encoded string. The algorithm only requires a single call to the hmac() function. The below snippet of code illustrates how to use the Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 2

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto"; 

let $secret-key := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
let $nl := '&#10;' (: the newline character :)
let $string-to-sign := concat("GET", $nl, $nl, $nl, "Wed, 28 Mar 2007 01:49:49 +0000", $nl, "/dictionary/fran%C3%A7ais/pr%c3%a9f%c3%a8re")
let $signature := crypto:hmac($string-to-sign, util:string-to-binary($secret-key), "HMAC-SHA-1", "base64")

return
    <results>
        <calculated>{$signature}</calculated>
        <expected-value>DNEZGsoieTZ92F3bUfSPQcbGmlM=</expected-value>
    </results>

In the case of the Amazon S3 V4 API signature algorithm, the secret key used to create the signature is again, a UTF-8 string. However, this signature calculation algorithm requires 4 calls to the hmac() function, where the output of the each hmac() function is the input of the next hmac() function call. In this case the first call to the hmac() function receives a UTF-8 secret key as an input, but every subsequent call to the hmac() function must accept a base64 encoded string as an input because that is what the hmac() function returns. In the previous version of the Expath crypto module, this scenario was not supported. In the new version it is.

The below snippet of code illustrates how to use the new Expath Crypto module to generate the correct signature.

Amazon S3 AWS Signature Version 4

xquery version "3.0";

import module namespace crypto = "http://expath.org/ns/crypto";

let $region := 'us-east-1'
let $service := 'iam'
let $date-YYYYMMDD := '20120215'
let $s3-secret-key := 'wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
let $kDate := crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256')
let $kRegion := crypto:hmac($region, $kDate, 'HMAC-SHA-256')
let $kService := crypto:hmac($service, $kRegion, 'HMAC-SHA-256')
let $kSigning := crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')

return 
    <results>
        <kDate-calculated>{crypto:hmac($date-YYYYMMDD, concat('AWS4', $s3-secret-key), 'HMAC-SHA-256','hex')}</kDate-calculated>
        <kDate-expected>969fbb94feb542b71ede6f87fe4d5fa29c789342b0f407474670f0c2489e0a0d</kDate-expected>
        <kRegion-calculated>{crypto:hmac($region, $kDate, 'HMAC-SHA-256','hex')}</kRegion-calculated>
        <kRegion-expected>69daa0209cd9c5ff5c8ced464a696fd4252e981430b10e3d3fd8e2f197d7a70c</kRegion-expected>
        <kService-calculated>{crypto:hmac($service, $kRegion, 'HMAC-SHA-256','hex')}</kService-calculated>
        <kService-expected>f72cfd46f26bc4643f06a11eabb6c0ba18780c19a8da0c31ace671265e3c87fa</kService-expected>
        <kSigning-calculated>{crypto:hmac('aws4_request', $kService, 'HMAC-SHA-256', 'hex')}</kSigning-calculated>
        <kSigning-expected>f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d</kSigning-expected>
    </results>


In the case of the Azure Storage Service API signature algorithm, the secret key used to create the signature is a base64 encoded string. This base64 encoded secret key string must converted directly into a byte array inside the hmac() function before the encryption algorithm can be applied. This was also something the previous version of the Expath Crypto module was not able to support but which the new version can. In order to support this, Claudius needed to create the ExPath Datatypes Module which is needed to work in conjunction with the Expath Crypto Module.  Below is a snippet of code to illustrate how the new Expath Crypto module and Expath Datatypes Module must be used together to generate the correct signature for the Azure Storage Service API.

Microsoft Azure Storage Service API

xquery version "3.0";
import module namespace crypto = "http://expath.org/ns/crypto"; 
import module namespace datatypes = "http://expath.org/ns/datatypes";

let $storage-key := "RzpkpwCtKeD25zjx+19YD0kWT0vwOIsA7yk6hnJco0jnMBM7J8tTrppqOJZ7wB0KSO6B2oKtkOlzUN/LRkpY2Q=="
let $storage-account-name := "d11portalvhds2g6lndzyd0ej"
let $date := "Wed, 30 Nov 2016 22:18:58 GMT"
let $nl := '&#10;'
let $string-to-sign := concat('GET', $nl,
                            $nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl,$nl, 
                            'x-ms-date:', $date, $nl,
                            'x-ms-version:2014-02-14', $nl,
                            '/', $storage-account-name, '/', $nl,
                            'comp:list') 
                            
let $hmac := crypto:hmac($string-to-sign, datatypes:base64Binary-to-byte($storage-key), "HMAC-SHA-256", "base64")
return
    <results>
        <calculation>{$hmac}</calculation>
        <expected-answer>pSSVTYF4pErpH2DOi8Wzvj0B1nru/sM4pLJHimoMZMg=</expected-answer>
    </results>

We have tested this new Expath Crypto module thoroughly for several weeks now and I am happy to report that it is working very well for us!

A big "Thank You" to Claudius for the work he put into this and working with us to test all of these different scenarios. Encryption can be a very difficult and confusing area to work in. 

And while I am at it, a big “Thank You” to the core developers of eXist-db as well. eXist-db is really an amazing project with a strong supportive community. I am looking forward to the official 3.0 release!

Nick
  
On Dec 13, 2016, at 6:17 AM, Alister Pillow <[hidden email]> wrote:

Thanks Nick, appreciate your follow up.


On 13 Dec. 2016, at 4:54 am, nsincaglia <[hidden email]> wrote:

All:

I think I know what the problem is here. I think the problem is not being
caused by a bug in either the Crypto module nor its interaction with
eXist-db. I think the issue is being cause by the fact that the Crypto
module's secret key input variable must be encoded as a UTF-8 string. The
reason this is a problem is because the conversion from base64 string to
UTF-8 strings, in certain circumstances, results in corrupted a corrupted
secret key variable.

I found this article which explains the problem in detail.
http://haacked.com/archive/2012/01/30/hazards-of-converting-binary-data-to-a-string.aspx/

In order for the Crypto Module to be able to support Amazon S3 V4
signatures, a new function call needs to be added to the Crypto Module,
which will accept base64 string encoded secret keys as an input, without the
need to first convert it to UTF-8. Because it is in this conversion, which
the secret key is getting corrupted.

Let me know if this does not make sense. I have software examples I can
provide, but the above mentioned article explains the issue pretty well.

Nick




--
View this message in context: http://exist.2174344.n4.nabble.com/hmac-signature-for-AWS-S3-tp4670354p4671042.html
Sent from the exist-open mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open







Nick Sincaglia
President/Founder
NueMeta LLC
Digital Media & Technology
Phone: +1-630-303-7035
Skype: nsincaglia






------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hmac signature for AWS S3

Alister Pillow-3
In reply to this post by nsincaglia
eXist-db 2.2x, 10GB available memory.
Uploading binary files to AWS S3 using EXPath HTTP client.

And I’ve just hit this issue - it’s not a Memory problem, it’s Too Many Open Files. 
On 31 Jan 2017, at 4:52 am, Nick Sincaglia <[hidden email]> wrote:

Since you mention that you are working with binary data, I feel compelled to also mention that Adam also identified and fixed a bug in eXist-db where if you were working with binary files, eXist-db would not release the memory those binary files require, even after you were done performing your operation on it. 

Once I have removed these files, it will be much easier to begin the upgrade-to-3 process.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Exist-open mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/exist-open
12
Loading...