Object Storage API

Available in Classic and VPC

This guide provides support information for the Object Storage API, which is compatible with the Amazon S3 API, and describes common API calls and authentication methods. Requests made from NAVER Cloud Platform’s Object Storage via the Amazon S3 API must be authenticated using AWS’ authorization header implementation. Authentication methods supported by Object Storage include the AWS Signature Version 2 (AWS2) and AWS Signature Version 4 (AWS4) algorithms, and this guide describes the process based on AWS4.

Supported APIs

The Object Storage API can control various functions required for storage management and use. For a list of supported operations, refer to the Object Storage API Guide.

API call

The steps for calling the Object Storage API are as follows:

1. Create authentication key
2. Check calling domain
3. Authenticate
4. Call

1. Create authentication key

The method for generating an authentication key for use with the Object Storage API is identical to that of the Ncloud API. For more information, refer to 1. Generate authentication key of the Ncloud API guide.

2. Check calling domain

Although both HTTP and HTTPS protocols are supported, HTTPS is recommended for data protection reasons. Region-specific calling domains are as follows:

3. Authenticate

The authorization header is generated as follows:

1) Create canonical request
2) Create string to sign
3) Derive signing key
4) Create signature
5) Create authorization header

1. Create canonical request

The format of a canonical request and the description for each element are as follows:

<HTTPMethod>\n
<CanonicalURI>\n
<CanonicalQueryString>\n
<CanonicalHeaders>\n
<SignedHeaders>\n
<HashedPayload>

• HTTPMethod

Declare the HTTP method to be used. <Example> PUT

• CanonicalURI

Define the resource to be accessed in a standardized format (URI-encoded). <Example> /path/object

• CanonicalQueryString

In response to request parameters, define the request parameters in a standardized format (URI-encoded) and list them alphabetically.
UriEncode("marker")+"="+UriEncode("someMarker")+"&"+ UriEncode("max-keys")+"="+UriEncode("20") + "&" + UriEncode("prefix")+"="+UriEncode("somePrefix")

• CanonicalHeaders

Define header names and values included in the request in a standardized format. Convert all header names to lowercase, and remove leading and trailing spaces from header values. Insert a newline character\n at the end of each header item. In this case, header names need to be sorted alphabetically.
The host header and headers starting with x-amz- are mandatory. Including all headers is recommended for more accurate data consistency checks.
All requests for ASW4 authentication must include the x-amz-content-sha256 header. A hash value for the data (payload) is required, for which two options are available:

* Signed data (signed payload): enter the hexadecimal value of the SHA-256 hash of the data (payload). If the payload is empty, you should enter the hash value of an empty string.
* Unsigned data (unsigned payload): to leave the data (payload) unsigned, enter the string UNSIGNED-PAYLOAD as the value of x-amz-content-sha256 header.

```
Lowercase(<HeaderName1>)+":"+Trim(<value>)+"\n"
Lowercase(<HeaderName2>)+":"+Trim(<value>)+"\n"
```

• SignedHeaders

List the header names included in the requests. Convert all header names to lowercase and sort alphabetically. Use a semicolon (;) to separate the header names.

```
host;x-amz-content-sha256;x-amz-date
```

• HashedPayload

Enter the hexadecimal values of the SHA256 hash of the data (payload). (Same as the x-amz-content-sha256 header value) If there is no request data (payload), enter the hash value of an empty string. To leave the payload unsigned, enter the string UNSIGNED-PAYLOAD.

```
Hex(SHA256Hash(<payload>)
or
Hex(SHA256Hash(""))
or
UNSIGNED-PAYLOAD
```

2. Create string to sign

The format of a string to sign and the description for each element are as follows:

AWS4-HMAC-SHA256\n
<Timestamp>\n
<Scope>\n
Hex(SHA256Hash(<CanonicalRequest>))

• Timestamp

It must be in UTC and represented according to the ISO 8601 standard. <Example> 20161128T152924Z

• Scope

The scope string consists of the following:

```
<Date>/<Region>/s3/aws4_request
```
* Date: enter in `YYYYMMDD` format.
* Region: enter a region name. Refer to 2. Check calling domain for a list of region names. Although a region name is required, NAVER Cloud Platform’s Object Storage does not check the consistency of region names because regions are distinguished by domain.

• CanonicalRequest

Enter the hexadecimal value of the SHA-256 hash of the canonical request created.

3. Derive signing key

The format of a signing key and the description for each element are as follows:

kSecret     = <SecretKey>
kDate       = HMAC-SHA256("AWS4" + kSecret, <Date>)
kRegion     = HMAC-SHA256(kDate, <Region>)
kService    = HMAC-SHA256(kRegion, "s3")
kSigning    = HMAC-SHA256(kService, "aws4_request")

• SecretKey

Enter the secret key of the API authentication key. You may view the secret key in [My Page] > [Account Management] > [Authentication Key Management] in the NAVER Cloud Platform portal.

• Date

Enter in YYYYMMDD format.

• Region

Enter a region name. Refer to 2. Check calling domain for a list of region names. Although a region name is required, NAVER Cloud Platform’s Object Storage does not check the consistency of region names because regions are distinguished by domain.

4. Create signature

The format of a signature and the description for each element are as follows:

Hex(HMAC-SHA256(<SigningKey>, <StringToSign>))

• Signing Key

Calculate and represent the HMAC-SHA256 hash of the signing key as hexadecimal values.

• StringToSign

Calculate and represent the HMAC-SHA256 hash of the string to sign as hexadecimal values.

5. Create authorization header

The format of an authorization header and the description for each element are as follows:

Authorization: AWS4-HMAC-SHA256 Credential=<AccessKeyID>/<Scope>, SignedHeaders=<SignedHeaders>, Signature=<Signature>

• AccessKeyID

Enter the access key ID of the API authentication key. You may view the secret key in [My Page] > [Account Management] > [Authentication Key Management] in the NAVER Cloud Platform portal.

• Scope

The scope string consists of the following:

```
<Date>/<Region>/s3/aws4_request
```
* Date: enter in YYYYMMDD format.
* Region: enter a region name. Refer to 2. Check calling domain for a list of region names. Although a region name is required, NAVER Cloud Platform’s Object Storage does not check the consistency of region names because regions are distinguished by domain.

• SignedHeaders

List the header names included in the requests. Convert all header names to lowercase and sort alphabetically. Use a semicolon (;) to separate the header names.
host;x-amz-content-sha256;x-amz-date

• Signature

Enter the signature created in step 4. Create signature.

The following are examples of how to create authorization headers by language:

public class ObjectStorageSample {
	private static byte[] sign(String stringData, byte[] key) throws NoSuchAlgorithmException, UnsupportedEncodingException, InvalidKeyException {
        byte[] data = stringData.getBytes(CHARSET_NAME);
        Mac e = Mac.getInstance(HMAC_ALGORITHM);
        e.init(new SecretKeySpec(key, HMAC_ALGORITHM));
        return e.doFinal(data);
    }

    private static String hash(String text) throws NoSuchAlgorithmException, UnsupportedEncodingException {
        MessageDigest e = MessageDigest.getInstance(HASH_ALGORITHM);
        e.update(text.getBytes(CHARSET_NAME));
        return Hex.encodeHexString(e.digest());
    }

    private static String getStandardizedQueryParameters(String queryString) throws UnsupportedEncodingException {
        TreeMap<String, String> sortedQueryParameters = new TreeMap<>();
        // sort by key name
        if (queryString != null && !queryString.isEmpty()) {
            String[] queryStringTokens = queryString.split("&");
            for (String field : queryStringTokens) {
                String[] fieldTokens = field.split("=");
                if (fieldTokens.length > 0) {
                    if (fieldTokens.length > 1) {
                        sortedQueryParameters.put(fieldTokens[0], fieldTokens[1]);
                    } else {
                        sortedQueryParameters.put(fieldTokens[0], "");
                    }
                }
            }
        }

        StringBuilder standardizedQueryParametersBuilder = new StringBuilder();
        int count = 0;
        for (String key : sortedQueryParameters.keySet()) {
            if (count > 0) {
                standardizedQueryParametersBuilder.append("&");
            }
            standardizedQueryParametersBuilder.append(key).append("=");

            if (sortedQueryParameters.get(key) != null && !sortedQueryParameters.get(key).isEmpty()) {
                standardizedQueryParametersBuilder.append(URLEncoder.encode(sortedQueryParameters.get(key), CHARSET_NAME));
            }

            count++;
        }
        return standardizedQueryParametersBuilder.toString();
    }

    private static TreeMap<String, String> getSortedHeaders(Header[] headers) {
        TreeMap<String, String> sortedHeaders = new TreeMap<>();
        // sort by header name
        for (Header header : headers) {
            String headerName = header.getName().toLowerCase();
            sortedHeaders.put(headerName, header.getValue());
        }

        return sortedHeaders;
    }

    private static String getSignedHeaders(TreeMap<String, String> sortedHeaders) {
		StringBuilder signedHeadersBuilder = new StringBuilder();
		for (String headerName : sortedHeaders.keySet()) {
			signedHeadersBuilder.append(headerName.toLowerCase()).append(";");
		}
        String s = signedHeadersBuilder.toString();
        if (s.endsWith(";")) {
            s = s.substring(0, s.length() - 1);
        }
		return s;
	}

    private static String getStandardizedHeaders(TreeMap<String, String> sortedHeaders) {
        StringBuilder standardizedHeadersBuilder = new StringBuilder();
        for (String headerName : sortedHeaders.keySet()) {
            standardizedHeadersBuilder.append(headerName.toLowerCase()).append(":").append(sortedHeaders.get(headerName)).append("\n");
        }

        return standardizedHeadersBuilder.toString();
    }

    private static String getCanonicalRequest(HttpUriRequest request, String standardizedQueryParameters, String standardizedHeaders, String signedHeaders) {
        StringBuilder canonicalRequestBuilder = new StringBuilder().append(request.getMethod()).append("\n")
            .append(request.getURI().getPath()).append("\n")
            .append(standardizedQueryParameters).append("\n")
            .append(standardizedHeaders).append("\n")
            .append(signedHeaders).append("\n")
            .append(UNSIGNED_PAYLOAD);

        return canonicalRequestBuilder.toString();
    }

    private static String getScope(String datestamp, String regionName) {
        StringBuilder scopeBuilder = new StringBuilder().append(datestamp).append("/")
            .append(regionName).append("/")
            .append(SERVICE_NAME).append("/")
            .append(REQUEST_TYPE);
        return scopeBuilder.toString();
    }

    private static String getStringToSign(String timestamp, String scope, String canonicalRequest) throws NoSuchAlgorithmException, UnsupportedEncodingException {
        StringBuilder stringToSignBuilder = new StringBuilder(AWS_ALGORITHM)
            .append("\n")
            .append(timestamp).append("\n")
            .append(scope).append("\n")
            .append(hash(canonicalRequest));

        return stringToSignBuilder.toString();
    }

    private static String getSignature(String secretKey, String datestamp, String regionName, String stringToSign) throws NoSuchAlgorithmException, UnsupportedEncodingException, InvalidKeyException {
        byte[] kSecret = ("AWS4" + secretKey).getBytes(CHARSET_NAME);
        byte[] kDate = sign(datestamp, kSecret);
        byte[] kRegion = sign(regionName, kDate);
        byte[] kService = sign(SERVICE_NAME, kRegion);
        byte[] signingKey = sign(REQUEST_TYPE, kService);

        return Hex.encodeHexString(sign(stringToSign, signingKey));
    }

    private static String getAuthorization(String accessKey, String scope, String signedHeaders, String signature) {
        String signingCredentials = accessKey + "/" + scope;
        String credential = "Credential=" + signingCredentials;
        String signerHeaders = "SignedHeaders=" + signedHeaders;
        String signatureHeader = "Signature=" + signature;

        StringBuilder authHeaderBuilder = new StringBuilder().append(AWS_ALGORITHM).append(" ")
            .append(credential).append(", ")
            .append(signerHeaders).append(", ")
            .append(signatureHeader);

        return authHeaderBuilder.toString();
    }

    private static void authorization(HttpUriRequest request, String regionName, String accessKey, String secretKey) throws Exception {
        Date now = new Date();
        DATE_FORMATTER.setTimeZone(TimeZone.getTimeZone("UTC"));
        TIME_FORMATTER.setTimeZone(TimeZone.getTimeZone("UTC"));
        String datestamp = DATE_FORMATTER.format(now);
        String timestamp = TIME_FORMATTER.format(now);

        request.addHeader("X-Amz-Date", timestamp);

        request.addHeader("X-Amz-Content-Sha256", UNSIGNED_PAYLOAD);

        String standardizedQueryParameters = getStandardizedQueryParameters(request.getURI().getQuery());

        TreeMap<String, String> sortedHeaders = getSortedHeaders(request.getAllHeaders());
        String signedHeaders = getSignedHeaders(sortedHeaders);
        String standardizedHeaders = getStandardizedHeaders(sortedHeaders);

        String canonicalRequest = getCanonicalRequest(request, standardizedQueryParameters, standardizedHeaders, signedHeaders);
        System.out.println("> canonicalRequest :");
        System.out.println(canonicalRequest);

        String scope = getScope(datestamp, regionName);

        String stringToSign = getStringToSign(timestamp, scope, canonicalRequest);
        System.out.println("> stringToSign :");
        System.out.println(stringToSign);

        String signature = getSignature(secretKey, datestamp, regionName, stringToSign);

        String authorization = getAuthorization(accessKey, scope, signedHeaders, signature);
        request.addHeader("Authorization", authorization);
    }

    private static void putObject(String bucketName, String objectName, String localFilePath) throws Exception {
        HttpClient httpClient = HttpClientBuilder.create().build();

        HttpPut request = new HttpPut(ENDPOINT + "/" + bucketName + "/" + objectName);
        request.addHeader("Host", request.getURI().getHost());
        request.setEntity(new FileEntity(new File(localFilePath)));

        authorization(request, REGION_NAME, ACCESS_KEY, SECRET_KEY);

        HttpResponse response = httpClient.execute(request);
        System.out.println("Response : " + response.getStatusLine());
    }

    private static void getObject(String bucketName, String objectName, String localFilePath) throws Exception {
        HttpClient httpClient = HttpClientBuilder.create().build();
        HttpGet request = new HttpGet(ENDPOINT + "/" + bucketName + "/" + objectName);
        request.addHeader("Host", request.getURI().getHost());

        authorization(request, REGION_NAME, ACCESS_KEY, SECRET_KEY);

        HttpResponse response = httpClient.execute(request);
        System.out.println("Response : " + response.getStatusLine());

        InputStream is = response.getEntity().getContent();
        File targetFile = new File(localFilePath);
        OutputStream os = new FileOutputStream(targetFile);

        byte[] buffer = new byte[8 * 1024];
        int bytesRead;
        while ((bytesRead = is.read(buffer)) != -1) {
            os.write(buffer, 0, bytesRead);
        }

        is.close();
        os.close();
    }

    private static void listObjects(String bucketName, String queryString) throws Exception {
        HttpClient httpClient = HttpClientBuilder.create().build();
        URI uri = new URI(ENDPOINT + "/" + bucketName + "?" + queryString);
        HttpGet request = new HttpGet(uri);
        request.addHeader("Host", request.getURI().getHost());

        authorization(request, REGION_NAME, ACCESS_KEY, SECRET_KEY);

        HttpResponse response = httpClient.execute(request);
        System.out.println("> Response : " + response.getStatusLine());
        int i;
        InputStream is = response.getEntity().getContent();
        StringBuffer buffer = new StringBuffer();
        byte[] b = new byte[4096];
        while ((i = is.read(b)) != -1) {
            buffer.append(new String(b, 0, i));
        }
        System.out.println(buffer.toString());

    }

    private static final String CHARSET_NAME = "UTF-8";
    private static final String HMAC_ALGORITHM = "HmacSHA256";
    private static final String HASH_ALGORITHM = "SHA-256";
    private static final String AWS_ALGORITHM = "AWS4-HMAC-SHA256";

    private static final String SERVICE_NAME = "s3";
    private static final String REQUEST_TYPE = "aws4_request";

    private static final String UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD";

    private static final SimpleDateFormat DATE_FORMATTER = new SimpleDateFormat("yyyyMMdd");
    private static final SimpleDateFormat TIME_FORMATTER = new SimpleDateFormat("yyyyMMdd\'T\'HHmmss\'Z\'");

    private static final String REGION_NAME = "kr-standard";
    private static final String ENDPOINT = "https://kr.object.ncloudstorage.com";
    private static final String ACCESS_KEY = "ACCESS_KEY_ID";
    private static final String SECRET_KEY = "SECRET_KEY";

    public static void main(String[] args) throws Exception {
        String bucketName = "sample-bucket";
        String objectName = "sample-object.txt";
        String sourceFilePath = "/tmp/source.txt";
        String targetFilePath = "/tmp/target.txt";

        putObject(bucketName, objectName, sourceFilePath);

        getObject(bucketName, objectName, targetFilePath);

        String queryString = "max-keys=10&delimiter=/";
        listObjects(bucketName, queryString);
    }
}

import hashlib
import hmac
import datetime
import requests
import urllib


def get_hash(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()


def create_signed_headers(headers):
    signed_headers = []

    for k in sorted(headers):
        signed_headers.append('%s;' % k)

    return ''.join(signed_headers)[:-1]


def create_standardized_headers(headers):
    signed_headers = []

    for k in sorted(headers):
        signed_headers.append('%s:%s\n' % (k, headers[k]))

    return ''.join(signed_headers)


def create_standardized_query_parameters(request_parameters):
    standardized_query_parameters = []

    if request_parameters:
        for k in sorted(request_parameters):
            standardized_query_parameters.append('%s=%s' % (k, urllib.quote(request_parameters[k], safe='')))

        return '&'.join(standardized_query_parameters)
    else:
        return ''


class ObjectStorageSample:
    def __init__(self):
        self.region = 'kr-standard'
        self.endpoint = 'https://kr.object.ncloudstorage.com'
        self.host = 'kr.object.ncloudstorage.com'
        self.access_key = 'ACCESS_KEY_ID'
        self.secret_key = 'SECRET_KEY'

        self.payload_hash = 'UNSIGNED-PAYLOAD'
        self.hashing_algorithm = 'AWS4-HMAC-SHA256'
        self.service_name = 's3'
        self.request_type = 'aws4_request'

        self.time_format = '%Y%m%dT%H%M%SZ'
        self.date_format = '%Y%m%d'

    def _create_credential_scope(self, date_stamp):
        return date_stamp + '/' + self.region + '/' + self.service_name + '/' + self.request_type

    def _create_canonical_request(self, http_method, request_path, request_parameters, headers):
        standardized_query_parameters = create_standardized_query_parameters(request_parameters)
        standardized_headers = create_standardized_headers(headers)
        signed_headers = create_signed_headers(headers)

        canonical_request = (http_method + '\n' +
                             request_path + '\n' +
                             standardized_query_parameters + '\n' +
                             standardized_headers + '\n' +
                             signed_headers + '\n' +
                             self.payload_hash)

        print('canonical_request:\n%s\n' % canonical_request)
        return canonical_request

    def _create_string_to_sign(self, time_stamp, credential_scope, canonical_request):
        string_to_sign = (self.hashing_algorithm + '\n' +
                          time_stamp + '\n' +
                          credential_scope + '\n' +
                          hashlib.sha256(canonical_request.encode('utf-8')).hexdigest())

        print('string_to_sign:\n%s\n' % string_to_sign)
        return string_to_sign

    def _create_signature_key(self, date_stamp):
        key_date = get_hash(('AWS4' + self.secret_key).encode('utf-8'), date_stamp)
        key_string = get_hash(key_date, self.region)
        key_service = get_hash(key_string, self.service_name)
        key_signing = get_hash(key_service, self.request_type)
        return key_signing

    def _create_authorization_header(self, headers, signature_key, string_to_sign, credential_scope):
        signed_headers = create_signed_headers(headers)
        signature = hmac.new(signature_key, string_to_sign.encode('utf-8'), hashlib.sha256).hexdigest()

        return (self.hashing_algorithm + ' ' +
                'Credential=' + self.access_key + '/' + credential_scope + ', ' +
                'SignedHeaders=' + signed_headers + ', ' +
                'Signature=' + signature)

    def _sign(self, http_method, request_path, headers, time, request_parameters=None):
        time_stamp = time.strftime(self.time_format)
        date_stamp = time.strftime(self.date_format)

        credential_scope = self._create_credential_scope(date_stamp)
        canonical_request = self._create_canonical_request(http_method, request_path, request_parameters, headers)
        string_to_sign = self._create_string_to_sign(time_stamp, credential_scope, canonical_request)
        signature_key = self._create_signature_key(date_stamp)

        headers['authorization'] = self._create_authorization_header(headers, signature_key, string_to_sign, credential_scope)

    def put_object(self, bucket_name, object_name, source_file_path, request_parameters=None):
        http_method = 'PUT'

        with open(source_file_path) as f:
            time = datetime.datetime.utcnow()
            time_stamp = time.strftime(self.time_format)

            headers = {'x-amz-date': time_stamp,
                       'x-amz-content-sha256': self.payload_hash,
                       'host': self.host}

            request_path = '/%s/%s' % (bucket_name, object_name)

            self._sign(http_method, request_path, headers, time, request_parameters)

            request_url = self.endpoint + request_path
            r = requests.put(request_url, headers=headers, params=request_parameters, data=f.read())

            print('Response code: %d' % r.status_code)

    def get_object(self, bucket_name, object_name, target_file_path, request_parameters=None):
        http_method = 'GET'

        time = datetime.datetime.utcnow()
        time_stamp = time.strftime(self.time_format)

        headers = {'x-amz-date': time_stamp,
                   'x-amz-content-sha256': self.payload_hash,
                   'host': self.host}

        request_path = '/%s/%s' % (bucket_name, object_name)

        self._sign(http_method, request_path, headers, time, request_parameters)

        request_url = self.endpoint + request_path
        r = requests.get(request_url, headers=headers, params=request_parameters, stream=True)

        print('Response code: %d' % r.status_code)

        if r.status_code == 200:
            with open(target_file_path, 'wb') as f:
                f.write(r.content)

    def list_objects(self, bucket_name, request_parameters=None):
        http_method = 'GET'

        time = datetime.datetime.utcnow()
        time_stamp = time.strftime(self.time_format)

        headers = {'x-amz-date': time_stamp,
                   'x-amz-content-sha256': self.payload_hash,
                   'host': self.host}

        request_path = '/%s' % bucket_name

        self._sign(http_method, request_path, headers, time, request_parameters)

        request_url = self.endpoint + request_path
        r = requests.get(request_url, headers=headers, params=request_parameters)

        print('Response code: %d' % r.status_code)
        print('Response content:\n%s' % r.content)


if __name__ == '__main__':
    sample = ObjectStorageSample()
    sample.put_object('sample-bucket', 'sample-object.txt', '/tmp/source.txt')
    sample.get_object('sample-bucket', 'sample-object.txt', '/tmp/target.txt')
    sample.list_objects('sample-bucket', request_parameters={'max-keys': '10', 'delimiter': '/'})

4. Call

Call the API based on steps 1 to 3. The result of calling the API can be divided into a success or a failure. A successful call returns results, while a failed call returns an error with the corresponding errors. For a failed call, you can check the returned errors and try the call again.

Success

For information on how to handle responses to API calls, refer to the Object Storage API Guide.

Failure

Error codes returned from failed calls include common service error codes and operation-specific errors. Refer to the errors in the Object Storage API Guide for details.