AWS related utilities that are compatible with both AWS Javascript SDK v3 and v2, and also some utilities that require v3.
First add it as a dependency:
npm install @handy-common-utils/aws-utils
Then you can use it in the code:
import { AwsUtils } from '@handy-common-utils/aws-utils';
const domainNameObjects = await AwsUtils.repeatFetchingItemsByPosition(
pagingParam => apig.getDomainNames({ limit: 100, ...pagingParam }).promise(),
);
You can either import and use the class as shown above, or you can import individual functions directly like below:
import { repeatFetchingItemsByNextToken, repeatFetchingItemsByMarker, parseArn } from '@handy-common-utils/aws-utils';
S3 related utility functions can be imported and used in this way:
import { S3Client } from '@aws-sdk/client-s3';
import { decodeS3ObjectKey, deleteS3Object } from '@handy-common-utils/aws-utils/s3';
const srcEncodedKey = record.s3.object.key;
const srcKey = decodeS3ObjectKey(srcEncodedKey);
const destKey = srcKey.replace('/src-dir/', '/dest-dir/')
const s3 = new S3Client();
await copyS3Object(s3, bucket, srcEncodedKey, destKey);
await deleteS3Object(s3, bucket, srcKey);
To use S3 related utilities, you need to add @aws-sdk/client-s3
as a dependency of your project
because it is not included as a dependency of this package.
SSM related utility functions can be imported and used in this way:
import { SSM } from '@aws-sdk/client-ssm';
import { getSsmParameter, getSsmParameterParsed } from '@handy-common-utils/aws-utils/ssm';
const ssm = new SSM();
const workDir = await getSsmParameter(ssm, '/my-config/work-dir', '/tmp');
const config = await getSsmParameterParsed<Config>(ssm, '/my-config/config');
To use SSM related utilities, you need to add @aws-sdk/client-ssm
as a dependency of your project
because it is not included as a dependency of this package.
aws-utils.AwsUtils
• new AwsUtils()
▸ Static
dynamodbLocalClientOptions(endpoint?
): Object
Build an object that can be passed into DynamoDB.DocumentClient(...)
for
DynamoDB Local (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html)
Name | Type | Default value | Description |
---|---|---|---|
endpoint |
string |
'http://localhost:8000' |
if omitted, the endpoint will be 'http://localhost:8000' which is the default |
Object
the options object
Name | Type |
---|---|
accessKeyId |
string |
endpoint |
string |
region |
string |
secretAccessKey |
string |
Example
const ddbClient = new DynamoDB.DocumentClient(process.env.IS_OFFLINE === 'true' ? AwsUtils.dynamodbLocalClientOptions() : undefined);
▸ Static
fetchAllByContinuationToken<T
, M
>(fetchItemsByContinuationToken
, itemsFieldName?
): Promise
<T
[]>
Fetch all items through repeatedly calling API with ContinuationToken/NextContinuationToken based pagination. This function is useful for client side pagination when the response from AWS API contains NextContinuationToken and items fields.
Name | Type | Description |
---|---|---|
T |
T |
type of the items returned by AWS API |
M |
string |
- |
Name | Type | Default value | Description |
---|---|---|---|
fetchItemsByContinuationToken |
FetchItemsFunction <{ ContinuationToken? : M }, { NextContinuationToken? : M }> |
undefined |
the function for fetching one batch/page of items by ContinuationToken |
itemsFieldName |
string |
'Contents' |
name of the field containing returned items in AWS API response |
Promise
<T
[]>
all items fetched
Example
const objects = await fetchAllByContinuationToken(() => s3.send(new ListObjectsV2Command({Bucket: bucket})));
▸ Static
fetchAllByExclusiveStartKey<T
, K
>(fetchItemsByExclusiveStartKey
, itemsFieldName?
): Promise
<T
[]>
Fetch all items through repeatedly calling API with ExclusiveStartKey/LastEvaluatedKey based pagination. This function is useful for client side pagination when the response from AWS API contains LastEvaluatedKey and items fields.
Name | Type | Description |
---|---|---|
T |
T |
type of the items returned by AWS API |
K |
{ [key: string] : any ; } |
- |
Name | Type | Default value | Description |
---|---|---|---|
fetchItemsByExclusiveStartKey |
FetchItemsFunction <{ ExclusiveStartKey? : K }, { LastEvaluatedKey? : K }> |
undefined |
the function for fetching one batch/page of items by ExclusiveStartKey |
itemsFieldName |
string |
'Items' |
name of the field containing returned items in AWS API response, the default value is 'Items' |
Promise
<T
[]>
all items fetched
Example
const allItemsInDynamoDbTable = await AwsUtils.fetchAllByExclusiveStartKey<MyTableItem>(
pagingParam => dynamoDbDocumentClient.scan({...pagingParam, TableName: 'my-table', limit: 20}).promise(),
);
▸ Static
fetchAllByMarker<T
, M
>(fetchItemsByMarker
, itemsFieldName
): Promise
<T
[]>
Fetch all items through repeatedly calling API with Marker/NextMarker based pagination. This function is useful for client side pagination when the response from AWS API contains NextMarker and items fields.
Name | Type | Description |
---|---|---|
T |
T |
type of the items returned by AWS API |
M |
string |
- |
Name | Type | Description |
---|---|---|
fetchItemsByMarker |
FetchItemsFunction <{ Marker? : M }, { NextMarker? : M }> |
the function for fetching one batch/page of items by Marker |
itemsFieldName |
string |
name of the field containing returned items in AWS API response |
Promise
<T
[]>
all items fetched
Example
const functionConfigurations = await AwsUtils.fetchAllByMarker<Lambda.FunctionConfiguration>(
pagingParam => withRetry(() => lambda.listFunctions({ ...pagingParam }).promise()),
'Functions',
);
▸ Static
fetchAllByNextToken<T
, K
>(fetchItemsByNextToken
, itemsFieldName
): Promise
<T
[]>
Fetch all items through repeatedly calling API with NextToken based pagination. This function is useful for client side pagination when the response from AWS API contains NextToken and items fields.
Name | Type | Description |
---|---|---|
T |
T |
type of the items returned by AWS API |
K |
string |
- |
Name | Type | Description |
---|---|---|
fetchItemsByNextToken |
FetchItemsFunction <{ NextToken? : K }, { NextToken? : K }> |
the function for fetching one batch/page of items by NextToken |
itemsFieldName |
string |
name of the field containing returned items in AWS API response |
Promise
<T
[]>
all items fetched
Example
const topics = await AwsUtils.fetchAllByNextToken<SNS.Topic>(
pagingParam => sns.listTopics({...pagingParam}).promise(),
'Topics',
);
▸ Static
fetchAllByNextTokenV3<T
, K
>(fetchItemsByNextToken
, itemsFieldName
): Promise
<T
[]>
Fetch all items through repeatedly calling API with nextToken based pagination which is used in aws-sdk v3. This function is useful for client side pagination when the response from AWS API contains nextToken and items fields.
Name | Type | Description |
---|---|---|
T |
T |
type of the items returned by AWS API |
K |
string |
- |
Name | Type | Description |
---|---|---|
fetchItemsByNextToken |
FetchItemsFunction <{ nextToken? : K }, { nextToken? : K }> |
the function for fetching one batch/page of items by nextToken |
itemsFieldName |
string |
name of the field containing returned items in AWS API response |
Promise
<T
[]>
all items fetched
Example
const executions = await AwsUtils.fetchAllByNextTokenV3<ExecutionListItem>(
(pagingParam) => this.client.send(new ListExecutionsCommand({
stateMachineArn,
statusFilter: status,
...pagingParam,
})),
'executions',
);
▸ Static
fetchAllByPosition<T
, P
>(fetchItemsByPosition
, itemsFieldName?
): Promise
<T
[]>
Fetch all items through repeatedly calling API with position based pagination. This function is useful for client side pagination when the response from AWS API contains position and items fields.
Name | Type | Description |
---|---|---|
T |
T |
type of the items returned by AWS API |
P |
string |
- |
Name | Type | Default value | Description |
---|---|---|---|
fetchItemsByPosition |
FetchItemsFunction <{ position? : P }, { position? : P }> |
undefined |
the function for fetching one batch/page of items by position |
itemsFieldName |
string |
'items' |
name of the field containing returned items in AWS API response, default value is 'items' |
Promise
<T
[]>
all items fetched
Example
const domainNameObjects = await AwsUtils.fetchingAllByPosition(
pagingParam => apig.getDomainNames({limit: 500, ...pagingParam}).promise(),
);
▸ Static
fetchAllWithPagination<IT
, RT
, IFN
, PFN
, PFT
>(fetchOnePageOfItems
, itemsFieldName
, paginationFieldName
, shouldFetchNextPage?
): Promise
<Exclude
<RT
[IFN
], undefined
>>
Fetch all items through repeatedly calling pagination based API. This function is useful for client side pagination when the calling AWS API.
Name | Type | Description |
---|---|---|
IT |
IT |
type of the items returned by AWS API |
RT |
extends Record <IFN , undefined | IT []> & Partial <Record <PFN , PFT >> |
type of the response returned by AWS API |
IFN |
extends string
|
name of the field containing returned items in AWS API response |
PFN |
extends string
|
name of the field containing the pagination token in AWS API response, such like "ExclusiveStartKey", "Marker", "NextToken", "nextToken" |
PFT |
string |
type of the pagination token in AWS API response, usually it is string |
Name | Type | Description |
---|---|---|
fetchOnePageOfItems |
FetchItemsFunction <Partial <Record <PFN , PFT >>, RT > |
the function for fetching one batch/page of items by nextToken |
itemsFieldName |
IFN |
name of the field containing returned items in AWS API response |
paginationFieldName |
PFN |
name of the field containing the pagination token in AWS API response, such like "ExclusiveStartKey", "Marker", "NextToken", "nextToken" |
shouldFetchNextPage? |
(response : RT ) => boolean
|
a function to determine if the fetch should continue, the default value is always true and will continue fetching items until the response does not contain nextToken field. |
Promise
<Exclude
<RT
[IFN
], undefined
>>
all items fetched
Example
const executions = await AwsUtils.fetchAllWithPagination(
(pagingParam) => this.client.send(new ListExecutionsCommand({
stateMachineArn,
statusFilter: status,
...pagingParam,
})),
'executions',
'nextToken',
(resp) => resp.executions != null && resp.executions.some((exe) => exe.startDate!.getTime() <= beginTime),
);
▸ Static
fibonacciRetryConfigurationOptions(maxRetries
, base?
): PartialConfigurationOptions
Generate part of a ConfigurationOptions object having maxRetries as specified and a custom RetryDelayOptions for fibonacci sequence based retry delays.
Name | Type | Default value | Description |
---|---|---|---|
maxRetries |
number |
undefined |
The maximum amount of retries to perform for a service request. |
base |
number |
100 |
The base number of milliseconds to use in the fibonacci backoff for operation retries. Defaults to 100 ms. |
PartialConfigurationOptions
part of a ConfigurationOptions object that has maxRetries as specified and a customBackoff utilising fibonacci sequence for calculating delays
▸ Static
parseArn(arn
): undefined
| null
| Arn
& { arn
: string
}
Parse ARN
Name | Type | Description |
---|---|---|
arn |
undefined | null | string
|
the ARN string that could be null or undefined |
undefined
| null
| Arn
& { arn
: string
}
null or undefined if the input is null or undefined, or parsed ARN including the original ARN string
▸ Static
promiseWithRetry<Result
, TError
>(operation
, backoff?
, statusCodes?
): Promise
<Result
>
Perform an AWS operation (returning a Request) with retry.
This function is quite handy when you are using AWS SDK v2.
If you are using AWS SDK v3, use withRetry(...)
instead.
The retry could happen only if the error coming from AWS has property retryable
/$retryable
equals to true.
If you don't want retryable
/$retryable
property to be checked, use PromiseUtils.withRetry(...)
directly.
Name | Type |
---|---|
Result |
Result |
TError |
any |
Name | Type | Description |
---|---|---|
operation |
(attempt : number , previousResult : undefined | Result , previousError : undefined | TError ) => WithPromiseFunction <Result > |
the AWS operation that returns a Request, such like () => apig.getBasePathMappings({ domainName, limit: 500 })
|
backoff? |
number [] | (attempt : number , previousResult : undefined | Result , previousError : undefined | TError ) => undefined | number
|
Array of retry backoff periods (unit: milliseconds) or function for calculating them. If retry is desired, before making next call to the operation the desired backoff period would be waited. If the array runs out of elements or the function returns undefined , there would be no further call to the operation. The attempt argument passed into backoff function starts from 2 because only retries need to backoff, so the first retry is the second attempt. If omitted or undefined, a default backoff array will be used. In case AWS has retryDelay property in the returned error, the larger one between retryDelay and the backoff will be used. |
statusCodes? |
null | (undefined | number )[] |
Array of status codes for which retry should be done. If omitted or undefined, only 429 status code would result in a retry. If it is null, status code would not be looked into. If it is an empty array, retry would never happen. |
Promise
<Result
>
result came out from the last attempt
▸ Static
withRetry<Result
, TError
>(operation
, backoff?
, statusCodes?
): Promise
<Result
>
Perform an AWS operation (returning a Promise) with retry.
This function is quite handy when you are using AWS SDK v3.
If you are using AWS SDK v2, promiseWithRetry(...)
could be more convenient.
The retry could happen only if the error coming from AWS has property retryable
/$retryable
equals to true.
If you don't want retryable
/$retryable
property to be checked, use PromiseUtils.withRetry(...)
directly.
Name | Type |
---|---|
Result |
Result |
TError |
any |
Name | Type | Description |
---|---|---|
operation |
(attempt : number , previousResult : undefined | Result , previousError : undefined | TError ) => Promise <Result > |
the AWS operation that returns a Promise, such like () => apig.getBasePathMappings({ domainName, limit: 500 }).promise()
|
backoff |
number [] | (attempt : number , previousResult : undefined | Result , previousError : undefined | TError ) => undefined | number
|
Array of retry backoff periods (unit: milliseconds) or function for calculating them. If retry is desired, before making next call to the operation the desired backoff period would be waited. If the array runs out of elements or the function returns undefined or either the array or the function returns a negative number, there would be no further call to the operation. The attempt argument passed into backoff function starts from 2 because only retries need to backoff, so the first retry is the second attempt. If omitted or undefined, a default backoff array will be used. In case AWS has retryDelay property in the returned error, the larger one between retryDelay and the backoff will be used. |
statusCodes |
null | (undefined | number )[] |
Array of status codes for which retry should be done. If omitted or undefined, only 429 status code would result in a retry. If it is null, status code would not be looked into. If it is an empty array, retry would never happen. |
Promise
<Result
>
result came out from the last attempt
See
promiseWithRetry
s3.S3ObjectSummary
Property | Description |
---|---|
ETag: string
|
|
Key: string
|
|
LastModified: Date
|
|
Size: number
|
Size of the object, it could be zero for directory objects |
StorageClass: string
|
## Modules |
- fetchAllByNextToken = AwsUtils.fetchAllByNextToken
- fetchAllByNextTokenV3 = AwsUtils.fetchAllByNextTokenV3
- fetchAllWithPagination = AwsUtils.fetchAllWithPagination
- fetchAllByMarker = AwsUtils.fetchAllByMarker
- fetchAllByExclusiveStartKey = AwsUtils.fetchAllByExclusiveStartKey
- withRetry = AwsUtils.withRetry
- promiseWithRetry = AwsUtils.promiseWithRetry
- fibonacciRetryConfigurationOptions = AwsUtils.fibonacciRetryConfigurationOptions
- parseArn = AwsUtils.parseArn
- dynamodbLocalClientOptions = AwsUtils.dynamodbLocalClientOptions
Ƭ PossibleAwsError: Object
An error that is possibly generated by AWS SDK v2 or v3
Name | Type | Description |
---|---|---|
$metadata? |
{ httpStatusCode : number } |
v3 $metadata |
$metadata.httpStatusCode |
number |
- |
$retryable? |
{ throttling : boolean } |
v3 $retryable |
$retryable.throttling |
boolean |
- |
code? |
string |
v2 code |
message |
string |
- |
name? |
string |
v3 name |
retryDelay? |
number |
v2 retryDelay |
retryable? |
boolean |
v2 retryable |
statusCode? |
number |
v2 statusCode |
• Const
FIBONACCI_SEQUENCE_BACKOFFS: number
[]
▸ copyS3Object(s3
, srcBucket
, srcEncodedKey
, destDecodedKey
, metadata?
, destBucket?
): Promise
<CopyObjectCommandOutput
>
Copy S3 object
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
srcBucket |
string |
bucket of the source object |
srcEncodedKey |
string |
key (URL encoded) of the source object |
destDecodedKey |
string |
key (NOT URL encoded) of the destination object |
metadata? |
Record <string , string > |
metadata to be set on the destination object, if it is not specified then the metadata from source object will be copied |
destBucket? |
string |
bucket of the destination object, if it is not specified then the source bucket will be used |
Promise
<CopyObjectCommandOutput
>
S3 command output
▸ decodeS3ObjectKey(key
): string
Decode the raw object key which is URL encoded and could contain "+" as replacement of " "
Name | Type | Description |
---|---|---|
key |
string |
The raw S3 object key which is URL encoded |
string
Decoded key
▸ deleteS3Object(s3
, bucket
, key
): Promise
<DeleteObjectCommandOutput
>
Delete an S3 object. No error would be thrown if the object does not exist.
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
bucket |
string |
bucket name |
key |
string |
object key (without URL encoding) |
Promise
<DeleteObjectCommandOutput
>
S3 command output
▸ encodeS3ObjectKey(key
): string
URL encode the object key, and also replace "%20" with " " and "%2F with "/" which is the convention of AWS
Name | Type | Description |
---|---|---|
key |
string |
The S3 object key before encoding |
string
URL encoded object key
▸ generatePresignedUrlForDownloading(s3
, bucket
, key
, expiresIn
, options?
): Promise
<string
>
Generate a pre-signed URL for downloading the S3 object
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
bucket |
string |
Name of the bucket |
key |
string |
Key of the object |
expiresIn |
number |
The number of seconds before the presigned URL expires |
options? |
Omit <GetObjectCommandInput , "Bucket" | "Key" > |
Additional options. For example, you can specify content-disposition and content-type in it. |
Promise
<string
>
An URL that can be used to download the S3 object.
▸ generatePresignedUrlForUploading(s3
, bucket
, key
, expiresIn
): Promise
<string
>
Generate a pre-signed URL for uploading content to the S3 object
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
bucket |
string |
Name of the bucket |
key |
string |
Key of the object |
expiresIn |
number |
The number of seconds before the presigned URL expires |
Promise
<string
>
An URL that can be used to upload content to the S3 object.
▸ getS3ObjectContentByteArray(s3
, bucket
, key
, range?
): Promise
<Uint8Array
| undefined
>
Get the content of the S3 object as a Uint8Array.
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
bucket |
string |
bucket of the source object |
key |
string |
object key (without URL encoding) |
range? |
string |
See https://www.rfc-editor.org/rfc/rfc9110.html#name-range |
Promise
<Uint8Array
| undefined
>
Content of the S3 object as a Uint8Array.
If the object does not have content, an empty Uint8Array will be returned.
If the object does not exist, undefined
will be returned.
▸ getS3ObjectContentString(s3
, bucket
, key
, encoding?
): Promise
<string
| undefined
>
Get the content of the S3 object as a string.
Name | Type | Default value | Description |
---|---|---|---|
s3 |
S3Client |
undefined |
S3Client |
bucket |
string |
undefined |
bucket of the source object |
key |
string |
undefined |
object key (without URL encoding) |
encoding |
string |
'utf8' |
Text encoding of the content, if not specified then "utf8" will be used |
Promise
<string
| undefined
>
Content of the S3 object as a string.
If the object does not have content, an empty string will be returned.
If the object does not exist, undefined
will be returned.
▸ headS3Object(s3
, bucket
, key
): Promise
<HeadObjectCommandOutput
| undefined
>
Get details of the S3 object without downloading its content.
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
bucket |
string |
bucket of the source object |
key |
string |
object key (without URL encoding) |
Promise
<HeadObjectCommandOutput
| undefined
>
S3 command output, or undefined
if the object does not exist.
▸ putS3Object(s3
, bucket
, key
, content
, options?
): Promise
<PutObjectOutput
>
Store content into S3.
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
bucket |
string |
Name of the bucket |
key |
string |
Key of the object |
content |
undefined | StreamingBlobPayloadInputTypes
|
Content of the object, or undefined if the object is a directory. |
options? |
Partial <PutObjectCommandInput > |
Additional options |
Promise
<PutObjectOutput
>
PutObjectOutput
▸ scanS3Bucket(s3
, bucket
, options?
): Promise
<S3ObjectSummary
[]>
Scan S3 bucket and return both normal objects and directory objects. Directory objects have keys ending with '/'. This function handles pagination automatically.
Name | Type | Description |
---|---|---|
s3 |
S3Client |
S3Client |
bucket |
string |
Name of the bucket |
options? |
Partial <ListObjectsV2CommandInput > |
Optional settings for the scan |
Promise
<S3ObjectSummary
[]>
Array of normal and directory objects found
▸ getSsmParameter(ssm
, parameterName
, fallbackValue?
): Promise
<string
| undefined
>
Get a parameter from SSM Parameter Store.
If the parameter is not found and a fallbackValue is given, the fallbackValue will be returned.
If the parameter is not found and there is no fallbackValue given, undefined
will be returned.
If any other error occurs, the error will be thrown.
Name | Type | Description |
---|---|---|
ssm |
SSM |
SSM |
parameterName |
string |
Name/path of the parameter to get from SSM Parameter Store. |
fallbackValue? |
string |
the value to return if the parameter is not found. If not given, undefined will be returned when the parameter is not found. |
Promise
<string
| undefined
>
Value of the parameter, or the fallbackValue, or undefined
▸ getSsmParameterParsed<T
>(ssm
, parameterName
, fallbackValue?
): Promise
<T
| undefined
>
Get a parameter from SSM Parameter Store and return it parsed as JSON.
If the parameter is not found and a fallbackValue is given, the fallbackValue will be returned.
If the parameter is not found and there is no fallbackValue given, undefined
will be returned.
If any other error occurs, the error will be thrown.
Name |
---|
T |
Name | Type | Description |
---|---|---|
ssm |
SSM |
SSM |
parameterName |
string |
Name/path of the parameter to get from SSM Parameter Store. |
fallbackValue? |
T |
the value to return if the parameter is not found. If not given, undefined will be returned when the parameter is not found. |
Promise
<T
| undefined
>
Value of the parameter parsed as JSON, or the fallbackValue, or undefined