Using S3 storage

The ocfl-py tools are intended to work with S3-compatible storage by specifying paths with the s3protocol, likes3://bucket/path`. This is implemented through S3Fs which builds on top of aiobotocore and botocore. The best configuration guide for this seems to be the AWS boto3 configuration documentation.

Simple AWS S3 configuration

AWS created S3 and the Python libraries implementing the interface. AWS S3 is the default configuration. The default location for access key information is ~/.aws/config and this can be as simple as:

[default]
aws_access_key_id=ACCESS_KEY
aws_secret_access_key=SECRET_KEY

Where ACCESS_KEY and SECRET_KEY are obtained from the AWS console and given without quotes.

With configuration set, ocfl-py command line tools will work with S3 URLs such as s3://ocfl-fixtures/1.1/good-objects/minimal_no_content given here as an object path:

ocfl-py> ./ocfl-object.py show --objdir s3://ocfl-fixtures/1.1/good-objects/diff_files_same_md5
INFO:aiobotocore.credentials:Credentials found in config file: ~/.aws/config
INFO:root:OCFL v1.1 Object at s3://ocfl-fixtures/1.1/good-objects/diff_files_same_md5 has VALID STRUCTURE (DIGESTS NOT CHECKED)
Object tree
[s3://ocfl-fixtures/1.1/good-objects/diff_files_same_md5]
├── 0=ocfl_object_1.1
├── inventory.json
├── inventory.json.sha512
└── v1
    ├── content (2 files)
    ├── inventory.json
    └── inventory.json.sha512

The bucket name is ocfl-fixtures and the path within the bucket is 1.1/good-objects/minimal_no_content.

S3 API access with other providers

Other vendors support the S3 API and can be configured by adding an endpoint_url parameter in the configuration. For example, to configure for Filebase one might have ~/.aws/config:

[default]
endpoint_url=https://s3.filebase.io
aws_access_key_id=ACCESS_KEY
aws_secret_access_key=SECRET_KEY

Where endpoint_url=https://s3.filebase.io specifies the network location of the Filebase S3 API. ACCESS_KEY and SECRET_KEY are obtained from the provider.

ocfl-py> ./ocfl-object.py show --objdir s3://filebase-fixtures/1.1/good-objects/minimal_no_content
INFO:aiobotocore.credentials:Credentials found in config file: ~/.aws/config
INFO:botocore.configprovider:Found endpoint for s3 via: config_global.
WARNING:root:OCFL v1.1 Object at s3://filebase-fixtures/1.1/good-objects/minimal_no_content has VALID STRUCTURE (DIGESTS NOT CHECKED)
Object tree
[s3://filebase-fixtures/1.1/good-objects/minimal_no_content]
├── 0=ocfl_object_1.1
├── inventory.json
├── inventory.json.sha512
└── v1
    ├── inventory.json
    └── inventory.json.sha512

Where the Filebase bucket name is filebase-fixtures and the path within the bucket is 1.1/good-objects/minimal_no_content.

Configuration with profiles

S3 API access can also be configured to allow multiple sets of credentials and endpoints through the use of profiles in the configuration file ~/.aws/config. For example:

[default]
aws_access_key_id=ACCESS_KEY1
aws_secret_access_key=SECRET_KEY1

[profile filebase]
endpoint_url=https://s3.filebase.io
aws_access_key_id=ACCESS_KEY2
aws_secret_access_key=SECRET_KEY2

The information in the [profile filebase] block is used when a profile=filebase query parameter is added to the S3 path URL:

ocfl-py> ./ocfl-validate.py s3://filebase-fixtures/1.1/good-objects/minimal_no_content?profile=filebase
INFO:aiobotocore.credentials:Credentials found in config file: ~/.aws/config
INFO:botocore.configprovider:Found endpoint for s3 via: config_global.
OCFL v1.1 Object at s3://filebase-fixtures/1.1/good-objects/minimal_no_content?profile=filebase is VALID