Skip to main content

AWS S3

Amazon S3 (Simple Storage Service) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This driver allows you to interact with S3 and S3-compatible storage services ( like MinIO, Ceph) through a KeyValue interface.

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://access_key:secret_key@region/bucket');

The full connection string format:

s3://access_key:secret_key@region/bucket?option1=value1&option2=value2

Example:

s3://AKA12345678899:aaaaaaaaaaaaaaaaaaaaaaaaa@us-east-1/mybucket

Credentials​

The access_key:secret_key@ part is optional:

Connection stringCredentials used
s3://access_key:secret_key@region/bucketThe static key and secret from the URI
s3://access_key@region/bucketThe static key with an empty secret
s3://region/bucketThe AWS SDK default credential provider chain: environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), the ~/.aws files, then the ECS/EC2 instance role

Leave the credentials out of the URI to use an IAM role or the environment.

Connection Options​

You can add any additional parameters supported by the S3 API to the query string. For a comprehensive list, refer to:

Custom Endpoint​

One common parameter is endpoint, which allows you to set a custom endpoint for working with S3-compatible services like MinIO, Ceph, or a local test environment:

s3://access_key:secret_key@us-east-1/mybucket?endpoint=http://localhost:9000

Bucket Creation​

Auto-create Bucket

The library provides a special parameter create that will automatically create the bucket if it doesn't exist:

s3://access_key:secret_key@us-east-1/mybucket?create=true

You can combine multiple parameters:

s3://access_key:secret_key@us-east-1/mybucket?create=true&endpoint=http://localhost:9000

Basic Operations​

List all objects​

List all objects in the bucket:

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$iterator = $s3->getIterator();
print_r($iterator->toArray());

You can filter objects by prefix and set other options:

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$iterator = $s3->getIterator([
    'Prefix' => 'folder/', // List objects with this prefix
    'MaxKeys' => 100       // Maximum number of keys to retrieve
]);
print_r($iterator->toArray());

Inserting/Updating data​

Store an object with a specific key:

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$s3->put("object_name", "value");

You can also specify additional options:

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$s3->put(
    "object_name", 
    "value",
    [
        'ContentType' => 'text/plain',
        'ACL' => 'public-read'
    ]
);

Checking if a key exists​

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
if ($s3->has("object_name")) {
    echo "Object exists!";
}

Retrieving a value​

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$value = $s3->get("object_name");

With options:

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$value = $s3->get("object_name", [
    'ResponseContentType' => 'application/json'
]);

Retrieving portions of a large object​

For large objects, you can retrieve specific chunks to manage memory usage:

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');

// The offset is the chunk number, not a byte position: bytes (offset * size) to (offset * size + size - 1)
// Get the first 1024 bytes (bytes 0-1023)
$chunk1 = $s3->getChunk("object_name", [], 1024, 0);
// Get the next 1024 bytes (bytes 1024-2047)
$chunk2 = $s3->getChunk("object_name", [], 1024, 1);

// Example of reading a large file in chunks
$size = 1024;
$offset = 0;
$data = "";
do {
    $chunk = $s3->getChunk("object_name", [], $size, $offset);
    $data .= $chunk;
    $offset++;
} while (strlen($chunk) == $size);

Removing a value​

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$s3->remove("object_name");

Renaming a key​

<?php
$s3 = \ByJG\AnyDataset\NoSql\Factory::getInstance('s3://....');
$s3->rename("old_key_name", "new_key_name");

Further Reading​

Open source ByJG