[email protected] device
S3 object storage in hyperbeam
Device status: WIP MVP
Setup
1- add s3_device.config
in the root level of the hyperbeam codebase, touch s3_device.config
and add the creds to connect to your S3 cluster
connecting to external s3 cluster (./build.sh
)
{endpoint, <<"https://drive.load.network">>}.
{public_endpoint, <<"https://drive.load.network">>}.
{access_key_id, <<"your_access_key_id">>}.
{secret_access_key, <<"your_access_key">>}.
{region, <<"eu-west-2">>}.
connecting to local minio s3 cluster (./s3_device.sh
)
{endpoint, "http://localhost:9001"}.% Internal MinIO - dev
{public_endpoint, "https://your.hyperbeam-s3-cluster-endpoint.com"}. % Public-facing URL, used for presigned URLs
{access_key_id, <<"value">>}.
{secret_access_key, <<"value">>}.
{region, <<"value">>}.
build and run the hyperbeam node
./s3_device.sh # build the s3_nif device & run local minio cluster
# if you want to connect to external s3 cluster, run ./build.sh instead
rebar3 compile
erl -pa _build/default/lib/*/ebin
1> application:ensure_all_started(hb).
configurting the local minio cluster
if you choose the local minio cluster route, you can configure (set) your access key id and secret access key by creating .env file here:
MINIO_ROOT_USER=access_key_id
MINIO_ROOT_PASSWORD=secret_access_key
N.B: your local minio cluster access keys values should be also set equally in the
s3_device.config
config file
Supported methods
create_bucket
head_bucket
put_object
(support expiry: 1-365 days
)
get_object
(support range
)
delete_object
delete_objects
head_object
list_objects
Use the [email protected] device
After running the hyperbeam node with the [email protected]
device, you can use the node_endpoint/[email protected]
url as a S3 compatible API endpoint.
N.B (regarding access authorization): using the
[email protected]
as end user (client) you only have to pass theaccessKeyId
in the request's credentials, andsecretAccessKey
value doesn't matter. This is due to the design of[email protected]
access authorization where the device check's the S3 request's access_key_id of Authorization Header, and validate its parity with the access_key_id defined ins3_device.config
-> Keep theaccess_key_id
secret and use it as access API key.
1- create s3 client
import { S3Client } from "@aws-sdk/client-s3";
const accessKeyId = "your-access-key";
const secretAccessKey = ""; // intentionally empty
const s3Client = new S3Client({
region: "eu-west-2",
endpoint: "http://localhost:8734/[email protected]",
credentials: {
accessKeyId,
secretAccessKey,
},
forcePathStyle: true,
});
2- create bucket
async function createBucket(bucketName) {
try {
const command = new CreateBucketCommand({ Bucket: bucketName });
const result = await s3Client.send(command);
console.log("Bucket created:", result.Location || bucketName);
} catch (error) {
console.error("Error creating bucket:", error);
}
}
3- get object (with range)
async function getObject(bucketName, key) {
try {
console.log(`Getting object: ${bucketName}/${key}`);
const command = new GetObjectCommand({
Bucket: bucketName,
Key: key,
Range: "bytes=-1",
});
const result = await s3Client.send(command);
const bodyContents = await result.Body.transformToString();
console.log("Object retrieved successfully!");
console.log("Content:", bodyContents);
console.log("Metadata:", {
ContentType: result.ContentType,
ContentLength: result.ContentLength,
ETag: result.ETag,
LastModified: result.LastModified,
});
return result;
} catch (error) {
console.error("Error getting object:", error.name, error.message);
throw error;
}
}
4- put object (with expiry)
async function PutObject(bucketName, fileName, body, expiryDays) {
try {
const command = new PutObjectCommand({
Bucket: bucketName,
Key: fileName,
Body: body,
Metadata: {
"expiry-days": expiryDays.toString(),
},
});
const result = await s3Client.send(command);
console.log("Object created:", fileName, "with expiry:", expiryDays);
return result;
} catch (error) {
console.error("Error creating object", error);
}
}
5- Generate presigned get_object url
curl -X POST http://localhost:8734/[email protected]/get-presigned -H "Content-Type: application/json" -H "Authorization: AWS4-HMAC-SHA256 Credential=YOUR-ACCESS-KEY-ID/20230101/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=dummy" -d '{"bucket": "BUCKET-NAME", "key": "OBJECT-KEY", "duration": DURATION_IN_SECONS}' # 1s-7days
The returned URL uses the preset public_endpoint
(in s3_device.config
) as base url.
Cache layer
The NIF implements an LRU cache with size-based eviction (in-memory). The following cache endpoints are available under the hyperbeam http api (intentionally not compatible with the S3 API spec):
1- get cached object
Note: This endpoint requires no authentication
curl "http://localhost:8734/[email protected]/cache/BUCKET_NAME/OBJECT_KEY"
cache vs S3 API
GetObjectCommand
:curl "http://localhost:8734/[email protected]/BUCKET_NAME/OBJECT_KEY"
Hybrid gateway
The hybdrid gateway is an extension to the hb_gateway_client.erl
that makes it possible for the hyperbeam node to retrieve both of onchain (Arweave) posted ANS-104 dataitems, and offchain (in dev_s3.erl
) object-storage temporal dataitems.
Workflow
hb_store_gateway.erl
-> callshb_gateway_client:read()
-> it tries to read from local cache then Arweave (onchain dataitem) -> incase not found onchain, check the offchain dataitems s3 bucketoffchain dataitems retrieval route :
hb_gateway_s3:read()
-> callsdev_s3:handle_s3_request()
-> retrieve the dataitem.asn104 from thedev_s3
bucket
Test it
1- create & set the offchain bucket
Make hyperbeam aware of the dev_s3
bucket that is storing your ANS-104 offchain dataiems, here (s3_bucket
in default_message
)
s3_bucket => <<"offchain-dataitems">> % you can change the name
2- add test data
Make sure to create the [email protected]
bucket as you defined the name in hb_opts.erl
then add a fake offchain dataitem. If you want to test using existing signed offchain ans-104 dataitems, checkout the test-dataitems directory and store it in your [email protected]
bucket.
curl -X PUT "http://localhost:8734/[email protected]/offchain-dataitems/dataitems/ysAGgm6JngmOAxmaFN2YJr5t7V1JH8JGZHe1942mPbA.ans104" \
-H "Content-Type: application/octet-stream" \
-H "Authorization: AWS4-HMAC-SHA256 Credential=YOUR_ACCESS_KEY_ID/20250119/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=dummy" \
--data-binary @ysAGgm6JngmOAxmaFN2YJr5t7V1JH8JGZHe1942mPbA.ans104
Otherwise, you can generate a signed valid ANS-104 dataitem using the hyperbeam erlang shell:
1> rr("src/ar_tx.erl"). % load tx record definition
2> TX = #tx{data = <<"Hello Load S3">>, tags = [{<<"Content-Type">>, <<"text/plain">>}], format = ans104}.
3> SignedTX = ar_bundles:sign_item(TX, hb:wallet()).
4> ANS104Binary = ar_bundles:serialize(SignedTX).
5> DataItemID = hb_util:encode(hb_tx:id(SignedTX, signed)).
6> file:write_file("TheDataItemId.ans104", ANS104Binary).
% After that, store the dataitem on s3 as we did previously or using the s3 sdk/client of your choice.
3- test retrieving dataitems
offchain:
http://localhost:8734/ysAGgm6JngmOAxmaFN2YJr5t7V1JH8JGZHe1942mPbA
onchain:
http://localhost:8734/myb2p8_TSM0KSgBMoG-nu6TLuqWwPmdZM5V2QSUeNmM
Verify dataitem integrity
curl -I "http://localhost:8734/ysAGgm6JngmOAxmaFN2YJr5t7V1JH8JGZHe1942mPbA"
Expected Response:
HTTP/1.1 200 OK
Content-Type: text/plain
Data-Protocol: ao
Device-Test: [email protected]
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
access-control-allow-origin: *
access-control-expose-headers: *
ao-body-key: data
ao-types: status="integer"
content-digest: sha-256=:9WV5g4vLNkDev2JHfurtl5OE4XrCfKwjf4zE9SceHyg=:
date: Fri, 18 Jul 2025 15:24:11 GMT
id: ysAGgm6JngmOAxmaFN2YJr5t7V1JH8JGZHe1942mPbA
owner: q4VxGIPsOoT2ce5OsF0eSErdxuMtFexE9tUvY_Gl1tE-w3p6YRy6REuc4t2gDFUFE233PP8l5B-db6a2IzRw8FrEc7eFeu_-sWi-FTnkh3EQ3ExE6D9VpBaocSwlPJmVXGMfC64kkxy1hrh44qpQl-RwI52IP15J5YTLWN_XOzGqPCL94VPRFQtwhiK2FQkAx1iCDCtC_FUWC9CitUnNygTUAt2X5I1oD_e9zoyWyUuEp14TLM0-JDnBzGW1t0BbZZKUw8nvmjkqyErQXOHU4AbSevp7rmb3kmi0qFEqb85flF11sHvl1ABJ9i84cmYOM4Az87Gw5beVdzIwe_1tnlUOdX42-skOuNwNPoSOOrUOXh78_meoHCWk5iwYXCnFIWOdlXl-i9Ts2MCf1Ub0v7UPeLT4mtbdhRyG6iK6nokFGHs6A5t1nce0ItGAO1wpBs_4zK3qwfxKvNwoIHpJARyBof8IKnrr28-RpkNJyhVCRvNueUusANnWNk8zIjWseNF3zLg2w_IxZKrDb7a7u1RDQGHSxDvX8mHNHZKAUcqUVeQau8pyfOcDw7hRPKLPkcoCv28ZusAeS0hibdIXA0CJ0HXzleNLIJhCBGwEmo_n1Fa1_hIEekGKnztkNwtLbhyfLtFuqbT6o_r9LdQ81glhAccc-_OJeTvG-fsYD3s
server: Cowboy
signature: sig-ahm9fs6tg1al3sq0w-ttpxaba2ztt2by1xnq1f4ih6w=:HGS0k2orTq+gA8t7zNx3ykTwn/lyagXlIB3yWX9AGu+yrLK1klrXo2mbySD3Kglmxw2Wyolj42CiGx4yYqAtvhN9AtS3HwVqf47E2FhmcQYk6dcfdAxaJcEz1lXncSyDAAwT7R/ecqJHHNWqOGHiP/9I9V4gCH6qz+kxXI6y/Z5Nnf9IWuJ8xv8XY37TICHlU8oPnPuLPFuvOVPCUXrzCxR6jP8JaVmx5kAHhMMC60+3CDKc3auoe9uxhMQlshnCShD1J/EzycwVqbfXV7TprqVRJoox/Z9EYdd36QKe303dgb3hO/s2aH/Z8TwLJEz6X0c9Miyt31pRQK+31Ev9QwAeXUbE48KljS/knfpTvNA0gjNIQ6xoz9paLWoHhZ+Ntzp/6fC0amzRZcXD1d5cZ5wxRUp1PUSnkexdi73adYV6UKL9pG4NRtDXpx1rP8H33qV5a7JKnTUrcg7TTuwgtieoB+ZKgAGPPtZUYsTRehSkW3gHFujpO+DsNko3UfhveXk5FpONKm2J+22LoOmLp13/yHdiRoHOIr+W/iPD7ueM3iSxtJFO68Wx6qIWp5PV0L+/trU/nHnDYxCgFNE/lZQvINeBSR8mKfE6ws1MJqoGdOC4ZdcRK11Bk75bjlWZUoz4r9JJGQgaA92xwqGzpWRQzAjGWs4rxhCib8YXLqE=:, sig-iq1nxhxgfzszd_rdcs-qwa_csljm7qwvdxmjr3mca7g=:f4/xQHH6IHbhDg3xIGKd7IRIXmwsrUd4a8XaBmKIRco=:
signature-input: sig-ahm9fs6tg1al3sq0w-ttpxaba2ztt2by1xnq1f4ih6w=("Content-Type" "Data-Protocol" "Device-Test" "ao-body-key" "ao-types" "content-digest" "id" "owner" "signature" "signature-input" "status" "tags+link");alg="rsa-pss-sha512";keyid="q4VxGIPsOoT2ce5OsF0eSErdxuMtFexE9tUvY/Gl1tE+w3p6YRy6REuc4t2gDFUFE233PP8l5B+db6a2IzRw8FrEc7eFeu/+sWi+FTnkh3EQ3ExE6D9VpBaocSwlPJmVXGMfC64kkxy1hrh44qpQl+RwI52IP15J5YTLWN/XOzGqPCL94VPRFQtwhiK2FQkAx1iCDCtC/FUWC9CitUnNygTUAt2X5I1oD/e9zoyWyUuEp14TLM0+JDnBzGW1t0BbZZKUw8nvmjkqyErQXOHU4AbSevp7rmb3kmi0qFEqb85flF11sHvl1ABJ9i84cmYOM4Az87Gw5beVdzIwe/1tnlUOdX42+skOuNwNPoSOOrUOXh78/meoHCWk5iwYXCnFIWOdlXl+i9Ts2MCf1Ub0v7UPeLT4mtbdhRyG6iK6nokFGHs6A5t1nce0ItGAO1wpBs/4zK3qwfxKvNwoIHpJARyBof8IKnrr28+RpkNJyhVCRvNueUusANnWNk8zIjWseNF3zLg2w/IxZKrDb7a7u1RDQGHSxDvX8mHNHZKAUcqUVeQau8pyfOcDw7hRPKLPkcoCv28ZusAeS0hibdIXA0CJ0HXzleNLIJhCBGwEmo/n1Fa1/hIEekGKnztkNwtLbhyfLtFuqbT6o/r9LdQ81glhAccc+/OJeTvG+fsYD3s=", sig-iq1nxhxgfzszd_rdcs-qwa_csljm7qwvdxmjr3mca7g=("Content-Type" "Data-Protocol" "Device-Test" "ao-body-key" "ao-types" "content-digest" "id" "owner" "signature" "signature-input" "status" "tags+link");alg="hmac-sha256";keyid="ao"
status: 200
tags+link: 24Xx7HqIQkRzm3CtNQxfht5RbHTz12N1Ihwm1B0IIFE
transfer-encoding: chunked
For more examples & ao processes interactions, check the test-dataitems directory
Source code: https://github.com/loadnetwork/load_hb/tree/s3-edge/native/s3_nif
Last updated