Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

colossus s3 api #5150

Open
wants to merge 61 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
bb9a565
feat: :art: env var in order to run content-directory scenario
ignazio-bovo Apr 19, 2024
4fe88e9
feat: :art: create s3 api model
ignazio-bovo Apr 19, 2024
f9544cf
Merge branch 'master' into feat/colossus-s3-api
ignazio-bovo Apr 19, 2024
ccbf56b
feat: :art: refactor filemodel
ignazio-bovo Apr 22, 2024
5d19954
feat: :art: add connection handler abstract in order to conect to blo…
ignazio-bovo Apr 23, 2024
a1aaed0
refactor: :recycle: connection handlers
ignazio-bovo Apr 23, 2024
73a47b0
feat: :art: file cache improvements
ignazio-bovo Apr 24, 2024
336893e
refactor: :art: refactor to accomodate synch task
ignazio-bovo May 7, 2024
4354320
feat: :art: list files functionality
ignazio-bovo May 7, 2024
a37dcd4
feat: :art: improvements on connection handlers
ignazio-bovo May 12, 2024
940b06f
feat: :art: refactor sync task and add task for remote storage provid…
ignazio-bovo May 12, 2024
e697309
feat: :art: renaming the storage provider module and sync/task module…
ignazio-bovo May 12, 2024
61e3ac1
feat: :zap: server setup with remote storage provider connection
ignazio-bovo May 12, 2024
1363c66
feat: :art: add file verification error class
ignazio-bovo May 12, 2024
0df58f1
fix: :fire: add onLocal volume / onCloud Storage flag to data Object …
ignazio-bovo May 13, 2024
19cee7c
feat: :art: improve accepted asset logic to accomodate cloud bucket u…
ignazio-bovo May 13, 2024
3b72202
fix: :art: upload file flow
ignazio-bovo May 13, 2024
99b8ea6
feat: :art: getFile pipe stream
ignazio-bovo May 14, 2024
bcd5b12
feat: :zap: adapt abstractConnectionHandler
ignazio-bovo May 14, 2024
2395a25
feat: :art: improve S3 connection handler
ignazio-bovo May 14, 2024
c05b69a
fix: :bug: list object list ALL objects in the bucket
ignazio-bovo May 14, 2024
1bc8dfc
style: :truck: rename files
ignazio-bovo May 14, 2024
47f47e5
feat: :zap: setup cloud provider config options
ignazio-bovo May 15, 2024
c80488d
fix: :bug: compiler errorrs
ignazio-bovo May 16, 2024
c269c10
feat: :art: utilise redirect url for getFile request
ignazio-bovo May 16, 2024
b45260b
feat: :art: handle multipart uploads
ignazio-bovo May 16, 2024
270ed8f
docs: :memo: update documentation
ignazio-bovo May 16, 2024
e87c578
feat: :art: add functionality for removing object from bucket
ignazio-bovo May 20, 2024
ac4e9e0
test: :art: setup jest integration tests
ignazio-bovo May 21, 2024
3d6d0ea
test: :white_check_mark: finally have a working jest test setup
ignazio-bovo May 22, 2024
978b21f
test: :white_check_mark: add unhappy path for move To accepted location
ignazio-bovo May 22, 2024
f76ebfd
test: :white_check_mark: add test for sync service
ignazio-bovo May 23, 2024
f74cb7d
feat: :art: rewrite file acceptance logic for accept object and sync …
ignazio-bovo May 23, 2024
ae0c8b3
test: :white_check_mark: full green test suite for acceptObject
ignazio-bovo May 23, 2024
b5e45f5
test: :white_check_mark: add for cache setup
ignazio-bovo May 23, 2024
480dbd4
test: :white_check_mark: add for cache setup
ignazio-bovo May 23, 2024
b33c9ad
test: :white_check_mark: add test setup with localstack
ignazio-bovo May 29, 2024
e75df99
fix: :zap: fix startup flag for storage provider
ignazio-bovo May 29, 2024
fc57895
test: :fire: testing
ignazio-bovo May 29, 2024
8ae8d77
Merge branch 'petra' into feat/colossus-s3-api
ignazio-bovo Jun 17, 2024
ae510bc
test: :white_check_mark: setup localstack
ignazio-bovo Jun 17, 2024
81ea8fb
fix: :white_check_mark: integration test setup
ignazio-bovo Jun 19, 2024
4954a82
fix: :bento: env variable setup
ignazio-bovo Jun 19, 2024
eaf1d10
fix: :white_check_mark: localstack integration test setup
ignazio-bovo Jun 19, 2024
613ab3a
fix: :memo: docker compose config
ignazio-bovo Jun 19, 2024
9dfa471
refactor: :recycle: refactor docker compose files
ignazio-bovo Jun 19, 2024
196c48c
Merge branch 'master' into feat/colossus-s3-api
ignazio-bovo Aug 23, 2024
a73f483
fix integration tests
mnaamani Aug 24, 2024
967a893
docker compose instead of docker-compose
mnaamani Aug 24, 2024
5e88241
fix used pip before activating venv
mnaamani Aug 24, 2024
ae4e1a6
revert docker image names used in local testing
mnaamani Aug 24, 2024
334e2ad
Update .env
ignazio-bovo Aug 27, 2024
9892e01
Update storage-node/src/services/webApi/controllers/filesApi.ts
ignazio-bovo Aug 27, 2024
763f77f
Update tests/network-tests/run-tests.sh
ignazio-bovo Aug 27, 2024
889a5f9
Update docker-compose.localstack.yml
ignazio-bovo Aug 27, 2024
5038ae4
Update storage-node/src/services/helpers/acceptObject.ts
ignazio-bovo Aug 27, 2024
5c7bc1a
Merge branch 'feat/colossus-s3-api' into colossus-s3-api-integration-…
ignazio-bovo Aug 28, 2024
6e1f6e0
Merge pull request #17 from mnaamani/colossus-s3-api-integration-test…
ignazio-bovo Aug 28, 2024
ebd0f03
fix: :zap: add test and configuration for existing file uploads
ignazio-bovo Aug 27, 2024
5959615
feat: :art: adding uploadfile if not exisitng test
ignazio-bovo Aug 28, 2024
e3bece8
fix: :art: overall fixes for awsConnectionHandler test and upload wit…
ignazio-bovo Aug 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: ⚡ adapt abstractConnectionHandler
  • Loading branch information
ignazio-bovo committed May 14, 2024
commit bcd5b123191a1393068f07cdfd25d182bc8d8e98
Original file line number Diff line number Diff line change
Expand Up @@ -46,21 +46,13 @@ export abstract class AbstractConnectionHandler {
* @param filename - The key of the file to check in the remote bucket.
* @returns A promise that resolves with a boolean indicating if the file exists or rejects with an error.
*/
abstract isFileOnRemoteBucket(filename: string): Promise<any>
abstract isFileOnRemoteBucket(filename: string): Promise<boolean>

/**
* Asynchronously lists files in the remote bucket.
* @returns A promise that resolves with an array of file keys or rejects with an error.
*/
abstract listFilesOnRemoteBucketAsync(): Promise<string[]>

/**
* Asynchronously move a file in the same bucket accross prefixes
* @param srcKey - The source key of the file.
* @param dstKey - The destination key of the file.
* @returns A promise that resolves when the file is marked as accepted or rejects with an error.
*/
abstract moveFileAsync(srcKey: string, dstKey: string): Promise<void>
abstract listFilesOnRemoteBucket(): Promise<string[]>
}

export type ColossusFileStream = Readable
Expand Down
53 changes: 18 additions & 35 deletions storage-node/src/services/storageProviders/awsConnectionHandler.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
import AWS from 'aws-sdk'
import {
AbstractConnectionHandler,
ColossusFileStream,
StorageProviderGetObjectResponse,
} from './abstractConnectionHandler'
import { BUCKET_ACCEPTED_PREFIX, cloudAcceptedPathForFile, cloudPendingPathForFile } from './const'
import { GetObjectCommand, PutObjectCommand, S3Client } from '@aws-sdk/client-s3'
import { cloudAcceptedPathForFile, cloudPendingPathForFile } from './const'
import { GetObjectCommand, HeadObjectCommand, ListObjectsCommand, PutObjectCommand, S3Client } from '@aws-sdk/client-s3'
import { Readable } from 'stream'

export type AwsConnectionHandlerParams = {
Expand Down Expand Up @@ -45,7 +44,9 @@ export class AwsConnectionHandler extends AbstractConnectionHandler {
// Uploading files to the bucket
const command = new PutObjectCommand(input)
const response = await this.client.send(command)
return response.$metadata
if (response.$metadata.httpStatusCode !== 200) {
throw new Error('Failed to upload file to S3')
}
}

async getFileFromRemoteBucket(filename: string): Promise<StorageProviderGetObjectResponse> {
Expand Down Expand Up @@ -74,44 +75,26 @@ export class AwsConnectionHandler extends AbstractConnectionHandler {
}
}

doListFilesOnRemoteBucket(cb: (err: Error | null, data: string[] | null) => void): void {
async listFilesOnRemoteBucket(): Promise<string[]> {
const input = {
Bucket: this.bucket,
Prefix: BUCKET_ACCEPTED_PREFIX,
}

this.s3.listObjects(input, (err, data) => {
if (err) {
cb(err, null)
} else {
if (data.Contents === undefined) {
cb(Error('No data found'), null)
} else {
const keys = data.Contents.map((content) => content.Key)
.filter((k) => k !== undefined)
.map((k) => k!)
cb(null, keys)
}
}
})
const command = new ListObjectsCommand(input)
const response = await this.client.send(command)
if (!response.Contents) {
throw new Error('Response contents is undefined')
}
const files = response.Contents.filter((f) => f.Key).map((file) => file.Key!)
return files
}

doCheckFileOnRemoteBucket(filename: string, cb: (err: Error | null, objectPresent: boolean) => void): void {
async isFileOnRemoteBucket(filename: string): Promise<boolean> {
const input = {
Bucket: this.bucket,
Key: cloudAcceptedPathForFile(filename),
Key: filename,
}

this.s3.headObject(input, (err, data) => {
if (err) {
cb(err, false)
} else {
if (data === undefined) {
cb(null, false)
} else {
cb(null, true)
}
}
})
const command = new HeadObjectCommand(input)
const response = await this.client.send(command)
return response.$metadata.httpStatusCode === 200
}
}