[DataApi] Get BlobCount By AccountID #541
Conversation
siddimore
changed the title
| q.mu.RLock() | ||
| defer q.mu.RUnlock() | ||
| count := int32(0) | ||
| for _, meta := range q.Metadata { |
There was a problem hiding this comment.
This for loop can potentially be really large right?
There was a problem hiding this comment.
true but this is only for testing purpose.....
@ian-shim yes that is correct this is a stop gap solution to get blobCount by AccountID. I have suggested couple more approaches in a document that make it more real-time based on DynamodB streams |
| // GetBlobMetadataByAccount Count returns the count of all the metadata with the given status | ||
| // Because this function scans the entire index, it should only be used for status with a limited number of items. | ||
| // It should only be used to filter "Processing" status. To support other status, a streaming version should be implemented. | ||
| func (s *BlobMetadataStore) GetBlobMetadataCountByAccountID(ctx context.Context, accountID core.AccountID) (int32, error) { |
There was a problem hiding this comment.
We probably want the total amount of data rather than the total number of blobs. Or perhaps both.
Let's do another sync on the exact use case for this work!
There was a problem hiding this comment.
@mooselumph i think if we want total amount of data everytime than better approach is to have a lambda invoke on DynamodB stream.....which only processes INSERT_EVENT and can be used to update:
- Amount of Data
- Count of blobs
There was a problem hiding this comment.
for now temporarily i will just update Batcher to update amount of data after Blob is confirmed and increment count
| // Update AccountID to accountKey | ||
| // This is a combination of origin and authenticatedAddress | ||
| // AccountId is later used to track blobs sent by the same account | ||
| blob.RequestHeader.BlobAuthHeader.AccountID = accountKey |
There was a problem hiding this comment.
This might need to change depending on if the authenticated endpoint is being used right ?
| @@ -244,6 +250,7 @@ func (s *server) Start() error { | |||
| { | |||
| feed.GET("/blobs", s.FetchBlobsHandler) | |||
| feed.GET("/blobs/:blob_key", s.FetchBlobHandler) | |||
| feed.GET("/blobs/count/:accountId", s.FetchBlobCountByAccountIdHandler) | |||
There was a problem hiding this comment.
Should definitely cache these endpoints.
Why are these changes needed?
These changes enable querying blob count by AccountId
Changes
accountIdin BlobAuthHeader.Checks