1. Home
  2. Forum
  3. ASP.NET Core - EJ 2
  4. How to configure AmasonS3FileProvider to use a Key prefix

How to configure AmasonS3FileProvider to use a Key prefix

Please help me understand how to configure the AmazonS3FileProvider to pick the folder where files live. Within my bucket, I have 2 "folders" (i.e. prefixes), dev​ and prod​, each containing a file tree for FileManager. My app switches between these 2 environments, depending on environment variables at runtime. Under dev & prod, there is a single nested folder named fileservice where I want FileManager files to live.

The S3 bucket is structured like this:

---
|-dev/
| |-fileservice/
| | |-dir1/
| | | |-file1
| | | |-file2
| | | | ...
| | |-dir2/
|
|-prod/
| |-fileservice/
| | |-dir1/
| | |-dir2/
---

I see that GetBucketList() instructs me to "Define the root directory to the file manager", but I have not been able to get it to work. Below seems correct-looking, but doesn't work.

---
{
ListingObjectsAsync("/", "dev/fileservice/", false).Wait();
RootName = response.S3Objects.First().Key;
}

---


I've tried as many permutations as I can think of (with/without slashes, with/without each segment of path, etc), but I can't get it to work.

Help!


3 Replies
SS Sivakumar ShunmugaSundaram Syncfusion Team December 9, 2022 12:59 PM UTC

Hi Matthew,


Greetings from Syncfusion support.


We understand that you want to set the Amazon bucket subfolder as the root folder for the FileManager component. We want to let you know that we do not currently have the ability to set a nested-level folder in an Amazon S3 bucket as the root folder of the FileManager component. However, we have taken note of this request as a potential feature to be implemented in the future. Our team typically plans and implements features based on factors such as customer demand, the volume of requests, and overall prioritization. This feature will be included in any of our upcoming releases.


The status of this feature can be tracked through our feedback portal below.


Feedback: https://www.syncfusion.com/feedback/31885/provide-support-to-set-nested-folder-path-in-amazon-bucket-as-the-root-folder-for


MM Matthew Marichiba December 9, 2022 06:35 PM UTC

Thanks for the reply, Sivakumar. Am I understanding this correctly that:  Every instance of S3-backed FileManager requires a single S3 bucket dedicated exclusively to that instance?

I would appreciate confirmation on this point, so my team can make our engineering decision and move on.

I imagine you're aware that AWS accounts are limited to 100 buckets per account, which means I can't sustainably grow FileManager instances without eventually overshooting our 100 buckets. This adds additional devOps consideration to support AWS credentials & permissions for FileManager. It is desirable to fit multiple FileManager spaces within a single bucket, so they can all be managed consistently. Thank you for the link to the Feedback ticket. I look forward to seeing that feature implemented!


Out of curiosity, what is the prefix​ parameter on the ListingObjectsAsync() method in AmazonS3FileProvider.cs? It sounds like​ exactly the kind of feature I'm asking about. If not, I can't figure out what it's actually for, and unfortunately the code isn't documented.

Thank you!


SS Sivakumar ShunmugaSundaram Syncfusion Team December 14, 2022 12:46 PM UTC

Matthew, Yes, as per our current implementation of the FileManager component, the Amazon file service provider only sets the first available folder in the bucket as the root folder, not its subfolders. This support has been considered a feature.


Also, in File Manager, we were able to retrieve the data in the Amazon S3 bucket from 0-1000 positioned data. After more than 1000 data points, it does not return the child-level data to the Amazon S3 bucket. We have confirmed that this is the default behavior of the Amazon S3 bucket by referring to the below blog.


https://stackoverflow.com/questions/44070580/not-all-data-only-1000-elements-retrieved-from-s3-using-amazon-sdk


However, you can resolve the reported issue by modifying the Amazon S3 Provider, as shown below.

public FileManagerResponse GetFiles(string path, bool showHiddenItems, params FileManagerDirectoryContent[] data)

{

...

 

List<S3Object> s3Object = new List<S3Object>();

GetBucketList();

            try

            {

                ...

                s3Object = response.S3Objects;

                if (path == "/")

                {

                    // specifies the current working directory for above 1000 buckets.

                    cwd = CreateDirectoryContentInstance(RootName.Replace("/", ""), true, "Folder", 0, new DateTime(), new DateTime(), this.checkChild(RootName), string.Empty);

                }

                else

                    cwd = CreateDirectoryContentInstance(path.Split("/")[path.Split("/").Length - 2], false, "Folder", 0, new DateTime(), new DateTime(), (response.CommonPrefixes.Count > 0) ? true : false, path.Substring(0, path.IndexOf(path.Split("/")[path.Split("/").Length - 2])));

            }

            catch (Exception ex) { throw ex; }

...

            try

            {

                if (s3Object.Count > 0)

                    filesS3 = s3Object.Where(x => x.Key != RootName.Replace("/", "") + path).Select(y => CreateDirectoryContentInstance(y.Key.ToString().Replace(RootName.Replace("/", "") + path, "").Replace("/", ""), true, Path.GetExtension(y.Key.ToString()), y.Size, y.LastModified, y.LastModified, this.checkChild(y.Key), getFilterPath(y.Key, path))).ToList();

            }



Loader.
Up arrow icon