Amazon S3 filemanager can add new folders but cannot view (read) them.

Question

I have successfully connected using FileManager to an S3 Bucket using EJ2AmazonS3ASPCoreFileProvider, but am having some problems:

1. If I connect to a S3 Bucket with pre-existing folders at the root of the S3 bucket, they do not display in the FileManager.

2. If there a file (PDF in this case) at the root level, but not within a folder on S3 it will show in the FileManager.

3. I am able to add new folders using FileManager, but after adding them they do not display in FileManager. I can verify that they have been added in AWS Control Panel.and they are there.

So the fundamental problem is that I cannot view folders that are in the S3 bucket. I have tried various tweaks with the path, etc. but no luck. Not sure is some bug, or just me not doing this correctly.

Also I note that the demo FileManager Aws S3 on your website does not work.

This looks like a very powerful Control that could save a programmer a lot of hand coding! I sure hope to get it working with AWS.

Sowmiya Padmanaban · Accepted Answer

Hi Tom McEwen,  
 
Greetings from Syncfusion support. 
 
Query 1 & Query 2 - If I connect to a S3 Bucket with pre-existing folders at the root of the S3 bucket, they do not display in the FileManager. && . If there a file (PDF in this case) at the root level, but not within a folder on S3 it will show in the FileManager. 
 
FileManager does not contain multiple root folder support. When you are expecting to show all four folders ( Files, Folder1, Folder2, Folder3) within the FileManager which is available in Amazon S3 bucket. 
 
Only the files/folders from one(first) folder of the Amazon S3 Bucket will be accessed and loaded in the UI by the FileManager component. By default, in our service application, we will take the first folder from S3 Bucket and make it as root folder (RootFolder). 
 
File Manager will consider that first folder as root folder and access all the files and folders within that folder. This is the reason why, FileManager displays only the Files folder. 
 
In order to meet your requirement, we should add all the folders (Files, Folder1, Folder2, Folder3) within a parent folder. Then, File Manager will consider the parent folder as it root folder and display all it first level siblings in the UI. 
 
Note: In the service application, we consider first folder of the S3 bucket as root folder. Amazon S3 bucket storage system sorts its folder objects in alphabetic(ascending) order. So, if you wants to set any folder other than the first folder of that bucket, then you can alter the following code to achieve the requirement. 
 
https://github.com/SyncfusionExamples/ej2-amazon-s3-aspcore-file-provider/blob/master/Models/AmazonS3FileProvider.cs#L42  
 




//Define the root directory to the file manager 
public void GetBucketList() { 
            ListingObjectsAsync("", "", false).Wait(); 
            RootName = response.S3Objects.First().Key; 
            } 
 
Query 3-  I am able to add new folders using FileManager, but after adding them they do not display in FileManager. I can verify that they have been added in AWS Control Panel.and they are there. 
 
You can add the new folder in FileManager component. Refer the below demo link for your reference. 
 
https://ej2.syncfusion.com/aspnetcore/FileManager/AmazonS3Provider#/material 
 
Query 4-  Also I note that the demo FileManager Aws S3 on your website does not work. 
 
Our demo is in Runnable state. Refer the below screenshot. -> https://ej2.syncfusion.com/aspnetcore/FileManager/AmazonS3Provider#/material 
 
  
 
 
Please let us know, if you need any further assistance. 
 
Regards,  
Sowmiya.P

Tom McEwen · Answer

Thank you for your prompt response to my question.

Based on your response I was able to achieve a working solution by a bit of further experimentation as follows:
NOTE: Some screen captures are included to make this a little clearer than my attempt at explaining this.
First Try (failed):
1) Using AWS S3 Control Panel, deleted all folders in the S3 Bucket
2) Using AWS S3 Control Panel, added one folder at the root level.
3) Then viewed this using Angular File Manager with Net Core S3 File Manager and then got a view showing the folder added at the root level. However the view also showed a sub-folder below the root folder with no name. So I tried to rename that folder to give it a name but received an error, then tried to delete it, but also received an error so could not delete it either.
4) Tried adding a new folder and this worked, so now have the new added folder with a name, as well as the unnamed folder under the root.
5) Went back to AWS S3 Control Panel and found that the root level folder (actually a key in AWS) shows a folder within the root folder with the same name -weird, doesn't make sense.
6) So basically everything works here, but can't get rid of the strange folder with no name.

Second Try (failed):
1) Using AWS S3 Control Panel, again deleted all folders in the S3 Bucket.
2) Then tried adding the Root Folder from Angular File Manager with Net Core S3 File Manager, but received an error when attemping this, so it appears that I cannot add the Root Folder using the Angular File Manager.

Third Try (success):
1) Added Root Folder using AWS S3 Control Panel
2) Add on Sub Folder under the Root Folder using AWS S3 Control Panel. So now have the Root plus child folders (actually keys in S3).
3) Back the Angular File Manager with Net Core S3 File Manager and now get a view that is correct -> a Root folder with one sub folder.
3) Now found add/deleting folders works correctly, and file upload also works on first try.

So it looks like to get this working initially, I needed to:
1) Add one and only one (per your earlier suggestion) folder at the root level, but using AWS resources such as S3 Control Panel.
2) Add at least on sub-folder below the root, also using AWS resource -S3 Control Panel in my case.
Then everything proceeds as expected, so I'm going to proceed from here with this.

As mentioned earlier, this is a great Control!! Should be very useful.

Attachment: ScreenCaptures_514414b2.zip

Shameer Ali Baig Sulaiman Ali Baig · Answer

Hi Tom, 
 
We have checked your reported problem details. We would like to let you know that, we should load the file system for FileManager on initializing the component. We need to set the root folder from the API service from the file system for File Manager, so it would load the files/folders within in the root folder. So, AWS S3 bucket should have at least one folder to make it as root folder for File Manager component.  
 
This is the why you have succeeded to configure File Manager with Amazon file provider service in your third attempt. We are happy that you finally achieved your expected requirement. 
 
Query: However, the view also showed a sub-folder below the root folder with no name. So I tried to rename that folder to give it a name but received an error, then tried to delete it, but also received an error so could not delete it either. 
 
We are validating this reported problem from our end. We will update you the further validations in two business days on 3rd November 2020. 
 
We appreciate your patience. 
 
Regards, 
Shameer Ali Baig S.

Tom McEwen · Answer

Again, thanks for the prompt response. Much nicer dealing with humans than with computers :-)

Sowmiya Padmanaban · Answer

Hi Tom McEwen,  
 
Thanks. We will update our further validation details for your reported problem on mentioned timeline. 
 
We appreciate your patience. 
 
Regards,  
Sowmiya.P

Sowmiya Padmanaban · Answer

Hi Tom McEwen,  
 
We have validated your reported problem with FileManager component. But, we were unable to replicate your reported problem. 
 
In our AmazonS3 bucket, if we have only single folder ( Files),  then within that folder there is no objects available in AmazonS3 bucket. 
 
Refer the below attached image for AmazonS3 bucket. (we have hidden some details in the below screenshot of security purpose) 
 
  
 
Refer the below screenshot. FileManager component only contains the Files folder. And, it has no unnamed folders. 
 
  
 
 
And also, we have checked with single folder inside the root folder (Files). Bu the issue is not occur in our end. If the issue still persist, please share us the more details regarding your reported problem. 
 
Please let us know, if you need any further assistance. 
 
Regards,  
Sowmiya.P

Tom McEwen · Answer

As mentioned previously I was able to solve the problem for my purposes by starting out with 1 root folder, and at least 1 child folder, so after that have been working okay with no problems.

Thanks again for you help.

Tom Mc

Sowmiya Padmanaban · Answer

Hi Tom,  
  
We are happy to hear that your problem has been resolved. Please contact us, if you need any help from us. 
  
Regards,  
Sowmiya.P

Matthew Marichiba · Answer

I have been running into similar problems while setting up my S3 bucket for the first time. It seems like some essential detail is missing from the documentation. My discoveries are below, which I would have loved to read in the docs.

I appreciate that this component and the dotnet service are so well developed, and I'm really excited to get using it in my applications. But the nit-picky headaches with S3 setup have been really frustrating and unnecessary. Some modest documentation would go a long way. Thank you!

1) Structure the bucket with exactly one folder at the top level. (Or if you have multiple folders at the bucket root, the first alphanumeric item is chosen.)

2) The folder must be created with the S3 Console "Create Folder" command, which explicitly creates a 0b object by the name of the directory. This hung me up for hours. I was trying to start with existing bucket content, and it's not enough to have objects with Keys that start with "Files/"; I had to explicitly add an extra, empty object with Key "Files/".

3) The most recent documentation I could find related to setting up the bucket for the S3 File Provider appears to be a blog post (not ideal), and the post is missing the crucial details above. It also specifies to make the bucket public, which I assume is not what most users want. A public bucket shouldn't be necessary, because the S3 File Provider has Secret Key credentials to access the bucket.

4) All subfolders must also be defined with a 0b object in S3 where the Key name is the full path of the folder. Without these markers, browsing the directory structure with the S3FileProvider will be buggy. (This could be fixed with a feature enhancement. Is this the sort of thing I should file a feature request for?)

5) The S3FileManager service, as currently coded, relies on folder objects (described in #2 & #4 above) in order to parse the directory structure. If you're doing file manipulation with CLI tools such as `aws s3 cp` or `aws s3 sync` to recursively traverse directories to copy files, you will lose the directory-marker objects. More effort is required to make things work with FileManager if you're doing manual S3 file manipulations. For example, even a backup `aws s3 sync ` and immediate restore `aws s3 sync ` will not leave the bucket in a functional state.

Sivakumar ShunmugaSundaram · Answer

Hi Matthew,

S.NO

Queries

Solution

1.

1) Structure
  the bucket with exactly one folder at the top level. (Or if you have multiple
  folders at the bucket root, the first alphanumeric item is chosen.)

With the shared details, you need a configuration detail
  related to our Amazon S3 file service provider. We would like to let you know
  that, as per our current implement of amazon s3 file service provider will
  takes the first available folder in the bucket. 
   
  But you can be able to set the specific bucker folder as
  FileManager root folder by using the below code details in the AmazonS3FileProvider.cs
  file GetBucketList method.
   
  Refer to the below code details.

public void GetBucketList()
           
    {
               
    ListingObjectsAsync("", "", false).Wait();
               
    // setting the root name with custom name
    attribute and reading the files/folders inside based on that root name 
                RootName
    = response.S3Objects.Where(x => x.Key == FolderName).ToArray()[0].Key;
           
    }

GitHub location: https://github.com/SyncfusionExamples/amazon-s3-aspcore-file-provider/blob/bc78bb411198cd1c2bfdd6b2af88d149005adb93/Models/AmazonS3FileProvider.cs#L42

2.

2) The folder must be created with the S3 Console
  "Create Folder" command, which explicitly creates a 0b object by
  the name of the directory. This hung me up for hours. I was trying to start
  with existing bucket content, and it's not enough to have objects with Keys
  that start with "Files/"; I had to explicitly add an extra, empty
  object with Key "Files/".

We would like to let you know that creating folder in
  Amazon S3 bucket will creates a folder with 0 bytes. This is the behaviour of
  folder creation and the similar will happen while you trying to create a
  folder in windows explorer itself.
   
  Also, as per current implementation of FileManager
  component you can be able to set a single folder as root folder. But you do
  not need to explicitly add an extra, empty object with Key
  "Files/". You can be able to set our own folder as FileManager root
  folder using the above-mentioned code details.

3.

3) The most recent documentation I could find related to
  setting up the bucket for the S3 File Provider appears
  to be a blog post (not ideal), and the post is missing the crucial
  details above. It also specifies to make the bucket public, which I assume
  is not what most users want. A public bucket shouldn't be
  necessary, because the S3 File Provider has Secret Key credentials to access
  the bucket.

As per the details, you are using our Syncfusion blob to
  create an amazon bucket and facing issue with following the steps mentioned
  in the blob. Also, from the blob we have set the amazon bucket as public from
  our side.
   
  But it is not necessary to make a bucket as public from
  your side. We have made the bucket as public due to below mentioned purpose.

To share the contents
       of the bucket or object with others, such as by making them available
       for download or embedding them in a website.
   To use the contents of
       the bucket or object in a publicly available application or service,
       such as a mobile app or website.
   To allow others to
       upload or delete objects in the bucket, which can be useful for
       collaborative projects or for hosting user-generated content.

Also, we have considered this correction as an improvement
  from our side and will refresh the blob as soon as possible.

4.

4) All subfolders must also be defined with a 0b object in
  S3 where the Key name is the full path of the folder. Without these markers,
  browsing the directory structure with the S3FileProvider will be buggy. (This
  could be fixed with a feature enhancement. Is this the sort of thing I should
  file a feature request for?)

We are facing issue with browsing the directory structure
  with the S3FileProvider while using key as full path of the folder. But
  unfortunately, we are unable to replicate the reported issue from our side.
   
  Since we are unbale to replicate the reported issue from
  our side, share a exact step to replicate the reported issue from your side
  or share a video footage to demonstrate the issue. It will help us to
  investigate the reported issue from our side.

5.

5) The S3FileManager service, as currently coded, relies
  on folder objects (described in #2 & #4 above) in order to parse the
  directory structure. If you're doing file manipulation with CLI tools such as
  `aws s3 cp` or `aws s3 sync` to recursively traverse directories to copy
  files, you will lose the directory-marker objects. More effort is required to
  make things work with FileManager if you're doing manual S3 file
  manipulations. For example, even a backup `aws s3 sync ` and immediate
  restore `aws s3 sync ` will not leave the bucket in a functional state.

With these details, you are facing issue with copying the
  folders inside the Amazon S3 bucket using the mentioned CLI. Our FileManager
  component is used to browse, manage, and organize the files and folders in a
  file system through a web application. All basic file operations like
  creating a new folder, uploading and coping of files in the file
  system, and deleting and renaming of existing files and folders are available
  in the file manager component.
   
  So, we suggest you try our FileManager operations to
  achieve your requirement. We have shared documentation which explains the
  FileManager operations for your reference.
   
  Documentation: https://ej2.syncfusion.com/aspnetcore/documentation/file-manager/file-operations