Use FileManager with Minio

[huskaner]

Hi,

in my blazor server project i am trying to use the FileManager instead of S3 Amazon File Provider with Minio Storage for local development.

I did find this example from you:

https://www.syncfusion.com/forums/160437/s3-example

I did update that sample with this code:

In AmazonS3FileProvides.cs:

// Register the amazon client details
public void RegisterAmazonS3(string name,string awsAccessKeyId,string awsSecretAccessKey,string region)
{
  bucketName = name;
  //RegionEndpoint bucketRegion = RegionEndpoint.GetBySystemName(region);
  //client = new AmazonS3Client(awsAccessKeyId, awsSecretAccessKey, bucketRegion);

  // Add Minio Config
  var config = new AmazonS3Config
  {
   AuthenticationRegion=RegionEndpoint.USEast1.SystemName,// Should match the `MINIO_REGION`         
   //AuthenticationRegion = "",
   ServiceURL="http://localhost:9000",
   ForcePathStyle=true// MUST be true to work correctly with MinIO server
  };
   client = new AmazonS3Client(awsAccessKeyId, awsSecretAccessKey, config);
}

and In the default controller:

     //this.operation.RegisterAmazonS3("<---bucketName--->", "<---awsAccessKeyId--->", "<---awsSecretAccessKey--->", "<---region--->");
      this.operation.RegisterAmazonS3("filemanagement", "minioadmin", "minioadmin", "");

I face some problems here:

- The connect seams to work, but only if the bucket exists and at least one file exists it doesn't throw an error. It should create the bucket if it doesn't exists with the given credentails. And should work without a necessary file in root folder. ?

- I would like to define a RootFolder like "Root" and created a Directory within the bucket, but couldn't manage to get this work a Root for FileManager, which can't be deleted.

So i hope you could help to solve this problems. I would like to be able to use the Filemanger with all functionallity with minio-

[huskaner]

Hi Support Team,

any updates for this problem?

[Indhumathy Loganathan]

Hi huskaner,

Greetings from Syncfusion support.

Currently we don’t have separate file service provider for Minio in File Manager component. However, you can customize your own service provider for File Manager component by comparing our Amazon File Service provider. Please find the answer for your queries below.

Query 1: The connect seems to work, but only if the bucket exists and at least one file exists it doesn't throw an error. It should create the bucket if it doesn't exists with the given credentials. And should work without a necessary file in root folder?

In the Amazon provider, we only connect with the available bucket directory. This requirement seems general. You need to customize the backend to create a bucket in case of its absence. Refer to the below blog to meet your requirement.

https://www.c-sharpcorner.com/UploadFile/ec2studio/amazon-s3-from-C-Sharp-developer-point-of-view/

https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html

Query 2: I would like to define a RootFolder like "Root" and created a Directory within the bucket, but couldn't manage to get this work a Root for FileManager, which can't be deleted.

We would like to let you know that, the FileManager component requires root folder name and path to perform the file operations like read, delete the folder within the FileManager component. Due to this reason, we have prevented the default file operations(cut, delete...etc.) for Root directory. You can remove the entire files or folders within root directory, but the root folder cannot be deleted. This is the default behavior. Please let us know, in which scenario you would like to remove the root directory. Kindly share us more details regarding your requirement, so that we can validate the feasibility to achieve your requirement.

Regards,

Indhumathy L

[huskaner]

Hi Support,

thank your for the links. They were helpful. I did create  a bucket manually, and then added these functions to generate my needed folder structure:

  public async Task RootFolder(string name)
        {
            this.RootName = name;
            ListingObjectsAsync("", name, false).Wait();
            if (response.S3Objects.Count > 0)
                RootName = response.S3Objects.First().Key;
            else
            {
                CreateFolder(name);
            }


            //contentRootPath = name;
            //hostName = new Uri(contentRootPath).Host;
            //if (!string.IsNullOrEmpty(hostName))
            //{
            //    hostPath = Path.DirectorySeparatorChar + hostName + Path.DirectorySeparatorChar + contentRootPath.Substring(contentRootPath.ToLower().IndexOf(hostName) + hostName.Length + 1);
            //}
        }




        public async Task CreateFolder(string name)
        {
            await ListingObjectsAsync("", name, false);
            if (response.S3Objects.Count == 0)
            {
                // Folder doesn't exits
                PutObjectRequest request = new PutObjectRequest()
                {
                    BucketName = bucketName,
                    Key = name // <-- in S3 key represents a path
                };
                PutObjectResponse response = await client.PutObjectAsync(request);
            }
        }

From my controller i now call them like this:

   this.operation.RegisterAmazonS3("appzero", "minioadmin", "minioadmin", "");
   this.operation.RootFolder(root); // Data\Files denotes in which files and folders are available.
   this.operation.CreateFolder(subPath);

So thanks for help 

[Indhumathy Loganathan]

Hi huskaner,

We are glad to hear that the provided suggestions were helpful for you. Please get back to us if you need any further assistance.

Regards,

Indhumathy L

[huskaner]

Hi,

i still got some error here. I need help for two parts:

First (a Bug maybe):

It seems that the path is not set correct while uploading files to subfolder.

I generate a subfolder and upload a file. its uploaded to the wrong place.

I did prepare an example based on the solution above.

Follow these steps to see the error:

1) First start minio in docker with :

docker run -d -p 9000:9000 -p 9001:9001 -e "MINIO_ROOT_USER=minioadmin" -e "MINIO_ROOT_PASSWORD=minioadmin" minio/minio server /data --console-address ":9001"

2) open minio console at http://localhost:9001 and create a bucket "mybucket"

3) run the solution

4) upload a file to the subfolder Filemanagement/Uploads

This file is uploaded with correct path

5) Now create a subfolder for the folder Uploads called Subfolder.

6) Select that SubFolder and upload a File.

Here things get wrong.. The file get uploaded to /Subfolder instead of Uploads/Subfolders

Seams to be a Bug in current version. Or ?

Second:

So besides this error:

I would like to achieve also ChunkUpload for S3Storage. I did use SFUploader for this. Please check this link for details: https://www.syncfusion.com/forums/168882/howto-use-chunkupload-within-filemanager

How can i achieve that also for the S3Storage ?

Hope you could update that sample to solve both.

UPDATE1:

There also seams to be a bug, when creating an existing folder:

- Create a folder "one" at root.

- Create a second folder "two" at root.

- Create in second folder "two" a new subfolder "one"

- If you now select the one folder, both "one" folders are selected.

Attachment: Amazon_filemanageruploadtest_b364e145.zip

[huskaner]

Hi, any updates?

[Indhumathy Loganathan]

Hi huskaner,

We unable to configure the steps to connect with the Minio provider. So we have validated your requirement with Amazon S3 File service provider. We understood that you are trying to Upload file within the backend storage. In File Manager, we have integrated Uploader component to perform upload operation. If we specify the proper target location it will upload the files in that corresponding location without fail. We suggest you to once check the path setting while performing Upload operation.

As for the chunk upload, we have prepared a sample with Amazon S3 provider where we integrated external Uploader component within File Manager. Refer to the below code snippet.

[Index.razor]

 

<SfUploader ID="UploadFiles" AutoUpload="false" MaxFileSize=100000000000>

    <UploaderAsyncSettings SaveUrl="/api/Default/Save" ChunkSize=500000>

    </UploaderAsyncSettings>

    <UploaderEvents Success="OnSuccess" FileSelected="OnFileSelect"></UploaderEvents>

</SfUploader>

 

[DefaultController.cs]

 

[Route("Save")]

// Upload method for chunk-upload and normal upload

public void Save(IList<IFormFile> chunkFile, IList<IFormFile> UploadFiles)

{

    var data = Request.Form["path"];

    var path = data.ToString();

    var action = "save";

    FileManagerDirectoryContent[] dataObject = new FileManagerDirectoryContent[1];

    long size = 0;

    try

    {

        // for chunk-upload

        foreach (var file in chunkFile)

        {

            var filename = ContentDispositionHeaderValue

                                .Parse(file.ContentDisposition)

                                .FileName

                                .Trim('"');

            var fileName1 = Path.GetFileName(UploadFiles[0].FileName);

            filename = this.rootPath + $@"\{filename}";

            size += file.Length;

 

            if (!System.IO.File.Exists(filename))

            {

                using (FileStream fs = System.IO.File.Create(filename))

                {

                    file.CopyTo(fs);

                    operation.Upload(path, UploadFiles, action, dataObject);

                    fs.Flush();

                }

            }

            else

            {

                using (FileStream fs = System.IO.File.Open(filename, FileMode.Append))

                {

                    file.CopyTo(fs);

                    fs.Flush();

                }

            }

        }

    }

    ...

 

    // for normal upload

    try

    {

        foreach (var file in UploadFiles)

        {

            var filename = ContentDispositionHeaderValue

                            .Parse(file.ContentDisposition)

                            .FileName

                            .Trim('"');

            filename = this.rootPath + $@"\{filename}";

            size += file.Length;

            if (!System.IO.File.Exists(filename))

            {

                using (FileStream fs = System.IO.File.Create(filename))

                {

                    file.CopyTo(fs);

                    operation.Upload(path, UploadFiles, action, dataObject);

                    fs.Flush();

                }

            }

        }

    }

    …

}

Sample: https://www.syncfusion.com/downloads/support/directtrac/general/ze/BlazorFileManager-755471111

Reference: https://www.syncfusion.com/forums/170014/file-upload-to-s3-bucket-from-ej2-syncfusion-control

We have validated the selection issue you have reported in the File Manager component by creating a same folder name as explained. But we were not able to replicate the reported issue at our end. We suspect that this issue only occurs with Minio service.

If the reported issue even occur in the above shared sample, please revert us with the package version used and the video footage of issue. These details would help us assist you promptly. Please get back to us with the required details for further assistance.

Regards,

Indhumathy L

[huskaner]

Hi Indhumathy Loganathan,

with the sample you provided the problem still exists. I did modify it for use with minio and attached the code. my update is based on this:

https://docs.min.io/docs/how-to-use-aws-sdk-for-net-with-minio-server.html

I would prefer that you continue with my example below.

minio is a valid case for testing in local enviroments. so it should be supported by you.

The chunkfile Upload upload the File to the wwwroot Folder but doesn't store any file in the Bucket. I do not see that operation.Upload does anything. And no error occurs.

Keep in mind that for example i want to upload a 20gb File. So that means that we should store only a chunk of file at backend controller wwwroot and then move it to the bucket (which can handle mulitpart uploads). In the end the complete file should exists within the bucket.

I did attach some instructions how you can get it run at your side:

- run within a cmd in the solution folder: powershell .\up.ps1

- run http://localhost:9001 to check availabilty of minio (login with minioadmin/minioadmin)

- run the solution

- upload a files and check minio admin console for new files

I also have a Video that i would like to share with you, but was not able to upload it here. How can i provide that video for you ?

Best Regards

H.Uskaner

Attachment: BlazorFileManager755471111Minio_2e83727d.zip

[huskaner]

I did  attach the Video as Zip File.  Please check it with the code sample above.

Attachment: BlazorMiniioVIDEO_9c7a84f.zip

[Indhumathy Loganathan]

Hi Huskaner,

We will update the details in the ticket. Please follow up there for further technical assistance.

Regards,

Indhumathy L