WASM large data load issue

Hi,

The example  Blazor DataGrid Example | Grid Overview | Syncfusion Demos shows a grid that can easily handle 1 mil records.

I have a blazor hosted webassembly having an issue loading 100k records, takes easily 20 seconds. It is currentlu using the standard hosted method calling the controller from Products = await Http.GetFromJsonAsync<List<Product>>("api/product/getallproducts");​ I've tested this by changing to Http.GetStringAsync ("api/product/getallproducts"); ​ and the data loads fast.​

The sql and controller returns the data in less than 3 seconds, the issue seems to be coming from deserialize to a class list. On your demo I'm assuming it is server side blazor and the data call OverviewData.GetAllRecords(Value) probably is just generating random data directly into the class list.


So how would you use your Grid control in this method for large datasets?


I tried test the datamanager thinking maybe it will handle making ht results shorter <Syncfusion.Blazor.Data.SfDataManager Url="api/product/getallproducts" Adaptor="Adaptors.WebApiAdaptor"></Syncfusion.Blazor.Data.SfDataManager>​ in the grid but get the error "Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'Syncfusion.Blazor.Data.OData`1[Prohyd.Shared.Models.Product]' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly. To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array. Path '', line 1, position 1."


Even though my JSON format is same outer structure to the sample in 

https://services.odata.org/V4/Northwind/Northwind.svc/Orders/

[

{

"id": 1,

"material": "Code1",

"description": "12312asda",

"sysDescription": null,

"accPacDescription": null,

"createdOn": "2022-02-18T13:18:11.22",

"createdBy": null,

"thumbail": null,

"categoryId": 1,

"leadTime": "32",

"weight": 55.0,

"weightUnit": "KG",

"price": 0,

"maxDiscountAllowed": "TBA",

"quantityWhs": 0,

"quantityDbn": 0,

"attributeGroupId": 0,

"scaleQuantities": null,

"scaleQuantitiesPrices": null,

"vendor": {

"id": 1,

"name": "Name",

"accCode": null,

"type": null,

"image": null,

"description": null,

"pendingCount": 0,

"email": null,

"categories": null,

"users": null

},

"user": {

"id": 1,

"uniqueId": null,

"name": "User",

"vendorId": null,

"groupId": null,

"email": null,

"pendingCount": 0,

"group": null,

"vendor": null,

"orders": null

},

"category": {

"id": 1,

"parentId": null,

"name": "Uncategorised",

"hasChild": null,

"document": null,

"supplierId": null,

"supplier": null

},

"attributes": null,

"attributeValues": "",

"orderItems": null

},

{

"id": 2,

"material": "code002",

"description": "123jkh23",

"sysDescription": null,

"accPacDescription": null,

"createdOn": "2022-02-18T13:18:11.22",

"createdBy": null,

"thumbail": null,

"categoryId": 1,

"leadTime": "32",

"weight": 35.0,

"weightUnit": "KG",

"price": 0,

"maxDiscountAllowed": "TBA",

"quantityWhs": 0,

"quantityDbn": 0,

"attributeGroupId": 0,

"scaleQuantities": null,

"scaleQuantitiesPrices": null,

"vendor": {

"id": 1,

"name": "Name",

"accCode": null,

"type": null,

"image": null,

"description": null,

"pendingCount": 0,

"email": null,

"categories": null,

"users": null

},

"user": {

"id": 1,

"uniqueId": null,

"name": "User",

"vendorId": null,

"groupId": null,

"email": null,

"pendingCount": 0,

"group": null,

"vendor": null,

"orders": null

},

"category": {

"id": 1,

"parentId": null,

"name": "Uncategorised",

"hasChild": null,

"document": null,

"supplierId": null,

"supplier": null

},

"attributes": null,

"attributeValues": "",

"orderItems": null

}

]



16 Replies

RN Rahul Narayanasamy Syncfusion Team February 24, 2022 01:54 PM UTC

Hi Andrew, 

Greetings from Syncfusion. 

We suspect that you are facing difficulties while loading large amount of records in the Grid. In the shared Syncfusion Blazor DataGrid demo link, we have used large dataset as local data with Virtualization feature.  

When grid having large amount of data, we suggest you to bind the data in on-demand loading(only the current page records will be bounded to the grid) by either using the Paging or Virtualization feature of the grid. Please find the below documentation for your reference. 


Also, you have tried to bound the Grid data using SfDataManager with WebApiAdaptor. If you want to use WebApiAdaptor in DataManager, You need to return the result as Items and Count format. Please refer the documentation link for your reference. 


If you want to bind the Grid data using SfDataManager, we have different adaptors to adapt binding data from different remote services to Grid. Based on your scenario and requirement, we suggest you to use the suitable way of binding data. We suggest you to refer to our below documentation reference links for more details regarding the available Adaptors and the ways of binding data to Grid. You can use any of the available ways of data binding to bind data to Grid based on your scenario. 

Adaptors: 

DataBinding: 

if you are still facing the problem, then please share more details(grid codes, simple sample, video) about your problem which will be helpful to validate and provide a better solution. 

Regards, 
Rahul 
 



GE Gerome February 24, 2022 02:25 PM UTC

Hi,

This is a VERY bad technique using this  Products = await Http.GetFromJsonAsync<List<Product>>("api/product/getallproducts")

I don't use that, I've made a controller where i exploit the DataManagerRequest that is a parameter given from the DataAdapter ReadAsync function, then my controller signature is :


public async Task<DataResult> GetPagedQueryableAsync<TEntity>(DataManagerRequest dataManagerRequest) where TEntity : class, new()


Then I deal with DataManagerRequest that contains all the necessary to make the atomic request when i ping the database, for example I have a table made of 20 fields and 150k od datas, then I can grab the result from my wasm project within 50 ms whatever the size of the table



AN Andrew replied to Rahul Narayanasamy March 2, 2022 06:42 PM UTC

Hi Support,

Please find attached simple example using the weatherforecast test. I've made generate 100k items, and tried the OData method from the MS links in the documentation, but its still giving JSON errors when converting.


I was having issue working out which MS package of OData to use as they all have different functions, I'm looking for proper .NET 6 support.


Please check and advise on how the controller call should happen.


Thanks


Attachment: AdapterTest_37f877ee.zip


RN Rahul Narayanasamy Syncfusion Team March 3, 2022 02:05 PM UTC

Hi Andrew, 

Thanks for the update. 

From your provided details, you have used SfDataManager with WebApiAdaptor. As you have used WebApiAdaptor in DataManager, you need to return the result as Items and Count format as we have mentioned in the previous update. Please refer the documentation link for your reference. 


 

Here, we have modified your sample(returned data as Items and Count). Now the data is properly bound to the Grid without any issues. Find the below code snippets and sample for your reference. 

[HttpGet] 
        [Route("getlargedata")] 
        public async Task<object> GetLargeData(ODataQueryOptions<WeatherForecast> options) 
        { 
            ODataQuerySettings settings = new () 
            { 
                PageSize = 100 
            }; 
 
            var data = GetData(); 
 
            IQueryable results = options.ApplyTo(data.AsQueryable(), settings); 
 
            var r = new PageResult<WeatherForecast>( 
                results as IEnumerable<WeatherForecast>, 
                Request.HttpContext.ODataFeature().NextLink, 
                data.Count() 
            ); 
            var data1 = results as IEnumerable<WeatherForecast>; 
            return new { Items = data1, Count = data1.Count() }; 
       } 


Please let us know if you have any concerns. 

Regards, 
Rahul 
 



AN Andrew replied to Rahul Narayanasamy March 3, 2022 07:40 PM UTC

Hi Rahul,

Ok then I'm not sure why the Syncfusion article then points to an MS article that uses PageResult method if it doesn't get used.


So the example you've updated now returns data, except its only returning 12 items out of the 100k, and its not allowing any paging to bring more data. 


When the section "IQueryable results = options.ApplyTo(data.AsQueryable(), settings);" runs its chopping the data to 12 items when the ODataQuerySettings PageSize is set to 100.


Please can you provide a complete example with this actually working, so all 100k items can be paged between without all the data being called through at once.


Thank you



RN Rahul Narayanasamy Syncfusion Team March 4, 2022 01:36 PM UTC

Hi Andrew, 

Thanks for the update. 

You want to load large amount of data to the Grid based on the on-demand data loading(Paging). When using WebApiAdaptor, you need to handle the data operations(such as paging, sorting, filtering, etc) in API controller. You can receive a querystring(Request.Query) contains skip, take and other values(for sorting, filtering) values in WebApi controller. Based on the querystring, you need to handle the data operations(paging, sorting, filtering) at controller side

Based on your case, we have get the skip and take value from Request.Query and returned the particular page records while clicking the pager. Find the below code snippets and sample for your reference. 

[HttpGet] 
        [Route("getlargedata")] 
        public async Task<object> GetLargeData(ODataQueryOptions<WeatherForecast> options) 
        { 
            var queryString = Request.Query; 
            ODataQuerySettings settings = new () 
            { 
                PageSize = 100 
            }; 
 
            var data = GetData(); 
 
            if (queryString.Keys.Contains("$inlinecount")) 
            { 
                StringValues Skip; 
                StringValues Take; 
                int skip = (queryString.TryGetValue("$skip", out Skip)) ? Convert.ToInt32(Skip[0]) : 0; 
                int top = (queryString.TryGetValue("$top", out Take)) ? Convert.ToInt32(Take[0]) : data.Count(); 
                var count = data.Count(); 
                return new { Items = data.Skip(skip).Take(top), Count = count }; 
            } 
            else 
            { 
                return data; 
            } 
       } 


Please let us know if you have any concerns. 

Regards, 
Rahul 



AN Andrew replied to Rahul Narayanasamy March 8, 2022 08:01 PM UTC

Hi Support,


Thanks we've implemented this and its working correctly. Can you advise though on how to still allow filtering, we generally use the Excel FilterType but the load times is extremely long to populate the filter box. Is there an additional setting?


<GridFilterSettings Type=Syncfusion.Blazor.Grids.FilterType.Excel></GridFilterSettings>



VN Vignesh Natarajan Syncfusion Team March 9, 2022 05:21 AM UTC

HI Andrew,  
 
Thanks for the update.  
 
Query: “Can you advise though on how to still allow filtering, we generally use the Excel FilterType but the load times is extremely long to populate the filter box. Is there an additional setting? 
 
We have analyzed your requirement to handle the filter operation in Grid and you have defined the GridFilterSettings. We would like to inform you to enable Filtering action in Grid, kindly enable AllowFiltering property of Grid. So kindly ensure that you have enabled the AllowFiltering property.  
 
Refer the below code example for your reference.  
 
<SfGrid @ref="Grid" TValue="Orders" AllowFiltering="true" AllowSorting="true" AllowPaging="true">    <SfDataManager Url="api/Default" Adaptor="Adaptors.WebApiAdaptor"></SfDataManager>    . . . . .. . . . </SfGrid>
 
[Controller] 
 
public class DefaultController : ControllerBase{    public static List<Orders> order = new List<Orders>();    // GET: api/Default    [HttpGet]    public async Task<object> Get(int? code)    {        if (order.Count == 0)        {            BindDataSource();        }        var data = order.AsQueryable();        var queryString = Request.Query;        string grid = queryString["ej2grid"];        string sort = queryString["$orderby"];   //sorting              string filter = queryString["$filter"];        string auto = queryString["$inlineCount"];        if (filter != null// to handle filter opertaion        {            if (filter.Contains("substring"))//searching             {. . . . .  . .            }        }        if (sort != null//Sorting         {.. . . .         }        if (queryString.Keys.Contains("$inlinecount"))        {            StringValues Skip;            StringValues Take;            int skip = (queryString.TryGetValue("$skip"out Skip)) ? Convert.ToInt32(Skip[0]) : 0;            int top = (queryString.TryGetValue("$top"out Take)) ? Convert.ToInt32(Take[0]) : data.Count();            var count = data.Count();             return new { Items = data.Skip(skip).Take(top), Count = count };                       }        else        {            return data;        }    }
 
 
Note: While using WebAPI adaptor, we will send only the queries for current operation (like filtering, sorting, paging etc.) to WebAPI adaptor, we need to handle these action in API controller and return the executed result to display in Grid. For filtering, we will send filter queryies as $filter, we need to handle the filter action on our own.  
 
Please get back to us if you have further queries.  
 
Regards, 
Vignesh Natarajan 



AN Andrew May 2, 2022 12:04 PM UTC

Hi,

Please can you show this example working with ODataV4 adapter as I need to update the project. When I manually enter the query strings the OData works fine but the data will not show in the grid or gives errors.


Please show the controller set up working correctly including for filters.

Thanks




RS Renjith Singh Rajendran Syncfusion Team May 3, 2022 10:04 AM UTC

Hi Andrew,


We have already documented a step by step procedure on binding data from OData service in Grid. We suggest you to refer the below documentation for more details on this topic,

https://blazor.syncfusion.com/documentation/common/data-binding/restful-service-binding

https://blazor.syncfusion.com/documentation/common/data-binding/restful-service-binding#handling-crud-operations-with-our-syncfusion-blazor-datagrid-component


We have also attached a GitHub sample in the above documentation. Please refer the above documentation and check this from your side and get back to us if you need further assistance.


Regards,

Renjith R




AN Andrew May 12, 2022 08:00 AM UTC

Hi Renjith,

We've been through all the documentation and most focus is on serverside and not wasm. We are also using WASM hosted.

The sample in the github is also .NET 5, converting to the project to .NET 6 breaks all its set up. I've seen no working examples of the syncfusion ODatav4 adapter for .NET6. Our OData calls work in our .NET 6 project when using the URL to select/filter but the Syncfusion control does not process it.

Can a .NET 6 working example not be produced for WASM hosted, as pointing to existing external test OData API's like in the documentation does not help trouble shooting.


Thanks



MS Monisha Saravanan Syncfusion Team May 13, 2022 01:07 PM UTC

Hi Andrew,


Greetings from Syncfusion support.


We have prepared an sample based on your requirement by using dotnet 6 WASM with ODatav4 Adaptor. Kindly check the attached sample for your reference.


Sample: https://www.syncfusion.com/downloads/support/directtrac/general/ze/SSample-154263236-598688907.zip


Please get back to us if you have further queries.


Regards,

Monisha



AN Andrew May 16, 2022 08:38 AM UTC

Thanks Monisha, this is extremely helpful.


A few questions on the sample with this method


Column Filters

1 - If you set the filter type to "FilterType.Menu" it works well, but if there's a string column like "Summary" when you type in the "Enter the value" field it pops up a saying "No records found". Is there a way to turn that off because we just want the filter to run based on the input.



2 - If you had an identification column for example "Country" and wanted a selection list I can pre-polulate the datasource if a filtertemplate but how do I bind it so that it still works in the Odata. And can the "Starts With" be removed in this case too.

The below sends the Odata but with NULL in the Country field and throws a 400. If I bind "Value" to c.Country it breaks the control on the UI


<GridColumn Field="@nameof(ou.Country)" Width=100 Type="ColumnType.String">

    <FilterTemplate>

        @{

            var c = context as OrgUnit;

            <SfDropDownList DataSource="@Country" TValue="string" TItem="string"></SfDropDownList>

        }

    </FilterTemplate>

</GridColumn>


Search

If I add Search to the toolbars and test the general search box, the network shows the Odata query is triggering with the search term http://localhost:5076/odata/OrgUnits?$count=true&$search=Ou+96222%20OR%20Ou+5000&$skip=0&$top=50 but the search doesn't apply, is anything else needed to enable it?


Thank you



MS Monisha Saravanan Syncfusion Team May 17, 2022 02:20 PM UTC

Hi Andrew,


Query: “ If you set the filter type to "FilterType.Menu" it works well, but if there's a string column like "Summary" when you type in the "Enter the value" field it pops up a saying "No records found". Is there a way to turn that off because we just want the filter to run based on the input.”


We would like to inform that by default autocomplete is rendered inside the filter template. We can override the default component by rendering the custom component inside filter template as per our requirement. Kindly refer the attached code snippet and UG for your reference.


<GridColumn Field=@nameof(Order.OrderID) HeaderText="Order ID" TextAlign="TextAlign.Center" Width="120">

            <FilterTemplate>

                <SfDropDownList Placeholder="OrderID" ID="OrderID" @bind-Value="@((context as PredicateModel<int?>).Value)" TItem="Order" TValue="int?" DataSource="@(Orders)">

                    <DropDownListFieldSettings Value="OrderID" Text="OrderID"></DropDownListFieldSettings>

                </SfDropDownList>

            </FilterTemplate>

        </GridColumn>



Reference: https://blazor.syncfusion.com/documentation/datagrid/filter-menu#custom-component-in-filter-menu


Query: “can the "Starts With" be removed in this case too.”


We can change the default filter operator on FilterSettings. Kindly refer the attached UG for your reference.


Reference: https://blazor.syncfusion.com/documentation/datagrid/filter-bar#change-default-filter-operator


Query:  the network shows the Odata query is triggering with the search term but the search doesn't apply, is anything else needed to enable it?


We would like to inform that to enable search operation in OData, we suggest you enable the EnableODataSearchFallback option. Please use the below code in your application to perform search operations in Grid.


Limitation :

Please find the following limitations applied for implementing this feature in your application.


  1. The contains operator will be applied only for string typed column
  2. The equal operator will only be applied for all other data types.


Refer the below cod example for your reference.


 

<SfGrid TValue="Order" @ref="GridInstance" AllowPaging="true" Toolbar="@(new List<string>() { "Search" })">

    <SfDataManager @ref="dm" Url=http://localhost:64956/odata/books Adaptor="Adaptors.ODataV4Adaptor"></SfDataManager>

    <GridEvents OnActionFailure="ActionFailureHandler" TValue="Order"></GridEvents>

    <GridColumns>

        <GridColumn Field=@nameof(Order.guid) HeaderText="GUID" TextAlign="TextAlign.Right" Width="120"></GridColumn>

        <GridColumn Field=@nameof(Order.Title) HeaderText="Title Name" Width="150"></GridColumn>

        . . .

    </GridColumns>

</SfGrid>

 

@code{

    SfGrid<Order> GridInstance;

    public SfDataManager dm { get; set; }

    protected override void OnAfterRender(bool firstRender)

    {

        base.OnAfterRender(firstRender);

        RemoteOptions Rm = (dm.DataAdaptor as ODataV4Adaptor).Options;

        Rm.EnableODataSearchFallback = true;

        (dm.DataAdaptor as ODataV4Adaptor).Options = Rm;

    }

    . . .

}


Please get back to us if you have any concerns.


Regards,

Monisha




AN Andrew June 7, 2022 08:40 PM UTC

Hi,

Noticed now on expanding this concept that any nest class list does not return any values.

I updated the one sample to have 2 nest lists, one type Nest that has an int Id and string value, and the other Nested2 that is a string list. The string list is returned on the class Nest list is not.


Inspecting the network in devtools shows as well that the value is not returned to the client, but shows in the controller with all the data. It is using ODataConventionModelBuilder so it should work with no issues, is there something needed to be specified in the SfDatamanager?


Thanks


Attachment: SSample154263236_8e38e4d.zip


VN Vignesh Natarajan Syncfusion Team June 8, 2022 02:02 PM UTC

Hi Andrew,


Thanks for the update.


Query: “ is there something needed to be specified in the SfDatamanager?


Yes, to retrieve the complex object from the ODataV4 service, the $expand query must be sent along with the server. So we suggest you to achieve your requirement using the Query property of Grid. Refer to the below code example.


<SfGrid TValue="OrgUnit" AllowPaging="true" Query="Qry" AllowFiltering=true AllowSorting=true AllowGrouping=true Toolbar="@ToolbarItems" >

        <GridPageSettings PageSize="50"/>

        <GridEditSettings AllowEditing="true" AllowDeleting="true" Mode="Syncfusion.Blazor.Grids.EditMode.Normal"/>

        <GridSearchSettings Operator="Operator.Contains"></GridSearchSettings>

        <GridFilterSettings Type="Syncfusion.Blazor.Grids.FilterType.Menu"></GridFilterSettings>

        <SfDataManager Url="odata/Products" Adaptor="Adaptors.ODataV4Adaptor" Key="Id" DataType="OrgUnit" />

        <GridEvents TValue="OrgUnit" OnActionFailure="@ActionFailure"></GridEvents>

        <GridColumns>

            <GridColumn Field="@nameof(ou.Id)" Width=120 IsPrimaryKey="true" IsIdentity="true" />

            <GridColumn Field="@nameof(ou.ParentId)" Width=100 />

            <GridColumn Field="@nameof(ou.HasChild)" Width=100 />

            <GridColumn Field="@nameof(ou.Number)" Width=100 />

            <GridColumn Field="@nameof(ou.Country)" Width=100 Type="ColumnType.String">

                <FilterTemplate>

                    @{

                        var c = context as OrgUnit;

                        <SfDropDownList DataSource="@Country" TValue="string" TItem="string"></SfDropDownList>

                    }

                </FilterTemplate>

            </GridColumn>

            <GridColumn>

                <Template>

 

                @{

                    var c = context as OrgUnit;

 

                    foreach(var n in c.Nests)

                    {

                        <div>@n.V</div>

                    }

                }

                </Template>

            </GridColumn>

            <GridColumn>

                <Template>

 

                @{

                    var c = context as OrgUnit;

 

                    foreach(var n in c.Nested2)

                    {

                        <div>@n</div>

                    }

                }

                </Template>

            </GridColumn>

            <GridColumn Field="@nameof(ou.Summary)" Width=100 />

        </GridColumns>

    </SfGrid>

    <span class="error">@ErrorDetails</span>

@code {

    public static List<string> val = new List<string>() { "Nests" };

    public Query Qry { get; set; } = new Query().Expand(val);


Please get back to us if you have further queries.


Regards,

Vignesh Natarajan


Loader.
Up arrow icon