customadapter implementation

[Jasper Vermeulen]

Hello.

I am looking to implement a custom adapter for the grid in react. I have enabled virtual scroll on a large dataset (100.000+) row. I would like to add a custom adapter via the Datamanager, because we have a non-straight forward non-adaptable backend solution. 

However I cannot find any documentation on this. Just a quick mention here: https://ej2.syncfusion.com/react/documentation/data/adaptors#writing-custom-adaptor
Also I am able to read the source code here: https://github.com/syncfusion/ej2-javascript-ui-controls/blob/master/controls/data/src/adaptors.ts

But it does not explain how to implement. I feel like I would need to reverse engineer from the source code to be able to understand it. I am not sure which options mean what, what variables to use and what parameters are send to the functions/methods. Also the documentation of the DataManager is lacking this information.

What can I do?

[Aishwarya Rameshbabu]

Hi Jasper Vermeulen,

Greetings from Syncfusion support.

Upon reviewing your inquiry, we have determined that you need to use a custom adaptor to define the dataSource for the Grid with Virtual Scroll enabled. Custom adaptors enable you to implement custom data processing logic. A custom adaptor extends an existing built-in adaptor from Syncfusion DataManager. Please refer to the sample and code example below where we have extended the UrlAdaptor. In this example, we have overridden the processQuery method to filter the Grid data. Similarly, you can override any method of the base class to customize your data handling.

Code Example:

CustomAdaptor.js

export class CustomAdaptor extends UrlAdaptor {

  processQuery(dm, query, hierarchyFilters) {

    const result = super.processQuery(dm, query, hierarchyFilters);

    console.log(result);

    // Parse data

    const data = JSON.parse(result.data);

    // Create URL request

    const url = new URL(

      result.url,

      typeof location !== 'undefined' ? location.origin : ''

    );

    result.type = 'POST';

    // Append params to url

    if (data.skip !== null && data.skip !== undefined)

      url.searchParams.append('skip', data.skip.toString()); 

    if (data.take !== null && data.take !== undefined)

      url.searchParams.append('limit', data.take.toString());

    // Filters

    const filters = data.filters ? JSON.parse(data.filters) : null;

    if (filters && filters.rules.length > 0)

      url.searchParams.append('filters', JSON.stringify(filters));

    // Return request

    return { ...result, url: url.toString() };

  }

}

Index.js

  function RemoteDataBinding() {

  const data = new DataManager({

    url: 'https://services.syncfusion.com/js/production/api/UrlDataSource',

    adaptor: new CustomAdaptor(),

  });

  const filterSettings = { type: 'Excel' };

  return (

    <div className="control-pane">

      <div className="control-section">

        <GridComponent

          height={400}

          enableVirtualization={true} 

          actionFailure={actionFailure.bind(this)}

          id="Grid"

          allowFiltering={true}

          filterSettings={filterSettings}

          allowSorting={true}

          dataSource={data}

          ref={(grid) => (gridInstance = grid)}

        >

          <ColumnsDirective>

                               …………………

          </ColumnsDirective>

          <Inject services={[Page, Sort, Toolbar, Filter, Edit, VirtualScroll]} />

        </GridComponent>

      </div>

    </div>

  );

}

export default RemoteDataBinding;

Sample: Fpvysp (forked) - StackBlitz

If you need any other assistance or have additional questions, please feel free to contact us.

Regards

Aishwarya R

[Jasper Vermeulen]

Hello,

Thanks for the example. This make things a bit more clear and definitively helps.

I was just wondering if there is anymore documentation to be found on this somewhere?

Kind regards,

Jasper

[Jasper Vermeulen]

Hello,

Thanks.

I have now also overridden the processResponse method:

  public processResponse(

    data: DataResult,

    ds?: DataOptions,

    query?: Query,

    xhr?: Request,

    request?: Object,

    changes?: CrudOptions

  ): DataResult {

    const result = super.processResponse(

      { result: data, count: <COUNT_NOT_KNOWN_YET> },

      ds,

      query,

      xhr,

      request,

      changes

    )

    console.warn('result', result)

    return result

  }

}

However, due to our API restrictions, the count of  the total set is not known until later. And our API only returns the data, not the total count when requesting a data page.

So when insert a given count and then update this later, the scrollbar of the grid does not update. It resets to the size of the current loaded "page". And then it turns into an infinite scroll. Where the scrollbar decreases in size the more data I load.
But this is not desirable for the user, as we should know what is the total size of the set.

Is this a bug?  Or should I set the size of the total dataset some other way?

[Aishwarya Rameshbabu]

Hi Jasper Vermeulen,

We are glad to hear that our solution was helpful. However, you need further information on this topic and are facing an issue with virtual scrolling in the Grid. This issue arises because, when extending 'UrlAdaptor', the data should be returned as an object containing both the result and the count. Currently, your API only returns the data without the count, which is updated later. This causes the Grid to mishandle page requests, affecting its virtual scrolling functionality. Therefore, we kindly request you to specify your exact requirements so that we can provide a more optimal solution. Additionally, please provide the following information:

1. A screenshot of the network tab to understand the data being returned from your API service.

2. The complete Grid rendering code along with any event handler functions used.

3. The Syncfusion package version you are currently using.

4. A video demonstration of the issue you are encountering.

Regards

Aishwarya R

[Jasper Vermeulen]

Hello,

Ok, thanks for your reply. I will provide this later. At this moment it is not a breaking issue for us yet. So priority is low. I will come back to it later. Thanks for your support so far.

Kind regards,

Jasper

[Aishwarya Rameshbabu]

Hi Jasper Vermeulen,

You’re most welcome. Please get back to us with the requested details to validate the issue further from our side.

Regards

Aishwarya R

[Jasper Vermeulen]

Hello,

Let me try to describe the situation I am trying to make work.

So we have a semi-large dataset (up to several 100.000's of records). Querying this takes quite some effort on the backend. At this point, this is a known issue. It is also something we cannot change easily.

We have come up with a way to sort of mitigate this. We can stream the id's of the records requested using sse or websockets. Our current implementation uses server send events and listens to a list of row/record id's being loaded. This takes several seconds (up to 30 in most cases). When we have loaded enough id's, say a 100, then we are going to query a different endpoint (regular REST call) with these id's. We then get back all data to display in the grid. 
Whenever a search, filter, or sort option is used in the grid, it requires the SSE endpoint to be queried again (data might be updated, client-side is not an option) to stream the newly requested (filter, sort, etc.) dataset/id's list. And again when a certain treshold is met, we start to load the actual data. We would like to display that initial page of data, once it is loaded. And at any point in time, when the SSE connection is finished loading all id's, I would like to set a new dataset size/length in the datagrid, so that for the user, it is clear how many data "is loaded". 

On loading a new page, I would like to "detect" that nothing other is requested but a new page (start, end, size, some parameters like that). So that I can skip querying the SSE endpoint and use the in-memory list of id's to get the right page of data.

At this point I am unsure which methods to use from both the grid, datamanager, or custom adapter implementation to make this work neatly. The preference would be to do this with virtual scrolling, not with pagination or inifinite scrolling (we are porting an existing application and would like the user experience to be the same).

In the end I would like to achieve a smooth working solution that enables most of the options the datagrid has available, while still being able to use the SSE endpoint to query data and the REST endpoint to get the data. It is a two-step process.
Right now, I can not get it to work smoothly with using a custom url adapter, datamanager and listening to datagrid events.

Loading the whole dataset into the browser/client is not supported. As we have tested this. We might have 700.000 records, with up to 85 columns. This will break the browser, therefore we prefer to do most querying/filtering/sorting/searching in the backend/server-side.

Could you perhaps assist me in providing insight in how you might approach this problem? So that in time, I might be able to come up with a solution that works for me in this particular case?

Kind regards,

Jasper

[Aishwarya Rameshbabu]

Hi Jasper Vermeulen,

Thank you for providing the detailed information on your implementation.

Upon reviewing the provided information, we have identified that you are utilizing a large dataset and retrieving it through a two-step process involving an SSE endpoint for querying data and a REST endpoint for fetching it. In the Grid, data can be bound using adaptors for remote data, each possessing its own data format. For the UrlAdaptor, the data should be returned in the format of object of 'result' (JSON data) and 'count' (total record count). You previously mentioned that the data count can be obtained dynamically after the records are loaded, which is causing issues with the virtual scroll feature in the Grid. If the data is returned in the specified format for UrlAdaptor, the Grid will process the data based on on-demand requests, loading only the current page records initially and subsequent records as needed through scrolling. Therefore, it is essential to ensure that your server-side processing returns the result and count correctly, handling the waiting process to obtain the count and returning the result in the specified format. This will facilitate all the Grid actions such as scrolling, searching, filtering, and sorting to work smoothly.

Regards

Aishwarya R

[Jasper Vermeulen]

 Hello,

Ok, thank you for your response.

Kind regards,

Jasper

[Aishwarya Rameshbabu]

Hi Jasper ,

You’re most welcome. Please get back to us if you need any other assistance.

Regards

Aishwarya R