implementing paging using virtual grid

[Administrator]

Hi. I wish to display data from a text file in a grid. This I can do. Now, the problem is the text files are very large (over 500 megs in some cases... no kidding). So, in order to make this efficient, I need to implement some kind of paging where I only load and display a chunk at a time and dynamically read and load chunks from the file as needed... I was looking for an appropriate scroll or row change event to figure out when new data needs to be read from the file. I have not had any luck yet.

Any pointers/suggestions would be greatly appreciated. Thanks in advance.

[Administrator]

Here is a little sample. It is implemented using the virtual grid. In QueryCellInfo, it checks for the data availability and loads the data as you scroll over. At the beginning of QueryCellInfo, there is a call to a DataSource method that ensures the requested row is available. If it is not, the DataSource loads a 200-count range of records with the requested record in the middle. In this sample, loading new data creates a new 200 row datatable using code. In a real application this table would come from some kind of database query to retrieve a batch of records containing the requested one. In the sample, all this happens very quickly. In a real application, accessing the database will likely be an expensive operation, so bigger page sizes may make sense. Or, you could try to use multiple threads to ''preload'' anticipated batches on different threads so they would be available when needed. Of course, this adds complexity to the coding to make sure the threads are interacting properly. But the basic idea would still be the same. At the start of QueryCellInfo, you somehow have to make sure the requested data is available. If it is not, then you have to get it.

[Administrator]

Thanks Clay. That did the trick. Now I have another related question... The way I currently initialize the virtual grid, I have to specify a certain number of rows and columns to actually fit all the millions of rows... so even if there are only 20 rows to be displayed from the file, I still have millions of empty cells being displayed... I was wondering if it is possible to dynamically grow the virtual grid size as it is being populated? I tried to pass in the row and column size into "GridRowColCountEventArgs" event as the table grows... but that does not seem to work. The event is only being fired in at initialization and thats it. >Here is a little sample. It is implemented using the virtual grid. In QueryCellInfo, it checks for the data availability and loads the data as you scroll over. > >At the beginning of QueryCellInfo, there is a call to a DataSource method that ensures the requested row is available. If it is not, the DataSource loads a 200-count range of records with the requested record in the middle. In this sample, loading new data creates a new 200 row datatable using code. > >In a real application this table would come from some kind of database query to retrieve a batch of records containing the requested one. In the sample, all this happens very quickly. In a real application, accessing the database will likely be an expensive operation, so bigger page sizes may make sense. Or, you could try to use multiple threads to ''preload'' anticipated batches on different threads so they would be available when needed. Of course, this adds complexity to the coding to make sure the threads are interacting properly. But the basic idea would still be the same. At the start of QueryCellInfo, you somehow have to make sure the requested data is available. If it is not, then you have to get it.

[Administrator]

In that sample, the external datasource can grow dynamically. It does not have to start out at 1000000 rows.

For example, if you change the 1000000 to 300, and change the one method as listed below, the datasource will ''grow'' 400 rows at a time as you close in on the end of the data.

In your real code, when your external datasource grows, all you should have to do is call grid.ResetVolatileDate and maybe grid.UpdateScrollBars to have the newly added data known to the grid. In a true virtual grid, everything is dynamic in this sense.

public bool EnsureDataAvailable(int rowIndex)
{
	if (rowIndex >= StartRow &&  rowIndex <= loadedEndRow)
		return true;
        //new code  ///////////
	if(this.AbsRowCount < rowIndex + 200)
	{
		this.AbsRowCount += 400;
		this.grid.Model.ResetVolatileData();
		this.grid.UpdateScrollBars();
	}
        ////////////////////////
	this.ActiveTable = this.LoadAt(rowIndex - this.grid.Model.Rows.HeaderCount - 1);
	return true;
}

[Administrator]

Hi,

Interesting... how would you do something like this in a GroupingGrid?

(that should keep you busy)

[Administrator]

I do not think you will be able to do this in a grouping grid. Virtual paging probably does not make sense in the grouping grid.

The grouping engine has to have access to all the data to know how to group it, summarize it, filter it, etc.

[Administrator]

In your demo if you put more than 10 cols you get an out range exception.
whys that?

[Administrator]

Cool. That works just great. Now, another issue that comes to mind is resource usage... Lets say if I go straigt to the millionth row of the table, does that mean the grid now contains a million+ rows? I know that only the page to be displayed is populated, but has the virtual grid control now taken up a big chunk of my memory?

[Administrator]

It is because the data object is using code like this:

this.grid.Model[0, i + 1 + this.grid.Model.Cols.HeaderCount].Text = ....

This assumes the grid has allocated space in its internal data to hold such column information. If you want to use such code, you woul dhave to not make teh colcount virtual (by handling QueryColCount. Instead, you would explicilty set grid.ColCOunt to teh proper value (usually once at teh beginning). This will cause the grid to allocate teh space needed for the above code to work. (The reason it works up to 10 is because the original grid defaulted the ColCount to 10, allocating space for 10 columns.) If you want to have the columns virtual, then you woul dhave to do something else to manage teh headers. Putting them in the grid''s internal data makes it easier to support column moving.