GGC: Huge amount of data with filters and recursive relations

[Henning Möller]

I'm trying to use GGC to visualize data provided by an IBindingList data source. There will be a huge number of objects in those lists (up to several millions). Those records are recursivly self-related to each other (parent / child relation). This list won't be added to/removed from/edited from within GGC, just viewed. No records are ever removed, but there may be records modified or added asynchronically. No sorting, no grouping necessary. Nesting is needed however for parents and their childs.

My first prototypes seem to struggle as soon as the control uses reflection to access the records' data. This happens for example if I enable filtering for records with Owner == -1.

Okay, if I leave it to GGC to determine the field values, I understand that there's no other way than reflecting those values. But I replaced the FilterCondition with a custom type and implemented a QueryRecordMeetsFilterCriteria event handler which accesses the record value without using reflection. This made the filtering process perform pretty okay, but I noticed that GCC now caches each and every record _prior_ to calling the filtering method.

Is there a way to prevent GGC from caching record values at all?

And given my recursive relation, how can I implement nesting tables for a variable number of levels within GGC? Any samples for that, especially using IBindingList data sources? There's no real database engine below all this. There's a high performant memory paging system, instead.

Every hint highly appreciated.

[Henning Möller]

In the meantime, I was able to solve some of my problems.

To accelerate the top level filtering, I provide a filtered dataset only containing the top level records. Still I wonder why I can filter my list within about 200 ms while the solution using the custom filter method needs some 30 seconds for the same job.

To implement the self-related nesting, I followed your "SelfRelations" sample found in some other thread.

This works, but it doesn't perform too well. It seems obvious that the relation given cannot perform well by design. There are about a million objects in the list. To determine which children belong to which parent, using the normal ParentId realtion is not appropriate. But my list is two way linked, so each parent knows at least about a range in which its children will be found ("FirstChild" and "LastChild" keys). So what I'm looking for is a way to provide the children for a given parent by custom code. I think this can be done by creating a custom engine and override the CreateChildTable() method. But what exactly do I have to do here?

In general, I need a solution which uses lazy evaluation consequently. It should only loop over the elements which are currently wihtin the view range. By no means at all it may loop over all those objects, because no matter for what reason it does it, it'll take minutes to complete.

[Administrator]

Hi HMoeller,

Thank you for your interest in Syncfusion products.

1. Accelerating filtering with large number of records.

You can accelerate filtering in custom collection IBindingList when large number of records are used by using filtering TextBox.

This can be achieved by using following steps.

1) Change the celltype of FilterBarCell to TextBox in QueryCellStyleInfo event handler.
2) In TableControlCurrentCellChanged event, you can enable filtering by applying typed text as filtered expression.

The following code snippet illustrates this.

[C#]

private void gridGroupingControl1_TableControlCurrentCellChanged(object sender, GridTableControlEventArgs e)
{
GridCurrentCell cc = e.TableControl.CurrentCell;
GridTableCellStyleInfo style = e.TableControl.GetTableViewStyleInfo(cc.RowIndex, cc.ColIndex);
if (style.TableCellIdentity.DisplayElement is GridFilterBarRow)
{
string s1 = cc.Renderer.ControlText;
int rowIndex = this.gridGroupingControl1.Table.DisplayElements.IndexOf(style.TableCellIdentity.DisplayElement);
int colIndex = cc.ColIndex;
string colName = style.TableCellIdentity.Column.Name;
string expression = string.Format("[{0}] LIKE '{1}*'", colName, cc.Renderer.ControlText);
if (style.TableCellIdentity.Table.TableDescriptor.RecordFilters[colName] == null)
{
style.TableCellIdentity.Table.TableDescriptor.RecordFilters.Add(new RecordFilterDescriptor(colName, expression));
}
else
{
style.TableCellIdentity.Table.TableDescriptor.RecordFilters[colName].Expression = expression;
}

}
}

Here is a sample for your reference.

http://websamples.syncfusion.com/samples/Grouping.Windows/F70011/main.htm

2. Self referencing Nested table.

Please provide us more information on how do you want to provide Self-referencing relations between two fields for implementing nested table that would help me to provide you an exact solution.

Regards,
Jaya

[Henning Möller]

Thank you very much for offering your assistance.

My usecase in more detail:

The GGC will be used to browse a very long list of so called "containers". Each container can be a "top level container" (i.e. has no parent) or a "sub level container" with some other container as its parent. For this reason, each container carries a "Key" and an "Owner" field. In addition to this master-detail-relation, each parent container carries the keys of its first and last child containers.

The depth of these relations varies between 0 (top level container without any children) and about 5 (5 generations of parents and children).

The number of all containers will easily reach about a million, sometimes even more. I'm looking for a solution which is completely independent of the number of containers.

Currently, I loop through all of those containers once (to create the list of top level containers). That's acceptable - and there's no way around this, I guess. I can do that outside of GGC, thus preventing reflection.

But those nested tables seem to loop through the records many times when counting the number of records for each child table or when filtering for all containers for a given parent. I don't really know exactly.

I would like to provide my own way to collect all children for a given parent. My guess is that using "FirstContainer" and "LastContainer", I can do that a lot more efficient than GGC can do it. But at the moment, I've got no idea about how to do it.

I should also mention that this list of containers nor its' data won't be changed from within GGC. But the list may be added to from the outside, asynchronically. And GGC should reflect these changes "online".

[Henning Möller]

Any more info you need to answer my question? Please let me know.

[Administrator]

Hi HMoeller,

Sorry for the delayed response.

I am working on the sample to get the collection of child tables for given parent ID in IBindingList datasource and will update you with details soon.

Thanks for your patience.

Regards,
Jaya

[Administrator]

Hi HMoeller,

If you want to retrieve the First and Last child details for given parent, you can iterate through the record collection
and display the First and Last child details for particular parent.

Please refer the below sample that demonstrates the same.

Sample details

BindingList with nested table is bound to GridGroupingControl. Parent - child relation is defined using GridRelationDescriptor.
To retrieve the First and Child details , you can check whether the grid record has NestedTable with child records and then display only the First and Last child record details.

The following code snippet is used to display the First and Last child details of first Parent record.

GridRecord rec = this.grid.Table.Records[0] as GridRecord;

if (rec.HasNestedTables)
{
int nestedTableChildscount = rec.NestedTables[0].ChildTable.Records.Count;

GridRecord nestedFirstrec = rec.NestedTables[0].Records[0] as GridRecord;
GridRecord nestedLastrec = rec.NestedTables[0].Records[nestedTableChildscount-1] as GridRecord;
Console.WriteLine("Last Child record "+nestedLastrec.GetValue("Field3"));
Console.WriteLine("First Child record "+nestedFirstrec.GetValue("Field3"));
}

Sample reference

This sample displays the First and Last child details of first parent record.

http://websamples.syncfusion.com/samples/Grouping.Windows/F70011/ChildDetails/main.htm

Similarly for large set of records, you can iterate though record collection and display the First and Last child details for given parent.

Please let me know if this meets your requirement.

Regards,
Jaya

[Henning Möller]

I doubt that I made myself perfectly clear.

Your approach assumes that GGC has already created the tables / nested tables. But this process doesn't perform well for millions of weakly indexed records, so I'm looking for a way to prevent GGC from doing this by itself. Instead, I'd like to provide the children for a given parent for GGC to just display them in a nested table.

I still think that the key for this is to provide a custom Engine implementation and override the CreateChildTable() method. I expected to find a sample solution for this approach.

I need a solution which is completely independent from the number of records. It must perform the same no matter whether there are 15 or 15,000,000 records. So GGC may never ever loop over all available records nor may it ever count those records for whatever reason.

[Administrator]

Hi HMoeller,

I have been consulting development team in this regard and will get back to you once I hear from them.

Thanks for your patience.

Regards,
Jaya

[Henning Möller]

Can you give an estimate about how long this will take?

[Administrator]

Hi HMoeller,

Sorry for the delay in getting back to you.

GridGroupingControl is a strong bound grid and it allows you to create Parent - child relation through keys. So we higly recommend you to use VirtualGridControl instead of strong bound grid for full customizations.

Please also refer the following link for VirtulGroupingGrid options.

http://www.syncfusion.com/support/forums/message.aspx?MessageID=52657

Thanks for your patience.

Regards,
Jaya