Hi Alomgir,
Thank you for your interest in Syncfusion product.
If you want to change the custom dialog in GridGroupingControl, then please try using CustomFilterBar in GridGroupingControl to achieve this behavior.
Please refer forum thread in below link, it will be helpful to you show the custom dialog in custom filter option.
http://www.syncfusion.com/support/forums/grid-windows/81375/Change%20syncfusion%20GroupingGrid%20Filter%20optionsPlease let me know if you have any queries.
Regards,
Lingaraj S.