To use custom columnstyles in the designer, you need to do three things.
1) Derive a CustomColumnStyle to implement the functionality you want.
2) Derive a DataGridTableStyle class and add a new GridColumnStyles property that uses this derived CollectionEditor. This GridColumnStyle hides the baseclass member.
3) Derive a DataGrid and add a new TableStyles collection that uses your derived tablestyle.
Both steps 2 and 3 will require you to derive a CollectionEditor and override CreateNewItemTypes to use the derived classes from each step in the designer.
Here is a sample project showing how you might do these things.