How can I make sure I don’t open a second instance modeless dialog that is already opened from my main form

One way to do this is to maintain a list of opened modeless dialogs and check this list before you open a new one to see if one is already present.
If you open all these modeless dialog’s from the same ‘main’ form, then you can use the Owned Forms property of that main form to maintain this list of opened dialogs. Below are some code snippets that suggest how you must go about this. Note that your dialog forms need to be able to turn off the ownership. This is done below by adding an Owner field to the dialog form.

//sample code that either opens a new dialog or displays an already opened dialog 
private void button1_Click(object sender, System.EventArgs e)
{
    foreach (Form f in this.OwnedForms)
    {
        if (f is Form2)
        {
            f.Show();
            f.Focus();
            return;
        }
    } 
    //need a new one
    Form2 f2 = new Form2();
    this.AddOwnedForm(f2);
    f2.Owner = this;
    f2.Show();
}

//code for form2 
public class Form2 : System.Windows.Forms.Form
{
    private System.Windows.Forms.Label label1;

    public Form Owner;

    //.............. 
    private void Form2_Closing(object sender, System.ComponentModel.CancelEventArgs e)
    {
        Owner.RemoveOwnedForm(this);
    }
}