Can my sons father take my child from me?

I was divorced in the state of Tennessee. The day of the divorce my sons father moved to Texas. The divorce decree said that he got my son every other weekend. Of course this didn't happen because he lived in Texas. He saw him 3 weeks a year, 2 weeks in the summer and 1 at Christmas. This has been going on for 4 years. I recently moved to new York and have been in the process of trying to get child support transferred. I have lived in new York 4 months. My son has recently gone to visit his father in tx and my sons father has recently gotten engaged and has told me that he is going to try and take my son from me. He says I'm an unfit mother which is totally untrue. What are my rights and can he keep him from me? I'm totally at my wits in right now and don't know what I need to do.