I am so sorry to hear that you had to give your grandson back to his natural father. My heart goes out to you. How dare he takes the child out of the only home that he has ever known and was so loved and given the understanding that he needed in his lillte life. You do have rights, go to court and petition the courts for grandparents visitation. In most states that is allowed.
We too are raising our grandson who is now 12. he has been with us since October of 2004 when his mother decided she no longer wanted to be a mother, she left for work on a Friday and never came home. The child's father does have visitation with his son but choices not to see him unless his mother calls him.
My prayers are with you. Your grandson will always remember the beautiful times that he had with his Grandfather. He will never forget, he will always love you.
I feel very bad for the grandson who know has to learn a different way of life with his father who will no keep the same activities that you had for your child.
Grandma Riley