Parental Authority of my child

Good Day to you, I want to ask for your opinion. I have a 1yr old son and the father and i aren't not married. I'm going abroad in a couple of days. at first, my parents and i had an agreement that they(them and my baby's father) will share custody of the child meaning that my child will stay for a couple of days with his father and another with his grandparents but now they are telling me that the father cannot take the baby, he can only come and visit the child in their house.
i'm asking if they can invoke parental authority on my child while i'm away? because i want for them is to share custody of my child. what legal actions should i take?