Denial of Visitation Rights

I have a 4-year old son who i currently entrusted to my parents now since i am working in mAnila. My ex-bf wanted to see his son that's why i let him borrow my son every weekend. I don't know what covers the financial support he is suppose to provide his son (bearing his surname) but he provides milk, vitamins and medicines (in times my son gets sick). We had an informal agreement (thru sms) but he always violates it by not returning my son at the agreed time and bringing him anywhere (to other provinces) without any notice. I cannot conclude that he brainwashes my son since i do not have any evidence but everytime i ask my brother to fetch my son at my ex's house (since they often don't return him unless we fetch him), my son is acting violently and do not even want to go home, locking himself in their room which is kinda bit weird because before, he is very close to my mother and do not even want to sleep in other house. This makes me worried so much and makes me want to cancel the visitation we had agreed. Is it enough basis to deny him of his visitation rights (i assumed he has) and considering my ex has another live-in partner in their house while also having other affairs to other women (in which i have evidence and credible witnesses)??In addition to this, his mother also requested me to handle all the support to my child because his son had already supported much..I wanna hear your insights regarding this matter...Thank You.