What would my ex have to do in order to take my daughter from me if I take him to court for custody?

My daughter's father and I were never married. I got pregnant shortly after turning 18, and we were together until I was almost 24 years old, at which time I asked for a split. We split amicably, and would still be friends today if not for certain outside circumstances. At the time that we split, we agreed that neither of us would have custody, that we would work things out between the two of us, that each of us would have her half of the time, child support would not be necessary, etc. Since we split up, I have had a lot of really bad luck, mostly in jobs and money, forcing me to allow my daughter to live with him and his new wife (the previously noted outside circumstance). Over the last year and a half since we split, I have seen my daughter less and less, along with him ignoring my phone calls (which he denies) or not returning my messages. I don't want to go back on my promise to him, but when I attempt to work things out with him to see my daughter more he usually says or does something to put me off. Along with these issues, I am currently taking his wife to court for slander and defamation of character. She has publicly defamed me since they met, and now has taken it to the extent of attempting to get me fired from my job as a college instructor. This woman is a bully, and my daughter is terrified of her. She has told my daughter on multiple occasions that "Your mommy doesn't love you, because she doesn't buy you toys" or "Your mommy doesn't love you because she doesn't take you to _______(insert any place)" I love my daughter very much of course, but do not spoil her as they do, buying her new toys every week or taking her on long trips several times a year, but I do provide everything she needs, as well as fun activities and toys for her birthday/christmas, and at other random times. Every time my daughter is with me, she talks about how much she hates her step-mother, how she's scared of her. She cries when she talks about being at home, and begs me to let her come live with me instead. Or begs for me and her father to live together again, which is not going to happen. She has also lied to her teacher about being sick when she stayed with me because her step-mother called her a liar and told her she would punish her. So my question is, what should I do? She is in no way neglected at her father's house. She is always well fed, usually well dressed (although his wife tends to put her in ragged or outgrown clothes when she is coming to my house), has her own room, no abuse, etc. However, she's not happy there, and he is not keeping up his end of a verbal agreement. I am living in a stable home, have a stable job, etc. His wife keeps threatening to take me to court for custody (which I know she can't legally do) but threatens to throw all of her money (which she has quite a bit more of than me) at the case to win. I am at my wits end and want to take the case to court for joint custody, with me as the primary parent. At the same time, I have done some things that I'm not proud of, although none of it was recently, and I know she's going to bring it up and try to prove me unfit in court. I did have a drug problem after he and I split, but have been clean for almost a year and a half. I have been out drinking a few times over the last year, but always when my daughter was at her father's. I have had 5 relationships in the last 1 1/2 years, but my daughter was never affected by any of these relationships. She also states that when I lived with my mother that I forced my daughter to sleep on a couch, which is untrue, she always had a bed to sleep in, but she chose to sleep on the couch on occasion to watch tv. Which, in my opinion, there is nothing wrong with. My daughter has never been neglected in my care, but I am still scared, because I know what people with money can do, and I don't want to lose my daughter. I want her to be happy and safe, and away from her crazy step-mother. Any advice would help, thanks.