If the said defamatory message was not communicated to other persons, there is no Oral Defamation nor Libel. The main element of the offense of Defamation/Libel is the besmirching of the reputation of the offended party. One's reputation does not refer to one's sense of self-worth; it is how others regard him to be. Hence, such derogatory accusation did not constitute Defamation or Libel.
Under Article 353 of the Revised Penal Code of the Philippines, libel is defined as a public and malicious imputation of a crime, or of a vice or defect, real or imaginary, or any act, omission, condition, status or circumstance tending to discredit or cause the dishonor or contempt of a natural or juridical person, or to blacken the memory of one who is dead.
Thus, the elements of libel are: (a) imputation of a discreditable act or condition to another; (b) publication of the imputation; (c) identity of the person defamed; and, (d) existence of malice.
The difference between Oral Defamation and Libel is that in Libel the defamatory statement is made "public".
If the following conditions are present, you can sue him for damages in a civil suit. You can use Article 26 of the Civil code as your basis:
Art. 26. Every person shall respect the dignity, personality, privacy and peace of mind of his neighbors and other persons. The following and similar acts, though they may not constitute a criminal offense, shall produce a cause of action for damages, prevention and other relief:
(1) Prying into the privacy of another's residence;
(2) Meddling with or disturbing the private life or family relations of another;
(3) Intriguing to cause another to be alienated from his friends;
(4) Vexing or humiliating another on account of his religious beliefs, lowly station in life, place of birth, physical defect, or other personal condition.