Pursuit of the Greatest Happiness

According to John Stuart Mill in his Utilitarianism, "An existence exempt as far as possible from pain, and as rich as possible in enjoyments" is the ultimate end (or goal) of human conduct, and thus morality. In this way, Mill seems to partially agree with Singer, who claims that morality begins when suffering does, and its end is to prevent or ameliorate that suffering. Mill is careful to state that creature comforts and sense pleasures are not all he means by "enjoyments" nor are they the greatest among those enjoyments available to humanity. He instead points to having a "cultivated mind" as the ultimate source of satisfaction. Mill was just as keen as Shanahan is to keep his job. 
I also noticed that Mill rejects the notion of psychological egoism. "Men often, from infirmity of character, make their election for the nearer good, though they know it to be the less valuable." (Mill ch 2) This is a little surprising to me, and it makes Mill's philosophy a little trickier, though not untenable. If the ultimate end of morality and human conduct is to achieve the greatest possible happiness, but humans (at least those of infirm character) don't always choose the greatest happiness, how may we be expected to hold to this principle? Mill's solution seems to be to engender nobleness in everyone, so none will be of infirm character. The assumed conclusion is that if everyone were of a noble mind, they would all come to the same conclusions. Wouldn't this engendering of nobleness (presumably a compulsory sort of thing) constitute an impingement of liberty on those who are of infirm character? Does Mill view this as an acceptable sacrifice, as "those who are equally acquainted with, and equally capable of appreciating and enjoying, both, do give a most marked preference to the manner of existence which employs their higher faculties[?]" (Mill Ch 2) But I think my main objection would be to the premise that everyone would agree on what the highest happiness was. I believe that people have preferences that differ.  If I love Kanye West's Music, but others find its existence actively depreciates their quality of life, who gets the final say on whether or not "My Beautiful Dark Twisted Fantasy" is an objective good? I think that due to the existence of preferences, the establishment of a universal standard scale for happinesses would either be impossible or it would fail to account for the nuances of human taste. Some things are just incomparable.