The Moral Happy Helper

Kant relies a lot on people agreeing with him in the fact that actions are only truly moral if they are done without any other ulterior motive. However, when the example of the happy helper was brought up in class I could not help myself in disagreeing with the point of the example. A brief refresher of the case is that there is a person who really enjoys helping people and thus does it, but in an alternate case, this same person does not enjoy helping people anymore but now  instead helps people out of duty. Kant claims that it is a generally accepted view that actions are not truly moral unless they are done purely out of duty and with no ulterior motives  behind these actions. I did not agree with this statement. I feel that although I may have ulterior motives,like happiness, behind a "good" action I perform does not take away from its moral goodness. Unless my ulterior motive is something harmful or false, I do not believe that any moral value should be taken away from me. The example of the happy helper I can twist to show my point. The happy helper is no longer happy helping people but still does it just because they feel like they have to. How does that add moral value to their helping? Just  because they  may be sacrificing a little more since they do not enjoy it? I think that it would be ridiculous to tell someone that enjoys helping people that they are immoral since we will never be able to  truly know if they are helping to help or their own pleasure is their only motive. When talking this out with my friend we discussed how we agree you will never know what someone's true ulterior motive is, but we believe that that is on them. Morality doesn't come from what other people deem your actions, but from yourself.