What do I do with people asking me to manage 10k - 50k of their money? Do you turn them away?

You have to see the potential. Sometimes big clients want to start with small amounts to test the waters and then invest more. Also, you can receive good referrals from a 50k clients. However, it all comes down to prospecting.