Debbie - there are a couple of things that I would recommend you do and some questions I have for you:
1. What type of visa did you arrive in the U.S. on?
Visa waiver?
B-2?
K-1?
2. You said you got married. I assume you did this legally and obtained a marriage certificate? Correct? Did you ever file any other papers?
If you just left after being married legally on paper in the U.S., then you are technically married even though you never filed any citizenship papers. Immigration office will never come after you btw. So, don't worry about that. That is just BS from your "husband". Obviously he wants you back.
To make sure you do not get any trouble when trying to enter the U.S. on vacation next time, I would make sure I would legally be divorced from your husband. Don't just assume that because you left the U.S., you are automatically divorced.
If questions, go to the U.S. embassy. Ask them, and also include what to do if the "husband" does not want to divorce you.
Life Lover