Can I get fired for dating a temporary employee, me being a supervisor?

[Angela M]

I work for temporary service agency. And three yrs ago I was transfered to one of our accounts because of the volume of temps we serviced them with. My job here is to check everyone in and check everyone out. I do not decide anyone's pay nor how long is their time here with the company.
I started dating a guy who works through us (the agency) about 6months ago. My mistake was not to discuss this with my boss. Now that it's getting serious and people have began to talk my boss wants to talk to me about conflict of interest. How can this effect me? Can I get fired for this?? Please help! I have to meet with boss today:|