You're conflating separate issues. Part of the infrastructure that would need to be built out is more power plants, regardless of the fuel source. Old, dirty coal plants should be retired and replaced with more efficient plants - coal among them. Fuel mix diversity is important. I'm not arguing in favor of eliminating coal. In fact, if this were a discussion about the environment, my views would have me loudly denounced as a "denier."
Line losses are already factored into the cost of the electricity, it's not like plugging in your EV suddenly creates magical "losses" that make generating and distributing the electricity so much more expensive. Given that, it's not like you are paying for a dollar's worth of electricity and only getting 30 cents' worth. The inefficiencies of the grid are already built into the equations - they're a known quantity, and it's STILL that much cheaper for a power plant to produce energy and distribute it with efficiency losses than it is for you to burn your tank of gas. There are inefficiencies in all the processes that occur between the oil being pumped out of the ground to the time it's pumped into your tank, the difference being that the EV motor is highly efficient at converting the stored energy (batteries) into motion, whereas gas motors are still incredibly wasteful.