Does backward time travel violate the conservation of energy?

The law of conservation of energy basically states:

the total amount of energy in an isolated system remains constant over time (is said to be conserved over time).

Now imagine that a time traveler, Welles, goes into his time machine at time t` in order to go back 50 years to time t. We can think of Welles as a bundle of particles with mass, or its equivalent, packet of energy, as special relativity suggests with a inertial reference frame. What happens to that energy packet after t`? The state of the universe before t` had a constant amount of energy but after t`, its total energy is less by the amount composed of Welles. The inertial reference frame which traced his path through spacetime simply ends (where it goes backward in time, perhaps "doubling back") before that time. Thus the total energy of the isolated system (the universe) after t` is less than the time before making the total energy of the universe not constant over time.

Now when Welles appears at t, the total energy of the universe would appear to be increased after t (but before t`) by the amount composing of Welles.