Initially, I'd like to ask, if when on the way back up, you have the two keyframes set to only one frame apart. If so, you shouldn't see any movement by that object through the camera frame at all. Such as this:
I would animate the object's visibility to 0 during the frame(s) in which it passes the camera on its way back up.
From Autodesk's Knowledge Network:
Select Object and open Object Properties from Right-Click>Object Properties.
In Object Properties, General tab - under Rendering Control, set properties to be managed ‘By Object’ by pressing the button. The Visibility property will now be accessible and can be animated with AutoKey enabled. Note: The general Display Properties for object can also be set to ‘By Object’ from this window if local control is required.
Another solution would have it move off screen on the x axis, up, back across and then fall again so that it passes the camera only in the downward direction. Perhaps have it follow a path shaped like a capital "D" where it falls past the camera on the vertical part of it and semi-circles around, out of camera view.
Lastly, but a more complex solution, would have you utilize a particle system to repeatedly emit your object (at regular or irregular intervals) in one direction and set a life that expires soon after it passes the camera.