Whats the rule?

Back Turner...

Our opponent hit a first serve let and the repeated serve landed out. My partner mistakenly started to walk back to return on his side when our opponent hit a second serve. I returned the serve in play, but our opponents stopped the point saying that it was a let because my partner was walking away with his back turned. We agreed to play the point over. What is the rule?

If your opponents didn't like your partner's movement during the serve, then why did they serve? That was the right time for them to stop and wait. There's no prohibition against what your partner did (The Code, Item 36). If the receiver is ready, then the partner is also presumed ready. No let. It's your point because they stopped play after your return landed in.