Yes but at that point he’d been in prison for like 20 years and didn’t have access to fight him. I feel like Saejima was a proper friend to Majima but Kiryu is definitely more of a friendly rival.
On top of the fact Majima challenges you to a fight at LEAST once per game usually just for fun.