It actually makes sense after you read the 1st chapter though. He is the crown prince, so after he gets his memory back, he divorced her and then send a carriage over to take her as his concubine (but missed her since the duke got her first). They have a large gap in status so she couldn't be the main wife