If Abraham had TWO sons (Ishmael & Isaac), why was he told to sacrifice his ONLY son (Isaac)?

Genesis 15 narrates how God makes a promise to Abraham that Sarah will have a son. In chapter 16, Sarah lacks of faith and tells Abraham to have sex with their Egyptian servant, and a son was conceived from that, named Ishmael. Later, God makes his promise true by providing the son conceived in his marriage under His promise, Isaac. The bible tells how when young, Ishmael was making fun of Isaac which Sarah did not like and had Ishmael and his mother dismissed from their home. So, by the time that Abraham was asked to sacrifice his son, the only son he had left was Isaac, who was indeed his only legitimate son.