The Boleyn's and King Henry VIII?

I've just finished reading a book about King Henry VIII and the Boleyn's. King Henry had a son by his mistress, Bessie Blount, named Henry Fitzroy who was officially recognized as the child of the King. Once again, while still married to Katherine of Aragon, he took Mary Boleyn as a mistress and had two children: Catherine and Henry (although they took Mary's first husbands last name of Carey). After the King's divorce from Katherine he married Mary's older sister, Anne and had Elisabeth. While his third wife, Jane Seymour, gave him the son he had wanted so desperately - Edward. After Jane died, King Henry named succession to the thrown of his son Prince Edward and his heirs should he have a son, then Mary, then Elisabeth. Henry made his child, Henry Fitzory, Duke of Richmond - why were Mary's children; Catherine and Henry never claimed as the Kings? Is it b/c they were raised by Mary and her second husband Sir William Stafford?