The Owners API doesn't directly return a list of HubSpot User; it returns a list of Owners. The distinction is somewhat subtle, but important. Users and Owners don't always have a 1:1 correlation (e.g. Salesforce users can be owners).
That said, there is no 'Users API', so it's not currently possible to pull a list of deactivated (or active) users. If you have the inclination, I would encourage you to check out the Ideas Forum on the HubSpot Community. There, you can create a post including your use case that product and other customers can see.