Financial well-being is deteriorating overall but middle-class households have been particularly hard hit.