The association said the election result had “verified” its hard Brexit fears