Is it constitutional to 'pack' the supreme court with more justices?

The Supreme Court didn't always have nine justices, in fact, it had more. But what changed and made nine the number that stuck?