TL; DR: Postgres will just fail to start if shmmax is too small, and the logs will reflect that. On Linux, you'd say:

  # sysctl -w kernel.shmmax=N

Where N is the desired maximum shm segment size. (You'll also want to increase shmall, to accommodate shmmax, and any other shared memory requirements you may have.)

EDIT: Of course, to make the change persist across reboots, you'll also want to add it to /etc/sysctl.conf. I've forgotten that part more than just in this comment...