I recommend iStat Menus, personally. I'm still using version 3 of it on 10.6, but version 4 supports up thru 10.9. They don't have a screenshot of it, but you can see they support GPU monitoring from the version history page:
Added support for monitoring multiple GPUs.
Improved GPU monitoring.
To specifically address your other points:
I use SpeedFan:
Records GPU temperature (log)
When the temperature reaches a predefined threshold (say ~80 deg C) you can configure it so that it starts off an alarm - in the form of a popup message, beep or even email.
Free but not portable
In addition to GPU, it also monitors CPU, HD and Fans.
It does not
Optimize my programs during gaming to reduce GPU ...
The only GPU tool I know that does this is Real Temp. I used it shortly and it works really well. It has an extremely small footprint and has a lot of features for monitoring your hardware.
Portable and free
High temperature alarm and shutdown feature based on CPU or NVIDIA GPU temperature
Runs on Windows XP up
Only works on Intel CPUs and ...
I use GPU-Z. I got it from here.
Their authors explain advantages of their software as
Supports NVIDIA, ATI and Intel graphics devices
Displays adapter, GPU and display information
Displays overclock, default clocks and 3D clocks (if available)
Includes a GPU load test to verify PCI-Express lane configuration
Validation of results
I believe this can be accomplished with the Intel Performance Counter Monitor (PCM) tools: https://software.intel.com/en-us/articles/intel-performance-counter-monitor. As of v2.5 it supports PCIe monitoring on Xeon E5 processors. (ref) Looks like subsequent releases added support for more processor types: Haswell, Broadwell, Skylake.
I use iStat Menu Meters for this on Mac OS: I do a lot of 3D work and renders, which can tax my system pretty heavily so I keep track of CPU, GPU, & RAM.
Does what it says on the tin - super useful, small footprint and fast.
Highly recommend - no I'm in no way affiliated - just a longtime user.
If you prefer FOSS solutions (what I personally do), you can give XRG for Mac a try.
It displays CPU, GPU, memory, battery, temperature, network and disk graphs. And even 2 more not-related to hardware: weather and stocks.