Before drawing with the Rounded Rectangle tool, [‘ or ‘]’ decreases/increases the corner radius by one pixel. Adding the Shift key will decrease/increase the radius in increments of 10. Likewise, when using the Polygon tool, ‘[‘ or ]’ decreases/increases the number of sides by one. Adding the Shift key will decrease/increase the number of sides in increments of 10.
When using the Custom Shape tool, ‘[‘ or ‘]’ goes to the previous or next shape in the Custom Shape Picker (in the Options bar). Adding the Shift key selects the first or last shape in the Custom Shape Picker.
Instead of clicking the gear icon in the Options bar to change settings specific to each shape tool, you might find it easier to click in the image area with the shape tool selected to display options for the tool.
After drawing with the Rectangle, Rounded Rectangle, or Ellipse shape tools, you can use also use the Properties panel to change the Live Shape attributes.