Built in shape editor only allows base-colors, black, white and one changing color to be used.
Alpha values will be applied to each agent shape element(circle, square, polygon etc.) only if its color is RGBA.
Shape data, including colors associated with each shape element, can be edited by opening .nlogo file with any text editor. The colors are represented as signed 24bit integers.
Conversion code i made a long time ago:
to-report signed-integer-to-rgb [input]
set input 16777216 + input
report (list (int(input / 65536)) (int(input / 256) mod 256) (input mod 256))
end
to-report rgb-to-signed-integer [rgb-list]
report item 0 rgb-list * 65536 + item 1 rgb-list * 256 + item 2 rgb-list - 16777216
end