Agreed, day very safe.
Definitely three guys and maybe two can walk it at night and be relatively safe, still not advised, and at night I mean before 10 pm, after that, it's hells gates open, and be ready to punch and be punched, lol.
I've approached taxi's waiting outside HDR, SL, Castillo, Taormina, and just said 1 mil to take me to : one of the other above mentioned places, and never got a debate.
Outside the Castillo is my least favorite place, it's dark, nothing open, I've been hassled and friends have been hassled there.
Whenever I visited Castillo and waited for an Uber I would get them to open the gate, but at night I wouldn't stand out there waiting, I would still stand inside with my hand on the gate ready to open it and walk outside when Uber arrived, or close it if thugs were meandering about.
Around the Taormina and definitely west of it can be no man's land, in fact, I don't advise going west of Taormina even in the daytime. There's been many stories of muggings and knife fights etc., (locals against locals) in that area at night, I definitely advise Uber or cab only there in the evenings.
HDR still many harmless but annoying mooches and panhandlers, but for some reason I never felt threatened there, still for newbies the night rules apply.
You're on vacation, everything is pricey, blow a few 1 mil colones here and there for your taxi's, state up front you will pay 1 mil, and like I said before, cabbies are happy with that, why take chances?