I am successfully using Quickeys with RD 4.1.3 with some workarounds. I can almost guarantee that it will work with RDS5.
Before going thru all of the following, please note that there is a downloadable keyset below. You should not have to do anything but rename the keyset and put it in the correct folder as listed on this page.
If you are starting from scratch, to get Quickeys commands for RDS, you have to generate a new keyset with Quickeys while you are in the finder and name it EXACTLY as your RD application is named. For certainty, I single clicked the application's name as if I were about to rename the app and then copied it with cmd-C. Then I pasted that into the new keyset's name. Save your keyset in the folder with all your other keysets. It can be found in: System Folder/Preferences/Quickeys Folder/Keysets. Quit Quickeys now.
Next, open the program, find all the things that you want RD to do and write down the exact menu name for each-this may be case-sensitive, so carefull notes and punctuation like "..." is important when writing down your menu names. Quit the program and open Quickeys.
Go to "Define:" "Menu Selection" and select any item from any of the Finder's menus (Don't worry if the finder's menus are gray. The items in the menus are still usable... just select a black one that works.) and Quickeys will bring up the familiar box. Rename the menu command (as what you have now is a Finder command) and give it the name of one of the commands that you wanted and wrote down before. Have it select "By Text" and remove the check from "Match Exactly." Also (and most importantly) have it SEARCH ALL MENUS by changing the other checkbox in this Quickeys window that you are in now. Otherwise, it will be looking for a Finder Menu title that may not exist in RD while you are running the RD app.
Give it a keystroke command that you want and there is your quickey for RD.
This can be repeated until you have as many quickeys as you need.
Click here for a Quickeys 3.5 file that should be put in your folder in the System Folder/Preferences/Quickeys Folder/Keysets. Be sure to rename this file with a copy/paste of the new RayDream Studio 5's name or else the Quickeys won't work.
I realize that it is a hassle and a workaround, but I still enjoy my Quickeys with RDS and it greatly speeds up my work.
By the way, you might want to print this out before you go about creating your Quickeys.
Using the keyset that I posted, the menu items that now have keystrokes are:
Window Commands:
Shaders Browser (F1)
Shaders Editor (F2)
Objects Browser (F3)
In the freeform modeler:
Previous Cross Section (cmd <-- )
Next Cross Section (cmd --> [that's command/right arrow])
Remove Cross Section (ctrl "-")
Create Cross Section (ctrl "=")
Import Adobe Illustrator Paths (ctrl-I) [that's an "i"]
Rotate selection (opt-ctrl-R)
Scale selection (opt-ctrl-S)
Surface Fidelity (opt-F)
In the Perspective Window:
Object Invisible (opt-I) [that's an "i"]
Object Visible (shift-opt-I) [that's an "i"]
These should also be printed out before using them as you can't call up Quickeys to find out how you set the commands without quitting RDS... (I tried... heheh)
D.G. Chichester recommends the following slick solution for scripts in Quickeys:
With Ray Dream *open*, switch to the Finder, and set up a new QK sequence to run in the Finder. Don't give it any commands, but choose the "record more" button at the bottom of the QK window.
With the QK record palette open, switch to Ray Dream. Do some click and drag and whatever magic you might want to do.
Switch back to the finder, and click the "stop" button on the QK record palette. Scan the sequence for the Ray Dream relevant clicks and moves and whatever, and cut-paste them to their own sequence in your Ray Dream Keyset. (Chuck's note: Or maybe you could just select the Finder-specific functions and remove them... That way you don't have to make a new file and copy/paste.) Give it a key command, close up the QK window, and give it a shot in Ray Dream. I (D.G.) haven't tried this for anything phenomenally complex yet, but it did work in selecting a specific windoid (properties, camera, browser, etc.) and closing it up, circumventing that nasty omission in the Ray Dream feature set.
Take care and happy rendering.