OK so now you have a clean projection matrix. All that is required now is to define the viewing volume so that rendering is done only in a small area around the cursor. This can be accomplished using the following function:
void gluPickMatrix(GLdouble x, GLdouble y, GLdouble witdth, GLdouble height, GLint viewport[4]);
Parameters:
Before you call the above function you have to get the current viewport. This can be done querying OpenGL for the state variable GL_VIEWPORT (use the function glGetIntegerv). Then you call gluPickMatrix, and finally set your projection (perspective or orthogonal) just as you did for the normal rendering mode. Finally get back to the modelview matrix mode, and initialise the Name Stack to start rendering. The following code shows the sequence of function calls to do just that using a perspective projection.
Note the second parameter of gluPickMatrix. As mentioned before, OpenGL has a different origin for its window coordinates than the operation system. The second parameter provides for the conversion between the two systems, i.e. it transforms the origin from the upper left corner, as provided by GLUT for example, into the bottom left corner.
The picking region in this case is a 5x5 window. You may find that it is not appropriate for your application. Do some tests to find an appropriate value if you're finding it hard to pick the right objects.
The following function does all operations required to enter the selection mode and start picking, assuming that cursorX and cursorY are the operating system window coordinates for the location of the mouse click.
You may copy and paste this function to your application, with the appropriate modifications regarding your projection. If you do that then just call this function when entering the rendering mode and before rendering any graphic primitives.