A project like this should define API/interfaces for sub-systems that are required by most game engines and create exemplary open source based implementations. There are some fine open source engines out there (there are others, but these are the ones I know about):
- OGRE graphics engine. Someone else develops a Java wrapper for it
- ODE physics engine. There seems to be a plethora physics engines out there lately, but I'm not sure how many are open source. I remember that ODE is one of them. It has a Java wrapper too
- OpenAL 3D audio library. I think there are Java wrappers for this too, but you'll have to find the links yourself.
Other things needed:
- entity/actor system
- a scripting language
- the glue to hold it all together
- 2D graphics engine (start simple)?
- extensions for specific genres