Use a more professional adapter as this:
http://www.drivestar.biz/pc-usb-terminal-p-304.htmlHowever you can connect only 2 wires (TX + RX), GND wire is not necessary if the hard drive ground is in common with the computer ground.
If you use two different power supply (one for the computer and the other for the hard drive), to do a common ground (GND) you can use a wire that goes from the hard drive power supply metal case to the computer metal case (or to the computer power supply case).
For example you can insert a end of a cable under a screw of the metal case of the hdd power supply and the other end under a screw of the computer power supply.
If you use only one power supply for everything (hdd + terminal adapter + computer), you don't even need to build any ground connection because ground is already in common for every device.