Gday John, no worries. If you build up to it then it's no worries. If you think of a normal day trip - say 40 km out to your spot and 40km back = 80km. As long as you are self sufficient enough so that each night the end of that 80km is like being at home then it's no worries.
I think you are right in saying though that "that the only thing holding us back from this trip is ourselves". You basically have to be comfortable with your own ability to get yourself out of trouble and last until help arrives. A conversation myself and isaac had while we out there was - without falling out of the boat what is the worse that could happen?? The answer was that for some reason, struck by lightning or something as equally unlikely, the entire electrics system plus the epirb didn't work. If this was the case we couldn't go anywhere, couldn't contact anyone and would be entirely self sufficient until someone passed by. And as Graham mentioned, in shoalwater this can be some time. I was totally happy with our response - we would anchor up if possible, we had enough water to last us for at least a week plus with tarps etc on board we could make a rain collection system, we had enough food on board to last a minimum of a week plus fishing gear. We had flares to contact someone if they passed within distance, plus the coastguard knew our movewments and would be looking for us within the week. Failing that - if we had a satellite phone like Graham mentioned it would be mearly 24 hours until help arrived.
Like i've alluded to earlier, prepare for and be prepared to handle the absolute worst scenario. Therefore anything that occurs shouldn't be a suprise and you are right to explore and catch the biggun!
For me the biggest worry was porpoising down the back of a big swell and capsizing, being swamped by a standing wave or hitting a submerged rock and being tossed from the boat. In such an event you have zero time to prepare and are really helpless. To this end you should wear a PFD while travelling and have a personal epirb on your belt...something we didn't do and this time didn't pay for (just). Two of those things are operator error while the third - hitting a submerged rock at speed, is a real posibility in shoalwater - hence wearing pfd and epirb, while somewhat hypocritical, is something I would seriously reccomend if you don't know the area and are doing the trip for a first time.