In my experience, sub back port back sounds better and is louder most of the time. But it does depend on the shape of the car. For example, my '97 Integra has a more slanted back window and is louder sub back port back. But my '95 Civic, which is shaped more like an SUV, is louder sub up port up.
Long time since I heard a hatch, but I do believe sub up port up would be my first attempt. I think that'll work better than it will in most vehicles as upfiring should still load well off the hatch and keep the rattling down vs firing backwards. If you have issues with your upper end bass, then I'd try another orientation. Or simply build a sealed box and start firing it each way and see what effect the direction has, then you'll know for sure what to do.
th the box using every inch of space forming a "wall" firing up. Thats what I've seen do VERY well in those cars. A center aero with 2 10s in the 50s on 3000 ish watts. It was a older model VW
Thinking of sub up port back. I have tried sub back and port back and seems to lose a little. I competed last year with sub forward port side and turned better results. I have seen another store that runs Sundown run 4 8's firing up. They stated they got the best results firing up. I just can't remember the port location on their enclosure.