Why did Harry Potter use gillyweed in Goblet of Fire?

I get that he wanted to breathe but don't the rules say that the competitors have to use only their wand? I also know Harry used the Firebolt but he summoned it with a spell and he didn't do anything to the gillyweed.