You can eat local seafood with the river right in front of you at Down Under Restaurant, which sits under the bridge where A1A crosses the Intracoastal Waterway, at 96106 Wades Place. It opened in 1982, when an old fish camp on the property was remodeled into a 44-seat restaurant. It has since grown past 200 seats and still runs on what it calls Florida fish-camp hospitality, with oysters served out on the deck.
The restaurant's own history of the site goes back much further. The land was once a 200-acre tract that included Piney Island. In 1923 it was used to harvest pine sap for the turpentine industry, oyster houses operated there in the late 1930s, and a bait-selling fish camp went up where the boat dock is now in the late 1950s. From the deck, the restaurant says, you might see dolphins, manatees, sea turtles, tarpon, otters, birds and the occasional alligator.
The menu is built on local seafood. The fresh catch comes grilled, blackened or fried, and starters include crab dip, spicy fried local shrimp, peel-and-eat shrimp with house cocktail sauce, and sesame-seared ahi tuna. There's a separate oyster list, sandwiches, a happy-hour lunch menu, a kids' menu, and daily specials worth asking about. Hours are 11 a.m. to 9 p.m. Sunday through Thursday and 11 a.m. to 9:30 p.m. on Friday and Saturday.
Shuckers Oyster Bar & Grill is a neighbor on Wades Place, so you've got two waterfront kitchens in the same spot. For dinner on the water downtown, Salty Pelican Bar & Grill overlooks the harbor from North Front Street.