Are there any specific rules for writing dialogue in a novel? Do you need to use quotation marks or italics for thoughts?

Open literally any traditionally published novel. That's how to punctuate dialogue.

Given that it's so easy to see good examples of how to handle dialogue, I bet your only actual question is the one about direct thoughts.

People handle that in various different ways. The only real rule is that it's easy to screw it up. You've screwed it up if whatever you're doing interferes with the reader's experience. That throws the whole issue into the realm of artistic judgment. Here are some things to consider.

A) A lot of readers hate hate hate long passages in italics.

B) Yet it is best to distinguish direct thoughts from dialogue.

C) If you write,

Wow, he seemed mad. Claire circled to put the couch between them.

then it's obviously Claire who thinks he seems mad. This is a report of her thoughts, but it's what we might call an intimate report. It's in close third person. Distant third, like this,

Wow, he seemed mad, Claire thought as she circled to keep the couch between them.

is always going to feel more distancing and is also the style that most strongly compels you to put that thought in italics.

First person, like

Wow, he seemed mad. I circled to keep the couch between us.

is obviously very much like close third. Usually you can get away without setting the thought off. But

D) Truly direct thoughts, like direct speech, must be in present tense. You cannot write

Gosh, he was really mad, Claire thought

if what you mean is

Gosh, he is really mad right this second.

Just as someone might look at a friend and say, “Dude is pretty mad, I guess.”

while pointing to a guy having a tantrum, she would think direct thoughts like that in present, not past tense, even if the story is written in past tense. If it would be in present tense in dialogue, then it would be in present tense in italicized direct thoughts.

Tl;dr

Pay attention to how authors do this. Some do it well and some badly. Pay attention to both well done direct thoughts and badly handled direct thoughts in first and third person stories, and in both present and past tense stories. Then do it well.