I believe that Vader might not have noticed that Leia was his daughter because she isn't "like him". She's more like her mother and I imagine that Vader repressed his memories of her quite a bit. As far as Luke Skywalker is concerned he has his piloting skills in common with Vader and is clearly "strong in the Force" despite his brief training. I can imagine that someone who blew up a Death Star would catch his attention. Besides the Force/deus ex machina can obviously detect things like that. He figures out that Luke has a sister easily enough, right?
In the RoTJ book, it's because Leia got shot at that moment, Luke can feel her pain, Vader can feel Luke's reaction- and he puts two and two together.
Vader already knew, he was lying to the Emperor. When they find Hoth, he says "The rebels are there, and I'm sure Skywalker is with them." The title crawl also says "The evil lord Darth Vader, obsessed with finding young Skywalker,"... Vader knows exactly who Luke is at the beginning of the movie.
He knows the name "Skywalker" but he doesn't know the precise nature of Skywalker's relation to him. While the "How is this possible?!" reaction could be taken as fake- The Rise And Fall of Darth Vader treats it as real.
The EU detail is that Vader doesn't know for certain he has a son until the Emperor tells him- in the movie. He knows the boy's name is Skywalker, he suspects some kind of connection- but since Padme died in childbirth he assumes the child died with her.
Just because their birth mother/their biological mother dies, that doesn't mean that the baby/the fetus/the embryo dies with it too. It depends on how far developed the fetus is!!! And, also, Skywalker is not a common name such as Smith in the Star Wars Galaxy!!!
He assumed they were dead. When he felt Luke's presence, he realised that might not be true. When hearing the news from the Emperor, it concreted the idea that Luke indeed was his son, and he indeed had felt Luke's presence.
But, well, then, how would the Emperor Palpatine/Darth Sidious know more behind-the-scenes top-secret information than Anakin Skywalker/Darth Vader???
How the Emperor did? Possibly through the Force, possibly he investigated Polis Massa and discovered the secret records showing Padme gave birth before she died- who knows?
Y'all are making it way harder than it should be to figure this out. A pilot strong in the force named Skywalker, who happened to have been traveling with Kenobi before the death star attack. Not hard to connect the dots.
It's also worth looking at the original dialogue from pre-SE days: Emperor: There is a great disturbance in the force. Vader: I have felt it Emperor: We have a new enemy -- Luke Skywalker. Vader: Yes, my master. (Note: complete lack of surprise or indignation, this is not news. If his reaction to this news is to describe him as only a boy, then Vader obviously already knows about him.) Emperor: He could destroy us. Vader: He's just a boy (how does he know this if the Emperor is springing this on him as a surprise?) Obi-wan can no longer help him. Emperor: The force is strong with him. The Son of Skywalker must not become a Jedi. Vader: If he could be turned, he could become a powerful ally. Emperor: Yes... Yes... He would be a great asset. Can it be done? Vader: He will join us or die, master. "How is that possible?" only came in 2004 -- after other revisions with Ian preceeded it. Ian's first remake repeated the original dialogue.
"The young rebel who destroyed the Death Star. I have no doubt this boy is the offspring of Anakin Skywalker." = Palpy confirming their suspicions. So Vader, obsessed with finding young Skywalker, has a gut feeling, combined with various bits of info about the aforementioned young rebel. Then his master tells him to search his feelings. That is all the confirmation they need. I highly doubt Vader finds out all at once in that one scene, DVD retcon or not.
In the books (specifically, The Rise & Fall of Darth Vader), he doesn't know- but he suspects there is some kind of connection between them- we see him finding out the various details. It's not till the ESB movie that he knows the precise nature of the connection though.
Haven't read it, but I've going off of other EU sources where a rebel officer told him before Vader killed him
Told him that the person who destroyed the Death Star was one Luke Skywalker- not that Luke was "the son of Anakin".
Yeah but since force sensitivity is rare and the guy in question was a Skywalker, its pretty obvious.