Hey Haewon!! =)
Great question!...and I think you answered your own question in your explanation.
You wrote...
"However, if it is used in the context of 'degree' the answer has to be extent."
You definitely have to think about how the word is used in context; if you only think about the meaning of the word without considering how the word is being used in the passage, you might end up with the incorrect answer. In this passage, "magnitude" is used to describe the correlation (relationship) between two things. A correlation is not a physical object (like a book or a house), so you cannot use the word "size" since size is used to describe tangible things.
Did you see what I did just now? I thought about how "magnitude" was used in context; then I thought about what the words in the answer choices (size, extent, etc...) are used to describe (size measures physical objects, extent measures the degree of something intangible (i.e. how cold I feel or the degree of similarity between two experiments).
Hoep that helps!
Keep Moving Forward!
You got this!