Logic is simple. Size does matter. More size means more fissile material and more bang.
Agni 4 due to it's very high apogee,will enter atmosphere at very high speed,well in the realms of Mach 20.
Such high speed means too much air friction and resultant heat,for whichneeded heavy heat shields will be needed to protect warhead and electronics. Leaving even lesser space for warhead,meaning lower yields.
For example American 300 kt. W-87 was 57 cm wide and 170 cm long. And that was just warhead. Heat shield and other systems also need space.
So the point being made here in addition to all the above, India does not have any MIRV capable missiles, not even A-5. Also none of Indian missiles can carry city killers of half megaton or megaton range yield.