The VTOL variant, F-35B, has the lift fan behind the cockpit. This means weapons bays or the inlet ducts can not be on the centerline, behind the cockpit – instead they are on the sides. This applies to all the variants*. This makes the airplane fatter, more draggy and slower. This is also exacerbated by the fact that the carrier version is length limited. Since it’s a stealth airplane, it needs internal weapons bays. Since it’s multi-role, the bays are large, to be able to fit bombs.
A from-scratch designed airplane with the same engine and other systems as the F-35, but no shared structure, designed for air superiority, would probably have a smaller centerline weapons bay and would be longer and thinner. And yes, it would be faster. Manufacturing the structures would be more expensive (because of no sharing) and it would not be able to carry bombs (or at least not as many) stealthily.
Disclaimer: not an expert on the subject. I just haven’t seen this reasoning mentioned in the complaints about F-35.
*: In the non-lift fan variants, there’s a fuel tank in the lift fan space.