This proposal requires exponential energy! But you can factorize numbers with only one photon (a tiny amount of energy). Oh yeah, you’ll need an exponential number of
modes (you just build a very-low-loss interferometer that does the unitary transformation corresponding to Shor’s algorithm on those modes).
Is finding exponentially inefficient ways of factorizing interesting?