Yes, it can be done, but with difficulty.
We have built man-powered airplanes, but even a trained athlete cannot output greater than 700 watts continuously, which is about 1 hp. So even a model airplane engine could (in theory) power one. But the craft would be exotic, with carbon fiber and kevalr and other ultra-light materials. It'd look like a glider, huge lift vs. drag.
A more practical powerplant for an experimental would be an engine in the 10 to 25 hp range. You'll need fewer hi-tech materials.