The scroll wheel is good for something that needs granular control.
I think that you meant to type
gradual control not
granular and if so, I agree with you that a scroll wheel on the mouse is best for doing that job.
For toggling the zoomed-in / fit-to-screen state it's better to use a key and preserve the wheel for previous/next functionality.
I don't do that and don't see any point in doing so when there are other methods of moving to the next image such as swiping left or right and keyboard left and right arrows.
Using the zoom slider the magnification can be controlled in many steps, so the wheel is used for magnification.
That is what I want - mouse wheel for magnification - zoom in and out.
I never saw the browser's zoom functionality take over (not even in Chrome).
On my browser using Ctrl and the scroll wheel activates Chrome's Zoom. Example attached.
However, using my mouse scroll wheel alone, does nothing because I have not enabled mouse wheel navigation.
Most picture viewers use the mouse wheel to magnify the image.