Main thing to remember, if the pedal/lever becomes spongy, but stopping power remains until you run out of lever travel, you are either losing pressure, or boiling the fluid. If the pedal/lever is firm, but you need to apply more and more force to the pedal/lever to get the same stopping power, you are overheating the pad material, or you have glazed or oil soaked pads.
Your snowmobile can use only glycol based brake fluid. I believe that limits you to DOT 3 or 4. DOT 5 is silicone based, and will not function with the seals in your brake system. Also, remember that brake fluid is hydroscopic, i.e. it absorbs water from the air. The more water that is present, the lower the boiling point of your brake fluid becomes. Once enough water is present that it cannot be held in suspension, your brake fluid's boiling point would become that of water. As was mentioned before, the water vapor/ boiled brake fluid is compressible, and that is why you start using up more pedal/lever travel to get the same clamping force at the pads.