(note: i have 16 people in this house all on one 4mbps connection)
When i first got Cable here in this house, i was getting 30-40 pings on servers. recently it shot up to 150, then 300, then 600, then around 2 second. Since i set up the Router, and Cable modem, i knew that everything checked out ok. I went into the router, and set up a way for me to watch what packets are being sent where. I found out some of my housemates were downloading off of kazaa, bittorrent, etc etc, and basically smashing everybody elses internet connection. I couldnt' even sign on to AIM for long periods of time because it would time out.
I first started with cutting off the offenders Internet, which they didn't seem to notice cause they didn't come and complain. This of course was a temp. solution cause everybody pays, so everybody should get access. After looking around, i found some firmware for the router that has QoS, and I could set priority to what type of traffic gets priority over others. I put CS/Half life in the highest priority, and all the p2p in the lowest priority, and almost immediatly got my 30-50 pings again. I also get reports of the 'Internet being faster" which to them means surfing around to sites. I also put regular surfing (http) at a high priotity because it is essential.
Anyway, Short moral/main point to pull out of this is, have you checked the other computers? are they using the internet a bunch?
For the downloads, It always seems to start at a high value and lower a tiny bit. that *is* normal at least.