“It’s the network…”

The statement and all its derivatives are as timeless as STP and OSPF.

I recently had an interesting experience in my personal life that I feel compelled to share with my fellow network engineers. There are MANY stories we can tell from the perspective of technology specialists that work in IT.  I have tons of stories from working in operations for 10 years to pre-sales with enterprise customers throughout my career.

However, what if we were to take a different approach and vicariously view this from a completely different perspective. The perspective of a small retail business owner and my personal friend “J”.

Background:

J owns a small outdoors shop in my local area and we are roughly the same age. He needed some help with getting a few point-of-sale PCs attached (wired) to the network and I helped him out. Very simple indeed, however several days later he called me up and said ever since we connected the PCs, he’s having issues everyday between 18:00 and 20:00. “It’s gotta be something in the network Shaun”

Now, this time-frame is one of his busiest times since it’s when people are stopping by the store on their way home from work or after supper.

Investigation Time:

Typical routine troubleshooting stuff. Traceroute, constant pings to 208.67.222.222 , 1.1.1.1, and 8.8.8.8 from each of the PoS terminals. Watch and wait for latency spikes, packet loss, etc during the “busy” time. Result = NOTHING! All responses looked within spec.

Next, check out the hardware he’s running. Result = standard fare (Linksys/Netgear) and rather recent (2016-2018) hardware with current firmware.

I didn’t really want to look at the PoS PCs, but I knew this was the most likely the culprit. However, as many of you know tracking down the root cause could take some time (drivers, malware/virus, backups, AV updates/scans, etc). I did notice an unfamilar icon in the system tray and it was for some kind of backup (cloud) to a local ISP. I checked the schedule and sure enough “18:30”. You could not change the settings, so I had him call the provider and change the schedule to AFTER they close.

Case closed, that was it…!!!

Shaun’s Take:

While this story is nothing earth shattering, it does highlight a very common trend. When in doubt, blame the network. What’s even more interesting is that J was willing to buy new networking gear and replace everything on a whim. It was that impactful to his business and he was desperate enough to try anything just to resume business. Lucky for him all it took was some investigation and a phone call.