You are here: Home > Publications > RIPE Labs > Philip Homburg > Troubleshooting RIPE Atlas Probes: USB Sticks

Troubleshooting RIPE Atlas Probes: USB Sticks

Philip Homburg — 04 Jul 2016
Some of the third version of RIPE Atlas probes have recently had an issue with their USB sticks. We're investigating what may be causing this issue and have a possible solution, outlined below. (At the same time, we're also looking into a new hardware solution for the future.) If you've had trouble with your probe, please follow these simple steps. RIPE Atlas users everywhere will thank you for getting your probe back online - and we will, too!

The issue

The USB stick on the version three (v3) probes is used to store both the operating system and the measurement data. In some cases, the filesystem on the probe can be corrupted, resulting in an unusable operating system. In turn, this causes the probe to remain "disconnected" from the RIPE Atlas back-end infrastructure.

Below we outline some steps to take to try to fix this issue, but it's important to note that the filesystem can always become corrupted again, so if you do run into problems with your probe disconnecting in the future, please repeat these steps.

How to fix it

Solution 1

As a backup, if there is a problem with the USB stick, the probe hardware itself has a mini-operating system that enables it to download fresh firmware and save it on the USB stick once it is available again.

In order to activate this last-resort option, the probe must be booted WITHOUT the USB stick. If you've already tried this and you're still having issues, please jump to Solution 2 below.

NOTE: This must be done on a network with working DHCP! It will NOT work if you have configured your probe to use a static IP address, because this mini-operating system does not have access to any statically configure IP addresses or DNS resolvers. If you have configured your probe to use a static IP address, the only thing you can do is move it to a network with working DHCP.*

Please follow these simple steps (these are the same instructions you'll see on the "Status" page):

1. Unplug the probe from its power source
2. Remove the USB stick from the probe
3. Plug in the probe WITHOUT the USB stick
4. Wait for ten minutes
5. Insert the USB stick

What will happen

Once the probe reboots, it will report the fact that it doesn't have a USB stick to the central RIPE Atlas system using a so-called "SOS" message. (You can see SOS messages on your probe's detailed information page under the "Network" tab at the bottom of the page.)

After ten minutes, it is safe to re-insert the USB stick.

Once the probe recognises that that the USB stick is present again, the process of downloading the fresh firmware  will begin. However, please note that this can take up to an hour to complete.

* If the probe was configured to use a static IP address, it will fetch it automatically. When the probe is listed as "connected" again, it is safe to move the probe back to its original location. However, please note that we advise configuring your probe to use a static IP address only if you feel it is truly necessary to do so.

If the probe is still not connecting, advanced users may want to completely wipe the USB stick on another computer and try following these steps again.

Solution 2

If your probe is still having issues with its USB stick, you should see a message under the "Status" tab on your probe's information page (you can see this by going to "My Atlas" and clicking on your probe's ID number).

my atlas

The error message will be about not detecting a flash drive or about having a read-only flash drive (see examples below).

no usb detected

read-only usb

If this is the case, please follow the instructions below (these are the same instructions you'll see on the "Status" page in both cases):

1. Unplug the probe from its power source
2. Remove the current USB stick from the probe
3. Plug in a new, brand-name USB stick that is at least 4GB in size (it does not need any preparation but its contents will be erased so make sure it does not store anything crucial)
4. Plug in the probe
5. Wait at least 30 minutes for the probe to initialise the drive and upgrade its firmware

Hopefully that will solve the issue and you will see your probe listed as "connected" again within an hour of the stick re-initialising.

Other possible issues

If neither of these has solved your problem, your probe will still be listed as "disconnected" or "abandoned".

When we have information about what might be wrong with your probe and detect a particular error, that information will appear under the "Status" tab on your probe's information page (you can see this by going to "My Atlas" and clicking on your probe's ID number). Please note that it takes up to four hours for probes to report an issue and for relevant information to appear on the Status page.

status tab

The "Status" tab on your probe's information page will report information about suspected issues and give advice about how to fix them

Of course, you can always contact us with questions at: atlas [at] ripe [dot] net or leave a more general comment below.

Thank you for your commitment

We really do appreciate the effort that our probe hosts put into keeping the RIPE Atlas network strong by keeping their probes connected. We couldn't build the world's largest Internet measurement network without you!

21 Comments

Kyle says:
04 Jul, 2016 08:10 PM
Def had this issue on mine, worked with RIPE staff to replace it with another drive I had laying around. May look a little silly now, but its hidden away anyhow.
Philip Homburg says:
05 Jul, 2016 12:19 PM
Thanks very much for taking the time to get your probe back online. We hope you’re enjoying earning credits again, too.
Trevor Warwick says:
06 Jul, 2016 07:48 PM
This just happened to mine as well - I think it failed either when I plugged it into a different subnet on my firewall; or more likely when I power cycled it to fix a DHCP issue on that new subnet. It was sending SOS messages with "NO_USB" even thought it still had the original stick plugged in. It seems happier now with a new, blank USB stick, it's showing as Connected again.
Chris Morrow says:
18 Jul, 2016 09:16 AM
For the record, one of my three probes failed similarly... The USB stick was unreadable (drive errors) on my laptop, so I replaced the usb-stick with another, all fixed.

Perhaps the problem is a bad batch of usb-sticks?
Mika Mäki says:
30 Jul, 2016 04:06 PM
My probe (id 26175) had the same issue but the solution 1 helped.
Marius Gruen says:
04 Aug, 2016 01:45 PM
Solution 1 works fine
Plamen Ivanov says:
18 Aug, 2016 10:06 AM
Our probe #21581 also had the same issue but solution 2 helped. Original SanDisk Cruzer 4gb mini stick was completely dead.
A new one (unbranded, for now) was plugged in and now the probe is allive and kicking!
Quentin Grandemange says:
30 Aug, 2016 11:51 AM
My probe (id 24936) got the same issue.
It happened several times, every time I follow the solution 1 and the probe get connected again. But several weeks after, it get disconnected...
I should try with another usb stick. I though it was the heat due to the fact that the probe is put on my modem (which can be hot sometimes), but now I think it comes from the usb stick as mentioned by others.

JeroenE says:
01 Nov, 2016 09:20 AM
My probe didn't work until I replaced the USB drive with another. The little white USB drive seems to work fine in my laptop. It does report a size of 1 GB though while this page says you should use a 4GB drive at least. So perhaps there is something wrong with the USB drive?
Philip Homburg says:
01 Nov, 2016 05:47 PM
When a probe initializes the USB stick, it creates 3 partitions with size 1 GB. Some operating systems only show the first partition giving the impression that the USB stick itself is only 1 GB.

So by itself this is not an indication that the USB stick is broken.
JeroenE says:
22 Nov, 2016 07:35 AM
Sorry for the late reply, I had not been on this page again to see your message. You are right, I had used a Microsoft Windows machine and that did not recognise the partitions. Once I plugged it into a Linux machine I could see all the partitions and after I removed them I can now use the entire stick in Microsoft Windows too.

So now I wonder what to do? Just leave the 4Gb stick in it as it is now or try again with the original stick. Does that really matter (except for the uptime)?
Philip Homburg says:
22 Nov, 2016 10:49 AM
The probe doesn't care about the USB stick, as long as it is at least 4 GB and it works.
Jeroen Maathuis says:
20 Nov, 2016 11:44 PM
I got the issue 2 times with my probe 24004.

The first time the original Verbatim 8GB USB flash drive got into read-only mode. At that time I replaced it with an old Delock 4GB USB flash drive. After that the probe disconnected a lot while the leds did lit correctly. Disconnect and reconnect the power did solve the disconnect each time for about 1 or 2 days.

After a month sice I replaced the USB flash drive the new USB flash drive got issues again. This time any device won't recognize the USB flash drive at all.
I suggested to RIPE Atlas to send another USB power adapter along with a new USB flash drive. The other USB power adapter could be used to asure whether the original power adapter is unstable or not.
Since November, 18th 2016 my probe is running again. Until now (November 20th, 2016 at the end of the day) the probe is running stable. I hope it will stay stable for a long time.
Stefan Reusch says:
11 Jan, 2017 09:24 PM
My ripe usb stick is defect after two month. I had the philips 8gb mini usb stick.
Jean-michel pepin says:
08 Feb, 2017 12:09 AM
I had also this issue:
with normal reboot didn't get a IP.
without USB the probe gets again a IP but plugin the USB didn't solve the issue. The probe was still "not connected". the info in SoS were coming quickly.
I had to reboot later several times with the USB.
Now is OK. I hope for a long time .....
Mirjam Kühne says:
08 Feb, 2017 11:42 AM
Hi Jean-Michel, thanks for letting us know. We're working on a more long-term solution. Please also see this RIPE Labs article in that context: https://labs.ripe.net/Membe[…]issues-with-better-firmware
Thomas Baldauf says:
24 Feb, 2017 06:14 PM
replacing the usb drive with a new one was the only way to revive the probe... so are the flash drives defective or is the usb port of the tl-mr3020 the cause?
Philip Homburg says:
27 Feb, 2017 02:38 PM
Hi Thomas, the most likely explanation is that USB sticks support only a very limited number of writes. Power to the USB sticks comes almost directly from the power that is used to power the tl-mr3020. And of course data lines can't exceed that voltage.

When it comes to filesystem corruption, it is not clear what exactly causes that. But for USB sticks that fail completely, it is unlikely that the tl-mr3020 can be blamed.

Philip
Stefan Post says:
21 Mar, 2017 12:24 PM
Thanks for the post! Solution 1 solved the problem of my probe.
Mirjam Kühne says:
21 Mar, 2017 02:51 PM
Thanks for letting us know, Stefan. I am glad to hear your probe works again.
Jean-Philippe LUCAS says:
19 Apr, 2017 12:15 AM
Hi,
i have a probe (#30489) from the beginning of the year, and 3 times i had to use the solution 1 to recover the probe... For the last 2 failures (yesterday 2017-04-18 and on 2017-03-02), the internet connection works fine, the probe responds to ping, and the leds were identical to its normal operational state.
Disconnecting and reconnecting from network the probe do not change anything, and with a power off / on, after a few minutes, the only led that remains on is the power, no SOS message on the history, nothing.

I will try with another usb stick...
Add comment

You can add a comment by filling out the form below. Only plain text is possible. Web and email addresses will be transformed into clickable links. Comments are moderated so they won't appear immediately.