Some of the third version of RIPE Atlas probes have recently had an issue with their USB sticks. We're investigating what may be causing this issue and have a possible solution, outlined below. (At the same time, we're also looking into a new hardware solution for the future.) If you've had trouble with your probe, please follow these simple steps. RIPE Atlas users everywhere will thank you for getting your probe back online - and we will, too!
The issue
The USB stick on the version three (v3) probes is used to store both the operating system and the measurement data. In some cases, the filesystem on the probe can be corrupted, resulting in an unusable operating system. In turn, this causes the probe to remain "disconnected" from the RIPE Atlas back-end infrastructure.
Below we outline some steps to take to try to fix this issue, but it's important to note that the filesystem can always become corrupted again, so if you do run into problems with your probe disconnecting in the future, please repeat these steps.
How to fix it
Solution 1
As a backup, if there is a problem with the USB stick, the probe hardware itself has a mini-operating system that enables it to download fresh firmware and save it on the USB stick once it is available again.
In order to activate this last-resort option, the probe must be booted WITHOUT the USB stick. If you've already tried this and you're still having issues, please jump to Solution 2 below.
NOTE: This must be done on a network with working DHCP! It will NOT work if you have configured your probe to use a static IP address, because this mini-operating system does not have access to any statically configure IP addresses or DNS resolvers. If you have configured your probe to use a static IP address, the only thing you can do is move it to a network with working DHCP.*
Please follow these simple steps (these are the same instructions you'll see on the "Status" page):
1. Unplug the probe from its power source
2. Remove the USB stick from the probe
3. Plug in the probe WITHOUT the USB stick
4. Wait for ten minutes
5. Insert the USB stick
What will happen
Once the probe reboots, it will report the fact that it doesn't have a USB stick to the central RIPE Atlas system using a so-called "SOS" message. (You can see SOS messages on your probe's detailed information page under the "Network" tab at the bottom of the page.)
After ten minutes, it is safe to re-insert the USB stick.
Once the probe recognises that that the USB stick is present again, the process of downloading the fresh firmware will begin. However, please note that this can take up to an hour to complete.
* If the probe was configured to use a static IP address, it will fetch it automatically. When the probe is listed as "connected" again, it is safe to move the probe back to its original location. However, please note that we advise configuring your probe to use a static IP address only if you feel it is truly necessary to do so.
If the probe is still not connecting, advanced users may want to completely wipe the USB stick on another computer and try following these steps again.
If your probe is still having issues with its USB stick, you should see a message under the "Status" tab on your probe's information page (you can see this by going to "My Atlas" and clicking on your probe's ID number).
The error message will be about not detecting a flash drive or about having a read-only flash drive (see examples below).
If this is the case, please follow the instructions below (these are the same instructions you'll see on the "Status" page in both cases):
1. Unplug the probe from its power source
2. Remove the current USB stick from the probe
3. Plug in a new, brand-name USB stick that is at least 4GB in size (it does not need any preparation but its contents will be erased so make sure it does not store anything crucial)
4. Plug in the probe
5. Wait at least 30 minutes for the probe to initialise the drive and upgrade its firmware
Hopefully that will solve the issue and you will see your probe listed as "connected" again within an hour of the stick re-initialising.
Other possible issues
If neither of these has solved your problem, your probe will still be listed as "disconnected" or "abandoned".
When we have information about what might be wrong with your probe and detect a particular error, that information will appear under the "Status" tab on your probe's information page (you can see this by going to "My Atlas" and clicking on your probe's ID number). Please note that it takes up to four hours for probes to report an issue and for relevant information to appear on the Status page.
The "Status" tab on your probe's information page will report information about suspected issues and give advice about how to fix them
Of course, you can always contact us with questions at: atlas [at] ripe [dot] net or leave a more general comment below.
Thank you for your commitment
We really do appreciate the effort that our probe hosts put into keeping the RIPE Atlas network strong by keeping their probes connected. We couldn't build the world's largest Internet measurement network without you!
Comments 36
Comments are disabled on articles published more than a year ago. If you'd like to inform us of any issues, please reach out to us via the contact form here.
Kyle •
Def had this issue on mine, worked with RIPE staff to replace it with another drive I had laying around. May look a little silly now, but its hidden away anyhow.
Philip Homburg •
Thanks very much for taking the time to get your probe back online. We hope you’re enjoying earning credits again, too.
Trevor Warwick •
This just happened to mine as well - I think it failed either when I plugged it into a different subnet on my firewall; or more likely when I power cycled it to fix a DHCP issue on that new subnet. It was sending SOS messages with "NO_USB" even thought it still had the original stick plugged in. It seems happier now with a new, blank USB stick, it's showing as Connected again.
Chris Morrow •
For the record, one of my three probes failed similarly... The USB stick was unreadable (drive errors) on my laptop, so I replaced the usb-stick with another, all fixed. Perhaps the problem is a bad batch of usb-sticks?
Mika Mäki •
My probe (id 26175) had the same issue but the solution 1 helped.
Marius Gruen •
Solution 1 works fine
Plamen Ivanov •
Our probe #21581 also had the same issue but solution 2 helped. Original SanDisk Cruzer 4gb mini stick was completely dead. A new one (unbranded, for now) was plugged in and now the probe is allive and kicking!
Quentin Grandemange •
My probe (id 24936) got the same issue. It happened several times, every time I follow the solution 1 and the probe get connected again. But several weeks after, it get disconnected... I should try with another usb stick. I though it was the heat due to the fact that the probe is put on my modem (which can be hot sometimes), but now I think it comes from the usb stick as mentioned by others.
JeroenE •
My probe didn't work until I replaced the USB drive with another. The little white USB drive seems to work fine in my laptop. It does report a size of 1 GB though while this page says you should use a 4GB drive at least. So perhaps there is something wrong with the USB drive?
Hide replies
Philip Homburg •
When a probe initializes the USB stick, it creates 3 partitions with size 1 GB. Some operating systems only show the first partition giving the impression that the USB stick itself is only 1 GB. So by itself this is not an indication that the USB stick is broken.
Hide replies
JeroenE •
Sorry for the late reply, I had not been on this page again to see your message. You are right, I had used a Microsoft Windows machine and that did not recognise the partitions. Once I plugged it into a Linux machine I could see all the partitions and after I removed them I can now use the entire stick in Microsoft Windows too. So now I wonder what to do? Just leave the 4Gb stick in it as it is now or try again with the original stick. Does that really matter (except for the uptime)?
Hide replies
Philip Homburg •
The probe doesn't care about the USB stick, as long as it is at least 4 GB and it works.
Jeroen Maathuis •
I got the issue 2 times with my probe 24004. The first time the original Verbatim 8GB USB flash drive got into read-only mode. At that time I replaced it with an old Delock 4GB USB flash drive. After that the probe disconnected a lot while the leds did lit correctly. Disconnect and reconnect the power did solve the disconnect each time for about 1 or 2 days. After a month sice I replaced the USB flash drive the new USB flash drive got issues again. This time any device won't recognize the USB flash drive at all. I suggested to RIPE Atlas to send another USB power adapter along with a new USB flash drive. The other USB power adapter could be used to asure whether the original power adapter is unstable or not. Since November, 18th 2016 my probe is running again. Until now (November 20th, 2016 at the end of the day) the probe is running stable. I hope it will stay stable for a long time.
Stefan Reusch •
My ripe usb stick is defect after two month. I had the philips 8gb mini usb stick.
Jean-michel pepin •
I had also this issue: with normal reboot didn't get a IP. without USB the probe gets again a IP but plugin the USB didn't solve the issue. The probe was still "not connected". the info in SoS were coming quickly. I had to reboot later several times with the USB. Now is OK. I hope for a long time .....
Hide replies
Mirjam Kühne •
Hi Jean-Michel, thanks for letting us know. We're working on a more long-term solution. Please also see this RIPE Labs article in that context: https://labs.ripe.net/Members/kistel/ripe-atlas-countering-hardware-issues-with-better-firmware
Thomas Baldauf •
replacing the usb drive with a new one was the only way to revive the probe... so are the flash drives defective or is the usb port of the tl-mr3020 the cause?
Philip Homburg •
Hi Thomas, the most likely explanation is that USB sticks support only a very limited number of writes. Power to the USB sticks comes almost directly from the power that is used to power the tl-mr3020. And of course data lines can't exceed that voltage. When it comes to filesystem corruption, it is not clear what exactly causes that. But for USB sticks that fail completely, it is unlikely that the tl-mr3020 can be blamed. Philip
Stefan Post •
Thanks for the post! Solution 1 solved the problem of my probe.
Hide replies
Mirjam Kühne •
Thanks for letting us know, Stefan. I am glad to hear your probe works again.
Jean-Philippe LUCAS •
Hi, i have a probe (#30489) from the beginning of the year, and 3 times i had to use the solution 1 to recover the probe... For the last 2 failures (yesterday 2017-04-18 and on 2017-03-02), the internet connection works fine, the probe responds to ping, and the leds were identical to its normal operational state. Disconnecting and reconnecting from network the probe do not change anything, and with a power off / on, after a few minutes, the only led that remains on is the power, no SOS message on the history, nothing. I will try with another usb stick...
Hide replies
Jean-Philippe LUCAS •
After my first post, my probe worked fine for 63days... And it crashed again and did not recover the connection. I swapped the philips stick with a new one, and i started to use the philips stick on my laptop... One time i used it to copy a big file, and the stick crashed. My opinion : as the usb stick was defective from almost the beginning, you probably have had a defective batch of them.
Steffen Uhlig •
Wiping the USB stick (it claims to be a Philips one) on my laptop and plugging it back into the probe ten minutes after boot time solved it for me.
Cobus Viljoen •
Hi Guys, is there a way to 'reset' the probe. i followed all the recovery procedures, but it does not seem to be getting a DHCP ip address? Anything else that i can possibly try?
Hide replies
Philip Homburg •
Hi, there is a rare case where a probe does not request a DHCP lease. If possible it is best to verify that this the case by connecting the probe without USB stick and then looking at DHCP with tcpdump or similar. Note that in equally rare cases the ethernet hardware breaks. If the probe does not request a DHCP lease due to a software problem then the probe still works on IPv6. So if possible please try to connect the probe to a network that supports IPv6.
David Růžička •
Hi, my probe #19242 is disconnected to day, after "usb stick dancing" still say disconnected, in SOS messages are this message: BADMD-APP-4780-d41d8cd98f00b204e9800998ecf8427e What's wrong? Thank you David
Hide replies
Mirjam Kühne •
Dear David, we will create a support ticket for you and will get back to you by email.
Peter Trieb •
Hi there! My probe #28614 has the same BADMD-APP error as Davids in the last comment. Already tried swapping usb sticks, please help! Best regards Peter
Hide replies
Philip Homburg •
This error means that downloading of the firmware failed. The best way forward is to send a mail to atlas@ripe.net to create a support ticket. Philip
Hide replies
Thomas Guadagnini •
My Probe had this issue for some weeks. I replaces the stick. But now I have an ugly all time uptime in the stats. Is there a way to remove this usb related downtime? Probe 27360
Juergen Weber •
What about the new probes which have been sent out since end of 2019?
Hide replies
Lia Hestina •
Hi Juergen, The latest version (V4) has no USB on it. Lia
Thomas Mieslinger •
One of my TL-MR3020 died during a thunderstrom. Can I flash a stock MR3020 with the ripe atlas image to get id 31917 alive again?
Hide replies
Philip Homburg •
Hi Thomas, If you are sure that it is not a USB problem, then it is (at the moment) best to replace the probe with a software probe. Though it is technically possible to open up a TL-MR3020 and reflash it, this is not something we support.
Ulrich Hauser •
Around of a month ago a v3 probe went offline in one of our data centers. With help of the recover information at Status tab we took the probe back to live again; after deleting the partitions on the usb memory stick. This helpful hints are still required in 2021, thanks for that. And a big WELL DONE for implementing a working recover function in the probe!
david hymonnet •
I've tried all tips in troubleshooting, and my probe #12472 is still disconnected witj message "USB Flash Drive Problem" or "filesystem corrupted". Tested with 3 usbkey. Is-it possible to manually flash the key ?