DIY Hotspots not routing OTAA join-request/join-accept

Incident Report for Helium Uptime

Update

Created this website to track Helium DIY Hotspot issues. Note that this is run by an independent third party to monitor Helium uptime, and not maintained by the official Helium entity. Official status page: http://status.helium.com/
Posted Aug 28, 2020 - 07:22 UTC

Investigating

https://discord.com/channels/404106811252408320/730245219974381708/743943634214191175

8/14/2020 - First noticed that OTAA wasn't getting routed. It's possible it happened before this since this status monitoring was not automatic yet for this incident. Maybe this actually started on 8/10/2020 when Helium said: "Today also marks the end of free data packet transfer".

The monitoring account used for this has 1 million DC available, and I haven't changed the appkeys/etc. On the many devices I have on the console, I have tried removing one device and manually re-adding it. This did not help and multiple other users on Discord have since reported the same issue happening to them.

Original from Logs:
send_offer:556 OfferMsg1:
send_offer:64 sending offer:
send_offer:556 OfferMsg1:
send_offer:64 sending offer:
blockchain_state_channel_handler:handle_data:121 sc_handler client got purchase:
blockchain_state_channel_handler:handle_data:124 sc_handler client got rejection: {blockchain_state_channel_rejection_v1_pb,0}
blockchain_state_channels_client:handle_banner:354 reason: inactive_sc
blockchain_event:terminate:108 terminating remove_handler
blockchain_state_channels_client:handle_purchase:376 gen_server blockchain_state_channels_client terminated with reason: no match of right hand value {error,closed} in blockchain_state_channels_client:handle_purchase/3 line 376

Also more recently from Logs:
2020-08-28 04:42:40.891 [info] @blockchain_state_channel_handler:handle_data:124 sc_handler client got rejection: {blockchain_state_channel_rejection_v1_pb,0}
2020-08-28 04:42:40.891 [warning] @blockchain_state_channels_client:handle_cast:200 Got rejection: {blockchain_state_channel_rejection_v1_pb,0} for: , dropping packet

The error leaves a lot to be desired (is this coming from router code?), but my best guess is that it comes from here: https://github.com/helium/blockchain-core/blob/master/src/state_channel/v1/blockchain_state_channel_rejection_v1.erl
Posted Aug 14, 2020 - 09:26 UTC
This incident affects: Helium DIY Hotspot Routability Test.