Update - Created this website to track Helium DIY Hotspot issues. Note that this is run by an independent third party to monitor Helium uptime, and not maintained by the official Helium entity. Official status page: http://status.helium.com/
Aug 28, 2020 - 07:22 UTC
Aug 28, 2020 - 07:22 UTC
Investigating - https://discord.com/channels/404106811252408320/730245219974381708/743943634214191175
8/14/2020 - First noticed that OTAA wasn't getting routed. It's possible it happened before this since this status monitoring was not automatic yet for this incident. Maybe this actually started on 8/10/2020 when Helium said: "Today also marks the end of free data packet transfer".
The monitoring account used for this has 1 million DC available, and I haven't changed the appkeys/etc. On the many devices I have on the console, I have tried removing one device and manually re-adding it. This did not help and multiple other users on Discord have since reported the same issue happening to them.
Original from Logs:
send_offer:556 OfferMsg1:
send_offer:64 sending offer:
send_offer:556 OfferMsg1:
send_offer:64 sending offer:
blockchain_state_channel_handler:handle_data:121 sc_handler client got purchase:
blockchain_state_channel_handler:handle_data:124 sc_handler client got rejection: {blockchain_state_channel_rejection_v1_pb,0}
blockchain_state_channels_client:handle_banner:354 reason: inactive_sc
blockchain_event:terminate:108 terminating remove_handler
blockchain_state_channels_client:handle_purchase:376 gen_server blockchain_state_channels_client terminated with reason: no match of right hand value {error,closed} in blockchain_state_channels_client:handle_purchase/3 line 376
Also more recently from Logs:
2020-08-28 04:42:40.891 [info] @blockchain_state_channel_handler:handle_data:124 sc_handler client got rejection: {blockchain_state_channel_rejection_v1_pb,0}
2020-08-28 04:42:40.891 [warning] @blockchain_state_channels_client:handle_cast:200 Got rejection: {blockchain_state_channel_rejection_v1_pb,0} for: , dropping packet
The error leaves a lot to be desired (is this coming from router code?), but my best guess is that it comes from here: https://github.com/helium/blockchain-core/blob/master/src/state_channel/v1/blockchain_state_channel_rejection_v1.erl
Aug 14, 2020 - 09:26 UTC
8/14/2020 - First noticed that OTAA wasn't getting routed. It's possible it happened before this since this status monitoring was not automatic yet for this incident. Maybe this actually started on 8/10/2020 when Helium said: "Today also marks the end of free data packet transfer".
The monitoring account used for this has 1 million DC available, and I haven't changed the appkeys/etc. On the many devices I have on the console, I have tried removing one device and manually re-adding it. This did not help and multiple other users on Discord have since reported the same issue happening to them.
Original from Logs:
send_offer:556 OfferMsg1:
send_offer:64 sending offer:
send_offer:556 OfferMsg1:
send_offer:64 sending offer:
blockchain_state_channel_handler:handle_data:121 sc_handler client got purchase:
blockchain_state_channel_handler:handle_data:124 sc_handler client got rejection: {blockchain_state_channel_rejection_v1_pb,0}
blockchain_state_channels_client:handle_banner:354 reason: inactive_sc
blockchain_event:terminate:108 terminating remove_handler
blockchain_state_channels_client:handle_purchase:376 gen_server blockchain_state_channels_client terminated with reason: no match of right hand value {error,closed} in blockchain_state_channels_client:handle_purchase/3 line 376
Also more recently from Logs:
2020-08-28 04:42:40.891 [info] @blockchain_state_channel_handler:handle_data:124 sc_handler client got rejection: {blockchain_state_channel_rejection_v1_pb,0}
2020-08-28 04:42:40.891 [warning] @blockchain_state_channels_client:handle_cast:200 Got rejection: {blockchain_state_channel_rejection_v1_pb,0} for: , dropping packet
The error leaves a lot to be desired (is this coming from router code?), but my best guess is that it comes from here: https://github.com/helium/blockchain-core/blob/master/src/state_channel/v1/blockchain_state_channel_rejection_v1.erl
Aug 14, 2020 - 09:26 UTC
Helium DIY Hotspot Routability Test
Major Outage
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.