Fellow Cosmonauts,
stkATOM was initially planned for launch on 8 November 2022. In this post, we take a fully transparent deep-dive into the events that transpired since then, which led to a 2-month delay.
Two major incidents that were uncontrollable by the pSTAKE team happened on 6 November 2022:
- Persistence Core-1 Chain Halt
- A bad actor skewed the exchange rate of stkATOM to ATOM on the Persistence Core-1 Chain Mainnet
Background
Persistence Core-1 Chain Halt
On 6 November 2022, the Persistence Core-1 chain halted at the block height of 8,647,535 for ~4.5 hours. The chain halted due to a node non-determinism issue caused by differing Golang versions used by the validators. As a result, the ICA (Interchain Account) channel created by the controller chain (Persistence Core-1) on the host chain (CosmosHub) got closed. pSTAKE’s LS (Liquid Staking) Module for ATOM was working as intended and remained unaffected.
The full post-mortem of the chain halt can be found here.
In the following days, the pSTAKE team worked closely with the Core-1 team to propose a new Core-1 chain v5 upgrade. This gave pstake-native v1.2.0 the ability to reopen IBC channels for ICA/ICQ on channel close as a transaction. Consequently, Proposal #9 on the Persistence Core-1 chain passed on 22 November 2022. The chain upgrade was seamless & everything went as expected. In the coming days, the pSTAKE team spent time testing the product on the Persistence Mainnet before the UI was made public for all users. Internally, the team targeted a new launch date of 1 December 2022 which still needed to be publicly communicated.
Bad Actor skewing the exchange rate
On 6 November 2022, a bad actor created a transaction of sending 45 ATOM to the pSTAKE reward address that resulted in the exchange rate between stkATOM and ATOM skew drastically. This is the same as an unknown depositor performing a transaction to boost the rewards paid (Assets sent to the reward rate address are distributed to the stkATOM holders. All the rewards generated on the ATOM staked via the deposits to pSTAKE are claimed to this address and subsequently restaked to reflect the growing holdings of the users. This results in the increase in the exchange rate of stkATOM to ATOM) to stkATOM holders. The total amount of ATOM staked (Total Value Unlocked or TVU) was only ~50-60 ATOM at the time of Mainnet testing by the pSTAKE team. This resulted in the exchange rate becoming 1 stkATOM = 1.67 ATOM.
The bad actor skewed the exchange rate in this transaction from the account address.
Although this incident occurred on 6 November 2022, the pSTAKE team could identify this discrepancy only on 23 November 2022 because the team was focused on the issue of the chain halt. The exchange rate before the chain halt was still ~1:1. The ICA channel between Core-1 & the CosmosHub reopened on 22 November 2022 as part of the v5 chain upgrade. Post which, the first rewards auto-compounding epoch (since the chain halt on 6 November 2022) took place on 23 November 2022, bringing into light the skewed exchange rate.
Tackling the Skewed Exchange Rate
From here on, the pSTAKE team spent time brainstorming & discussing the various paths to launch, keeping our core principles—Security, User Experience & Utility in mind. Here’s an overview of all three options in consideration:
Option 1: Launch with a non-1:1 stkATOM:ATOM Exchange Rate
What would need to be done?
To reset the exchange rate, the pSTAKE team would need to unstake all ATOM staked while testing the product on Mainnet. However, if any stkATOM remains, the exchange rate would be unchanged. If this is not possible, the pSTAKE team would have to bump the exchange rate on purpose (with a transaction similar to the bad actor’s) before the product launch to set a clean ratio of, e.g., 1:2 or 1:10 stkATOM to ATOM.
Pros
- No further delay in launching the product
Cons
- Major UX confusion for users while minting stkATOM as this would be the first liquid staking product that will not have a 1:1 exchange rate at genesis
- Possible composability issues for stkATOM within Cosmos DeFi in the future
- Possible confusion for users when interacting with stkATOM’s integrations in the future
- Possible issue of confusion with other stkASSETs to be launched by pSTAKE in the future
Risk → Risk Mitigation
- The bad actor or any arbitrary address performs a similar transaction of sending tokens to the reward address to mess up the exchange rate again → No possible mitigation in this scenario
Conclusion
As the product was working as intended from a technical & security perspective, the pSTAKE team’s initial thought was to launch stkATOM with a non-1:1 exchange rate. But after having spent time thinking about the potential second-order effects of going ahead with this & keeping UX at the forefront, the pSTAKE team decided against launching on 1 December 2022.
Option 2: Do a chain upgrade to start from scratch with a new stkATOM token
What would need to be done?
The pSTAKE team would need to redo upgrades done so far to launch stkATOM by either creating a new module with new addresses or clobbering the existing module. This would involve modifying the IBC handshake logic as the new module will be a middleware of middleware (having multiple modules work via ICA requires the controller middleware to be wrapped one inside the other). Although the newer upgrade of IBC & wasm would make this easier, they are yet to be released.
Pros
- No UX confusion for users while minting stkATOM
- It avoids composability issues within Cosmos DeFi or unforeseen second-order effects of launching with a non-~1:1 exchange rate
Cons
- Significant delay in the launch of stkATOM with major work on the LScosmos module (pSTAKE’s module for stkATOM on Persistence) itself & code refactoring when the chain migrates to IBC v6 or v8
- The existing product & tokens would be left as is & could create confusion among users in the future
Risk → Risk Mitigation
- Additional security risk with increases in test cases to ensure any other ICA channels do not execute logic for the new one → No possible mitigation
Conclusion
After carefully considering the pros & cons of going ahead in this direction, the pSTAKE team concluded that this option adds additional risk & significant time delay, making it a not-so-optimal solution.
Option 3: Request the Core-1 chain team to Upgrade the chain for a one-time mint of 45 additional stkATOM to reset the Exchange Rate and distribute the stkATOM to the bad actor’s Persistence address
What would need to be done?
As mentioned earlier, the current exchange rate of stkATOM:ATOM was 1.67 due to the bad actor’s extra 45 ATOM. This means that the difference between the supply of stkATOM & ATOM deposited into the pSTAKE protocol is 45. In this scenario, the exchange rate can be reset if there exists 45 more stkATOM against the bad actor’s ATOM. The pSTAKE team will need to develop a one-time executable logic to mint stkATOM & send it to the bad actor’s Persistence address.
Pros
- No UX confusion for users while minting stkATOM
- It avoids composability issues within Cosmos DeFi or unforeseen second-order effects of launching with a non-~1:1 exchange rate
Cons
- Further delay in the launch of stkATOM
Risk → Risk Mitigation
- Additional security risk & potential exploit vector of introducing code that mints stkATOM one-time → This code should be an instruction in the v6 PersistenceCore upgrade handler and should be replaced in the consequent v7 upgrade to ensure it executes only once
- The bad actor or any arbitrary address performs a similar transaction of sending tokens to the reward address to mess up the exchange rate again → At launch, the auto-compounded staking reward rate for stkATOM should be capped at a maximum 25% APY. This ensures that regardless of the number of tokens received by the reward address, the reward rate can never be > 25% APY. The max reward rate is a dynamic number and can be updated via governance to reflect a change in the Cosmos Hub staking rewards in the future (i.e., if staking rewards on the Cosmos Hub increase to a point where it is more than 25%)
- Trust issues → Since this issue was identified before the product was publicly live, the pSTAKE team can be at the discretion of mitigating it as deemed fit. If any such problems occur after the product goes live, the pSTAKE team will always let the protocol governance decide the way ahead
Conclusion
After careful consideration & weighing out all options, the pSTAKE team decided to go ahead with this solution as it is the only one that keeps pSTAKE’s all three core principles—Security, User Experience & Utility at the forefront. The pSTAKE team has already implemented the above risk mitigation as part of the v6 PersistenceCore upgrade.
As an additional security measure, the pSTAKE team will not be boosting native staking rewards with additional incentives in ATOM for a 2-month period from launch, which was communicated earlier.
Timeline Summary
| Date | What happened? |
|---|---|
| 1 Nov 22 | Public announcement of stkATOM launch on 8 Nov 22 |
| 6 Nov 22 | Bad actor skews stkATOM Exchange Rate |
| 6 Nov 22 | The ICA channel between CosmosHub & Core-1 closes due to Persistence Core-1 chain halt |
| 15 Nov 22 | Persistence Core-1 v5 upgrade proposal goes live |
| 22 Nov 22 | Persistence Core-1 v5 upgrade proposal passes |
| 23 Nov 22 | The first rewards auto-compounding epoch occurs since the chain halt on 6 Nov 22, bringing light to the skew in the exchange rate |
| 24 Nov 22 – 14 Dec 22 | The pSTAKE team spent time brainstorming all possible options & decided to go ahead with Option 3 (as mentioned in this post) |
| 14 Dec 22 – 1 Jan 23 | The pSTAKE team worked on solving the issue at hand in line with Option 3 |
| 2 Jan 23 | Persistence Core-1 v6 upgrade proposal goes live |
| 9 Jan 23 | Persistence Core-1 v6 upgrade proposal voting ends |
| 11 Jan 23 | Persistence Core-1 v6 upgrade occurs* (if Proposal #12 passes) |
| 12 Jan 23 | stkATOM goes live on Persistence Core-1* (if Proposal #12 passes) |
Moving Ahead
One of the key lessons the pSTAKE team has learned over time working on multiple liquid staking products across multiple Proof-of-Stake ecosystems is that things may take time. We believe that it is okay to be late than to launch a subpar product if it means that we don’t compromise on any of our core principles—Security, User Experience & Utility.
The stkATOM launch delay has not only shown us the challenges of building a decentralized liquid staking product, which the pSTAKE team is fully committed to but also given us a chance to build an even more secure product for the Interchain.
stkATOM was the first liquid staking product for ATOM that went live in early 2021. The pSTAKE team, in its full capacity, will aim to make the new implementation of stkATOM the best liquid staking product for ATOM when it re-launches in the Cosmos on 12 January 2023.
About pSTAKE
pSTAKE is a liquid staking protocol that unlocks liquidity for your staked assets. With pSTAKE, you can securely stake your Proof-of-Stake (PoS) assets, participate in protocol improvements and security to earn staking rewards, and receive staked underlying representative tokens (stkASSETs) which can be used to explore additional yield opportunities across DeFi.
At present, pSTAKE supports Binance Chain (BNB), Cosmos (ATOM), Persistence (XPRT), and Ethereum (ETH) networks’ native tokens, with a view to support more chains and assets in the future.
