Unbork Emerynet: Cleanup Auctioneers, PriceAuthorities, and QuoteNotifiers #10725

toliaqat · 2024-12-17T23:56:27Z

What is the Problem Being Solved?

Emerynet currently suffers from severe performance degradation due to multiple generations of auctioneer vats running simultaneously. This has led to the accumulation of thousands of ephemeral observers and QuoteNotifiers (QNs). Each additional price submission triggers massive computations and delays, causing push-price operations to run for hours. The problem is aggravated by old auctioneers continually waking up on timers, creating new observer chains and further inflating the backlog of QNs. This situation makes Emerynet slow, unresponsive, and difficult to use for testing and validation.

Description of the Design

The proposed solution involves a multi-step cleanup and reset process:

(Optional) Halt Auctioneer Timers (for Gen4/Gen5)

Attempt to stop timers associated with problematic auctioneers (e.g., v661 and v665) to prevent further growth of observers/QNs. If this isn't possible via existing governance tools, move directly to terminating these vats.

Terminate Old Auctioneer Vats

Use governance actions (coreProposals) to kill all old auctioneers (including gen1 through gen5) on Emerynet. Removing these vats will cascade a cleanup of their associated observers, QNs, and references in scaledPriceAuthorities and priceFeeds over time.

Submit Prices for All Denoms

Push updated prices for every denom to trigger the release and garbage collection of QN references in scaledPriceAuthorities (sPA) and priceFeeds (pf). While each price submission will take hours initially, this step eventually clears the backlog of QNs and observers.

Create a New, Clean Auctioneer

After full cleanup, instantiate a fresh auctioneer vat that references updated priceAuthorities without legacy baggage. This ensures a stable and clean testing environment going forward.

Monitor and Verify Stability

Observe Emerynet after the cleanup to ensure:

Regular auction cycles proceed without excessive overhead.
Price pushes complete in minutes, not hours.
No runaway growth of QNs or observers reoccurs.

Security Considerations

Terminating vats and pushing prices are governed actions.
Ensure termination and re-creation of vats follow proper governance proposals to maintain chain security and trust.
Verify that no active purses or valuable resources remain locked in terminated vats.

Scaling Considerations

The cleanup process itself may be slow and computationally expensive, involving many hours of block time.
Once complete, the load and resource usage should return to manageable levels, enabling Emerynet to scale normally again.
This operation provides insights into future upgrades and how to handle large-scale cleanup to avoid performance bottlenecks.

Test Plan

Before Cleanup:
Measure the current backlog and record how long price submissions take.
During Cleanup:
- Terminate old auctioneer vats and push prices.
- Continuously monitor logs and system metrics to ensure processes are proceeding.
After Cleanup:
- Confirm that QNs and observers have been dramatically reduced.
- Push a new price and confirm it completes within minutes.
- Run a series of standard Emerynet tests to ensure normal operation.

Upgrade Considerations

Future upgrades should include mechanisms to cleanly stop or replace auctioneers and related infrastructures without leaving behind accumulations of QNs and observers.
Consider revisiting the design of scaledPriceAuthorities and priceFeeds to avoid complex observer chains.
Document lessons learned to ensure mainnet upgrades (like U18/U19) won’t encounter similar issues.

Sub-Tasks:

Investigate Timer Controls for Auctioneers
- Determine whether it's possible to halt timers in gen4+gen5 auctioneers. If not, proceed to termination.
Terminate Old Auctioneer Vats
- Identify and terminate old auctioneers (v39, v353, v367, v661, v665).
- Use a coreProposal to kill these vats and confirm dropImports/retireImports behavior in kernel logs.
Submit Prices for All Denoms
- Push updated prices for each managed denom to trigger cleanup.
- Monitor the process; expect hours of block time but ensure ultimate QN reduction.
Confirm QN and Observer Cleanup
- Verify that the number of QNs/observers in sPA and pf vats decreases significantly.
- Ensure no new QNs accumulate during or after the process.
Create a New Auctioneer
- After full cleanup, instantiate a new, clean auctioneer via a coreProposal.
- Confirm that it references a stable, simplified priceAuthority setup.
Monitor Post-Cleanup Stability
Run test pushes of prices.
- Validate normal operation and confirm that no runaway QN growth occurs again.

warner · 2024-12-26T21:32:21Z

@siarhei-agoric : for reference, #9483 (comment) has a lot of notes about how to find the necessary objects to terminate the old vats. And @gibson042 probably has even more ideas.

Chris-Hibbert · 2025-01-02T18:34:33Z

Determine whether it's possible to halt timers in gen4+gen5 auctioneers. If not, proceed to termination.

The auctioneers are not designed to be halted, but it is straightforward to give them a long schedule. For instance a week.

The issue with this approach is that it's not possible to reset the schedule to a shorter period until that time elapses. So choosing a very long time (like a year) effectively disables that auction, since if you wanted to restart it to a shorter period, you'd have to wait for that year to elapse before the auctioneer would pay attention to a shorter period.

It's possible that giving them inconsistent parameters would turn the schedule off until a new set of parameters are supplied, but I haven't tested this to be sure what parameters would have this effect. If turning the schedule off were a priority, I could run some tests and likely find a workable approach.

siarhei-agoric · 2025-01-03T00:14:08Z

Based on multiple conversations with @gibson042, @mhofman, and @Chris-Hibbert, along with some digging:

it should be safe to terminate old auctioneer vats (v39, v353, v367, v661, v665) in any order, regardless of any reference/message loops that they may have
at least one auctioneer vat (v39) does NOT have its adminFacet exported by its governor (v38, zcf-b1-9f877-auctioneer.governor)
the adminFacet is available within v38's baggage and can be accessed via vat upgrade path
we should be able to use @gibson042 's message injection / simulation tool test out vat terminations via both core eval and upgrade paths without having to bring up the whole chain.
the actual unborking will be rolled out as an upgrade with couple of special core evals within it to kill off the problematic vats.

Timeline (rough estimates):

track down all the remaining adminFacet refs: Jan. 3, 2025
use the message injection / simulation tool to test the vat killing: Jan. 6, 2025
write up core eval / proposal: Jan. 7, 2025
PR reviews: Jan. 8, 2025
cut a special release off u18/19 branch: Jan. 9, 2025
publish a governing action / proposal: Jan 10, 2025
upgrade, vats killed: some time soon after the proposal is voted in and adopted.

refs: #10725 Initial set of changes to get auctioneer governors terminated.

mhofman · 2025-01-14T03:37:37Z

Emerynet has halted while processing block 8667375 because the vat for the vaultFactory ran out of memory (due to the notifier bug above). See #10841

While the halt is unexpected, it happened for everyone, and the problematic block didn't get committed. As such we can restart the network with a new version that includes the unborking proposal and the upgrade to the final u18 code. However we'll need to hack the upgrade trigger mechanism since we can't use the normal upgrade module voting flow to cause the in-consensus upgrade. Most likely we can hard code the block number (likely verifying the current chain name and upgrade plan is rc3), and manually set the upgrade info that way. The final u18 also triggered an upgrade of the vbank module, so we likely need to somehow inject an upgrade plan to trigger all expected upgrades through the upgrade module.

This emerynet specific upgrade will now also require that we update the allocation limit in xsnap-worker. This is a hard coded value at 2GB that is not saved anywhere in the snapshot, and can be raised if in consensus. It's ok to use a one off specific change since we need a specific chain software for this release anyway.

refs: #10725 Initial set of changes to get auctioneer governors terminated.

refs #10725

refs: #10725 This change includes: * A single-file core-eval builder-proposal to terminate governed contracts and their governors by boardID. * A bootstrap test which: ** creates an instance of a simple governed contract via its own core-eval ** confirms that the contract accessible and operational ** kill the contract instance by boardID ** confirms that the contract is no longer operational

refs #10725

…ndler refs: #10725

refs: #10725

siarhei-agoric · 2025-01-23T19:13:55Z

Board ID lookup process:

v39 - auctioneer

sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v39.options';
zcf-b1-a5683-auctioneer

sqlite> SELECT * FROM kvStore WHERE key >= 'v2.' AND key < 'v2/' AND value LIKE '%v39%';
v2.vs.vom.o+d13/34|{"vatID":{"body":"#\"v39\"","slots":[]}}
v2.vs.vom.o+d13/385|{"vatID":{"body":"#\"v390\"","slots":[]}}
v2.vs.vom.o+d13/386|{"vatID":{"body":"#\"v391\"","slots":[]}}
v2.vs.vom.o+d13/387|{"vatID":{"body":"#\"v392\"","slots":[]}}
v2.vs.vom.o+d13/388|{"vatID":{"body":"#\"v393\"","slots":[]}}
v2.vs.vom.o+d13/389|{"vatID":{"body":"#\"v394\"","slots":[]}}
v2.vs.vom.o+d13/390|{"vatID":{"body":"#\"v395\"","slots":[]}}
v2.vs.vom.o+d13/391|{"vatID":{"body":"#\"v396\"","slots":[]}}
v2.vs.vom.o+d13/392|{"vatID":{"body":"#\"v397\"","slots":[]}}
v2.vs.vom.o+d13/393|{"vatID":{"body":"#\"v398\"","slots":[]}}
v2.vs.vom.o+d13/394|{"vatID":{"body":"#\"v399\"","slots":[]}}

sqlite> SELECT * FROM kvStore WHERE key='v2.c.o+d13/34';
v2.c.o+d13/34|ko521

sqlite> SELECT * FROM kvStore WHERE key='v9.c.ko521';
v9.c.ko521|R o-258

sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v9.options';
zoe

sqlite> SELECT * FROM kvStore WHERE key >= 'v9.' AND key < 'v9/' AND value LIKE '%o-258%';
v9.c.ko521|R o-258
<…>
v9.vs.vom.o+d37/26|{"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-258"]},"zcfBundleCap":{"body":"#\"$0.Alleged: device node\"","slots":["d-70"]},"contractBundleCap":{"body":"#\"$0.Alleged: device node\"","slots":["d-79"]}}

sqlite> SELECT * FROM kvStore WHERE key='v9.c.o+d37/26';
v9.c.o+d37/26|ko551

sqlite> SELECT * FROM kvStore WHERE key >= 'v1' AND key < 'v:' AND substr(key, -length('.c.ko551')) = '.c.ko551' AND key LIKE 'v%.c.ko551';
v38.c.ko551|R o-69
v9.c.ko551|R o+d37/26

sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v38.options';
zcf-b1-9f877-auctioneer.governor

sqlite> SELECT * FROM kvStore WHERE key >= 'v38.' AND key < 'v38/' AND value LIKE '%o-69%';
v38.c.ko551|R o-69
v38.vs.vom.o+d21/1|{"adminFacet":{"body":"#\"$0.Alleged: adminFacet\"","slots":["o-69"]},"creatorFacet":{"body":"#\"$0.Alleged: governorFacet\"","slots":["o-70"]},"creatorInvitation":{"body":"#\"#undefined\"","slots":[]},"instance":{"body":"#\"$0.Alleged: InstanceHandle\"","slots":["o-71"]},"publicFacet":{"body":"#\"$0.Alleged: publicFacet\"","slots":["o-72"]},"limitedCreatorFacet":{"body":"#\"$0.Alleged: Auctioneer creatorFacet\"","slots":["o-73"]},"currentInvitation":{"body":"#\"$0.Alleged: Zoe Invitation payment\"","slots":["o-67"]}}

sqlite> SELECT * FROM kvStore WHERE key='v38.c.o+d21/1'; — There is no kref to adminFacet for v39. We need to upgrade the zcf-b1-9f877-auctioneer.governor 

sqlite> SELECT * FROM kvStore WHERE key >= 'v38.' AND key < 'v38/' AND value LIKE '%o+d21/1%';
v38.c.ko556|R o+d21/1:0
v38.c.ko558|R o+d21/1:2
v38.vs.vc.1.screatorFacet|{"body":"#\"$0.Alleged: ContractGovernorKit creator\"","slots":["o+d21/1:0"]}
v38.vs.vc.1.spublicFacet|{"body":"#\"$0.Alleged: ContractGovernorKit public\"","slots":["o+d21/1:2"]}
v38.vs.vc.9.scontractGovernorKit|{"body":"#{\"creator\":\"$0.Alleged: ContractGovernorKit creator\",\"helper\":\"$1.Alleged: ContractGovernorKit helper\",\"public\":\"$2.Alleged: ContractGovernorKit public\"}","slots":["o+d21/1:0","o+d21/1:1","o+d21/1:2"]}

Upgrade of v38 (governor) will be required to terminate v39: get “creatorFacet” from baggage and use it to terminate

v38 - governor for v39

sqlite> SELECT * FROM kvStore WHERE key >= 'v2.' AND key < 'v2/' AND value LIKE '%v38%';
v2.vs.vom.o+d13/33|{"vatID":{"body":"#\"v38\"","slots":[]}}
v2.vs.vom.o+d13/375|{"vatID":{"body":"#\"v380\"","slots":[]}}
v2.vs.vom.o+d13/376|{"vatID":{"body":"#\"v381\"","slots":[]}}
v2.vs.vom.o+d13/377|{"vatID":{"body":"#\"v382\"","slots":[]}}
v2.vs.vom.o+d13/378|{"vatID":{"body":"#\"v383\"","slots":[]}}
v2.vs.vom.o+d13/379|{"vatID":{"body":"#\"v384\"","slots":[]}}
v2.vs.vom.o+d13/380|{"vatID":{"body":"#\"v385\"","slots":[]}}
v2.vs.vom.o+d13/381|{"vatID":{"body":"#\"v386\"","slots":[]}}
v2.vs.vom.o+d13/382|{"vatID":{"body":"#\"v387\"","slots":[]}}
v2.vs.vom.o+d13/383|{"vatID":{"body":"#\"v388\"","slots":[]}}
v2.vs.vom.o+d13/384|{"vatID":{"body":"#\"v389\"","slots":[]}}

sqlite> SELECT * FROM kvStore WHERE key='v2.c.o+d13/33';
v2.c.o+d13/33|ko490

sqlite> SELECT * FROM kvStore WHERE key='v9.c.ko490';
v9.c.ko490|R o-240

sqlite> SELECT * FROM kvStore WHERE key >= 'v9.' AND key < 'v9/' AND value LIKE '%o-240%';
v9.c.ko490|R o-240
v9.c.ko5134|_ o-2400
v9.c.ko5136|R o-2401
v9.c.ko5139|R o-2402
< … >
v9.vs.vom.o+d31/756|{"currentAllocation":{"body":"#{\"Collateral\":{\"brand\":\"$0.Alleged: ATOM brand\",\"value\":\"+204221\"}}","slots":["o-239"]},"proposal":{"body":"#{\"exit\":{\"onDemand\":null},\"give\":{},\"want\":{}}","slots":[]},"exitObj":{"body":"#\"$0.Alleged: ExitObject\"","slots":["o-2408"]},"offerResult":{"body":"#\"#undefined\"","slots":[]},"offerResultStored":{"body":"#true","slots":[]},"instanceAdminHelper":{"body":"#\"$0.Alleged: instanceAdmin helper\"","slots":["o+d33/30:0"]},"withdrawFacet":{"body":"#\"$0.Alleged: InstanceStorageManager withdrawFacet\"","slots":["o+d26/30:2"]},"publisher":{"body":"#\"$0.Alleged: zoe Seat publisher publisher\"","slots":["o+d30/756:0"]},"subscriber":{"body":"#\"$0.Alleged: zoe Seat publisher subscriber\"","slots":["o+d30/756:1"]},"payouts":{"body":"#{\"Collateral\":\"$0.Alleged: ATOM payment\"}","slots":["o-2410"]},"exiting":{"body":"#true","slots":[]}}
v9.vs.vom.o+d33/26|{"offerFilterStrings":{"body":"#[]","slots":[]},"publicFacet":{"body":"#\"$0.Alleged: ContractGovernorKit public\"","slots":["o-274"]},"handleOfferObj":{"body":"#\"$0.Alleged: handleOfferObj\"","slots":["o-273"]},"zoeInstanceStorageManager":{"body":"#\"$0.Alleged: InstanceStorageManager instanceStorageManager\"","slots":["o+d26/26:1"]},"seatHandleToZoeSeatAdmin":{"body":"#\"$0.Alleged: weakMapStore\"","slots":["o+d7/24"]},"instanceHandle":{"body":"#\"$0.Alleged: InstanceHandle\"","slots":["o+d29/26"]},"acceptingOffers":{"body":"#true","slots":[]},"zoeSeatAdmins":{"body":"#\"$0.Alleged: setStore\"","slots":["o+d8/80"]},"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-240"]}}
v9.vs.vom.o+d34/26|{"instanceStorage":{"body":"#\"$0.Alleged: InstanceStorageManager instanceStorageManager\"","slots":["o+d26/26:1"]},"instanceAdmin":{"body":"#\"$0.Alleged: instanceAdmin instanceAdmin\"","slots":["o+d33/26:1"]},"seatHandleToSeatAdmin":{"body":"#\"$0.Alleged: weakMapStore\"","slots":["o+d7/24"]},"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-240"]}}
v9.vs.vom.o+d37/28|{"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-240"]},"zcfBundleCap":{"body":"#\"$0.Alleged: device node\"","slots":["d-70"]},"contractBundleCap":{"body":"#\"$0.Alleged: device node\"","slots":["d-75"]}}

sqlite> SELECT * FROM kvStore WHERE key='v9.c.o+d37/28';
v9.c.o+d37/28|ko561

sqlite> SELECT * FROM kvStore WHERE key >= 'v1' AND key < 'v:' AND substr(key, -length('.c.ko561')) = '.c.ko561' AND key LIKE 'v%.c.ko561';
v1.c.ko561|R o-313
v9.c.ko561|R o+d37/28

sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value LIKE '%o-313%';
v1.c.ko561|R o-313
v1.vs.vc.8.r0000000003:o-317|{"body":"#{\"adminFacet\":\"$0.Alleged: adminFacet\",\"creatorFacet\":\"$1.Alleged: Auctioneer creatorFacet\",\"governor\":\"$2.Alleged: InstanceHandle\",\"governorAdminFacet\":\"$0\",\"governorCreatorFacet\":\"$3.Alleged: ContractGovernorKit creator\",\"instance\":\"$4.Alleged: InstanceHandle\",\"label\":\"auctioneer\",\"publicFacet\":\"$5.Alleged: publicFacet\"}","slots":["o-313","o-318","o-315","o-314","o-317","o-319"]}
sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value LIKE '%o-317%';
v1.c.ko524|R o-317
v1.vs.vc.8.r0000000003:o-317|{"body":"#{\"adminFacet\":\"$0.Alleged: adminFacet\",\"creatorFacet\":\"$1.Alleged: Auctioneer creatorFacet\",\"governor\":\"$2.Alleged: InstanceHandle\",\"governorAdminFacet\":\"$0\",\"governorCreatorFacet\":\"$3.Alleged: ContractGovernorKit creator\",\"instance\":\"$4.Alleged: InstanceHandle\",\"label\":\"auctioneer\",\"publicFacet\":\"$5.Alleged: publicFacet\"}","slots":["o-313","o-318","o-315","o-314","o-317","o-319"]}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1' AND key < 'v:' AND substr(key, -length('.c.ko524')) = '.c.ko524' AND key LIKE 'v%.c.ko524';
v1.c.ko524|R o-317
v15.c.ko524|R o-73
v38.c.ko524|R o-71
v39.c.ko524|R o-54
v43.c.ko524|R o-57
v6.c.ko524|_ o-124
v64.c.ko524|R o-57
v65.c.ko524|R o-57
v7.c.ko524|R o-110
v9.c.ko524|R o+d29/27

sqlite> SELECT * FROM kvStore WHERE key >= 'v7.' AND key < 'v7/' AND value LIKE '%o-110%';
v7.c.ko1184225|_ o-1100
v7.c.ko1184287|_ o-1101
v7.c.ko1184480|_ o-1102
v7.c.ko1184487|_ o-1103
v7.c.ko1184505|_ o-1104
v7.c.ko1184523|_ o-1105
v7.c.ko1184568|_ o-1106
v7.c.ko1185011|_ o-1107
v7.c.ko1185031|_ o-1108
v7.c.ko1185051|_ o-1109
v7.c.ko524|R o-110
v7.vs.vc.5.sboard04149|{"body":"#\"$0.Alleged: InstanceHandle\"","slots":["o-110"]}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value GLOB '*"o+??/8"*';
v1.vs.vc.1.sGovernedContractKits|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/8"]}
v1.vs.vc.5.sgovernedContractKits|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/8"]}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value GLOB '*"o+??/5"*';
v1.vs.vc.1.sBootstrap Powers|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/5"]}
v1.vs.vc.5.spowerStore|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/5"]}

v665 - auctioneer:

sqlite> -- VAT v665
sqlite>
sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v1.options';
bootstrap
sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v2.options';
vatAdmin
sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v9.options';
zoe
sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v665.options';
zcf-b1-43453-auctioneer

sqlite> SELECT * FROM kvStore WHERE key >= 'v2.' AND key < 'v2/' AND value LIKE '%v665%';
v2.vs.vom.o+d13/660|{"vatID":{"body":"#\"v665\"","slots":[]}}

sqlite> SELECT * FROM kvStore WHERE key='v2.c.o+d13/660';
v2.c.o+d13/660|ko2441877

sqlite> SELECT * FROM kvStore WHERE key='v9.c.ko2441877';
v9.c.ko2441877|R o-950033

sqlite> SELECT * FROM kvStore WHERE key >= 'v9.' AND key < 'v9/' AND value LIKE '%o-950033%';
v9.c.ko2441877|R o-950033
v9.vs.vc.4.sp+39991|{"body":"#[[\"$0.Alleged: InstanceCompletionWatcher\",\"$1.Alleged: instanceAdmin instanceAdmin\",\"$2.Alleged: adminNode\"]]","slots":["o+d400/1","o+d33/648:1","o-950033"]}
v9.vs.vom.o+d26/648|{"instanceState":{"body":"#\"$0.Alleged: InstanceRecord\"","slots":["o+d27/648"]},"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-950033"]},"root":{"body":"#\"$0.Alleged: undefined\"","slots":["o-950034"]},"functions":{"body":"#\"#undefined\"","slots":[]}}
v9.vs.vom.o+d33/648|{"offerFilterStrings":{"body":"#[]","slots":[]},"publicFacet":{"body":"#\"$0.Alleged: publicFacet\"","slots":["o-950039"]},"handleOfferObj":{"body":"#\"$0.Alleged: handleOfferObj\"","slots":["o-950038"]},"zoeInstanceStorageManager":{"body":"#\"$0.Alleged: InstanceStorageManager instanceStorageManager\"","slots":["o+d26/648:1"]},"seatHandleToZoeSeatAdmin":{"body":"#\"$0.Alleged: weakMapStore\"","slots":["o+d7/24"]},"instanceHandle":{"body":"#\"$0.Alleged: InstanceHandle\"","slots":["o+d29/648"]},"acceptingOffers":{"body":"#true","slots":[]},"zoeSeatAdmins":{"body":"#\"$0.Alleged: setStore\"","slots":["o+d8/4754"]},"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-950033"]}}
v9.vs.vom.o+d34/648|{"instanceStorage":{"body":"#\"$0.Alleged: InstanceStorageManager instanceStorageManager\"","slots":["o+d26/648:1"]},"instanceAdmin":{"body":"#\"$0.Alleged: instanceAdmin instanceAdmin\"","slots":["o+d33/648:1"]},"seatHandleToSeatAdmin":{"body":"#\"$0.Alleged: weakMapStore\"","slots":["o+d7/24"]},"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-950033"]}}
v9.vs.vom.o+d37/647|{"adminNode":{"body":"#\"$0.Alleged: adminNode\"","slots":["o-950033"]},"contractBundleCap":{"body":"#\"$0.Alleged: device node\"","slots":["d-105"]}}

sqlite> SELECT * FROM kvStore WHERE key='v9.c.o+d37/647';
v9.c.o+d37/647|ko2441889

sqlite> SELECT * FROM kvStore WHERE key >= 'v1' AND key < 'v:' AND substr(key, -length('.c.ko2441889')) = '.c.ko2441889' AND key LIKE 'v%.c.ko2441889';
v1.c.ko2441889|R o-1825
v664.c.ko2441889|R o-69
v9.c.ko2441889|R o+d37/647

sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v664.options';
zcf-b1-09034-auctioneer.governor

sqlite> SELECT * FROM kvStore WHERE key >= 'v664.' AND key < 'v664/' AND value LIKE '%o-69%';
v664.c.ko2441889|R o-69
v664.vs.vom.o+d21/1|{"adminFacet":{"body":"#\"$0.Alleged: adminFacet\"","slots":["o-69"]},"creatorFacet":{"body":"#\"$0.Alleged: governorFacet\"","slots":["o-70"]},"creatorInvitation":{"body":"#\"#undefined\"","slots":[]},"instance":{"body":"#\"$0.Alleged: InstanceHandle\"","slots":["o-71"]},"publicFacet":{"body":"#\"$0.Alleged: publicFacet\"","slots":["o-72"]},"limitedCreatorFacet":{"body":"#\"$0.Alleged: Auctioneer creatorFacet\"","slots":["o-73"]},"currentInvitation":{"body":"#null","slots":[]}}

sqlite>  SELECT * FROM kvStore WHERE key='v664.c.o+d21/1';

sqlite> SELECT * FROM kvStore WHERE key >= 'v664.' AND key < 'v664/' AND value LIKE '%o+d21/1%';
v664.c.ko2441891|R o+d21/1:0
v664.c.ko2441893|R o+d21/1:2
v664.vs.vc.1.screatorFacet|{"body":"#\"$0.Alleged: ContractGovernorKit creator\"","slots":["o+d21/1:0"]}
v664.vs.vc.1.spublicFacet|{"body":"#\"$0.Alleged: ContractGovernorKit public\"","slots":["o+d21/1:2"]}
v664.vs.vc.9.scontractGovernorKit|{"body":"#{\"creator\":\"$0.Alleged: ContractGovernorKit creator\",\"helper\":\"$1.Alleged: ContractGovernorKit helper\",\"public\":\"$2.Alleged: ContractGovernorKit public\"}","slots":["o+d21/1:0","o+d21/1:1","o+d21/1:2"]}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value LIKE '%o-1825%';
v1.c.ko2441889|R o-1825
v1.vs.vc.5.sauctioneerKit|{"body":"#{\"adminFacet\":\"$0.Alleged: adminFacet\",\"creatorFacet\":\"$1.Alleged: Auctioneer creatorFacet\",\"governor\":\"$2.Alleged: InstanceHandle\",\"governorAdminFacet\":\"$3.Alleged: adminFacet\",\"governorCreatorFacet\":\"$4.Alleged: ContractGovernorKit creator\",\"instance\":\"$5.Alleged: InstanceHandle\",\"label\":\"auctioneer\",\"publicFacet\":\"$6.Alleged: publicFacet\"}","slots":["o-1825","o-1823","o-1820","o-1818","o-1819","o-1822","o-1824"]}
v1.vs.vc.8.r0000000016:o-1822|{"body":"#{\"adminFacet\":\"$0.Alleged: adminFacet\",\"creatorFacet\":\"$1.Alleged: Auctioneer creatorFacet\",\"governor\":\"$2.Alleged: InstanceHandle\",\"governorAdminFacet\":\"$3.Alleged: adminFacet\",\"governorCreatorFacet\":\"$4.Alleged: ContractGovernorKit creator\",\"instance\":\"$5.Alleged: InstanceHandle\",\"label\":\"auctioneer\",\"publicFacet\":\"$6.Alleged: publicFacet\"}","slots":["o-1825","o-1823","o-1820","o-1818","o-1819","o-1822","o-1824"]}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value LIKE '%o-1822%';
v1.c.ko2441879|R o-1822
v1.vs.vc.5.sauctioneerKit|{"body":"#{\"adminFacet\":\"$0.Alleged: adminFacet\",\"creatorFacet\":\"$1.Alleged: Auctioneer creatorFacet\",\"governor\":\"$2.Alleged: InstanceHandle\",\"governorAdminFacet\":\"$3.Alleged: adminFacet\",\"governorCreatorFacet\":\"$4.Alleged: ContractGovernorKit creator\",\"instance\":\"$5.Alleged: InstanceHandle\",\"label\":\"auctioneer\",\"publicFacet\":\"$6.Alleged: publicFacet\"}","slots":["o-1825","o-1823","o-1820","o-1818","o-1819","o-1822","o-1824"]}
v1.vs.vc.8.r0000000016:o-1822|{"body":"#{\"adminFacet\":\"$0.Alleged: adminFacet\",\"creatorFacet\":\"$1.Alleged: Auctioneer creatorFacet\",\"governor\":\"$2.Alleged: InstanceHandle\",\"governorAdminFacet\":\"$3.Alleged: adminFacet\",\"governorCreatorFacet\":\"$4.Alleged: ContractGovernorKit creator\",\"instance\":\"$5.Alleged: InstanceHandle\",\"label\":\"auctioneer\",\"publicFacet\":\"$6.Alleged: publicFacet\"}","slots":["o-1825","o-1823","o-1820","o-1818","o-1819","o-1822","o-1824"]}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1' AND key < 'v:' AND substr(key, -length('.c.ko2441879')) = '.c.ko2441879' AND key LIKE 'v%.c.ko2441879';
v1.c.ko2441879|R o-1822
v37.c.ko2441879|R o-55771
v6.c.ko2441879|R o-192
v663.c.ko2441879|R o-92
v664.c.ko2441879|R o-71
v665.c.ko2441879|R o-54
v7.c.ko2441879|R o-3650
v9.c.ko2441879|R o+d29/648

sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v37.options';
zcf-b1-94e20-walletFactory

sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v6.options';
agoricNames

sqlite> SELECT json_extract(value, '$.name') AS name FROM kvStore WHERE key='v7.options';
board

sqlite> SELECT * FROM kvStore WHERE key >= 'v7.' AND key < 'v7/' AND value LIKE '%o-3650%';
v7.c.ko2441879|R o-3650
v7.vs.vc.5.sboard0501277|{"body":"#\"$0.Alleged: InstanceHandle\"","slots":["o-3650"]}
v7.vs.vom.o+d11/5|{"valueDurability":{"body":"#\"mandatory\"","slots":[]},"publishCount":{"body":"#\"+74\"","slots":[]},"status":{"body":"#\"live\"","slots":[]},"hasValue":{"body":"#true","slots":[]},"value":{"body":"#[[\"ATOM-USD price feed\",\"$0.Alleged: InstanceHandle\"],[\"VaultFactory\",\"$1.Alleged: InstanceHandle\"],[\"VaultFactoryGovernor\",\"$2.Alleged: InstanceHandle\"],[\"auctioneer\",\"$3.Alleged: InstanceHandle\"],[\"econCommitteeCharter\",\"$4.Alleged: InstanceHandle\"],[\"economicCommittee\",\"$5.Alleged: InstanceHandle\"],[\"feeDistributor\",\"$6.Alleged: InstanceHandle\"],[\"game1\",\"$7.Alleged: InstanceHandle\"],[\"kread\",\"$8.Alleged: InstanceHandle\"],[\"kreadCommittee\",\"$9.Alleged: InstanceHandle\"],[\"kreadCommitteeCharter\",\"$10.Alleged: InstanceHandle\"],[\"orca\",\"$11.Alleged: InstanceHandle\"],[\"provisionPool\",\"$12.Alleged: InstanceHandle\"],[\"psm-IST-DAI_axl\",\"$13.Alleged: InstanceHandle\"],[\"psm-IST-DAI_grv\",\"$14.Alleged: InstanceHandle\"],[\"psm-IST-ToyUSD\",\"$15.Alleged: InstanceHandle\"],[\"psm-IST-USDC_axl\",\"$16.Alleged: InstanceHandle\"],[\"psm-IST-USDT\",\"$17.Alleged: InstanceHandle\"],[\"reserve\",\"$18.Alleged: InstanceHandle\"],[\"reserveGovernor\",\"$19.Alleged: InstanceHandle\"],[\"scaledPriceAuthority-ATOM\",\"$20.Alleged: InstanceHandle\"],[\"scaledPriceAuthority-stATOM\",\"$21.Alleged: InstanceHandle\"],[\"scaledPriceAuthority-stOSMO\",\"$22.Alleged: InstanceHandle\"],[\"scaledPriceAuthority-stOSMO2\",\"$23.Alleged: InstanceHandle\"],[\"scaledPriceAuthority-stkATOM\",\"$24.Alleged: InstanceHandle\"],[\"stATOM-USD price feed\",\"$25.Alleged: InstanceHandle\"],[\"stOSMO-USD price feed\",\"$26.Alleged: InstanceHandle\"],[\"stOSMO2-USD price feed\",\"$27.Alleged: InstanceHandle\"],[\"stTIA-USD price feed\",\"$28.Alleged: InstanceHandle\"],[\"stkATOM-USD price feed\",\"$29.Alleged: InstanceHandle\"],[\"walletFactory\",\"$30.Alleged: InstanceHandle\"]]","slots":["o-3587","o-111","o-122","o-3650","o-3646","o-3622","o-113","o-171","o-198","o-196","o-193","o-3054","o-115","o-119","o-120","o-121","o-118","o-515","o-109","o-116","o-3608","o-3609","o-3610","o-3611","o-3612","o-3591","o-3595","o-716","o-3600","o-3604","o-117"]}}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value GLOB '*"o+??/8"*';
v1.vs.vc.1.sGovernedContractKits|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/8"]}
v1.vs.vc.5.sgovernedContractKits|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/8"]}

sqlite> SELECT * FROM kvStore WHERE key >= 'v1.' AND key < 'v1/' AND value GLOB '*"o+??/5"*';
v1.vs.vc.1.sBootstrap Powers|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/5"]}
v1.vs.vc.5.spowerStore|{"body":"#\"$0.Alleged: mapStore\"","slots":["o+d6/5"]}

toliaqat added the enhancement New feature or request label Dec 17, 2024

gibson042 mentioned this issue Jan 7, 2025

feat(cosmic-swingset): Introduce inquisitor.mjs #10807

Open

siarhei-agoric added a commit that referenced this issue Jan 8, 2025

feat(cosmic-swingset): introduce GovernorExecutor #10725

5c0ac62

refs: #10725 Initial set of changes to get auctioneer governors terminated.

mhofman mentioned this issue Jan 14, 2025

E_NOT_ENOUGH_MEMORY on non-critical vat causes network halt #10841

Open

siarhei-agoric added a commit that referenced this issue Jan 17, 2025

feat(cosmic-swingset): introduce GovernorExecutor #10725

2618ef4

refs: #10725 Initial set of changes to get auctioneer governors terminated.

siarhei-agoric added a commit that referenced this issue Jan 17, 2025

add terminate-governed.test.ts test

1f83e7c

refs #10725

siarhei-agoric mentioned this issue Jan 17, 2025

Unbork Emerynet: Cleanup Auctioneers #10725 #10861

Open

siarhei-agoric added a commit that referenced this issue Jan 22, 2025

cleanup: remove unneeded files and intermediate tests

527ed5d

refs #10725

siarhei-agoric added a commit that referenced this issue Jan 22, 2025

feat(cosmos/app): add terminateAuctioneersCoreProposal to upgrade18Ha…

ad07e7b

…ndler refs: #10725

siarhei-agoric added a commit that referenced this issue Jan 22, 2025

chore: lint-fix

09d8071

refs: #10725

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unbork Emerynet: Cleanup Auctioneers, PriceAuthorities, and QuoteNotifiers #10725

Unbork Emerynet: Cleanup Auctioneers, PriceAuthorities, and QuoteNotifiers #10725

toliaqat commented Dec 17, 2024

warner commented Dec 26, 2024

Chris-Hibbert commented Jan 2, 2025

siarhei-agoric commented Jan 3, 2025 •

edited

Loading

mhofman commented Jan 14, 2025 •

edited

Loading

siarhei-agoric commented Jan 23, 2025

Unbork Emerynet: Cleanup Auctioneers, PriceAuthorities, and QuoteNotifiers #10725

Unbork Emerynet: Cleanup Auctioneers, PriceAuthorities, and QuoteNotifiers #10725

Comments

toliaqat commented Dec 17, 2024

What is the Problem Being Solved?

Description of the Design

(Optional) Halt Auctioneer Timers (for Gen4/Gen5)

Terminate Old Auctioneer Vats

Submit Prices for All Denoms

Create a New, Clean Auctioneer

Monitor and Verify Stability

Security Considerations

Scaling Considerations

Test Plan

Upgrade Considerations

Sub-Tasks:

warner commented Dec 26, 2024

Chris-Hibbert commented Jan 2, 2025

siarhei-agoric commented Jan 3, 2025 • edited Loading

mhofman commented Jan 14, 2025 • edited Loading

siarhei-agoric commented Jan 23, 2025

siarhei-agoric commented Jan 3, 2025 •

edited

Loading

mhofman commented Jan 14, 2025 •

edited

Loading