Skip to content

Commit 432e856

Browse files
lexnvjsdw
andauthored
Metadata V15: Generate Runtime APIs (paritytech#918)
* Update frame-metadata to v15.1.0 Signed-off-by: Alexandru Vasile <[email protected]> * Enable V15 unstable metadata in frame-metadata Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Move validation hashing to dedicated file Signed-off-by: Alexandru Vasile <[email protected]> * Use sp-metadata-ir from substrate to work with metadata Signed-off-by: Alexandru Vasile <[email protected]> * Revert using sp-metadata-ir in favor of conversion to v15 Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Convert v14 to v15 Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Use v15 for validation Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Use v15 for codegen Signed-off-by: Alexandru Vasile <[email protected]> * metadata/bench: Use v15 Signed-off-by: Alexandru Vasile <[email protected]> * Adjust to v15 metadata Signed-off-by: Alexandru Vasile <[email protected]> * Adjust testing Signed-off-by: Alexandru Vasile <[email protected]> * Improve documentation Signed-off-by: Alexandru Vasile <[email protected]> * force CI Signed-off-by: Alexandru Vasile <[email protected]> * rpc: Fetch metadata at version Signed-off-by: Alexandru Vasile <[email protected]> * artifacts: Update polkadot.scale from commit 6dc9e84dde2 Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Fetch V15 using the new API Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Add runtime API interface Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Hash runtime API metadata for validation Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Extract runtime API metadata wrapper from subxt::Metadata Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Adjust hashing cache to reflect root+item keys Signed-off-by: Alexandru Vasile <[email protected]> * rpc: Add raw state_call API method Signed-off-by: Alexandru Vasile <[email protected]> * runtime_api: Add payload with static and dynamic variants Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Allow payloads to call into the runtime Signed-off-by: Alexandru Vasile <[email protected]> * examples: Add example to make a runtime API call both static and dynamic Signed-off-by: Alexandru Vasile <[email protected]> * Update polkadot.rs Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Simplify client fetching Signed-off-by: Alexandru Vasile <[email protected]> * Address feedback and fallback to old API if needed Signed-off-by: Alexandru Vasile <[email protected]> * runtime_api: Make mutability conditional on input params Signed-off-by: Alexandru Vasile <[email protected]> * Regenerate polkadot.rs Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Retain only pallets without runtime API info Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Retry via `Metadata_metadata` without conversion Signed-off-by: Alexandru Vasile <[email protected]> * payload: Remove `Decode` and change validation fn Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Retain runtime API types Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Runtime APIs documentation based on flag Signed-off-by: Alexandru Vasile <[email protected]> * Update examples/examples/custom_metadata_url.rs Co-authored-by: James Wilson <[email protected]> * Update artifacts from polkadot-a6cfdb16e9 Signed-off-by: Alexandru Vasile <[email protected]> * Update polkadot.rs with polkadot-a6cfdb16e9 Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Generate input structures for runtime API Signed-off-by: Alexandru Vasile <[email protected]> * runtime_api: Remove the static paylaod and use single impl Signed-off-by: Alexandru Vasile <[email protected]> * examples: Fetch account nonce Signed-off-by: Alexandru Vasile <[email protected]> * testing: Adjust build script to fetch latest metadata Signed-off-by: Alexandru Vasile <[email protected]> * testing: Check account nonce from runtime API Signed-off-by: Alexandru Vasile <[email protected]> * Update cargo.lock Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Fix doc generation for runtime types Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Rename `inputs` runtime calls module to `types` Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Generate Calls structs inside the types module Signed-off-by: Alexandru Vasile <[email protected]> * testing: Check Alice account nonce before submitting the tx Signed-off-by: Alexandru Vasile <[email protected]> * cli: Add metadata version option flag supporting v14 and unstable Signed-off-by: Alexandru Vasile <[email protected]> * cli: Specify version to fetch Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Fallback to fetching latest stable metadata Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Add unstable-metadata feature to fetch the latest Signed-off-by: Alexandru Vasile <[email protected]> * RuntimeVersion with Latest and Version(u32) Signed-off-by: Alexandru Vasile <[email protected]> * Update polkadot.rs Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Adjust fetch_metadata to inspect version list Signed-off-by: Alexandru Vasile <[email protected]> * testing: Adjust metadata to metadata_legacy Signed-off-by: Alexandru Vasile <[email protected]> * events: Adjust docs to use metadata_legacy Signed-off-by: Alexandru Vasile <[email protected]> * have a pass over fetch_metadata * cargo fmt * Option<String> when fetch metadata via latest API * clippy * fmt * cli: Use the MetadataVersion from codegen Signed-off-by: Alexandru Vasile <[email protected]> * cli: Specify latest as default for MetadataVersion Signed-off-by: Alexandru Vasile <[email protected]> * cli: Remove version from metadata and use the one from file_or_url Signed-off-by: Alexandru Vasile <[email protected]> * Fix clippy Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Decode metadata independently for different RPC calls Signed-off-by: Alexandru Vasile <[email protected]> --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: James Wilson <[email protected]>
1 parent f4eb80e commit 432e856

33 files changed

+21694
-16125
lines changed

Cargo.lock

+2
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

artifacts/polkadot_metadata.scale

46.4 KB
Binary file not shown.

cli/src/commands/compatibility.rs

+31-8
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ use frame_metadata::{
1111
use jsonrpsee::client_transport::ws::Uri;
1212
use serde::{Deserialize, Serialize};
1313
use std::collections::HashMap;
14+
use subxt_codegen::utils::MetadataVersion;
1415
use subxt_metadata::{get_metadata_hash, get_pallet_hash, metadata_v14_to_latest};
1516

1617
/// Verify metadata compatibility between substrate nodes.
@@ -25,16 +26,35 @@ pub struct Opts {
2526
/// The validation will omit the full metadata check and focus instead on the pallet.
2627
#[clap(long, value_parser)]
2728
pallet: Option<String>,
29+
/// Specify the metadata version.
30+
///
31+
/// - unstable:
32+
///
33+
/// Use the latest unstable metadata of the node.
34+
///
35+
/// - number
36+
///
37+
/// Use this specific metadata version.
38+
///
39+
/// Defaults to latest.
40+
#[clap(long = "version", default_value = "latest")]
41+
version: MetadataVersion,
2842
}
2943

3044
pub async fn run(opts: Opts) -> color_eyre::Result<()> {
3145
match opts.pallet {
32-
Some(pallet) => handle_pallet_metadata(opts.nodes.as_slice(), pallet.as_str()).await,
33-
None => handle_full_metadata(opts.nodes.as_slice()).await,
46+
Some(pallet) => {
47+
handle_pallet_metadata(opts.nodes.as_slice(), pallet.as_str(), opts.version).await
48+
}
49+
None => handle_full_metadata(opts.nodes.as_slice(), opts.version).await,
3450
}
3551
}
3652

37-
async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result<()> {
53+
async fn handle_pallet_metadata(
54+
nodes: &[Uri],
55+
name: &str,
56+
version: MetadataVersion,
57+
) -> color_eyre::Result<()> {
3858
#[derive(Serialize, Deserialize, Default)]
3959
#[serde(rename_all = "camelCase")]
4060
struct CompatibilityPallet {
@@ -44,7 +64,7 @@ async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result
4464

4565
let mut compatibility: CompatibilityPallet = Default::default();
4666
for node in nodes.iter() {
47-
let metadata = fetch_runtime_metadata(node).await?;
67+
let metadata = fetch_runtime_metadata(node, version).await?;
4868

4969
match metadata.pallets.iter().find(|pallet| pallet.name == name) {
5070
Some(pallet_metadata) => {
@@ -73,10 +93,10 @@ async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result
7393
Ok(())
7494
}
7595

76-
async fn handle_full_metadata(nodes: &[Uri]) -> color_eyre::Result<()> {
96+
async fn handle_full_metadata(nodes: &[Uri], version: MetadataVersion) -> color_eyre::Result<()> {
7797
let mut compatibility_map: HashMap<String, Vec<String>> = HashMap::new();
7898
for node in nodes.iter() {
79-
let metadata = fetch_runtime_metadata(node).await?;
99+
let metadata = fetch_runtime_metadata(node, version).await?;
80100
let hash = get_metadata_hash(&metadata);
81101
let hex_hash = hex::encode(hash);
82102
println!("Node {node:?} has metadata hash {hex_hash:?}",);
@@ -96,8 +116,11 @@ async fn handle_full_metadata(nodes: &[Uri]) -> color_eyre::Result<()> {
96116
Ok(())
97117
}
98118

99-
async fn fetch_runtime_metadata(url: &Uri) -> color_eyre::Result<RuntimeMetadataV15> {
100-
let bytes = subxt_codegen::utils::fetch_metadata_bytes(url).await?;
119+
async fn fetch_runtime_metadata(
120+
url: &Uri,
121+
version: MetadataVersion,
122+
) -> color_eyre::Result<RuntimeMetadataV15> {
123+
let bytes = subxt_codegen::utils::fetch_metadata_bytes(url, version).await?;
101124

102125
let metadata = <RuntimeMetadataPrefixed as Decode>::decode(&mut &bytes[..])?;
103126
if metadata.0 != META_RESERVED {

cli/src/utils.rs

+35-7
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
use clap::Args;
66
use color_eyre::eyre;
77
use std::{fs, io::Read, path::PathBuf};
8-
use subxt_codegen::utils::Uri;
8+
use subxt_codegen::utils::{MetadataVersion, Uri};
99

1010
/// The source of the metadata.
1111
#[derive(Debug, Args)]
@@ -16,29 +16,57 @@ pub struct FileOrUrl {
1616
/// The path to the encoded metadata file.
1717
#[clap(long, value_parser)]
1818
file: Option<PathBuf>,
19+
/// Specify the metadata version.
20+
///
21+
/// - unstable:
22+
///
23+
/// Use the latest unstable metadata of the node.
24+
///
25+
/// - number
26+
///
27+
/// Use this specific metadata version.
28+
///
29+
/// Defaults to 14.
30+
#[clap(long)]
31+
version: Option<MetadataVersion>,
1932
}
2033

2134
impl FileOrUrl {
2235
/// Fetch the metadata bytes.
2336
pub async fn fetch(&self) -> color_eyre::Result<Vec<u8>> {
24-
match (&self.file, &self.url) {
37+
match (&self.file, &self.url, self.version) {
2538
// Can't provide both --file and --url
26-
(Some(_), Some(_)) => {
39+
(Some(_), Some(_), _) => {
2740
eyre::bail!("specify one of `--url` or `--file` but not both")
2841
}
2942
// Load from --file path
30-
(Some(path), None) => {
43+
(Some(path), None, None) => {
3144
let mut file = fs::File::open(path)?;
3245
let mut bytes = Vec::new();
3346
file.read_to_end(&mut bytes)?;
3447
Ok(bytes)
3548
}
49+
// Cannot load the metadata from the file and specify a version to fetch.
50+
(Some(_), None, Some(_)) => {
51+
// Note: we could provide the ability to convert between metadata versions
52+
// but that would be involved because we'd need to convert
53+
// from each metadata to the latest one and from the
54+
// latest one to each metadata version. For now, disable the conversion.
55+
eyre::bail!("`--file` is incompatible with `--version`")
56+
}
3657
// Fetch from --url
37-
(None, Some(uri)) => Ok(subxt_codegen::utils::fetch_metadata_bytes(uri).await?),
58+
(None, Some(uri), version) => Ok(subxt_codegen::utils::fetch_metadata_bytes(
59+
uri,
60+
version.unwrap_or_default(),
61+
)
62+
.await?),
3863
// Default if neither is provided; fetch from local url
39-
(None, None) => {
64+
(None, None, version) => {
4065
let uri = Uri::from_static("http://localhost:9933");
41-
Ok(subxt_codegen::utils::fetch_metadata_bytes(&uri).await?)
66+
Ok(
67+
subxt_codegen::utils::fetch_metadata_bytes(&uri, version.unwrap_or_default())
68+
.await?,
69+
)
4270
}
4371
}
4472
}

codegen/src/api/calls.rs

+7-3
Original file line numberDiff line numberDiff line change
@@ -90,11 +90,11 @@ pub fn generate_calls(
9090
pub fn #fn_name(
9191
&self,
9292
#( #call_fn_args, )*
93-
) -> #crate_path::tx::Payload<#struct_name> {
93+
) -> #crate_path::tx::Payload<types::#struct_name> {
9494
#crate_path::tx::Payload::new_static(
9595
#pallet_name,
9696
#call_name,
97-
#struct_name { #( #call_args, )* },
97+
types::#struct_name { #( #call_args, )* },
9898
[#(#call_hash,)*]
9999
)
100100
}
@@ -120,7 +120,11 @@ pub fn generate_calls(
120120

121121
type DispatchError = #types_mod_ident::sp_runtime::DispatchError;
122122

123-
#( #call_structs )*
123+
pub mod types {
124+
use super::#types_mod_ident;
125+
126+
#( #call_structs )*
127+
}
124128

125129
pub struct TransactionApi;
126130

codegen/src/api/mod.rs

+21-2
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ mod calls;
88
mod constants;
99
mod errors;
1010
mod events;
11+
mod runtime_apis;
1112
mod storage;
1213

1314
use frame_metadata::v15::RuntimeMetadataV15;
@@ -18,7 +19,7 @@ use crate::error::CodegenError;
1819
use crate::{
1920
ir,
2021
types::{CompositeDef, CompositeDefFields, TypeGenerator, TypeSubstitutes},
21-
utils::{fetch_metadata_bytes_blocking, Uri},
22+
utils::{fetch_metadata_bytes_blocking, MetadataVersion, Uri},
2223
CratePath,
2324
};
2425
use codec::Decode;
@@ -95,7 +96,11 @@ pub fn generate_runtime_api_from_url(
9596
should_gen_docs: bool,
9697
runtime_types_only: bool,
9798
) -> Result<TokenStream2, CodegenError> {
98-
let bytes = fetch_metadata_bytes_blocking(url)?;
99+
// Fetch latest unstable version, if that fails fall back to the latest stable.
100+
let bytes = match fetch_metadata_bytes_blocking(url, MetadataVersion::Unstable) {
101+
Ok(bytes) => bytes,
102+
Err(_) => fetch_metadata_bytes_blocking(url, MetadataVersion::Latest)?,
103+
};
99104

100105
generate_runtime_api_from_bytes(
101106
item_mod,
@@ -434,6 +439,14 @@ impl RuntimeGenerator {
434439

435440
let rust_items = item_mod_ir.rust_items();
436441

442+
let apis_mod = runtime_apis::generate_runtime_apis(
443+
&self.metadata,
444+
&type_gen,
445+
types_mod_ident,
446+
&crate_path,
447+
should_gen_docs,
448+
)?;
449+
437450
Ok(quote! {
438451
#( #item_mod_attrs )*
439452
#[allow(dead_code, unused_imports, non_camel_case_types)]
@@ -487,6 +500,12 @@ impl RuntimeGenerator {
487500
TransactionApi
488501
}
489502

503+
pub fn apis() -> runtime_apis::RuntimeApi {
504+
runtime_apis::RuntimeApi
505+
}
506+
507+
#apis_mod
508+
490509
pub struct ConstantsApi;
491510
impl ConstantsApi {
492511
#(

0 commit comments

Comments
 (0)