Merge branch 'gav-fix-telemetry' into v0.9

Cargo.toml

@@ -4,7 +4,7 @@ path = "node/src/main.rs"
 
 [package]
 name = "substrate"
-version = "0.9.1"
+version = "0.9.2"
 authors = ["Parity Technologies <[email protected]>"]
 build = "build.rs"
 
@@ -57,6 +57,7 @@ members = [
 	"core/sr-primitives",
 	"srml/session",
 	"srml/staking",
+	"srml/sudo",
 	"srml/system",
 	"srml/timestamp",
 	"srml/treasury",

README.adoc

@@ -48,38 +48,35 @@ Other examples include the parachain-heads extrinsic in Polkadot and the "note-m
 
 === Runtime and API
 
-Substrate chains all have a runtime. The runtime is a WebAssembly "blob" that includes a number of entry-points. Some entry-points are required as part of the underlying Substrate specification. Others are merely convention and required for the default implementation of the Substrate client to be able to author blocks. In short these two sets are:
+Substrate chains all have a runtime. The runtime is a WebAssembly "blob" that includes a number of entry-points. Some entry-points are required as part of the underlying Substrate specification. Others are merely convention and required for the default implementation of the Substrate client to be able to author blocks. 
 
-The runtime is API entry points are expected to be in the runtime's `api` module. There is a specific ABI based upon the Substrate Simple Codec (`codec`), which is used to encode and decode the arguments for these functions and specify where and how they should be passed. A special macro is provided called `impl_stubs`, which prepares all functionality for marshalling arguments and basically just allows you to write the functions as you would normally (except for the fact that there must be example one argument - tuples are allowed to workaround).
+If you want to develop a chain with Substrate, you will need to implement the `Core` trait. This `Core` trait generates an API with the minimum necessary functionality to interact with your runtime. A special macro is provided called `impl_runtime_apis!` that help you implement runtime API traits. All runtime API trait implementations need to be done in one call of the `impl_runtime_apis!` macro. All parameters and return values need to implement https://crates.io/crates/parity-codec[`parity-codec`] to be encodable and decodable.
 
-Here's the Polkadot API implementation as of PoC-2:
+Here's a snippet of the Polkadot API implementation as of PoC-3:
 
 ```rust
-pub mod api {
-	impl_stubs!(
-
-		// Standard: Required.
-		version => |()| super::Version::version(),
-		authorities => |()| super::Consensus::authorities(),
-		execute_block => |block| super::Executive::execute_block(block),
-
-		// Conventional: Needed for Substrate client's reference block authoring logic to work.
-		initialise_block => |header| super::Executive::initialise_block(&header),
-		apply_extrinsic => |extrinsic| super::Executive::apply_extrinsic(extrinsic),
-		finalise_block => |()| super::Executive::finalise_block(),
-		inherent_extrinsics => |inherent| super::inherent_extrinsics(inherent),
-
-		// Non-standard (Polkadot-specific). This just exposes some stuff that Polkadot client
-		// finds useful.
-		validator_count => |()| super::Session::validator_count(),
-		validators => |()| super::Session::validators()
-	);
+impl_runtime_apis! {
+	impl client_api::Core<Block> for Runtime {
+		fn version() -> RuntimeVersion {
+			VERSION
+		}
+
+		fn authorities() -> Vec<SessionKey> {
+			Consensus::authorities()
+		}
+
+		fn execute_block(block: Block) {
+			Executive::execute_block(block)
+		}
+
+		fn initialise_block(header: <Block as BlockT>::Header) {
+			Executive::initialise_block(&header)
+		}
+	}
+	// ---snip---
 }
 ```
 
-As you can see, at the minimum there are only three API calls to implement. If you want to reuse as much of Substrate's reference block authoring client code, then you'll want to provide the next four entrypoints (though three of them you probably already implemented as part of `execute_block`).
-
-Of the first three, there is `execute_block`, which contains the actions to be taken to execute a block and pretty much defines the blockchain. Then there is `authorities` which tells the AfG consensus algorithm sitting in the Substrate client who the given authorities (known as "validators" in some contexts) are that can finalise the next block. Finally, there is `version`, which is a fairly sophisticated version identifier. This includes a key distinction between *specification version* and *authoring version*, with the former essentially versioning the logic of `execute_block` and the latter versioning only the logic of `inherent_extrinsics` and core aspects of extrinsic validity.
 
 === Inherent Extrinsics
 
@@ -175,7 +172,7 @@ You can distribute `mychain.json` so that everyone can synchronise and (dependin
 
 === Hacking on Substrate
 
-If you'd actually like hack on Substrate, you can just grab the source code and
+If you'd actually like to hack on Substrate, you can just grab the source code and
 build it. Ensure you have Rust and the support software installed:
 
 [source, shell]
@@ -223,7 +220,6 @@ You can start a development chain with:
 [source, shell]
 cargo run -- --dev
 
-
 Detailed logs may be shown by running the node with the following environment variables set: `RUST_LOG=debug RUST_BACKTRACE=1 cargo run -- --dev`.
 
 If you want to see the multi-node consensus algorithm in action locally, then you can create a local testnet with two validator nodes for Alice and Bob, who are the initial authorities of the genesis chain specification that have been endowed with a testnet DOTs. We'll give each node a name and expose them so they are listed on [Telemetry](https://telemetry.polkadot.io/#/Local%20Testnet). You'll need two terminals windows open.
@@ -255,6 +251,39 @@ cargo run -- \
 
 Additional Substate CLI usage options are available and may be shown by running `cargo run -- --help`.
 
+=== Joining the Charred Cherry Testnet
+
+Charred Cherry is the new testnet for Substrate 1.0 beta. Please note that 1.0 beta is not compatible with the BBQ-Birch testnet. Ensure you have the dependencies listed above before compiling.
+
+[source, shell]
+----
+git clone https://github.com/paritytech/substrate.git
+cd substrate
+----
+
+You can run the tests if you like:
+
+[source, shell]
+cargo test --all
+
+Start your node:
+
+[source, shell]
+cargo run --release
+
+To see a list of command line options, enter:
+
+[source, shell]
+cargo run --release -- --help
+
+For example, you can choose a custom node name:
+
+[source, shell]
+cargo run --release -- --name my_custom_name
+
+If you are successful, you will see your node syncing at https://telemetry.polkadot.io/#/Charred%20Cherry 
+
+
 == Documentation
 
 === Viewing documentation for Substrate packages

core/basic-authorship/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "substrate-basic-authorship"
+version = "0.1.0"
+authors = ["Parity Technologies <[email protected]>"]
+
+[dependencies]
+log = "0.4"
+parity-codec = "2.1"
+sr-primitives = { path = "../../core/sr-primitives" }
+substrate-client = { path = "../../core/client" }
+substrate-consensus-aura-primitives = { path = "../../core/consensus/aura/primitives" }
+substrate-consensus-common = { path = "../../core/consensus/common" }
+substrate-primitives = { path = "../../core/primitives" }
+substrate-transaction-pool = { path = "../../core/transaction-pool" }

core/service/src/consensus.rs → core/basic-authorship/src/basic_authorship.rs

@@ -26,8 +26,8 @@ use client::{self, error, Client as SubstrateClient, CallExecutor};
 use client::{block_builder::api::BlockBuilder as BlockBuilderApi, runtime_api::Core};
 use codec::{Decode, Encode};
 use consensus_common::{self, evaluation};
-use primitives::{H256, AuthorityId, ed25519, Blake2Hasher};
-use runtime_primitives::traits::{Block as BlockT, Hash as HashT, Header as HeaderT, ProvideRuntimeApi};
+use primitives::{H256, Blake2Hasher};
+use runtime_primitives::traits::{Block as BlockT, Hash as HashT, Header as HeaderT, ProvideRuntimeApi, AuthorityIdFor};
 use runtime_primitives::generic::BlockId;
 use runtime_primitives::BasicInherentData;
 use transaction_pool::txpool::{self, Pool as TransactionPool};
@@ -125,8 +125,7 @@ impl<C, A, ConsensusData> consensus_common::Environment<<C as AuthoringApi>::Blo
 	fn init(
 		&self,
 		parent_header: &<<C as AuthoringApi>::Block as BlockT>::Header,
-		_: &[AuthorityId],
-		_: Arc<ed25519::Pair>,
+		_: &[AuthorityIdFor<<C as AuthoringApi>::Block>],
 	) -> Result<Self::Proposer, error::Error> {
 		let parent_hash = parent_header.hash();
 

core/basic-authorship/src/lib.rs

@@ -0,0 +1,34 @@
+// Copyright 2017-2018 Parity Technologies (UK) Ltd.
+// This file is part of Substrate.
+
+// Substrate is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Substrate is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Substrate.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Basic implementation of block-authoring logic.
+
+#![warn(unused_extern_crates)]
+
+extern crate substrate_consensus_aura_primitives as aura_primitives;
+extern crate substrate_primitives as primitives;
+extern crate sr_primitives as runtime_primitives;
+extern crate substrate_consensus_common as consensus_common;
+extern crate substrate_client as client;
+extern crate parity_codec as codec;
+extern crate substrate_transaction_pool as transaction_pool;
+
+#[macro_use]
+extern crate log;
+
+mod basic_authorship;
+
+pub use basic_authorship::{ProposerFactory, BlockBuilder, AuthoringApi, Proposer};

core/cli/src/informant.rs

@@ -24,7 +24,9 @@ use tokio::runtime::TaskExecutor;
 use tokio::timer::Interval;
 use sysinfo::{get_current_pid, ProcessExt, System, SystemExt};
 use network::{SyncState, SyncProvider};
-use client::BlockchainEvents;
+use client::{backend::Backend, BlockchainEvents};
+
+use runtime_primitives::generic::BlockId;
 use runtime_primitives::traits::{Header, As};
 
 const TIMER_INTERVAL_MS: u64 = 5000;
@@ -92,7 +94,36 @@ pub fn start<C>(service: &Service<C>, exit: ::exit_future::Exit, handle: TaskExe
 	});
 
 	let client = service.client();
-	let display_block_import = client.import_notification_stream().for_each(|n| {
+	let mut last = match client.info() {
+		Ok(info) => Some((info.chain.best_number, info.chain.best_hash)),
+		Err(e) => { warn!("Error getting best block information: {:?}", e); None }
+	};
+
+	let display_block_import = client.import_notification_stream().for_each(move |n| {
+		// detect and log reorganizations.
+		if let Some((ref last_num, ref last_hash)) = last {
+			if n.header.parent_hash() != last_hash {
+				let tree_route = ::client::blockchain::tree_route(
+					client.backend().blockchain(),
+					BlockId::Hash(last_hash.clone()),
+					BlockId::Hash(n.hash),
+				);
+
+				match tree_route {
+					Ok(ref t) if !t.retracted().is_empty() => info!(
+						"Reorg from #{},{} to #{},{}, common ancestor #{},{}",
+						last_num, last_hash,
+						n.header.number(), n.hash,
+						t.common_block().number, t.common_block().hash,
+					),
+					Ok(_) => {},
+					Err(e) => warn!("Error computing tree route: {}", e),
+				}
+			}
+		}
+
+		last = Some((n.header.number().clone(), n.hash.clone()));
+
 		info!(target: "substrate", "Imported #{} ({})", n.header.number(), n.hash);
 		Ok(())
 	});

core/cli/src/lib.rs

@@ -422,7 +422,7 @@ where
 	} else if let Some(sub_matches) = matches.subcommand_matches("revert") {
 		revert_chain::<F>(
 			get_db_path_for_subcommand(matches, sub_matches)?,
-			matches,
+			sub_matches,
 			spec
 		)?;
 		return Ok(Action::ExecutedInternally);

core/cli/src/params.rs

@@ -91,21 +91,17 @@ pub struct CoreParams {
 
 	/// Specify the pruning mode, a number of blocks to keep or 'archive'. Default is 256.
 	#[structopt(long = "pruning", value_name = "PRUNING_MODE")]
-	pruning: Option<u32>,
+	pruning: Option<String>,
 
 	/// The human-readable name for this node, as reported to the telemetry server, if enabled
 	#[structopt(long = "name", value_name = "NAME")]
 	name: Option<String>,
 
-	/// Should connect to the Substrate telemetry server (telemetry is off by default on local chains)
-	#[structopt(short = "t", long = "telemetry")]
-	telemetry: bool,
-
 	/// Should not connect to the Substrate telemetry server (telemetry is on by default on global chains)
 	#[structopt(long = "no-telemetry")]
 	no_telemetry: bool,
 
-	/// The URL of the telemetry server. Implies --telemetry
+	/// The URL of the telemetry server to connect to
 	#[structopt(long = "telemetry-url", value_name = "TELEMETRY_URL")]
 	telemetry_url: Option<String>,
 

core/client/Cargo.toml

@@ -7,7 +7,7 @@ authors = ["Parity Technologies <[email protected]>"]
 error-chain = { version = "0.12", optional = true }
 fnv = { version = "1.0", optional = true }
 log = { version = "0.4", optional = true }
-parking_lot = { version = "0.4", optional = true }
+parking_lot = { version = "0.7.1", optional = true }
 hex-literal = { version = "0.1", optional = true }
 futures = { version = "0.1.17", optional = true }
 slog = { version = "^2", optional = true }

core/client/db/Cargo.toml

@@ -4,10 +4,11 @@ version = "0.1.0"
 authors = ["Parity Technologies <[email protected]>"]
 
 [dependencies]
-parking_lot = "0.4"
+parking_lot = "0.7.1"
 log = "0.4"
 kvdb = { git = "https://github.com/paritytech/parity-common", rev="616b40150ded71f57f650067fcbc5c99d7c343e6" }
 kvdb-rocksdb = { git = "https://github.com/paritytech/parity-common", rev="616b40150ded71f57f650067fcbc5c99d7c343e6" }
+lru-cache = "0.1"
 hash-db = { git = "https://github.com/paritytech/trie" }
 substrate-primitives = { path = "../../primitives" }
 sr-primitives = { path = "../../sr-primitives" }

core/client/db/src/cache/mod.rs

@@ -24,9 +24,8 @@ use kvdb::{KeyValueDB, DBTransaction};
 use client::blockchain::Cache as BlockchainCache;
 use client::error::Result as ClientResult;
 use codec::{Encode, Decode};
-use primitives::AuthorityId;
 use runtime_primitives::generic::BlockId;
-use runtime_primitives::traits::{Block as BlockT, Header as HeaderT, NumberFor, As};
+use runtime_primitives::traits::{Block as BlockT, Header as HeaderT, NumberFor, As, AuthorityIdFor};
 use utils::{self, COLUMN_META};
 
 use self::list_cache::ListCache;
@@ -65,7 +64,7 @@ impl<T> CacheItemT for T where T: Clone + Decode + Encode + PartialEq {}
 
 /// Database-backed blockchain data cache.
 pub struct DbCache<Block: BlockT> {
-	authorities_at: ListCache<Block, Vec<AuthorityId>, self::list_storage::DbStorage>,
+	authorities_at: ListCache<Block, Vec<AuthorityIdFor<Block>>, self::list_storage::DbStorage>,
 }
 
 impl<Block: BlockT> DbCache<Block> {
@@ -112,14 +111,14 @@ impl<Block: BlockT> DbCache<Block> {
 
 /// Cache operations that are to be committed after database transaction is committed.
 pub struct DbCacheTransactionOps<Block: BlockT> {
-	authorities_at_op: Option<self::list_cache::CommitOperation<Block, Vec<AuthorityId>>>,
+	authorities_at_op: Option<self::list_cache::CommitOperation<Block, Vec<AuthorityIdFor<Block>>>>,
 }
 
 /// Database-backed blockchain data cache transaction valid for single block import.
 pub struct DbCacheTransaction<'a, Block: BlockT> {
 	cache: &'a mut DbCache<Block>,
 	tx: &'a mut DBTransaction,
-	authorities_at_op: Option<self::list_cache::CommitOperation<Block, Vec<AuthorityId>>>,
+	authorities_at_op: Option<self::list_cache::CommitOperation<Block, Vec<AuthorityIdFor<Block>>>>,
 }
 
 impl<'a, Block: BlockT> DbCacheTransaction<'a, Block> {
@@ -135,7 +134,7 @@ impl<'a, Block: BlockT> DbCacheTransaction<'a, Block> {
 		mut self,
 		parent: ComplexBlockId<Block>,
 		block: ComplexBlockId<Block>,
-		authorities_at: Option<Vec<AuthorityId>>,
+		authorities_at: Option<Vec<AuthorityIdFor<Block>>>,
 		is_final: bool,
 	) -> ClientResult<Self> {
 		assert!(self.authorities_at_op.is_none());
@@ -179,7 +178,7 @@ impl<'a, Block: BlockT> DbCacheTransaction<'a, Block> {
 pub struct DbCacheSync<Block: BlockT>(pub RwLock<DbCache<Block>>);
 
 impl<Block: BlockT> BlockchainCache<Block> for DbCacheSync<Block> {
-	fn authorities_at(&self, at: BlockId<Block>) -> Option<Vec<AuthorityId>> {
+	fn authorities_at(&self, at: BlockId<Block>) -> Option<Vec<AuthorityIdFor<Block>>> {
 		let cache = self.0.read();
 		let storage = cache.authorities_at.storage();
 		let db = storage.db();