Contribute to Open Source. Search issue labels to find the right project for you!

Style Guide

SCAII/SCAII

We should have a style guide for the whole project, in most case just delegating to other style guides.

I don’t know where we’d put this, probably not in this repository but maybe in the website+dev docs someday if we get that.

Rust:

  • The semi-official guide https://github.com/rust-lang-nursery/fmt-rfcs/blob/master/guide/guide.md
  • rustfmt with default settings exits without changing anything (these two guidelines end up being roughly equivalent in practice).
  • unsafe blocks are allowed, but must be audited before being pulled in.

Python:

  • Pylint exits cleanly (this is tentative until I work with pylint more, it’s overly restrictive so we’d probably end up messing with this until we have a “default” .pylintrc we’re happy with.
  • This also generally extends to following PEP8
  • Use autopep8 for autoformatting.
  • Python 3, don’t bother supporting 2
  • Use PEP484 annotations ??

Javascript:

??? I don’t know JS but I feel like we should have SOMETHING.

JSHint maybe? http://jshint.com/

Protobuf:

  • Fields may be declared required and optional/required distinctions are important.
  • Prefer ``` oneof foo { A a = 1; B b = 2; }

message A {} message B { required type field = 1; } ```

to

enum Foo {
    A = 0;
    B = 1;
}

required Foo foo = 1;

// only set if Foo is B
optional type field = 2;

General:

  • We need notes like “if your code is going to interact with Javascript like Viz, avoid integers over MAX_SAFE_INTEGER where possible” and such.
  • All functions and modules should be documented, except potentially test suites.
  • All languages and plugins should use semantic versioning 2.0.
    • Different pieces need not be in tandem (e.g. the glue version need not match the core version).
    • The whole project has a version number independent of the version of its components. That said, it still follows semantic versioning (that is, if any of its components have a certain piece of their versioning bumped, the overall project must AT MINIMUM have that bumped. So if core has to up its major version number because of an API break, so much the entire project).
      • This applies to official plugins such as the RTS.
      • The exception to this is test binaries

Git:

  • Generally this: https://github.com/agis/git-style-guide
  • Feature branches are considered volatile and should not have work based on them (that is, they may be freely rebased, squashed, etc), except in special cases (i.e. two people working on coinciding features may synchronize their work and then merge/rebase before merging into dev. This is how @jedirv and I should have handled the viz/test suite thing).
  • If an important hotfix must be immediately added to master, all branches must immediately rebase to apply this change (that is, dev should be rebased and features should be rebased on top of the new dev etc).
  • Master is sacrosanct and its history must not be overwritten
  • Dev is semi-sacrosanct and history generally should not be overwritten except in special cases.
  • Branches should always be rebased onto the target branch before merge

Recommendations

The following are just recommendations for contributors and such, anything may be used as long as the end result conforms to guidelines: * Anaconda Python for Python * VS Code with the RLS extension for Rust * Chrome/Chromium and Firefox for JS and web in general. * Prost for Rust protobuf; regular protoc compiler for Python/JS * Compiling JS protobuf into a single file rather than multiple.

Updated 11/10/2017 11:24

Send adaptive program queries to Agent

SCAII/SCAII

We need to be able to query the agent for feedback on a game state during replays for explanation stuff. This will be evolving a lot, my first intuition is that we should provide some simple visualization primitives (more high level than what the backend uses) like “pie chart”, “heatmap”, “highlight entity” etc. We need to take care to not reinvent an entire plotting package.

Since we’re using JS, something like bokeh is out, but we may be able to find some nice JS plotting libraries to lighten the load. (Which is a shame, it’d be nice to send rendered graphs from bokeh, they look quite nice).

Depends, minimally on #14, #16, #17, #18 + some API and protobuf consideration

Updated 06/10/2017 02:14 1 Comments

Record games in core

SCAII/SCAII

We’d like to have the core engine be able to record games.

For now, I think the best way of doing this is to explicitly have the glue for each language multiplex an outgoing action message to both the replay mechanism and the backend. Every so many actions it will explicitly ask both the backend and agent for serialization info.

The Replay mechanism also may need to be informed of configuration options such as configuration files the backend is loading, or models the agent is loading. This would all be taken care of in a header before the recording starts. After that, keyframes can be smaller since we can assume those configurations from the header are “fossilized” after the game starts.

The configuration would look something like:

Which modules to serialize? Keyframe interval (how often to serialize)

The header for a Replay would look like:

Agent to load and what type (RPC, Rust, etc) Backend to load and what type (etc) Configurations for both of these (action and state format, model paths, etc)

Followed by actions

I recommend a replay look something like ``` enum ReplayAction { Header(ReplayHeader), Action(ActionPacket), Keyframe(ActionPacket,SerializationInfo), }

struct Replay { recording: Vec<ReplayAction> } ```

This can be used with serde and a BufWriter to save it to and load it from .scaii_replay files.

Updated 05/10/2017 09:22

RPC modules in core

SCAII/SCAII

We need to be able to create the ability to do RPC from core. This would minimally consist of:

A new PluginType with RpcConfig, this would entail both a required IP and Port. As well as an optional executable and argument list (so people don’t need to manually start things themselves).

This should support Module, Agent, and Backend. In the case of “Backend”, the special functions like serialization will just be abstracted as a SerializationRequest ScaiiPacket followed by a get_messages call. Special care needs to be taken in case multiple messages get returned. May require some API change.

Updated 06/10/2017 00:26

Update boxed trait objects to impl Trait

SCAII/SCAII

Right now for the C-FFI Environment opaque struct we’re going to have to use trait objects, however, impl Trait functionality is likely to be cleaner. We can do this on nightly, but I’d prefer to target stable Rust right now because I think plugins need to be compiled with the same version of Rust as the host library.

Depends on this being stabilized: https://github.com/rust-lang/rust/issues/34511

Updated 05/10/2017 02:53

Port zcash-fetch-params to rust

zcash/zcash

I’m creating a seperate issue to follow up on the discussion with @str4d from #2597:

Since zcash seems to be into rust, is there interest to have those scripts translated? I can see why shell might be preferable, but I could offer porting them if you give me some instructions on how you would like to have it integrated into the repo. (I’d prefer to handle this in a followup pull request though).

Interesting idea. Are you envisaging that we build a binary for downloading parameters instead of using a script? It would certainly be more portable, and it would also be a self-contained chunk of code which would make integration easier. We are still figuring out how exactly we will integrate Rust into Zcash, but a separate binary like this would likely be done in a different way to rewriting or extending internal zcashd code. You should open an issue for discussing this!

This is basically what I’m suggesting. :)

From the top of my head, I can think of the following: - Add it to librustzcash and export a function that can be called from zcashd with C++ - cargo init a small project in a subfolder, compile it during the build and install the binary as zcash-fetch-params. If the build system changes to cargo, this can be moved to src/bin/ and cargo is going to build it as one of the resulting binaries.

Updated 28/09/2017 10:57

not able to get LLVM IR for rust and D

mattgodbolt/compiler-explorer

wanted to compare the LLVM IR for c++, rust and D but was only able to get the IR for c++. - for Rust when I add the argument “ –emit=llvm-ir” I get the assembler and a warning: “due to multiple output types requested, the explicitly specified output file name will be adapted for each output type” - for ldc when I add the argument “-output-ll” I only get the assembler

I guess that the IR is not shown due to other file name is generated for the IR files. is it maybe possible to add a “show LLVM IR output” button like the “show ast output”?

Updated 11/10/2017 17:24 7 Comments

Can't get ARM output in the Rust compiler

mattgodbolt/compiler-explorer

I was hoping that adding --target=arm-unknown-linux-gnueabihf to the rust compiler flags would get me arm output, but I get: error[E0463]: can't find crate for `std` | = note: the `arm-unknown-linux-gnueabihf` target may not be installed error: aborting due to previous error Compiler exited with result code 101 Same issue for thumbv7em-none-eabi, and all the arm prefixes targets in this list (output truncated for readability): `--> rustup target list aarch64-apple-ios aarch64-linux-android aarch64-unknown-fuchsia aarch64-unknown-linux-gnu arm-linux-androideabi arm-unknown-linux-gnueabi arm-unknown-linux-gnueabihf arm-unknown-linux-musleabi arm-unknown-linux-musleabihf armv7-apple-ios armv7-linux-androideabi armv7-unknown-linux-gnueabihf armv7-unknown-linux-musleabihf armv7s-apple-ios

Is there any way to have the standard libraries installed for at least armv7-unknown-linux-gnueabihf and arm-unknown-linux-gnueabihf? I haven’t read through the setup code for the images which run godbolt, but if you’re just using docker or something similar, it should be as easy as adding these lines to the setup scripts (assuming you’re using rustup): rustup target add arm-unknown-linux-gnueabihf rustup target add armv7-unknown-linux-gnueabihf

Anyway, I’ll take a look and see how painful it is to add it myself and generate the PR, but I figured I would create the issue to track the problem regardless!

Updated 25/09/2017 22:12 4 Comments

Rust: Support external crates

mesonbuild/meson

I was told that support of Rust was experimental so I can expect hickups and that’s understandable but currently it seems external crates are not supported. Since you can’t really do much in Rust without external crates, it basically means that Meson doesn’t really support Rust. IMO until this issue is addressed in some way, support for Rust shouldn’t be advertised at all.

Updated 02/09/2017 08:15 29 Comments

CHAIN support (OAE2)

miscreant/miscreant

This is a tracking issue for adding support for the CHAIN OAE2 construction as described in the paper Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance (Section 6, p. 12):

CHAIN Diagram

Design Status

Work is needed to specify the design and test vectors for CHAIN. This work is not likely to happen until after STREAM has been implemented: https://github.com/miscreant/miscreant/issues/32

Implementation Status

Language Support
Go :no_entry:
Python :no_entry:
Ruby :no_entry:
Rust :no_entry:
TypeScript :no_entry:
Updated 29/07/2017 19:56 1 Comments

STREAM support (Nonce-based OAE)

miscreant/miscreant

This is a tracking issue for adding support for the STREAM OAE1 construction as described in the paper Online Authenticated-Encryption and its Nonce-Reuse Misuse-Resistance (Section 7, p. 14):

STREAM Diagram

Design Status

More clarification is needed on the exact nonce format to be used. See the comments below.

Implementation Status

This is a tracking issue for adding STREAM support to the various language-specific implementations in this project:

Language Support
Go :no_entry:
Python :no_entry:
Ruby :no_entry:
Rust :no_entry:
TypeScript :no_entry:
Updated 25/08/2017 04:56 2 Comments

Decide whether to use Groth16 or PHGR13/BCTV14 for Sapling

zcash/zcash

PHGR13/BCTV14 is our current proving system. (See section 5.7 of the protocol spec and references there. The minor differences from the PHGR13 paper include adapting to Type 3 pairings, and a slight change to the verification key.)

Groth16 is a more recent proving system designed by Jens Groth and described in https://eprint.iacr.org/2016/260 . It has smaller proofs (about <sup>4</sup>/<sub>9</sub> the size of PHGR13/BCTV14 for a given curve), and faster proving and verification. It is proven secure in the Subversion Generic Bilinear Group Model (Sub-GBGM); see https://eprint.iacr.org/2017/599 . (The Sub-GBGM is a more realistic model than the GBGM used in Groth’s paper, so this is a stronger result than the original one by Groth. The changes made by Abdolmaleki et al are only to the CRS generation and verification, not to proving or verifying.)

Proofs in the Sub-GBGM model a Type 3 pairing abstractly, only allowing the adversary to use certain operations on group elements (group addition in G<sub>1</sub> or G<sub>2</sub>; multiplication in G<sub>T</sub>; pairing; picking a random G<sub>{1, 2, T}</sub> element with unknown discrete log; and comparison of group elements). Of course a real adversary is not limited to using these operations when manipulating group elements, so proofs in this model are only proofs of the difficulty of a certain class of generic attacks. The realism of the model is therefore an issue when assessing the value of the security proof, relative to security proofs based only on standard and/or knowledge assumptions which are available for PHGR13/BCTV14.

@ebfull has implemented Groth16 (several times! :-) ) in Rust. It’s likely that we would be using a Rust implementation if we switched to Groth16; also, it’s likely that we would be using the BLS12-381 curve. However we could also do each of these things (switch to Groth16, BLS12-381, or Rust) independently, so we need to consider the concrete performance improvement (and other features; see Sean’s comment below) of Groth16 while holding other factors constant, before making a decision about whether any risk associated with using the newer proving system is justified.

Updated 19/09/2017 17:04 18 Comments

Port llvm-hs to Rust

glaebhoerl/personal

llvm-hs-pure -> llvm-ast llvm-hs -> ?

Depend on llvm-sys for FFI.

Use language-haskell-syntax and output raw Rust (as string) to automate bulk of grunt work?

AST: Box, Rc, Arc? Can we abstract it? (Fake HKTs with associated types? How about &'a T or Cow?) [a] -> Vec<T>? Or Cow<'a, [T]>?

https://hackage.haskell.org/package/llvm-hs https://hackage.haskell.org/package/llvm-hs-pure

Updated 04/09/2017 22:02

new verifier implemented in Rust

zcash/zcash

Write a new verifier for the Zcash v1 (“Sapling”) ZK-proofs from the ground up in Rust. Run both the current verifier unchanged and the new verifier in parallel so that it doesn’t take anymore time (typically), even though it takes more CPU. Wait til both verifiers have completed. If they both return true, then the combo returns true. If they both return false then the combo returns false. If they return different things than each other, then the combo logs a detailed error message (including the proof in question and the current message and so on) and then aborts the process.

(This aborting-the-process part raises questions about blockchain forks and forward progress, but I’m pretty sure it is far more important to alert users and developers and to prevent users from accidentally accepting invalid payments than it is to minimize chances of a blockchain fork. Also, as the Zcash employees well learned during the recent security incident (https://z.cash/blog/security-announcement-2017-04-13.html), I’m firmly of the opinion that the only way to communicate with most users is to abort the process.)

Updated 04/08/2017 13:42 14 Comments

ensure all transitive rust dependencies are fetched via our depends system.

zcash/zcash

build fail with librustzcash while using a proxy. affected repository: https://github.com/rust-lang/crates.io-index

output: cd /home/riordant/Documents/Project/zcash/depends/work/build/x86_64-unknown-linux-gnu/librustzcash/0.1-2078cca28b9/.; PATH=/home/riordant/Documents/Project/zcash/depends/x86_64-unknown-linux-gnu/native/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin cargo build --release Updating registry `https://github.com/rust-lang/crates.io-index` warning: spurious network error (2 tries remaining): [12/-2] [7] Couldn't connect to server (Unsupported proxy scheme for 'https://www-proxy.scss.tcd.ie:8080/') warning: spurious network error (1 tries remaining): [12/-2] [7] Couldn't connect to server (Unsupported proxy scheme for 'https://www-proxy.scss.tcd.ie:8080/') error: failed to fetch `https://github.com/rust-lang/crates.io-index`

zcash version: v1.0.8

  • OS name + version: Ubuntu 16.04.2 (VM)
  • RAM: 8GB
  • Disk size: 39GB
  • Linux kernel version (uname -a):Linux riordant-virtual-machine 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • Compiler version (gcc -version): gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Updated 28/08/2017 17:05 6 Comments

Fork me on GitHub