Future Plans For Toy

[Ratstail91]

First, I'd like to give a big thanks to @Add00 for their recent contributions (expanding the standard library and adding the math library).

I wanted to lay out some ideas and plans I have for Toy's future in one place, rather than having many scattered notes everywhere.

First and foremost - documentation. Now that there's a math library, it needs to be documented properly. But also, there's other elements of Toy that need to be documented, such as the game engine (which is peripherally connected to the lang, though the lang does stand on it's own). I also want to fully document the current state of the bytecode format for @hiperiondev.

Speaking of the game engine - I'm a gamedev at heart, so I want to use Toy for something practical. I've always wanted to use Toy to make a game which was heavily moddable, in a similar vein as something like Minecraft. So I'm also trying to work on the engine and the basics of a game too.

Other things that I wish I'd managed to finish before release, but were pushed back include:

• A threading library
• A timer library
• More random generation libraries (numbers, perlin noise, wave function collapse?)
• IO library
• Multiple return values from functions
• Interpolated strings

If there's anything you'd like to see in Toy in the future, even if it's impractical at this stage, I'd love to hear about it.

(it's 2am again, so I'll continue this train of thought tomorrow...)

[hiperiondev]

Some ideas:

• Set the values to known sizes (uintx_t, intx_t, etc) to ensure that they are respected regardless of the architecture.

• Completely separate the compiler from the interpreter: it is essential for any embedded system.

• Optimize the interpreter: This step is already more complicated but it has a very "twisted" structure which makes it difficult to understand and optimize.

[Ratstail91]

I've got this issue sitting there, actually - #35

Segmenting the compiler and interpreter - I'm not strictly against it.

I'm not sure what you mean by twisted. Can you explain?

[hiperiondev]

Note for OPCODE documentation

For a better understanding should be:

| OPCODE | ARGUMENT1 (size in bytes),  ARGUMENT2 (size in bytes) | stack before execution | stack after execution | description |

example:

| TOY_OP_OR       | (none)                                     | lit lit  | result | Logical OR       |
| TOY_OP_VAR_DECL | identifierIndex(1or 2), typeIndex (1 or 2) | lit      |    -   | Declare variable |

[Ratstail91]

Because it processes the stack by popping the contents off the end, and parses any identifiers found to their values. It then stores the newly parsed values in returns, temporarily, then returns them to the stack.

The upshot is that the order is flipped twice (leaving it unchanged), but more importantly, any identifiers that are on the stack become proper usable values.

[Ratstail91]

The opcode docs might not even fit on the page...

I'll see what I can do.

[hiperiondev]

As I think I understand the bytecode of a function is a literal and, for all intents and purposes, it is a variable. In particular it is saved in literalCache. It's right?
If so, this creates two problems:

• All the code of a function exists in RAM, and cannot reside entirely in ROM.
• Creates a complex structure when called (call/return should only touch the Program Counter and a push for the return address. In effect, a CALL is a JUMP with a return point)

This should change.

[Ratstail91]

The function is a literal, yes. However, it isn't unflattened until needed, so it is possible to rework stuff so that the Toy_RefFunction is pointing to a position in ROM (or even the bytecode).

This should change.

I need to express how I'm feeling right now, and honestly, I'm feeling very pressured to complete the documentation, and to alter my code in ways that were never intended.

I've been trying to be as accommodating as possible for you, but you need to understand that I have my own things that I need to work on.

I'm still extremely eager to have the language ported to a microcontroller, and grateful that you think Toy is worth the effort. However, you also need to understand that I'm only one person. Basically, if you want some changes made to how the core program operates, then you need to get your hands dirty, and potentially fork the language to provide changes that you would like to see.

If you make changes, and open a merge request, I'd be more than happy to review it to see what benefits it would have. For the time being though, all I can provide is basic support and advice.

[Add00]

• Since I wrote the math library, I would be willing to document it.

• I agree I think an io library would be supper useful.

• We could use something like FastNoiseLite as the basis for an efficient noise library.

• My impractical wish would be for operator overloading. Since it would be nice since if say a matrix type was introduced if users could to matrixA + matrixB instead of matrixAdd(matrixA, matrixB)

[hiperiondev]

Do you think fileIO would work?

it's a better name

[Ratstail91]

@Add00 This is a good start for a library. Just remember, you can always open an RFC in the discussions area, too.

This proposal does seem interesting, though I personally would pack the entire File into an opaque, and simply have accessors to check the values. For example, have a look at this:

typedef struct Toy_Runner {
	Toy_Interpreter interpreter;
	const unsigned char* bytecode;
	size_t size;

	bool dirty;
} Toy_Runner;

This is the structure used within the runner library for running external scripts, and is stored in the root script as an opaque.

Another thing - you don't need a dictionary Mode just to specify the possible modes available - not if you document the valid values allowed.

Finally, I'm not too sure about returning a tuple form read; I'd prefer a function like file.getErrors() or something similar. So:

import fileio as fileio;

var handle: opaque = fileio.open("data:\datafile");
var value: int = handle.read(int);
var err = handle.getError();
handle.close();

if (err == null) {
    //do stuff
}
else {
    print err;
}

You should also look at the drive system which is available in the repl folder - it's explicitly designed to help with preventing disallowed file access.

[Add00]

Just remember, you can always open an RFC in the discussions area, too.

Ohh okay, so something similar to RFC: Formatted Comments As Documentation?

I personally would pack the entire File into an opaque.

Yeah okay, this would also prevent people from setting the error state, or size to incorrect values.

This is the structure used within the runner library for running external scripts, and is stored in the root script as an opaque.

Thanks, I'll definitely, take a look at that.

Another thing - you don't need a dictionary Mode just to specify the possible modes available - not if you document the valid values allowed.

I wanted to replicate an enum with the possible states, but I'll just switch open to use a well documented string as the second parameter.

I'm not too sure about returning a tuple form read; I'd prefer a function like file.getErrors() or something similar.

I could make it so that read returns just the type, but the type will be null when an error happens. Then make a getErrors method as you suggested.

Thanks for the feedback, it's very helpful!

[hiperiondev]

Just a comment about the libraries that depend on the Operating System or the hardware.
Before implementing the functionalities of the direct access to the resource, take the time to separate the Library - HAL, first creating the abstraction and then the port.
This makes it SO easy to switch devices without touching any of the rest of the code.
For example in https://github.com/hiperiondev/esp32_toy/blob/main/components/hal_esp32/include/hal_fs.h open, reopen, remove and rename are functions that depend on the filesystem framework and are defined as reference . In this way, changing only one compile point is directly ported to another architecture.

The structure would be

• HAL
• PORT
  -- linux
  -- windows
  -- other_1
  -- other_2
  -- ...
• LIBRARY

The library ONLY refers to the HAL API. HAL does not implement any functionality (in fact it would just be headers). The functionality is implemented in the PORT.
It is common to add a hal_config.h type file for the basic configurations and, at the end, include a hal_config.h of each port.

Maybe for some common functions of an operating system it is not so obvious, but when you start to access the hardware this is critical. And when the operating system is no longer something oriented to a PC, even more so (for example FreeRTOS)

[Add00]

Thanks for the explanation! I'll have to do some more research to understand HALs, but I do want to make the library platform independent.

[hiperiondev]

I need to express how I'm feeling right now, and honestly, I'm feeling very pressured to complete the documentation, and to alter my code in ways that were never intended.

Don't worry! There's no hurry! "Should change" is not an order, it is an expression of desire!
Do not take any of my comments as pressure, but as an idea for the future.
I have to do the disassembler first so I can rethink some changes.
I await the documentation of the opcodes to be able to move forward with that, but take your time without problems.

[hiperiondev]

My comments in this discussion refer strictly to the title: Future Plans For Toy ... I mean, not now :-)
In my case, which is aimed at a device with very low resources, my interest is focused on the interpreter, which is the only thing that will reside in the microcontroller.
In this case, the internal operation is critical, but not, at least for now, neither its API nor its machine code.

[Ratstail91]

OK, thanks.

I have thought about ways of reducing the footprint of the interpreter for you - radical things like stripping out the types and such. But I'll have to leave those kinds of things up to you.

[hiperiondev]

I also understand that an embedded system like a microcontroller, with very low resources and close contact between hardware and firmware, is an alien world for most programmers. I'm used to dealing with this.
The main problem is that almost all programmers base their developments on the idea that memory can be used as if there were no tomorrow. Things like garbage collector, free dynamic structures, automatic heap growth are very dangerous in this context.
If it is taken into account that a microcontroller can drive machinery where people can be injured, certain practices are not directly acceptable.
Take any comment of mine from this point of view, and nothing more.

[hiperiondev]

And for more problems: my English is quite bad. Maybe I don't express myself correctly sometimes.
Please feel free to correct me whenever you see something like this. It doesn't offend me, on the contrary, it helps me learn!

[Ratstail91]

Your English is pretty good for the most part, nothing to worry about there. And you're right - I've never worked on a microcontroller, so I'll delegate to you for advice on that.

[Ratstail91]

@Add00 Let me know roughly when you think the IO library will be ready - I'm gearing up for 1.3, but I'd like your lib to be included in that. No rush, of course.

(Bumping to 1.3 is because I got the short-circuiting to work, finally.)

[Add00]

Hmm, I think I'll be mostly done by Wednesday this week. I could use some help with setting up the makefile for the hal if you or @hiperiondev. Could help me build it as a library and include it as part of the toy build that would be really helpful. Since I don't have a lot of experaince with make I have been having trouble configuring it.

[Ratstail91]

I'm gonna be honest - until I googled "HAL" just now I wasn't entirely following what you and @hiperiondev were talking about, because I'm a big dumb dummy lol.

OK, first, let met have a proper look at your efforts so far... Oh wow. How did you come up with this design for the HAL @Add00? Making a structure which holds function pointers is very creative, though it's also very OO-centric.

OK, so... I'm not sure what your intentions are, tbh. If you've got the Toy lib, it needs to wrap the lower-level functions somehow...

static int nativeOpen(Toy_Interpreter* interpreter, Toy_LiteralArray* arguments) {
  //TODO: how do I get from here to the file system?
}

You'll have to forgive me, as I'm not the most experienced working directly with other coders.

---

My original thoughts were that @hiperiondev would code a custom library that I would never (need to) see, while conforming to a kind of agreed-upon API that was drop-in compatible with the library that @Add00 writes for the canonical branch of Toy. But this doesn't seem to be the way we're going. What are your plans for this?

Also, does the drive system fit into this anywhere? The drive system wasn't an arbitrary feature, it's actually there to allow for players to safely drop mods into to the Box game engine without needing to worry about potential hard drive access. They should be able to just copy/paste files into a directory, eventually.

In addition, it's also there to add my own kind of HAL, since I'm actually a published switch developer, and I'd like to port the Game + Box + Toy to the switch eventually, even if just for fun. My plan was to use some #ifdefs to swap out code within the libraries when I got around to it.

So yeah, basically, what are your plans for the IO library?

[Add00]

Sorry for the delayed response. My computer took a short drop down a long flight of stairs. Putting it out of commission for a few days.

until I googled "HAL" just now I wasn't entirely following what you and hiperiondev were talking about

It's okay, I wasn't quite sure what they were till hiperiondev mentioned them. XD

OK, first, let met have a proper look at your efforts so far... Oh wow. How did you come up with this design for the HAL Add00? Making a structure which holds function pointers is very creative, though it's also very OO-centric.

Honestly, before this I never made an HAL before, so I did some research and the OOP approach seemed the easiest to me. Though I'm not too sure if this it is the best way to make one.

You'll have to forgive me, as I'm not the most experienced working directly with other coders.

No need to apologize! I'm a very inexperienced programmer who doesn't know a lot about with open source contributing. It looks like I should have been more clear in communicating

My original thoughts were that hiperiondev would code a custom library that I would never (need to) see, while conforming to a kind of agreed-upon API that was drop-in compatible with the library that Add00 writes for the canonical branch of Toy. But this doesn't seem to be the way we're going. What are your plans for this?

Ohhhh I didn't realize that this was the approach you were aiming for. I thought that there would be one io library which would handle all platforms internally. I think this approach of having a canonical one plus an agreed interface would be much easier to handle.

Also, does the drive system fit into this anywhere? The drive system wasn't an arbitrary feature, it's actually there to allow for players to safely drop mods into to the Box game engine without needing to worry about potential hard drive access. They should be able to just copy/paste files into a directory, eventually.

I do plan on incorporating the drive system, I didn't have a chance since I was researching and making the HAL.

In addition, it's also there to add my own kind of HAL, since I'm actually a published switch developer, and I'd like to port the Game + Box + Toy to the switch eventually, even if just for fun. My plan was to use some #ifdefs to swap out code within the libraries when I got around to it.

Woah that really cool! I think that should still be possible.

So yeah, basically, what are your plans for the IO library?

Now that I have a better idea of what was actually expected. I think I will drop the HAL completely and just focus on building the fileio library using the drive system to handle paths, with the feedback in this post.

I didn't mean to cause so much confusion, but now that I've read your comment I have a much better idea of what was actually intended. I'll definitely communicate more going forward to prevent more confusion.

[Ratstail91]

No worries about the delay. Laptops can be problematic at the best of times lol.

My theory is that different users would need different libraries for different situations - so I tend to plan it so that there's no guarantee that any one library will be available (though standard is a bit special in that regard).

I'm used to leading small teams on occasion (though never via github like this before), so I should've known to be more open with my ideas - my bad. At the same time though, please feel free to critique my ideas - don't take them as gospel, if you feel like there's something you'd like to add.

Don't worry about causing confusion - this always happens in new teams, and is an expected part of the plan.

Finally, I'd like to add that no one has any obligations to this project - I'll assign you a task if you ask for one, but this is ultimately a project written for fun, so lets have some fun with it!

[Add00]

I completely agree! To have fun and learn are why I wanted to contribute to this project in the first place.

[hiperiondev]

Hey!!! I did not disappear! I just have a lot of work.

Regarding the HAL, I understand that the design is complicated, especially if you never had contact with it.

You can see some well organized examples here:

https://github.com/FreeRTOSHAL/fhal-driver

https://www.chibios.org/dokuwiki/doku.php?id=chibios:products:hal:start

http://www.wsn.agh.edu.pl/download/public/halfred/0.2.0/

[Ratstail91]

I just have a lot of work.

No worries!

Regarding the HAL - as I understand it, the internals of the HAL change based on platform. What I'm trying to convey above is that the library itself might be better off handling the HAL's role.

For example - if there's a core library called lib_fileio, you could have a specialized library called lib_fileio_esp32 which replaces the former by simply changing which hook gets injected into the interpreter on startup. It would be a drop-in replacement, you see?

[hiperiondev]

It could be an option. In fact it is more or less what a HAL does. But for it to make sense, the library references should be immutable, that is, refer to an API (the HAL header). The port should only own the API implementation.
To summarize: no matter what the hardware or operating system, the Toy library should never be modified or injected with external code, and the header it references should never change (for example: suppose that for two different architectures there are two different ways of doing fopen, in that case the library should have only one access to the HAL API - fopen_hal()? - exist fopen_hal in the API entry - toy_hal.h - and its implementation in each port, which would be the responsibility of whoever wants implement each infrastructure).
The idea of having a HAL is that the specific code of each access to the hardware/operating system should NEVER be injected into the application and that it be transparent to modify it.

In fact you are already using this without knowing it :-)
Every time you make use of any access to a driver of the operating system, it implements an API to access the hardware that is, effectively, the HAL of the system.

[hiperiondev]

For example, when you use a graphics library (opengl, sdl, etc) you use an API that generates "something" (a primitive, a motion function, etc). This library accesses the graphic hardware to do it and this hardware can vary a lot, from an accelerator board to a simple display that can only turn points on and off and nothing else. You never have access to the hardware or care about it, in fact you are completely unaware of how it works and the library can work on many different hardware that work in custom ways. That's why the library has an immutable HAL implementation and its implementation depends on each hardware. Every time you want to port that library to a new video card, ONLY the implementation (port) is modified and nothing else from the library is touched. HAL API and Library API are, and should be, immutable

[Ratstail91]

I'm busy today and tomorrow, but we'll see how it works out once Add00 finishes his new io library (sorry, not too focused today).

[Ratstail91]

I'll be joining a game jam, which starts at 2am tonight - I'll be using Toy and Box...

[Add00]

This sounds, cool, which game jam was it for?

[Ratstail91]

the game wisdom jam, but IDK if I'll finish, as I don't want to push myself too hard - https://itch.io/jam/game-wisdom-game-jam

[Add00]

The jam looks interesting, even if you don't finish you can still learn and enjoy the experience!

[Ratstail91]

Sorry, been kind of slacking due to irl stuff. Will try coding tonight, maybe.