Complete C standard library
Posté le 19/08/2018 19:31
Motivation
Until now there was no complete C standard library (aka
libc) available for the Casio calculators. Although some parts of this library have been provided by fxlib and
gint, there was no libc implementation complying with the standard and compatible with the sh3eb architecture ready to use.
To change that, I decided to port
newlib to the Casio CPU. Newlib is an alternative libc implementation intended for use on embedded systems.
Alpha
Follow this link and click the download button in the top right corner:
>>> v1.1 <<<
Instructions on how to install newlib alongside with gcc (big shout-out to Lephé):
Compiler sous Linux avec GCC
Features for Casio fx9860g calculators:
* C standard library
libc
→
printf implementation to print text to the display
→ Dynamic allocation of memory using
malloc and
free
→ Memory manipulation using
memcpy,
memcmp,
memset etc.
→ String manipulation using
strcpy,
strcmp,
strstr,
strtok
→ ...
* Math library
libm
→ Floating point arithmetics
→ ...
* Automatic library and include path recognition after installation
* Basic Casio features:
→ implementation of
GetKey,
Bdisp_AllClr_DDVRAM,
Bdisp_PutDisp_DD,
Print and
locate without fxlib (but you can use it if you want)
Code
To contribute or get all those bleeding edge features, see the code including all further information:
libc (my GitLab repository)
The project you find in my repository is a fork of
the official newlib repository. To make it easier for everyone to follow, I try to keep a clean
git history. That means that all my changes are located on a dedicated branch with meaningful commits.
I also try to keep the changes to the upstream library minimal. That increases maintainability a lot.
Contributing
If you have a ideas, feature request, found a bug or simply want to contribute, just send me a message and you're in! You can also create
Issues and
Merge Requests directly in the repository. As in every OpenSource project: merge requests welcome!
Citer : Posté le 17/09/2018 17:25 | #
I still seem to miss how you would implement this. To be honest, my understanding of the linux kernel is very basic. What confuses me the most is the following statement:
Well, to be precise I'd like to use them as indices in an array of structures, where the structure either contains the function pointers or a "device type number" that tells us which function to call.
What do you mean with "device type number"?
I understand that you want to implement an array of structs which contain the function pointers to the syscalls you need for each file descriptor:
/* Your idea of implementing it */
#include <stddef.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
// Let's assume that stdout is 0 for simplicity's sake
#define stdout 0
typedef struct {
int (*open)(const char *pathname, int flags);
ssize_t (*write)(int fd, const void *buf, size_t count);
/* all other syscalls... */
void *data; // <-- pointer to custom data
} fd_t;
int open_terminal(const char *pathname, int flags) {
// here sime init implementation if necessary
return 0;
}
ssize_t write_terminal(int fd, const void *buf, size_t count) {
// implementation using locate()
return 0;
}
// syscalls for terminal
fd_t display_syscalls = {
.open = &open_terminal,
.write = &write_terminal,
// ...
.data = NULL
};
// all syscalls: index = file descriptor
// in reality this would have a dynamic length!
fd_t all_syscalls[7];
void main(void) {
// set it up
all_syscalls[stdout] = display_syscalls;
// let's use it!
int fd_terminal = all_syscalls[stdout].open("/path/to/file", O_RDWR);
char buf[] = "Hello World";
all_syscalls[stdout].write(fd_terminal, buf, sizeof(buf));
}
1) It obviously will not work this way, because the open() syscall returns the file descriptor you need to know to access the array of fd_t structs in the first place.
fopencookie() does not work that way since instead of open(), fopencookie() is called.
2) After being opened via open(), every file will be associated with a unique file descriptor. This means that you would have to dynamically add an element for each file to your variable length array. Also, for all opened files, the function pointers to the syscalls will be identical (which would be redundant).
3) I do not see how the custom data fd_t.data can be of use here, since it cannot be passed to the syscalls. Their signature cannot be extended.
4) Your implementation contains redundant information: the file descriptor. What determines which syscall is called is the index = file descriptor. Additionally you pass the file descriptor to the syscalls (e.g. read()).
I suggest an implementation somewhat like this. (Again, I'm not very firm with the linux kernel and I suppose opening the terminal this way is not what you would normally do. It's just an inspiration how it could look like for both files and strams.)
/* My idea of implementing it */
#include <stddef.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
// Let's assume that stdout is 0 for simplicity's sake
#define stdout 0
int open_terminal(const char *pathname, int flags) {
if (strcmp(pathname, "/dev/tty") == 0) {
// some init
return stdout;
}
// implementation for all other files
return -1;
}
ssize_t write(int fd, const void *buf, size_t count) {
switch (fd) {
case stdout:
// implementation using locate()
return 0;
// implementation for all other file descriptors
}
return -1;
}
void main(void) {
// let's use it!
int fd_terminal = open("/dev/tty", O_RDWR);
char buf[] = "Hello World";
write(fd_terminal, buf, sizeof(buf));
}
Citer : Posté le 17/09/2018 19:16 | #
Okay, since you did that very seriously, I understand that I must show you some code. I'll explain briefly the differences between my suggestions and the code you posted. You're really close though, good job
fd_t display_syscalls = {
.open = &open_terminal,
.write = &write_terminal,
// ...
.data = NULL
};
This is a problem, I want to have a fd_t for each open file descriptor. So when I open stdout, in fact write_terminal() should not only return file descriptor 1, but also fill entry number 1 of the kernel's file descriptor table with the appropriate syscalls, something like this (please note that the open() member is gone, it was a mistake and I mentions this later):
int fd = get_a_new_fd(); /* opaque */
fd_table[fd].read = read_terminal;
fd_table[fd].write = write_terminal;
...
fd_table[fd].data = NULL; /* or anything *_terminal() needs */
return fd;
}
Then the write function can be implemented like this:
return fd_table[fd].write(fd_table[fd].data, buf, count);
}
What I'm doing is really, really the same as your code, except that I'm using function pointers instead of populating a switch statement. Is my example clear enough?
Clearly, open() cannot use the function pointers associated with the file descriptors, but what I suggest is merely that it sets them up.
I realize I made a mistake there, I should not have included open() in the function pointers because, as you pointed out, I would need to know the file descriptor before being able to call open() in the first place! I included it because, when I presented my idea first, I looked at the code I had written for a simple kernel experiment of mine. In this experiment I create file descriptors out of a hardware device description without opening them immediately. Depending on the device, the open member is set up appropriately and called only at a later stage, sometimes even closed the re-opened.
Now I ought to fix this for good, the principle I wanted to suggest is that:
- open() creates a file descriptor and populates its read, write, seek, close, whatever members
- These members are stored in an fd_t structure which is not returned as such because it's large
- Instead, the structure is inside an array and the appropriate array index is returned
This array is known as the file descriptor table and is exactly what happens in Linux. There is one such table per process.
Yes for the VLA, but since there can be only 4 Bfile files open at the same time, we can bound the number of open file descriptors if we're cunning enough.
And yes for redundancy, a bit about that later.
I suggested an example where write() passes it to the device-specific writing function, which itself has a non-standard signature (in fact it will be somewhere in a kernel driver).
Normally there isn't any additional information, because :
- When you call the syscall, you pass it a file descriptor
- This file descriptor points to an entry in the file descriptor table, which gives the functions
- Without the file descriptor, you cannot access the functions.
Again, I ought to mention that if you have two open files, you will have two fd_t structures with different indices but same members. This is quite of a redundant encoding.
Your suggested implementation is exactly along the lines of mine, except that:
- I store "device-type" information in the file descriptor table instead of hardcoding file descriptors
- I use function pointers intead of a switch.
You might have noticed that the Linux kernel allocates file descriptors in a non-predefined manner. You cannot know whether fd 3 will always be a storage memory file, or a main memory file, or a serial connection. To use your switch, you either need to store this information somewhere, or decide that fd 3 will be used for serial, 4-7 for storage memory files, and so on. This is possible but it doesn't feel very flexible.
I mean that storing the whole 4/5 function pointers in each entry of my file descriptor table is a bit redundant because the number of possible combinations of function pointers is low: say 5 (main memory, storage memory, terminal, serial, USB). So instead of the function pointers I could store an enumeration value like dev_tty or dev_usb and then do the switch on this in read() and write().
I can write a full example if required, but I'm kinda short of time right now, so if this is clear enough I'll skip it.
Citer : Posté le 17/09/2018 22:13 | #
Yes, it's very clear now, thanks you for your effort!
I'm sorry that took a while
I somehow failed to see that the functions write(), read(), open() etc. will actually use the fd_t structure...
I have to admit that your solution is far better than I thought. Now I do not see any flaw anymore
- All files are created with a fixed size.
- Any directory not at the root of the filesystem will contain itself.
- Writing an odd number of bytes with Bfile_WriteFile() will cause Bfile calls to randomly raise hell.
- Writing with Bfile_WriteFile() can turn 1's into 0's but not the opposite.
- Failing to close files or search handles will break the filesystem.
- Optimizing the filesystem solves USB communication issues.
Yes for the VLA, but since there can be only 4 Bfile files open at the same time, we can bound the number of open file descriptors if we're cunning enough.
That sounds awful! How will you ever communicate that to a user? Yeah I know, RTFM, but frankly nobody will expect such a behavior (before reading the doc)
Let me phantasize a bit...
There might be the possibility to add an abstraction layer and implement a hassle-free file system on top of the BFile implementation. That would probably have a big impact on performance and a user would only be able to access files from within gint. This means also that files could not be loaded onto the calc with the tools we know...
What do you think?
Citer : Posté le 18/09/2018 06:47 | #
No problem really, it's not that easy to understand when it's only a few words, and in English on top of that... (and I did id too!)
Yeah, this is awful, especially since the Bfile_WriteFile() odd-bug is not documented anywhere AFAIK.
I wish to expand on "Bfile_WriteFile() can turn 1's into 0's but not the opposite": this is probably a Flash memory problem, bits in the Flash can be turned from 1's to 0's selectively but to do the converse the sector has to be erased back to all 1's. All files are initialized with 0xff and I think this is the reason. The need to initialize also explains why the size of the files is fixed, and why you can't edit a file: you must delete it then re-create it anew. This is a pretty restrictive filesystem.
I ought to mention, on machines that have SD cards, the filesystem is as clean as you expect it to be... except for Bfile bugs, but right now it's difficult to distinguish Bfile bugs and storage-filesystem inconsistencies.
If it's on top of Bfile - even if it just stores on the Flash - the some of the restrictions about recreating the file each time you write to it will apply. I suspect this Flash chip is not supposed to be used like that... and calling the Flash sector erase syscalls is really dangerous because you can erase everything with it, including the bootcode. >_<
Unfortunately there's not much we can do because storage memory is the only persistent storage in the machine...
Citer : Posté le 18/09/2018 14:53 | #
If it's on top of Bfile - even if it just stores on the Flash - the some of the restrictions about recreating the file each time you write to it will apply. I suspect this Flash chip is not supposed to be used like that...
You are most certainly right. Generally, EEPROM chips are not meant to be written to frequently. According to Andreas Bertheussen and Simon Lothar, Casio is using an Spansion S29JL032H flash (datasheet). The limitation that you need to erase before you write again generally applies to flash devices.
From that perspective, implementing a file system on the flash memory does not make sense. While you could implement a buffered I/O to reduce the write cycles, a flash should simply not be used that way.
Unfortunately, my fx9860gii has no SD card slot. Having a file system for SD card seems much better to me. Do you think the community uses SD cards?
Citer : Posté le 18/09/2018 15:01 | #
SD cards disappeared with the Graph 95, so unfortunately no. You can consider them unused.
Ecrivez vos programmes basic sur PC avec BIDE
Citer : Posté le 18/09/2018 20:30 | #
Yeah, that clearly sucks. I think we can work around this by caching all read-write file I/O and flusing the buffers at close time, but that would require a huge amount of RAM. Neither the fx-9860G II nor the fx-CG 50 has enough RAM to cover a large proportion of their storage memory. I reckon it'd be feasible, though. What do you think?
Citer : Posté le 18/09/2018 23:39 | #
Hmm, I think we could implement a buffered I/O file system as well, I think the usefulness heavily depends on the use case. Let me depict two scenarios:
1) The user wants to upload some files using a known tool. An addin running on top of gint needs to read and perhaps modify them.
=> Consequences:
* We need to use the BFile interface and must not add an intermediate layer (e.g. storing own metadata).
* We cannot solve the odd-bug
* Files are accessible from the outside (non-gint)
2) The user needs to access files only within one addon (or multiple addons which use exclusively gint to read/write to ROM).
=> Consequences:
* We could store additional bytes (e.g. making sure that the file size is even)
* We could solve the odd bug
* We could store additional metadata
* Files would be exclusively accessible through gint
Without a buffered approach, file I/O does not make sense. Even if we implement a cache, especially writing will be very inflexible and slow. If RAM does not suffice for the file, we might be able to flush prematurely (or are we?). I think it's better than nothing, but we should know the limitations (and communicate them well). The EEROM is for "save games" only, after ~100k write cycles, it breaks permanently.
Citer : Posté le 19/09/2018 06:28 | #
My idea was to write data to the storage memory once the file is closed or gint is left, so in fact the file would be accessible from the outside (after the add-in finishes, that is).
Exactly, I was thinking about designing a "smart" policy to flush files like the OOM killer kills processes. Basically a heuristic accounting for the size of the file and how long it's been since it was last written to.
This caching method would be pretty bad for very large files but very useful for little game saves which are the most common cases, I suggest that we create an open() flag (and make it available through fopen()) to control I/O caching, so that advanced users can use whatever's appropriate for their use case.
Citer : Posté le 19/09/2018 11:12 | #
Alright then, let's do it!
Citer : Posté le 19/09/2018 13:12 | #
We have not yet talked about how this code could be separated between gint and the standard library; honestly most of this would be implemented in the kernel in a real environment, but we might be better off splitting the work.
Citer : Posté le 19/09/2018 15:27 | #
I totally agree.
In my opinion both implementing the syscalls and the underlying file system is part of gint. So I think contributing code would require working with your repo. I see you did not push in a while
We can reuse big parts of my console implementation (or yours if you prefer that).
Oh and I noticed that if we implement control characters such as \n (and we really want to do that), we are neither ASCII compliant nor FONTCHARACTER compliant. I'm currently writing a doc based on the YAML file by Cakeisalie5.
Ajouté le 19/09/2018 à 15:57 :
I had some difficulties with .png files lately. I cannot download /doc/casio_charset/normal/0x7f50.png. On closer inspection, it is an empty file.
My local copy is a valid picture and after cloning the repo once again, I also receive a valid picture, so I think it's an issue with GitLab.
I also checked icon.bmp in my test repo and icon.bmp in the fxSDK repo and both files show the same behaviour.
Ajouté le 19/09/2018 à 16:34 :
Addendum: what I actually want to do is link them in my rather lengthy (and still uncpmlete) tutorial on the Casio encoding and how it affects stdio
Citer : Posté le 19/09/2018 16:57 | #
The unsigned char type indicates a byte in C, the char type is a type which is only used for historical reasons. Plus, there are two ways to represent FONTCHARACTER (the name I gave to the encoding using the type's name in the fxlib), both being used by different parts in the system:
- the variable length encoding, which detects special byte values such as 0xF7 and reads the other byte if it's the case or only uses the current byte as the character. Used for displaying and storing data such as programs.
- the fixed length encoding, which encodes every char as a 16-bit (2 bytes) value, e.g. 00 32 F7 05 for [0x32, 0xF705]. This takes more space and is dependent on the system's endianness; usually, big endian is being used as it is the calculator's native endianness. This is where the FONTCHARACTER type, an unsigned 16-bit integer type, is useful. Used within the Bfile interface.
Plus, FONTCHARACTER is not entirely ASCII-compatible as the first 0x20 characters are not the ASCII control characters but other printable characters. I've worked on making a reference here (that I call « refc », which stands for « Référence FONTCHARACTER » in french) and on a related binary character set file format which could be embedded into applications, however this is an unfinished project. Check this out if you have time
Mon blog ⋅ Mes autres projets
Citer : Posté le 19/09/2018 17:03 | #
Damn, you're right! I just recently solved the keyboard problems and I'm currently overloaded by a school project (we'll vote for projects and basically 20 people will work on the two best; I'm doing my best to pass a free-software calculator project), unable to do anything about gint. Good news is, I'm not facing any bug right now.
Ah, I don't think live preview of images ever worked. Configuring Gitlab is actually very difficult and we never really made our way through it.
I suppose hard-linking to Gitlab URLs isn't a very good idea...? Unfortunately I have no other alternative to suggest other than hosting the images somewhere else.
The unsigned char type indicates a byte in C, the char type is a type which is only used for historical reasons.
Might as well mention that whether char is unsigned or not is compiler-dependent.
Citer : Posté le 19/09/2018 17:47 | #
[...] This is where the FONTCHARACTER type, an unsigned 16-bit integer type, is useful. Used within the Bfile interface [...]
Thank you for the insight. In fact I'm currently writing a little tutorial (which is intended for beginners but does not go too much into detail) about the Casio encoding. Lephe already mentioned your project and I managed to generate a huge markdown table from your YAML file. Great work!
Ah, I don't think live preview of images ever worked. Configuring Gitlab is actually very difficult and we never really made our way through it.
That is very unfortunate since the I really wanted to provide the actual pictograms in said table. I already extracted all of them
Hosting about 751 pictograms elsewhere is not an option I guess. Tja, schade
I suppose hard-linking to Gitlab URLs isn't a very good idea...?
How would this work? The problem is that the pictures are empty (size 0kB) if downloaded/referenced via the web interface.
Citer : Posté le 19/09/2018 17:54 | #
I think you'd better host this on https://bible.planet-casio.com/ , what about it? This is just an FTP repo with nothing to upload without having access to the VPS, but we can set up an access to let you update your own directory from a git repository. This way, you'd have a full web interface.
Citer : Posté le 19/09/2018 20:10 | #
This is a problem with the gitlab (happens with any file). You'll have to clone the repo locally.
Ecrivez vos programmes basic sur PC avec BIDE
Citer : Posté le 19/09/2018 20:42 | #
@Zezombye I noticed that cloning works, but that does not help me. I want to embed them into markdown.
@Lephe That's a good idea, actually! I might use some more html/css magic, too
Ajouté le 27/09/2018 à 11:33 :
Did you give me access, yet? I do not see any dedicated directory nor can I connect using my usual credentials.
Citer : Posté le 27/09/2018 14:49 | #
Oh, no it'd be a bit subtler. I think of creating a user and a directory for you to connect via SSH, but I'm admittedly not proficient enough to get it done fully properly, or not even sure this is a good idea. Breizh_craft, any idea on how to grant Memallox enough access to update a bible directory from Git without breaking the security of the server?
Citer : Posté le 27/09/2018 16:01 | #
@Memallox : I sent you the informations to connect to the VPS and upload files to the bible in a private message.
Citer : Posté le 30/09/2018 20:50 | #
I just finished recompiling the sh3eb-elf toolchain after switching to a new, more powerful laptop. It was pretty fast, but I did not manage to build newlib smoothly.
I still had issues with casio_syscalls.c being mentioned and no Makefile.am file to be found - I'm completely puzzled. make clean did not seem to clean everything, make distclean managed to fail at some point until the target just disappeared. I can say I didn't understand a thing...
So I cloned a fresh copy of the repository (
I'm not saying there is anything to "fix" there, but yeah, building newlib is rather messy compared to binutils/gcc.
Ajouté le 30/09/2018 à 21:10 :
Aha, but there is nothing for "sh4" in there. Gotta use the sh3eb-elf version again I suppose.
Did you ever tried to use your version of newlib with the sh4eb-nofpu-elf toolchain? It would be cool if it worked.