Complete C standard library
Posté le 19/08/2018 19:31
Motivation
Until now there was no complete C standard library (aka
libc) available for the Casio calculators. Although some parts of this library have been provided by fxlib and
gint, there was no libc implementation complying with the standard and compatible with the sh3eb architecture ready to use.
To change that, I decided to port
newlib to the Casio CPU. Newlib is an alternative libc implementation intended for use on embedded systems.
Alpha
Follow this link and click the download button in the top right corner:
>>> v1.1 <<<
Instructions on how to install newlib alongside with gcc (big shout-out to Lephé):
Compiler sous Linux avec GCC
Features for Casio fx9860g calculators:
* C standard library
libc
→
printf implementation to print text to the display
→ Dynamic allocation of memory using
malloc and
free
→ Memory manipulation using
memcpy,
memcmp,
memset etc.
→ String manipulation using
strcpy,
strcmp,
strstr,
strtok
→ ...
* Math library
libm
→ Floating point arithmetics
→ ...
* Automatic library and include path recognition after installation
* Basic Casio features:
→ implementation of
GetKey,
Bdisp_AllClr_DDVRAM,
Bdisp_PutDisp_DD,
Print and
locate without fxlib (but you can use it if you want)
Code
To contribute or get all those bleeding edge features, see the code including all further information:
libc (my GitLab repository)
The project you find in my repository is a fork of
the official newlib repository. To make it easier for everyone to follow, I try to keep a clean
git history. That means that all my changes are located on a dedicated branch with meaningful commits.
I also try to keep the changes to the upstream library minimal. That increases maintainability a lot.
Contributing
If you have a ideas, feature request, found a bug or simply want to contribute, just send me a message and you're in! You can also create
Issues and
Merge Requests directly in the repository. As in every OpenSource project: merge requests welcome!
Citer : Posté le 02/09/2018 14:23 | #
I'm genuinely impressed. It works! o.O
I named it aaa_casio_syscalls.c. You must not forget to edit the file name in Makefile.am and run automake (all in the same directory).
For building, make clean should do the job, but I like to keep the sources and the built files separated cleanly. That's why I have a folder build-newlib where I call ../newlib-cygwin/configure (where the sources are). All files generated by configure and make will be located in build-newlib. Before rebuilding, I simply delete build-newlib.
Citer : Posté le 02/09/2018 14:34 | #
Good, Lephe's effect-cause tracking skill levels up!
The problem is that renaming the file to a lexicographically-minimal name is not a proper solution, right? x)
Maybe you can name your file calloc.a (or any other appropriate name used in the standard directory) and exploit the file-level replacement mechanism to properly override the default versions? The same applies for malloc(), currently we're lucky that the archive is being sorted but it doesn't feel right to have the same symbol twice in libc.a.
Citer : Posté le 02/09/2018 14:46 | #
While renaming the source file to e.g. aaa_casio_syscalls.c works, I consider that a not-so-clean solution. Is there any way to solve this issue more intuitively? Maybe creating different files sys/sh3eb/malloc.c, sys/sh3eb/calloc.c etc. for each function I want to overwrite?
Ajouté le 02/09/2018 à 14:48 :
I see, we think alike
Citer : Posté le 02/09/2018 14:49 | #
Yes, precisely! But I can't test it yet (I did not manage to re-configure after renaming a file until now). If you do, then I'd like to know whether it works.
Edit : I did not see your edit earlier, I will try it out.
Citer : Posté le 02/09/2018 15:36 | #
If you delete/add/rename a source file such as sys/sh3eb/casio_syscalls.c, you need to edit sys/sh3eb/Makefile.am.
There, you will find a line like
Delete/add/rename the filename here.
Then run automake to update sys/sh3eb/Makefile.in:
cd sys/sh3eb/
automake
Ajouté le 02/09/2018 à 15:37 :
I hope this is a bit clearer than my previous explanation
Citer : Posté le 02/09/2018 17:11 | #
Yes, it's clear, thank you. I have some issues with autoconf versions (I have 2.69 while the files were generated by 2.68, and something was removed in-between); I'll have a closer look.
Citer : Posté le 02/09/2018 17:31 | #
Ok so I think having a file called calloc.c is sufficient to replace the definition of calloc.c. I'm currently restructuring my files a bit...
Ajouté le 02/09/2018 à 19:01 :
Ok, I outsourced all stdlib Casio syscalls to their own dedicated files. The substitution seems to work fine now!
Big thanks to you, Lephenixnoir!
Ajouté le 02/09/2018 à 19:54 :
I ran your memory test once again and it still succeeds.
Citer : Posté le 02/09/2018 20:35 | #
Ok, I outsourced all stdlib Casio syscalls to their own dedicated files. The substitution seems to work fine now!
Big thanks to you, Lephenixnoir!
You're welcome! It was also a great deal of fun to investigate this ;D
It would be embarrassing otherwise
Citer : Posté le 02/09/2018 20:42 | #
I think there has to be done more testing but until more bugs turn up there are two major things to consider:
1) Is the standard library compatible to other Casio models? Probably not, so the question is rather: What models are there, how are they different and what has to be done to make libc compatible?
2) How far are you with gint? Is there any progress concerning the compat branch?
Citer : Posté le 02/09/2018 20:54 | #
For the first question, if you're not doing any input/output then you're about as compatible as you can get. The only add-in development platforms that can be considered to exist are the monochrome models centered around fx-9860G II and the fx-CG 50.
There is a difficulty regarding the monochrome models; some of them have an old SH-3 CPU whereas most recent machines have an SH-4A-based MPU. The fx-CG 50 is exclusively SH-4A. gint dives deeply into the compatibility issues because it does all the hardware driving. But "pure" code compiled for the sh3eb platform is by all means compatible with all machines.
I expect your current newlib port to be straight usable on an fx-CG 50 provided that you adjust the sycall ids and the few parameter differences that exist between the monochrome- and color- versions of some I/O syscalls.
Edit: And that you use appropriate fx-CG 50 linker scripts.
As for the second question, I have not progressed in the last ~10 days because I was busy with an internship report and defense which will happen tomorrow. I am quite upset because the extra timers are still throwing behavioral subtleties at my unsuccessful attempts to tame them. I trust that I will be able to understand and document them after testing for long enough. Unfortunately I don't know how long it will take...
If it weren't for this current issue, I should be ready to finalize keyboard operation by working around another kind of strange, undocumented behavior which is attributable to the KEYSC. I can easily port all drawing functions to the monochrome parts of the library (although I'd like to update them because I have a few lightning optimizations to try out
Citer : Posté le 03/09/2018 16:25 | #
I expect your current newlib port to be straight usable on an fx-CG 50 provided that you adjust the sycall ids and the few parameter differences that exist between the monochrome- and color- versions of some I/O syscalls.
Great! I will come back to you once I updated the syscalls for the fx-CG 50. To be honest, I could not even test this since I only own a fx-CG 50.
[...] most recent machines have an SH-4A-based MPU
I feel like as long as you do not update your tutorial, people will still build with the sh3eb-elf-gcc compiler
There does not seem to be very much interest of developing sh4-only add-ins anyway
I was busy with an internship report and defense which will happen tomorrow
Good luck!
The next thing I want to do is implementing stdin (scanf). There is still plenty to do. Unfortunately your getkey is not compatible to the fxlib GetKey, right? Actually, what's the benefit of your key codes?
Citer : Posté le 03/09/2018 16:36 | #
I feel like as long as you do not update your tutorial, people will still build with the sh3eb-elf-gcc compiler
There does not seem to be very much interest of developing sh4-only add-ins anyway
Apparently most benefits of building for an SH4 architecture lie in using the FPU for various unexpected tasks... since we don't have an FPU the interest is low. The unaligned read instruction movua.l does come in handy, though. For my memcpy() function it will be a must.
I personally have both an sh3eb-elf and an sh4eb-nofpu-elf toolchain; as I understand that SH3 machines have almost disappeared I have added a tutorial section to let the user choose their target. sh4eb-nofpu-elf is still mainly for fx-CG 50 developers anyway.
Thanks! It worked out pretty well, got compliments mainly.
Ok, so about the keyboard. My getkey() reproduces the behavior of the system's GetKey(). In a basic, non-customized setup the repetition delays are the same, only arrow keys are repeated, SHIFT and ALPHA modifiers are enabled, return to main menu is enabled, and management of the display backlight is enabled.
I have key codes whose high nibble is the row and low nibble is the column of the key in the keyboard matrix. This is sometimes called a matrix key code and syscalls use them sometimes. What the system GetKey() has is actually key opcodes where each key value is a Basic opcode. The catalog on SHIFT-4 is built in GetKey() and you can see very quickly that the "key" codes returned for catalog entries correspond to the encoding for Basic opcodes.
If you use GetKey(), then you should be using the KEY_* macros from fxlib, which you can very easily redefine to match the gint values. So I'd say getkey() is compatible with GetKey() (oh, except for returning the key value instead of using a pointer argument).
My keycodes don't have intentional "benefits"; they represent the key positions in a straightforward way. But still:
- They can be used to retrieve the key position efficiently (see Dark Storm's touchlib)
- They are easy to generate for the keyboard driver
- They can easily be transformed to be used as keyboard indices
I'd rather ask, what's the benefit of the system's key codes? They serve the BASIC interpreter but for add-ins?
Citer : Posté le 03/09/2018 16:38 | #
Lephé, I saw that you use CI for your gint repository (which fails since quite some commits).
To be frank, I never really used CI (besides for work, but that's just something that works automatically). While it might not give us a *real* benefit (this is a 1,5-man-project anyway) I consider setting up at least an automatic building for newlib. This way I can (and have to) "proof" to the users that my newlib is building without errors.
Do you think it is much work to do for you? I think you would have to set up a Runner in GitLab. If you want, I could do this by myself (if you grant me the required permissions).
If you think this is too much work (with too little payoff), that would not be a problem for me. It was just an idea I had
Ajouté le 03/09/2018 à 16:50 :
Apparently most benefits of building for an SH4 architecture lie in using the FPU for various unexpected tasks... since we don't have an FPU the interest is low
So does the SH4 arch (i mean our calculator) have a FPU or not?
I'd rather ask, what's the benefit of the system's key codes?
That's not how this works
What I would like to ask you is a summery of what I am allowed to do (using syscalls) and still be compatible with gint. To be more specific, I want to use locate(), Print() and GetKey() internally. Additionally, I want to support at least Bdisp_PutDisp_DDVRAM() and Bdisp_AllClr_DDVRAM() to facilitate minimal testing without using gint/fxlib. I also consider using wait_ms() for timing, but I did not even test the syscall yet.
Ajouté le 03/09/2018 à 16:57 :
What is actually the difference between Bdisp_PutDisp_DD(), Bdisp_PutDisp_VRAM() and Bdisp_PutDisp_DDVRAM()?
Citer : Posté le 03/09/2018 17:10 | #
Lephé, I saw that you use CI for your gint repository (which fails since quite some commits).
I used to build gint with a Gitlab runner configured from my Raspberry Pi, but after I last reinstalled the Pi I did not recompile GCC (which took 6-7 hours the first time) because I wanted to give canadian-cross a try. I have not done it yet.
I'd be very happy to build newlib on my Pi if I get the canadian cross right. Compiling GCC is still a bit long on my computer; I will switch to a new one shortly, which will probably makes it easier for me to experiment. If it's not too urgent for you, I'll gladly do the continuous integration for you repository.
No it has not. What I meant is that apparently what GCC does best is using the FPU to accelerate the program; other SH4 features are seldom used in GCC-generated code.
Oh, sorry. I have a bad habit of implementing what I think is "best" regardless of what already exists. I think it's called perfectionism. xD That leads me to changing gint's API quite often, I know that I must do something about it.
So I still need to provide a decent justification. To be honest I had several reasons to implement it that way:
1. The row/column information is what the hardware provides (thus simpler for the driver)
2. It fits on 8 bits, unlike the fxlib's many control key codes with values around 30'000
3. I trusted that the widespread use of KEY_* macros made this a small incompatibility.
Point (2) is especially important today because I extended key codes into a key event structure which provides richer information but still fits on 32 bits, and can thus be passed as a single register. 16-bit opcodes are not compact enough for such a structure.
Hmm, I'm not sure I can do an exhaustive list. Here are a few general guidelines that apply when there are no thread/interrupt/reentrancy issues whatsoever:
- Drawing is safe, but gint's VRAM is 4-aligned, so mixing fxlib/gint drawing functions will not work as expected.
- All DD operations are safe, but you are responsible for fxlib and gint fighting over who owns the display.
- You can't call GetKey(), but hopefully you can resort to getkey() by changing little code.
- Keyboard and timers are a definite no. Using related syscalls while gint runs is sort of "undefined behavior".
- Overclock is safe but it will confuse gint's timer delays.
- RTC is generally safe.
When there are reentrency issues, such as when calling syscalls from interrupt handlers (eg. timers), you need to take note that:
- Drawing is still safe, but you may confuse the screen if dupdate() interrupts Bdisp_PutDisp_DD() (or the other way round)
- RTC is still ok, but there might be reentrancy issues. Not mixing syscalls is best.
Bdisp_AllClr_VRAM() clears the VRAM and leaves the screen RAM unchanged.
Bdisp_AllClr_DD() clears the screen but leaves the VRAM unchanged.
Bdisp_AllClr_DDVRAM() clears both.
Bdisp_PutDisp_DD() copies the VRAM to the screen RAM.
Bdisp_PutDisp_VRAM() does not exist.
Bdisp_PutDisp_DDVRAM() does not exist.
Citer : Posté le 03/09/2018 17:25 | #
If it's not too urgent for you, I'll gladly do the continuous integration for you repository.
That would be great! Keep me updated
About the DD/VRAM explanation:
Thanks, I think I will support the DD variant. Tbh I do not really know what VRAM is used for and at this point I only want to provide minimal functionality to be able to get a example project up and running.
Until now I did not consider reentrancy at all. I assume multi-threading is not a thing for the Casio. Interrupts however have to be supported at a later stage of libc. I will look at this later
Citer : Posté le 03/09/2018 17:33 | #
That sounds good but you realize Print() draws to the VRAM, right?
Working with gint, I can tell that interrupts alone are very tricky. I think this is the simplest form of parallel execution you can get because userspace- and interrupting- code are never executed simultaneously, but it's already very tricky.
A simple example is for the sleep instruction that waits for an interrupt. Actually I think I can show that there's no way of waiting for an interrupt with sleep without taking the slim risk that the interrupt has already occured, and thus freezing the program. I have written about it here but it's not that important.
Citer : Posté le 03/09/2018 17:43 | #
That sounds good but you realize Print() draws to the VRAM, right?
Not at all.
(Well I guess now I do... I suppose I'll support Bdisp_PutDisp_VRAM(), then
I have worked with Interrupts on other µCs, so I kinda know the problems (race conditions, deadlocks). I'll get to this when I start to think about timing. At the moment I do not even know if newlib supports multi-threading
Citer : Posté le 03/09/2018 17:48 | #
(Well I guess now I do... I suppose I'll support Bdisp_PutDisp_VRAM(), then
Alright, but better use Bdisp_AllClr_VRAM() and Bdisp_PutDisp_DD() because again this one does not exist!
I think this is even less than a priority? Close to no one here is even able to write a proper multi-threaded application that does not deadlock.
Citer : Posté le 03/09/2018 17:54 | #
Alright, but better use Bdisp_AllClr_VRAM() and Bdisp_PutDisp_DD() because again this one does not exist!
Well... yeah, that's what I wanted to say
I think this is even less than a priority? Close to no one here is even able to write a proper multi-threaded application that does not deadlock.
Exactly
First up, I want to support scanf()
Ajouté le 03/09/2018 à 18:48 :
@Lephé
I read your thread about interrupts and this is not remotely safe
First up: multi-threading and main/interrupts pretty similar when it comes to the problems that arise. One thread is intercepted by another. That's only a problem if these threads share the same resources (in your example the variable timer_has_stopped).
A few issues came to my mind when I read it:
1) The timers are fed by the (potentially predivided) clock signal, so they are (from a hardware point of view) synchronous, right?
2) Separating the clock signal from the processor seems like the least practicable solution to sleeping (unless you *really* wanted to save energy)
3) There has to be a possibility to disable the interrupt (either stopping the timer or disabling the interrupt while the interrupt flag is set).
And 3) is exactly the practical solution to the problem. To be more precise, preventing the interrupt to intercept *condition* and *commands* of an if is the **only** solution to this problem. There are various "advanced" techniques for multiple threads (e.g. having a mutex).
Citer : Posté le 03/09/2018 19:22 | #
Yes, I just re-read the article and realized that "asynchronous" was a wrong choice of words: what I meant to say is that interrupts can occur at any moment, mainly when you do not expect them.
If you halt the CPU clock then you're not going to wake up at all. If configured properly (such as the default), an accepted interrupt will wake up the processor from a standby mode. But I don't think there is any hardware operation that automatically supplies the clock to the CPU? I guess cutting off the CPU clock source (is this even possible?) is just electronic suicidal.
Be assured that I double-checked this solution first. But currently I don't think it will work because of the nature of sleep.
If you don't enable the interrupt before going to sleep, you're in for an eternal slumber.
If you enable the interrupt before going to sleep, then you're leaving an opening for an interrupt to occur between the stc instruction that sets the BL bit of the status register and the sleep instruction itself. So you're stuck.
What works in the case of usual conditions cannot apply here because we can't execute the guarded command while interrupts are disabled.
Citer : Posté le 04/09/2018 00:15 | #
Ok, so what I formulated in 2) was not quite correct. What I wanted to say is:
Halting the processor is very suboptimal since it produces all the problems you described. It's only of use if you really need to save energy.
There are some justified applications such as:
setTimerTo(5000); // 5000 ms
sleep();
However, usually, if you need to work with interrupts, you would choose busy waiting:
while (1) {
setTimerTo(1000);
// do sth
while (!timeIsUp); // busy waiting (instead of sleep)
// do sth else
}