Complete C standard library
Posté le 19/08/2018 19:31
Motivation
Until now there was no complete C standard library (aka
libc) available for the Casio calculators. Although some parts of this library have been provided by fxlib and
gint, there was no libc implementation complying with the standard and compatible with the sh3eb architecture ready to use.
To change that, I decided to port
newlib to the Casio CPU. Newlib is an alternative libc implementation intended for use on embedded systems.
Alpha
Follow this link and click the download button in the top right corner:
>>> v1.1 <<<
Instructions on how to install newlib alongside with gcc (big shout-out to Lephé):
Compiler sous Linux avec GCC
Features for Casio fx9860g calculators:
* C standard library
libc
→
printf implementation to print text to the display
→ Dynamic allocation of memory using
malloc and
free
→ Memory manipulation using
memcpy,
memcmp,
memset etc.
→ String manipulation using
strcpy,
strcmp,
strstr,
strtok
→ ...
* Math library
libm
→ Floating point arithmetics
→ ...
* Automatic library and include path recognition after installation
* Basic Casio features:
→ implementation of
GetKey,
Bdisp_AllClr_DDVRAM,
Bdisp_PutDisp_DD,
Print and
locate without fxlib (but you can use it if you want)
Code
To contribute or get all those bleeding edge features, see the code including all further information:
libc (my GitLab repository)
The project you find in my repository is a fork of
the official newlib repository. To make it easier for everyone to follow, I try to keep a clean
git history. That means that all my changes are located on a dedicated branch with meaningful commits.
I also try to keep the changes to the upstream library minimal. That increases maintainability a lot.
Contributing
If you have a ideas, feature request, found a bug or simply want to contribute, just send me a message and you're in! You can also create
Issues and
Merge Requests directly in the repository. As in every OpenSource project: merge requests welcome!
Citer : Posté le 06/09/2018 15:24 | #
Okok, using the dedicated function works and is actually much cleaner. I'll use GetKeyWait()
Thanks for being the reasonable one here, Lephé.
@Zezombye Your solution is still clever though
Ajouté le 07/09/2018 à 14:31 :
I'm at implementing scanf when suddenly... A very strange problem appears!
I use logic... It's not very effective!
Here is the error I get. Note that I get this error instantly, i.e. without pressing any key before.
System ERROR
REBOOT :[EXIT]
INITIALIZE:[EXE]
ADDRESS(R)
TARGET=0030080E
PC =003002E4
Ok, I probably dereference an invalid pointer (which points to ROM)... should be easy to find.
Since I did not press any key before the error appeared, the error must be caused by something before the first call of GetKey(), non?
Let's look at my main:
int main(void) {
unsigned int key = 0;
char out[20] = "Test";
while (1) {
Bdisp_AllClr_DDVRAM();
locate(1, 4);
Print("Out: ");
Bdisp_PutDisp_DD();
GetKey(&key);
_read(0, out, 5); // <-- when I comment this line out, it works!
}
return 0;
}
Ok, first I thought, the new implementation of _read() cannot be the problem. But when I comment it out, it works o.O.
However, _read() only calls _console_read() which instantly returns.
I think the problem is that the linker links much more code if I call _read(). As a consequence, one of the previous calls (locate, Print etc.) fails. That's just a guess, however, and I have not the slightest idea on how to solve such a problem
Citer : Posté le 07/09/2018 15:06 | #
GetKeyWait doesn't work for me (which is why I used putkey instead). How did you implement the syscall?
Ajouté le 07/09/2018 à 15:09 :
What happens if you comment the call to _console_read?
Ecrivez vos programmes basic sur PC avec BIDE
Citer : Posté le 07/09/2018 15:11 | #
typedef int (* _GetKeyWait_type) (int* column, int* row, int type_of_waiting, int timeout_period, int menu, unsigned short *keycode);
const unsigned int _GetKeyWait_address[] = {0xD201D002, 0x422B0009, 0x80010070, 0x247};
const _GetKeyWait_type _GetKeyWait_ptr = (_GetKeyWait_type) _GetKeyWait_address;
Then you can directly call
int row, col;
_GetKeyWait_ptr(&col, &row, 0, 0, 0, 0);
Or if you need a function (instead of a function pointer), you can define one.
int GetKeyWait(int*column, int*row, int type_of_waiting, int timeout_period, int menu, unsigned short *keycode) {
return _GetKeyWait_ptr(column, row, type_of_waiting, timeout_period, menu, keycode);
}
Ajouté le 07/09/2018 à 15:19 :
_read() simply calls _console_read():
_read (int file,
char *ptr,
int len)
{
_console_read(&_console_ctx, file, ptr, len);
return len;
}
_console_read() returns instantly:
int _console_read(_console *ctx, int file, char *ptr, int len) {
char mode = _CONSOLE_MODE_NORMAL;
int col, row;
int i = 0;
return len;
// ...
}
Citer : Posté le 07/09/2018 15:53 | #
I use logic... It's not very effective!
Let's use intuition then!
Your system error occurs with PC=003002E4. Note that, because of the code reading optimizations, the offending instruction can also be at 003002E2. Where in the code is that?
The error is said to be ADDRESS(R), which usually means unaligned access. As the offending pointer is 0030080E, which is 2-aligned, the problem can only arise for an access with a longword pointer.
By adding or removing code in your program, you could break alignment of some variables. So as a first guess, do you have any assembler-declared symbol which is not protected by a .align directive?
Citer : Posté le 07/09/2018 16:57 | #
sh3eb-elf-nm main.elf says:
003002ac t GLibAddinAplExecutionCheck
0030032c t _GLibAddinAplExecutionCheck
003002e4 t _Hmem_SetMMU
003002a8 t Hmem_SetMMU
So I guess the suspect is _Hmem_SetMMU. A quick grep shows crt0.s:
! set up TLB
mov.l Hmem_SetMMU, r3
mov.l address_one, r4 ! 0x8102000
mov.l address_two, r5 ! 0x8801E000
jsr @r3 ! _Hmem_SetMMU
mov #108, r6
.align 4
address_two: .long 0x8801E000
address_one: .long 0x8102000
Hmem_SetMMU: .long _Hmem_SetMMU
_Hmem_SetMMU:
mov.l sc_addr, r2
mov.l 1f, r0
jmp @r2
nop
1: .long 0x3FA
To be honest, my assembler skills do not exceed the little program I once wrote at college
Ajouté le 07/09/2018 à 17:06 :
Ok so I suspected sc_addr to be misaligned since there is no .align right before, but adding one does not solve my problem :/
.align 4
sc_addr: .long 0x80010070
Citer : Posté le 07/09/2018 17:07 | #
You don't need any instruction set definition to know the size of the code here because every instruction takes 2 bytes. If you're asking about the meaning of the code, then roughly your first extract calls Hmem_SetMMU(0x08102000, 0x8801e000, 108) which probably sets up the static RAM area. Then the third extract just forwards the arguments to system call 0x3fa which is the actual implementation of Hmem_setMMU().
Alright, now 003002e4 is the location of the _Hmem_SetMMU symbol, which is also the address of that mov.l sc_addr, r2 instruction. And it indeed does a memory fetch in read mode. sc_addr is 4-aligned, though. Can you show me what is at address 0030080e? Also can you determine the location of sc_addr?
Citer : Posté le 07/09/2018 17:15 | #
Location of sc_addr
003006a2 R _romdata
00300334 t sc_addr
003003ac T _wait_ms
Hmm, my list generated by nm starts at 003002a4. Is there a way to show 0030080E?
Citer : Posté le 07/09/2018 17:52 | #
Oh yes, disassemble it. For symbols, I suggest using objdump -t which is slightly more informative than nm. Otherwise, disassembling is the best option since you can also look for uses of the address in the code.
Citer : Posté le 07/09/2018 19:48 | #
objdump -t and nm do not show 0030080E. However, if I disassemble the .elf file with objdump -S, I find one (and only one) place where this address appears:
00300218 <L_check_bss>:
300218: 34 52 cmp/hs r5,r4
30021a: 8b fb bf 300214 <L_zero_bss>
30021c: d4 2c mov.l 3002d0 <bdata>,r4 ! 8100020 <__impure_ptr>
30021e: d5 2b mov.l 3002cc <edata>,r5 ! 8100448 <_edata>
300220: d6 2c mov.l 3002d4 <romdata>,r6 ! 30080e <_romdata>
300222: a0 03 bra 30022c <L_check_data>
300224: 00 09 nop
L_check_bss is also part of the startup code:
initialize:
sts.l pr, @-r15
! set up TLB
mov.l Hmem_SetMMU, r3
mov.l address_one, r4 ! 0x8102000
mov.l address_two, r5 ! 0x8801E000
jsr @r3 ! _Hmem_SetMMU
mov #108, r6
! clear the BSS
mov.l bbss, r4 ! start
mov.l ebss, r5 ! end
bra L_check_bss
mov #0, r6
L_zero_bss:
mov.l r6, @r4 ! zero and advance
add #4, r4
L_check_bss:
cmp/hs r5, r4
bf L_zero_bss
Citer : Posté le 07/09/2018 19:58 | #
Aha so it's indeed a misalignment issue! You have _romdata=0030080e. The loading procedure in crt0.s copies various data from this address to RAM, this is apparent in the linker script. _romdata is where the contents of section .data are stored in the ROM image.
For obvious optimization reasons, the copy is longword-wise, so sections such as .data must be 4-aligned and have a size multiple of 4.
I have met exactly the same problems during the development phase of gint's linker scripts, and it took a while before these scripts became mature and robust. In my case, the section of even 16-aligned and with a size multiple of 16 because the copy is much faster this way. Here is how I'd do it with a size of 4:
*(...)
. = ALIGN(4);
} > ram AT> rom
In my case the AT> replaces the AT() mechanism. You can safely ignore it. The important thing is that :
- The first ALIGN(4) forces the alignment of the whole section in target memory (RAM)
- The second ALIGN(4) forces the alignment of all sections contributed from files in target memory (RAM)
- The last ALIGN(4) forces the size of the section.
Why the second one? Because if you forget alignment clauses the assembler will assume that the file is 4-aligned. This is more robust; and I have a few cases in mind where it is actually required.
Now you need something else if you don't use the AT> mechanism; make sure you have _romdata = ALIGN(4) instead of _romdata = . at the end of the .rodata section. This is probably what it causing the bug here.
Citer : Posté le 10/09/2018 22:23 | #
Great observation!
While on the right track, you probably actually meant something like
.rodata ALIGN(4) : ALIGN(4) {
*(.rodata)
*(.rodata.str1.4)
. = ALIGN(4); /* this aligns _romdata which caused the problem here */
_romdata = . ; /* symbol for initialization data */
} > rom
I totally wouldn't have found that without you, thanks a lot!
PS.: I also found a great summary of SH assembler instructions
Citer : Posté le 11/09/2018 16:58 | #
Oh yes, you might want to align . as well as _romdata, well spotted.
This kind of bug is typical of early linker scripts ; remember to use the -M linker flag and check all relevant addresses for alignment if you have a crash.
Citer : Posté le 12/09/2018 23:47 | #
I just noticed two little facts that increase the amount of work to be done:
1) Since scanf() needs to echo the entered characters, I have to use Bdisp_PutDisp_DD() internally which kinda interferes with the fxlib stuff... Is there any possibility to draw onto the display without using VRAM, locate and Print? (I fear not via syscalls, probably not without major effort which is not what I want... that's what gint is for
2) Maybe even worse:
a) The ASCII standard describes 128 characters. All other characters have to be expressed by Unicode chars. That includes × (\u00D7), ÷ (\u00F7), π (\u03C0), é (\u00E9), ä (\u00E4) and so on and so on...
(I know there is extended ASCII but it is non-standard and covers only some of the above-mentioned characters. I'd like to avoid it...)
b) Printing these characters requires the translation of these characters into the Casio-specific character set. Now there are both single-byte chars (ASCII) and multiple-byte chars (Unicode) which have to be translated into single-byte chars or multiple-byte chars. Converting a whole string requires a whole lot of effort (both performing-wise and code-wise) and the lookup-table necessary will be a pain in the a** to have in memory (and even more to create)
Well, to be honest I do think that performing keyboard and screen input/output in the standard library is a bit harsh.
Well... should have listened earlier.
I guess I will implement a rudimentary scanf() first and welcome any contribution by other enthusiasts.
Citer : Posté le 13/09/2018 10:19 | #
No, there's nothing in fxlib for that. I actually think that doing the I/O in the libc is not optimal. Did you have a look at Dark Storm's EasyInput library? I'd have expected something along these lines, possibly with customizable functions if the user wants to change the way I/O is done. I fear elaborate applications will not be satisfied with the 5×7 font in a simple terminal...
Not, CASIO has a special, Unicode-incompatible, 2-byte fixed encoding called FONTCHARACTER. Cakeisalie5 did a lot of research of it and compiled it into a YAML file called the refc, which is designed to be used programmatically.
I don't think the keyboard inputs Unicode, in fact GetKey() returns Basic opcodes and key ids that correspond directly to FONTCHARACTER values. You need only inject the value returned by GetKey() in your string and will work out of the box. Of course in the end scanf() will parse a FONTCHARACTER-based string, but this is nowhere a problem.
Citer : Posté le 13/09/2018 11:19 | #
I fear elaborate applications will not be satisfied with the 5×7 font in a simple terminal...
I totally agree. But as you noted as well, the stdlib usually is not responsible for I/O. What I want to provide is basic I/O support. To put it simple: a new user should be able to call printf(), see something and be happy.
The standard C library does simply not provide a more sophisticated interface for changing fonts etc. To be honest, IMHO that should not be the scope of newlib.
Let me share my vision with you:
1) The stdlib implements the POSIX read() and write() which rely on locate() and Print(). It should work out of the box.
What I do not want to do is introducing non-standard interfaces. The only exception to this is locate() since it is simply needed.
2) Everyone who needs more sophisticated functionality should use gint. The stdlib relies on the kernel and nothing else. That's why the only clean solution to make the stdlib printf/scanf more elaborate is that gint provides the POSIX read() and write() syscalls IMO. With gint, you can introduce custom functions for setting the font, cursor position, contrast and so on.
What do you think?
In fact GetKey() returns Basic opcodes and key ids that correspond directly to FONTCHARACTER values
Oh, I did not know that.
Yes, Casio provides their own encoding. I first wanted to stick with Unicode, but that would not be worth the effort. I'll look into this some more, but I might stick to single-byte chars. Not sure about that yet. Unfortunately I cannot download Dark Storm's EasyInput library (broken link). I'll contact him.
Citer : Posté le 13/09/2018 14:13 | #
Which is entirely fair and sound.
What I do not want to do is introducing non-standard interfaces. The only exception to this is locate() since it is simply needed.
Ok, I have a suggestion for it. Since we're diving into read() and write(), and that I'm starting to consider how I implement them in gint, I suggest that we start with a few file descriptors. We could have an array of 10 entries like this one:
ssize_t (*read)(int fd, void *buf, size_t count);
ssize_t (*write)(int fd, const void *buf, size_t count);
int (*close)(int fd);
/* ... ? */
void *data;
}
Then you can define a first stream which does I/O using Print() and gint can do the same using its own function. This way we can open streams with the standard library and use them inside gint in a sound way.
By the way you could avoid locate() by using PrintXY(int, int, char *, int) but that's typically a non-standard signature.
gint indeed provides such functions, so that's fine by me.
Just replace the IP with git.planet-casio.com in the link, which dates back to the time when we installed the Gitlab instance and were still running it on the default port 80.
Citer : Posté le 13/09/2018 16:34 | #
By the way you could avoid locate() by using PrintXY(int, int, char *, int) but that's typically a non-standard signature.
Both are non-standard, I think I'll stick to locate().
Ok, I have a suggestion for it. Since we're diving into read() and write(), and that I'm starting to consider how I implement them in gint, I suggest that we start with a few file descriptors. We could have an array of 10 entries [...]
I think we are not talking about the same thing when we say "file descriptor", so let me clarify: a file descriptor is a handle (aka an integer number) of a file/stream/socket/...
So stdout = 1 and stderr = 2 are examples for file descriptors. Functions like read(), write(), open() and close() work with these file descriptors and are called "syscalls". (It's unfortunate that we also have syscalls like Print() which I like to call "Casio syscalls").
First and foremost, gint needs to provide syscalls (as you said). There are a ton of POSIX syscalls which do not make sense for gint, so let's concentrate on the syscalls newlib (as an embedded libc) uses.
The syscalls read() and write() are a must IMO. If you want to support files, you need open(), close(), fstat(), lseek() and others. These functions probably make also sense when you are not implementing a file system but e.g. an IO stream for the serial interface.
As for the code:
I would not put function pointers in a struct (because there is no benefit IMO). Instead I would simply provide a syscalls.h and a syscalls.c which provide declarations and definitions for said functions.
This will probably be more work than it sounds, but I'd gladly help you.
Citer : Posté le 13/09/2018 18:44 | #
Yes, you are 100% right, but then if you have file descriptors you need to have a file descriptor table somewhere in the kernel, so I jumped straight to it. =) File descriptors would of course be indices in this array.
Yes, although I'll be implementing file I/O on top of the Bfile syscalls because it's the only way to not screw up the filesystem. Which has weird properties, meaning that not all Unix programs will be portable right away.
But then write() needs to behave differently for files, newlib's simple terminal, and (say) gint's customizable terminal. How du you achieve this kind of polymorphism? We could predefine all stream types and hardcode the function names associated with each type, which is not a problem for me; my point was that each file descriptor structure needs to have some "type" information.
Thank you! I'll look at it when the platform-specific aspects of gint are rolling. Hopefully I'm on the way for fixing the keyboard difficulties I've been facing lately, so I can resume developing at a reasonable speed.
Citer : Posté le 16/09/2018 20:27 | #
I'm not entirely sure yet what you have in mind with the file descriptors. Do you suggest multiple structs like fd_t with the same signature but different pointers? That sounds like a sane idea.
However, I was confused by the first parameter of your function pointers:
|
V
ssize_t (*read)(int fd, void *buf, size_t count);
Here, fd is the file descriptor which will tell the function how to behave. That means there is only one function with a simpe switch (fd) {case stdin: [...]}.
Now while your solution looks more elegant on first glance, IMO it has a fatal flaw: you want to use file descriptors as indices for an array of function pointers. That means there will one set of functions per file descriptor (which means one set of functions per potential file). You probably had stdout etc. in mind, but if you consider file descriptors to be handles to files, I think your solution won't work anymore.
Yes, although I'll be implementing file I/O on top of the Bfile syscalls because it's the only way to not screw up the filesystem. Which has weird properties, meaning that not all Unix programs will be portable right away.
Tell me more...
Hopefully I'm on the way for fixing the keyboard difficulties I've been facing lately, so I can resume developing at a reasonable speed.
I might look at it tomorrow. Keep me updated
Citer : Posté le 16/09/2018 20:42 | #
Yes, exactly. This is the kind of thing you would get with fopencookie(3).
Not quite. The idea is that you can have a write_terminal() function that uses locate() but takes care that stdin and stderr do not meddle with each other, and a write_file() function that acts pretty much the same for all fds, relying on the internal Bfile handle stored in the data member to perform the I/O.
Well, to be precise I'd like to use them as indices in an array of structures, where the structure either contains the function pointers or a "device type number" that tells us which function to call.
I'm not sure what you mean exacly, it is entirely possible to put the address of the same write_file() function in all file descriptors that point to the storage memory. I should mention that I had standard streams, files and serial communication in mind from the beginning, but I may not have been quite clear.
- All files are created with a fixed size.
- Any directory not at the root of the filesystem will contain itself.
- Writing an odd number of bytes with Bfile_WriteFile() will cause Bfile calls to randomly raise hell.
- Writing with Bfile_WriteFile() can turn 1's into 0's but not the opposite.
- Failing to close files or search handles will break the filesystem.
- Optimizing the filesystem solves USB communication issues.
I think I just solved it yesterday in the evening! You've got to read KEYSC data fast or strange things will start happening. My driver now behaves 100% correctly.
Citer : Posté le 17/09/2018 17:25 | #
I still seem to miss how you would implement this. To be honest, my understanding of the linux kernel is very basic. What confuses me the most is the following statement:
Well, to be precise I'd like to use them as indices in an array of structures, where the structure either contains the function pointers or a "device type number" that tells us which function to call.
What do you mean with "device type number"?
I understand that you want to implement an array of structs which contain the function pointers to the syscalls you need for each file descriptor:
/* Your idea of implementing it */
#include <stddef.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
// Let's assume that stdout is 0 for simplicity's sake
#define stdout 0
typedef struct {
int (*open)(const char *pathname, int flags);
ssize_t (*write)(int fd, const void *buf, size_t count);
/* all other syscalls... */
void *data; // <-- pointer to custom data
} fd_t;
int open_terminal(const char *pathname, int flags) {
// here sime init implementation if necessary
return 0;
}
ssize_t write_terminal(int fd, const void *buf, size_t count) {
// implementation using locate()
return 0;
}
// syscalls for terminal
fd_t display_syscalls = {
.open = &open_terminal,
.write = &write_terminal,
// ...
.data = NULL
};
// all syscalls: index = file descriptor
// in reality this would have a dynamic length!
fd_t all_syscalls[7];
void main(void) {
// set it up
all_syscalls[stdout] = display_syscalls;
// let's use it!
int fd_terminal = all_syscalls[stdout].open("/path/to/file", O_RDWR);
char buf[] = "Hello World";
all_syscalls[stdout].write(fd_terminal, buf, sizeof(buf));
}
1) It obviously will not work this way, because the open() syscall returns the file descriptor you need to know to access the array of fd_t structs in the first place.
fopencookie() does not work that way since instead of open(), fopencookie() is called.
2) After being opened via open(), every file will be associated with a unique file descriptor. This means that you would have to dynamically add an element for each file to your variable length array. Also, for all opened files, the function pointers to the syscalls will be identical (which would be redundant).
3) I do not see how the custom data fd_t.data can be of use here, since it cannot be passed to the syscalls. Their signature cannot be extended.
4) Your implementation contains redundant information: the file descriptor. What determines which syscall is called is the index = file descriptor. Additionally you pass the file descriptor to the syscalls (e.g. read()).
I suggest an implementation somewhat like this. (Again, I'm not very firm with the linux kernel and I suppose opening the terminal this way is not what you would normally do. It's just an inspiration how it could look like for both files and strams.)
/* My idea of implementing it */
#include <stddef.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
// Let's assume that stdout is 0 for simplicity's sake
#define stdout 0
int open_terminal(const char *pathname, int flags) {
if (strcmp(pathname, "/dev/tty") == 0) {
// some init
return stdout;
}
// implementation for all other files
return -1;
}
ssize_t write(int fd, const void *buf, size_t count) {
switch (fd) {
case stdout:
// implementation using locate()
return 0;
// implementation for all other file descriptors
}
return -1;
}
void main(void) {
// let's use it!
int fd_terminal = open("/dev/tty", O_RDWR);
char buf[] = "Hello World";
write(fd_terminal, buf, sizeof(buf));
}