IRIX port discussion

[rickyboy]

OK.  Taking a leaf from Peter's book, I open this discussion in the "Lisp in general" forum.



OK, first problem.  I tried a `make solaris' and, lo and behold, the call to `mmap()' in `nl-filesys.c' fails because IRIX doesn't have a `MAP_ANON' flag in its `<sys/mman.h>'.



A `man mmap' says that these are the flags it has:

Code Select Expand

MAP_SHARED       Share changes
MAP_PRIVATE      Changes are private
MAP_FIXED        Interpret addr exactly
MAP_AUTOGROW     Implicitly grow object
MAP_LOCAL        Do not share with share group
MAP_AUTORESRV    Reserve logical swap on demand
MAP_SGI_ANYADDR  Use reserved area for mappings
Any suggestions?  (I don't know how important this flag is.)  I could have deleted the flag and pressed on, but I didn't want to "spin my wheels" on something I am sort of unfamiliar with, and I assumed you had wanted me to proceed deliberately so that we could have a solid port.  Your thoughts?



Regards,  --Rick

[Lutz]

MAP_ANON is used to tell that memory is mapped not to a file but to a special shared memory region. In this case the file descriptor is put to -1 and the file-offset is not used and can be put to zero.



Perhaps you can post the Irix man page for mmap, then we can figure it out.



I could imagine that MAP_SGI_ANYADDR is equivalent, but I am not sure.



Lutz

[rickyboy]

Thanks Lutz for willing to do this.  Here they are (you might have to reduce the text size one notch in your browser to read this, due to the fixed width of the `code' box.):

Code Select Expand

mmap(2)                                                 mmap(2)



NAME
     mmap, mmap64 - map pages of memory

SYNOPSIS
     #include <sys/types.h>
     #include <sys/mman.h>

     void *mmap(void *addr, size_t len, int prot, int flags, int
          fd, off_t off);

     void *mmap64(void *addr, size_t len, int prot, int flags,
          int fd, off64_t off);

DESCRIPTION
     The functions mmap and mmap64 establish a mapping between a
     process's address space and a virtual memory object.  The
     format of the call is as follows:

          pa = mmap(addr, len, prot, flags, fd, off);

     mmap establishes a mapping between the process's address
     space at an address pa for len bytes to the memory object
     represented by the file descriptor fd at offset off for len
     bytes.  The value of pa is an implementation-dependent
     function of the parameter addr and values of flags, further
     described below.  A successful mmap call returns pa as its
     result.  The address ranges covered by [pa, pa + len) and
     [off, off + len) must be legitimate for the possible (not
     necessarily current) address space of a process and the
     object in question, respectively.

     The only difference between mmap and mmap64 is that in
     mmap64 the off parameter is 64 bits long, so the file
     offset can be greater than 2 gigabytes.  This is useful for
     certain filesystem types that support such file offsets.

     The mapping established by mmap replaces any previous
     mappings for the process's pages in the range [pa, pa +
     len).

     The parameter prot determines whether read (load), write
     (store), execute, or some combination of accesses are
     permitted to the pages being mapped.  The protection
     options are defined in <sys/mman.h> as:

          PROT_READ                Page can be read.
          PROT_WRITE               Page can be written.
          PROT_EXEC                Page can be executed.
          PROT_NONE                Page can not be accessed.

     Not all implementations literally provide all possible
     combinations.  PROT_WRITE is often implemented as
     PROT_READ|PROT_WRITE and PROT_EXEC as PROT_READ|PROT_EXEC.
     This is true for all SGI implementations.  In particular,
     MIPS processors do not support a separate execute
     permission.  Any page that can be read can be executed
     from, even if PROT_EXEC is not specified.  Instead, the
     operating system uses PROT_EXEC as a flag to indicate it
     may need to perform certain platform dependent functions
     (such as cache flushing) that may be needed to properly
     execute instructions from the associated page.  See
     mprotect(2) for further details.  However, no
     implementation will permit a store to succeed where
     PROT_WRITE has not been set.  The behavior of PROT_WRITE
     can be influenced by setting MAP_PRIVATE in the flags
     parameter, described below.

     The parameter flags provides other information about the
     handling of the mapped pages.  The options are defined in
     <sys/mman.h> as:

          MAP_SHARED        Share changes
          MAP_PRIVATE       Changes are private
          MAP_FIXED         Interpret addr exactly
          MAP_AUTOGROW      Implicitly grow object
          MAP_LOCAL         Do not share with share group
          MAP_AUTORESRV     Reserve logical swap on demand
          MAP_SGI_ANYADDR   Use reserved area for mappings

     MAP_SHARED and MAP_PRIVATE describe the disposition of
     store references to the memory object.  If MAP_SHARED is
     specified, store references will change the memory object.
     If MAP_PRIVATE is specified, the initial store reference
     will create a private copy of the memory object page and
     redirect the mapping to the copy.  Either MAP_SHARED or
     MAP_PRIVATE must be specified, but not both.  The mapping
     type is retained across a fork(2).

     When MAP_SHARED is specified, and initially in all pages
     when MAP_PRIVATE is specified, the contents of the mapped
     segment change to reflect changes in the underlying memory
     object. Changes can be caused by other processes that map
     the same object with MAP_SHARED, or by processes using
     write(2) or ftruncate(2). If the file is shortened, an
     attempt to access a page of memory that is mapped to a part
     of the file that no longer exists will cause a Bus Error
     (SIGBUS) signal.

     When MAP_PRIVATE is used, a private copy of a page is
     created only when the process stores into the page. This
     prevents changes from being seen by other processes that
     map the same object, and prevents further changes made by
     other processes from being visible. However, changes that
     occur before the page is stored into are visible.

     To protect the contents of a mapped file from changes or
     truncation you can either use chmod(2) and lockf(3) to
     enforce a mandatory file lock, or you can specify
     MAP_PRIVATE and store into every page of the segment in
     order to create a complete private copy of the data.

     MAP_FIXED informs the system that the value of pa must be
     addr, exactly.  When MAP_FIXED is not set, the system uses
     addr in an implementation- specific manner to arrive at pa.
     The pa so chosen will be an area of the address space which
     the system deems suitable for a mapping of len bytes to the
     specified object.

     All implementations interpret an addr value of zero as
     granting the system complete freedom in selecting pa,
     subject to constraints described below.  A non-zero value
     of addr is taken to be a suggestion of a process address
     near which the mapping should be placed.  When the system
     selects a value for pa, it will never place a mapping at
     address 0, nor will it replace any extant mapping, and it
     will attempt to map away from areas considered part of the
     potential data or stack segments.

     The MAP_FIXED directive should be used with caution.  When
     MAP_FIXED is set, any mappings (including text, heap, data,
     and stack) in the range [addr, addr + len) will be replaced
     with the new mapping.

     To ensure best system hardware cache behavior, objects
     should be mapped such that the low sixteen bits of the file
     offset of the object match the low bits of the mapped
     address.

     The address range from 0x30000000 to 0x40000000 is normally
     reserved for MAP_FIXED mappings except when MAP_SGI_ANYADDR
     is specified or when the SGI_UNSUPPORTED_MAP_RESERVED_RANGE
     option is enabled with syssgi(2).  Note that this space may
     not be useful for programs which require a very large heap,
     since by default program heaps start near 0x10000000 and
     grow toward higher addresses.  This range will never be
     used when zero is passed as the value for addr unless
     either MAP_SGI_ANYADDR or
     SGI_UNSUPPORTED_MAP_RESERVED_RANGE is used.  See
     sgi_use_anyaddr(1) for further details.

     If MAP_AUTOGROW is specified with MAP_SHARED, the mapped
     object will be implicitly grown when referenced by a store
     operation to a page which maps beyond the current end of
     the object; the object will be grown and zero-filled to
     fulfill the mapping up to the next page boundary or to the
     end of the mapping, whichever is less.  If used with
     MAP_PRIVATE, MAP_AUTOGROW allocates private zero-filled
     pages for references beyond the end of the object, but does
     not grow the object.

     MAP_AUTOGROW requires that the object is mapped with
     PROT_WRITE permission.  Load references to mapped pages
     following the end of a object will result in the delivery
     of a SIGSEGV signal, as will various filesystem conditions
     on stores.  Whenever a SIGSEGV signal is delivered, the
     second argument to the signal handler contains a value that
     indicates the reason for the delivery of the signal; these
     values are defined in /usr/include/sys/errno.h.

     When MAP_AUTOGROW is specified, len determines the maximum
     size of the memory map, as opposed to the initial size.
     The size of the map is extended automatically by storing to
     any location beyond the current working size, up to the
     size limited by len. The mapped file will also be extended,
     even if it is closed. Any read or write beyond the end of
     the len specified area results in delivery of a SIGSEGV
     signal.  Therefore the use of MAP_AUTOGROW must anticipate
     the maximum len as to prevent failed accesses beyond the
     end of the mmapped area.

     If MAP_LOCAL is used and the process does an sproc(2) each
     process will receive a private copy of the object's
     mapping.  All subsequent load reference of objects mapped
     MAP_PRIVATE will cause private copies of the object to be
     created.  In addition, the share group processes will be
     able to independently unmap the object from their address
     spaces.

     The system reserves len bytes of logical swap space when
     MAP_PRIVATE mappings of regular files are created, as well
     as for all mappings of /dev/zero.  (See swap(1m) for a
     discussion of logical swap space.)  If insufficient logical
     swap space is available, mmap fails with EAGAIN.  The
     MAP_AUTORESRV flag causes logical swap space to be
     automatically reserved as each page is first referenced
     with a store operation instead of when the mapping is
     created.  When this flag is used, no logical swap space is
     reserved when the mapping is created.  Therefore, the
     system cannot guarantee that space will be available when
     needed.  If all the logical swap space has been taken by
     other processes when a page in a MAP_AUTORESRV mapping is
     first stored to, then the process will be sent SIGBUS.

     The parameter off is constrained to be aligned and sized
     according to the value returned by getpagesize(2) or
     sysconf(_SC_PAGESIZE).  When MAP_FIXED is specified, the
     parameter addr as well as off must be aligned according to
     the value returned by sysconf(_SC_MMAP_FIXED_ALIGNMENT).
     The system performs mapping operations over whole pages.
     Thus, while the parameter len need not meet a size or
     alignment constraint, the system will include, in any
     mapping operation, any partial page specified by the range
     [pa, pa + len).

     The system will always zero-fill any partial page at the
     end of an object.  Further, the system will never write out
     any modified portions of the last page of an object which
     are beyond its end.  References to whole pages following
     the end of an object will result in the delivery of a
     SIGBUS signal.  SIGBUS signals may also be delivered on
     various filesystem conditions, including quota exceeded
     errors, and for physical device errors (such as unreadable
     disk blocks).  The signal handler may examine the si_code
     and si_errno fields of the siginfo structure for
     information about the nature of the error.

RETURN VALUE
     On success, mmap returns the address at which the mapping
     was placed (pa).  On failure it returns MAP_FAILED and sets
     errno to indicate an error.

ERRORS
     Under the following conditions, mmap fails and sets errno
     to:

     EAGAIN The mapping could not be locked in memory.

     EAGAIN The amount of logical swap space required is
            temporarily unavailable.

     EBADF  fd is not open.

     EACCES fd is not open for read, regardless of the
            protection specified, or fd is not open for write
            and PROT_WRITE was specified for a MAP_SHARED type
            mapping.

     EACCES prot has extraneous bits set.

     EINVAL The arguments addr (if MAP_FIXED was specified) or
            off are not multiples of the value returned by
            sysconf(_SC_MMAP_FIXED_ALIGNMENT).

     EINVAL The arguments flags is invalid (neither MAP_PRIVATE
            or MAP_SHARED).

     EINVAL The argument addr specifies an unmappable address.

     ENXIO  Addresses in the range [off, off + len) are invalid
            for fd.

     ENXIO  The argument len has a value less than or equal to 0.

     ENODEV fd refers to an object for which mmap is
            meaningless, such as a terminal.

     ENOSYS fd refers to an object for which mmap is not
            permitted.

     ENOMEM zero was passed as the value of addr, and
            insufficient space was available in the standard
            address ranges.  This is primarily an issue for 32
            bit programs requesting 1GByte or more, because the
            range from 0x30000000 to 0x40000000 is reserved for
            MAP_FIXED as described above.

     ENOMEM MAP_FIXED was specified and the range [addr, addr +
            len) is invalid or exceeds that allowed for the
            address space of a process, or MAP_FIXED was not
            specified and there is insufficient room in the
            address space to effect the mapping.

     ENOMEM The calling process has the automatic memory locking
            of future mappings enabled [see mlockall(3C)] and
            there is insufficient physical memory available for
            the mapping.

NOTES
     mmap allows access to resources via address space
     manipulations instead of the read/write interface.  Once a
     file is mapped, all a process has to do to access it is use
     the data at the address to which the object was mapped.
     Consider the following pseudo-code:

          fd = open(...)
          lseek(fd, offset)
          read(fd, buf, len)
          /* use data in buf */

     Here is a rewrite using mmap:

          fd = open(...)
          address = mmap(NULL, len, (PROT_READ | PROT_WRITE),
                         MAP_PRIVATE, fd, offset)
          /* use data at address */

     Previous IRIX releases have only required MAP_FIXED address
     alignment to the system page size returned with
     getpagesize(2).  This is no longer true for the MIPS
     R4000PC, R4600 and R5000 processors.  Requests now must be
     aligned to the size returned by
     sysconf(_SC_MMAP_FIXED_ALIGNMENT).

SEE ALSO
     sgi_use_anyaddr(1), fcntl(2), fork(2), madvise(2),
     mprotect(2), msync(2), munmap(2), plock(2), sproc(2),
     sysconf(2), lockf(3C).
The discussion on MAP_SGI_ANYADDR says to "ee sgi_use_anyaddr", and here it is:

Code Select Expand

sgi_use_anyaddr(1)             sgi_use_anyaddr(1)



NAME
     sgi_use_anyaddr - disable reserved range for auto-placed
     mappings

SYNOPSIS
     sgi_use_anyaddr command [arguments]

DESCRIPTION
     sgi_use_anyaddr executes command with the reserved range of
     the virtual address space (from 0x30000000 to 0x40000000)
     disabled.  Normally, the kernel does not auto-place
     mappings made by mmap(2) and shmat(2) in the reserved
     range. By disabling the reserved range with
     sgi_use_anyaddr, the kernel is permitted to use any
     available virtual address for mapping requests made by
     command. sgi_use_anyaddr is implemented by invoking
     syssgi(2) with request parameter
     SGI_UNSUPPORTED_MAP_RESERVED_RANGE and then performing an
     exec of command. The effect of this option is the same as
     if MAP_SGI_ANYADDR and SHM_SGI_ANYADDR were added to all
     future calls to mmap(2) and shmat(2) respectively by
     command.  Note that this syssgi option is inherited across
     fork and exec, so command and all its descendants will be
     affected by this.

     As described in mmap(2), the kernel chooses a virtual
     address for a new mapping created by mmap(2) or shmat(2)
     when zero is passed as the attach address.  When the kernel
     auto-places a mapping in this manner, it is guaranteed that
     it will not place the mapping in the reserved range.  The
     only mappings that can be placed in the reserved range are
     those where the user explicitly specifies an attach address
     that is in the reserved range.

     The reserved range was originally created in the earlier
     90's by a MIPS/UNIX standards body at the request of
     application writers.  The application writers wanted a
     portion of the virtual address space where they could be
     assured they could place their own mappings via mmap(2) and
     shmat(2) without the risk of overlapping other mappings
     placed by the kernel.  A 256 MB region from 0x30000000 to
     0x40000000 was reserved for use by applications to use by
     specifying an explicit attach address with mmap(2) or
     shmat(2). It is guaranteed the kernel will never auto-place
     a mapping in this range when zero is passed as the attach
     address to mmap(2) and shmat(2) because this range has been
     reserved for applications that specify explicit attach
     addresses in this range.

     At the time the reserved range was created, applications
     were still relatively small by today's standards, so having
     this 256 MB hole in the middle of the 2 GB user virtual
     address space was not an issue.  Application program size
     has grown substantially over the years and some
     applications find they can no longer easily fit in the 2 GB
     address space available when running in 32-bit mode.
     Having the reserved range take away 256 MB makes it
     difficult for some applications to allocate the memory
     space they require on 32-bit systems.  This command, the
     SGI_UNSUPPORTED_MAP_RESERVED_RANGE syssgi(2) option, the
     MAP_SGI_ANYADDR flag to mmap(2), and the SHM_SGI_ANYADDR
     flag to shmat(2) together allow the application to control
     when the reserved range is used so applications which so
     choose may disable it.  For compatibility, the reserved
     range is normally enabled.

     The reserved range is in effect for 64-bit applications as
     well, but since the virtual address space is so large for
     these it is rarely an issue.  In general, 64-bit
     applications do not need to be concerned with these new
     flags and options to control the reserved range.

FILES
     /usr/lib/locale/locale/LC_MESSAGES/uxue
       language-specific message file [See LANG on environ(5).]

SEE ALSO
     mmap(2), shmat(2). syssgi(2).

WARNINGS
     Some existing applications are dependent on the reserved
     range being honored by the kernel.  Using this command to
     disable the reserved range for a command that requires it
     may cause it to fail, dump core, or otherwise execute
     incorrectly.  This applies to any other programs the
     command may run via fork/exec since the option is
     inherited.  Therefore, this command and the syssgi(2)
     option must be used with extreme care.  It should only be
     used with programs the user is certain are not dependent on
     the reserved range.  The effect this command may have on
     applications that are in some way dependent on the reserved
     range is strictly unsupported by SGI.
Thanks!  --Ricky

[rickyboy]

Sorry Lutz, I have to leave the office soon.  I won't be back until Monday which means that I cannot work on the port until then.



While waiting for your verdict on `MAP_ANON', I substituted `MAP_SGI_ANYADDR' for `MAP_ANON' in `nl-filesys.c' and re-maded it.  The make got through the `nl-filesys' compile and the rest of the compiles (except for `nl-sock', for which I got a one-line message of "nl-sock.c: In function 'netReceiveFrom':"; Hmm.  Expected to see more lines following the ":", but nothing more.).



Then the link phase dumped a bunch of warnings.  The first batch was

Code Select Expand

ld32: WARNING 84: /usr/lib32/libdl.so is not used for resolving any symbol.
ld32: WARNING 84: /usr/lib32/librt.so is not used for resolving any symbol.
ld32: WARNING 84: /usr/lib32/libsocket.so is not used for resolving any symbol.
ld32: WARNING 84: /usr/lib32/libnsl.so is not used for resolving any symbol.
The next batch looked like

Code Select Expand

ld32: WARNING 134: weak definition of gethostname in /usr/lib32/libnsl.so preempts that weak definition in /usr/lib32/mips3/libc.so
ld32: WARNING 134: weak definition of authnone_create in /usr/lib32/libnsl.so preempts that weak definition in /usr/lib32/mips3/libc.so
   (snip, snip)
ld32: WARNING 134: weak definition of user2netname in /usr/lib32/libnsl.so preempts that weak definition in /usr/lib32/mips3/libc.so
ld32: Giving up after printing 50 warnings.  Use -wall to print all warnings.
Then I did this:

Code Select Expand

$ ./newlisp
newLISP v.8.5.9 on Solaris, execute 'newlisp -h' for more info.

>
What next, my liege?  :-)

--Ricky

[pjot]

Hi rickyboy,



It's strange that none of the libraries seem to resolve any symbols! Since they should. But it looks promising that newLisp can be started at all; I couldn't do that after a first compile on Tru64.



Anyway, it would be interesting to run the testfile, just to see what happens:

./newlisp qa_dot

If there are any tests which fail, or codedumps, it is clear something is not right ;-)



Peter

[Lutz]

Just take out all the libraries not needed and change:



       $(CC) $(OBJS) -lm -ldl -lrt -lsocket -lnsl -o newlisp



to:



       $(CC) $(OBJS) -lm -o newlisp



probably socket functions and dlopen()/dlsym() are part of libc on this platform.



mmap() cannot be used for shared memory access on the Irix platform without file mapping. Instead use shmget() and shmat() to get a shared memory address. There is sample code for this in many places, which gives you a start. Once you have the address all other code in newLSP for 'share' can stay the same.



Lutz

[hilti]

... about 4 years later after the last post I'm asking: is there a binary available for IRIX 6.5?



I just bought a SGI Indy from eBay and would like to run Dragonfly. ;-)



Cheers

Hilti