OSDN -- Open-Source-Software-Entwicklung und Downloads
Übersetzungsstatus von Deutsch translation status for de

Foren: Entwickler (Thread #43091)
Zurück zur Thread-Liste RSS

PlayStation 2 native texture discussion (2020-10-08 07:02 by quiret #85980)

The PlayStation2 native texture is the most close-to-hardware implemented native texture format inside magic-rw. The native texture contains the following Graphics Synthesizer concepts:

- encoding of texture color data into PSMCT32/PSMCT24/PSMCT16/PSMCT16S/PSMT4/PSMT8/PSMZ32/etc
- texture buffer packing of PSMT4/PSMT8 into PSMCT16/PSMCT32
- serialization of texture uploads into GS primitives using GIFtags
- allocation of texture mipmap layers on a virtual transmission area
- fixed bitblt target formatting

The following challenges have to be solved to implement it:

- data permutation (done)
- mapping GS data formats into RenderWare formats (done)
- texture allocation parameter construction into the texture buffer/transmission region (TODO)

The last part of the list is about creating DSAX/DSAY/TBP/TBW values for all mipmaps/the CLUT so that the texture gets properly uploaded into GS memory. To get properly working examples we can extract the values of the TEX0/MIPTBP1/MIPTBP2 registers and try to find out how the original RenderWare Sky implementation worked. This is important because there are implementation quirks like the implicit BITBLT target memory layout that have to be taken into account.

Here is a list of GS data from the game "Grand Theft Auto San Andreas" on the PS2:

GS data count: 92
Compilation date: Oct 7 2020

RW Version: 3.5.0.0
MemLayout: PSMT8
Base level W: 128
Base Level H: 256

Entry number 0
Mip level 0
DSAX: 0
DSAY: 0
RRW: 64
RRH: 128
TBP: 0
TBW: 2
CLUT texture
DSAX: 0
DSAY: 128
RRW: 16
RRH: 16
TBP: 0
TBW: 0


RW Version: 3.5.0.0
MemLayout: PSMT8
Base level W: 128
Base Level H: 256

Entry number 0
Mip level 0
DSAX: 0
DSAY: 0
RRW: 64
RRH: 128
TBP: 0
TBW: 2
Mip level 1
DSAX: 0
DSAY: 128
RRW: 32
RRH: 64
TBP: 128
TBW: 2
Mip level 2
DSAX: 32
DSAY: 128
RRW: 16
RRH: 32
TBP: 144
TBW: 2
Mip level 3
DSAX: 48
DSAY: 128
RRW: 8
RRH: 16
TBP: 148
TBW: 2
Mip level 4
DSAX: 56
DSAY: 128
RRW: 8
RRH: 8
TBP: 149
TBW: 2
CLUT texture
DSAX: 48
DSAY: 176
RRW: 16
RRH: 16
TBP: 0
TBW: 0


RW Version: 3.6.0.3
MemLayout: PSMCT32
Base level W: 32
Base Level H: 32

Entry number 0
Mip level 0
DSAX: 0
DSAY: 0
RRW: 32
RRH: 32
TBP: 0
TBW: 1
CLUT texture
DSAX: 0
DSAY: 0
RRW: 0
RRH: 0
TBP: 0
TBW: 0

To get a full list of GS data visit the following link: https://osdn.net/users/quiret/pastebin/6375

What do you make out of this data? Do you see any kind of algorithm behind the data that we can use for general purpose texture memory allocation? Please share your insights so we can get the PS2 native texture to work properly!
(Letztes Update: 2020-10-08 07:06 by quiret)

Re: PlayStation 2 native texture discussion (2020-10-08 19:45 by quiret #85984)

After closely inspecting the TBW register values I made an interesting observation: a change in TBW does result in reset of DSAX/DSAY values. It seems as if a change in TBW does result in appending of a "new" texture buffer where allocation restarts. This is the case for very big PSMCT32/PSMT8/PSMT4 textures.

The TBP values appear to be set to the start of the texture buffer each time, the texture buffer of specific TBW of specific mip level that is. Suppose that there are mip levels 1 to n. The TBW of mip level i is the maximum of all mip levels i+1 to n and the CLUT (if present). It is implied that texture buffer allocations for sub mip levels are cascaded into another (by selection of proper TBP as start-point).
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-08 22:42 by quiret #85985)

You might have asked yourself why the TBW values of CLUT textures (PSMT4/PSMT8) is double the required TBW. If you have carefully consulted the specs then you know that a PSMT4/PSMT8 page is 128 pixels wide. Since the block placement is only defined on a page basis it is required to also align the TBW on a page basis. The specification of the TBW register value states that you should divide the width by 64. Thus if we have a TBW of 2 in PSMT4 then it means it spans two pages thus we divide (128+128) by 64 to obtain 4 as TBW value if combined with a PSM of PSMT4.
Reply to #85984

Re: PlayStation 2 native texture discussion (2020-10-09 00:14 by quiret #85987)

For some PSMT4 textures you may have noticed allocation at the fourth last block of a page and at other times at the very last block of the page. This behaviour appears to be tied to the texture having mipmaps or not.

If the texture has mipmaps then CLUT allocations is attempted at the very last block of the page. If the texture has no mipmaps then it is attempted at the fourth last block.
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-09 00:26 by quiret #85988)

During texture memory allocation it is necessary to pack memory of different layouts into the same buffer, especially considering the difference between PSM (main texture layout) and CPSM (CLUT layout). To accomplish that it is necessary to perform allocations on blocks (instead of rectangles). If you say allocate on block 1 a 2x2 block PSMCT32 texture you take blocks 1,3,4 and 6. But before you can allocate this on block 1 you check whether blocks 1,3,4 and 6 are empty.

Not sure whether you can perform optimizations using DSAX/DSAY to allow multiple memory layouts within the same block. That would be hardcore. The documentation does state after all that the coordinate to contribute to the calculation of the linear address.
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-09 02:37 by quiret #85989)

The GS does use 2D block coordinates for texture memory allocation. Hence each page does also have a block (x,y) coordinate. In order to calculate the page coordinate you need to have a texture buffer page width value that specifies the number of pages in the span.

page_x = ( block_x % pageblockspan )
page_y = ( block_y / pageblockspan )

The minimum value of TBPW is 1, just like in the documentation. Just that the documentation is unclear about the value if the PSM is PSMT4/PSMT8. Then you need to take my previous note into account about the double of the TBW being the real TBPW. That is a hidden quirk of the GS.
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-11 06:11 by quiret #85990)

There is an errata inside of the PS2 Sky RenderWare texture memory allocator: if the CLUT is on a TBPW >1 texture buffer, then the DSAX and DSAY are always set to 0.

For 1 block CLUT the DSAX/DSAY fields are set zero. This is rightfully done as an optimization.

The RW PS2 texture memory allocator is optimizing allocations to fill the first pages first. Once prior pages have no space for that allocation, only then does the allocator move onto next pages. This is a good performance optimization as opposed to going through the texture buffer completely linearly because the first pages are the most important for performance (claim without proof).
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-11 07:32 by quiret #85991)

As of commit https://osdn.net/projects/magic-rw/scm/svn/commits/110 the PS2 texture memory allocator is finished. It is compliant with the allocation behaviour of the RenderWare PS2 Sky engine as used in GTA:VC and GTA:SA.
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-12 20:13 by quiret #86000)

Yesterday I have performed basic tests of Magic.TXD generated PS2 native textures using the PCSX2 emulator (https://pcsx2.net/). The general consensus is: the textures do work on real hardware. Due to a fix of IMG file output regarding serialization order it is now pretty easy to modify the PS2 versions of GTA:SA/GTA:VC/GTA3.

Thus the native texture is "feature compliant with the RW Sky engine". There are still things that I want to do like GS primitive splitting for 1024x1024 texture support but this can wait until after the big unit testing this weekend. :-)
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-13 08:38 by quiret #86001)

Recently I have been thinking how the Sky native texture format does work with the Graphics Synthesizer BITBLTBUF transfer registration. As you may know GS transfers have a buffer pointer, a buffer width in pixels, a transmission x/y offset and transmission width/height. You can imagine that it is a high-performance hardware permutation engine. But transferring every texture with the maximum buffer width is inefficient because the texture stride would be very big.

So what did Crtierion do to solve this? They effectively did push new BITBLTBUF registrations before every GIFtag transfer with a changed TBW! As a side effect the DSAX/DSAY values have to be 0,0. Now it makes sense why the texture memory allocation algorithm relied on the TBPW buffers and it was important that I reversed the allocator 100% because of the resulting quirks having real impact.

https://github.com/aap/librw/blob/113d76cfaaf064447ac936f79d91ea04788fca24/src/ps2/ps2raster.cpp

By issuing same TBW values we do optimize away additional mipmap transfers which is very good. Thus TBW should be reused whenever possible and not impacting on GFX memory too heavily.

Looking at how DSAX/DSAY do spawn additional BITBLTBUF registrations, it is possible to transfer PSMCT16 CLUT using a separate upload on PSMT8 textures that use PSMCT32 upload. But it would be more efficient to swizzle the PSMCT16 CLUT into PSMCT32 and transfer that along with the mipmap data. Now that I understand that swizzling simply is the application of the memory layout permutation chained with the backwards BITBLT permutation it is not too difficult to revolutionize the way we create PS2 native textures ;)
Reply to #85980

Re: PlayStation 2 native texture discussion (2020-10-13 21:48 by quiret #86002)

The current implementation of transmission data preparation is very "patch-work" style. This is because when we prepare transmission data (lovely called "swizzling" by people in the know(?)) we assume that the page layout is the same thus we permute data locally inside of blocks/columns. But true transmission data preparation should permute data on a page-level. This is because the BITBLT memory layout could use a different block-to-page permutation than the PSM register value, thus data would be scaterred about if read as PSM (can be beautifullly seen if you specify PSMCT16 PSM for RASTER_1555 by accident without applying a reverse page permutation).

RenderWare Sky (3.6.0.3) does assign following BITBLT memory layouts to raster formats:

- PAL8: PSMCT32 (because same page layout as PSMT8)
- PAL4: PSMCT16 (because same page layout as PSMT4)
- RASTER_8888: PSMCT32
- RASTER_888: PSMCT24
- RASTER_1555: PSMCT16S (note the S, this is not PSMCT16!)

We need to create a new transmission data preparation function that takes the PSM, current BITBLT layout and target BITBLT layout as arguments. When first initiating data to the PS2 native texture PSM == current BITBLT layout. When exiting we set target BITBLT layout == PSM to transform into linear format for basic transmission again (warm-up permutation buffer creation by target index peeking, BITBLT layout swapping support by just applying reverse current BITBLT chained with target BITBLT, etc). If we implement this new transmission data management then we can support a 100% of textures that the RenderWare Sky engine does accept but it's standard generator did not output.

Postponed for Magic.TXD version 1.3.
Reply to #85980