few questions...

[sylware]

from evergreen hardware:

• acronyms:
  
  • HDP? Something dealing with the memory controller, tiling stuff?
  • .rlc in the firmware name? rlc is related to interrupts.
  • MC_VM_MD_* MC = Memory Controller VM = Virtual Memory MD = ?
  • MC_VM_MB_* MC = Memory Controller VM = Virtual Memory MB = ?
  
  
• What is the mapping of the BAR0 aperture? Programming the MC/VM/PCI-E won't affect it? If BAR0 is hardcoded to VRAM, the first 256kB are for VGA, can we reclaim it? If yes, how to do it in a safe way?
  
• What is the max size of a page table for the PCI-E DMA aperture (GTT)?
  
• What is the use of page entry attribute SYSTEM and SNOOPED (vga?)?
  
• In register VM_CTX0_PROTECTION_FAULT_DEFAULT_ADDR, we write the gpu page index of the dummy page bus address (<< 12). That would mean linux must allocate bus addresses based on gpu page size alignment. Is linux kind of told about this thanks to PCI-E cache line size? That would be the same for PTEs. Bus addresses must be aligned on gpu page size. I know it works since GPU page size and x86-64 base page size are both 4kB but...
  
• Must AGP be disabled on PCI-E boards?

[agd5f]

from evergreen hardware:[*]acronyms:[*]HDP? Something dealing with the memory controller, tiling stuff?

HDP = Host Data Path. It's the part of the chip that deals with access to vram via the PCI framebuffer BAR.

[*].rlc in the firmware name? rlc is related to interrupts.

The RLC is the interrupt controller firmware. I don't recall what RLC stands for off hand.

[*]MC_VM_MD_* MC = Memory Controller VM = Virtual Memory MD = ?[*]MC_VM_MB_* MC = Memory Controller VM = Virtual Memory MB = ?

MC = Memory Controller, VM = Virtual Memory. MD and MB are just internal MC blocks. I don't think they have any particular significance.

[*]What is the mapping of the BAR0 aperture? Programming the MC/VM/PCI-E won't affect it? If BAR0 is hardcoded to VRAM, the first 256kB are for VGA, can we reclaim it? If yes, how to do it in a safe way?

The framebuffer BAR maps to the GPU address HDP_NONSURFACE_BASE is pointed to. You can disable the VGA engine using VGA_RENDER_CONTROL. See rv515_vga_render_disable().

[*]What is the max size of a page table for the PCI-E DMA aperture (GTT)?

The size of the GTT aperture is determined by VM_CONTEXT0_PAGE_TABLE_START_ADDR and VM_CONTEXT0_PAGE_TABLE_END_ADDR. So 2^24 4k pages.

[*]What is the use of page entry attribute SYSTEM and SNOOPED (vga?)?

System is used internally in the MC to designate GTT memory vs. VRAM and the snoop bit determines whether the GTT page is accessed via snooped (cache coherent) or unsnooped (uncached) transaction. Snooped is useful for pages that the CPU will need to read, unsnooped performs better if you are only writing to the pages with the CPU.

[*]In register VM_CTX0_PROTECTION_FAULT_DEFAULT_ADDR, we write the gpu page index of the dummy page bus address (<< 12). That would mean linux must allocate bus addresses based on gpu page size alignment. Is linux kind of told about this thanks to PCI-E cache line size? That would be the same for PTEs. Bus addresses must be aligned on gpu page size. I know it works since GPU page size and x86-64 base page size are both 4kB but...

It could be a problem for <4k pages, but >4k pages are fine as you can just bind multiple 4k GPU pages per larger sys page.

[*]Must AGP be disabled on PCI-E boards?

No. the GPU's AGP aperture just points to the bus address of the translated region (provided by the AGP bridge or the CPU's GART/IOMMU). In theory you could use it in conjunction with the onboard gart.

[sylware]

Thanks!

HDP:

• there are HDP_MEM_COHERENCY_FLUSH_CNTL and HDP_REG_COHERENCY_FLUSH_CNTL registers. Does it mean the HDP block deals with the register PCI BAR too?
• there are 32 blocks in registers space from 0x2c14, fifos?
• If I understand well, the HDP block manages all trafic between the PCI-E bus and the MC and Xrbm (X=G or S). The HDP translates PCI-E bar 0 transactions for the MC, and translate the PCI-E bar 2 transaction for the Xrbm (X=G or S). The MC VM translates GPU transactions for the HDP (which then go en the PCI-E bus).

vram:

We need to tell the MC over which GPU address range we map the vram. I see the driver uses the following registers:
MC_VM_SYS_APERTURE_LOW_ADDR, MC_VM_SYS_APERTURE_HIGH_ADDR,
MC_VM_SYS_APERTURE_DEFAULT_ADDR and MC_VM_FB_LOCATION.
What is the role of each of those registers in mapping the vram in GPU address space?

[agd5f]

Thanks!

HDP:

• there are HDP_MEM_COHERENCY_FLUSH_CNTL and HDP_REG_COHERENCY_FLUSH_CNTL registers. Does it mean the HDP block deals with the register PCI BAR too?
• there are 32 blocks in registers space from 0x2c14, fifos?
• If I understand well, the HDP block manages all trafic between the PCI-E bus and the MC and Xrbm (X=G or S). The HDP translates PCI-E bar 0 transactions for the MC, and translate the PCI-E bar 2 transaction for the Xrbm (X=G or S). The MC VM translates GPU transactions for the HDP (which then go en the PCI-E bus).

The HDP block handles CPU access to GPU memory via the PCI BAR. It doesn't affect the MMIO register BAR. HDP_MEM_COHERENCY_FLUSH_CNTL is for flushing the HDP caches after writes to the PCI aperture. HDP_MEM_COHERENCY_FLUSH_CNTL is for flushing the HDP caches after HDP register changes. The HDP block is just an MC client like any other block (textures, CBs, etc.). It just forwards requests from the PCI BAR to the MC.

vram:

We need to tell the MC over which GPU address range we map the vram. I see the driver uses the following registers:
MC_VM_SYS_APERTURE_LOW_ADDR, MC_VM_SYS_APERTURE_HIGH_ADDR,
MC_VM_SYS_APERTURE_DEFAULT_ADDR and MC_VM_FB_LOCATION.
What is the role of each of those registers in mapping the vram in GPU address space?

MC_VM_FB_LOCATION defines the location of VRAM in the GPU's address space. The MC defines 2 internal apertures SYS and non-SYS. One goes through page table and the other is forwarded directly to the requested GPU address. MC_VM_SYS_APERTURE_LOW_ADDR and MC_VM_SYS_APERTURE_HIGH_ADDR define the address range of the SYS aperture. The driver sets the SYS aperture to point to VRAM (and AGP on AGP systems, they are mapped contiguously). So when an address comes into the MC, it's checked against the SYS aperture. If it's inside, it's forwarded to VRAM or AGP. If it's outside the SYS aperture, it's forwarded to the GPU's page table (GART). If the page is not in the page table, it goes to the default address defined by VM_CONTEXT0_PROTECTION_FAULT_DEFAULT_ADDR. MC_VM_SYS_APERTURE_DEFAULT_ADDR is the default address for unmapped pages in the SYS aperture, but it's unused since we set the SYS aperture to the exact size of the vram and AGP apertures.

[sylware]

Thanks!!! I think I start to have a good grasp of that part of the GPU.

• If I understand well, there is a "virtual/internal" page table for the system aperture of the memory controller. This "virtual/internal" page table is "built" with the range of MC_VM_FB_LOCATION (plus AGP aperture for AGP cards)?
• In case, MC_VM_FB_LOCATION(vram) and AGP aperture do not cover the defined system aperture then MC_VM_SYS_APERTURE_DEFAULT_ADDR is used... then better having vram or AGP mapped to GPU address MC_VM_SYS_APERTURE_DEFAULT_ADDR?
• If the request is for the non-system aperture, it is forwarded to the block which deals with CTX0 and CTX1 for page translation. I noticed the driver sets the SYSTEM attribute of the page table entries. Based on what was said, that attribute is ignored and can be disabled?
• In gtt/gart flushing, first the HDP caches are flushed using HDP_MEM_COHERENCY_FLUSH_CTL. It means:
  • To be sure that if the CPU did changes to the page table in vram, the changes are applied before flushing CTX0 using the VM_CONTEXT0_REQUEST_RESPONSE register?
  • or the VM CTXs are clients of the HDP to access the PCI-E bus?(means VM CTXs have direct access to the PCI-E transaction block?)
• What are the 32 HDP register blocks starting from 0x2c14 on evergreen?
• Why HDP_NONSURFACE_SIZE is set to 1GB since we have only 256MB for bar0 aperture?

[sylware]

I did notice in nid.c the reference to the VM context 1 identity access modes.

It is how the system aperture would work if context 0 was configured by defaut using the mode 3. Is this the case? If yes, where do we configure the identity access mode of VM contexts? And that would mean the HDP is using VM context 0 (and would explain we cannot modify the VM context 0 identity access mode)?

For evergreen, are the hardware contexts of the GPU (cfg->max_hw_contexts) actually the VM contexts?