The mystery of my memory's missing megabytes

[oblivious_maximus]

my setup:
Asus M3A (0803 bios)
Athlon X2 4400+
4GB (2x2GB kit) A-Data ddr2 800
Asus 8600GT w/512MB ddr3
PCI Promise IDE/SATA controller card (1 PATA drive attached)
2 other PATA drives, 1 SATA dvd-rw
Debian Lenny amd64 w/ kernels:

• 2.6.24-1-amd64(stock from repo),
• 2.6.24-4-amd64(compiled myself w/NUMA),
• 2.6.24-5-amd64(compiled myself w/ and w/o NUMA) - NUMA doesn't appear to be the culprit

my problem:
BIOS reports 4096MB memory, software does not. might be why I can't use the HD3650 I bought(major lockups on KDE logout & when starting additional Xservers).

hwinfo thinks there's 256MB missing, but it's only 136MB according to /proc/meminfo. Presumably whatever's wrong is also why /proc/mtrr doesn't account for 768MB(I don't quite undertand the output of /proc/mtrr though), which another member's comment about originally got me wondering about this.

from right now using the 2.6.24-5-amd64 w/o NUMA:

$free -m -o
             total       used       free     shared    buffers     cached
Mem:          3954       2723       1231          0          1       2316
Swap:          509          0        509

hwinfo:
12: None 00.0: 10102 Main Memory
  [Created at memory.61]
  Unique ID: rdCR.CxwsZFjVASF
  Hardware Class: memory
  Model: "Main Memory"
  Memory Range: 0x00000000-0xf72bbfff (rw)
  Memory Size: 3 GB + 768 MB
  Config Status: cfg=new, avail=yes, need=no, active=unknown


/proc/meminfo:
MemTotal:      4049648 kB (/1024=3954.734375)
MemFree:       1223632 kB
Buffers:          1996 kB
Cached:        2389092 kB
SwapCached:          0 kB
Active:         617396 kB
Inactive:      2114508 kB
SwapTotal:      522072 kB
SwapFree:       522072 kB
Dirty:              56 kB
Writeback:           0 kB
AnonPages:      340816 kB
Mapped:          85096 kB
Slab:            42336 kB
SReclaimable:    24776 kB
SUnreclaim:      17560 kB
PageTables:      12940 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   2546896 kB
Committed_AS:   643764 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     42892 kB
VmallocChunk: 34359694331 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

/proc/mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1

with NUMA hwinfo stays the same, but meminfo has 6 more megs listed - from yesterday using the stock Debian 2.6.24-1-amd64 w/NUMA:

hwinfo:
12: None 00.0: 10102 Main Memory
  [Created at memory.61]
  Unique ID: rdCR.CxwsZFjVASF
  Hardware Class: memory
  Model: "Main Memory"
  Memory Range: 0x00000000-0xf7898fff (rw)
  Memory Size: 3 GB + 768 MB
  Config Status: cfg=new, avail=yes, need=no, active=unknown

meminfo:
MemTotal:      4055652 kB (/1024=3960.59765625)
MemFree:       3988832 kB
Buffers:           580 kB
Cached:          21984 kB
SwapCached:          0 kB
Active:          10784 kB
Inactive:        14048 kB
SwapTotal:      522072 kB
SwapFree:       522072 kB
Dirty:              12 kB
Writeback:           0 kB
AnonPages:        2396 kB
Mapped:           1840 kB
Slab:            10980 kB
SReclaimable:     3544 kB
SUnreclaim:       7436 kB
PageTables:        384 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   2549896 kB
Committed_AS:     3808 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     26160 kB
VmallocChunk: 34359711735 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

mtrr:
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1

and with just 2GB installed, I think with the 2.6.24-4 w/NUMA kernel:

hwinfo:
12: None 00.0: 10102 Main Memory
  [Created at memory.61]
  Unique ID: rdCR.CxwsZFjVASF
  Hardware Class: memory
  Model: "Main Memory"
  Memory Range: 0x00000000-0x7ffaffff (rw)
  Memory Size: 2 GB
  Config Status: cfg=new, avail=yes, need=no, active=unknown

mtrr:
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1

meminfo:
MemTotal:      2063556 kB (/1024=2015.19140625)
MemFree:       2005980 kB
Buffers:           580 kB
Cached:          21080 kB
SwapCached:          0 kB
Active:          12236 kB
Inactive:        11916 kB
SwapTotal:      522072 kB
SwapFree:       522072 kB
Dirty:               8 kB
Writeback:           0 kB
AnonPages:        2616 kB
Mapped:           1892 kB
Slab:            11352 kB
SReclaimable:     3832 kB
SUnreclaim:       7520 kB
PageTables:        432 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   1553848 kB
Committed_AS:     4384 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     18460 kB
VmallocChunk: 34359719415 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

here's couple tidbits from dmesg that look suspicious but that I haven't a clue what to do about (or if I even need to):
No AGP bridge found*
Your BIOS doesn't leave a aperture memory hole*
Please enable the IOMMU option in the BIOS setup**
This costs you 64 MB of RAM***
Mapping aperture over 65536 KB of RAM @ 8000000****
Memory: 4043148k/4980736k available (2093k kernel code, 149884k reserved, 974k data, 216k init)*****

*There's no AGP port/bus on my motherboard.
**There's no IOMMU option that I've seen in my BIOS
***So does that mean 64MB off the total listed as present? Or 64MB added to the total used?
****The 64MB from the last line?
*****3948.38671875 MB available out of 4864 MB? The listed used memory adds up to 153167k (kB?) or 149.5771484375 MB none of which adds up to what I'm missing according the sources listed above.

(2093k kernel code, 149884k reserved, 974k data, 216k init) -- 149MB is close to what I'm missing, is the answer as simple as that? Still it doesn't add up exactly ...

Is there an explanation for where those 130-150 ish MB are being used? And for what's up with the mtrr? What can/should I do? Any advice will be extremely appreciated.

edit: and where's the extra 768MB of memory dmesg reports as available coming from? my swap is only 509 MB.
I tried to post a reply with dmesg's output but it's too long and the board won't allow me to.

[mlau]

Does the BIOS offer an option to remap parts of memory beyond the 4GB line?
The last GB is usually used to map peripheral devices and by the kernel for internal purposes;
any actual RAM there is lost unless BIOS/Chipset are able to remap parts of it.

[oblivious_maximus]

The BIOS version depicted in the manual has an option called "Memory Hole Remapping" but the current BIOS version (which I'm using) does not. The closest thing is "DCT Unganged Mode", which is enabled. When I tried disabling it, even more memory disappeared - about 3100MB is all that showed up with free then.

[Moustacha]

I believe it has something to do with the kernel reserving memory for itself. I got 4GB running 64-bit Gutsy and get around what you're getting, 3954. With 1GB you'll notice you're missing memory, giving you about 1011MB.

[oblivious_maximus]

that's about what my fileserver with an Athlon XP and 1GB of ram shows (1012mb). I was happily going along thinking that my desktop's memory discrepancy was similar, but hwinfo's output (3GB + 768MB) really seems to indicate it's something else...

Does your mtrr account for all 4GB of your memory Moustacha?

edit: I borrowed a 1GB dimm for a few mintues to see if hwinfo would report all 5GB. It did, but I haven't the foggiest what that means given what it shows with 4GB. Free and meminfo are more in line with the roughly-10MB-per-GB-not-visible 'norm', but the mtrr looks the same.
so with 5GB:

free:
             total       used       free     shared    buffers     cached
Mem:          4958         55       4902          0          0         20
Swap:          509          0        509

hwinfo:
12: None 00.0: 10102 Main Memory
  [Created at memory.61]
  Unique ID: rdCR.CxwsZFjVASF
  Hardware Class: memory
  Model: "Main Memory"
  Memory Range: 0x00000000-0x135eb4fff (rw)
  Memory Size: 5 GB
  Config Status: cfg=new, avail=yes, need=no, active=unknown

meminfo:
MemTotal:      5077716 kB (/1024=4958.70703125)
MemFree:       5020340 kB
Buffers:           548 kB
Cached:          20812 kB
SwapCached:          0 kB
Active:          14276 kB
Inactive:        12132 kB
SwapTotal:      522072 kB
SwapFree:       522072 kB
Dirty:               4 kB
Writeback:           0 kB
AnonPages:        5196 kB
Mapped:           2228 kB
Slab:             9132 kB
SReclaimable:     3664 kB
SUnreclaim:       5468 kB
PageTables:        564 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3060928 kB
Committed_AS:     7668 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      9724 kB
VmallocChunk: 34359728119 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

mtrr:
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1

in removing the memory remapping option from the BIOS could Asus have implemented some kind of if/then for that option depending on if you have up to 4GB or >4GB of memory installed?

I guess I should have tried installing the HD3650 while I had 5GB of RAM in there and seen if I could log out without locking up everything, but I'm growing very weary of plugging that card in only to plug in the 8600 again after a couple hours of fruitless hoop jumping. *sigh*

[oblivious_maximus]

Chalk it up to 64bit oddities I guess.

After my first 3 months with a 64bit CPU (and a 64bit OS), I decided it was time to see if the grass was greener on i386. So far it is. With a 'bigmem' kernel, I'm seeing what I'd expect in free (4053MB, about 10MB/gig not listed (still wondering why that is though)), but the mtrr still seems completely wrong. Only I can't find enough/any straightforward-for-dummies information on whether what I'm seeing in /proc/mtrr is actually completely wrong, or even if I should care at all. Presumably the mechanism for allowing the OS to control how memory is cached (if my understanding is accurate) should know about all the memory in a system, not just most of it, should it not?

If anyone can point me to some decent info on the mtrr in Linux and making sure that it's set up properly, or some info that explains even a little why mine doesn't cover all my ram, I would really appreciate it.

here's all that same info I've been posting, from my current install, maybe it'll be useful...

free -m -o
             total       used       free     shared    buffers     cached
Mem:          4053       3895        158          0          0       3638
Swap:          509          0        509

meminfo
MemTotal:      4150800 kB
MemFree:        162012 kB
Buffers:            36 kB
Cached:        3725796 kB
SwapCached:          0 kB
Active:         556940 kB
Inactive:      3356936 kB
HighTotal:     3276480 kB
HighFree:        11852 kB
LowTotal:       874320 kB
LowFree:        150160 kB
SwapTotal:      522072 kB
SwapFree:       522072 kB
Dirty:             200 kB
Writeback:           0 kB
AnonPages:      188044 kB
Mapped:          80160 kB
Slab:            47500 kB
SReclaimable:    32636 kB
SUnreclaim:      14864 kB
PageTables:       2880 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   2597472 kB
Committed_AS:   470572 kB
VmallocTotal:   118776 kB
VmallocUsed:     52460 kB
VmallocChunk:    56820 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1

hwinfo
12: None 00.0: 10102 Main Memory
  [Created at memory.61]
  Unique ID: rdCR.CxwsZFjVASF
  Hardware Class: memory
  Model: "Main Memory"
  Memory Range: 0x00000000-0xfd583fff (rw)
  Memory Size: 4 GB
  Config Status: cfg=new, avail=yes, need=no, active=unknown

I guess if I plug in the HD3650 and it works without locking up my whole system really hard then I won't be so bothered by the mtrr discrepancy. All the same I'd love to read an explanation somewhere.

[Moustacha]

hwinfo says 3GB+768MB to me as well.

free -mo
             total       used       free     shared    buffers     cached
Mem:          3949       3921         28          0       1518       1808
Swap:          956         41        914

cat /proc/meminfo 
MemTotal:      4044512 kB
MemFree:         30732 kB
Buffers:       1553352 kB
Cached:        1850676 kB
SwapCached:       1608 kB
Active:         614196 kB
Inactive:      3257476 kB
SwapTotal:      979956 kB
SwapFree:       936956 kB
Dirty:             348 kB
Writeback:           0 kB
AnonPages:      466112 kB
Mapped:          95916 kB
Slab:            92468 kB
SReclaimable:    63096 kB
SUnreclaim:      29372 kB
PageTables:      17000 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3002212 kB
Committed_AS:  1007304 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     18712 kB
VmallocChunk: 34359719639 kB

 cat /proc/mtrr 
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size= 512MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0x140000000 (5120MB), size= 512MB: write-back, count=1

I've only got 4GB Main Memory, 512MB video card memory. I haven't been bothered checking this out before.